Thread: [Docstring-develop] dps_visit.py - first pass at XML output

As always, they're at:

   	http://www.tibsnjoan.co.uk/reST/dps_visit.py
   	http://www.tibsnjoan.co.uk/reST/test.py

The addition is an "xml" command line switch, which causes a (rather
nasty) emission of XML to the standard output. No attempt to process the
docstrings yet. No attempt to think about a nice schema for the XML,
either.

Note that this module now depends on the DPS module, since it imports
(its of) it. I fully intend to produce a separate module at some stage
which is useful *just* as an example of mining the relevant information
out of the AST - that should simply be a matter of stripping stuff out.

NB: Jeremy - do we believe the compiler module to work under 2.2a1? I
had a problem last night that I avoided by using an earlier version of
it - sorry, I don't have the exception stack to hand, and unfortunately
I don't have internet connectivity at home at the moment, so can't
download 2.2a2 (which, for all I know, may fix it). I'll try to research
more tonight, in case this is something not known about.

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
Give a pedant an inch and they'll take 25.4mm
(once they've established you're talking a post-1959 inch, of course)
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

[Neal, Jeremy - do you still want me to CC you, or shall I drop back to
just the docstring-develops list? I'll take no answer as meaning make no
change...]

Had fun this weekend getting the quick-and-dirty HTML output started. In
the end, I've stopped faffing around with DOM trees (all say "yeah!")
and am using the dps/nodes.py tree (DPS nodes tree?) directly - it's
pleasantly simple to use.

As always, latest version is at:

    	http://www.tibsnjoan.co.uk/reST/pydps.tgz

As you might expect (!) the colours are garish - this is my oh-so-subtle
way of indicating that this is a PROTOTYPE.

Still to do:

* add some more functions to the html.py, so it can cope
  with more stuff

* expand nodes.py to transfer more of my initial data
  into the DPS nodes tree.

* expand visit.py to work properly (!) - various things
  need doing - see the file.

* deal with cross-references.

* add more documentation in various places.

* extract the command line stuff to its own file, make
  the directory a proper package, etc., etc.

* integrate as a basic DPS Formatter, plus extra subclass
  to handle the Python stuff.

As you can see, there's still plenty of fun in store (I estimate at
least another two major refactorings, for a start).

For what its worth, I'm finding the DPS (and also the compiler package)
good fun to play with - it's nice to leave someone else to handle the
parsing!

Tibs
--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
"Bounce with the bunny. Strut with the duck.
 Spin with the chickens now - CLUCK CLUCK CLUCK!"
BARNYARD DANCE! by Sandra Boynton
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

Ho hum, bla bla,

     	http://www.tibsnjoan.co.uk/reST/pydps.tgz

I finally fixed the stupid bugs I'd left in sorting out the left
alignment of docstrings (before they get handed to DPS). That's what I
get for programming before bedtime.

The "dumb" HTML formatter is now working, and is split into a Writer
class (which knows about some of the DPS nodes) and a PythonWriter
subclass (which knows about some of the Python "outer level"
infrastructure).

It handles enough of reST to be usefully run on its own modules
(packages aren't supported yet, I'm afraid) and get a reasonably
coherent result.

I seriously like the paradigm of "instantiate a Writer class once and
then call it as needed" - this allows good encapsulation of the baggage
one wants to define only once for a multiplicity of documents, and also
allows one to reduce the number of arguments to the "write" call without
needing scads of optional arguments. I still remember how neat I
realised the __call__ method was when I finally internalised its use.

Speaking of which (hmm - this is probably a Doc-SIG question, but
still), which "special" (i.e., __xxx__) methods do we want to *normally*
report on with DPS? Obviously __init__ and __call__ are extra special,
since their arguments matter. (of course, *removing* things from the
output is unlikely to happen until a long way down the line - so it's
not a decision that's needed Real Soon Now).

Next thing is to transfer more information across from visit.py (where I
remember the information derived from the AST) to nodes.py (where I
place it into a DPS nodes tree) and thus make it available to html.py
(for presentation).

[of course, the school year has now started again, so mornings are now
more rushed getting Michael to school, and evenings have less time, so
Glacial Software Development is back in action, I'm afraid...]

Tibs

(ps: thanks for the nightly tarballs - that makes my life a LOT neater -
a couple of times I've noticed a bug, and magically had it go away the
next day - great stuff!)

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
You said "run as root" and "securely" in the same sentence relating to
CGI. You're funny! -- Ignacio Vazquez-Abrams, on the Python list
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

And again,

      	http://www.tibsnjoan.co.uk/reST/pydps.tgz

Well, I should have been working, but instead I've added rudimentary
package support to pydps. The above contains a new directory within the
pydps directory, html, which contains two example outputs, produced by
cd'ing to the pydps directory and doing::

    python pydps.py html html.py > html/html.html
    python pydps.py html ..\pydps > html/pydps-package.html

(yes, I'm afraid it was on Windows NT).

Since "verbose" is enable by default, if you try those you'll get *lost*
of stuff printed to stderr - never mind.

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
Which is safer, driving or cycling?
Cycling - it's harder to kill people with a bike...
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

It's still

      	http://www.tibsnjoan.co.uk/reST/pydps.tgz

I've started adding support for function signatures. This has meant
working on producing representations for the RHS of assignments, which
is partially completed (it does, for instance, list comprehensions, but
not basic arithmetic). See my other message on the Doc-SIG as such today
(Re: [Doc-SIG] DPS DTDs) for an example of what it handles.

Jeremy - this means that I'm getting a *much* better understanding of
some of the nodes in the compiler tree. I realise that the table in

	http://www.python.org/~jeremy/compiler/module-compiler.ast.html

is generated from LaTeX source, presumably that at


http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/To
ols/compiler/doc/asttable.tex

(well, when I browse it). I'm willing to update that, in whatever form
is useful, with more information about what each "entry" is. What's the
best way for me to do that, so far as you're concerned? (taking a copy
of the LaTeX and working on it would be OK by me).

Two other questions:

1. Is there a possibility that the compiler module might get shifted
from Tools into the standard library? It seems rather too useful to be
stuck "in the hinterlands".

2. Should the various representation functions I'm working on eventually
be merged in as methods on the compiler nodes? I'm a bit chary of having
code outwith the compiler module that has to know details about a good
number of the nodes inside (e.g., the problems if a new class gets
added). On the other hand, cruft that (maybe) no-one else would want
would be a bad thing...

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
You said "run as root" and "securely" in the same sentence relating to
CGI. You're funny! -- Ignacio Vazquez-Abrams, on the Python list
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

>>>>> "TJI" == Tibs  <Tony> writes:

  TJI> Jeremy - this means that I'm getting a *much* better
  TJI> understanding of some of the nodes in the compiler tree. I
  TJI> realise that the table in

  TJI> 	http://www.python.org/~jeremy/compiler/module-compiler.ast.html

  TJI> is generated from LaTeX source, presumably that at

  TJI> http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/To
  TJI> ols/compiler/doc/asttable.tex

That's right. 

  TJI> (well, when I browse it). I'm willing to update that, in
  TJI> whatever form is useful, with more information about what each
  TJI> "entry" is. What's the best way for me to do that, so far as
  TJI> you're concerned? (taking a copy of the LaTeX and working on it
  TJI> would be OK by me).

The LaTeX source is the right place to make changes.  If you've got
updates, it's probably best to use the patch manager to submit
changes.

  TJI> Two other questions:

  TJI> 1. Is there a possibility that the compiler module might get
  TJI> shifted from Tools into the standard library? It seems rather
  TJI> too useful to be stuck "in the hinterlands".

I think it's primarily a question of user demand.  If people are using
it enough, it might move.

  TJI> 2. Should the various representation functions I'm working on
  TJI> eventually be merged in as methods on the compiler nodes? I'm a
  TJI> bit chary of having code outwith the compiler module that has
  TJI> to know details about a good number of the nodes inside (e.g.,
  TJI> the problems if a new class gets added). On the other hand,
  TJI> cruft that (maybe) no-one else would want would be a bad
  TJI> thing...

I think you can count on the organization of the ast classes staying
basically the same.  The names of the classes and the attributes of
their instances are a crucial part of the API.  Is that sufficient?

Jeremy

Jeremy Hylton responded:
> >>>>> "TJI" == Tibs  <Tony> writes:

s'OK - I answer indiscriminately to my usename and my bornname.

> The LaTeX source is the right place to make changes.

OK. I'll try to have a go "at some stage".

> If you've got updates, it's probably best to use the patch
> manager to submit changes.

Oh dear - another new thing to learn. Ah well.

>   TJI> 1. Is there a possibility that the compiler module might get
>   TJI> shifted from Tools into the standard library? It seems rather
>   TJI> too useful to be stuck "in the hinterlands".
>
> I think it's primarily a question of user demand.  If people are using
> it enough, it might move.

My thinking was more that it is going to be fundamental to pydps (or
whatever it gets called), and it's a pain if we have to get people to
install something else (which they might not have around - Tools doesn't
come with the Windows installer, I think).

>   TJI> 2. Should the various representation functions I'm working on
>   TJI> eventually be merged in as methods on the compiler nodes?
>   ...deletia...

> I think you can count on the organization of the ast classes staying
> basically the same.  The names of the classes and the attributes of
> their instances are a crucial part of the API.  Is that sufficient?

Hmm - maybe. I'll try to work up a "proper" description of anything I
think might benefit the innards of the compiler module as such, if I do
decide it isn't.

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
Well we're safe now....thank God we're in a bowling alley.
- Big Bob (J.T. Walsh) in "Pleasantville"
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

As ever:

	http://www.tibsnjoan.co.uk/reST/pydps.tgz

This lunchtime's work is to change how names are handled, such that the
HTML can report on what a name is *set to*. This is (obviously!) limited
in usefulness, and can only be done if the name is assigned to exactly
once at the particular level it "belongs to". Amendment by methods and
functions will, of course, not be seen.

Look at html/html.py for an example of what this looks like.

It *can* be misleading, as if one does::

    a = 1
    b = a
    a = 2
    c = a

you will get "a" reported with no value (since it has more than one
assignment), but "b" and "c" both reported as being assigned "a",
despite the fact their values are different. It may be that more of the
reports need suppressing.

On the other hand, if one is doing::

    def __init__(self):
        self.postfix = 0

then being told that "self.postfix = 0" is quite useful...

(and yes, I *know* I'm reporting names in places that DPS doesn't want
me to, but information suppression isn't coded yet)

(note for anyone reading the code (yuck) that the *old* method is still
in there as well, and also that the *gathering* of this information is,
erm, rather inefficient since it's done by yet another scan of the AST)

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
.. "equal" really means "in some sense the same, but maybe not
.. the sense you were hoping for", or, more succinctly, "is
.. confused with". (Gordon McMillan, Python list, Apr 1998)
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)