#@+leo-ver=5-thin
#@+node:ekr.20031218072017.329: * @file ../doc/leoNotes.txt
#@@nocolor

#@+all
#@+node:ekr.20090202064534.4: **  Your mission, should you choose to accept it
@language rest

#@+node:ekr.20100223100750.5843: *3* Original post by Robin Dunn
@nocolor-node

Here are the emacs features that I use very often that any editor would
need to have in order for me to switch.  I've seen some editors with
some of these, but none with all unless it is an emacs clone.  I'll
leave out the obvious things like platform independence, good syntax
highlighting, calltips or auto-completion.  Also, these features are
just dealing with the code editor portion of the app, if it is more than
that (like a full IDE) then some of these things may or may not apply to
the non code editor parts:

* (done) Python should be just one of the languages that this editor supports,
not the primary target.  I spend as much time in C/C++ as I do Python,
and my editor of choice needs to help me with C/C++ coding just as much
as it does with Python.  So some sort of support for calltips and
auto-completion would be marvelous, and also being able to act as a
front-end for gdb since I currently use emacs for that most of the time.

* (done) Absolutely every feature or action must be able to be done with just
the keyboard.  Moving the hand back and forth to the mouse wastes time,
breaks concentration and contributes to RSI.  Multi-key sequences are
fine as long as they are grouped in a logical fashion.  For example in
emacs all of the version control features are accessible via the
Ctrl-x,v sequence plus one more letter.

* (done) Incremental search, both forward and reverse, and wrapping around
after you've reached the end or the beginning of the document.  I like
to have the same key to start the search and also do a search-next after
you've typed all the characters you are searching for, and also to have
backspace go back one search position and/or remove one character from
the search text.

* (done) Multiple top level windows, and able to show any buffer in any TLW,
including those that are already displayed in another TLW.  Of course
there should be key-bindings available for opening a new TLW, cycling
forward and backward through the buffer list, and a way to select a
buffer from a popup list of buffer/file names.

* (to be improved) The Kill-Ring.  For those of you that have never used an emacs-like
editor it works like this:  There is a collection of the N previous
blocks of text that have been cut or copied (in emacs 'cut' == 'kill'
more or less)  When I do a yank (paste) it uses the last thing put in
the kill-ring.  If I then immediately use another key-binding then it
replaces that pasted text with the next item in the kill ring, and so on
until I eventually wrap around get back to the first one in the ring, or
I do some other command or move the cursor somewhere else.

* (done) Registers.  A text snippet can be copied into a register, which is
like the kill ring except you refer to each one by name, where the names
are 'a' through 'z'.  You can also append to a register that already has
text in it, and you can paste the contents of a register into the
document at the current cursor location.

* (done) Able to have selections be either a stream of characters or a
rectangle.  A stream selection is like what you have in all text
editors, it starts from position a on line N and continues forward or
back to position b on line M and includes all the characters in between.

  A rectangle selection is all the characters between position a and b
on lines N to M.  In other words, it has width and height and it might
be something like positions 5 through 10 on lines 20 to 25.  Cutting or
deleting a rectangle removes the text in the rectangle and shifts any
text to the right of the rectangle over.  It does not remove any lines
although they may end up being empty.  Pasting a rectangle inserts the
new text with the upper-left of the rectangle at the current cursor
position, shifts existing text to the right if needed, and fills with
spaces on the left if a line affected by the paste is not long enough.
New lines are not added unless the file needs to be extended to
accommodate the rectangle paste.  Rectangles can also be put into registers.

* (to be improved) Good keystroke macro recording and the ability to save and load
keystroke macros, and the ability to assign a key-binding to a saved
recorded macro. Any time I need to make the same edits to a bunch of
lines or groups of lines I'll record doing it on the first one including
the keystrokes needed to reposition for the next line, and then stop
recording and then it's just one keystroke to replay the keystrokes for
every other line that needs it done.  I record, use and throw away up to
a dozen or so macros per day.

* (done, and better than asserted) If you must have a toolbar make it optional
and keep it simple. Toolbars require the mouse and the goal is to keep the hand
off the mouse as much as possible.

* (done) Similarly, avoid using popup dialogs whenever possible.  This includes
things like the file dialog.  I don't mind seeing the file dialog if I
select a menu item, because most likely my hand is already on the mouse,
but the rest of the time I just want to hit a key, type a path name
(with tab-completion to help find stuff, up/down keys to cycle through
past selections) and press enter.  So I would prefer this editor to have
something like emacs' minibuffer, or the QuickFind panel in Firefox.  In
other words, when there is something you would normally use a dialog for
just create a small panel that rolls up from the bottom of the frame,
put the keyboard focus there, perhaps do stuff in the main buffer as
they are typing if appropriate, and then when the user is done the panel
rolls out of sight again and keyboard focus is restored to their active
buffer.  This can be done for file open/saves, search & replace,
specifying build or grep commands (see next item) choosing to execute
some editor function by name that may not have some key-binding yet (see
item after next) etc.

* (done, with user @commands)

Flexible build/grep commands.  Emacs handles both of these in almost
the same way so I'll list them together here.  I hit a key and am
presented with either the default, or the most recently used compile or
grep command.  I can edit the command or use the up/down arrows to
select previous commands that I've used.  I then hit enter and emacs
runs the command putting the output in an editor buffer.  There is a key
I can hit to kill the compile if needed.  It then parses the output and
there is a key I can use to find the file listed in the compile or grep
output, load it, and position the cursor on the reported line.  (This
can even be done while the compile/grep is still running.)

* (done) For access to editor commands/functionality that may not be bound to a
keystroke it's real nice to have the ability to hit a key, type the
command name, press enter and then it's done.  This can also allow for
commands that might need to prompt for parameters, be interactive, etc.
  All editor commands should be named and can be bound to keys by name
or executed by name in this way.

* (done) Regex search.  Emacs has support for regular expression search modes
for all of the search types, incremental search, search/replace,
although I don't use it that much.

* (done, or not needed, depending on your point of view)
Multi-file search and replace.  Be able to select files interactively,
or by wildcard, or both.  Enter search string, or regex, and replace
text.  The editor loads each file and does the search, allowing you to
choose for each one whether to do the replacement, or replace all.

* If it is a full IDE it would be nice to have a way to start just the
code editor portion for quick edits.

Things that would be nice to have, but that I could live without:

(All of these things can be done easily with @command)

* Interactive diffs, merges and applying of patches.

* Able to be a front-end for gdb.

* Able to be a front-end for CVS, SVN, etc.

* (done) Be able to run shell commands, or the shell itself in an editor buffer.

* (easy) have a built-in psychotherapist or be able to play towers of hanoi.  ;-) 
#@+node:ekr.20100223100750.5842: *3* Post to pyxides, 2010/02/23
http://groups.google.com/group/pyxides

Robin Dunn's post, reproduced at:
http://groups.google.com/group/leo-editor/browse_thread/thread/4f76a0f57759aba
continues to be one of the benchmarks for Leo.

Leo 4.7 went out the door today.  It contains many important
improvements, but few directly related to Robin's important post.
That doesn't mean Robin's post is irrelevant, but it does mean that
other considerations were more relevant :-)  In particular, Leo passes
all unit tests with Python 2.6 and Python 3.1.

Leo 4.8 will concentrate on better support for vim-like bindings.  As
a happy side effect, this will make Leo compliant with almost all
unfinished aspects of Robin's mission.

There are two major items from Robin's list that are incomplete in
Leo:

* The Kill-Ring.  Leo does have a kill ring.  The vim work will fix
discrepancies between how Leo, emacs and vim handle the kill ring.

* Good keystroke macro recording and the ability to save and load
keystroke macros.  This happen as part of support for vim's "dot"
command.  To some extent, Leo's execute-script command compensates for
wimpy macro support, but I'd like to do better.

The following could be done easily using Leo's @command or @button
features.  There doesn't seem to be much demand for them in Leo, but
I'll list them here for completeness.

- Flexible build/grep commands.
- Interactive diffs, merges and applying of patches.
- Able to be a front-end for gdb. (Leo has a plugin to do this).
- Able to be a front-end for CVS, SVN, etc.
- Have a built-in psychotherapist or be able to play towers of
hanoi.  ;-)

As always, I invite you all to try Leo, and to ask for features that
would be important to you.

======

P.S. Leo does have auto-completion.  It will be improved in Leo 4.9.

#@+node:ekr.20110616084347.14800: *3* Post to pyxides, 2011/07/10
http://groups.google.com/group/pyxides

@nocolor-node

On 2010/02/23 I commented about Leo 4.7 as it relates to Robin Dunn's post,
http://groups.google.com/group/leo-editor/browse_thread/thread/4f76a0f57759aba

A few weeks ago Leo 4.9 went out the door. Imo, this version of Leo has
accomplished the mission. Leo has all the important features that Leo's users
have requested. Yes, wishlist items remain. See:
https://bugs.launchpad.net/leo-editor/+bugs

None of these wish-list items interferes in any way with Leo's day-to-day
operation. Furthermore, many of Leo's essential features moot the need for more
traditional features.

For example, Leo 4.9 adds full support for macros. Recording, saving, editing
and retrieving macros is easier in Leo than in other editors because Leo stores
macros in @macro nodes, not external files. But few, if any, of Leo's users are
likely to use macros because Leo's @button nodes make all of Python's scripting
abilities easily available on a node-by-node or outline-wide basis.

It may be that Leo could benefit from some Emacs-like or vim-like features, but
that doesn't seem so likely.  Leo has many users who also use Emacs and vim, and
they seem happy enough :-)

Finally, Leo offers features that have no counterpart at all in editors like
Emacs and vim. For example, the rst3 command converts an outline to restructured
text. See: http://webpages.charter.net/edreamleo/rstplugin3.html Yes, one could
imagine an org-mode command that does this, but the fact is that Leo's outline
orientation has given it abilities possessed by no other editor or IDE.

I invite you to try Leo. If, after using Leo for
real work, you find you would like some new feature, then by all means ask.

Edward
#@+node:ekr.20061116060847: ** @url http://www.jhorman.org/wikidPad/
#@+node:ekr.20091218120633.6299: ** Bzr
#@+node:ekr.20080917153158.10: *3* Bzr notes
@nocolor

How to create launchpad key pairs
=================================

http://help.launchpad.net/YourAccount/CreatingAnSSHKeyPair

How to create/use branches
=========================

pushes a branch after it has been created on launchpad::

    bzr push --use-existing-dir lp:leo-editor/whatever (private)

    bzr push --use-existing-dir bzr+ssh://edreamleo@bazaar.launchpad.net/~edreamleo/leo-editor/whatever (private)

remembers the push::

    bzr push --remember bzr+ssh://edreamleo@bazaar.launchpad.net/~edreamleo/leo-editor/whatever

creates branch on local machine: do this from leo.repo directory::

    bzr branch old-branch-name new-branch-name

    bzr branch lp:leo-editor new-branch-name 

    bzr branch bzr+ssh://edreamleo@bazaar.launchpad.net/~edreamleo/leo-editor/whatever new-branch-name

To resolve conflicts
====================

kdiff3 file1 file2 file3 -m


Docs
====
- http://doc.bazaar-vcs.org/latest/en/user-guide/index.html
    - http://bazaar-vcs.org/BzrWhy (referenced in sec 1.1.3 of the guide)
        - http://ianclatworthy.files.wordpress.com/2007/10/dvcs-why-and-how3.pdf
        - PQM: https://launchpad.net/pqm (google)
        - BB Bundle buggy: http://bundlebuggy.aaronbentley.com/help (google)
        - Spike solution: http://www.extremeprogramming.org/rules/spike.html (google)
- My log was at: http://bzr.arbash-meinel.com/irc_log/bzr/2008/02/bzr-2008-02-25.html


Configuration files
===================

.bzrignore in each repository contains the patterns to ignore.

EKR:  do ed-bzr

Windows: c:\Users\AppData\Roaming\bazaar\2.0\.bazaar.conf

Linux:  ~/.bazaar/bazaar.conf
#@+node:ekr.20090713080429.6042: *3* Bzr workflow notes
@nocolor-node

http://groups.google.com/group/leo-editor/browse_thread/thread/bd3e06a8a34a1938

This is the first time we have done this ("stable" branch + trunk),
so it might be good to bring this up:

- When doing a bugfix/improvement, always do it in the oldest branch that will
  receive it.

- Only after that, merge it to trunk

This prevents "manual" merges, and accidental incorporation of unwanted fixes.
It also gives us clean merge history.

I think this is the process python-the-project will use with mercurial. If they
find a bug in python 2.6.whatever, they fix it in that branch first, then merge
from that branch to trunk (so python 2.7 will receive it).

===================

http://groups.google.com/group/leo-editor/browse_thread/thread/3f24628c7f735c42

> I'll probably open a branch (based on the 4-6-final branch) to
  attempt a fix for an rst bug.

You don't need to create the branch on launchpad to do little fixes. The local
branch you create with "bzr branch" is a full-blown branch.

Here's how I do all my commits to trunk:

I have ~/leotrunk. I always keep this up to date with "bzr pull", but never
develop here.  (But I merge from leo-editor, as shown below.)
EKR: I call this main-trunk.

I have ~/leo-editor, also created from trunk. I develop here normally.
Occasionally, I just "bzr push", but often it fails because "branches have
diverged". I resist the temptation to "bzr merge" here, because it screws up
history. Rather, I:

- cd ~/leotrunk
- bzr pull (this always succeeds)
- bzr merge ~/leo-editor
- (investigate diffs)
- bzr qcommit (select the files I really want to commit--usually .py files)
- bzr push

Then, to get my ~/leo-editor up to date again:

cd ~/leo-editor
bzr pull 
#@+node:ekr.20081113095540.1: *3* Bzr/ubuntu notes
/etc/apt/sources.list.d is a directory
It contains, on my machine, files called edgy-universe.list.x
The prefix doesn't matter, but the contents of these files must contain the proper ubuntu distro name,
for example, gutsy, hardy, intrepid.
#@+node:ekr.20101024062147.6004: ** Documentation notes
#@+node:ekr.20060306194040: *3* The curse of knowledge
@nocolor

The curse of knowledge
http://groups.google.com/group/leo-editor/browse_thread/thread/3e75787223ee9303

(Rich) I'd like to see something like:

[Buttons]
...[What are Buttons good for?]
...[How do I make my own buttons?]
......[Some commands you can use with buttons]
......[Where to find button commands]
#@+node:ekr.20100828074347.5828: *3* Slide show resources
http://sourceforge.net/forum/message.php?msg_id=3615177
By: ktenney

High on my ToLearn list is vnc2swf
http://www.unixuser.org/~euske/vnc2swf/

http://sourceforge.net/forum/message.php?msg_id=3615278
By: James

A lot of people are now using Wink for demonstrations
(http://www.debugmode.com/wink/), it's is free and seems to work well.

Check out http://murl.se/11332
At the bottom they talk about tools and techniques.
http://showmedo.com seems like it would be a good
place to host vids also.

I've listened/watched a fair number of things like this;
my recomendation is to get a good microphone and
pre-amp to record your voice, and prepare the audio
track carefully. It is so aggravating when
it's hard to discern the words being spoken.

http://sourceforge.net/forum/message.php?msg_id=3758271

From: Rich

Tutorials would be great. I use Liberty BASIC, (http://libertybasic.conforums.com)
and it has a very good tutorial -- leads the beginner by the hand through much
of the language. Also the help file has working code snippets
to cut-n-paste-n-play-with. 
#@+node:ekr.20100904134301.8336: *3* Generate pdf on Linux
El 01/05/09 15:12, Ville M. Vainio escribió:

- make latex
- cd _build/latex
- make all-pdf
#@+node:ekr.20091218120633.6300: ** Other notes
#@+node:ekr.20090601083544.6066: *3* Cool plugins & Leo code
These are plugins/projects that I'd like to investigate further.

interact.py:  Add buttons so leo can interact with command line environments.
#@+node:ekr.20070614094933: *3* EasyInstall installation notes--XP
C:\prog>c:\python25\python ez_setup.py
Downloading http://cheeseshop.python.org/packages/2.5/s/setuptools/setuptools-0.6c6-py2.5.egg
Processing setuptools-0.6c6-py2.5.egg
Copying setuptools-0.6c6-py2.5.egg to c:\python25\lib\site-packages
Adding setuptools 0.6c6 to easy-install.pth file
Installing easy_install-script.py script to c:\python25\Scripts
Installing easy_install.exe script to c:\python25\Scripts
Installing easy_install-2.5-script.py script to c:\python25\Scripts
Installing easy_install-2.5.exe script to c:\python25\Scripts

Installed c:\python25\lib\site-packages\setuptools-0.6c6-py2.5.egg
Processing dependencies for setuptools==0.6c6
Finished processing dependencies for setuptools==0.6c6

C:\prog>c:\python24\python ez_setup.py
Downloading http://cheeseshop.python.org/packages/2.4/s/setuptools/setuptools-0.6c6-py2.4.egg
Processing setuptools-0.6c6-py2.4.egg
creating c:\python24\lib\site-packages\setuptools-0.6c6-py2.4.egg
Extracting setuptools-0.6c6-py2.4.egg to c:\python24\lib\site-packages
Adding setuptools 0.6c6 to easy-install.pth file
Installing easy_install-script.py script to c:\python24\Scripts
Installing easy_install.exe script to c:\python24\Scripts
Installing easy_install-2.4-script.py script to c:\python24\Scripts
Installing easy_install-2.4.exe script to c:\python24\Scripts

Installed c:\python24\lib\site-packages\setuptools-0.6c6-py2.4.egg
Processing dependencies for setuptools==0.6c6
#@+node:ekr.20071104222805: *3* Emacs/Pymacs notes
#@+node:ekr.20071102191642.1: *4* xemacs/pymacs install notes
@nocolor

Added the following line to setup.py and setup files.

# -*- coding: utf-8 -*-

The second installation script installs the Emacs Lisp part only.
[snip]
I couldn't get this script to work.  Instead, I just created a pymacs folder at::

    C:\XEmacs\xemacs-packages\lisp\pymacs

For Win32 systems, I created create c:\Windows\pymacs-services.bat containing::

    c:\Python25\python C:\prog\Pymacs-0.22\scripts\pymacs-services

To check that pymacs.el is properly installed, start Emacs and do::

    M-x load-library RET pymacs

You should not receive any error.
(works)

To check that pymacs.py is properly installed, start an interactive Python session and type::

    from Pymacs import lisp

you should not receive any error.
(works)

To check that pymacs-services is properly installed, type the following in a console::

    pymacs-services </dev/null

You should then get a line ending with (pymacs-version version), and another saying : Protocol error : `>' expected..
(works, mostly: I omitted the </dev/null

The rest is from Leo's Chapter 18::

    ; Step 1: load leoPymacs if it has not already been loaded.
    (setq reload nil)
    (if (or reload (not (boundp 'leoPymacs)))
        (setq leoPymacs (pymacs-load "leoPymacs" "leo-"))
        (message "leoPymacs already loaded")
    )

    ; Step 2: compute the path to leo/test/ut.leo using a Leo script.
    (setq script
        "g.app.scriptResult = g.os_path_abspath(
            g.os_path_join(g.app.loadDir,'..','test','ut.leo'))"
    )
    (setq fileName (leo-run-script nil script))

    ; Step 3: execute a script in ut.leo.
    (setq c (leo-open fileName))
    (setq script "print 'c',c.shortFileName() ,'current:',c.p.h")
    (leo-run-script c script)
#@+node:ekr.20071103090504: *4* Pymacs docs
@killcolor
#@+node:ekr.20071103090504.1: *5* Emacs Lisp structures and Python objects
#@+node:ekr.20071103090504.2: *6* Emacs lisp structures and Python
Conversions

Whenever Emacs Lisp calls Python functions giving them arguments, these arguments are Emacs Lisp structures that should be converted into Python objects in some way. Conversely, whenever Python calls Emacs Lisp functions, the arguments are Python objects that should be received as Emacs Lisp structures. We need some conventions for doing such conversions.

Conversions generally transmit mutable Emacs Lisp structures as mutable objects on the Python side, in such a way that transforming the object in Python will effectively transform the structure on the Emacs Lisp side (strings are handled a bit specially however, see below). The other way around, Python objects transmitted to Emacs Lisp often loose their mutability, so transforming the Emacs Lisp structure is not reflected on the Python side.

Pymacs sticks to standard Emacs Lisp, it explicitly avoids various Emacs Lisp extensions. One goal for many Pymacs users is taking some distance from Emacs Lisp, so Pymacs is not overly pushing users deeper into it.
#@+node:ekr.20071103090504.3: *6* Simple objects
Simple objects

Emacs Lisp nil and the equivalent Emacs Lisp () yield Python None. Python None and the Python empty list [] are returned as nil in Emacs Lisp.

Emacs Lisp numbers, either integer or floating, are converted in equivalent Python numbers. Emacs Lisp characters are really numbers and yield Python numbers. In the other direction, Python numbers are converted into Emacs Lisp numbers, with the exception of long Python integers and complex numbers.

Emacs Lisp strings are usually converted into equivalent Python narrow strings. As Python strings do not have text properties, these are not reflected. This may be changed by setting the pymacs-mutable-strings option : if this variable is not nil, Emacs Lisp strings are then transmitted opaquely. Python strings, except Unicode, are always converted into Emacs Lisp strings.

Emacs Lisp symbols yield the special lisp.symbol or lisp[string] notations on the Python side. The first notation is used when the Emacs Lisp symbol starts with a letter, and contains only letters, digits and hyphens, in which case Emacs Lisp hyphens get replaced by Python underscores. This convention is welcome, as Emacs Lisp programmers commonly prefer using dashes, where Python programmers use underlines. Otherwise, the second notation is used. Conversely, lisp.symbol on the Python side yields an Emacs Lisp symbol with underscores replaced with hyphens, while lisp[string] corresponds to an Emacs Lisp symbol printed with that string which, of course, should then be a valid Emacs Lisp symbol name.
#@+node:ekr.20071103090504.4: *6* Sequences
Sequences

The case of strings has been discussed in the previous section.

Proper Emacs Lisp lists, those for which the cdr of last cell is nil, are normally transmitted opaquely to Python. If pymacs-forget-mutability is set, or if Python later asks for these to be expanded, proper Emacs Lisp lists get converted into Python lists, if we except the empty list, which is always converted as Python None. In the other direction, Python lists are always converted into proper Emacs Lisp lists.

Emacs Lisp vectors are normally transmitted opaquely to Python. However, if pymacs-forget-mutability is set, or if Python later asks for these to be expanded, Emacs Lisp vectors get converted into Python tuples. In the other direction, Python tuples are always converted into Emacs Lisp vectors.

Remember the rule : Round parentheses correspond to square brackets!. It works for lists, vectors, tuples, seen from either Emacs Lisp or Python.

The above choices were debatable. Since Emacs Lisp proper lists and Python lists are the bread-and-butter of algorithms modifying structures, at least in my experience, I guess they are more naturally mapped into one another, this spares many casts in practice. While in Python, the most usual idiom for growing lists is appending to their end, the most usual idiom in Emacs Lisp to grow a list is by cons'ing new items at its beginning :

     (setq accumulator (cons 'new-item accumulator))


or more simply :

     (push 'new-item accumulator)


So, in case speed is especially important and many modifications happen in a row on the same side, while order of elements ought to be preserved, some (nreverse ...) on the Emacs Lisp side or .reverse() on the Python side side might be needed. Surely, proper lists in Emacs Lisp and lists in Python are the normal structure for which length is easily modified.

We cannot so easily change the size of a vector, the same as it is a bit more of a stunt to modify a tuple. The shape of these objects is fixed. Mapping vectors to tuples, which is admittedly strange, will only be done if the Python side requests an expanded copy, otherwise an opaque Emacs Lisp object is seen in Python. In the other direction, whenever an Emacs Lisp vector is needed, one has to write tuple(python_list) while transmitting the object. Such transmissions are most probably to be unusual, as people are not going to blindly transmit whole big structures back and forth between Emacs and Python, they would rather do it once in a while only, and do only local modifications afterwards. The infrequent casting to tuple for getting an Emacs Lisp vector seems to suggest that we did a reasonable compromise.

In Python, both tuples and lists have O(1) access, so there is no real speed consideration there. Emacs Lisp is different : vectors have O(1) access while lists have O(N) access. The rigidity of Emacs Lisp vectors is such that people do not resort to vectors unless there is a speed issue, so in real Emacs Lisp practice, vectors are used rather parsimoniously. So much, in fact, that Emacs Lisp vectors are overloaded for what they are not meant : for example, very small vectors are used to represent X events in key-maps, programmers only want to test vectors for their type, or users just like bracketed syntax. The speed of access is hardly an issue then.
#@+node:ekr.20071103090504.5: *5* Opaque objects
#@+node:ekr.20071103090504.6: *6* Emacs lisp handles
Emacs Lisp handles

When a Python function is called from Emacs Lisp, the function arguments have already been converted to Python types from Emacs Lisp types and the function result is going to be converted back to Emacs Lisp.

Several Emacs Lisp objects do not have Python equivalents, like for Emacs windows, buffers, markers, overlays, etc. It is nevertheless useful to pass them to Python functions, hoping that these Python functions will operate on these Emacs Lisp objects. Of course, the Python side may not itself modify such objects, it has to call for Emacs services to do so. Emacs Lisp handles are a mean to ease this communication.

Whenever an Emacs Lisp object may not be converted to a Python object, an Emacs Lisp handle is created and used instead. Whenever that Emacs Lisp handle is returned into Emacs Lisp from a Python function, or is used as an argument to an Emacs Lisp function from Python, the original Emacs Lisp object behind the Emacs Lisp handle is automatically retrieved.

Emacs Lisp handles are either instances of the internal Lisp class, or of one of its subclasses. If object is an Emacs Lisp handle, and if the underlying Emacs Lisp object is an Emacs Lisp sequence, then whenever object[index], object[index] = value and len(object) are meaningful, these may be used to fetch or alter an element of the sequence directly in Emacs Lisp space. Also, if object corresponds to an Emacs Lisp function, object(arguments) may be used to apply the Emacs Lisp function over the given arguments. Since arguments have been evaluated the Python way on the Python side, it would be conceptual overkill evaluating them again the Emacs Lisp way on the Emacs Lisp side, so Pymacs manage to quote arguments for defeating Emacs Lisp evaluation. The same logic applies the other way around.

Emacs Lisp handles have a value() method, which merely returns self. They also have a copy() method, which tries to open the box if possible. Emacs Lisp proper lists are turned into Python lists, Emacs Lisp vectors are turned into Python tuples. Then, modifying the structure of the copy on the Python side has no effect on the Emacs Lisp side.

For Emacs Lisp handles, str() returns an Emacs Lisp representation of the handle which should be eq to the original object if read back and evaluated in Emacs Lisp. repr() returns a Python representation of the expanded Emacs Lisp object. If that Emacs Lisp object has an Emacs Lisp representation which Emacs Lisp could read back, then repr() value is such that it could be read back and evaluated in Python as well, this would result in another object which is equal to the original, but not neccessarily eq.
#@+node:ekr.20071103090504.7: *6* Python handles
Python handles

The same as Emacs Lisp handles are useful for handling Emacs Lisp objects on the Python side, Python handles are useful for handling Python objects on the Emacs Lisp side.

Many Python objects do not have direct Emacs Lisp equivalents, including long integers, complex numbers, Unicode strings, modules, classes, instances and surely a lot of others. When such are being transmitted to the Emacs Lisp side, Pymacs use Python handles. These are automatically recovered into the original Python objects whenever transmitted back to Python, either as arguments to a Python function, as the Python function itself, or as the return value of an Emacs Lisp function called from Python.

The objects represented by these Python handles may be inspected or modified using the basic library of Python functions. For example, in :

     (setq matcher (pymacs-eval "re.compile('pattern').match"))
     (pymacs-call matcher argument)


the initial setq above could be decomposed into :

           (setq compiled (pymacs-eval "re.compile('pattern')")
            matcher (pymacs-call "getattr" compiled "match"))


This example shows that one may use pymacs-call with getattr as the function, to get a wanted attribute for a Python object.
#@+node:ekr.20071103090504.8: *5* Usages on the Emacs lisp side
#@+node:ekr.20071103090504.9: *6* pymacs-eval/apply

pymacs-eval

Function (pymacs-eval text) gets text evaluated as a Python expression, and returns the value of that expression converted back to Emacs Lisp.

pymacs-call

Function (pymacs-call function argument...) will get Python to apply the given function over zero or more argument. function is either a string holding Python source code for a function (like a mere name, or even an expression), or else, a Python handle previously received from Python, and hopefully holding a callable Python object. Each argument gets separately converted to Python before the function is called. pymacs-call returns the resulting value of the function call, converted back to Emacs Lisp.

pymacs-apply

Function (pymacs-apply function arguments) will get Python to apply the given function over the given arguments. arguments is a list containing all arguments, or nil if there is none. Besides arguments being bundled together instead of given separately, the function acts pretty much like pymacs-call.

We do not expect that pymacs-eval, pymacs-call or pymacs-apply will be much used, if ever. In practice, the Emacs Lisp side of a Pymacs application might call pymacs-load a few times for linking into the Python modules, with the indirect effect of defining trampoline functions for these modules on the Emacs Lisp side, which can later be called like usual Emacs Lisp functions.
#@+node:ekr.20071103090504.10: *6* pymacs-load
pymacs-load

Function (pymacs-load module prefix) imports the Python module into Emacs Lisp space.

module is the name of the file containing the module, without any .py or .pyc extension. If the directory part is omitted in module, the module will be looked into the current Python search path. Dot notation may be used when the module is part of a package. Each top-level function in the module produces a trampoline function in Emacs Lisp having the same name, except that underlines in Python names are turned into dashes in Emacs Lisp, and that prefix is uniformly added before the Emacs Lisp name (as a way to avoid name clashes).

prefix may be omitted, in which case it defaults to base name of module with underlines turned into dashes, and followed by a dash.

Whenever pymacs_load_hook is defined in the loaded Python module, pymacs-load calls it without arguments, but before creating the Emacs view for that module. So, the pymacs_load_hook function may create new definitions or even add interaction attributes to functions.

The return value of a successful pymacs-load is the module object. An optional third argument, noerror, when given and not nil, will have pymacs-load to return nil instead of raising an error, if the Python module could not be found.

When later calling one of these trampoline functions, all provided arguments are converted to Python and transmitted, and the function return value is later converted back to Emacs Lisp. It is left to the Python side to check for argument consistency. However, for an interactive function, the interaction specification drives some checking on the Emacs Lisp side. Currently, there is no provision for collecting keyword arguments in Emacs Lisp.
#@+node:ekr.20071103091052: *5* Usage on the Python side
#@+node:ekr.20071103091052.1: *6* Python setup
Python setup

Pymacs requires little or no setup in the Python modules which are meant to be used from Emacs, for the simple situations where these modules receive nothing but Emacs nil, numbers or strings, or return nothing but Python None, numbers or strings.

Otherwise, use from Pymacs import lisp. If you need more Pymacs features, like the Let class, write from Pymacs import lisp, Let.
#@+node:ekr.20071103091052.2: *6* Response mode
Response mode

When Python receives a request from Emacs in the context of Pymacs, and until it returns the reply, Emacs keeps listening to serve Python requests. Emacs is not listening otherwise. Other Python threads, if any, may not call Emacs without very careful synchronisation.
#@+node:ekr.20071103091052.3: *6* Emacs lisp symbols
Emacs Lisp symbols

lisp is a special object which has useful built-in magic. Its attributes do nothing but represent Emacs Lisp symbols, created on the fly as needed (symbols also have their built-in magic).

lisp.nil or lisp["nil"], are the same as None.

Otherwise, lisp.symbol and lisp[string] yield objects of the internal Symbol type. These are genuine Python objects, that could be referred to by simple Python variables. One may write quote = lisp.quote, for example, and use quote afterwards to mean that Emacs Lisp symbol. If a Python function received an Emacs Lisp symbol as an argument, it can check with == if that argument is lisp.never or lisp.ask, say. A Python function may well choose to return lisp.t.

In Python, writing lisp.symbol = value or lisp[string] = value does assign value to the corresponding symbol in Emacs Lisp space. Beware that in such cases, the lisp. prefix may not be [omitted] spared. After result = lisp.result, one cannot hope that a later result = 3 will have any effect in the Emacs Lisp space : this would merely change the Python variable result, which was a reference to a Symbol instance, so it is now a reference to the number 3.

The Symbol class has value() and copy() methods. One can use either lisp.symbol.value() or lisp.symbol.copy() to access the Emacs Lisp value of a symbol, after conversion to some Python object, of course. However, if value() would have given an Emacs Lisp handle, lisp.symbol.copy() has the effect of lisp.symbol.value().copy(), that is, it returns the value of the symbol as opened as possible.

A symbol may also be used as if it was a Python function, in which case it really names an Emacs Lisp function that should be applied over the following function arguments. The result of the Emacs Lisp function becomes the value of the call, with all due conversions of course.
#@+node:ekr.20071103091052.4: *6* Dynamic bindings (The let class)
Dynamic bindings

As Emacs Lisp uses dynamic bindings, it is common that Emacs Lisp programs use
let for temporarily setting new values for some Emacs Lisp variables having
global scope. These variables recover their previous value automatically when
the let gets completed, even if an error occurs which interrupts the normal flow
of execution.

Pymacs has a Let class to represent such temporary settings. Suppose for example
that you want to recover the value of lisp.mark() when the transient mark mode
is active on the Emacs Lisp side. One could surely use lisp.mark(lisp.t) to
force reading the mark in such cases, but for the sake of illustration, let's
ignore that, and temporarily deactivate transient mark mode instead. This could
be done this way :

        try :
        let = Let()
        let.push(transient_mark_mode=None)
        ... user code ...
        finally :
        let.pop()

let.push() accepts any number of keywords arguments. Each keyword name is
interpreted as an Emacs Lisp symbol written the Pymacs way, with underlines. The
value of that Emacs Lisp symbol is saved on the Python side, and the value of
the keyword becomes the new temporary value for this Emacs Lisp symbol. A later
let.pop() restores the previous value for all symbols which were saved together
at the time of the corresponding let.push(). There may be more than one
let.push() call for a single Let instance, they stack within that instance. Each
let.pop() will undo one and only one let.push() from the stack, in the reverse
order or the pushes.

When the Let instance disappears, either because the programmer does del let or
let = None, or just because the Python let variable goes out of scope, all
remaining let.pop() get automatically executed, so the try/finally statement may
be omitted in practice. For this omission to work flawlessly, the programmer
should be careful at not keeping extra references to the Let instance.

The constructor call let = Let() also has an implied initial .push() over all
given arguments, so the explicit let.push() may be omitted as well. In practice,
this sums up and the above code could be reduced to a mere :

     let = Let(transient_mark_mode=None)
     ... user code ...

Be careful at assigning the result of the constructor to some Python variable.
Otherwise, the instance would disappear immediately after having been created,
restoring the Emacs Lisp variable much too soon.

Any variable to be bound with Let should have been bound in advance on the Emacs
Lisp side. This restriction usually does no kind of harm. Yet, it will likely be
lifted in some later version of Pymacs.

The Let class has other methods meant for some macros which are common in Emacs
Lisp programming, in the spirit of let bindings. These method names look like
push_* or pop_*, where Emacs Lisp macros are save-*. One has to use the matching
pop_* for undoing the effect of a given push_* rather than a mere .pop() : the
Python code is clearer, this also ensures that things are undone in the proper
order. The same Let instance may use many push_* methods, their effects nest.

push_excursion() and pop_excursion() save and restore the current buffer, point
and mark. push_match_data() and pop_match_data() save and restore the state of
the last regular expression match. push_restriction() and pop_restriction() save
and restore the current narrowing limits. push_selected_window() and
pop_selected_window() save and restore the fact that a window holds the cursor.
push_window_excursion() and pop_window_excursion() save and restore the current
window configuration in the Emacs display.

As a convenience, let.push() and all other push_* methods return the Let
instance. This helps chaining various push_* right after the instance
generation. For example, one may write :

         let = Let().push_excursion()
         if True :
         ... user code ...
         del let

The if True: (use if 1: with older Python releases, some people might prefer
writing if let: anyway), has the only goal of indenting user code, so the scope
of the let variable is made very explicit. This is purely stylistic, and not at
all necessary. The last del let might be omitted in a few circumstances, for
example if the excursion lasts until the end of the Python function.
#@+node:ekr.20071103091052.5: *6* Raw Emacs lisp expression
Raw Emacs Lisp expressions

Pymacs offers a device for evaluating a raw Emacs Lisp expression, or a sequence of such, expressed as a string. One merely uses lisp as a function, like this :

     lisp("""
     ...
     possibly-long-sequence-of-lisp-expressions
     ...
     """)


The Emacs Lisp value of the last or only expression in the sequence becomes the value of the lisp call, after conversion back to Python.
#@+node:ekr.20071103091052.6: *6* User interaction
User interaction

Emacs functions have the concept of user interaction for completing the specification of their arguments while being called. This happens only when a function is interactively called by the user, it does not happen when a function is programmatically called by another. As Python does not have a corresponding facility, a bit of trickery was needed to retrofit that facility on the Python side.

After loading a Python module but prior to creating an Emacs view for this
module, Pymacs decides whether loaded functions will be interactively callable
from Emacs, or not. Whenever a function has an interaction attribute, this
attribute holds the Emacs interaction specification for this function. The
specification is either another Python function or a string. In the former case,
that other function is called without arguments and should, maybe after having
consulted the user, return a list of the actual arguments to be used for the
original function. In the latter case, the specification string is used verbatim
as the argument to the (interactive ...) function on the Emacs side. To get a
short reminder about how this string is interpreted on the Emacs side, try C-h f
interactive within Emacs. Here is an example where an empty string is used to
specify that an interactive has no arguments::

    from Pymacs import lisp

    def hello_world() :
        "`Hello world' from Python."
        lisp.insert("Hello from Python!")
        hello_world.interaction = ''

Versions of Python released before the integration of PEP 232 do not allow users
to add attributes to functions, so there is a fallback mechanism. Let's presume
that a given function does not have an interaction attribute as explained above.
If the Python module contains an interactions global variable which is a
dictionary, if that dictionary has an entry for the given function with a value
other than None, that function is going to be interactive on the Emacs side.
Here is how the preceeding example should be written for an older version of
Python, or when portability is at premium::

    from Pymacs import lisp
    interactions = {}

    def hello_world() :
        "`Hello world' from Python."
        lisp.insert("Hello from Python!")
        interactions[hello_world] = ''

One might wonder why we do not merely use lisp.interactive(...) from within
Python. There is some magic in the Emacs Lisp interpreter itself, looking for
that call before the function is actually entered, this explains why
(interactive ...) has to appear first in an Emacs Lisp defun. Pymacs could try
to scan the already compiled form of the Python code, seeking for
lisp.interactive, but as the evaluation of lisp.interactive arguments could get
arbitrarily complex, it would a real challenge un-compiling that evaluation into
Emacs Lisp.
#@+node:ekr.20071103091052.7: *6* Key bindings
Keybindings

An interactive function may be bound to a key sequence.

To translate bindings like C-x w, say, one might have to know a bit more how
Emacs Lisp processes string escapes like \C-x or \M-\C-x in Emacs Lisp, and
emulate it within Python strings, since Python does not have such escapes. \C-L,
where L is an upper case letter, produces a character which ordinal is the
result of subtracting 0x40 from ordinal of L. \M- has the ordinal one gets by
adding 0x80 to the ordinal of following described character. So people can use
self-inserting non-ASCII characters, \M- is given another representation, which
is to replace the addition of 0x80 by prefixing with `ESC', that is 0x1b.

So \C-x in Emacs is '\x18' in Python. This is easily found, using an interactive
Python session, by givin it : chr(ord('X') - ord('A') + 1). An easier way would
be using the kbd function on the Emacs Lisp side, like with lisp.kbd('C-x w') or
lisp.kbd('M-<f2>').

To bind the F1 key to the helper function in some module :

     lisp.global_set_key((lisp.f1,), lisp.module_helper)

(item,) is a Python tuple yielding an Emacs Lisp vector. lisp.f1 translates to
the Emacs Lisp symbol f1. So, Python (lisp.f1,) is Emacs Lisp [f1]. Keys like
[M-f2] might require some more ingenuity, one may write either (lisp['M-f2'],)
or (lisp.M_f2,) on the Python side.
#@+node:ekr.20071103092153: *5* Debugging
#@+node:ekr.20071103092153.1: *6* The *pymacs* buffer
The *Pymacs* buffer

Emacs and Python are two separate processes (well, each may use more than one process). Pymacs implements a simple communication protocol between both, and does whatever needed so the programmers do not have to worry about details. The main debugging tool is the communication buffer between Emacs and Python, which is named *Pymacs*. As it is sometimes helpful to understand the communication protocol, it is briefly explained here, using an artificially complex example to do so. Consider :

     (pymacs-eval "lisp('(pymacs-eval \"`2L**111`\")')")
     "2596148429267413814265248164610048L"

Here, Emacs asks Python to ask Emacs to ask Python for a simple bignum computation. Note that Emacs does not natively know how to handle big integers, nor has an internal representation for them. This is why I use backticks, so Python returns a string representation of the result, instead of the result itself. Here is a trace for this example. The < character flags a message going from Python to Emacs and is followed by an expression written in Emacs Lisp. The > character flags a message going from Emacs to Python and is followed by a expression written in Python. The number gives the length of the message.

     <22   (pymacs-version "0.3")
     >49   eval("lisp('(pymacs-eval \"`2L**111`\")')")
     <25   (pymacs-eval "`2L**111`")
     >18   eval("`2L**111`")
     <47   (pymacs-reply "2596148429267413814265248164610048L")
     >45   reply("2596148429267413814265248164610048L")
     <47   (pymacs-reply "2596148429267413814265248164610048L")

Python evaluation is done in the context of the Pymacs.pymacs module, so for example a mere reply really means Pymacs.pymacs.reply. On the Emacs Lisp side, there is no concept of module namespaces, so we use the pymacs- prefix as an attempt to stay clean. Users should ideally refrain from naming their Emacs Lisp objects with a pymacs- prefix.

reply and pymacs-reply are special functions meant to indicate that an expected result is finally transmitted. error and pymacs-error are special functions that introduce a string which explains an exception which recently occurred. pymacs-expand is a special function implementing the copy() methods of Emacs Lisp handles or symbols. In all other cases, the expression is a request for the other side, that request stacks until a corresponding reply is received.

Part of the protocol manages memory, and this management generates some extra-noise in the *Pymacs* buffer. Whenever Emacs passes a structure to Python, an extra pointer is generated on the Emacs side to inhibit garbage collection by Emacs. Python garbage collector detects when the received structure is no longer needed on the Python side, at which time the next communication will tell Emacs to remove the extra pointer. It works symmetrically as well, that is, whenever Python passes a structure to Emacs, an extra Python reference is generated to inhibit garbage collection on the Python side. Emacs garbage collector detects when the received structure is no longer needed on the Emacs side, after which Python will be told to remove the extra reference. For efficiency, those allocation-related messages are delayed, merged and batched together within the next communication having another purpose.

Variable pymacs-trace-transit may be modified for controlling how and when the *Pymacs* buffer, or parts thereof, get erased.
#@+node:ekr.20071103092153.2: *6* Usual Emacs debugging
Emacs usual debugging

If cross-calls between Emacs Lisp and Python nest deeply, an error will raise
successive exceptions alternatively on both sides as requests unstack, and the
diagnostic gets transmitted back and forth, slightly growing as we go. So,
errors will eventually be reported by Emacs. I made no kind of effort to
transmit the Emacs Lisp backtrace on the Python side, as I do not see a purpose
for it : all debugging is done within Emacs windows anyway.

On recent Emacses, the Python backtrace gets displayed in the mini-buffer, and
the Emacs Lisp backtrace is simultaneously shown in the *Backtrace* window. One
useful thing is to allow to mini-buffer to grow big, so it has more chance to
fully contain the Python backtrace, the last lines of which are often especially
useful. Here, I use :

         (setq resize-mini-windows t
          max-mini-window-height .85)

in my .emacs file, so the mini-buffer may use 85% of the screen, and quickly
shrinks when fewer lines are needed. The mini-buffer contents disappear at the
next keystroke, but you can recover the Python backtrace by looking at the end
of the *Messages* buffer. In which case the ffap package in Emacs may be yet
another friend! From the *Messages* buffer, once ffap activated, merely put the
cursor on the file name of a Python module from the backtrace, and C-x C-f RET
will quickly open that source for you.
#@+node:ekr.20071103092153.3: *6* Auto-reloading on save
Auto-reloading on save

I found useful to automatically pymacs-load some Python files whenever they get
saved from Emacs. This can be decided on a per-file or per-directory basis. To
get a particular Python file to be reloaded automatically on save, add the
following lines at the end :

     # Local Variables :
     # pymacs-auto-reload : t
     # End :

Here is an example of automatic reloading on a per-directory basis. The code
below assumes that Python files meant for Pymacs are kept in
~/share/emacs/python.

    (defun fp-maybe-pymacs-reload ()
        (let ((pymacsdir (expand-file-name "~/share/emacs/python/")))
         (when (and (string-equal (file-name-directory buffer-file-name)
                  pymacsdir)
              (string-match "\\.py\\'" buffer-file-name))
          (pymacs-load (substring buffer-file-name 0 -3)))))
         (add-hook 'after-save-hook 'fp-maybe-pymacs-reload)
#@+node:ekr.20071103092153.4: *5* Example 1: defining an Emacs command in Python
@nocolor
Let's say I have a a module, call it manglers.py, containing this simple python
function::

    def break_on_whitespace(some_string) :
         words = some_string.split()
         return '\n'.join(words)

The goal is telling Emacs about this function so that I can call it on a region
of text and replace the region with the result of the call. We shall also bind
this function to the key [f7].

Here is the Python side::
@color

    from Pymacs import lisp
    interactions = {}

    def break_on_whitespace():
        # start and end may be given in any order.
        start,end = lisp.point(),lisp.mark(lisp.t)
        words = lisp.buffer_substring(start, end).split()
        lisp.delete_region(start,end)
        lisp.insert('\n'.join(words))

    interactions[break_on_whitespace] = ''
@nocolor

Here is the emacs side::

    (pymacs-load "manglers")
    (global-set-key [f7] 'manglers-break-on-whitespace)
#@+node:ekr.20071103093725: *5* Example 3: defining a rebox tool
For comments held within boxes, it is painful to fill paragraphs, while
stretching or shrinking the surrounding box by hand, as needed. This piece of
Python code eases my life on this. It may be used interactively from within
Emacs through the Pymacs interface, or in batch as a script which filters a
single region to be reformatted.

In batch mode, the reboxing is driven by command options and arguments and expects a
complete, self-contained boxed comment from a file.

Emacs function rebox-region also presumes that the region encloses a single
boxed comment.

Emacs rebox-comment is different, as it has to chase itself the extent of the
surrounding boxed comment.

#@+node:ekr.20071103094355: *6* The python side
Design notes for rebox.py:

Pymacs specific features are used exclusively from within the pymacs_load_hook
function and the Emacs_Rebox class. In batch mode, Pymacs is not even imported.

In batch mode, as well as with rebox-region, the text to handle is turned over
to Python, and fully processed in Python, with practically no Pymacs interaction
while the work gets done. On the other hand, rebox-comment is rather Pymacs
intensive: the comment boundaries are chased right from the Emacs buffer, as
directed by the function Emacs_Rebox.find_comment. Once the boundaries are
found, the remainder of the work is essentially done on the Python side.

Once the boxed comment has been reformatted in Python, the old comment is
removed in a single delete operation, the new comment is inserted in a second
operation. This occurs in Emacs_Rebox.process_emacs_region. But by doing so, if
point was within the boxed comment before the reformatting, its precise position
is lost. To well preserve point, Python might have driven all reformatting
details directly in the Emacs buffer. We really preferred doing it all on the
Python side : as we gain legibility by expressing the algorithms in pure Python,
the same Python code may be used in batch or interactively, and we avoid the
slowdown that would result from heavy use of Emacs services.

To avoid completely loosing point, I kludged a Marker class, which goal is to
estimate the new value of point from the old. Reformatting may change the amount
of white space, and either delete or insert an arbitrary number characters meant
to draw the box. The idea is to initially count the number of characters between
the beginning of the region and point, while ignoring any problematic character.
Once the comment has been reboxed, point is advanced from the beginning of the
region until we get the same count of characters, skipping all problematic
characters. This Marker class works fully on the Python side, it does not
involve Pymacs at all, but it does solve a problem that resulted from my choice
of keeping the data on the Python side instead of handling it directly in the
Emacs buffer.

We want a comment reformatting to appear as a single operation, in the context
of Emacs Undo. The method Emacs_Rebox.clean_undo_after handles the general case
for this. Not that we do so much in practice : a reformatting implies one
delete-region and one insert, and maybe some other little adjustements at
Emacs_Rebox.find_comment time. Even if this method scans and mofifies an Emacs
Lisp list directly in the Emacs memory, the code doing this stays neat and
legible. However, I found out that the undo list may grow quickly when the Emacs
buffer use markers, with the consequence of making this routine so Pymacs
intensive that most of the CPU is spent there. I rewrote that routine in Emacs
Lisp so it executes in a single Pymacs interaction.

Function Emacs_Rebox.remainder_of_line could have been written in Python, but it
was probably not worth going away from this one-liner in Emacs Lisp. Also, given
this routine is often called by find_comment, a few Pymacs protocol interactions
are spared this way. This function is useful when there is a need to apply a
regexp already compiled on the Python side, it is probably better fetching the
line from Emacs and do the pattern match on the Python side, than transmitting
the source of the regexp to Emacs for it to compile and apply it.

For refilling, I could have either used the refill algorithm built within in
Emacs, programmed a new one in Python, or relied on Ross Paterson's fmt,
distributed by GNU and available on most Linuxes. In fact, refill_lines prefers
the latter. My own Emacs setup is such that the built-in refill algorithm is
already overridden by GNU fmt, and it really does a much better job. Experience
taught me that calling an external program is fast enough to be very bearable,
even interactively. If Python called Emacs to do the refilling, Emacs would
itself call GNU fmt in my case, I preferred that Python calls GNU fmt directly.
I could have reprogrammed GNU fmt in Python. Despite interesting, this is an
uneasy project : fmt implements the Knuth refilling algorithm, which depends on
dynamic programming techniques; Ross did carefully fine tune them, and took care
of many details. If GNU fmt fails, for not being available, say, refill_lines
falls back on a dumb refilling algorithm, which is better than none.
#@+node:ekr.20071103094355.1: *6* The emacs side
For most Emacs language editing modes, refilling does not make sense
outside comments, one may redefine the `M-q' command and link it to this
Pymacs module.  For example, I use this in my `.emacs' file::

    (add-hook 'c-mode-hook 'fp-c-mode-routine)
    (defun fp-c-mode-routine ()
        (local-set-key "\M-q" 'rebox-comment))
    (autoload 'rebox-comment "rebox" nil t)
    (autoload 'rebox-region "rebox" nil t)

with a "rebox.el" file having this single line:

    (pymacs-load "Pymacs.rebox")

The Emacs function `rebox-comment' automatically discovers the extent of the
boxed comment near the cursor, possibly refills the text, then adjusts the box
style. When this command is executed, the cursor should be within a comment, or
else it should be between two comments, in which case the command applies to the
next comment.

The Emacs function `rebox-region' does the same, except that it takes the
current region as a boxed comment. Both commands obey numeric prefixes to add or
remove a box, force a particular box style, or to prevent refilling of text.
Without such prefixes, the commands may deduce the current box style from the
comment itself so the style is preserved.

The default style initial value is nil or 0.  It may be preset to another
value through calling `rebox-set-default-style' from Emacs LISP, or changed
to anything else though using a negative value for a prefix, in which case
the default style is set to the absolute value of the prefix.

A `C-u' prefix avoids refilling the text, but forces using the default box
style.  `C-u -' lets the user interact to select one attribute at a time.

Adding new styles
-----------------

Let's suppose you want to add your own boxed comment style, say:

    //--------------------------------------------+
    // This is the style mandated in our company.
    //--------------------------------------------+

You might modify `rebox.py' but then, you will have to edit it whenever you
get a new release of `pybox.py'.  Emacs users might modify their `.emacs'
file or their `rebox.el' bootstrap, if they use one.  In either cases,
after the `(pymacs-load "Pymacs.rebox")' line, merely add:

    (rebox-Template NNN MMM ["//-----+"
                             "// box  "
                             "//-----+"])

In batch mode [If you use the `rebox' script rather than Emacs], the simplest is to make
your own.  This is easy, as it is very small.  For example, the above
style could be implemented by using this script instead of `rebox':

    #!/usr/bin/env python
    import sys
    from Pymacs import rebox
    rebox.Template(226, 325, ('//-----+',
                              '// box  ',
                              '//-----+'))
    apply(rebox.main, tuple(sys.argv[1:]))

In all cases, NNN is the style three-digit number, with no zero digit.
Pick any free style number, you are safe with 911 and up.  MMM is the
recognition priority, only used to disambiguate the style of a given boxed
comments, when it matches many styles at once.  Try something like 400.
Raise or lower that number as needed if you observe false matches.

Usually, the template uses three lines of equal length.  Do not worry if
this implies a few trailing spaces, they will be cleaned up automatically
at box generation time.  The first line or the third line may be omitted
to create vertically opened boxes.  But the middle line may not be omitted,
it ought to include the word `box', which will get replaced by your actual
comment.  If the first line is shorter than the middle one, it gets merged
at the start of the comment.  If the last line is shorter than the middle
one, it gets merged at the end of the comment and is refilled with it.
#@+node:ekr.20071103093725.1: *6* rebox.py
@color
@language python
@tabwidth -4

@others
<< templates >>

if __name__ == '__main__':
    apply(main, sys.argv[1:])
#@+node:ekr.20071103093725.2: *7* rebox declarations
#!/usr/bin/env python
# Copyright © 1991-1998, 2000, 2002 Progiciels Bourbeau-Pinard inc.
# François Pinard <pinard@iro.umontreal.ca>, April 1991.

# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

"""\
Handling of boxed comments in various box styles.

Introduction
------------

For comments held within boxes, it is painful to fill paragraphs, while
stretching or shrinking the surrounding box "by hand", as needed.  This piece
of Python code eases my life on this.  It may be used interactively from
within Emacs through the Pymacs interface, or in batch as a script which
filters a single region to be reformatted.  I find only fair, while giving
all sources for a package using such boxed comments, to also give the
means I use for nicely modifying comments.  So here they are!

Box styles
----------

Each supported box style has a number associated with it.  This number is
arbitrary, yet by _convention_, it holds three non-zero digits such the the
hundreds digit roughly represents the programming language, the tens digit
roughly represents a box quality (or weight) and the units digit roughly
a box type (or figure).  An unboxed comment is merely one of box styles.
Language, quality and types are collectively referred to as style attributes.

When rebuilding a boxed comment, attributes are selected independently
of each other.  They may be specified by the digits of the value given
as Emacs commands argument prefix, or as the `-s' argument to the `rebox'
script when called from the shell.  If there is no such prefix, or if the
corresponding digit is zero, the attribute is taken from the value of the
default style instead.  If the corresponding digit of the default style
is also zero, than the attribute is recognised and taken from the actual
boxed comment, as it existed before prior to the command.  The value 1,
which is the simplest attribute, is ultimately taken if the parsing fails.

A programming language is associated with comment delimiters.  Values are
100 for none or unknown, 200 for `/*' and `*/' as in plain C, 300 for `//'
as in C++, 400 for `#' as in most scripting languages, 500 for `;' as in
LISP or assembler and 600 for `%' as in TeX or PostScript.

Box quality differs according to language. For unknown languages (100) or
for the C language (200), values are 10 for simple, 20 for rounded, and
30 or 40 for starred.  Simple quality boxes (10) use comment delimiters
to left and right of each comment line, and also for the top or bottom
line when applicable. Rounded quality boxes (20) try to suggest rounded
corners in boxes.  Starred quality boxes (40) mostly use a left margin of
asterisks or X'es, and use them also in box surroundings.  For all others
languages, box quality indicates the thickness in characters of the left
and right sides of the box: values are 10, 20, 30 or 40 for 1, 2, 3 or 4
characters wide.  With C++, quality 10 is not useful, it is not allowed.

Box type values are 1 for fully opened boxes for which boxing is done
only for the left and right but not for top or bottom, 2 for half
single lined boxes for which boxing is done on all sides except top,
3 for fully single lined boxes for which boxing is done on all sides,
4 for half double lined boxes which is like type 2 but more bold,
or 5 for fully double lined boxes which is like type 3 but more bold.

The special style 221 is for C comments between a single opening `/*'
and a single closing `*/'.  The special style 111 deletes a box.

Batch usage
-----------

Usage is `rebox [OPTION]... [FILE]'.  By default, FILE is reformatted to
standard output by refilling the comment up to column 79, while preserving
existing boxed comment style.  If FILE is not given, standard input is read.
Options may be:

  -n         Do not refill the comment inside its box, and ignore -w.
  -s STYLE   Replace box style according to STYLE, as explained above.
  -t         Replace initial sequence of spaces by TABs on each line.
  -v         Echo both the old and the new box styles on standard error.
  -w WIDTH   Try to avoid going over WIDTH columns per line.

So, a single boxed comment is reformatted by invocation.  `vi' users, for
example, would need to delimit the boxed comment first, before executing
the `!}rebox' command (is this correct? my `vi' recollection is far away).

Batch usage is also slow, as internal structures have to be reinitialised
at every call.  Producing a box in a single style is fast, but recognising
the previous style requires setting up for all possible styles.

Emacs usage
-----------

For most Emacs language editing modes, refilling does not make sense
outside comments, one may redefine the `M-q' command and link it to this
Pymacs module.  For example, I use this in my `.emacs' file:

     (add-hook 'c-mode-hook 'fp-c-mode-routine)
     (defun fp-c-mode-routine ()
       (local-set-key "\M-q" 'rebox-comment))
     (autoload 'rebox-comment "rebox" nil t)
     (autoload 'rebox-region "rebox" nil t)

with a "rebox.el" file having this single line:

     (pymacs-load "Pymacs.rebox")

Install Pymacs from `http://www.iro.umontreal.ca/~pinard/pymacs.tar.gz'.

The Emacs function `rebox-comment' automatically discovers the extent of
the boxed comment near the cursor, possibly refills the text, then adjusts
the box style.  When this command is executed, the cursor should be within
a comment, or else it should be between two comments, in which case the
command applies to the next comment.  The function `rebox-region' does
the same, except that it takes the current region as a boxed comment.
Both commands obey numeric prefixes to add or remove a box, force a
particular box style, or to prevent refilling of text.  Without such
prefixes, the commands may deduce the current box style from the comment
itself so the style is preserved.

The default style initial value is nil or 0.  It may be preset to another
value through calling `rebox-set-default-style' from Emacs LISP, or changed
to anything else though using a negative value for a prefix, in which case
the default style is set to the absolute value of the prefix.

A `C-u' prefix avoids refilling the text, but forces using the default box
style.  `C-u -' lets the user interact to select one attribute at a time.

Adding new styles
-----------------

Let's suppose you want to add your own boxed comment style, say:

    //--------------------------------------------+
    // This is the style mandated in our company.
    //--------------------------------------------+

You might modify `rebox.py' but then, you will have to edit it whenever you
get a new release of `pybox.py'.  Emacs users might modify their `.emacs'
file or their `rebox.el' bootstrap, if they use one.  In either cases,
after the `(pymacs-load "Pymacs.rebox")' line, merely add:

    (rebox-Template NNN MMM ["//-----+"
                             "// box  "
                             "//-----+"])

If you use the `rebox' script rather than Emacs, the simplest is to make
your own.  This is easy, as it is very small.  For example, the above
style could be implemented by using this script instead of `rebox':

    #!/usr/bin/env python
    import sys
    from Pymacs import rebox
    rebox.Template(226, 325, ('//-----+',
                              '// box  ',
                              '//-----+'))
    apply(rebox.main, tuple(sys.argv[1:]))

In all cases, NNN is the style three-digit number, with no zero digit.
Pick any free style number, you are safe with 911 and up.  MMM is the
recognition priority, only used to disambiguate the style of a given boxed
comments, when it matches many styles at once.  Try something like 400.
Raise or lower that number as needed if you observe false matches.

On average, the template uses three lines of equal length.  Do not worry if
this implies a few trailing spaces, they will be cleaned up automatically
at box generation time.  The first line or the third line may be omitted
to create vertically opened boxes.  But the middle line may not be omitted,
it ought to include the word `box', which will get replaced by your actual
comment.  If the first line is shorter than the middle one, it gets merged
at the start of the comment.  If the last line is shorter than the middle
one, it gets merged at the end of the comment and is refilled with it.

History
-------

I first observed rounded corners, as in style 223 boxes, in code from
Warren Tucker, a previous maintainer of the `shar' package, circa 1980.

Except for very special files, I carefully avoided boxed comments for
real work, as I found them much too hard to maintain.  My friend Paul
Provost was working at Taarna, a computer graphics place, which had boxes
as part of their coding standards.  He asked that we try something to get
him out of his misery, and this how `rebox.el' was originally written.
I did not plan to use it for myself, but Paul was so enthusiastic that I
timidly started to use boxes in my things, very little at first, but more
and more as time passed, still in doubt that it was a good move.  Later,
many friends spontaneously started to use this tool for real, some being very
serious workers.  This convinced me that boxes are acceptable, after all.

I do not use boxes much with Python code.  It is so legible that boxing
is not that useful.  Vertical white space is less necessary, too.  I even
avoid white lines within functions.  Comments appear prominent enough when
using highlighting editors like Emacs or nice printer tools like `enscript'.

After Emacs could be extended with Python, in 2001, I translated `rebox.el'
into `rebox.py', and added the facility to use it as a batch script.
"""

## Note: This code is currently compatible down to Python version 1.5.2.
## It is probably worth keeping it that way for a good while, still.

## Note: a double hash comment introduces a group of functions or methods.

import re, string, sys

#@+node:ekr.20071103093725.3: *7* main
def main(*arguments):
    refill = 1
    style = None
    tabify = 0
    verbose = 0
    width = 79
    import getopt
    options, arguments = getopt.getopt(arguments, 'ns:tvw:', ['help'])
    for option, value in options:
        if option == '--help':
            sys.stdout.write(__doc__)
            sys.exit(0)
        elif option == '-n':
            refill = 0
        elif option == '-s':
            style = int(value)
        elif option == '-t':
            tabify = 1
        elif option == '-v':
            verbose = 1
        elif option == '-w':
            width = int(value)
    if len(arguments) == 0:
        text = sys.stdin.read()
    elif len(arguments) == 1:
        text = open(arguments[0]).read()
    else:
        sys.stderr.write("Invalid usage, try `rebox --help' for help.\n")
        sys.exit(1)
    old_style, new_style, text, position = engine(
        text, style=style, width=width, refill=refill, tabify=tabify)
    if text is None:
        sys.stderr.write("* Cannot rebox to style %d.\n" % new_style)
        sys.exit(1)
    sys.stdout.write(text)
    if verbose:
        if old_style == new_style:
            sys.stderr.write("Reboxed with style %d.\n" % old_style)
        else:
            sys.stderr.write("Reboxed from style %d to %d.\n"
                             % (old_style, new_style))

#@+node:ekr.20071103093725.4: *7* pymacs_load_hook
def pymacs_load_hook():
    global interactions, lisp, Let, region, comment, set_default_style
    from Pymacs import lisp, Let
    emacs_rebox = Emacs_Rebox()
    # Declare functions for Emacs to import.
    interactions = {}
    region = emacs_rebox.region
    interactions[region] = 'P'
    comment = emacs_rebox.comment
    interactions[comment] = 'P'
    set_default_style = emacs_rebox.set_default_style

#@+node:ekr.20071103093725.5: *7* class Emacs_Rebox
class Emacs_Rebox:
    @others
#@+node:ekr.20071103093725.6: *8* __init__ (Emacs_Rebox)

def __init__(self):
    self.default_style = None

#@+node:ekr.20071103093725.7: *8* set_default_style
def set_default_style(self, style):
    """\
Set the default style to STYLE.
"""
    self.default_style = style

#@+node:ekr.20071103093725.8: *8* region
def region(self, flag):
    """\
Rebox the boxed comment in the current region, obeying FLAG.
"""
    self.emacs_engine(flag, self.find_region)

#@+node:ekr.20071103093725.9: *8* comment
def comment(self, flag):
    """\
Rebox the surrounding boxed comment, obeying FLAG.
"""
    self.emacs_engine(flag, self.find_comment)

#@+node:ekr.20071103093725.10: *8* emacs_engine
def emacs_engine(self, flag, find_limits):
    """\
Rebox text while obeying FLAG.  Call FIND_LIMITS to discover the extent
of the boxed comment.
"""
    # `C-u -' means that box style is to be decided interactively.
    if flag == lisp['-']:
        flag = self.ask_for_style()
    # If FLAG is zero or negative, only change default box style.
    if type(flag) is type(0) and flag <= 0:
        self.default_style = -flag
        lisp.message("Default style set to %d" % -flag)
        return
    # Decide box style and refilling.
    if flag is None:
        style = self.default_style
        refill = 1
    elif type(flag) == type(0):
        if self.default_style is None:
            style = flag
        else:
            style = merge_styles(self.default_style, flag)
        refill = 1
    else:
        flag = flag.copy()
        if type(flag) == type([]):
            style = self.default_style
            refill = 0
        else:
            lisp.error("Unexpected flag value %s" % flag)
    # Prepare for reboxing.
    lisp.message("Reboxing...")
    checkpoint = lisp.buffer_undo_list.value()
    start, end = find_limits()
    text = lisp.buffer_substring(start, end)
    width = lisp.fill_column.value()
    tabify = lisp.indent_tabs_mode.value() is not None
    point = lisp.point()
    if start <= point < end:
        position = point - start
    else:
        position = None
    # Rebox the text and replace it in Emacs buffer.
    old_style, new_style, text, position = engine(
        text, style=style, width=width,
        refill=refill, tabify=tabify, position=position)
    if text is None:
        lisp.error("Cannot rebox to style %d" % new_style)
    lisp.delete_region(start, end)
    lisp.insert(text)
    if position is not None:
        lisp.goto_char(start + position)
    # Collapse all operations into a single one, for Undo.
    self.clean_undo_after(checkpoint)
    # We are finished, tell the user.
    if old_style == new_style:
        lisp.message("Reboxed with style %d" % old_style)
    else:
        lisp.message("Reboxed from style %d to %d"
                     % (old_style, new_style))

#@+node:ekr.20071103093725.11: *8* ask_for_style
def ask_for_style(self):
    """\
Request the style interactively, using the minibuffer.
"""
    language = quality = type = None
    while language is None:
        lisp.message("\
Box language is 100-none, 200-/*, 300-//, 400-#, 500-;, 600-%%")
        key = lisp.read_char()
        if key >= ord('0') and key <= ord('6'):
            language = key - ord('0')
    while quality is None:
        lisp.message("\
Box quality/width is 10-simple/1, 20-rounded/2, 30-starred/3 or 40-starred/4")
        key = lisp.read_char()
        if key >= ord('0') and key <= ord('4'):
            quality = key - ord('0')
    while type is None:
        lisp.message("\
Box type is 1-opened, 2-half-single, 3-single, 4-half-double or 5-double")
        key = lisp.read_char()
        if key >= ord('0') and key <= ord('5'):
            type = key - ord('0')
    return 100*language + 10*quality + type

#@+node:ekr.20071103093725.12: *8* find_region
def find_region(self):
    """\
Return the limits of the region.
"""
    return lisp.point(), lisp.mark(lisp.t)

#@+node:ekr.20071103093725.13: *8* find_comment
def find_comment(self):
    """\
Find and return the limits of the block of comments following or enclosing
the cursor, or return an error if the cursor is not within such a block
of comments.  Extend it as far as possible in both directions.
"""
    let = Let()
    let.push_excursion()
    # Find the start of the current or immediately following comment.
    lisp.beginning_of_line()
    lisp.skip_chars_forward(' \t\n')
    lisp.beginning_of_line()
    if not language_matcher[0](self.remainder_of_line()):
        temp = lisp.point()
        if not lisp.re_search_forward('\\*/', None, lisp.t):
            lisp.error("outside any comment block")
        lisp.re_search_backward('/\\*')
        if lisp.point() > temp:
            lisp.error("outside any comment block")
        temp = lisp.point()
        lisp.beginning_of_line()
        lisp.skip_chars_forward(' \t')
        if lisp.point() != temp:
            lisp.error("text before start of comment")
        lisp.beginning_of_line()
    start = lisp.point()
    language = guess_language(self.remainder_of_line())
    # Find the end of this comment.
    if language == 2:
        lisp.search_forward('*/')
        if not lisp.looking_at('[ \t]*$'):
            lisp.error("text after end of comment")
    lisp.end_of_line()
    if lisp.eobp():
        lisp.insert('\n')
    else:
        lisp.forward_char(1)
    end = lisp.point()
    # Try to extend the comment block backwards.
    lisp.goto_char(start)
    while not lisp.bobp():
        if language == 2:
            lisp.skip_chars_backward(' \t\n')
            if not lisp.looking_at('[ \t]*\n[ \t]*/\\*'):
                break
            if lisp.point() < 2:
                break
            lisp.backward_char(2)
            if not lisp.looking_at('\\*/'):
                break
            lisp.re_search_backward('/\\*')
            temp = lisp.point()
            lisp.beginning_of_line()
            lisp.skip_chars_forward(' \t')
            if lisp.point() != temp:
                break
            lisp.beginning_of_line()
        else:
            lisp.previous_line(1)
            if not language_matcher[language](self.remainder_of_line()):
                break
        start = lisp.point()
    # Try to extend the comment block forward.
    lisp.goto_char(end)
    while language_matcher[language](self.remainder_of_line()):
        if language == 2:
            lisp.re_search_forward('[ \t]*/\\*')
            lisp.re_search_forward('\\*/')
            if lisp.looking_at('[ \t]*$'):
                lisp.beginning_of_line()
                lisp.forward_line(1)
                end = lisp.point()
        else:
            lisp.forward_line(1)
            end = lisp.point()
    return start, end

#@+node:ekr.20071103093725.14: *8* remainder_of_line
def remainder_of_line(self):
    """\
Return all characters between point and end of line in Emacs buffer.
"""
    return lisp('''\
(buffer-substring (point) (save-excursion (skip-chars-forward "^\n") (point)))
''')

#@+node:ekr.20071103093725.15: *8* clean_undo_after_old
def clean_undo_after_old(self, checkpoint):
    """\
Remove all intermediate boundaries from the Undo list since CHECKPOINT.
"""
    # Declare some LISP functions.
    car = lisp.car
    cdr = lisp.cdr
    eq = lisp.eq
    setcdr = lisp.setcdr
    # Remove any `nil' delimiter recently added to the Undo list.
    cursor = lisp.buffer_undo_list.value()
    if not eq(cursor, checkpoint):
        tail = cdr(cursor)
        while not eq(tail, checkpoint):
            if car(tail):
                cursor = tail
                tail = cdr(cursor)
            else:
                tail = cdr(tail)
                setcdr(cursor, tail)

#@+node:ekr.20071103093725.16: *8* clean_undo_after
def clean_undo_after(self, checkpoint):
    """\
Remove all intermediate boundaries from the Undo list since CHECKPOINT.
"""
    lisp("""
(let ((undo-list %s))
(if (not (eq buffer-undo-list undo-list))
  (let ((cursor buffer-undo-list))
(while (not (eq (cdr cursor) undo-list))
  (if (car (cdr cursor))
      (setq cursor (cdr cursor))
    (setcdr cursor (cdr (cdr cursor)))))))
nil)
"""
         % (checkpoint or 'nil'))

#@+node:ekr.20071103093725.17: *7* engine
def engine(text, style=None, width=79, refill=1, tabify=0, position=None):
    """\
Add, delete or adjust a boxed comment held in TEXT, according to STYLE.
STYLE values are explained at beginning of this file.  Any zero attribute
in STYLE indicates that the corresponding attribute should be recovered
from the currently existing box.  Produced lines will not go over WIDTH
columns if possible, if refilling gets done.  But if REFILL is false, WIDTH
is ignored.  If TABIFY is true, the beginning of produced lines will have
spaces replace by TABs.  POSITION is either None, or a character position
within TEXT.  Returns four values: the old box style, the new box style,
the reformatted text, and either None or the adjusted value of POSITION in
the new text.  The reformatted text is returned as None if the requested
style does not exist.
"""
    last_line_complete = text and text[-1] == '\n'
    if last_line_complete:
        text = text[:-1]
    lines = string.split(string.expandtabs(text), '\n')
    # Decide about refilling and the box style to use.
    new_style = 111
    old_template = guess_template(lines)
    new_style = merge_styles(new_style, old_template.style)
    if style is not None:
        new_style = merge_styles(new_style, style)
    new_template = template_registry.get(new_style)
    # Interrupt processing if STYLE does not exist.
    if not new_template:
        return old_template.style, new_style, None, None
    # Remove all previous comment marks, and left margin.
    if position is not None:
        marker = Marker()
        marker.save_position(text, position, old_template.characters())
    lines, margin = old_template.unbuild(lines)
    # Ensure only one white line between paragraphs.
    counter = 1
    while counter < len(lines) - 1:
        if lines[counter] == '' and lines[counter-1] == '':
            del lines[counter]
        else:
            counter = counter + 1
    # Rebuild the boxed comment.
    lines = new_template.build(lines, width, refill, margin)
    # Retabify to the left only.
    if tabify:
        for counter in range(len(lines)):
            tabs = len(re.match(' *', lines[counter]).group()) / 8
            lines[counter] = '\t' * tabs + lines[counter][8*tabs:]
    # Restore the point position.
    text = string.join(lines, '\n')
    if last_line_complete:
        text = text + '\n'
    if position is not None:
        position = marker.get_position(text, new_template.characters())
    return old_template.style, new_style, text, position

#@+node:ekr.20071103093725.18: *7* guess_language
def guess_language(line):
    """\
Guess the language in use for LINE.
"""
    for language in range(len(language_matcher) - 1, 1, -1):
        if language_matcher[language](line):
            return language
    return 1

#@+node:ekr.20071103093725.19: *7* guess_template
def guess_template(lines):
    """\
Find the heaviest box template matching LINES.
"""
    best_template = None
    for template in template_registry.values():
        if best_template is None or template > best_template:
            if template.match(lines):
                best_template = template
    return best_template

#@+node:ekr.20071103093725.20: *7* left_margin_size
def left_margin_size(lines):
    """\
Return the width of the left margin for all LINES.  Ignore white lines.
"""
    margin = None
    for line in lines:
        counter = len(re.match(' *', line).group())
        if counter != len(line):
            if margin is None or counter < margin:
                margin = counter
    if margin is None:
        margin = 0
    return margin

#@+node:ekr.20071103093725.21: *7* merge_styles
def merge_styles(original, update):
    """\
Return style attributes as per ORIGINAL, in which attributes have been
overridden by non-zero corresponding style attributes from UPDATE.
"""
    style = [original / 100, original / 10 % 10, original % 10]
    merge = update / 100, update / 10 % 10, update % 10
    for counter in range(3):
        if merge[counter]:
            style[counter] = merge[counter]
    return 100*style[0] + 10*style[1] + style[2]

#@+node:ekr.20071103093725.22: *7* refill_lines
def refill_lines(lines, width):
    """\
Refill LINES, trying to not produce lines having more than WIDTH columns.
"""
    # Try using GNU `fmt'.
    import tempfile, os
    name = tempfile.mktemp()
    open(name, 'w').write(string.join(lines, '\n') + '\n')
    process = os.popen('fmt -cuw %d %s' % (width, name))
    text = process.read()
    os.remove(name)
    if process.close() is None:
        return map(string.expandtabs, string.split(text, '\n')[:-1])
    # If `fmt' failed, do refilling more naively, wihtout using the
    # Knuth algorithm, nor protecting full stops at end of sentences.
    lines.append(None)
    new_lines = []
    new_line = ''
    start = 0
    for end in range(len(lines)):
        if not lines[end]:
            margin = left_margin_size(lines[start:end])
            for line in lines[start:end]:
                counter = len(re.match(' *', line).group())
                if counter > margin:
                    if new_line:
                        new_lines.append(' ' * margin + new_line)
                        new_line = ''
                    indent = counter - margin
                else:
                    indent = 0
                for word in string.split(line):
                    if new_line:
                        if len(new_line) + 1 + len(word) > width:
                            new_lines.append(' ' * margin + new_line)
                            new_line = word
                        else:
                            new_line = new_line + ' ' + word
                    else:
                        new_line = ' ' * indent + word
                        indent = 0
            if new_line:
                new_lines.append(' ' * margin + new_line)
                new_line = ''
            if lines[end] is not None:
                new_lines.append('')
                start = end + 1
    return new_lines

#@+node:ekr.20071103093725.23: *7* class Marker
class Marker:
    @others
#@+node:ekr.20071103093725.24: *8* save_position

## Heuristic to simulate a marker while reformatting boxes.

def save_position(self, text, position, ignorable):
    """\
Given a TEXT and a POSITION in that text, save the adjusted position
by faking that all IGNORABLE characters before POSITION were removed.
"""
    ignore = {}
    for character in ' \t\r\n' + ignorable:
        ignore[character] = None
    counter = 0
    for character in text[:position]:
        if character in ignore:
            counter = counter + 1
    self.position = position - counter

#@+node:ekr.20071103093725.25: *8* get_position
def get_position(self, text, ignorable, latest=0):
    """\
Given a TEXT, return the value that would yield the currently saved position,
if it was saved by `save_position' with IGNORABLE.  Unless the position lies
within a series of ignorable characters, LATEST has no effect in practice.
If LATEST is true, return the biggest possible value instead of the smallest.
"""
    ignore = {}
    for character in ' \t\r\n' + ignorable:
        ignore[character] = None
    counter = 0
    position = 0
    if latest:
        for character in text:
            if character in ignore:
                counter = counter + 1
            else:
                if position == self.position:
                    break
                position = position + 1
    elif self.position > 0:
        for character in text:
            if character in ignore:
                counter = counter + 1
            else:
                position = position + 1
                if position == self.position:
                    break
    return position + counter

#@+node:ekr.20071103093725.26: *7* class Template
## Template processing.

class Template:
    @others
#@+node:ekr.20071103093725.27: *8* __init_ (Template)_

def __init__(self, style, weight, lines):
    """\
Digest and register a single template.  The template is numbered STYLE,
has a parsing WEIGHT, and is described by one to three LINES.
STYLE should be used only once through all `declare_template' calls.

One of the lines should contain the substring `box' to represent the comment
to be boxed, and if three lines are given, `box' should appear in the middle
one.  Lines containing only spaces are implied as necessary before and after
the the `box' line, so we have three lines.

Normally, all three template lines should be of the same length.  If the first
line is shorter, it represents a start comment string to be bundled within the
first line of the comment text.  If the third line is shorter, it represents
an end comment string to be bundled at the end of the comment text, and
refilled with it.
"""
    assert style not in template_registry, \
           "Style %d defined more than once" % style
    self.style = style
    self.weight = weight
    # Make it exactly three lines, with `box' in the middle.
    start = string.find(lines[0], 'box')
    if start >= 0:
        line1 = None
        line2 = lines[0]
        if len(lines) > 1:
            line3 = lines[1]
        else:
            line3 = None
    else:
        start = string.find(lines[1], 'box')
        if start >= 0:
            line1 = lines[0]
            line2 = lines[1]
            if len(lines) > 2:
                line3 = lines[2]
            else:
                line3 = None
        else:
            assert 0, "Erroneous template for %d style" % style
    end = start + len('box')
    # Define a few booleans.
    self.merge_nw = line1 is not None and len(line1) < len(line2)
    self.merge_se = line3 is not None and len(line3) < len(line2)
    # Define strings at various cardinal directions.
    if line1 is None:
        self.nw = self.nn = self.ne = None
    elif self.merge_nw:
        self.nw = line1
        self.nn = self.ne = None
    else:
        if start > 0:
            self.nw = line1[:start]
        else:
            self.nw = None
        if line1[start] != ' ':
            self.nn = line1[start]
        else:
            self.nn = None
        if end < len(line1):
            self.ne = string.rstrip(line1[end:])
        else:
            self.ne = None
    if start > 0:
        self.ww = line2[:start]
    else:
        self.ww = None
    if end < len(line2):
        self.ee = line2[end:]
    else:
        self.ee = None
    if line3 is None:
        self.sw = self.ss = self.se = None
    elif self.merge_se:
        self.sw = self.ss = None
        self.se = string.rstrip(line3)
    else:
        if start > 0:
            self.sw = line3[:start]
        else:
            self.sw = None
        if line3[start] != ' ':
            self.ss = line3[start]
        else:
            self.ss = None
        if end < len(line3):
            self.se = string.rstrip(line3[end:])
        else:
            self.se = None
    # Define parsing regexps.
    if self.merge_nw:
        self.regexp1 = re.compile(' *' + regexp_quote(self.nw) + '.*$')
    elif self.nw and not self.nn and not self.ne:
        self.regexp1 = re.compile(' *' + regexp_quote(self.nw) + '$')
    elif self.nw or self.nn or self.ne:
        self.regexp1 = re.compile(
            ' *' + regexp_quote(self.nw) + regexp_ruler(self.nn)
            + regexp_quote(self.ne) + '$')
    else:
        self.regexp1 = None
    if self.ww or self.ee:
        self.regexp2 = re.compile(
            ' *' + regexp_quote(self.ww) + '.*'
            + regexp_quote(self.ee) + '$')
    else:
        self.regexp2 = None
    if self.merge_se:
        self.regexp3 = re.compile('.*' + regexp_quote(self.se) + '$')
    elif self.sw and not self.ss and not self.se:
        self.regexp3 = re.compile(' *' + regexp_quote(self.sw) + '$')
    elif self.sw or self.ss or self.se:
        self.regexp3 = re.compile(
            ' *' + regexp_quote(self.sw) + regexp_ruler(self.ss)
            + regexp_quote(self.se) + '$')
    else:
        self.regexp3 = None
    # Save results.
    template_registry[style] = self

#@+node:ekr.20071103093725.28: *8* __cmp__
def __cmp__(self, other):
    return cmp(self.weight, other.weight)

#@+node:ekr.20071103093725.29: *8* characters
def characters(self):
    """\
Return a string of characters which may be used to draw the box.
"""
    characters = ''
    for text in (self.nw, self.nn, self.ne,
                 self.ww, self.ee,
                 self.sw, self.ss, self.se):
        if text:
            for character in text:
                if character not in characters:
                    characters = characters + character
    return characters

#@+node:ekr.20071103093725.30: *8* match
def match(self, lines):
    """\
Returns true if LINES exactly match this template.
"""
    start = 0
    end = len(lines)
    if self.regexp1 is not None:
        if start == end or not self.regexp1.match(lines[start]):
            return 0
        start = start + 1
    if self.regexp3 is not None:
        if end == 0 or not self.regexp3.match(lines[end-1]):
            return 0
        end = end - 1
    if self.regexp2 is not None:
        for line in lines[start:end]:
            if not self.regexp2.match(line):
                return 0
    return 1

#@+node:ekr.20071103093725.31: *8* unbuild
def unbuild(self, lines):
    """\
Remove all comment marks from LINES, as hinted by this template.  Returns the
cleaned up set of lines, and the size of the left margin.
"""
    margin = left_margin_size(lines)
    # Remove box style marks.
    start = 0
    end = len(lines)
    if self.regexp1 is not None:
        lines[start] = unbuild_clean(lines[start], self.regexp1)
        start = start + 1
    if self.regexp3 is not None:
        lines[end-1] = unbuild_clean(lines[end-1], self.regexp3)
        end = end - 1
    if self.regexp2 is not None:
        for counter in range(start, end):
            lines[counter] = unbuild_clean(lines[counter], self.regexp2)
    # Remove the left side of the box after it turned into spaces.
    delta = left_margin_size(lines) - margin
    for counter in range(len(lines)):
        lines[counter] = lines[counter][delta:]
    # Remove leading and trailing white lines.
    start = 0
    end = len(lines)
    while start < end and lines[start] == '':
        start = start + 1
    while end > start and lines[end-1] == '':
        end = end - 1
    return lines[start:end], margin

#@+node:ekr.20071103093725.32: *8* build
def build(self, lines, width, refill, margin):
    """\
Put LINES back into a boxed comment according to this template, after
having refilled them if REFILL.  The box should start at column MARGIN,
and the total size of each line should ideally not go over WIDTH.
"""
    # Merge a short end delimiter now, so it gets refilled with text.
    if self.merge_se:
        if lines:
            lines[-1] = lines[-1] + '  ' + self.se
        else:
            lines = [self.se]
    # Reduce WIDTH according to left and right inserts, then refill.
    if self.ww:
        width = width - len(self.ww)
    if self.ee:
        width = width - len(self.ee)
    if refill:
        lines = refill_lines(lines, width)
    # Reduce WIDTH further according to the current right margin,
    # and excluding the left margin.
    maximum = 0
    for line in lines:
        if line:
            if line[-1] in '.!?':
                length = len(line) + 1
            else:
                length = len(line)
            if length > maximum:
                maximum = length
    width = maximum - margin
    # Construct the top line.
    if self.merge_nw:
        lines[0] = ' ' * margin + self.nw + lines[0][margin:]
        start = 1
    elif self.nw or self.nn or self.ne:
        if self.nn:
            line = self.nn * width
        else:
            line = ' ' * width
        if self.nw:
            line = self.nw + line
        if self.ne:
            line = line + self.ne
        lines.insert(0, string.rstrip(' ' * margin + line))
        start = 1
    else:
        start = 0
    # Construct all middle lines.
    for counter in range(start, len(lines)):
        line = lines[counter][margin:]
        line = line + ' ' * (width - len(line))
        if self.ww:
            line = self.ww + line
        if self.ee:
            line = line + self.ee
        lines[counter] = string.rstrip(' ' * margin + line)
    # Construct the bottom line.
    if self.sw or self.ss or self.se and not self.merge_se:
        if self.ss:
            line = self.ss * width
        else:
            line = ' ' * width
        if self.sw:
            line = self.sw + line
        if self.se and not self.merge_se:
            line = line + self.se
        lines.append(string.rstrip(' ' * margin + line))
    return lines

#@+node:ekr.20071103093725.33: *7* regexp_quote
def regexp_quote(text):
    """\
Return a regexp matching TEXT without its surrounding space, maybe
followed by spaces.  If STRING is nil, return the empty regexp.
Unless spaces, the text is nested within a regexp parenthetical group.
"""
    if text is None:
        return ''
    if text == ' ' * len(text):
        return ' *'
    return '(' + re.escape(text.strip() + ') *'

#@+node:ekr.20071103093725.34: *7* regexp_ruler
def regexp_ruler(character):
    """\
Return a regexp matching two or more repetitions of CHARACTER, maybe
followed by spaces.  Is CHARACTER is nil, return the empty regexp.
Unless spaces, the ruler is nested within a regexp parenthetical group.
"""
    if character is None:
        return ''
    if character == ' ':
        return '  +'
    return '(' + re.escape(character + character) + '+) *'

#@+node:ekr.20071103093725.35: *7* unbuild_clean
def unbuild_clean(line, regexp):
    """\
Return LINE with all parenthetical groups in REGEXP erased and replaced by an
equivalent number of spaces, except for trailing spaces, which get removed.
"""
    match = re.match(regexp, line)
    groups = match.groups()
    for counter in range(len(groups)):
        if groups[counter] is not None:
            start, end = match.span(1 + counter)
            line = line[:start] + ' ' * (end - start) + line[end:]
    return string.rstrip(line)

#@+node:ekr.20071103093725.36: *7* make_generic
## Template data.

# Matcher functions for a comment start, indexed by numeric LANGUAGE.
language_matcher = []
for pattern in (r' *(/\*|//+|#+|;+|%+)',
                r'',            # 1
                r' */\*',       # 2
                r' *//+',       # 3
                r' *#+',        # 4
                r' *;+',        # 5
                r' *%+'):       # 6
    language_matcher.append(re.compile(pattern).match)

# Template objects, indexed by numeric style.
template_registry = {}

def make_generic(style, weight, lines):
    """\
Add various language digit to STYLE and generate one template per language,
all using the same WEIGHT.  Replace `?' in LINES accordingly.
"""
    for language, character in ((300, '/'),  # C++ style comments
                                (400, '#'),  # scripting languages
                                (500, ';'),  # LISP and assembler
                                (600, '%')): # TeX and PostScript
        new_style = language + style
        if 310 < new_style <= 319:
            # Disallow quality 10 with C++.
            continue
        new_lines = []
        for line in lines:
            new_lines.append(string.replace(line, '?', character))
        Template(new_style, weight, new_lines)

#@+node:ekr.20071103093725.37: *7* << templates >>

make_generic(11, 115, ('? box',))

make_generic(12, 215, ('? box ?',
                       '? --- ?'))

make_generic(13, 315, ('? --- ?',
                       '? box ?',
                       '? --- ?'))

make_generic(14, 415, ('? box ?',
                       '???????'))

make_generic(15, 515, ('???????',
                       '? box ?',
                       '???????'))

make_generic(21, 125, ('?? box',))

make_generic(22, 225, ('?? box ??',
                       '?? --- ??'))

make_generic(23, 325, ('?? --- ??',
                       '?? box ??',
                       '?? --- ??'))

make_generic(24, 425, ('?? box ??',
                       '?????????'))

make_generic(25, 525, ('?????????',
                       '?? box ??',
                       '?????????'))

make_generic(31, 135, ('??? box',))

make_generic(32, 235, ('??? box ???',
                       '??? --- ???'))

make_generic(33, 335, ('??? --- ???',
                       '??? box ???',
                       '??? --- ???'))

make_generic(34, 435, ('??? box ???',
                       '???????????'))

make_generic(35, 535, ('???????????',
                       '??? box ???',
                       '???????????'))

make_generic(41, 145, ('???? box',))

make_generic(42, 245, ('???? box ????',
                       '???? --- ????'))

make_generic(43, 345, ('???? --- ????',
                       '???? box ????',
                       '???? --- ????'))

make_generic(44, 445, ('???? box ????',
                       '?????????????'))

make_generic(45, 545, ('?????????????',
                       '???? box ????',
                       '?????????????'))

# Textual (non programming) templates.

Template(111, 113, ('box',))

Template(112, 213, ('| box |',
                    '+-----+'))

Template(113, 313, ('+-----+',
                    '| box |',
                    '+-----+'))

Template(114, 413, ('| box |',
                    '*=====*'))

Template(115, 513, ('*=====*',
                    '| box |',
                    '*=====*'))

Template(121, 123, ('| box |',))

Template(122, 223, ('| box |',
                    '`-----\''))

Template(123, 323, ('.-----.',
                    '| box |',
                    '`-----\''))

Template(124, 423, ('| box |',
                    '\\=====/'))

Template(125, 523, ('/=====\\',
                    '| box |',
                    '\\=====/'))

Template(141, 143, ('| box ',))

Template(142, 243, ('* box *',
                    '*******'))

Template(143, 343, ('*******',
                    '* box *',
                    '*******'))

Template(144, 443, ('X box X',
                    'XXXXXXX'))

Template(145, 543, ('XXXXXXX',
                    'X box X',
                    'XXXXXXX'))
# C language templates.

Template(211, 118, ('/* box */',))

Template(212, 218, ('/* box */',
                    '/* --- */'))

Template(213, 318, ('/* --- */',
                    '/* box */',
                    '/* --- */'))

Template(214, 418, ('/* box */',
                    '/* === */'))

Template(215, 518, ('/* === */',
                    '/* box */',
                    '/* === */'))

Template(221, 128, ('/* ',
                    '   box',
                    '*/'))

Template(222, 228, ('/*    .',
                    '| box |',
                    '`----*/'))

Template(223, 328, ('/*----.',
                    '| box |',
                    '`----*/'))

Template(224, 428, ('/*    \\',
                    '| box |',
                    '\\====*/'))

Template(225, 528, ('/*====\\',
                    '| box |',
                    '\\====*/'))

Template(231, 138, ('/*    ',
                    ' | box',
                    ' */   '))

Template(232, 238, ('/*        ',
                    ' | box | ',
                    ' *-----*/'))

Template(233, 338, ('/*-----* ',
                    ' | box | ',
                    ' *-----*/'))

Template(234, 438, ('/* box */',
                    '/*-----*/'))

Template(235, 538, ('/*-----*/',
                    '/* box */',
                    '/*-----*/'))

Template(241, 148, ('/*    ',
                    ' * box',
                    ' */   '))

Template(242, 248, ('/*     * ',
                    ' * box * ',
                    ' *******/'))

Template(243, 348, ('/******* ',
                    ' * box * ',
                    ' *******/'))

Template(244, 448, ('/* box */',
                    '/*******/'))

Template(245, 548, ('/*******/',
                    '/* box */',
                    '/*******/'))

Template(251, 158, ('/* ',
                    ' * box',
                    ' */   '))

#@+node:ekr.20070215183046: *3* IronPython notes
@nocolor

- IronPython does not accept 'from __future__ import x'
  I could work around this, but perhaps it is time to abandon Python 2.2.2.

- Amazingly, it is possible to add Python24\Lib to IronPython's path!
  Almost all of those modules import correct.

- IronPython has troubles with the xml modules.
  It complains about a missing 'strict' codec.

- IronPython has trouble with the pdb module, so some other way must be found to debug IronPython programs.
#@+node:ekr.20050214055018: *3* Mac Notes
#@+node:ekr.20050221054932: *4* How to make monolithic Leo app on MacOS X
 @killcolor
http://sourceforge.net/forum/message.php?msg_id=3007062
By: jgleeson

Sorry to take so long to reply.  I've been buried in work and haven't kept up
with some email.

Here's the link to the site where I posted the folder you have:
<http://homepage.mac.com/jdgleeson/>  It's the small file named "Leo.zip" (23
KB), not the large file "Leo-4.3-alpha-2.dmg" (20 MB).

I agree that I did not write very clear instructions, beginnng with the first
step, where I should have also said:  "It is important to use version 1.1.8
of py2app, which is only available through svn.  The version on the py2app website
is 1.1.7, which creates buggy Tkinter apps. If you try to use version 1.1.7,
the Leo app it creates will give you a message saying that Tkinter is not properly
installed.  Your installation is fine; otherwise you could not have even built
Leo.app with py2app, because py2app copies the essential parts of Tcl/Tk into
the application bundle to make the app completely standalone."

I haven't tried intalling the Fink subversion -- I'm using DarwinPorts
<http://darwinports.opendarwin.org/>.  But there's a simpler alternative than
DarwinPorts. Metissian releases OS X packages of Subversion clients
<http://metissian.com/projects/macosx/subversion/>

AFAIK, the command "python setup.py bdist_mpkg --open" only applies to the py2app
1.1.8 distribution.  By the way, bdist_mpkg is distributed with py2app. It creates
a package around the setup.py script (more specialized than Platypus).  I don't
have any experience with bdist_mpkg yet.

'Copy the leo folder into this directory' is horrible. I'm glad you figured
it out -- I'm not sure I could have.

"python setup.py py2app -a" should be run in the folder with the readme file,
which also contains the setup.py file that the command refers to.  Most importantly,
the folder in which this command is run must contain the leo folder -- which
it does only if you are brilliant enough to decode my instructions.   ;) 

HTH

-John
#@+node:ekr.20050214055018.4: *4* @url http://idisk.mac.com/genthaler-Public/Leo.zip (download)
#@+node:ekr.20050214055018.5: *4* @url http://www.wordtech-software.com/leo.html  (Mac Bundle)
#@+node:ekr.20050513164506: *4* Problems with run script command on Mac x11
@killcolor

Jon Schull <jschull@softlock.com>  
Date:  2003/12/30 Tue PM 05:50:51 EST 
To:  edreamleo@charter.net 
Subject:  Leo, Mac OS X 10.3, and VPython 

I've been evaluating leo or vpython programming on  Mac OS X 10.3, and 
have some observations and a suggestion.

Observations:
- Leo runs under X11 as well as under OS X.
- My X11 python configuration was created using the recipe at XXX (which enables vpython).
- The OS X configuration is vanilla MacPython from MacPython.org, along with AquaTclTk batteries included XXX.

In both environments I can run leo under python leo.py and under idle.
Under OS X we get font smoothing, but we can't run visual python programs (python crashes;  this is a known incompatibility with  MacPython.)

- Under X11 we can run visual python programs like this one
    #box.py
    from visual import *
    box()

And we can even run them under leo (under X11). HOWEVER, when the visual python program is terminated, leo vanishes (leo and the vp program apparently run in the same space)

Under x11, we can keep leo alive by putting the vp program in its own space:

    os.popen3('/sw/bin/python /Users/jis/box.py')

However,  this doesn't let us see the output of stderr and stdout.  
Those text streams are available...

    def do(cmd='ls'):
        from os import popen3
        pIn,pOut,pErr=0,1,2
        popenResults=popen3(cmd)
        print popenResults[pOut].read()
        print popenResults[pErr].read()

    import os
    do('/sw/bin/python /Users/jis/box.py')

...but only when the vpython program terminates.

Here's the good news:  if we execute our vp program with 
/sw/bin/idle.py rather than with python, we get to see the program 
output in real time (under idle, under X11).

    import os
    os.chdir('/sw/lib/python2.3/idlelib')
    os.popen3('/sw/bin/python idle.py -r /Users/jis/box.py')

#this runs as an executed script in leo, and produces a live idle 
with real time ongoing output.

Now, while idle is running, leo sits in suspended animation.  But when 
the vpython program terminates, we are left in idle, and when idle is 
terminated, leo becomes active again.

It would be even better if leo were not suspended (using os.spawn, 
perhaps) but the real point is that I would really really like leo's 
"Execute script" command to execute code this way and spare me having 
to  hard-write the path to box.py.  It ought to be possible to 
eliminate os.chdir as well.

------------------
Jon Schull, Ph.D.
Associate Professor
Information Technology
Rochester Institute of Technology
schull@digitalgoods.com 585-738-6696
#@+node:ekr.20040104162835.8: *4* Linux/Mac notes: Dan Winkler
#@+node:ekr.20040104162835.13: *5* Fink & aqua
Yes, fink does have pre-built Pythons, both 2.1 and 2.2.  (If you don't 
see them it probably means you don't have the right servers listed in 
your /sw/etc/apt/sources.list file.)  However, the versions of Python 
you'd get through fink are set up to run under X Windows, which I don't 
think is what you want.

I think what you want is MacPython which can run Tk programs like Leo 
under Aqua.  That's what I use these days.

I can tell from your question that you don't understand the following 
differences between the versions of Python available:

1) The version that comes with OS X is a text only one which doesn't 
have Tk.  Leo can't run under that.  Also, I hate Apple for including 
this instead of one that does have Tk and I hope they'll fix it some 
day.

2) You can get a version of Python from fink with has Tk but which runs 
under X Windows.  I don't think you want that.

3). You can also get MacPython which has Tk but it's a version of Tk 
that uses the Aqua windowing system, not X Windows.

So Tk can either be present or not and if it is present it can use 
either X Windows or Aqua.  You want it present and using Aqua, I think.


#@+node:ekr.20040104162835.14: *5* Mac, Fink, etc.
> 1. The python that FC installs is MacPython.  I think that because the
> MacPython docs talk about Fink.

Nope.  The python installed by FC knows nothing about the Mac.  It 
thinks it's running on a Unix machine.  And it uses a version of Tk 
which thinks it's running on a Unix machine.  The window standard on 
Unix is called X (or X11 or XFree86, all the same thing).  So the main 
reason to run Leo this way would be to get an idea of how it works for 
Unix/Linux users.  But when programs run under X, they don't look like 
Mac programs.  They don't get all those glossy, translucent widgets 
that Aqua provides.  They really look like they would on a Unix/Linux 
machine.

Aqua is the native windowing system on Mac.  MacPython is set up to 
work with it.  Most Mac users will want Leo to work this way.  That's 
what I do.

>
>
> I have the TkTclAquBI (Batteries included) installer.  Is installing 
> this
> enough to get Leo to work with Aqua?  Do I have to de-install the
> present tk stuff that I installed with FC?

Yes, I think that's all I installed to get Tk to work under Aqua.  You 
don't have to deinstall the FC stuff.  All the FC stuff lives in its 
own world under /sw and runs under X.  It won't conflict with the Mac 
world.

#@+node:ekr.20040104162835.15: *5* Double clicking on Linux
Double-clickable things (i.e. Macintosh applications) are usually 
actually folders with a name that ends in .app.  The file you found is 
probably executable only from the command line, not by double clicking 
it.  So I think if you run it from the command line it will work but 
will not know about Tk because Apple's version was built without Tk 
support.

You can also execute the .app programs from the command line by using 
the open command, so "open foo.app" will do the same thing as double 
clicking on foo in the finder (the .app extension is suppressed).  The 
idea behind this is that an application can look like just one opaque 
icon in the finder but actually have all its resources nicely organized 
in subfolders.
#@+node:ekr.20060111112513.1: *3* New jyLeo notes
@nocolor
http://sourceforge.net/forum/message.php?msg_id=3516227
By: nobody

Some highlights:
* simpler startup:
jyleo leo.py
should be sufficient to start it up.
* new editor colorization
* the JythonShell is much more powerful and cooler
* new plugins
* Chapters support
* mod_script is in place.
* dyna-menu was converted.  I guess 'e' will have to judge the conversion.
* multi-language script support.
* drag and drop
* some powerful new editor commands.  Try keyword completing on the language
in effect.  Say if it is python:
se(Tab)
becomes
self

Some warnings:
1. Be careful about reading your regular leo files into jyleo and saving them.
Its quite conceivable that jyleo will write it out to an XML format that regular
leo can't handle.  Why?  Well jyleo is using an XML library to spit its XML
out while leo uses a home grown method.  The library can handle leo's XML, but
Ive seen regular leo not be able to handle jyleo's XML.  Its based around <tag/>
I believe.

2. If you move jyleo after executing it you will need to clear out your compiled
py files as the __file__ attribute is hard compiled into the resulting objects.
Not what we want.  We want it to be set at runtime.  Ive been waiting a long
time for jython to release again and hopefully fix this, but Im not holding
my breath anymore.

----------
Its hard to give this thing a number, I want to call it jyleo2, but jyleo is
sufficient.  Dependent upon bug reports the next release could be much sooner
than before, maybe even weeks.  I hope one thing, that the dreaded "I can't
get it to start" problems are gone.  I took the snapshot and expanded it in
Windows XP.  Went to the src directory and typed: jython leo.py
and it started.  That's what I wanted to see.  I didn't have to mess with the
CLASSPATH or anything.

things needed:
java 5
a jython2.2a1 or beyond.  jython2.2a1 is the most recent snapshot.

Beyond bug fixing, I will be planning to add more SwingMacs command as time
goes along.  But I think most major features are in place.  Of course the 3D
experiments in the future could change that... :D

A NOTE ON STARTUP TIMES: In my experience it takes awhile for jyleo to start.
It will take much longer the first time you execute it because the py files
are being compiled.  Ive haven't been able to figure out what eats the time,
it may just have a slow startup in the aggregate.  So don't think its not doing
anything, it probably is.

leouser
#@+node:ekr.20031218072017.392: *3* Python Notes...
#@+node:ekr.20031218072017.398: *4* How to call any Python method from the C API
In general, everything you can do in Python is accessible through the C API.

    lines = block.split('\n');

> That will be

    lines = PyObject_CallMethod(block, "split", "s", "\n");
#@+node:ekr.20031218072017.399: *4* How to run Python programs easily on NT,2K,XP
#@+node:ekr.20031218072017.400: *5* setting the PATHEXT env var
It is worth noting that NT, Win2K and XP all have an alternative which is
to add .PY to the PATHEXT environment variable. Then you can run any .PY
file directly just by typing the name of the script without the extension. 

e.g.
C:\>set PATHEXT=.COM;.EXE;.BAT;.CMD

C:\>set PATH=%PATH%;c:\python22\tools\Scripts

C:\>google
'google' is not recognized as an internal or external command,
operable program or batch file.

C:\>set PATHEXT=.COM;.EXE;.BAT;.CMD;.PY

C:\>google
Usage: C:\python22\tools\Scripts\google.py querystring

C:\>
#@+node:ekr.20031218072017.401: *5* Yet another Python .bat wrapper
>> It has a header of just one line. All the ugly stuff is at the end.
>>
>> -------------------------------------------------------------------
>> goto ="python"
>>
>> # Python code goes here
>>
>> ''' hybrid python/batch footer:
>> @:="python"
>> @python.exe %0 %1 %2 %3 %4 %5 %6 %7 %8 %9
>> @if errorlevel 9009 echo Python may be downloaded from
>www.python.org/download
>> @rem '''
>> -------------------------------------------------------------------
>>
>>         Oren
>>
>

It's for running python scripts on windows, without having to type:

[<path to python>\]python[.exe] <scriptname> [<arguments>*]

and almost takes the place of the "shabang" line at the top of *nix
scripts.

#@+node:ekr.20080110082845: *3* pyxides: code completion
@nocolor

Python code completion module


From: "Tal Einat" <talei...@gmail.com>
Date: Wed, 6 Jun 2007 20:57:18 +0300

I've been developing IDLE over the past 2 years or so. Even before
that, I helped a friend of mine, Noam Raphael, write IDLE's
auto-completion, which is included in recent versions of IDLE.

Noam wrote the original completion code from scratch, and AFAIK every
Python IDE which features code completion has done the same. Surely
there is -some- functionality which could be useful cross-IDE?
Retrieving possible completions from the namespace, for example. And
we should be learning from each-others' ideas and experiences.

So how about we design a generic Python completion module, that
each IDE could extend, and use for the completion logic?



From: "Ali Afshar" <aafs...@gmail.com>
Date: Wed, 6 Jun 2007 19:06:01 +0100

I am very keen for this. I will help where it is required. PIDA
currently has no code completion (outside what vim/emacs provide),



From: "phil jones" <inters...@gmail.com>
Date: Wed, 6 Jun 2007 11:07:33 -0700

What functions would we ask for a code completion module?

Presumably recognition of the beginnings of
- a) python keywords
- b) classes and functions defined earlier in this file?
- c) in scope variables?

As python is dynamically typed, I guess we can't expect to know the
names of methods of objects?



From: "Ali Afshar" <aafs...@gmail.com>
Date: Wed, 6 Jun 2007 19:13:10 +0100

> Presumably recognition of the beginnings of
> - a) python keywords
> - b) classes and functions defined earlier in this file?
> - c) in scope variables?

does c) include: d) imported modules



From: Nicolas Chauvat <nicolas.chau...@logilab.fr>
Date: Wed, 6 Jun 2007 20:17:30 +0200

> >Presumably recognition of the beginnings of
> >- a) python keywords
> >- b) classes and functions defined earlier in this file?
> >- c) in scope variables?

> does c) include: d) imported modules

For code-completion, I suppose astng[1] could be useful.

1: http://www.logilab.org/project/eid/856


From: Stani's Python Editor <spe.stani...@gmail.com>
Date: Wed, 06 Jun 2007 20:48:41 +0200

A good point. I think we all have been thinking about this. Important
issues for the design is the extraction method and the sources.

*the method*
Importing is a lazy, but accurate way of importing, but is security wise
not such a good idea. Parsing throught an AST compiler is better,
however more difficult. Here are two options.

From version 2.5 the standard Python compiler converts internally the
source code to an abstract syntax tree (AST) before producing the
bytecode. So probably that is a good way to go as every python
distribution has this battery included.

As Nicolas suggested earlier on this mailing list, there is another
option: the AST compiler in python or PyPy:

On Mar 14 2006, 12:16 am, Nicolas Chauvat <nicolas.chau...@logilab.fr>
wrote:

> > WingIDE use anASTgenerator written in C (but cross-platform),
> > lightningly quick, and open sourced. This could be a potential
> > starting point.

> > Additionally isn't Python2.5 planned to have a C-written compiler?

> PyPy also produced an improved parser/compiler.

> http://codespeak.net/pypy/dist/pypy/doc/index.html
> http://codespeak.net/pypy/dist/pypy/module/recparser/

But if it could be done with the standard one it is one dependency less.

*the sources*
In the design we could define first the sources:
1 external imported modules from the pythonpath
2 local modules relative to the current file or context dependent
(Blender, Gimp, ...)
3 inner code

For 1:
It might be a good idea to have a function which scans all the modules
from the pythonpath or one specific module to cache all autocompletion
and calltip information of all classes, methods and doc strings. Why?
Modules in the pythonpath don't change so often. With some criteria
(file name, time stamp, size, ...) you could check if updates are
necessary at startup. Having a readymade 'database' (could be python
dictionary or sqlite database) for autocompletion/call tips would speed
up things (and is also more secure if you are importing rather than
parsing. For example trying to provide gtk autocompletion in a wxPython
by importing is problematic).

For 2:
Here you load the parser on demand. Autocompletion/calltip information
can be added to the database.

For 3:
A different kind of parser needs to be used here as per definition code
you edit contains errors while typing. External modules are retrieved
from 1 and 2, for internal code you can scan all the words and add them
to the autocomplete database. As a refinement you can give special
attention to 'self'. Also for calltips you can inherit when there are
assignments, eg
frame = Frame()
than frame inherits autocomplete & calltip information from Frame.

So autocompletion & calltips deals with two steps: extraction and
'database'. If someone has a good parser already, we could use it.
Otherwise we can define an API for the extraction and maybe lazily
implement it first with importing and concentrate first on the
'database'. When the database is ready we can implement the parsing. You
could also implement the parsing first, but than it takes longer before
you have results. Of course the library is GUI independent, it only
works with strings or lists.

What concerns SPE, it uses importing for autocompletion (1+2) and does
internal code analysis for local code (however without the inheriting).

Tal, how does IDLE's autocompletion works?

Stani



From: Stani's Python Editor <spe.stani...@gmail.com>
Date: Wed, 06 Jun 2007 20:53:10 +0200

Nicolas Chauvat wrote:
> On Wed, Jun 06, 2007 at 07:13:10PM +0100, Ali Afshar wrote:
>>> Presumably recognition of the beginnings of
>>> - a) python keywords
>>> - b) classes and functions defined earlier in this file?
>>> - c) in scope variables?
>> does c) include: d) imported modules

> For code-completion, I suppose astng[1] could be useful.

> 1: http://www.logilab.org/project/eid/856

How dependent/independent is this from the standard AST compiler or
PyPy? Is it more IDE friendly? Is it based on it or a total independent
implementation?



From: "Ali Afshar" <aafs...@gmail.com>
Date: Wed, 6 Jun 2007 19:59:13 +0100

> A good point. I think we all have been thinking about this. Important
> issues for the design is the extraction method and the sources.

> *the method*
> Importing is a lazy, but accurate way of importing, but is security wise
> not such a good idea. Parsing throught an AST compiler is better,
> however more difficult. Here are two options.

> From version 2.5 the standard Python compiler converts internally the
> source code to an abstract syntax tree (AST) before producing the
> bytecode. So probably that is a good way to go as every python
> distribution has this battery included.

> As Nicolas suggested earlier on this mailing list, there is another
> option: the AST compiler in python or PyPy:

What concerns me about these is whether they would work in a module
which has a syntax error.

I believe Wing's compiler bit of their code completion is open source.
I remember having seen the code.



From: Stani <spe.stani...@gmail.com>
Date: Wed, 06 Jun 2007 12:08:00 -0700

> What concerns me about these is whether they would work in a module
> which has a syntax error.

> I believe Wing's compiler bit of their code completion is open source.
> I remember having seen the code.

It is indeed, but is implemented in C, which means an extra dependency
and not a 100% python solution. Normally modules (especially in the
pythonpath) which you import don't have syntax errors. Maybe logilabs
implementation handles syntax errors well as it is developed for
PyLint. Nicolas?



From: "Tal Einat" <talei...@gmail.com>
Date: Wed, 6 Jun 2007 22:34:41 +0300

> As python is dynamically typed, I guess we can't expect to know the
> names of methods of objects?

Well, the dir() builtin does just that, though there can be attributes
which won't be included therein. However, the builtin dir() can be
overridden... and ignoring it can break libraries like RPyC which
define a custom dir() function just for this purpose.

This issue has already been run in to by RPyC (an Python RPC lib). The
main developr went ahead and suggested adding a __dir__ method which
will return a list of attributes, and IIRC he has already implemented
a patch for this, and it will likely enter Python2.6.

Until then, I guess we're going to have to rely on dir for this.



From: "Josiah Carlson" <josiah.carl...@gmail.com>
Date: Wed, 6 Jun 2007 12:42:01 -0700

For reference, PyPE auto-parses source code in the background, generating
(among other things) a function/class/method hierarchy.  Its autocomplete
generally sticks to global functions and keywords, but when doing
self.method lookups, it checks the current source code line, looks up in its
index of classes/methods, and trims the results based on known methods in
the current class in the current source file.

It certainly isn't complete (it should try to check base classes of the
class in the same file, it could certainly pay attention to names assigned
in the current scope, the global scope, imports, types of objects as per
WingIDE's assert isinstance(obj, type), etc.), but it also makes the
computation fairly straightforward, fast, and only in reference to the
current document.



From: "Tal Einat" <talei...@gmail.com>
Date: Wed, 6 Jun 2007 22:52:08 +0300

> Tal, how does IDLE's autocompletion works?

Much like Stani said, since Python is interpreted, collection of
possible completions splits into two methods:
1) source code analysis
2) dynamic introspection

Of course, we could do either or a combination of both.

IDLE just uses introspection: since IDLE always has a python shell
running, it just completes according to the shell's state (plus
built-in keywords and modules). This is a very simple method,
obviously lacking. It does allow the user some control of the
completion, though - just import whatever you want to be completable
in the shell. However, introspection is all that is needed in a Python
shell, which is the major reason this is the method used in IDLE.



From: Nicolas Chauvat <nicolas.chau...@logilab.fr>
Date: Wed, 6 Jun 2007 23:59:32 +0200


> How dependent/independent is this from the standard AST compiler or
> PyPy? Is it more IDE friendly? Is it based on it or a total independent
> implementation?

It is independent from PyPy.

The above web page says:

"""
Python Abstract Syntax Tree New Generation

The aim of this module is to provide a common base representation of
python source code for projects such as pychecker, pyreverse,
pylint... Well, actually the development of this library is essentialy
governed by pylint's needs.

It extends class defined in the compiler.ast [1] module with some
additional methods and attributes. Instance attributes are added by a
builder object, which can either generate extended ast (let's call
them astng ;) by visiting an existant ast tree or by inspecting living
object. Methods are added by monkey patching ast classes.Python
Abstract Syntax Tree New Generation

The aim of this module is to provide a common base representation of
python source code for projects such as pychecker, pyreverse,
pylint... Well, actually the development of this library is essentialy
governed by pylint's needs.

It extends class defined in the compiler.ast [1] module with some
additional methods and attributes. Instance attributes are added by a
builder object, which can either generate extended ast (let's call
them astng ;) by visiting an existant ast tree or by inspecting living
object. Methods are added by monkey patching ast classes.
"""

From: "Sylvain Thénault" <thena...@gmail.com>
Date: Wed, 13 Jun 2007 10:51:04 +0200

> Please let me involve Sylvain in the discussion. As the main author of
> pylint and astng, he will provide better answers.

well logilab-astng is basically a big monkey patching of the compiler
package from the stdlib, so you can't get an astng representation from a
module with syntax errors in. However inference and most others
navigation methods (which are basically the value added by astng) are
"syntax error resilient" : if a dependency module (direct or indirect)
contains a syntax error, you don't get any exception, though since some
information is missing you can miss some results you'ld get if the
faulting module were parseable.



From: "Tal Einat" <talei...@gmail.com>
Date: Tue, 31 Jul 2007 10:33:33 +0300

Since astng already does some inference (which we definitely want!)
and is based on the standard Python AST compiler, it sounds like our
#1 candidate. I think we should give the code a serious once-over and
see how well it fits our requirements, and if it can be adapted to
better handle errors. Any volunteers?

Also, has anyone used astng for completion, calltips, or something
similar? Or the standard AST compiler, for that matter?



From: "Tal Einat" <talei...@gmail.com>
Date: Tue, 31 Jul 2007 10:40:11 +0300

How does PyPE parse code? Home-rolled, standard AST compiler, something else?

It seems to me we should try to come up with an algorithm for parsing,
before getting to the code. All of the details you mentioned -
noticing assignments, using base-class methods, etc. - could be better
defined and organized this way. Perhaps we could brainstorm on this in
a wiki?



From: "Tal Einat" <talei...@gmail.com>
Date: Tue, 31 Jul 2007 11:38:40 +0300

Sorry for being away for such a long time. I hope we can get this
conversation rolling again, and get started with the actual work.

I'll try to sum up what has been said so far, and how I see things.

== Top Priorities ==
* Can we implement a parser based on the standard Python AST compiler
(or astng)? For example, can syntax errors be handled well?
* Is importing reasonable security-wise? If not, can it be made secure?

== General issues ==
* Do we aim for just completion, or also calltips? Perhaps also other
meta-data, e.g. place defined, source code, ... (see IPython's '??')
* Dependencies - do we want to allow C-extensions, or are we going for
a Python-only solution? (IDLE would only use such a Python-only tool.)
It seems that we want to pre-process most of the data in the
background, so I don't see why we would want to do this in C for
efficiency reasons.

== Completion sources ==
1) Importing "external" modules
2) Importing/Parsing "local" modules
3) Parsing the current file
4) Using objects/modules from the shell (e.g. IDLE has both editor
windows and a Python shell)

== Importing ==
* Stani mentioned that importing is problematic from a security point
of view. What are the security issues? Are they really an issue for an
IDE? If so, perhaps we could overcome this by importing in some kind
of "sandbox"?
* What are the pros and cons of Importing vs. Parsing?
* If importing is always preferable to parsing unless there's a syntax
error, perhaps try to import and parse on failure?

== Parsing ==
* This is going to be the most complex method - I think we should have
a general idea of how this should work before starting an
implementation. I suggest hashing ideas out on a wiki, since there a
lot of details to consider.
* Can a parser based on the standard AST compiler (or astng) work? Is
there a way to deal with errors? (HIGH PRIORITY!)
* There are other existing, open-source implementations out there -
WingIDE, PyPE have been mentioned. Any others? We should collect these
so we can use the code for learning, and perhaps direct use (if
possible license-wise).

== Shell ==
This is relatively straight-forward - just use dir(). This should be
optional, for use by IDEs which have a shell (support multiple
shells?).

Some known issues from IDLE and PyCrust:
* Handle object proxies such as RPC proxies (e.g. RPyC)
* Handle ZODB "ghost" objects
* Watch out for circular references
* Watch out for objects with special __getattr__/__hasattr__
implementations (for example xmlrpc, soap)

== Persistence ==
* Stani mentioned a 'database'. I feel Sqlite should be at most
optional, to reduce dependencies.
* Do we really want to have the data persistent (between IDE
sessiosns)? If so, we need to support simultaneous instances of the
IDE so they don't corrupt the data. Any other issues? (I have a
feeling this would better be left for later stages of development.)



From: "Tal Einat" <talei...@gmail.com>
Date: Tue, 31 Jul 2007 12:22:59 +0300

One more note: We should distinguish between completion in an editor
and completion in a shell. The conversation up until now has focused
on editors, which is reasonable since that is the problematic scene. I
think a generic Python completion library should support completion in
both contexts, especially if it uses can use a shell's namespace for
completion in the editor.



From: "Ali Afshar" <aafs...@gmail.com>
Date: Tue, 31 Jul 2007 11:20:19 +0100

I have just implemented a completion mockup using Rope (which is a
refactoring library). It works quite nicely, and definitely worth a
look.

http://rope.sourceforge.net/

It even achieves this kind of completion:

class Banana(object):
    def do_something(self):
         return

def foo():
    return [Banana(), Banana()]

foo()[0].<complete> includes do_something

Which seems pretty impressive to me.



From: "Tal Einat" <talei...@gmail.com>
Date: Tue, 31 Jul 2007 20:12:50 +0300

Wow, Rope does look very impressive! A quick look at the code tells me
that a lot of work has been invested in it.

So we have one existing Python-only solution. We should evaluate it -
see what it can and can't do, and perhaps take a look at the overall
design.

I'm CC-ing Rope's developer, Ali. Hopefully Ali can help us quickly
understand Rope's code analysis capabilities.

Ali, could you elaborate a bit on what kinds of completion Rope can
do, and the methods it uses? We would especially like to know how your
static and dynamic inference work, what they can accomplish, and what
their limitations are.



From: "Ali Afshar" <aafs...@gmail.com>
Date: Tue, 31 Jul 2007 19:45:15 +0100

> Ali, could you elaborate a bit on what kinds of completion Rope can
> do, and the methods it uses? We would especially like to know how your
> static and dynamic inference work, what they can accomplish, and what
> their limitations are.

Well, I haven't really looked at the code. But I can tell you this:

from rope.ide.codeassist import PythonCodeAssist
from rope.base.project import Project
for compl in PythonCodeAssist(Project(package_root)).assist(buffer,
offset).completions:
    print compl

And that is as far as I really got. I expect to get a better look at
it later in the week though...


From: "Josiah Carlson" <josiah.carl...@gmail.com>
Date: Wed, 1 Aug 2007 00:26:14 -0700

> How does PyPE parse code? Home-rolled, standard AST compiler, something else?

The compiler for syntactically correct Python, a line-based compiler
for broken Python.  TO generate a method list for self.methods, using
the current line number, I discover the enclosing class, check the
listing of methods for that class (generated by the compiler or
line-based parsers), and return a valid list for the specified prefix.
 It doesn't walk the inheritance tree, it doesn't do imports, etc.

> It seems to me we should try to come up with an algorithm for parsing,
> before getting to the code. All of the details you mentioned -
> noticing assignments, using base-class methods, etc. - could be better
> defined and organized this way. Perhaps we could brainstorm on this in
> a wiki?

A wiki would be fine, the one for this mailing list would likely be
best (if it is still up and working).  Then again, Rope looks quite
nifty.  I may have to borrow some of that source ;)


Discussion subject changed to "Fwd: Python code completion module" by Tal Einat

From: Ali Gholami Rudi <aligr...@gmail.com>
Date: Aug 1, 2007 5:50 PM

First of all I should note that rope's main goal was being a
refactoring tool and a refactoring tool needs to know a lot about
python modules.  `rope.base` package provides information about python
modules.

Actually what ropeide provides as auto-completion is defined in
`rope.ide.codeassist` module.  This module almost does nothing but use
`rope.base`.  Since `rope.ide` package is not included in the rope
library (which has been separated from ropeide since 0.6m4) it lacks
good documentation and the API might not be easy to use (most of it is
written in the first months of rope's birth).

> ..., could you elaborate a bit on what kinds of completion Rope can
> do, ...

I don't know what to say here.  Well, actually it tries to use the
source code as much as possible and infer things from it.  So I can
say that it can complete any obvious thing that can be inferred by a
human.  Like this is the first parameter of a method and after dots
its attributes can appear or these modules are imported so their names
and contents are available or this is an instance of some known type
and we know its attributes and ... .  Try ropeide (it uses emacs-like
keybinding, C-/ for completion; see ~/.rope if you want to change
that); it completes common cases (and sometimes completes things you
don't expect it to!).

> ..., and the methods it uses?

Rope analyzes python source code and AST.  Rope used to use the
`compiler` module till 0.5 and now it uses `_ast` module.

> We would especially like to know how your
> static and dynamic inference work, what they can accomplish

There are a few examples in docs/overview.txt.  Unit-test modules like
`ropetest.base.objectinfertest` and `advanced_oi_test` might help,
too.  Also have a look at `rope.base.oi.__init__` pydoc for an
overview of how they work; (I'm afraid it is a bit out of date and
carelessly written.)  The idea behind rope's object inference is to
guess what references (names in source-code) hold.  They collect
information about code when they can and use them later.

>..., and what their limitations are.

Many things in rope are approximations that might be exact if some
conditions hold.  For instance rope might assume that every normal
reference in module scope holds only one kind of object.  Apart from
these assumptions both SOI and DOI have their own disadvantages; For
instance SOI fails when dynamic code is evaluated while DOI does not.
Or DOI is slower than SOI.  (Well, after recent enhancements to rope's
SOI I rarely use DOI).

I tried to answer as short as possible.  If there are questions on
specific parts of rope, I'll be happy to answer.

By the way, I tried to reply this mail to the group, but it seems that
your group requires subscription for posting, so I've sent it to you,
instead.
#@+node:ekr.20070308062440: *3* Thread notes
#@+node:ekr.20070308062440.1: *4* Posting 2
On 2/26/07, Edward Ream <edreamleo@charter.net> wrote:

> threads will swap after sys.getcheckinterval() bytecodes have been
> processed for that thread.

Many thanks for this detailed summary.  I think this is the guarantee I need
a) to experiment with threads and b) to fiddle with settings should that 
appear to be necessary.

Just be careful.  Test your assumptions before you rely on them, especially regarding threads.  Generally threading is seen as a "hard" problem.  If you want to help make them easier, use Queues to handle inter-thread communication. 


> you can use a technique known as 'cooperative multithreading with 
> generators'.

Googling this leads directly to an entry in the Python Cookbook.  The site
is down at present.  I'll study this entry when it's back up.

The basic idea is to have each task be a generator, with each generator giving up control after some amount of work.  Here's a variant of the recipe in the cookbook... 

 - Josiah

import collections

tasks = collections.deque()

def busy():
    while 1:
        yield None

def delay(v):
    import time
    while 1:
        time.sleep(v)
        yield None 

def xpasses(x):
    while x > 0:
        x -= 1
        yield None

def runtasks():
    while 1:
        task = tasks.popleft()
        try:
            task.next()
        except StopIteration: 
            pass
        else:
            tasks.append(task)
#@+node:ekr.20050306070535: *3* Tk Notes
@killcolor
#@+node:ekr.20050306070535.3: *4* How to detect changes in text
http://sourceforge.net/forum/message.php?msg_id=1864564
By: btheado

WAS:RE: Leo 3.10 comments
edream wrote:

>This is due to apparent glitches in the Tk event dispatching. The problem is
that pressing a control or alt or shift key _all by themselves_ will generate
keypress events that are passed on to Leo's key handlers

This should be easy to make simpler--just bind an empty script to <Alt-KeyPress>,
<Shift-KeyPress>, etc.  Tk chooses the most specific event it can find, so the
more general <KeyPress> handler will not fire.

On a broader note, when programming the text widget in Tcl/Tk, watching key
events is not the easiest way to detect changes in the text.  The only way the
text in a text widget can change is if either the delete or the insert subcommands
(methods) are called.  Any keypresses that end up changing text will have called
one of these subcommands.

So the simplest way to detect changes is to just intercept the calls to insert
and delete.  In Tcl/Tk intercepting these calls is pretty straightforward. 
I don't know if the same is true in Tkinter.

Also note the text widget in Tk8.4 (http://www.tcl.tk/man/tcl8.4/TkCmd/text.htm#M72)
has a built-in way of seeing if the text has changed

Brian Theado
#@+node:ekr.20031218072017.365: ** How to...
#@+node:ekr.20060208112908: *3* BZR stuff...
#@+node:ekr.20060331094112: *4* How to generate keys using putty
To generate a SSH key using PuTTY:

Execute c:\"Program Files"\tortoiseCVS\PUTTYGEN.EXE

Select "SSH2 DSA", within the "Parameters" section.

Click on the "Generate" button. Follow the instruction to move the mouse over
the blank area of the program in order to create random data used by PUTTYGEN to
generate secure keys. Key generation will occur once PUTTYGEN has collected
sufficient random data.

Enter edream@cvs.sourceforge.net for the key comment (depends on what host the
key is for)

(Omit) Enter the desired passphrase in the "Key passphrase" and "Confirm passphrase"
fields. If the key will be used for automation of operations (i.e. as part of a
script), you may choose to omit this step from the key generation process.

Click on the "Save private key" button. Use the resulting dialog to save your
private key data for future use. You may use a filename such as
"SourceForge-Shell.ppk" or "SourceForge-CF.ppk". The .ppk extension is used for
PuTTY Private Key files.

Go to the SSH key posting page on the SourceForge.net site: http://sourceforge.net/account/

Copy your public key data from the "Public key for pasting into OpenSSH
authorized_keys2 file" section of the PuTTY Key Generator, and paste the key
data to the provided form on the SourceForge.net site. Click on the "Update"
button to complete the posting process.

Exit the PuTTY Key Generator (PUTTYGEN).

Key data sync to hosts from the SourceForge.net site occurs on regular
intervals. Your key data will be synchronized to the designated servers (either
shell and CVS, or the Compile Farm) after a short delay.
#@+node:ekr.20031218072017.366: *4* How to add and remove files from CVS repository
use the command line option in the admin menu to do the following:

add leoConfig.py and leoConfig.txt
    cvs add leoConfig.txt
    cvs add leoConfig.py
    (then do commit)

remove readme*.doc
    remove files from working area (done)
    cvs remove readme1.doc
    cvs remove readme2.doc
    ...
    (then do commit)
#@+node:ekr.20031218072017.391: *4* How to use CVS branches
@nocolor

I have a fair bit of expertise on CVS branches. It's late at night, so I don't have time for a long soapbox spiel at the moment. I will try to post something tomorrow. 

The brief picture is: 

* Check out code from CVS at the point you want to create the branch. 

* Make sure none of the files in your sandbox is modified. 

* Create the branch (cvs tag -b branchname). The branch name must start with a letter (upper or lower case) and thereafter can have alphanumeric characters, hyphens, and underscores (no periods or spaces). 

* The branch is created on the repository, but your sandbox is still checked out on the main branch. To check out on the new branch, do "cvs up -r branchname". 

When you want to merge changes back into the main branch, you can use "cvs up -r MAIN" to retrieve the main branch, then "cvs up -j branchname" to merge changes, then "cvs commit" to commit the merged version to the main branch AFTER YOU HAVE VERIFIED IT. 

I would recommend caution with merging because as you have noted, leo files are not well set up for CVS. They don't merge well because of inconsistent sentinel values. 

You may want to look at manually merging changes back into the main branch until leo implements invariant unique (UUID) sentinel indices. 

This will not hurt your ability to use branches, only your ability to automatically merge changes from one branch onto another.
#@+node:ekr.20031218072017.367: *3* How to add support for a new language
@nocolor

- Add new entries in the following Python dictionariues in leoApp.py:
  self.language_delims_dict, self.language_extension_dict and self.extension_dict

- Add an entry to the languages list in <<configure language-specific settings>>

- Add a list of the keywords of the language to << define colorizer keywords >>

  N.B.: the name of this list must be x_keywords, where x is the entry in language in step a.

- Add any language-specifig code to leoColor.colorizeAnyLanguage.
  For most languages nothing need be done.

- If the language is case insensitive, add it to the list of
case_insensitiveLanguages found in  << define global colorizer data >>

- Create the files theLanguage.xml and theLanguage.py file was added to the leo\modes directory.
  See Chapter 15 of Leo's users guide for full details.

TESTS

- Test the syntax coloring for the new language by using the @language directive.

@color
#@+node:ekr.20080814134319.1: *3* How to create and push to a private bzr branch
@nocolor

C:\leo.repo\trunk>bzr push --use-existing-dir bzr+ssh://edreamleo@bazaar.launchpad.net/~edreamleo/leo-editor/leo-4-5-1

C:\leo.repo\trunk>bzr push --use-existing-dir bzr+ssh://edreamleo@bazaar.launchpad.net/~edreamleo/leo-editor/leo-4-5-1

#@+node:ekr.20051203084725: *3* How to expand java .jar files
- Put whatever.jar in c:\prog
- cd: c:\prog
- jar xvf whatever.jar
#@+node:ekr.20031218072017.384: *3* How to export syntax colored code preserving colors
Scite has the option to "Export as html" and "export as rtf", and it will be
full of colour and fonts - and you can define them in properties, so it will be
the same as during editing.
#@+node:ekr.20031218072017.385: *3* How to Increase environment space
To increase the size of environment space, add the following to config.sys:

shell=C:\windows\command\command.com /p:4096

Notes:

1. The path C:\windows\command\command.com may vary.
Check you system for the location of command.com.

2. This works for versions of Windows prior to Me.
On Me you set the registry somehow.
No information on XP.
#@+node:ekr.20051203084725.1: *3* How to install and run jythonShell
Install:

Put JythonShellEA.jar in c:\prog\JythonShell

(optional) Expand the jar so you can see the code:

jar xvf JythonShellEA.jar

Run:

Here is the contents of jythonShell.bat:

cd c:\prog\jythonShell
java -cp c:\jython-2.2a1\jython.jar;c:\prog\jythonShell\JythonShellEA2.1.jar org.leo.shell.JythonShell
#@+node:ekr.20050316092232: *3* How to install jyLeo
- Unpack the .zip file, placing the result somewhere, say in c:\prog\jyleo-Jan-11-06

- Edit jleo.bat so it refers to jyleo-Jan-11-06.  For example:

rem open jyLeo
set ARGS= 
:loop 
if [%1] == [] goto end 
set ARGS=%ARGS% %1 
shift 
goto loop 
:end 

cd c:\prog\jyleo-Jan-11-06
java -jar c:\jython-2.2a1\jython.jar src\leo.py
#@+node:ekr.20050716104357: *4* Old instructions
@nocolor

- put the jyleo-nnn.jar file in c:\prog

- Execute the following command in a console window
    cd c:\prog
    jar xvf j-leo-nnn.jar

This creates a folder called j-leo-nnn

- Do the following, or execute jleo.bat

cd c:\prog\j-leo-nnn\src
java -jar c:\jython22a0\jython.jar leo.py

Note:  at present this gives KeyError: HOME

In leo.py, in computeHomeDir, I changed:
@color

home = os.getenv('HOME' )#,default=dotDir)

to:

try:
    home = os.getenv('HOME' )#,default=dotDir)
except Exception:
    home = ''
#@+node:ekr.20050317153447: *4* jy-Leo install instructions by Paul Paterson
@killcolor

http://sourceforge.net/forum/message.php?msg_id=3053534
By: paulpaterson

Very interesting indeed - great work! 

I didn't have Java/Jython installed so for others in the same boat here's what I had to do to get it work on my platform (Win2k). Some of this is in the README but I had to do some extra but I'm not sure why. 

1. Install 1.5 JDK  
http://java.sun.com/j2se/1.5.0/download.jsp 

2. Install Jython 
http://www.jython.org/jython22a1.zip 

3. Edit Jython.bat file - the part that calls Java.exe to ... 
"C:\Program Files\Java\jdk1.5.0_02\jre\bin\java" -cp "C:\Program Files\Java\jdk1.5.0_02\jre\lib";"c:\Apps\Python23\Jython";"C:\Apps\jLeo\j-leo-MAR15\Icons";"C:\Apps\jLeo\j-leo-MAR15\skins";"C:\Apps\jLeo\j-leo-MAR15\src";"C:\Apps\jLeo\j-leo-MAR15\skinimages" -Dpython.home="c:\Apps\Python23\Jython" -jar jython.jar %ARGS% 

Where  
- Java installed at C:\Program Files\Java\jdk1.5.0_02 
- Jython at c:\Apps\Python23\Jython 
- jLeo at C:\Apps\jLeo\j-leo-MAR15 

Change your paths as appropriate! There must be a better way to do this - Java confuses me! 

4. Edit leo.py in jleo/src directory to fix failure to find HOME env variable. 

line 241 becomes ... 

....try:home = os.getenv('HOME' )#,default=dotDir) 
....except KeyError:home="" 


Then, from the Jython install directory ... 

Jython " 
C:\Apps\jLeo\j-leo-MAR15\src\leo.py" 

Works a treat!  

Paul
#@+node:ekr.20051129084430: *3* How to install jython
@nocolor

- Download jython_Release_2_2alpha1.jar and put it anywhere (say on the desktop)

- Double-click the file.  This brings up an installer.  Follow the direction.
  (I installed to c:\jython-2.2a1

- Using the Control Panel, System, Advanced tab, environment variables,
  add c:\jython-2.2a1\jython.jar to CLASSPATH (in user variables)
#@+node:ekr.20051129084430.1: *4* @url http://www.jython.org/install.html
#@+node:ekr.20101004092958.6050: *3* How to make codewise work
@nocolor-node

http://groups.google.com/group/leo-editor/browse_thread/thread/ac3f8789010c882e/a1558a10eb8537c0?lnk=gst&q=codewise#a1558a10eb8537c0

1. Make sure you have exuberant ctags (not just regular ctags)
installed.  It's an Ubuntu package, so easy if you're using Ubuntu.

2. Install Ville's python module "codewise".  This is a small module on
which the Leo plugin relies.

   bzr branch lp:codewise
   cd codewise
   sudo python setup.py install

3. You need a recent trunk version of leo to get the plugin which uses
the above module.

4. Enable the plugin by putting "codewisecompleter.py" on an
uncommented line in your @enabled-plugins @settings node.

5. On the command line:

if you have an existing ~/.ctags for some reason, and it's nothing you
need to keep:

  rm ~/.ctags

then

  codewise setup
  codewise init
  codewise parse .../path/to/leo/  # assuming you want completion on
                                   # leo code
  codewise parse .../some/other/project/

Then, after restarting leo if necessary, type

c.op<Alt-0> in the body editor to find all the c. methods starting
with 'op' etc.

Nice work Ville, thanks.

==================

Thanks for this, I hope others will take a stab at it as well, given
sane instructions (I burned my free cycles frantically coding this
thing and neglected the all-important HOWTO). This is important
because functional completion is the single most important thing still
missing from Leo. Or, well, was ;-).

Especially the presentation part (QCompleter) needs some care, so you
can operate it from your keyboard alone. It should probably be moved
to core (qtgui, perhaps leoQTextEditWIdget), so codewise completer can
just invoke w.complete(list_of_completions) that will bring up the
QCompleter popup.

> Then, after restarting leo if necessary, type

> c.op<Alt-0> in the body editor to find all the c. methods starting
> with 'op' etc.

Also, try the explicit declarations:

# w : SomeClass

w.<alt+0>

And self.<alt+0> 
#@+node:ekr.20070623150151: *3* How to make Leo commands undoable
The chapter commands provide a good example.  In this file, see the node:

Code-->Core classes...-->@thin leoChapters.py-->class chapterController-->Undo

The general plan is this:

1. The command handler calls a **beforeCommand** method before changing the outline.

The beforeCommand method creates a g.Bunch that contains all the information needed to
restore the outline to its previous state. Typically, the beforeCommand method
will call c.undoer.createCommonBunch(p), where p is, as usual, c.p.

2. After changing the outline the command handler calls an **afterCommand** method.

This method should take as one argument the g.Bunch returned by the
beforeCommand method. In the discussion below, denote this bunch by b. The
afterCommand method adds any information required to redo the operation after
the operation has been undone.

The afterCommand method also sets b.undoHelper and b.redoHelper to two method
that actually perform the undo and redo operations. (Actually, the beforeCommand
method could also set these two entries).

When the afterCommand method has 'filled in' all the entries of b, the
afterCommand method must call u.pushBead(b). This pushes all the undo
operation on a stack managed by the Leo's undoer, i.e., c.commands.undoer.

3. The undoer calls the undoHelper and redoHelper methods to perform the actual undo and redo operations.

The undoer handles most of the housekeeping chores related to undo and redo.  All the undoHelper and redoHelper methods have to do is actually alter Leo's outline.

**Note**: the undoer creates an ivar (instance variable) of the *undoer* class for every entry in the bunch b passed as an argument to u.pushBead(b).  For example, suppose u = c.commands.under and that b has ivars 'a','b' and 'c'.  Then, on entry to the undoHelper and the redoHelper the u.a, u.b and u.c ivars will be defined.  This makes it unnecessary for the undoHelper or the redoHelper to 'unpack' b explicitly.

Writing correct undo and redo helpers is usually a bit tricky.  The code is often subtly different from the original code that implements a command.  That just can't be helped.




#@+node:ekr.20091217112515.6070: *3* How to make the codewise completer work
@nocolor-node

http://groups.google.com/group/leo-editor/browse_thread/thread/ac3f8789010c882e

Ville's completer is working and very cool, here are instructions for
making it go.  They're like the instructions Ville gave, only usable ;-)

1. (done) Make sure you have exuberant ctags (not just regular ctags)
installed.  It's an Ubuntu package, so easy if you're using Ubuntu.

2. (done) Install Ville's python module "codewise".  This is a small module on
which the Leo plugin relies.

   bzr branch lp:codewise
   cd codewise
   sudo python setup.py install

3. (done) You need a recent trunk version of leo to get the plugin which uses
the above module.

4. (done) Enable the plugin by putting "codewisecompleter.py" on an
uncommented line in your @enabled-plugins @settings node.

5. On the command line:

if you have an existing ~/.ctags for some reason, and it's nothing you
need to keep:

  rm ~/.ctags

then

  codewise setup
  codewise init
  codewise parse .../path/to/leo/  # assuming you want completion on
                                   # leo code
  codewise parse .../some/other/project/

Then, after restarting leo if necessary, type

c.op<Alt-0> in the body editor to find all the c. methods starting
with 'op' etc.


===== Ville's response

Especially the presentation part (QCompleter) needs some care, so you
can operate it from your keyboard alone. It should probably be moved
to core (qtgui, perhaps leoQTextEditWIdget), so codewise completer can
just invoke w.complete(list_of_completions) that will bring up the
QCompleter popup.

> Then, after restarting leo if necessary, type

> c.op<Alt-0> in the body editor to find all the c. methods starting
> with 'op' etc.

Also, try the explicit declarations:

# w : SomeClass

w.<alt+0>

And self.<alt+0>
#@+node:ekr.20091217112515.6069: *4* Others posts
@nocolor-node

1. codewisecompleter.py now completes by explicit type hints (as seen in
screenhots). p, c also work, as does 'self'.

self works by scanning for parent headlines looking for "class Foo"

Work remains for presentation part (it's mouse only now) but Edward
will probably do it :-).

2. > Would codewise work outside of leo, as stand-alone plugin for a text
> editor?

Yes, currently Leo uses it as an external program ("codewise m
MyClass" dumps the methods in MyClass to stdout).

Someone just has to write the vim integration plugin (or whatever they
call it). OTOH, vim already has "pysmell" and the likes that do the
same thing.

==============

The version of codewise completer that works with Tk is now on trunk.
#@+node:ekr.20091217112515.6071: *4* plugin docs
@nocolor-node

- You need to create ctags file to ~/.leo/tags. Example::

    cd ~/.leo
    ctags -R /usr/lib/python2.5 ~/leo-editor ~/my-project

- Enter text you want to complete and press alt+0 to show completions
  (or bind/execute ctags-complete command yourself).

Attempting to complete 'foo->' is useless, but 'foo->ba' will work (provided you
don't have 2000 functions/methods starting with 'ba'. 'foo->' portion is ignored
in completion search.
#@+node:ekr.20031218072017.386: *3* How to remove cursed newlines: use binary mode
teknico ( Nicola Larosa ) 
 RE: Removing '\r' characters?   
2002-09-16 14:27  
> I am plowing through old bug reports, and I found the following, from whom 
> I don't know: 

That's from me, *again*. You are kindly advised to stop forgetting the attribution to all my bug reports. ;^) 

>> - Source files still have the dreaded \r in them. Why don't you switch 
>> to \n only, once and for all, and live happily ever after? ;^) 

> I sure whould like to do that, and I'm not sure how to do this. All 
> versions of the read code attempt to remove '\r' characters, and all 
> versions of the write code write '\n' only for newlines. 

Sorry for being a bit vague, I was talking about the Leo source files themselves. I don't know what you use to edit them, ;^))) but in version 3.6 they still have \r\n as end-of-line. 

If Leo itself does not solve the problem, may I suggest the 
Tools/scripts/crlf.py script in the Python source distibution? It's nice and simple, and skips binary files, too. That's what I use every time I install a new version of Leo. :^) 

#@+node:ekr.20031218072017.387: *4* The solution
Under unix, python writes "\n" as "\n"; under windows, it writes it as "\r\n". The unix python interpreter ignores trailing "\r" in python source files. There are no such guarantees for other languages. Unix users should be able to get rid of the cosmetically detrimental "\r" either by running dos2unix on the offending files, or, if they're part of a .leo project, reading them into leo and writing them out again.  


By: edream ( Edward K. Ream ) 
 RE: Removing '\r' characters?   
2002-09-17 09:34  
Oh, I see. Thanks very much for this clarification. 

Just to make sure I understand you: the problem with '\r' characters is that: 

1. I am creating LeoPy.leo and LeoDocs.leo on Windows and 
2. People are then using these files on Linux. 

and the way to remove the '\r' characters: 

1. I could run dos2unix on all distributed files just before committing to CVS or making a final distribution or 
2. People could, say, do the following: 

Step 1: Read and Save the .leo files, thereby eliminating the '\r' in those files and 
Step 2: Use the Write @file nodes command on all derived files to clear the '\r' in those files. 

Do you agree so far? 

> Under unix, python writes "\n" as "\n"; under windows, it writes it as "\r\n". 

I am going to see if there is any way to get Python to write a "raw" '\n' to a file. I think there must be. This would solve the problem once and for all. 

Thanks again for this most helpful comment. 

Edward
#@+node:ekr.20031218072017.388: *4* cursed newline answer
In 2.3 you can open files with the "U" flag and get "universal newline"
support: 

% python
Python 2.3a0 (#86, Sep 4 2002, 21:13:00) 
[GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.62mdk)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open("crlf.txt")
>>> line = f.readline()
>>> line
'This is an example of what I have come to call the "cursed newline"
problem,\r\n'
>>> f = open("crlf.txt", "rU")
>>> line = f.readline()
>>> line
'This is an example of what I have come to call the "cursed newline" problem,\n'

#@+node:ekr.20031218072017.389: *4* cursed newline answer 2
> You can open the file in 'binary' mode (adding 'b' to the mode string) and
> the file will contain '\r\n' on both platforms (and any other platforms.)

Nope. Exactly wrong. In 2.2 and those before, when files are opened in
*text* mode (no "b") then reading them will provide Unix-style line endings
(newline only). When you open files in binary mode then you see the bytes
stored in the file.

On Unix systems there's no difference in the contents of a file whether in
binary or text mode. On Windows a file is shorter by the number of carriage
returns. On the Mac I have no idea what they do. Probably just carriage
returns, to be different :-)

2.3 will be a bit more flexible about such mattrers.
#@+node:ekr.20061023153133: *3* How to run patch
patch -p1 < patchfile
#@+node:ekr.20050510071834: *3* How to use a temp file with pdb
@killcolor

http://sourceforge.net/forum/message.php?msg_id=3137690
By: nobody

I dont know if anyone has solved this for regular Leo, but in the JyLeo JythonShell,
when the user executes a script with Pdb it:
1. dumps the script in a tmp file system's tmp directory.
2. Executes pdb based off of that tmp file.

that way you get all the goodness that pdb can offer.
#@+node:ekr.20090105132011.6: ** Regex notes
@nocolor-node

# *** match pat2 if not preceded by pat1::

(?<!pat1)pat2

.  any char
^  start line
$  end of line
\w alphanum: [a-zA-Z0-9_]
\W non-alphanum
\s whitespace
\S non-whitespace

These can be done with regexps:

copy-to-end-of-each-line:   (.)$        -->  \1x
copy-to-start-of-each-line: ^([ \t]+)   -->  \1x
remove-leading-ws:  ^[ \t]+             -->  empty
remove-trailing-ws: [ \t]+$             -->  empty
paste-at-column:    ^(.{4})             -->  \1x
paste-after-lws:    ^([ \t]+)           -->  \1x

[]
    Used to indicate a set of characters. Characters can be listed individually,
    or a range of characters can be indicated by giving two characters and
    separating them by a '-'. Special characters are not active inside sets.

    [akm$] will match any of the characters 'a', 'k', 'm', or '$';
    [a-z] will match any lowercase letter.
    [a-zA-Z0-9] matches any letter or digit.

    Character classes such as \w or \S (defined below) are also
    acceptable inside a range, although the characters they match depends on
    whether LOCALE or UNICODE mode is in force. If you want to include a ']' or
    a '-' inside a set, precede it with a backslash, or place it as the first
    character. The pattern []] will match ']', for example.

    You can match the characters not within a range by complementing the set.
    This is indicated by including a '^' as the first character of the set; '^'
    elsewhere will simply match the '^' character. For example, [^5] will match
    any character except '5', and [^^] will match any character except '^'.

(?=...)
    Matches if ... matches next, but doesnt consume any of the string.
    This is called a lookahead assertion.
    For example, Isaac (?=Asimov) will match 'Isaac ' only if it's followed by 'Asimov'.

(?!...)
    Matches if ... doesn't match next.
    This is a negative lookahead assertion.
    For example, Isaac (?!Asimov) will match 'Isaac ' only if it is not followed by 'Asimov'.

(?<=...)
    Matches if the current position in the string is preceded by a match for ...
    that ends at the current position. This is called a positive lookbehind assertion.
    The contained pattern must only match strings of some fixed length,
    meaning that abc or a|b are allowed, but a* and a{3,4} are not.

(?<!...) Matches if the current position in the string is not preceded by a
    match for .... This is called a negative lookbehind assertion. Similar to
    positive lookbehind assertions, the contained pattern must only match
    strings of some fixed length. Patterns which start with negative lookbehind
    assertions may match at the beginning of the string being searched.
#@+node:ekr.20060217111834: *3* @url http://docs.python.org/lib/re-syntax.html
#@+node:ekr.20110527084258.18374: ** Summary of the Ashland sprint
http://groups.google.com/group/leo-editor/browse_thread/thread/9b1dbebd56d50e14/d5a690127ddad38e
 
"Sprint" isn't really the correct term.  We wrote no code.  Instead,
we discussed what seemed to each of us as the most important
directions for Leo.

After much pleasant discussion, we reached agreement, dreaded or not,
on just about everything. I'll summarize the topics here, and
elaborate about file format issues in a separate thread.

1.  (Done) Make .leo files as standards-compliant as possible.

This will demonstrate to newbies that we have some sophistication re
web standards, and it will allow external tools to handle .leo files
in the easiest possible way.

I've delegated the design of this project to Kent.  I'll be in charge
of implementation.

2. Simplify and revise Leo's file format.

Details in a separate thread: some items deserve just a bit more
discussion.  In particular, we want the about-to-be-renamed <t>
elements to contain headline text so that <t> elements represent key/
value pairs directly.  The question is, should the about-to-be-renamed
<v> elements contain headline text (readable, but "denormalized")?

3. Support reading and writing Leo outlines in JSON format.

This will allow closer cooperation with databases and other tools.
I'll do this.

4. Complete the transition to Terry's free_layout plugin.

A. Place separate body editors in free_layout areas.  This should
*easy* to do!  Almost nothing changes in the code, but the visual
effect should be much better.

B. Allow any pane to be "tabified" (placed in a tab in the Log pane)
and "untabified."  There are a few details to be handled, but nothing
major.

Terry and I will collaborate on this.

5. Add global search to the quicksearch plugin and to Leo's find
command.

6. (Abandoned) Add node-specific undo asap.

The present undo is almost useless after a few levels.  Node-specific
undo would be much more useful. This has been on the list forever.  It
should be done yesterday.

7.  Rejected direct support for .ini files instead of Leo's @settings
nodes.

After some discussion we decided that the present .ini importer should
suffice.  In other words, it seems like a bad idea to support .ini
settings *instead* of, or in *addition* to, .ini files.  However,
scripts or commands to import/export Leo settings to one or more .ini
files would be fine.

8.  (done) Make uA's first class citizens.

There should be commands to get and set uA's.  This is easy to do:
it's just an oversight.

Summary
=======

Looking back on the discussions, I am struck once again by how minor
the suggestions are.  Most of these items can be done in a day or two,
or a week or two at most.  The conclusion is that Leo has reached a
mature state.

Kent, Terry, did I omit anything?  Misstate anything? 
#@+node:ekr.20101127152442.5878: ** Url's
@color

@language rest
#@+node:ekr.20101127154340.5929: *3*  To investigate first
#@+node:ekr.20101127154340.5928: *4* @url pyexpect: spawn child apps
http://pexpect.sourceforge.net/pexpect.html

@language rest

pexpect: a Python module for spawning child applications and
controlling them automatically. 
#@+node:ekr.20101127154340.5930: *4* @url pyjamas: AJAX tool kit
http://pyjs.org/

@language rest

Pyjamas: AJAX tool kit.
#@+node:ekr.20101127154340.5936: *4* @url proto of a Leo forum
http://groups.google.com/group/leo-editor/browse_thread/thread/db6e75d82da4b41d

@language rest

A few buttons turns Leo into a competitor for google groups.

#@+node:ekr.20101129064803.6060: *4* @url server code to interact with a running Leo
http://groups.google.com/group/leo-editor/browse_thread/thread/278aa85d7298a319/ef6446fcf6268c0d

leoRemote.py
#@+node:ekr.20101129064803.6061: *4* @url problem solving with graph traversals
http://groups.google.com/group/leo-editor/browse_thread/thread/2e1b240b023b545e/8e9164b52ff25199

Cool slide show: very technical, about graph traversals, from AT&T technical talk

http://www.slideshare.net/slidarko/problemsolving-using-graph-traversals-searching-scoring-ranking-and-recommendation
#@+node:ekr.20101127154340.5935: *3* Apps
#@+node:ekr.20101127154340.6806: *4* @url review board: code review
http://www.reviewboard.org/

@language rest
#@+node:ekr.20101127154340.6824: *4* @url scrivener: cool MacOS outliner
http://www.literatureandlatte.com/scrivener.html

@language rest
#@+node:ekr.20101127154340.5934: *4* @url tomboy: wiki-like notes
http://projects.gnome.org/tomboy/index.html

http://groups.google.com/group/leo-editor/browse_thread/thread/18d4af19686f2ead
#@+node:ekr.20101127152442.5879: *3* Autocompletion
#@+node:ekr.20101127154340.6852: *4* @url codewise: ville's code completer
http://www.mail-archive.com/leo-editor@googlegroups.com/msg10145.html
#@+node:ekr.20101127152442.5881: *4* @url ctagscompleter.py (Ville's suggestion)
http://groups.google.com/group/leo-editor/browse_thread/thread/c537f3bc8328a938
#@+node:ekr.20101127152442.5880: *4* @url pydiction (autocompletion for vim)
http://www.vim.org/scripts/script.php?script_id=850
#@+node:ekr.20101127154340.5931: *4* @url pysmell: python IDE completion helper
http://code.google.com/p/pysmell/

@language rest
#@+node:ekr.20101127154340.6828: *4* @url SciTe Java api
http://www.burgaud.com/scite-java-api/

@language rest
#@+node:ekr.20101127154340.6829: *5* @url tags2api.py: python
http://www.scintilla.org/tags2api.py

@language rest

Produces a .api file for SciTE's identifier completion and calltip features.
#@+node:ekr.20101127154340.6830: *5* @url generates python api for scite
http://www.koders.com/python/fid7000B9C96CF2C6FB5BCE9DF700365C5B2A1F36A7.aspx?s=gtk#L53

@language rest

gen_python_api.py generates a python.api file for SciTE
#@+node:ekr.20101127201907.5952: *3* Data bases
#@+node:ekr.20101127154340.6856: *4* @url lazysoft: cool db (sentences)
http://www.lazysoft.com/

@language rest
#@+node:ekr.20101127201907.5953: *4* @url persistent trees: couchDB
http://eclipsesource.com/blogs/2009/12/13/persistent-trees-in-git-clojure-and-couchdb-data-structure-convergence/

@language rest


Thread: Interesting post on data tree design

http://groups.google.com/group/leo-editor/browse_thread/thread/e4646371478cd30/44eba97b45bd53b3
#@+node:ekr.20101127201907.5947: *3* Docs
#@+node:ekr.20101127201907.5949: *4* @url new design for colorer
http://groups.google.com/group/leo-editor/browse_thread/thread/9fb569af95eee493/00c3bbe120567771

Thread: A new design for the incremental colorer

This long posting will discuss a new design for an incremental
colorizer using QSyntaxHighlighter.  The essential features of this
design became apparent during yesterday's walk. 
#@+node:ekr.20101127201907.5945: *4* @url docs for my successor
http://groups.google.com/group/leo-editor/browse_thread/thread/440bbf170787c5ed/c228a9fd1d429fa6

@language rest
#@+node:ekr.20101127201907.5946: *4* @url Leo's MVC architecture
http://groups.google.com/group/leo-editor/browse_thread/thread/6b77a59a3a5c7cbb/f26164f24bee68d2

@language rest
#@+node:ekr.20101127201907.5948: *4* @url one damn fine checkin
http://groups.google.com/group/leo-editor/browse_thread/thread/278442febf1a1965/6a6640976c23935f

@language rest

configure_tags killing performance again

EKR:

Rev 2119 contains what appears to be a major improvement in speed:

- leoQTextEditWidget.SetAllText now sets a lockout that prevents a duplicate
  recoloring of text. This doubles the speed of the syntax colorer, and almost
  doubles the speed of unit tests!

- updateSyntaxColorer and colorize set self.flag = False for large body text.
  This doesn't prevent recolor from being called, but it does short-circuit
  recolor.

And yes, configure_tags is called only by the ctor, that is, just once. 

Ville:

Now that was one damn fine checkin... 
#@+node:ekr.20101127201907.5951: *4* @url Stupendous Aha re unit tests
http://groups.google.com/group/leo-editor/browse_thread/thread/47dfb2e1767d2cda/8e658c5b73406a8d

@language rest

I have gotten *zero* responses to my 42 post:
http://groups.google.com/group/leo-and-pylint/browse_thread/thread/3c...

I should have remembered that nobody reads long posts. So here is the
Aha in a nutshell:

Unit tests are not just for testing!  They are *the* master tool for
programming, design, testing, refactoring, studying code, or *anything
else*. 
#@+node:ekr.20101127154340.6805: *3* Editors
#@+node:ekr.20101127154340.6807: *4* @url pyxides
http://groups.google.com/group/pyxides

@language rest
#@+node:ekr.20101127154340.6813: *4* Vim
#@+node:ekr.20101127154340.6814: *5* @url vim cheat sheet
http://www.fprintf.net/vimCheatSheet.html

@language rest
#@+node:ekr.20101127154340.6815: *5* @url vim reference guide
http://www.dc.turkuamk.fi/docs/soft/vim/vim.html

@language rest
#@+node:ekr.20101127154340.6816: *5* @url slight advance intro to vim
http://linuxgazette.net/152/srinivasan.html

@language rest
#@+node:ekr.20101127154340.6817: *5* @url voom: vim 2-page outliner
http://vim-voom.webs.com/

@language rest
#@+node:ekr.20101127154340.6818: *5* @url why do those nutheads use vim
http://www.viemu.com/a-why-vi-vim.html

@language rest
#@+node:ekr.20101127154340.6819: *5* @url viper
http://www.delorie.com/gnu/docs/emacs/viper.html

@language rest
#@+node:ekr.20101127154340.6820: *5* @ulr learning vim the pragmatic way
http://jrmiii.com/2009/03/06/learning-vim-the-pragmatic-way.html

@language rest
#@+node:ekr.20101127154340.6822: *4* @url python-UNO (Open Office)
http://udk.openoffice.org/python/python-bridge.html

@language rest

http://wiki.services.openoffice.org/wiki/Documentation/DevGuide/ProUNO/Professional_UNO
#@+node:ekr.20101127154340.6823: *4* @url open komodo forums
http://community.activestate.com/forums/komodo/open-komodo

@language rest

archives:

http://lists.openkomodo.com/pipermail/openkomodo-dev/
#@+node:ekr.20101127154340.6825: *4* @url org-babel
http://orgmode.org/worg/org-contrib/babel/intro.php

@language rest
#@+node:ekr.20101127152442.5884: *3* Feature requests
#@+node:ekr.20101127152442.5885: *4* Better find/replace dialog
For newbies (and for power users too, for that matter), it may be helpful to
have a "Find and Replace" dialog box similar to the one below:

.. image:: c:/prog/findReplace.jpg

#@+node:ekr.20101127152442.5886: *4* @image find/replace dialog
c:/prog/findReplace.jpg
#@+node:ekr.20101127154340.6808: *3* Graphics
#@+node:ekr.20101127154340.6843: *4* @url inkscape: open source svg editor
http://inkscape.org/

@language rest

#@+node:ekr.20101127154340.6835: *4* @url interactive map of Linux kernel
http://www.makelinux.net/kernel_map

@language rest
#@+node:ekr.20101127154340.6836: *4* @url blender: open source content creation
http://www.blender.org/

@language rest
#@+node:ekr.20101127154340.6809: *4* @url tinkerpop: graphics tools
http://www.tinkerpop.com/
#@+node:ekr.20101129064803.6061: *4* @url problem solving with graph traversals
http://groups.google.com/group/leo-editor/browse_thread/thread/2e1b240b023b545e/8e9164b52ff25199

Cool slide show: very technical, about graph traversals, from AT&T technical talk

http://www.slideshare.net/slidarko/problemsolving-using-graph-traversals-searching-scoring-ranking-and-recommendation
#@+node:ekr.20101127152442.5887: *3* Gui
#@+node:ekr.20101127152442.5888: *4* @url cool hack: detach body editor
http://groups.google.com/group/leo-editor/browse_thread/thread/8616f4e171e1a24b

@language rest

This features a long an important discussion of using Leo with vim.
#@+node:ekr.20101127152442.5889: *4* @url fast syntax highlighting
http://groups.google.com/group/leo-editor/browse_thread/thread/bdcbe3a4ffb5ac61/7dbeea85f671cfe9

@language rest

Source-highlight Qt Library 0.2.2
http://srchiliteqt.sourceforge.net/source-highlight-qt.html

This seems to be a (c++ based) syntax highlighting package that would
allow us to avoid running any python code for syntax highlighting.
Needless to say this is probably very fast.

Investigation needed how easy this would be to deploy in python & leo.
I might(!) make a plugin that allows you to use it instead of leo's
own highlighters...

t seems to support lots of languages already:

http://www.gnu.org/software/src-highlite/

seems you can define your own languages:

http://www.gnu.org/software/src-highlite/source-highlight.html#Language-Definitions

lots of stuff there, but the simple case (keywords) seem to be simple:

http://www.gnu.org/software/src-highlite/source-highlight.html#Simple-definitions

EKR: We would have to support coloring of Leo directives.

#@+node:ekr.20101127152442.5890: *4* @url code bubbles: an alternative to clones
http://groups.google.com/group/leo-editor/browse_thread/thread/d030a9eccfae2aa2/3a951894e0c15f72

@language rest

This is a cool video:

http://www.cs.brown.edu/people/acb/codebubbles_site.htm
#@+node:ekr.20101127154340.6840: *4* @url SIKULI: test gui's
http://groups.csail.mit.edu/uid/sikuli/

@language rest
#@+node:ekr.20101127152442.5882: *3* Leo projects
#@+node:ekr.20101127152442.5883: *4* @url @data contextmenu_commands
http://groups.google.com/group/leo-editor/browse_thread/thread/e236feb5bd2a097a/d261f6dbdb7eb950

@language rest

As requested, there is now user-friendly way of adding minibuffer
commands to context menu (in trunk, qt only).

I also added example to quickstart.leo, but all you need to know is
that you need to create

@settings
  @data contextmenu_commands

And the content is like:

# The format is <command> SPACE <description>

stickynote Create a sticky note
read-at-file-nodes Read file nodes 
#@+node:ekr.20101127152442.5912: *4* @url fuse: Leo as a file system
http://leo.zwiki.org/LeoAsAFileSystem

@language rest

#@+node:ekr.20101127154340.5936: *4* @url proto of a Leo forum
http://groups.google.com/group/leo-editor/browse_thread/thread/db6e75d82da4b41d

@language rest

A few buttons turns Leo into a competitor for google groups.

#@+node:ekr.20101129064803.6062: *4* @url stickynotes_plus and markdown
http://groups.google.com/group/leo-editor/browse_thread/thread/f8234a8fdeb08d22/4f42eafa955eae9b

Rich text editing in Leo

markdown: http://www.freewisdom.org/projects/python-markdown/

his is a Python implementation of John Gruber's Markdown. It is almost
completely compliant with the reference implementation.

http://daringfireball.net/projects/markdown/

Markdown is a text-to-HTML conversion tool for web writers. Markdown allows you
to write using an easy-to-read, easy-to-write plain text format, then convert it
to structurally valid XHTML (or HTML).
#@+node:ekr.20101129064803.6063: *4* @url leo+emacs+pida
http://groups.google.com/group/leo-editor/browse_thread/thread/f65470074f6573ab/c73d5b313526d5af

pida: http://pida.co.uk/

PIDA is an IDE (integrated development environment). PIDA is different from
other IDEs in that it will use the tools you already have available rather than
attempting to reinvent each one. PIDA is written in Python with the PyGTK
toolkit, and although is designed to be used to program in any language, PIDA
has fancy Python IDE.
#@+node:ekr.20101129064803.6065: *4* @url leoremote.py
http://groups.google.com/group/leo-editor/browse_thread/thread/66bcdc5cac03f5ad/0528c66838f1726d

communicating with a running Leo

@language rest

Terry Brown

For some time I've wanted a simple way to make an running leo session
load a file from the command line, like emacs-client.\

Now, thanks to Ville, I also want a way to make an running leo session
pop up a stickynote window (which would correspond to an automatically
created node which would probably use current data and time as a
headstring).

Ville

First working version of this (using qt) is now on trunk.

Testing it requires some manual work:

1. bzr pull
2. Put this script (it's a simple example client) to your leo-editor
directory: http://pastebin.com/f2bcbfd36
3. Enable leoremote.py plugin
4. Launch leo, do alt-x leoserv-start. The leo process where you last
ran leoserv-start is always the session that will receive your
commands
5. Try running the script (it's a python script, not leo script) you
created at #2 from another command prompt 

Terry

Very very cool Ville, thanks.

Here http://pastebin.com/mf678e24 is my version of a functioning
stickynote script. 

And here: http://pastebin.com/m236194c

is a quick hack at a script to edit (or create and edit) a file in leo
from the command line. 

Terry -> EKR

> Can you explain in more detail.  I am totally lost about why this
> gives us anything new.  Thanks.

It's probably of little benefit to people who work with mouse, menus, and icons
all the time. But if you do everything from the command line (i.e. your OS's
shell), then it makes moving things into leo much smoother.

Suppose I've run leo and checked my todo items for the day, and now
leo's buried under some other window and I'm working in the shell in
some directory I've just created where I've just unzipped something and
now I want to edit a file that was in the .zip.

I can either:

  - find the leo window
  - insert a node
  - active the open file dialog
  - navigate to the directory containing this file
  - select the file
  - and finally do the editing I want to do

or, with Ville's communication to the running leo:

  - enter on the command line `led foo.txt`
  - and do the editing I want to do

where led is a script which causes the running leo to create an @edit
node containing foo.txt and pop to the front with the node selected.

Previously I was much more likely to use emacs, just because it was
easier to invoke that way from the command line.

So, opening files, creating sticky notes, invoking leo to handle output
from grep or diff or whatever - all these things are better now. 


Matt Wilkie

The corresponding point and click process for this scenario is

a) select > r-click > Edit with Leo
b) or drag'n'drop from folder to Leo icon on task bar (or window if visible)

In short, I see this being a productivity boost for all users. 
#@+node:ekr.20101129064803.6066: *5* @url  Ville's script
http://pastebin.com/f2bcbfd36

from leo.external import lproto
import os

addr = open(os.path.expanduser('~/.leo/leoserv_sockname')).read()
print "will connect to",addr
pc  = lproto.LProtoClient(addr)
pc.send('''

g.es("hello world from remote") 
c = g.app.commanders()[0]

''')
#@+node:ekr.20101129064803.6067: *5* @url Terry's led script
http://pastebin.com/m236194c
#@+node:ekr.20101127152442.5903: *4* Ville
#@+node:ekr.20101127152442.5904: *5* @url use json to represent trees
http://mail.google.com/mail/#label/Cool/121a1f23e75fc348

@language rest

Often, it' useful to decouple node creation from data processing. Case
in point is the read code & the hash based speedup scheme I described
(and referred to again in previous thread!)

(for every instance of 'json' below, you can substitute "hierarchical
python object". I'm thinking of json because of interoperability &
human-readablity of json vs. pickle).

What I mean is:

- Provide a way to create tree structure from json object. This is
easy, just add function

c.createSubTreeFromJson(p, json)

The 'json' arg is just a string with subtree encoded in json. Something like

[ ('h1', 'body1', 'gnx1', [ ('h1.1' , 'body1.1' , 'gnx1.1'), ('h1.2' ,
'body1.2' , 'gnx1.2')], ('h2', ......]

i.e. it's a recursive data structure with list of nodes as tuples (h,
b, gnx, children) where 'children' is the recursive part. So far, so
good. That's the easy part - it's just like xml, apart from the fact
that xml is overkill for this problem, requiring nontrivial parser
code (and thus being of lower usability & speed). Yes, the json is
pretty much the same as repr(python_object).

Now, for the important part.

All code that creates trees (read code as the most relevant example!)
probably shouldn't create the tree directly. Rather, it should use c
and p to learn what it needs to, then compose a result json object and
pass it on to c.createSubTreeFromJson(p, json). This is important &
elegant distinction from current behaviour where the data processing
code also modify the tree. We can basically have tests for most of the
stuff without altering the tree in any way, allowing the tree to stay
"read only" as long as possible.

The hash speedup I've been talking about will execute the auto/thin
parsing code exactly once for the particular file content, store away
the json, and just read that json file on subsequent runs. There is no
need to speculate on feasibility of this scheme, we've already had
that particular discussion. It really is the simplest & fastest
possible solution.

We could also add lazy loading (though this is not as important), but
*also this* will be easier with this scheme. Basically, the lazy
loading would:

- Run in a thread in the background, collecting stuff from external
files to json objects
- Every once in a while, do the "destructive" thing and update the
tree from these json objects, redrawing if necessary.

Also this will be perfectly safe - the background thread never does
gui manipulation. At any given time, it's only reading a file, and
writing the json obj to buffer. No communication with leo gui is
necessary. If position with root obj no longer exist, we just throw
the json away and rescan if needed.
#@+node:ekr.20101127152442.5905: *5* @url dump snippets as yaml
http://mail.google.com/mail/#label/Cool/121b225322eb0a6a

@language rest

I experimented with creating a "direct" recursive data structure for leo trees.

It's pretty simple (who needs Haskell & friends when we have python):

def p_to_obj(p):
   return [p.h, p.b, p.gnx, [p_to_obj(po) for po in p.children_iter()]]

Yes, I removed utf8 encoding to keep it more elegant ;-).

The whole snippet is somewhat like this:

http://pastebin.com/f6a30f859

It creates a python object like this:

http://pastebin.com/f17a3a285

(list of p,h,gnx, children...)

Now, the interesting part. This can be directly encoded as Yaml, to
render the rather readable:

http://pastebin.com/f3d750af3

The part that needs work is the body text, it should probably be
improvable by using "block scalars".

This idea is related to my "jsonification" refactoring suggestion a
while ago. If functions like read code emitted stuff that p_to_obj
returns, the yaml structure could be used to debug / trace the data
(but still in machine-readable form). This could also be used to
communicate outline structure so that it can be both read, and copy
pasted from emails to real tree structure.

No need to take any action regarding this email. Just a food for
thought, perhaps for inspiration...
#@+node:ekr.20101127152442.5906: *5* @url creating debian packages
http://groups.google.com/group/leo-editor/browse_thread/thread/f83704ecc4ba225a/9e75ecc105aab953

@language rest

> Would it be possible to publish somewhere the scripts or other files you use
> to create the .deb files?

It's somewhat of a nontrivial effort (that I do manually), but the
debian/ directory is here:

https://code.launchpad.net/~leo-editor-team/leo-editor/packaging-jaunty

In order to build the package, I have set up the ubuntu "ppa" (you can
register it at launchpad), use "debuild -S -sa" to build the package
when everything is at correct place (it's picky about directory & file
names and such), and use "dput" to upload it to launchpad servers.
After a while (one hour?), the package is built.

I have a .leo file that I use to trace these steps. I can push it to
trunk. But there is no automatic script to do it; it's possible to
just run "dpkg-buildpackage -rfakeroot" when the debian/ directory is
at leo-editor directory, but that's not the "proper" way, as your
environment is almost never clean enough to get a reliable package
that directly reflects the source snapshot at that tag. 
#@+node:ekr.20101127154340.5927: *5* @url objtrees: object trees for unit testing
http://groups.google.com/group/leo-editor/browse_thread/thread/32c3a295d2dae35b
#@+node:ekr.20101127154340.6852: *5* @url codewise: ville's code completer
http://www.mail-archive.com/leo-editor@googlegroups.com/msg10145.html
#@+node:ekr.20101129064803.6060: *5* @url server code to interact with a running Leo
http://groups.google.com/group/leo-editor/browse_thread/thread/278aa85d7298a319/ef6446fcf6268c0d

leoRemote.py
#@+node:ekr.20101127154340.6826: *3* Leo urls
#@+node:ekr.20101127154340.6827: *4* @url NSIS manual
http://nsis.sourceforge.net/Docs/Contents.html

@language rest
#@+node:ekr.20101127154340.6833: *4* @url leo-editor-files
http://tinyurl.com/35ddr4w

@language rest
#@+node:ekr.20101127154340.6851: *4* @url upload to source forge
https://sourceforge.net/project/admin/explorer.php?group_id=3458

@language rest
#@+node:ekr.20101127154340.6837: *3* Math
#@+node:ekr.20101127154340.6838: *4* @url sage: mathematics software system
http://sagemath.org/

@language rest
#@+node:ekr.20101127154340.6839: *4* @url viewdog: viewer for math functions
http://gul.sourceforge.net/viewdog-manual/node3.html

@language rest
#@+node:ekr.20101127152442.5910: *3* Python tools
#@+node:ekr.20101127152442.5911: *4* @url clonedigger: find similar code
http://clonedigger.sourceforge.net/
#@+node:ekr.20101127154340.5928: *4* @url pyexpect: spawn child apps
http://pexpect.sourceforge.net/pexpect.html

@language rest

pexpect: a Python module for spawning child applications and
controlling them automatically. 
#@+node:ekr.20101129064803.6059: *4* @url pynotify: wait for a file to change
http://groups.google.com/group/leo-editor/browse_thread/thread/2b6cceebd7cd2e3/dddcb73e2a6469b9

@language rest

Linux only

For using inotify from Python, I've used pyinotify; it seems to be a
bit more mature: 

http://pypi.python.org/pypi/pyinotify/0.9.1

There's also inotifyx: (more portable?)

http://pypi.python.org/pypi/inotifyx/0.1.1 
#@+node:ekr.20101127154340.6804: *4* @url python trace module
http://docs.python.org/library/trace.html

@language rest

See the child node for a cool script.
(It is also in scripts.leo).
#@+node:ekr.20101127154340.6803: *5* Call hierarchy tracing (using python 'trace' module)
@language python
@tabwidth -4

"""
Analyzing program flow.

Run (ctrl+b) this script after 

cd ~/leo-editor
python -m trace --trackcalls launchLeo.py --gui=qt >trace.txt

"""

tracefile = '~/leo-editor/trace.txt'

import os

tr = open(os.path.expanduser(tracefile))
print tr
top = p.insertAsLastChild().copy()
top.h = 'trace session'
cur = None
no = None
for l in tr:
    if l.startswith('***'):
        cur = top.insertAsLastChild().copy()
        cur.h = os.path.basename(l.split()[1])
    elif l.startswith('  -->'):
        no = cur.insertAsLastChild().copy()
        no.h = os.path.basename(l.split()[1].strip())
    else:
        if no:
            no.b += l.strip() + '\n'

    print ".",
#@+node:ekr.20101127154340.5932: *4* @url pythoscope: create unit tests automatically
http://pythoscope.org/

@language rest
#@+node:ekr.20101127154340.5933: *4* @url rope: refactoring library
http://rope.sourceforge.net/

@language rest
#@+node:ekr.20101127152442.5891: *3* Scientific
#@+node:ekr.20101127152442.5892: *4* @url calioPY Scientific environment using Leo
http://www.caliopywork.org/
#@+node:ekr.20101127154340.6812: *4* @url reinteract: scientific platform
http://fishsoup.net/software/reinteract/

@language rest

#@+node:ekr.20101127152442.5893: *3* Text
#@+node:ekr.20101127152442.5894: *4* Related to viewrendered plugin
#@+node:ekr.20101127152442.5895: *5* @url enthought editor for restructured text
http://blog.enthought.com/?p=127

rst project of interest 2009/07/14

http://mail.google.com/mail/#label/Cool/1227cc2c66c976ec

It may be that the codebase could be worth looking at.  I find the
prospect of doing a full sphinx rendering of a document in real time
quite fascinating :-).

Requires ETS (Enthought Tool Suite)
http://code.enthought.com/projects/index.php
#@+node:ekr.20101127152442.5896: *5* @url autosphinx (related to enthought editor)
https://code.launchpad.net/~villemvainio/leo-editor/autosphinx

http://groups.google.com/group/leo-editor/browse_thread/thread/f292f7c9f2fd66d8

bzr branch lp:~villemvainio/leo-editor/autosphinx 

@language rest


No, we want *another window*, like with @button rst-preview, that is
updated in real time. The html output is not editable anyway.

>> Now, the thing is to
>> update this in real time (as you are editing, i.e. not on save or
>> explicit tangle step).

> How about doing it at idle time, that is, at most every half second?

It needs to happen in another process in order to not bog down normal
editing (if we stop doing that, the ui will hang). Generating sphinx
output is somewhat expensive operation.

> I don't want to involve any part of the file logic in rendering: it's too
> complex as it is.

With @auto-rst, we get this for free (i.e. rendering every time you
save). Barring that, we could do the rendering just for current node,
but that is not faithful to the final output.

I think the best implementation is a new process that monitors a file.
That way, it's not really leo-specific, and it's easy to do. Leo just
needs to tell it what file to monitor.

-- 
#@+node:ekr.20101127152442.5913: *5* @url manuel: rST testing tool
http://pypi.python.org/pypi/manuel

@language rest

Manuel parses documents, evaluates their contents,
then formats the result of the evaluation.
#@+node:ekr.20101127152442.5897: *4* @url controlling tex parameter from rst markup
http://groups.google.com/group/leo-editor/browse_thread/thread/20ec9f3ec33eb174/dacfce8823e727c1

@language rest

#@+node:ekr.20101127152442.5898: *5* solution 1: use @raw
@language rest

> On a related note, is there a good example somewhere of embedding
> LaTeX markup in rst?

You can use the ..raw directive

Here is a quote from another thread
http://groups.google.com/group/leo-editor/browse_thread/thread/16272e...

QQQ
One of the difficulties was how to "convince" Leo to write raw code, I
wanted to write some equations and tables, so it was necessary to use
symbols like \, { }, and so on, i. e., what is required by LaTeX. The
problem was that all these symbols were "interpreted" wrongly and
replaced by an unreadable code. The solution was to use the raw
directive. An example is much better to show this. Assume the
following LaTeX code for an equation and a table in a Leo node:

@
.. raw:: latex html

 \begin{equation}  f(x)=\frac{e^{X\beta}}{1-e^{X\beta}}  \end
{equation}

 \newline
 \begin{tabular}[b]{|r|l||c|r|}
    <content deleted ...>
 \end{tabular}

@c

Attention: RsT code requires an indentation with respect to the raw
directive. I didn't know it. It is here a leading blank space for all
LaTeX code inside the raw directive. And then it worked fine.
QQQ 
#@+node:ekr.20101127152442.5899: *5* solution 2: use make
I think you get the most control by having leo generate an rst file and
then processing it yourself.

Like this somewhat dated page (skip the first section):
http://leo.zwiki.org/RstEmacs

For a project I'm currently working on, currently :), my set up is like
this:

Edit rst in leo using an @auto-rst node.

Run this script (it is (pointlessly) a Makefile):

.. sourcecode:: make

  all:
        rst2html.py report2.rst report2.html
        itex2MML < report2.html > report2.xhtml
        rst2latex.py --documentoptions=letterpaper --stylesheet-path=myprefs.inc \
          --reference-label="ref*" --use-latex-citations report2.rst report2.tex
        echo | pdflatex -draftmode report2 >/dev/null 2>&1
        echo | pdflatex -draftmode report2 >/dev/null 2>&1
        echo | pdflatex report2 2>&1 | tr \\n \\r
        rst2latex.py --documentoptions=letterpaper --stylesheet-path=myprefs.inc \
          report2.rst report2.tex

The two -draftmode parses take care of references etc. much quick than
non-draft, because they don't chew up time turning pngs and pdfs into
part of the pdf output.  I.e. they don't produce pdf output.

``myprefs.inc`` looks like:

.. sourcecode:: latex

  \usepackage{pslatex}
  \usepackage{mathptmx}
  \usepackage[scaled=.90]{helvet}
  \usepackage{courier}
  \renewcommand\sfdefault{phv}%               use helvetica for sans serif
  \renewcommand\familydefault{\sfdefault}%    use sans serif by default
  \renewcommand{\topfraction}{0.85}
  \renewcommand{\textfraction}{0.1}
  \renewcommand{\floatpagefraction}{0.75}
  \setcounter{bottomnumber}{2}
  \renewcommand{\bottomfraction}{0.8}

  \setlength{\parskip}{2ex}
  \setlength{\parindent}{0pt}

  \addtolength{\oddsidemargin}{-1.5cm}
  \addtolength{\evensidemargin}{-1.5cm}
  \addtolength{\textwidth}{3cm}
  \addtolength{\topmargin}{-2cm}
  \addtolength{\textheight}{4cm}

  \usepackage{fancyhdr}
  \pagestyle{fancy}
  \lhead{Synoptic mapping of Native Plant Communities}
  \rhead{}
  \lfoot{\footnotesize 20091203-draft }

itex2MML (referenced in the ``Makefile``) converts latex math notation to MathML in the .xhtml output.

To insert raw latex in the latex output from rst, use

.. sourcecode:: rst

  .. raw:: latex

     \some{latex here}

Cheers -Terry 
#@+node:ekr.20101127152442.5900: *4* @url rst to anything
http://rst2a.com/

@language rest

A web service that converts reStructuredText to pdf or other formats.

You can use that to, say, quickly generate pdf from an (@auto-) rst file.

There is an api for the web service:

http://rst2a.com/api/
#@+node:ekr.20101127154340.6845: *4* @url mxTextTools: python text tools
http://www.egenix.com/products/python/mxBase/mxTextTools/

@language rest
#@+node:ekr.20101127154340.6847: *4* @url 2die4 games: TL (very cool rst stuff)
http://www.2die4games.com/

Thread: TL's 2die4games web site

http://groups.google.com/group/leo-editor/browse_thread/thread/cace15fe101e4844/6acd66a982bc063b
#@+node:ekr.20101127201907.5950: *4* @url idea - presentation tool
http://groups.google.com/group/leo-editor/browse_thread/thread/4ea2d3f7d2c68106/478c773f875815db

@language rest

- Create one QWebView window, zoom it in to have large fonts
- Create @button that converts current node containing
  restructuredtext to html, and pushes that html to QWebView.

Voila', instant presentation tool. The webview window would be on projector, and
leo would be in your private computer. You can easily edit the text, or find new
interesting slides to present in privacy of your own screen.

=====

Terry:
#@+node:ekr.20101127201907.5954: *4* @url rst/html in email
http://groups.google.com/group/leo-editor/browse_thread/thread/d119424cbccc96df/379a0600dfb8f4ca?lnk=gst&q=terry+brown#379a0600dfb8f4ca

@language rest

Was: "Anchors" as pseudo-persistent positions

Now: OT: rst/html in email

Important : You should try and view the HTML version of this message!

On Thu, 10 Dec 2009 10:46:05 -0600 Terry Brown <terry_n_brown@yahoo.com> wrote:

> Hey - would it be cool to have an email system which lets you write
> in rst and then sends both text (rst) and html forms...

Despite the fact y'all failed to chorus "Yes it would" ;-) I went ahead and set
it up in Claws-mail

   1.

      Add an rst-preview button to the Claws-mail compose window. It just runs a
      user command on the body text:

      | rst2pyg >~/.tmp.html; x-www-browser ~/.tmp.html

      rst2pyg is included below.
   2.

      Add an indent button to the Claws-mail compose window. It just runs a user command on the selected text:

          | sed 's/^/  /' |

   3.

      Write rst2email, included below.
   4.

      Use Send Later for messages you want to process, so rst2email can get at them in the queue directory.
   5.

      You can get rst2email_pygments.py (as imported by both rst2email and rst2pyg) from my blog, it's the file at the end of this page.

rst2email::

    #!/usr/bin/python
    """rst2email - look for messages in an email queue folder and
    add an html part by processing the text part as rst

    Terry Brown, terry_n_brown@yahoo.com
    """

    import email
    import mailbox
    from email.mime.text import MIMEText
    from docutils.core import publish_string

    # this import adds the sourcecode:: directive to rst
    import rst2email_pygments

    queue = "/home/tbrown/Mail/queue"

    mbox = mailbox.MH(queue, email.message_from_file)

    for msgkey in mbox.iterkeys():
        # d = email.utils.parsedate(msg.get('Date'))

        msg = mbox[msgkey]

        if not msg.is_multipart():
            txt = msg.get_payload()
            html = publish_string(txt, writer_name='html')
            part1 = MIMEText(txt, 'plain')
            part2 = MIMEText(html, 'html')
            part1["X-rst2email"] = "rst"
            part2["X-rst2email"] = "html"
            msg.set_type("multipart/alternative")
            msg.set_payload([])
            msg.attach(part1)
            msg.attach(part2)
            mbox[msgkey] = msg
        else:
            txtpart = None
            htmlpart = None
            for part in msg.walk():
                if part.get_content_type() == "text/plain":
                    if txtpart:  # can't handle more than one
                        txtpart = None
                        break
                    txtpart = part
                if part.get_content_type() == "text/html":
                    if htmlpart:  # can't handle more than one
                        htmlpart = None
                        break
                    htmlpart = part
            if txtpart and htmlpart and not txtpart.is_multipart():
                htmlpart.set_payload(
                    publish_string(txtpart.get_payload(),
    writer_name='html')) txtpart["X-rst2email"] = "rst"
                htmlpart["X-rst2email"] = "html"
                msg.set_type("multipart/alternative")
                mbox[msgkey] = msg

rst2pyg::



    #!/usr/bin/python

    from docutils.core import publish_string
    import rst2email_pygments
    import sys

    print publish_string(sys.stdin.read(), writer_name='html')

===== Ville M. Vainio	

This kind of stuff makes me think html email is actually a somewhat tolerable concept.

For thunderbird, there is pasteCode:

https://addons.mozilla.org/en-US/thunderbird/addon/4046

===== Terry Brown

Maybe :-) even as I wrote it it was more because I thought email
rendered (to html) from rst was cool, rather than that I think html
email is necessary.  It's funny looking at postings in this list vs. my
main inbox, here very very few msgs have html parts, whereas in the
main inbox at least 50% do.

Did occur to me that it would probably be possibly to set up some
combination of unicode chrs and css which would render leo trees
nicely in an html email.
#@+node:ekr.20101129064803.6064: *4* @url rst/latex tricks
http://groups.google.com/group/leo-editor/browse_thread/thread/d052979864a278bc/3ff8f82b2a80774d

@language rest

python in leo to generate rst

Here's a fun example of rst / latex tricks.  I want (well, the report
recipients want) 16 similar figures included.  I write a little python
in an rst comment, use ``Ctrl-b`` to execute it, and then an rst ``..
include::`` directive to include its output.  Here's the rst:

.. sourcecode:: rst

  The range of each input variable which occurs within each cluster
  could be examined to assign a biological meaning to each cluster.
  Figures `Upland cluster 1 and variables`_ through
  `Wetland cluster 8 and variables`_ show the
  relationships between clusters and variables.  The maps
  give a more general view of the clusters, and
  a variable by variable interpretation of each cluster may
  not add much useful information.

  ..  python

    out = file('/home/tbrown/Desktop/Proj/AitkinMap/varvsclust.rst', 'w')
    def pnt(x): out.write(x+'\n')
    for uw in 'Upland', 'Wetland':
      for i in range(1,9):
        pnt(".. figure:: plots/%s_Cluster_%d.pdf" % (uw,i))
        pnt("   :width: 95%")
        pnt("")
        pnt("   %s cluster %d and variables" % (uw,i))
        pnt("")
    out.close()

  .. include:: varvsclust.rst

Note: The .. python is just a comment.  "Empty" comments like::

    ..
      print 40+2
      print 'done'

don't work, because empty comments don't consume indented blocks
#@+node:ekr.20101127152442.5901: *3* Video tools
#@+node:ekr.20101127154340.6841: *4* @url screenr: instant screencasts for twitter
http://screenr.com/
#@+node:ekr.20101127154340.6842: *4* @url jing
http://jingproject.com
#@+node:ekr.20101127152442.5907: *3* Web technologies
#@+node:ekr.20101127154340.6811: *4* @url 0mq: socket library
http://www.zeromq.org/
#@+node:ekr.20101127154340.6810: *4* @url flask: micro devel framework for python
http://flask.pocoo.org/docs/

@language rest
#@+node:ekr.20101127154340.6848: *4* @url google app engine
http://code.google.com/appengine/

@language rest
#@+node:ekr.20101127154340.6849: *4* @url browser shots: web page testing
http://browsershots.org/

@language rest
#@+node:ekr.20101127154340.6850: *4* @url goosh: google shell
http://goosh.org/

@language rest
#@+node:ekr.20101127152442.5908: *4* @url json
http://www.json.org/

@language rest

**object**:

{   string : value,
    string : value,
    ...
} 

**array**:

[ value, value...]

**value**: string object array true false null

**number**: 
    int
    int frac
    int e digits
    int frac e digits 
#@+node:ekr.20101127152442.5909: *4* @url places to ask questions
serverfault.com
stackoverflow.com

@language rest

Quora
#@+node:ekr.20101127154340.5930: *4* @url pyjamas: AJAX tool kit
http://pyjs.org/

@language rest

Pyjamas: AJAX tool kit.
#@+node:ekr.20101127154340.6846: *4* @url w3c: web standards page
http://www.w3.org/

@language rest
#@+node:ekr.20101127154340.6853: *3* Windows
#@+node:ekr.20101127154340.6854: *4* @url cmd
http://commandwindows.com/command1.htm

@language rest
#@+node:ekr.20101127154340.6844: *4* @url process monitor for windows
http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx?PHPSESSID=d926

@language rest
#@+node:ekr.20101127154340.6855: *4* @url windows shortcuts
http://www.codeproject.com/Articles/36538/Windows-7-Tricks-and-Keyboard-Shortcuts.aspx

@language rest
#@+node:ekr.20111003130143.15533: ** Unused settings (from leoSettings.leo)
@ignore
#@+node:ekr.20111003130143.15534: *3* @bool trace_masterClickHandler = False
#@+node:ekr.20111003130143.15535: *3* Unused Options for jEdit colorizer
#@+node:ekr.20111003130143.15536: *4* @int colorizer_interrupt_count2 = 5000
The number of times the colorizer's main loop in colorOneChunk executes before actually colorizing text.
A reasonable value is 5000.
#@+node:ekr.20111003130143.15537: *4* @int colorizer_interrupt_count1 = 10
The number of times the colorizer's main loop in colorOneChunk executes before pausing to check for events.
A reasonable value is 10.
#@+node:ekr.20111024091133.16662: *3* Common colors (At least some *are* used by new colorizer)
#@+node:ekr.20111024091133.16663: *4* @@@color comment_color = firebrick3
The color of comments in code parts.
#@+node:ekr.20111024091133.16664: *4* @@@color cweb_section_name_color = red
The color of name in @<name@>
#@+node:ekr.20111024091133.16665: *4* @@@color directive_color = blue
The color of C/C++ preprocessor directive lines.
#@+node:ekr.20111024091133.16666: *4* @@@color forth_bracket_range_color = orange
#@+node:ekr.20111024091133.16667: *4* @@@color keyword_color = blue
The color of keywords of the present language.

For example, the color of "if", "try" and "except" in Python.
#@+node:ekr.20111024091133.16668: *4* @@@color latex_background_color = None
#@+node:ekr.20111024091133.16669: *4* @@@color section_name_brackets_color = blue
The color of the double angle brackets in << name >>

#@+node:ekr.20111024091133.16670: *4* @@@color string_color = #00aa00
The color of strings and their delimiters.

#@+node:ekr.20111103102153.9715: ** Unused scripts
#@+node:ekr.20111018143804.15945: *3* Not used from find-cmd-docstrings
#@+node:ekr.20111017211256.15900: *4* find_all_commands_classes
def find_all_commands_classes(c):
    
    g.trace(c.shortFileName())
    result = []
    
    for p in c.all_positions():
        if p.h.find('getPublicCommands') > -1:
            for parent in p.parents():
                if g.match_word(parent.h,0,'class'):
                    g.trace('commands class',parent.h)
                    result.append(parent.v)

    return result
#@+node:ekr.20111017211256.15902: *4* get_commanders
def get_commanders (files):
    
    '''Return a list of commanders corresponding to the given files.'''
    
    def normalize(s):
        return s.replace('\\','/').lower()
    
    paths = []
    for fn in files:
        args = [g.app.loadDir,'..']
        args.extend(fn.split('/'))
        paths.append(g.os_path_finalize_join(*args))
        
    result = []
    for path in paths:
        for f in g.app.windowList:
            if normalize(path) == normalize(f.c.fileName()):
                result.append(f.c) ; break

    return result
#@+node:ekr.20111017211256.15901: *4* find_all_docstrings
def find_all_docstrings(commanders):

    class_list = []
    for c in commanders:
        aList = find_all_commands_classes(c)
        if aList:
            class_list.extend([(c,v) for v in aList])
            
    result = []

    return result
#@+node:ekr.20111023210405.16576: *3* Script: find color tags
'''Find all tags arguments to colorRangeWithTag'''

# Not useful: most tag arguments are non-literal.

# .colorRangeWithTag(self,s,i,j,tag,delegate='',exclude_match=False):
    
for p in p.self_and_subtree():
    s,i,tag = p.b,0,'.colorRangeWithTag('
    while True:
        progress = i
        i = s.find(tag,i)
        if i > -1:
            j = s.find(')',i)
            k = s.find('\n',i)
            if -1 < j < k:
                args = s[i+len(tag):j].strip()
                print(args)
                i = j
            elif k > -1:
                i = k
            else:
                break
            assert progress < i
        else:
            break
print('done')
#@+node:ekr.20120319174225.9743: ** Unused documentation (from LeoDocs.leo)
#@+node:ekr.20120319174225.9766: *3* Using Outlines
#@+node:ekr.20120319174225.9767: *4* @@@file outlines.txt
@language rest
@tabwidth -4

@ @rst-options
call_docutils=False
code_mode=False
generate_rst=True
http_server_support = False
show_organizer_nodes=True
show_headlines=True
show_leo_directives=True
stylesheet_path=..\doc
write_intermediate_file = True
verbose=True
@c

@all
#@+node:ekr.20120319174225.9768: *5* @rst html\outlines.html
##############
Using Outlines
##############

This chapter tells how to use Leo's outlines. Most of the commands can be
invoked by clicking with the pointer in the outline pane or by using the pointer
to select the command from the ``Outline`` menu. The key bindings mentioned here
are those of the default EKR bindings.

.. contents::
    :depth: 2

#@+node:ekr.20120319174225.9769: *6* @rst-no-head links
.. _`A Tutorial Introduction to Leo`:   intro.html
.. _`Writing Programs in Leo`:          directives.html
.. _`Clones and views`:                 intro.html#clones-views
#@+node:ekr.20120319174225.9770: *6* @rst-no-head Outline Related
#@+node:ekr.20120319174225.9771: *7* Navigating through the outline
Leo has many commands that select nodes in the outline. These commands can be
found in the Outline:Go To menu.

As described in the tutorial, you can move about the outline by clicking on the
headlines or using Alt+arrow keys.
#@+node:ekr.20120319174225.9772: *7* Expanding & contracting nodes
You can expand or contract a node by clicking in the tree view icon to the
left of the headline. The icon in the Qt gui matches the native OS's tree
view icon, i.e. for Mac's, a triangle pointing right or down; on Windows, a
square containing a plus or minus. Expanding a node shows its immediate
children; contracting a node hides all its children. The corresponding
commands are ``expand-node`` and ``contract-node``. For more convenient
navigation, there are ``expand-and-go-right`` (Alt-Right) and
``contract-or-go-up`` (Alt-Left) commands.

The ``expand-all`` command expands every node in the outline.
``contract-all`` (Alt-hypen) contracts every node in the outline. Both
commands are available in the Outline->Expand/Contract... submenu.
``contract-all`` In all but the smallest outlines, ``expand-all`` is rarely
used, and is does not have a key binding.
#@+node:ekr.20120319174225.9773: *7* Creating and destroying nodes
The ``insert-node`` (Ctrl-I or Insert) command inserts a new node into the
outline. When invoked, (from any pane), it inserts a new node below the
presently selected node, and at the same level as that node, or at the
child level if it has a visible child. The ``delete-node`` command deletes
a node and all its children; it is initially unbound. If you want to retain
the children you must promote all the children before you do the delete.
#@+node:ekr.20120319174225.9774: *7* Cutting, pasting and deleting nodes
The ``cut-node`` (Ctrl-Shift-X) ``paste-node`` (Ctrl-Shift-V),
``copy-node`` (Ctrl-Shift-C) and ``delete-node`` commands work on nodes
rather than text. The ``cut-node`` and ``copy-node`` commands copy a text
representation of the outline to the clipboard. This representation is the
same as Leo's .leo file format with some information deleted. You may copy
this text representation into a body pane (or into any other text editor)
using Edit->Paste in the menus, Ctrl-V, or Alt-X paste-text.

**Warning**: If you want to preserve the "cloned" attribute of a node, or
want to paste the node as a clone of the node you cut or copied, use the
``past-retaining-clones`` command, which in the Outline menu is called
"Paste Node as Clone". The ``paste-node`` command instead creates a new,
distinct version of the node you previously cut or copied, though if there
were descendant nodes which were clones of each other, the new version will
have parallel, distinct nodes that are also clones of each other (just not
of the originals). You may paste a node between .leo files, but there can
be no clone relationship across files.

The ``paste-retaining-clones`` command is disabled if it would cause a node
to become a parent of itself. The Leo outline is thus mathematically a
*directed acyclic graph*: clones make it more flexible than a tree, but not
a generalized graph.
#@+node:ekr.20120319174225.9775: *7* Moving & Reorganizing nodes
The ``move-outline-up`` (Ctrl-U or Alt-Shift-Up), ``move-outline-down``
(Ctrl-D or Alt-Shift-Down), ``move-outline-left`` (Ctrl-L or
Alt-Shift-Left), and ``move-outline-right`` (Ctrl-R or Alt-Shift-Right)
commands move the currently selected node. **Important**: When focus is in
the outline pane, you can move nodes whithout adding the Alt modifier.
Shift-Up moves the select node up, etc.

The ``promote`` (Ctrl-[) command makes all the children of
a node siblings of the node. The ``demote`` (Ctrl-]) command makes all following
siblings of a node children of the node.
#@+node:ekr.20120319174225.9776: *7* Cloning nodes
A cloned node is a copy of a node that changes when the original changes.
One may also think of it as a single node that is hooked into the outline
at multiple positions. Because that single node brings along all its
descendants, changes are maintained across all the the clones of a node,
along with changes to its offspring (children, grandchildren, etc.), i.e.,
any changes are simultaneously made to the corresponding offspring of all
of those clones. A small red arrow in the icon box marks cloned nodes. You
can think of the arrow as pointing out that there are other paths to get to
this same node. There is no real distinction between the "original" node
and any of its clones. Any headline or body update of a clone headed
subtree affects all of its clones simultaneously. A cloned node becomes a
regular node whenever deletion of its other clones makes it the only one
left. Clones are useful for making alternate views of a program. See
`Clones and views`_ for full details.

The ``clone-node`` (Ctrl-\`) command creates a clone as the immediate
sibling of a selected node. You have to place it where you want it by
either using move commands, or cutting and paste the clone.
#@+node:ekr.20120319174225.9777: *7* Marking nodes
You can mark nodes in several ways:

- With the ``mark`` (Ctrl-M), ``mark-changed-items``, or ``mark-subheads``
  commands. The ``mark`` command unmarks the selected node if it is already
  marked.

- With the Find or Change commands. The "mark changes" and "mark finds"
  options are initially cleared; you can change them with the
  ``toggle-find-mark-changes-option`` and ``toggle-find-mark-finds-option``
  commands. Using those commands will put focus in the Find tab of the log
  pane. When the Find tab is visible, Alt in combination with the
  underlined option character will put focus on that option, and then the
  space bar will toggle the option. You can also change the options by
  clicking on them in the Find tab.

The ``goto-next-marked`` command selects the next marked node.
#@+node:ekr.20120319174225.9778: *7* Dragging nodes
You may drag a node (including all its descendants) from one place to
another in an outline. To start a drag, press the main (left) mouse button
while the cursor is over the icon for a node. The cursor will change to a
hand icon. If you release the mouse button while the hand cursor is above
the icon for another node, Leo will move the dragged node after that node.
If you release the mouse button when the hand cursor is not over an icon,
Leo will leave the outline pane as it is. Leo scrolls the outline pane as
the result of mouse-moved events, so to continue scrolling you must keep
moving the mouse.
#@+node:ekr.20120319174225.9779: *6* @rst-no-head Body Related
#@+node:ekr.20120319174225.9780: *7* Editing body text
-   Clicking mouse button 1 positions the insertion cursor just before the character
    underneath the mouse cursor, sets the input focus to this widget, and clears any
    selection in the widget. Dragging with mouse button 1 strokes out a selection
    between the insertion cursor and the character under the mouse. 

-   Double-clicking mouse button 1 selects the word under the mouse and
    positions the insertion cursor at the end of the word.

-   The ends of the selection can be adjusted by dragging with mouse button 1 while
    the Shift key is down; this will adjust the end of the selection that was
    nearest to the mouse cursor when button 1 was pressed. 

-   If there is no selection, normal printing characters are inserted at the point of the insertion cursor. 

-   If there is a selection, normal printing characters replace the selection.

-   If the mouse is dragged out of the body pane while button 1 is pressed, the
    entry will automatically scroll to make more text visible.

-   Left Arrow and Right Arrow move the cursor one character to the left or right
    and clear any selection in the text.

-   Shift Left or Shift Right move the cursor and modify the selection.

-   Control-Left and Control-Right move the insertion cursor by words,
    and Control-Shift-Left and Control-Shift-Right move the insertion
    cursor by words and also extend the selection.

-   Up Arrow and Down Arrow move the insertion cursor one line up or down and
    clear any selection in the text. 

-   Shift Up and Shift Right move the cursor and extend the selection.

-   Control-Up and Control-Down move the insertion cursor by paragraphs.

-   Control-Shift-Up and Control-Shift-Down move the insertion cursor by
    paragraphs and extend the selection.

-   Next (Page Down) and Prior (Page Up) keys move the
    insertion cursor one screen and clear any text selection.

-   Shift Next and Shift Prior move the cursor one screen and extend the selection.

-   Line movement is by text lines terminated by hard returns (newlines), not by
    displayed lines; if a text line is long and wraps across more than one display
    line, then the Up and Down movement will skip the extra wrapped display lines.

-   Home moves the insertion cursor to the beginning of its line and clears any
    selection in the widget.

-   Shift-Home moves the insertion cursor to the beginning of the line and extends the selection. 

-   End moves the insertion cursor to the end of the line and clears any selection in
    the widget.

-   Shift-End moves the cursor to the end of the line and extends the selection. 

-   Control-Home moves the insertion cursor to the beginning of the text and clears
    any selection in the widget.

-   Control-Shift-Home moves the insertion cursor to the beginning of the text and extends the selection. 

-   Control-End moves the insertion cursor to the end of the text and clears any
    selection.

-   Control-Shift-End moves the cursor to the end of the text and extends the selection. 

-   Select and Control-Space set the selection anchor to the position of the
    insertion cursor. They don't affect the current selection.

-   Shift-Select and Control-Shift-Space adjust the selection to the current position of the
    insertion cursor, selecting from the anchor to the insertion cursor if there was
    no previous selection. 

-   F16  (Copy on many Sun workstations) or Control-c copies the
    selection in the widget to the clipboard, if there is a selection.

-   F20 (Cut on many Sun workstations) or Control-x copies the
    selection in the widget to the clipboard and deletes the selection.
    These keys have no effect if no text is selected.

-   F18 (Paste on many Sun workstations) or Control-v inserts the
    contents of the clipboard at the position of the insertion cursor.

-   Delete deletes the text selection, or the character to the right of the cursor
    if there is no text selection.

-   Backspace deletes the selection, or the character to the left of the cursor
    if there is no text selection.
#@+node:ekr.20120319174225.9781: *7* Indenting body text automatically
Leo auto indents unless @nocolor is in effect.  Typing a newline
automatically inserts the same leading whitespace present on the previous line.

If Python is the present language, Leo inserts an additional tab if the previous
line ends with a colon.  When the smart_auto_indent setting is True, Leo uses Emacs-style
auto-indentation instead.  This style of auto-indent aligns newly created lines
with unmatched ( [ or { brackets in the previous line.
#@+node:ekr.20120319174225.9782: *7* Creating and destroying multiple body editors
Three commands in the Cmds->Body Editors menu allow one to create, destroy, or focus on separate editors in the body pane.
- The ``add-editor`` (Cmds->Body Editors->Add Body Editor) command adds a new editor in the body pane and gives it the body editor focus.
- The ``delete-editor`` (Cmds->Body Editors->Delete Body Editor) command deletes the editor with body editor focus.
- The ``cycle-editor-focus`` (Cmds->Body Editors->Change Body Editor) command cycles body editor focus between editors in the body text.

There is a single body editor focus which determines which body editor shows the content of the selected outline node and which allows that node to be edited; other body editors continue to show the node contents they last had when they had the body editor focus.
#@+node:ekr.20120319174225.9783: *6* @rst-no-head Miscellaneous
#@+node:ekr.20120319174225.9784: *7* Undoing operations
Leo supports unlimited undo for all typing and all commands. The
'undo_granularity' setting controls the granularity of undo. There are four
possible values:

node
    Starts a new undo unit when typing moves to a new node.

line (default)
    Starts a new undo unit when typing moves to new line.

word
    Starts a new undo unit when typing starts a new word.

char (not recommended)
    Starts a new undo unit for each character typed.
    This wastes lots of computer memory.

Internally, setUndoTypingParams() calls recognizeStartOfTypingWord() to recognize the
start of words. Plugins can modify recognizeStartOfTypingWord(). It should
return True if the typing indicated by the params starts a new 'word' for the
purposes of undo with 'word' granularity. setUndoTypingParams() calls this
method only when the typing could possibly continue a previous word. In other
words, undo will work safely regardless of the value returned. See the actual
code for recognizeStartOfTypingWord() for more details.
#@+node:ekr.20120319174225.9785: *7* Resizing panes
You can change the relative sizes of the outline and body panes by dragging the
splitter bar. The ``equal-sized-panes`` (Window->Equal Sized Panes) command resizes the panes so that each
fills half of the main window.
#@+node:ekr.20120319174225.9786: *7* Opening URL's automatically
Double-clicking the icon box of a node whose headline has the form::

    @url <any url>

executes the URL in your default web browser.

Leo checks that the URL is valid before doing so. A valid URL is:

-   3 or more lowercase alphas
-   followed by one :
-   followed by one or more of:
-   ``$%&'()*+,-./0-9:=?@A-Z_a-z{}~``
-   followed by one of: ``$%&'()*+/0-9:=?@A-Z_a-z}~`` (i.e., comma, hyphen and open curly brace may not be the last character)

URL's should contain no spaces: use %20 to indicate spaces. You may use any
type of URL that your browser supports: http, mailto, ftp, file, etc.
#@+node:ekr.20120319174225.9787: *7* Using chapters
Chapters are regions of a Leo outline whose root is an ``@chapter`` node. They
are available in an outline if the ``@bool usechapters`` option is True.
``@chapter`` nodes may appear anywhere in an outline, but the ``create-chapter``
command (see below) creates ``@chapter`` nodes as children of the first
``@chapters`` (note the ``s``) node in the outline.

One selects a chapter with the ``select-chapter`` command, after which Leo shows
only the nodes in the selected chapter; in this respect, chapters are like
hoists. The ``main`` chapter represents the entire outline and can not be
deleted by name. When chapters are in effect, Leo creates an ``@chapters`` node
for the use of ``create-chapter``.

Associated settings:

- The ``@bool use_chapters`` setting determines whether chapters are enabled.
- The ``@bool use_chapter_tabs`` setting determines whether the chapters
  pop-up menu appears in the icon area. Choosing a chapter name from this list selects a chapter.

When chapters are enabled, the Cmds->Chapters menu shows all available chapter commands:

- The ``chapter-create`` command creates an @chapter node and populates it with a single node.
- The ``chapter-remove`` command deletes the currently selected chapter.
- The ``chapter-select`` command prompts for a chapter name and makes only the nodes of the selected chapter visible.
- The ``chapter-move-node-to``, ``chapter-clone-node-to`` and ``chapter-copy-node-to`` commands
  prompt for a chapter name and add the currently selected node (and its descendants) to another chapter.
#@+node:ekr.20031218072017.434: ** Unused code
@ignore
@language python
@color
#@+node:ekr.20050920084036.207: *3* queryReplaceCommandsClass (limited to single node)
class queryReplaceCommandsClass (baseEditCommandsClass):

    '''A class to handle query replace commands.'''

    @others
#@+node:ekr.20050920084036.208: *4*  ctor & init (queryReplaceCommandsClass)
def __init__ (self,c):

    baseEditCommandsClass.__init__(self,c) # init the base class.
    self.regexp = False # True: do query-replace-regexp.  Set in stateHandler.

def init (self):

    self.qQ = None
    self.qR = None
    self.replaced = 0 # The number of replacements.
#@+node:ekr.20050920084036.209: *4*  getPublicCommands
def getPublicCommands (self):

    return {
        'query-replace':        self.queryReplace,
        'query-replace-regex':  self.queryReplaceRegex,
    }
#@+node:ekr.20050920084036.210: *4* Entry points
def queryReplace (self,event):

    '''Interactively find and replace text.
    This is not recommended: Leo's other find and change commands are more capable.'''
    self.regexp = False
    self.stateHandler(event)

def queryReplaceRegex (self,event):
    '''Interactively find and replace text using regular expressions.
    This is not recommended: Leo's other find and change commands are more capable.'''
    self.regexp = True
    self.stateHandler(event)
#@+node:ekr.20051005151838: *4* Helpers
#@+node:ekr.20050920084036.212: *5* doOneReplace
def doOneReplace (self,event):

    w = self.editWidget(event)
    if not w: return

    i = w.tag_ranges('qR')
    w.delete(i[0],i[1])
    ins = w.getInsertPoint()
    w.insert(ins,self.qR)
    self.replaced += 1
#@+node:ekr.20050920084036.219: *5* findNextMatch (query-replace)
def findNextMatch (self,event):

    '''Find the next match and select it.
    Return True if a match was found.
    Otherwise, call quitSearch and return False.'''

    k = self.k
    w = self.editWidget(event)
    if not w: return

    if g.app.gui.guiName() != 'tkinter':
        return g.es('command not ready yet',color='blue')

    w.tag_delete('qR')
    if self.regexp:
        << handle regexp >>
    else:
        << handle plain search >>
#@+node:ekr.20051005155611: *6* << handle regexp >>
try:
    regex = re.compile(self.qQ)
except Exception:
    self.quitSearch(event,'Illegal regular expression')
    return False

i = w.getInsertPoint()
txt = w.get(i,'end')
match = regex.search(txt)

if match:
    start = match.start()
    end = match.end()
    length = end - start
    i = w.getInsertPoint()
    w.setInsertPoint(i+start)
    w.tag_add('qR','insert','insert +%sc' % length)
    w.tag_config('qR',background='lightblue')
    i = w.getInsertPoint()
    txt = w.get(i,i+length)
    return True
else:
    self.quitSearch(event)
    return False
#@+node:ekr.20051005160923: *6* << handle plain search >> (tag_add & tag_config) LATER
i = w.search(self.qQ,'insert',stopindex='end')

if i:
    w.setInsertPoint(i)
    w.tag_add('qR','insert','insert +%sc' % len(self.qQ))
    w.tag_config('qR',background='lightblue')
    return True
else:
    self.quitSearch(event)
    return False
#@+node:ekr.20050920084036.211: *5* getUserResponse
def getUserResponse (self,event):

    w = self.editWidget(event)
    char = event and event.char or ''
    
    if not w or not char return

    if char in ('Y','y'):
        self.doOneReplace(event)
        if not self.findNextMatch(event):
            self.quitSearch(event)
    elif char in ('Q','q','\n','Return'):
        self.quitSearch(event)
    elif char == '!':
        while self.findNextMatch(event):
            self.doOneReplace(event)
    elif char in ('N','n','Delete'):
        # Skip over the present match.
        i = w.getInsertPoint()
        w.setInsertPoint(i + len(self.qQ))
        if not self.findNextMatch(event):
            self.quitSearch(event)

    w.seeInsertPoint()
#@+node:ekr.20050920084036.220: *5* quitSearch
def quitSearch (self,event,message=None):

    k = self.k
    w = self.editWidget(event)
    if not w: return

    w.tag_delete('qR')
    k.clearState()
    if message is None:
        message = 'Replaced %d occurences' % self.replaced
    k.setLabelGrey(message)
#@+node:ekr.20050920084036.215: *5* stateHandler
def stateHandler (self,event):

    k = self.k ; state = k.getState('query-replace')

    prompt = g.choose(self.regexp,'Query replace regexp','Query replace')

    if state == 0: # Get the first arg.
        self.init()
        k.setLabelBlue(prompt + ': ',protect=True)
        k.getArg(event,'query-replace',1,self.stateHandler)
    elif state == 1: # Get the second arg.
        self.qQ = k.arg
        if len(k.arg) > 0:
            prompt = '%s %s with: ' % (prompt,k.arg)
            k.setLabelBlue(prompt)
            k.getArg(event,'query-replace',2,self.stateHandler)
        else:
            k.resetLabel()
            k.clearState()
    elif state == 2: # Set the prompt and find the first match.
        self.qR = k.arg # Null replacement arg is ok.
        k.setLabelBlue('Query replacing %s with %s\n' % (self.qQ,self.qR) +
            'y: replace, (n or Delete): skip, !: replace all, (q or Return): quit',
            protect=True)
        k.setState('query-replace',3,self.stateHandler)
        self.findNextMatch(event)
    elif state == 3:
        self.getUserResponse(event)
#@+node:ekr.20101031172539.5880: *3* Existing autocompleter: do not delete
#@+node:ekr.20061031131434.43: *4* completeSelf (not used yet)
def completeSelf (self):

    g.trace(g.callers(4))

    # This scan will be fast if an instant object already exists.
    className,obj,p,s = self.classScanner.scan()
    # g.trace(className,obj,p,s and len(s))

    # First, look up the className.
    if not obj and className:
        obj = self.allClassesDict.get(className)
        # if obj: g.trace('found in allClassesDict: %s = %s' % (className,obj))

    # Second, create the object from class definition.
    if not obj and s:
        theClass = self.computeClassObjectFromString(className,s)
        if theClass:
            obj = self.createProxyObjectFromClass(className,theClass)
            if obj:
                self.selfObjectsDict [className] = obj
                # This prevents future rescanning, even if the node moves.
                self.selfVnodesDict [p.v] = obj
    if obj:
        self.selfClassName = className
        self.push(self.theObject)
        self.theObject = obj
        self.membersList = self.getMembersList(obj=obj)
    else:
        # No further action possible or desirable.
        self.selfClassName = None
        self.theObject = None
        self.clear()
        self.membersList = []
#@+node:ekr.20061031131434.60: *4* class forgivingParserClass (not used)
class forgivingParserClass:

    '''A class to create a valid class instances from
    a class definition that may contain syntax errors.'''

    @others
#@+node:ekr.20061031131434.61: *5* ctor (forgivingParserClass)
def __init__ (self,c):

    self.c = c
    self.excludedTnodesList = []
    self.old_putBody = None # Set in parse for communication with newPutBody.
#@+node:ekr.20061031131434.62: *5* parse
def parse (self,p):

    '''The top-level parser method.

    It patches c.atFileCommands.putBody, calls the forgiving parser and finally
    restores c.atFileCommands.putBody.'''

    c = self.c

    # Create an ivar for communication with newPutBody.
    self.old_putBody = c.atFileCommands.putBody

    # Override atFile.putBody.
    c.atFileCommands.putBody = self.newPutBody

    try:
        s = None
        s = self.forgivingParser(p)
    finally:
        c.atFileCommands.putBody = self.old_putBody

    return s # Don't put a return in a finally clause.


#@+node:ekr.20061031131434.63: *5* forgivingParser (leoKeys)
def forgivingParser (self,p,suppress=False):

    c = self.c ; root = p.copy()
    self.excludedTnodesList = []
    s = g.getScript(c,root,useSelectedText=False)
    while s:
        try:
            if not g.isPython3:
                s = g.toEncodedString(s)
            compile(s+'\n','<string>','exec')
            break
        except SyntaxError:
            fileName, n = g.getLastTracebackFileAndLineNumber()
            p = self.computeErrorNode(c,root,n,lines=g.splitLines(s))
            if not p or p == root:
                if not suppress:
                    g.es_print('syntax error in class node: can not continue')
                s = None ; break
            else:
                # g.es_print('syntax error: deleting',p.h)
                self.excludedTnodesList.append(p.v)
                s = g.getScript(c,root,useSelectedText=False)
        except Exception:
            g.trace('unexpected exception')
            g.es_exception()
            break
    return s or ''
#@+node:ekr.20061031131434.64: *5* computeErrorNode (leoKeys)
def computeErrorNode (self,c,root,n,lines):

    '''The from c.goToLineNumber that applies to scripts.
    Unlike c.gotoLineNumberOpen, this function returns a position.'''

    if n == 1 or n >= len(lines):
        return root

    # vnodeName, junk, junk, junk, junk = c.convertLineToVnodeNameIndexLine(
        # lines, n, root, scriptFind = True)

    goto = goToLineNumber(c)
    vnodeName,junk,junk,junk = goto.findVnode(
        root,lines,n,ignoreSentinels)

    if vnodeName:
        for p in root.self_and_subtree():
            if p.matchHeadline(vnodeName):
                return p

    return None
#@+node:ekr.20061031131434.65: *5* newPutBody
def newPutBody (self,p,oneNodeOnly=False,fromString=''):

    if p.v in self.excludedTnodesList:
        pass
        # g.trace('ignoring',p.h)
    else:
        self.old_putBody(p,oneNodeOnly,fromString)
#@+node:ekr.20061031131434.66: *4* class classScannerClass (not used)
# Called by completeSelf, which is not used.

class classScannerClass:

    '''A class to find class definitions in a node or its parents.'''

    @others
#@+node:ekr.20061031131434.67: *5* ctor (classScannerClass)
def __init__ (self,c):

    self.c = c

    # Ignore @root for now:
    # self.start_in_doc = c.config.getBool('at_root_bodies_start_in_doc_mode')

    self.start_in_doc = False
#@+node:ekr.20061031131434.68: *5* scan
def scan (self):

    c = self.c

    className,obj,p = self.findParentClass(c.p)
    # g.trace(className,obj,p)

    if p and not obj:
        parser = c.k.autoCompleter.forgivingParser
        s = parser.parse(p)
    else:
        s = None

    return className,obj,p,s
#@+node:ekr.20061031131434.69: *5* findParentClass
def findParentClass (self,root):

    autoCompleter = self.c.k.autoCompleter

    # First, see if any parent has already been scanned.
    for p in root.self_and_parents():
        obj = autoCompleter.selfVnodesDict.get(p.v)
        if obj:
            # g.trace('found',obj,'in',p.h)
            return None,obj,p

    # Next, do a much slower scan.
    # g.trace('slow scanning...')
    for p in root.self_and_parents():
        className = self.findClass(p)
        if className:
            # g.trace('found',className,'in',p.h)
            return className,None,p

    return None,None,None
#@+node:ekr.20061031131434.70: *5* findClass & helpers
def findClass (self,p):

    lines = g.splitLines(p.b)
    inDoc = self.start_in_doc
    # g.trace(p.h)
    for s in lines:
        if inDoc:
            if self.endsDoc(s):
                inDoc = False
        else:
            if self.startsDoc(s):
                inDoc = True
            else:
                # Not a perfect scan: a triple-string could start with 'class',
                # but perfection is not important.
                className = self.startsClass(s)
                if className: return className
    else:
        return None
#@+node:ekr.20061031131434.71: *6* endsDoc
def endsDoc (self,s):

    return s.startswith('@c')
#@+node:ekr.20061031131434.72: *6* startsClass
def startsClass (self,s):

    if s.startswith('class'):
        i = 5
        i = g.skip_ws(s,i)
        j = g.skip_id(s,i)
        word = s[i:j]
        # g.trace(word)
        return word
    else:
        return None
#@+node:ekr.20061031131434.73: *6* startsDoc
def startsDoc (self,s):

    for s2 in ('@doc','@ ','@\n', '@r', '@\t'):
        if s.startswith(s2):
            return True
    else:
        return False
#@+node:ekr.20061031131434.57: *4* Proxy classes and objects
#@+node:ekr.20061031131434.58: *5* createProxyObjectFromClass
def createProxyObjectFromClass (self,className,theClass):

    '''Create a dummy instance object by instantiating theClass with a dummy ctor.'''

    if 0: # Calling the real ctor is way too dangerous.
        # Set args to the list of required arguments.
        args = inspect.getargs(theClass.__init__.im_func.func_code)
        args = args[0] ; n = len(args)-1
        args = [None for z in range(n)]

    def dummyCtor (self):
        pass

    try:
        obj = None
        old_init = hasattr(theClass,'__init__') and theClass.__init__
        theClass.__init__ = dummyCtor
        obj = theClass()
    finally:
        if old_init:
            theClass.__init__ = old_init
        else:
            delattr(theClass,'__init__')

    g.trace(type(theClass),obj)

    # Verify that it has all the proper attributes.
    # g.trace(g.listToString(dir(obj)))
    return obj
#@+node:ekr.20061031131434.59: *5* createClassObjectFromString
def computeClassObjectFromString (self,className,s):

    try:
        # Add the the class definition to the present environment.
        exec(s) # Security violation!

        # Get the newly created object from the locals dict.
        theClass = locals().get(className)
        return theClass

    except Exception:
        if 1: # Could be a weird kind of user error.
            g.es_print('unexpected exception in',computeProxyObject)
            g.es_exception()
        return None
#@+node:ekr.20080924032842.3: *4* getExternalCompletions
def getExternalCompletions (self,s,p=None,language='python'):

    '''Return the completions possible at the end of string 's'.
       Return (theObject,completions):
    - theObject is None unless the last character is 's' is a period.
    - completions is the list of valid completions.'''

    c = self.c ; k = c.k
    if not p: p = c.p

    # Use a separate widget containing just s.
    self.widget = w = leoFrame.stringTextWidget(c,'compute-completions-widget')
    w.setAllText(s)

    # Scan back for the first period.
    i = len(s)-1
    # while i > 0 and s[i] != '.':
    while i > 0 and g.isWordChar(s[i]):
        i -= 1
    if s[i] == '.': i += 1
    prefix = s[i:].strip()

    # Remember the prefix, but put the insert before the period.
    w.setSelectionRange(i, len(s)-1, insert=i)

    # Init the ivars...
    self.language = p and g.scanForAtLanguage(c,p) or language
    self.tabName = ''
    old_enable = c.k.enable_autocompleter

    # Get the completions.
    try:
        c.k.enable_autocompleter = True
        self.useTabs = False
        self.start(prefix=prefix)
    finally:
        c.k.enable_autocompleter = old_enable
        self.useTabs = True

    theObject,tabList = self.theObject,self.tabList
    self.exit() # Not called from the autocompleter itself.
    return theObject,tabList
#@+node:ekr.20110916215321.8056: *3* Unused (C_to_python)
# These caused lots of pylint errors.
if 0:
    @others
#@+node:ekr.20110917104720.6871: *4* Drivers & helpers
#@+node:ekr.20110916215321.7989: *5* convertLeo1to2 & helper
def convertLeo1to2 (self):

    for p in self.p.self_and_subtree():
        g.es("converting:",p.h)
        s=convertStringLeo1to2(p.b)
        p.b = s

    g.es("done")
#@+node:ekr.20110916215321.7990: *6* convertStringLeo1to2
def convertStringLeo1to2 (self,s):
    
    aList = [z for z in s]
    outputList = []
    i = 0
    while i < len(aList):
        j = skipCodePart(aList,i)
        if j > i:
            code = aList[i:j]
            convertCodeList1to2(code)
            i = j
            #print "-----code:", listToString(code)
            for item in code:
                outputList.append(item)
        j = skipDocPart(aList,i)
        if j > i:
            doc = aList[i:j]
            convertDocList(doc) # same as in c2py
            #print "-----doc:", listToString(doc)
            i = j
            for item in doc:
                outputList.append(item)

    result = listToString(outputList)
    global printFlag
    if printFlag: print("-----:\n", result)
    return result
#@+node:ekr.20110916215321.7994: *5* convertLeoTree
def convertLeoTree (self):

    for p2 in self.p.self_and_subtree():
        print("converting:",v.h)
        s = convertCStringToPython(p2.b,leoFlag=True)
        p2.b = s

    g.es('done')
#@+node:ekr.20110916215321.7995: *5* convertCFileToPython
def convertCFileToPython(fn):

    f=open(fn, 'r')
    if not f: return
    s = f.read()
    f.close()

    f=open(fn + ".py", 'w')
    if not f: return
    s = convertCStringToPython(s,leoFlag=False)
    f.write(s)
    f.close()
#@+node:ekr.20110916215321.7991: *5* convertCodeList1to2
@ We do _not_ replace @root by @file or insert @others as needed.
Inserting @others can be done easily enough by hand,
and may take more global knowledge than we can reasonably expect to have.
@c

def convertCodeList1to2 (self,aList):

    if 0: # There isn't much reason to do this.
        removeAtRoot(aList)

    self.safeReplace(aList,"@code","@c")
    self.replaceSectionDefs(aList)
    self.removeLeadingAtCode(aList)
#@+node:ekr.20110916215321.7996: *5* convertCStringToPython
def convertCStringToPython (self,s,leoFlag):

    # print "convertCStringToPython:start\n", s
    firstPart = True
    aList = [z for z in s]

    if not leoFlag:
        self.convertCodeList(aList,firstPart,dontDoLeoTranslations)
        return ''.join(aList)

    outputList = []
    i = 0
    while i < len(aList):
        j = skipCodePart(aList,i)
        if j > i:
            code = aList [i: j]
            convertCodeList(code,firstPart,doLeoTranslations)
            i = j
            #print "-----code:", ''.join(code)
            for item in code:
                outputList.append(item)
        firstPart = False # don't remove @c from here on.
        j = skipDocPart(aList,i)
        if j > i:
            doc = aList [i: j]
            convertDocList(doc)
            #print "-----doc:", ''.join(doc)
            i = j
            for item in doc:
                outputList.append(item)

    result = ''.join(outputList)
    global printFlag
    if printFlag: print("-----:\n",result)
    return result
#@+node:ekr.20110917104720.6876: *5* get_default_user_types
def get_default_user_types (self):
    
    '''Return default user types.'''
    
    self.class_list = [
        # "vnode", "tnode", "Commands",
        # "wxString", "wxTreeCtrl", "wxTextCtrl", "wxSplitterWindow",
    ]

    self.ivars_dict = {
        # "atFile": [
            # "mCommands", "mErrors", "mStructureErrors",
            # "mTargetFileName", "mOutputFileName", "mOutputStream",
            # "mStartSentinelComment", "mEndSentinelComment", "mRoot",
        # ],
        # "vnode": [
            # "mCommands", "mJoinList", "mIconVal", "mTreeID", "mT", "mStatusBits",
        # ],
        # "tnode": [
            # "mBodyString", "mBodyRTF", "mJoinHead", "mStatusBits", "mFileIndex",
            # "mSelectionStart", "mSelectionLength", "mCloneIndex",
        # ],
        # "LeoFrame": [
            # "mNextFrame", "mPrevFrame", "mCommands",
        # ],
        # "Commands": [
            # # public
            # "mCurrentVnode", "mLeoFrame", "mInhibitOnTreeChanged", "mMaxTnodeIndex",
            # "mTreeCtrl", "mBodyCtrl", "mFirstWindowAndNeverSaved",
            # #private
            # "mTabWidth", "mChanged", "mOutlineExpansionLevel", "mUsingClipboard",
            # "mFileName", "mMemoryInputStream", "mMemoryOutputStream", "mFileInputStream",
            # "mInputFile", "mFileOutputStream", "mFileSize", "mTopVnode", "mTagList",
            # "mMaxVnodeTag",
            # "mUndoType", "mUndoVnode", "mUndoParent", "mUndoBack", "mUndoN",
            # "mUndoDVnodes", "mUndoLastChild", "mUndoablyDeletedVnode",
        # ],
    }
#@+node:ekr.20110919211949.6921: *4* Multi-character code
# This is all fairly horrible code...

# None of it has been tested...
#@+node:ekr.20110919211949.6922: *5* match
def match (self,s,i,pat):
    
    '''Return True if s[i:] matches the pat string.
    
    We can't use g.match because s is usually a list.
    '''
    
    assert pat
    
    aList,i,n = [],0,0
    while n < len(pat) and i < len(s):
        token = s[i]
        n += len(token)
        aList.append(token)
    
    return pat.startswith(''.join(aList))
#@+node:ekr.20110919211949.6923: *5* match_word
def match_word (self,s,i,pat):
    
    '''Return True if s[i:] word matches the pat string.'''
    
    aList,i,n = [],0,0
    while i < len(s) and n < len(pat):
        token = s[i]
        i += 1
        n += len(token)
        aList.append(token)
    if n == len(pat):
        return pat == ''.join(aList)
    elif n > len(pat):
        s2 = ''.join(aList)
        if pat.startswith(s2):
            ch = s2[len(pat)]
            return not ch.isalnum() and not ch == '_'
        else:
            return False
    else: # n < len(pat)
        assert i == len(s)
        return False
#@+node:ekr.20110917104720.6873: *4* Scanning
#@+node:ekr.20110916215321.8002: *5* convertLeadingBlanks
def convertLeadingBlanks(self,aList):

    w = self.tab_width
    if w < 2: return

    i = 0
    while i < len(aList):
        n = 0
        while i < len(aList) and aList[i] == ' ':
            n += 1 ; i += 1
            if n == w:
                aList[i-w:i] = ['\t']
                i = i - w + 1
                n = 0
        i = self.skip_past_line(aList, i)
#@+node:ekr.20110916215321.7998: *5* convertDocList
def convertDocList (self,docList):

    # print "convertDocList:", docList
    if self.match_word(docList,0,"@doc"):
        i = self.skip_ws(docList,4)
        if self.match(docList,i,"\n"):
            i += 1
        docList [0: i] = [z for z in ("@ ")]
#@+node:ekr.20110916215321.8000: *5* skipCodePart
def skipCodePart (self,aList,i):

    # print "skipCodePart", i
    if self.match_word(aList,i,"@doc") or self.match_word(aList,i,"@"):
        return i
    while i < len(aList):
        if self.match(aList,i,"//"):
            i = skipPastLine(aList,i)
        elif self.match(aList,i,"/*"):
            i = skipCBlockComment(aList,i)
        elif self.match(aList,i,'"') or self.match(aList,i,"'"):
            i = skipString(aList,i)
        elif self.match(aList,i,"\n"):
            i += 1
            if self.match_word(aList,i,"@doc") or self.match_word(aList,i,"@"):
                break
        else: i += 1
    return i
#@+node:ekr.20110916215321.7999: *5* skipDocPart
def skipDocPart (self,aList,i):

    # print "skipDocPart", i
    while i < len(aList):
        if self.match_word(aList,i,"@code") or self.match_word(aList,i,"@c"):
            break
        elif isSectionDef(aList,i):
            break
        else: i = skipPastLine(aList,i)
    return i
#@+node:ekr.20110917104720.6872: *4* Utils
#@+node:ekr.20110916215321.8018: *5* findInCode
def findInCode (self,aList,i,findString):

    while i < len(aList):
        if self.is_string_or_comment(aList,i):
            i = self.skip_string_or_comment(aList,i)
        elif self.match(aList,i,findString):
            return i
        else:
            i += 1

    return -1
#@+node:ekr.20110916215321.8019: *5* findInList
def findInList (self,aList,i,findString):

    while i < len(aList):
        if self.match(aList,i,findString):
            return i
        else:
            i += 1

    return-1
#@+node:ekr.20110916215321.7986: *4* speedTest
def speedTest (self,passes):

    import time
    fn = r"c:\prog\LeoPy\LeoPy.leo"
    f=open(fn)
    if not f:
        print("not found: ",fn)
        return
    s=f.read()
    f.close()
    print("file:", fn, " size:", len(s), " passes:", passes)
    print("speedTest start")
    time1 = time.clock()
    p = passes
    while p > 0:
        n = len(s) ; i = 0 ; lines = 0
        while -1 < i < n:
            if s[i] == '\n':
                lines += 1 ; i += 1
            else:
                i = s.find('\n',i) # _much_ faster than list-based-find.
            continue
            # match is about 9 times slower than simple test.
            if s[i]=='\n': # match(s,i,'\n'): # 
                i += 1
            else:
                i += 1
        p -= 1
    time2 = time.clock()
    print("lines:", lines)
    print("speedTest done:")
    print("elapsed time:", time2-time1)
    print("time/pass:", (time2-time1)/passes)
#@+node:ekr.20110916215321.7981: *4* test
def test(self):
    
    << define test data >>
    
    self.print_flag = True
    for s in testData:
        self.convertCStringToPython(s,leoFlag=True)
#@+node:ekr.20110917104720.6875: *5* << define test data >>
testData = [
"\n@doc\n\
This is a doc part: format, whilest, {};->.\n\
<<\
section def>>=\n\
LeoFrame::LeoFrame(vnode *v, char *s, int i)\n\
{\n\
    // test ; {} /* */.\n\
    #if 0 //comment\n\
        if(gLeoFrameList)gLeoFrameList -> mPrevFrame = this ;\n\
        else\n\
            this -> mNextFrame = gLeoFrameList ;\n\
    #else\n\
        \n\
        vnode *v = new vnode(a,b);\n\
        Commands *commander = (Commands) NULL ; // after cast\n\
        this -> mPrevFrame = NULL ;\n\
    #endif\n\
    if (a==b)\n\
        a = 2;\n\
    else if (a ==c)\n\
        a = 3;\n\
    else return; \n\
    /* Block comment test:\n\
        if(2):while(1): end.*/\n\
    for(int i = 1; i < limit; ++i){\n\
        mVisible = FALSE ;\n\
        mOnTop = TRUE ;\n\
    }\n\
    // trailing ws.	 \n\
    mCommands = new Commands(this, mTreeCtrl, mTextCtrl) ;\n\
    gActiveFrame = this ;\n\
}\n\
    ", "<<" +
"vnode methods >>=\n\
\n\
void vnode::OnCopyNode(wxCommandEvent& WXUNUSED(event))\n\
{\n\
    mCommands -> copyOutline();\n\
}\n\
\n@doc\n\
another doc part if, then, else, -> \n<<" +
"vnode methods >>=\n\
void vnode::OnPasteNode(wxCommandEvent& WXUNUSED(event))\n\
{\n\
    mCommands -> pasteOutline();\n\
}\n" ]
#@+node:ekr.20110916215321.8023: *4* removeAllCComments (Not used)
def removeAllCComments (self,aList,delim):

    i = 0
    while i < len(aList):
        progress = i
        if self.match(aList,i,"'") or self.match(aList,i,'"'):
            i = self.skip_string(aList,i)
        elif self.match(aList,i,"//"):
            j = self.skip_to_end_of_line(aList,i)
            print("deleting single line comment:", ''.join(aList[i:j]))
            del aList[i:j]
        elif self.match(aList,i,"/*"):
            j = self.skip_c_block_comment(aList,i)
            print("deleting block comment:", ''.join(aList[i:j]))
            del aList[i:j]
        else:
            i += 1
        assert i > progress
#@+node:ekr.20110916215321.8024: *4* removeAllCSentinels
def removeAllCSentinels (self,aList,delim):

    i = 0
    while i < len(aList):
        if self.match(aList,i,"'") or self.match(aList,i,'"'):
            # string starts a line.
            i = self.skip_string(aList,i)
            i = self.skip_past_line(aList,i)
        elif self.match(aList,i,"/*"):
            # block comment starts a line
            i = self.skip_c_block_comment(aList,i)
            i = self.skip_past_line(line,i)
        elif self.match(aList,i,"//@"):
            j = self.skip_past_line(aList,i)
            print("deleting sentinel:", ''.join(aList[i:j]))
            del aList[i:j]
        else:
            i = self.skip_past_line(aList,i)
#@+node:ekr.20110916215321.8025: *4* removeAllPythonComments
def removeAllPythonComments (self,aList,delim):

    i = 0
    while i < len(aList):
        if self.match(aList,i,"'") or self.match(aList,i,'"'):
            i = self.skip_string(aList,i)
        elif self.match(aList,i,"#"):
            j = self.skip_past_line(aList,i)
            print("deleting comment:", ''.join(aList[i:j]))
            del aList[i:j]
        else:
            i += 1
#@+node:ekr.20110916215321.8026: *4* removeAllPythonSentinels
def removeAllPythonSentinels (self,aList,delim):

    i = 0
    while i < len(aList):
        if self.match(aList,i,"'") or self.match(aList,i,'"'):
            # string starts a line.
            i = self.skip_string(aList,i)
            i = self.skip_past_line(aList,i)
        elif self.match(aList,i,"#@"):
            j = self.skip_past_line(aList,i)
            print("deleting sentinel:", ''.join(aList[i:j]))
            del aList[i:j]
        else:
            i = self.skip_past_line(aList,i)
#@+node:ekr.20110916215321.8027: *4* removeAtRoot
def removeAtRoot (self,aList):

    i = self.skip_ws(aList, 0)
    if self.match_word(aList,i,"@root"):
        j = self.skip_past_line(aList,i)
        del aList[i:j]

    while i < len(aList):
        if self.is_string_or_comment(aList,i):
            i = self.skip_string_or_comment(aList,i)
        elif self.match(aList,i,"\n"):
            i = self.skip_ws(aList, i+1)
            if self.match_word (aList,i,"@root"):
                j = self.skip_past_line(aList,i)
                del aList[i:j]
        else: i += 1
#@+node:ekr.20110916215321.8031: *4* removeLeadingAtCode
def removeLeadingAtCode (self,aList):

    i = self.skip_ws_and_nl(aList,0)
    if self.match_word(aList,i,"@code"):
        i = self.skip_ws_and_nl(aList,5)
        del aList[0:i]
    elif self.match_word(aList,i,"@c"):
        i = self.skip_ws_and_nl(aList,2)
        del aList[0:i]
#@-all
#@-leo
