This is ../info/internals.info, produced by makeinfo version 4.7 from
internals/internals.texi.

INFO-DIR-SECTION XEmacs Editor
START-INFO-DIR-ENTRY
* Internals: (internals).       XEmacs Internals Manual.
END-INFO-DIR-ENTRY

   Edition History:

   Created November 1995 (?) by Ben Wing.  XEmacs Internals Manual
Version 1.0, March, 1996.  XEmacs Internals Manual Version 1.1, March,
1997.  XEmacs Internals Manual Version 1.4, March, 2001.  XEmacs
Internals Manual Version 21.5, October, 2004.

   Copyright (C) 1992 - 2004 Ben Wing.  Copyright (C) 1996, 1997 Sun
Microsystems.  Copyright (C) 1994 - 1998, 2002, 2003 Free Software
Foundation.  Copyright (C) 1994, 1995 Board of Trustees, University of
Illinois.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that the
entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the Foundation.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided also
that the section entitled "GNU General Public License" is included
exactly as in the original, and provided that the entire resulting
derived work is distributed under the terms of a permission notice
identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that the section entitled "GNU General Public License"
may be included in a translation approved by the Free Software
Foundation instead of in the original English.


File: internals.info,  Node: Future Work -- Version Number and Development Tree Organization,  Next: Future Work -- Improvements to the `xemacs.org' Website,  Prev: Future Work -- Extents,  Up: Future Work

40.13 Future Work - Version Number and Development Tree Organization
====================================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* The purpose of this proposal is to present a coherent
plan for how development branches in XEmacs are managed.  This will
cover such issues as stable versus experimental branches, creating new
branches, synchronizing patches between branches, and how version
numbers are assigned to branches.

   A development branch is defined to be a linear series of releases of
the XEmacs code base, each of which is derived from the previous one.
When the XEmacs development tree is forked and two branches are created
where there used to be one, the branch that is intended to be more
stable and have fewer changes made to it is considered the one that
inherits the parent branch, and the other branch is considered to have
begun at the branching point.  The less stable of the two branches will
eventually be forked again, while this will not happen usually to the
more stable of the two branches, and its development will eventually
come to an end.  This means that every branch has a definite ending
point.  For example, the 20.x branch began at the point when the
released 19.13 code tree was split into a 19.x and a 20.x branch, and a
20.x branch will end when the last 20.x release (probably numbered 20.5
or 20.6) is released.

   I think that there should always be three active development
branches at any time.  These branches can be designated the stable, the
semi-stable, and the experimental branches.  This situation has existed
in the current code tree as soon as the 21.0 development branch was
split.  In this situation, the stable branch is the 20.x series.  The
semi-stable branch is the 21.0 release and the stability releases that
follow.  The experimental branch is the branch that was created as the
result of the 21.0 development branch split.  Typically, the stable
branch has been released for a long period of time.  The semi-stable
branch has been released for a short period of time, or is about to be
released, and the experimental branch has not yet been released, and
will probably not be released for awhile.  The conditions that should
hold in all circumstances are:

  1.  There should be three active branches.

  2.  The experimental branch should never be in feature freeze.


   The reason for the second condition is to ensure that active
development can always proceed and is never throttled, as is happening
currently at the end of the 21.0 release cycle.  What this means is
that as soon as the experimental branch is deemed to be stable enough
to go into feature freeze:

  1.  The current stable branch is made inactive and all further
     development on it ceases.

  2.  The semi-stable branch, which by now should have been released
     for a fair amount of time, and should be fairly stable, gets
     renamed to the stable branch.

  3.  The experimental branch is forked into two branches, one of which
     becomes the semi-stable branch, and the other, the experimental
     branch.


   The stable branch is always in high resistance, which is to say that
the only changes that can be made to the code are important bug fixes
involving a small amount of code where it should be clear just by
reading the code that no destabilizing code has been introduced.  The
semi-stable branch is in low resistance, which means that no major
features can be added, but except right before a release fairly major
code changes are allowed.  Features can be added if they are
sufficiently small, if they are deemed sufficiently critical due to
severe problems that would exist if the features were not added (for
example, replacement of the unexec mechanism with a portable solution
would be a feature that could be added to the semi-stable branch
provided that it did not involve an overly radical code re-architecture,
because otherwise it might be impossible to build XEmacs on some
architectures or with some compilers), or if the primary purpose of the
new feature is to remedy an incompleteness in a recent architectural
change that was not finished in a prior release due to lack of time (for
example, abstracting the mouse pointer and list-of-colors interfaces,
which were left out of 21.0).  There is no feature resistance in place
in the experimental branch, which allows full development to proceed at
all times.

   In general, both the stable and semi-stable branches will contain
previous net releases.  In addition, there will be beta releases in all
three branches, and possibly development snapshots between the beta
releases.  It's obviously necessary to have a good version numbering
scheme in order to keep everything straight.

   First of all, it needs to be immediately clear from the version
number whether the release is a beta release or a net release.  Steve
has proposed getting rid of the beta version numbering system, which I
think would be a big mistake.  Furthermore, the net release version
number and beta release version number should be kept separate, just as
they are now, to make it completely clear where any particular release
stands.  There may be alternate ways of phrasing a beta release other
than something like 21.0 beta 34, but in all such systems, the beta
number needs to be zero for any release version.  Three possible
alternative systems, none of which I like very much, are:

  1.  The beta number is simply an extra number in the regular version
     number.  Then, for example, 21.0 beta 34 becomes 21.0.34.  The
     problem is that the release version, which would simply be called
     21.0, appears to be earlier than 21.0 beta 34.

  2.  The beta releases appear as later revisions of earlier releases.
     Then, for example, 21.1 beta 34 becomes 21.0.34, and 21.0 beta 34
     would have to become 21.-1.34.  This has both the obvious ugliness
     of negative version numbers and the problem that it makes beta
     releases appear to be associated with their previous releases,
     when in fact they are more closely associated with the following
     release.

  3.  Simply make the beta version number be negative.  In this scheme,
     you'd start with something like -1000 as the first beta, and then
     21.0 beta 34 would get renumbered to 21.0.-968.  Obviously, this
     is a crazy and convoluted scheme as well, and we would be best to
     avoid it.


   Currently, the between-beta snapshots are not numbered, but I think
that they probably should be.  If appropriate scripts are handled to
automate beta release, it should be very easy to have a version number
automatically updated whenever a snapshot is made.  The number could be
added either as a separate snapshot number, and you'd have 21.0 beta 34
pre 1, which becomes before 21.0 beta 34; or we could make the beta
number be floating point, and then the same snapshot would have to be
called 21.0 beta 33.1.  The latter solution seems quite kludgey to me.

   There also needs to be a clear way to distinguish, when a net
release is made, which branch the release is a part of.  Again, three
solutions come to mind:

  1.  The major version number reflects which development branch the
     release is in and the minor version number indicates how many
     releases have been made along this branch.  In this scheme, 21.0
     is always the first release of the 21 series development branch,
     and when this branch is split, the child branch that becomes the
     experimental branch gets version numbers starting with 22.  This
     scheme is the simplest, and it's the one I like best.

  2.  We move to a three-part version number.  In this scheme, the
     first two numbers indicate the branch, and the third number
     indicates the release along the branch.  In this scheme, we have
     numbers like 21.0.1, which would be the second release in the 21.0
     series branch, and 21.1.2, which would be the third release in the
     21.1 series branch.  The major version number then gets increased
     only very occasionally, and only when a sufficiently major
     architectural change has been made, particularly one that causes
     compatibility problems with code written for previous branches.  I
     think schemes like this are unnecessary in most circumstances,
     because usually either the major version number ends up changing
     so often that the second number is always either zero or one, or
     the major version number never changes, and as such becomes
     useless.  By the time the major version number would change, the
     product itself has changed so much that it often gets renamed.
     Furthermore, it is clear that the two version number scheme has
     been used throughout most of the history of Emacs, and recently we
     have been following the two number scheme also.  If we introduced a
     third revision number, at this point it would both confuse
     existing code that assumed there were two numbers, and would look
     rather silly given that the major version number is so high and
     would probably remain at the same place for quite a long time.

  3.  A third scheme that would attempt to cross the two schemes would
     keep the same concept of major version number as for the three
     number scheme, and would compress the second and third numbers of
     the three number scheme into one number by using increments of
     ten.  For example, the current 21.x branch would have releases No.
     21.0, 21.1, etc.  The next branch would be No. 21.10, 21.11, etc.
     I don't like this scheme very much because it seems rather
     kludgey, and also because it is not used in any other product as
     far as I know.

  4.  Another scheme that would combine the second and third numbers in
     the three number scheme would be to have the releases in the
     current 21.x series be numbered 21.0, then 21.01, then 22.02, etc.
     The next series is 21.1, then 21.11, then 21.12, etc.  This is
     similar to the way that version numbers are done for DOS in
     Windows.  I also think that this scheme is fairly silly because,
     like the previous scheme, its only purpose is to avoid increasing
     the major version number very much.  But given that we have
     already have a fairly large major version number, there doesn't
     seem to be any particular problem with increasing this number by
     one every year or two.  Some people will object that by doing
     this, it becomes impossible to tell when a change is so major that
     it causes a lot of code breakage, but past releases have not been
     accurate indicators of this.  For example, 19.12 caused a lot of
     code breakage, but 20.0 caused less, and 21.0 caused less still.
     In the GNU Emacs world, there were byte code changes made between
     19.28 and 19.29, but as far as I know, not between 19.29 and 20.0.


   With three active development branches, synchronizing code changes
between the branches is obviously somewhat of a problem.  To make things
easier, I propose a few general guidelines:

  1.  Merging between different branches need not happen that often.  It
     should not happen more often than necessary to avoid undue burden
     on the maintainer, but needs to be done at all defined
     checkpoints.  These checkpoints need to be noted in all of the
     places that track changes along the branch, for example, in all of
     the change logs and in all of the CVS tags.

  2.  Every code change that can be considered a self-contained unit, no
     matter how large or small, needs to have a change log entry,
     preferably a single change log entry associated with it.  This is
     an absolute requirement.  There should be no code changes without
     an associated change log entry.  Otherwise, it is highly likely
     that patches will not be correctly synchronized across all
     versions, and will get lost.  There is no need for change log
     entries to contain unnecessary detail though, and it is important
     that there be no more change log entries than necessary, which
     means that two or more change log entries associated with a single
     patch need to be grouped together if possible.  This might imply
     that there should be one global change log instead of change logs
     in each directory, or at the very least, the number of separate
     change logs should be kept to a minimum.

  3.  The patch that is associated with each change log entry needs to
     be kept around somewhere.  The reason for this is that when
     synchronizing code from some branch to some earlier branch, it is
     necessary to go through each change log entry and decide whether a
     change is worthy to make it into a more stable branch.  If so, the
     patch associated with this change needs to be individually applied
     to the earlier branch.

  4.  All changes made in more stable branches get merged into less
     stable branches unless the change really is completely unnecessary
     in the less stable branch because it is superseded by some other
     change.  This will probably mean more developers making changes to
     the semi-stable branch than to the experimental branch.  This
     means that developers should strive to do their development in the
     most stable branch that they expect their code to go into.  An
     alternative to this which is perhaps more workable is simply to
     insist that all developers make all patches based off of the
     experimental branch, and then later merge these patches down to
     the more stable branches as necessary.  This means, however, that
     submitted patches should never be combinations of two or more
     unrelated changes.  Whenever such patches are submitted, they
     should either be rejected (which should apply to anybody who
     should know better, which probably means everybody on the beta
     list and anybody else who is a regular contributor), or the
     maintainer or some other designated party needs to filter the
     combined patch into separate patches, one per logical change.

  5.  The maintainer should keep all the patches around in some data
     base, and the patches should be given an identifier consisting of
     the author of the patch, the date the patch was submitted, and
     some other identifying characteristic, such as a number, in case
     there is more than one patch on the same date by the same author.
     The database should hopefully be correctly marked at all times
     with something indicating which branches the patch has been
     applied to, and this database should hopefully be publicly visible
     so that patch authors can determine whether their patches have
     been applied, and whether their patches have been received, so
     that patches do not get needlessly resubmitted.

  6.  Global automatable changes such as textual renaming, reordering,
     and additions or deletions of parameters in function calls should
     still be allowed, even with multiple development branches.
     (Sometimes these are necessary for code cleanliness, and in the
     long run, they save a lot of time, even through they may cause
     some headaches in the short-term.)  In general, when such changes
     are made, they should occur in a separate beta version that
     contains only such changes and no other patches, and the changes
     should be made in both the semi-stable and experimental branches
     at the same time.  The description of the beta version should make
     it very clear that the beta is comprised of such changes.  The
     reason for doing these things is to make it easier for people to
     diff between beta versions in order to figure out the changes that
     were made without the diff getting cluttered up by these code
     cleanliness changes that don't change any actual behavior.



File: internals.info,  Node: Future Work -- Improvements to the `xemacs.org' Website,  Next: Future Work -- Keybindings,  Prev: Future Work -- Version Number and Development Tree Organization,  Up: Future Work

40.14 Future Work - Improvements to the `xemacs.org' Website
============================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   The `xemacs.org' web site is the face that XEmacs presents to the
outside world.  In my opinion, its most important function is to present
information about XEmacs in such a way that solicits new XEmacs users
and co-contributors.  Existing members of the XEmacs community can
probably find out most of the information they want to know about XEmacs
regardless of what shape the web site is in, or for that matter, perhaps
even if the web site doesn't exist at all.  However, potential new users
and co-contributors who go to the XEmacs web site and find it out of
date and/or lacking the information that they need are likely to be
turned away and may never return.  For this reason, I think it's
extremely important that the web site be up-to-date, well-organized, and
full of information that an inquisitive visitor is likely to want to
know.

   The current XEmacs web site needs a lot of work if it is to meet
these standards.  I don't think it's reasonable to expect one person to
do all of this work and make continual updates as needed, especially
given the dismal record that the XEmacs web site has had.  The proper
thing to do is to place the web site itself under CVS and allow many of
the core members to remotely check files in and out.  This way, for
example, Steve could update the part of the site that contains the
current release status of XEmacs. (Much of this could be done by a
script that Steve executes when he sends out a beta release
announcement which automatically HTML-izes the mail message and puts it
in the appropriate place on the web site.  There are programs that are
specifically designed to convert email messages into HTML, for example
`mhonarc'.)  Meanwhile, the `xemacs.org' mailing list administrator
(currently Jason Mastaler, I think) could maintain the part of the site
that describes the various mailing lists and other addresses at
`xemacs.org'.  Someone like me (perhaps through a proxy typist) could
maintain the part of the site that specifies the future directions that
XEmacs is going in, etc., etc.

   Here are some things that I think it's very important to add to the
web site.

  1.  A page describing in detail how to get involved in the XEmacs
     development process, how to submit and where to submit various
     patches to the XEmacs core or associated packages, how to contact
     the maintainers and core developers of XEmacs and the maintainers
     of various packages, etc.

  2.  A page describing exactly how to download, compile, and install
     XEmacs, and how to download and install the various binary
     distributions.  This page should particularly cover in detail how
     exactly the package system works from an installation standpoint
     and how to correctly compile and install under Microsoft Windows
     and Cygwin.  This latter section should cover what compilers are
     needed under Microsoft Windows and Cygwin, and how to get and
     install the Cygwin components that are needed.

  3.  A page describing where to get the various ancillary libraries
     that can be linked with XEmacs, such as the JPEG, TIFF, PNG,
     X-Face, DBM, and other libraries.  This page should also cover how
     to correctly compile it and install these libraries, including
     under Microsoft Windows (or at least it should contain pointers to
     where this information can be found).  Also, it should describe
     anything that needs to be specified as an option to `configure' in
     order for XEmacs to link with and make use of these libraries or
     of Motif or CDE.  Finally, this page should list which versions of
     the various libraries are required for use with the various
     different beta versions of XEmacs.  (Remember, this can change
     from beta to beta, and someone needs to keep a watchful eye on
     this).

  4.  Pointers to any other sites containing information on XEmacs.
     This would include, for example, Hrvoje's XEmacs on Windows FAQ
     and my Architecting XEmacs web site.  (Presumably, most of the
     information in this section will be temporary.  Eventually, these
     pages should be integrated into the main XEmacs web site).

  5.  A page listing the various sub-projects in the XEmacs development
     process and who is responsible for each of these sub-projects, for
     example development of the package system, administration of the
     mailing lists, maintenance of stable XEmacs versions, maintenance
     of the CVS web interface, etc.  This page should also list all of
     the packages that are archived at `xemacs.org' and who is the
     maintainer or maintainers for each of these packages.


Other Places with an XEmacs Presence
------------------------------------

We should try to keep an XEmacs presence in all of the major places on
the web that are devoted to free software or to the "open source"
community.  This includes, for example, the open source web site at
http://opensource.oreilly.com (../../opensource.oreilly.com/default.htm)
(I'm already in the process of contacting this site), the Freshmeat site
at http://www.freshmeat.net (../../www.freshmeat.net/default.htm), the
various announcement news groups (for example, comp.os.linux.announce
(news:comp.os.linux.announce), and the Windows announcement news group)
etc.


File: internals.info,  Node: Future Work -- Keybindings,  Next: Future Work -- Byte Code Snippets,  Prev: Future Work -- Improvements to the `xemacs.org' Website,  Up: Future Work

40.15 Future Work - Keybindings
===============================

* Menu:

* Future Work -- Keybinding Schemes::
* Future Work -- Better Support for Windows Style Key Bindings::
* Future Work -- Misc Key Binding Ideas::


File: internals.info,  Node: Future Work -- Keybinding Schemes,  Next: Future Work -- Better Support for Windows Style Key Bindings,  Prev: Future Work -- Keybindings,  Up: Future Work -- Keybindings

40.15.1 Future Work - Keybinding Schemes
----------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* We need a standard mechanism that allows a different
global key binding schemes to be defined.  Ideally, this would be the
keyboard action interface (keyboard-actions.html) that I have proposed,
however this would require a lot of work on the part of mode
maintainers and other external Elisp packages and will not be rady in
the short term.  So I propose a very kludgy interface, along the lines
of what is done in Viper currently.  Perhaps we can rip that key munging
code out of Viper and make a separate extension that implements a global
key binding scheme munging feature.  This way a key binding scheme could
rearrange all the default keys and have all sorts of other code, which
depends on the standard keys being in their default location, still
work.


File: internals.info,  Node: Future Work -- Better Support for Windows Style Key Bindings,  Next: Future Work -- Misc Key Binding Ideas,  Prev: Future Work -- Keybinding Schemes,  Up: Future Work -- Keybindings

40.15.2 Future Work - Better Support for Windows Style Key Bindings
-------------------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* This page describes how we could create an XEmacs
extension that modifies the global key bindings so that a Windows user
would feel at home when using the keyboard in XEmacs.  Some of these
bindings don't conflict with standard XEmacs keybindings and should be
added by default, or at the very least under Windows, and probably under
X Windows as well. Other key bindings would need to be implemented in a
Windows compatibility extension which can be enabled and disabled on the
fly, following the conventions outlined in Standard interface for
enabling extensions (enabling-extensions.html) Ideally, this should be
implemented using the keyboard action interface (keyboard-actions.html)
but these wil not be available in the short term, so we will have to
resort to some awful kludges, following the model of Michael Kifer's
Viper mode.

   We really need to make XEmacs provide standard Windows key bindings
as much as possible.  Currently, for example, there are at least two
packages that allow the user to make a selection using the shifted arrow
keys, and neither package works all that well, or is maintained.  There
should be one well-written piece of code that does this, and it should
be a standard part of XEmacs.  In fact, it should be turned on by
default under Windows, and probably under X as well. (As an aside here,
one point of contention in how to implement this involves what happens
if you select a region using the shifted arrow keys and then hit the
regular arrow keys.  Does the region remain selected or not?  I think
there should be a variable that controls which of these two behaviors
you want.  We can argue over what the default value of this variable
should be.  The standard Windows behavior here is to keep the region
selected, but move the insertion point elsewhere, which is unfortunately
impossible to implement in XEmacs.)

   Some thought should be given to what to do about the standard Windows
control and alt key bindings.  Under NTEmacs, there is a variable that
controls whether the alt key behaves like the Emacs meta key, or whether
it is passed on to the menu as in standard Windows programs.  We should
surely implement this and put this option on the *Options* menu.
Making `Alt-f' for example, invoke the *File* menu, is not all that
disruptive in XEmacs, because the user can always type `ESC f' to get
the meta key functionality.  Making `Control-x', for example, do *Cut*,
is much, much more problematic, of course, but we should consider how
to implement this anyway.  One possibility would be to move all of the
current Emacs control key bindings onto control-shift plus a key, and
to make the simple control keys follow the Windows standard as much as
possible.  This would mean, for example, that we would have the
following keybindings:
`Control-x' ==> *Cut*
`Control-c' ==> *Copy*
`Control-v' ==> *Paste*
`Control-z' ==> *Undo*
`Control-f' ==> *Find*
`Control-a' ==> *Select All*
`Control-s' ==> *Save*
`Control-p' ==> *Print*
`Control-y' ==> *Redo*
(this functionality _is_ available in XEmacs with Kyle Jones' `redo.el'
package, but it should be better integrated)
`Control-n' ==> *New*
`Control-o' ==> *Open*
`Control-w' ==> *Close Window*
The changes described in the previous paragraph should be put into an
extension named `windows-keys.el' (see Standard interface for enabling
extensions (enabling-extensions.html)) so that it can be enabled and
disabled on the fly using a menu item and can be selected as the
default for a particular user in their custom options file. Once this
is implemented, the Windows installer should also be modified so that
it brings up a dialog box that allows the user to make a selection of
which key binding scheme they would prefer as the default, either the
XEmacs standard bindings, Vi bindings (which would be Viper mode),
Windows-style bindings, Brief, CodeWright, Visual C++, or whatever we
manage to implement.


File: internals.info,  Node: Future Work -- Misc Key Binding Ideas,  Prev: Future Work -- Better Support for Windows Style Key Bindings,  Up: Future Work -- Keybindings

40.15.3 Future Work - Misc Key Binding Ideas
--------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   * M-123 ... do digit arg

   * However, M-( group commands together until M-)

   * Nested M-() are allowed.

   * Number repeating plus () repeats N times each group of commands as
     a unit.

   * M-() by itself forms an anonymous macro, and there should be a
     command to repeat, like VI (execute macro), but when no () before,
     it repeats the last command of same amount of complication - or
     more like, somewhere there is a repeats all command back to make
     to act that stopping like VI's dot command.

   * C-numbers switches to a particular window.  maybe 1-3 or 1-4 does
     this.

   * C-4 or 5 to 9 (or ()? maybe reserved) switches to a particular
     frame.

   * Possibly C-Sh-numbers select more windows or frames.

   * M-C-1 M-C-2 M-C-3 M-C-4 M-C-5 M-C-6 M-C-7 M-C-8 M-C-9 M-C-0

     maybe should be execute anonymous macros (other possibility is
     insert register but you can easily simulate with a keyboard macro)

   * What about C-S M-C-S M-S??

   * I think there should be default fun key binding for *ILLEGIBLE*
     similar to what I have - load, save, cut, copy, paste, kill line,
     start/end macro, do macro


File: internals.info,  Node: Future Work -- Byte Code Snippets,  Next: Future Work -- Lisp Stream API,  Prev: Future Work -- Keybindings,  Up: Future Work

40.16 Future Work - Byte Code Snippets
======================================

Author: Ben Wing (mailto:ben@xemacs.org)

   * For use in time critical (e.g. redisplay) places such as display
     tables - a simple piece of code is evalled, e.g.
          (int-to-char (1+ c))
     where c is the arg, specbound.

   * can be compiled like
          (byte-compile-snippet (int-to-char (1+ c)) (c))
                                                     ^^^
                                          environment of local vars

   * need eval with bindings (not hard to implement) (extendable when
     lexical scoping present)

   * What's the return value of byte-compile-snippet?  (Look to see how
     this might be implemented)

* Menu:

* Future Work -- Autodetection::
* Future Work -- Conversion Error Detection::
* Future Work -- Unicode::
* Future Work -- BIDI Support::
* Future Work -- Localized Text/Messages::


File: internals.info,  Node: Future Work -- Autodetection,  Next: Future Work -- Conversion Error Detection,  Prev: Future Work -- Byte Code Snippets,  Up: Future Work -- Byte Code Snippets

40.16.1 Future Work - Autodetection
-----------------------------------

There are various proposals contained here.

New Implementation of Autodetection Mechanism
---------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   The current auto detection mechanism in XEmacs Mule has many
problems. For one thing, it is wrong too much of the time. Another
problem, although easily fixed, is that priority lists are fixed rather
than varying, depending on the particular locale; and finally, it
doesn't warn the user when it's not sure of the encoding or when there's
a mistake made during decoding. In both of these situations the user
should be presented with a list of likely encodings and given the
choice, rather than simply proceeding anyway and giving a result that is
likely to be wrong and may result in data corruption when the file is
saved out again.

   All coding systems are categorized according to their type. Currently
this includes ISO2022, Big 5, Shift-JIS, UTF8 and a few others. In the
future there will be many more types defined and this mechanism will be
generalized so that it is easily extendable by the Lisp programmer.

   In general, each coding system type defines a series of subtypes
which are handled differently for the purpose of detection. For
example, ISO 2022 defines many different subtypes such as 7 bit, 8 bit,
locking shift, designating and so on. UCS2 may define subtypes such as
normal and byte reversed.

   The detection engine works conceptually by calling the detection
methods of all of the defined coding system types in parallel on
successive chunks of data (which may, for example, be 4K in size, but
where the size makes no difference except for optimization purposes)
and watching the results until either a definite answer is determined
or the end of data is reached. The way the definite answer is
determined will be defined below. The detection method of the coding
system type is passed some data and a chunk of memory, which the method
uses to store its current state (and which is maintained separately for
each coding system type by the detection engine between successive
calls to the coding system type's detection method). Its return value
should be an alist consisting of a list of all of the defined subtypes
for that coding system type along with a level of likelihood and a list
of additional properties indicating certain features detected in the
data. The extra properties returned are defined entirely by the
particular coding system type and are used only in the algorithm
described below under "user control." However, the levels of likelihood
have a standard meaning as follows:

   Level 4 means "near certainty" and typically indicates that a
signature has been detected, usually at the beginning of the data,
indicating that the data is encoded in this particular coding system
type. An example of this would be the byte order mark at the beginning
of UCS2 encoded data or the GZIP mark at the beginning of GZIP data.

   Level 3 means "highly likely" and indicates that tell-tale signs have
been discovered in the data that are characteristic of this particular
coding system type. Examples of this might be ISO 2022 escape sequences
or the current Unicode end of line markers at regular intervals.

   Level 2 means "strongly statistically likely" indicating that
statistical analysis concludes that there's a high chance that this
data is encoded according to this particular type. For example, this
might mean that for UCS2 data, there is a high proportion of null bytes
or other repeated bytes in the odd-numbered bytes of the data and a
high variance in the even-numbered bytes of the data. For Shift-JIS,
this might indicate that there were no illegal Shift-JIS sequences and
a fairly high occurrence of common Shift-JIS characters.

   Level 1 means "weak statistical likelihood" meaning that there is
some indication that the data is encoded in this coding system type. In
fact, there is a reasonable chance that it may be some other type as
well. This means, for example, that no illegal sequences were
encountered and at least some data was encountered that is purposely
not in other coding system types. For Shift-JIS data, this might mean
that some bytes in the range 128 to 159 were encountered in the data.

   Level 0 means "neutral" which is to say that there's either not
enough data to make any decision or that the data could well be
interpreted as this type (meaning no illegal sequences), but there is
little or no indication of anything particular to this particular type.

   Level -1 means "weakly unlikely" meaning that some data was
encountered that could conceivably be part of the coding system type
but is probably not. For example, successively long line-lengths or
very rarely-encountered sequences.

   Level -2 means "strongly unlikely" meaning that typically a number
of illegal sequences were encountered.

   The algorithm to determine when to stop and indicate that the data
has been detected as a particular coding system uses a priority list,
which is typically specified as part of the language environment
determined from the current locale or the user's choice. This priority
list consists of a list of coding system subtypes, along with a minimum
level required for positive detection and optionally additional
properties that need to be present. Using the return values from all of
the detection methods called, the detection engine looks through this
priority list until it finds a positive match. In this priority list,
along with each subtype is a particular coding system to return when
the subtype is encountered. (For example, in a Japanese-language
environment particular subtypes of ISO 2022 will be associated with the
Japanese coding system version of those subtypes). It is perfectly
legal and quite common in fact, to list the same subtype more than once
in the priority list with successively lower requirements. Other facts
that can be listed in the priority list for a subtype are "reject",
meaning that the data should never be detected as this subtype, or
"ask", meaning that if the data is detected to be this subtype, the
user will be asked whether they actually mean this. This latter
property could be used, for example, towards the bottom of the priority
list.

   In addition there is a global variable which specifies the minimum
number of characters required before any positive match is reported.
There may actually be more than one such variable for different sources
of data, for example, detection of files versus detection of subprocess
data.

   Whenever a file is opened and detected to be a particular coding
system, the subtype, the coding system and the associated level of
likelihood will be prominently displayed either in the echo area or in
a status box somewhere.

   If no positive match is found according to the priority list, or if
the matches that are found have the "ask" property on them, then the
user will be presented with a list of choices of possible encodings and
asked to choose one. This list is typically sorted first by level of
likelihood, and then within this, by the order in which the subtypes
appear in the priority list. This list is displayed in a special kind
of dialog box or other buffer allowing the user, in addition to just
choosing a particular encoding, to view what the file would look like
if it were decoded according to the type.

   Furthermore, whenever a file is decoded according to a particular
type, the decoding engine keeps track of status values that are output
by the coding system type's decoding method. Generally, this status
will be in the form of errors or warnings of various levels, some of
which may be severe enough to stop the decoding entirely, and some of
which may either indicate definitely malformed data but from which it's
possible to recover, or simply data that appears rather questionable.
If any of these status values are reported during decoding, the user
will be informed of this and asked "are you sure?"  As part of the "are
you sure" dialog box or question, the user can display the results of
the decoding to make sure it's correct. If the user says "no, they're
not sure," then the same list of choices as previously mentioned will
be presented.

RFC: Autodetection
------------------

Also appeared under heading "Implementation of Coding System Priority
Lists in Various Locales" ?

   Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   Date: 11/1/1999 2:48 AM

     >>>>> "Hrvoje" == Hrvoje Niksic <hniksic@srce.hr> writes:

         [Ben sez:]

         >> You are perfectly free to set up your XEmacs like this, but
         >> XEmacs/Mule *will* autodetect by default if there is no
         >> Content-Type: info and no reason to believe we are dealing with
         >> binary files.

         Hrvoje> In that case, it will be a serious mistake to make
         Hrvoje> --with-mule the default, ever.  I think more care should
         Hrvoje> be shown in meeting the need of European users.

   Hrvoje, I don't understand what you are worrying about.  I suspect
you are worrying about Handa's hyperactive and obstinate Mule, not what
Ben has in mind.  Yes, Ben has said "better guessing," but that's
simply not reasonable without substantial language environment
information.  I think trying to detect Latin-1 vs Latin-2 in the POSIX
locale would be a big mistake, I think trying to guess Big 5 v. Shift
JIS in a European locale would be a big mistake.

   If Ben doesn't mean "more appropriate use of language environment
information" when he writes "better guessing," I, as much as you, want
to see how he plans to do that.  Ben?  ("Yes/no/oops I need to think
about it" is good enough if you have specifics you intend to put in the
RFC you're planning to present.)

   Let me give a formal proposal of what I would like to see in the
autodetection specification.

  1. Definitions

       1. "Autodetection" means detecting and making available to Mule
          the external file's encoding.  See (5), below.  It doesn't
          imply any specific actions based on that information.

       2. The "default" case is POSIX locale, and no environment
          information in ~/.emacs.

          N.B.  This *will* cause breakage for all 1-byte users because
          the default case can no longer assume Latin-1.  You *may* be
          able to use the TTY font or the Xt -font option to fake this,
          and default to iso8859-1; I would hope that we would not use
          such a kludge in the beta versions, although it might be
          satisfactory for general use.  In particular, encodings like
          VISCII (Vietnamese) and I believe KOI-8 (Cyrillic) are not
          ISO-2022-clean, but using C1 control characters as a heuristic
          for detecting binary files is useful.

          If we do allow it, I think that XEmacs should bitch and warn
          that the practices of implicitly specifying language
          environment by -font and defaulting on TTYs is deprecated and
          likely to be obsoleted.

       3. The "European" case is any Latin-* locale, either implied by
          setlocale() and friends or set in ~/.emacs.  Latin-1 is
          specifically not given precedence over other Latin-*, or
          non-Latin or non-ISO-8859 for that matter.  I suspect but am
          not sure that this case extends to all ISO-8859 encodings, and
          possibly to non-ISO-8859 single-byte encodings like KOI-8r (in
          particular when combined in a class with ISO-8859 encodings).

       4. The "CJK" case is any CJK locale.  Japanese is specifically
          not given precedence over other Asian locales.

       5. For completeness, define the "Unicode" case (Unicode
          unfortunately has lots of junk such as precomposed characters,
          language tags, and directionality indicators in it; we
          probably don't care yet, but we should also not claim
          compliance) and the "general" case (which has a lot of
          features similar to Unicode, but lacks the advantage of a
          unified encoding).  This proposal has no idea how to handle
          the special features of these, or even if that matters.  The
          general case includes stuff that nobody here really knows how
          it works, like Tibetan and Ethiopic.

     Each of the following cases is given in the order of priority of
     detection.  I'm not sure I'm serious about the top priority given
     the (optional) Unicode detection.  This may be appropriate if Ben
     is right that ISO-2022 is going to disappear, but possibly not
     until then (two two-byte sequences out of 65536 is probably 1.99
     too many).  It probably isn't too risky if (6)(c) is taken pretty
     seriously; a Unicode file should contain _no_ private use
     characters unless the encoding is explicitly specified, and that's
     a block of 1/10 of the code space, which should help a lot in
     detecting binary files.

  2. Default locale

       1. Some Unicode (fixed width; maybe UTF-8, too?) may optionally
          be detected by the byte-order-mark magic (if the first two
          bytes are 0xFE 0xFF, the file is Unicode text, if 0xFF 0xFE,
          it is wrong-endian Unicode; if legal in UTF-8, it would be
          0xFE 0xBB 0xBF, either-endian).  This is probably an
          optimization that should not be on by default yet.

       2. ISO-2022 encodings will be detected as long as they use
          explicit designation of all non-ASCII character sets.  This
          means that many 7-bit ISO-2022 encodings would be detected
          (eg, ISO-2022-JP), but EUC-JP and X Compound Text would not,
          because they implicitly designate character sets.

          N.B. Latin-1 will be detected as binary, as for any Latin-*.

          N.B. An explicit ISO-2022 designation is semantically
          equivalent to a Content-Type: header.  It is more dangerous
          because shorter, but I think we should recognize them by
          default despite the slight risk; XEmacs is a text editor.

          N.B. This is unlikely to be as dangerous as it looks at first
          glance.  Any file that includes an 8-bit-set byte before the
          first valid designation should be detected as binary.

       3. Binary files will be detected (eg, presence of NULs, other
          non-whitespace control characters, absurdly long lines, and
          presence of bytes >127).

       4. Everything else is ASCII.

       5. Newlines will be detected in text files.

  3. European locales

       1. Unicode may optionally be detected by the byte-order-mark
          magic.

       2. ISO-2022 encodings will be detected as long as they use
          explicit designation of all non-ASCII character sets.

       3. A locale-specific class of 1-byte character sets (eg,
          '(Latin-1)) will be detected.

          N.B.  The reason for permitting a class is for cases like
          Cyrillic where there are both ISO-8859 encodings and
          incompatible encodings (KOI-8r) in common use.  If you want to
          write a Latin-1 v. Latin-2 detector, be my guest, but I don't
          think it would be easy or accurate.

       4. Binary files will be detected per (2)(c), except that only
          8-bit bytes out of the encoding's range imply binary.

       5. Everything else is ASCII.

       6. Newlines will be detected in text files.

  4. CJK locales

       1. Unicode may optionally be detected by the byte-order-mark
          magic.

       2. ISO-2022 encodings will be detected as long as they use
          explicit designation of all non-ASCII character sets.

       3. A locale-specific class of multi-byte and wide-character
          encodings will be detected.  N.B. No 1-byte character sets
          (eg, Latin-1) will be detected.  The reason for a class is to
          allow the Japanese to let Mule do the work of choosing EUC v.
          SJIS.

       4. Binary files will be detected per (3)(d).

       5. Everything else is ASCII.

       6. Newlines will be detected in text files.

  5. Unicode and general locales; multilingual use

       1. Hopefully a system general enough to handle (2)-(4) will
          handle these, too, but we should watch out for gotchas like
          Unicode "plane 14" tags which (I think _both_ Ben and Olivier
          will agree) have no place in the internal representation, and
          thus must be treated as out-of-band control sequences.  I
          don't know if all such gotchas will be as easy to dispose of.

       2. An explicit coding system priority list will be provided to
          allow multilingual users to autodetect both Shift JIS and Big
          5, say, but this ability is not promised by Mule, since it
          would involve (eg) heuristics like picking a set of code
          points that are frequent in Shift JIS and uncommon in Big 5
          and betting that a file containing many characters from that
          set is Shift JIS.

  6. Relationship to decoding semantics

       1. Autodetection should be run on every input stream unless the
          user explicitly disables it.

       2. The (conceptual) default procedure is

       3. Read the file into the buffer

          Announce the result of autodetection to the user.

          User may request decoding, with autodetected encoding(s)
          given priority in a list of available encodings.

          zations (see (e) below) should avoid introducing data tion
          that this default procedure would avoid.

          sly, it can't be perfect if any autodecoding is done; like
          Hrvoje should have an easily available option to  to this
          default (or an optimized approximation which t actually read
          the whole file into a buffer) or simply y everything as
          binary (with the "font" for binary files a user option).

       4. This implies that we should be detecting conditions in the
          tail of the file which violate the implicit assumptions of the
          coding system autodetected (eg, in UTF-8 illegal UTF-8
          sequences, including those corresponding to surrogates) should
          raise a warning; the buffer should probably be made read-only
          and the user prompted.

          This could be taken to extremes, like checking by table
          whether all characters in a Japanese file are actually
          legitimate JIS codes; that's insane (and would cause corporate
          encodings to be recognized as binary).  But we should think
          about the idea that autodetection shouldn't mean XEmacs can't
          change its mind.

       5. A flexible means for the user to delegate the decision
          (conditional on the result of autodetection) to decode or not
          to XEmacs or a Lisp program should be provided (eg, the
          coding priority list and/or a file-coding-alist).

       6. Optimized operations (eg, the current lstreams) should be
          provided, with the recognition that if they depend on sampling
          the file they are risky.

       7. Mule should provide a reasonable set of default delegations
          (as in (d) above) for as many locales as possible.

  7. Implementation

       1. I think all the decision logic suggested above can be
          accomplished through a coding-priority-list and appropriate
          initializations for different language environments, and a
          file-coding-alist.

       2. Many of the tests on the file's tail shouldn't be very
          expensive; in particular, all of the ones I've suggested are
          O(n) although they might involve moderate-sized auxiliary
          tables for efficiency (eg, 64kB for a single Unicode-oriented
          test).

   Other comments:

   It might be reasonable given Hrvoje's objections to require that any
autodetection that could cause data loss (any coding system that
involves escape sequences, and only those AFAIK: by design translation
to Unicode is invertible) by default prompt the user (presumable with a
novice-like ability to retain the prompt, always default to binary, or
always default to the autodetected encoding) in the future, at least in
locales that don't need it (POSIX, Latin-any).

   Ben thinks that we can remember the input data; I think it's going to
be hard to comprehensively test that a highly optimized version works.
Good design will help, but ISO-2022 is enormously complex, and there
are many encodings that violate even its lax assumptions.  On the other
hand, memory is the only way to get non-rewindable streams right.

   Hrvoje himself said he would like to have an XEmacs that
distinguishes between Latin-1 and Latin-2 text.  Where it is possible
to do that, this is exactly what autodetection of ISO-2022 and Unicode
gives you.  Many people would want that, even at some risk of binary
corruption.

   >> Once again I remind you that XEmacs is a *text* editor.  There
>> are lots of files that potentially may have Japanese etc. in     >>
them without this marked, e.g. C or Elisp files in the XEmacs     >>
source.  Surely you're not arguing that we interpret even these     >>
files as binary by default?

   Hrvoje> I am.  If I want to see Japanese, I'll setup my     Hrvoje>
environment that way.  But I don't, and neither do 99% of     Hrvoje>
Croatian users.  I can't speak for French, Italian, and     Hrvoje>
others, but I'd assume similar.

   Hrvoje> If there is Japanese in the source files, I will see it as
 Hrvoje> escape sequences, which is perfectly fine, because I don't
Hrvoje> read Japanese.

   And some (European) people will have their terminals scrambled,
because Shift-JIS contains sequences that can change the state of XTerm
(as do fixed-width Unicode and Big5).  This may also be a problem with
some Windows-12xx encodings; I'm not sure they all are ISO-2022-clean.
(This isn't a problem for XEmacs native X11 frames or native MS-Windows
frames, and the XEmacs sources themselves are all in 7-bit ISO-2022 now
IIRC.  But it is a potential source of great frustration for many
users.)

   I think that should be considered too, although it is presumably
lower priority than the data corruption of binary files.

Response to RFC: Autodetection
------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Date: 11/1/1999 7:24 AM

   Stephen, thank you very much for writing this up.  I think it is a
good start, and definitely moving in the direction I would like to see
things going: more proposals, less arguing. (aka "more light, less
heat") However, I have some suggestions for cleaning this up:

   You should try to make it more layered.  For example, you might have
one section devoted to the workings of autodetection, which starts out
like this (the section numbers below are totally arbitrary):

Section 5
.........

`Autodetect()' is a function whose arguments are (1) a readable stream,
(2) some hints indicating how the autodetection is to proceed, and (3)
a value indicating the maximum number of characters to examine at the
beginning of the stream.  (Possibly, the value in (3) may be some
special symbol indicating that we only go as far as the next line, or a
certain number of lines ahead; this would be used as part of
"continuous autodetection", e.g. we are decoding the results of an
interactive terminal session, where the user may periodically switch
encodings, line terminations, etc. as different programs get run and/or
telnet or similar sessions are entered into and exited.) We assume the
stream is rewindable; if not, insert a "rewinding" stream in front of
the non-rewinding stream; this kind of stream automatically buffers the
data as necessary.  [You can use pseudo-code terminology here.  No need
for straight C or ELisp.]  [Then proceed to describe what the hints
look like - e.g. you could portray it as a property list or whatever.
The idea is that, for each locale, there is a corresponding hints value
that is used at least by default.  The hints structure also has to be
set up to allow for two or more competing hints specifications to be
merged together.  For example, the extension of a file might provide an
additional hint or hints about how to interpret the data of that file,
and the caller of `autodetect()', when calling `autodetect()' on such a
file, would need to have a way of gracefully merging the default hints
corresponding to the locale with the more specific hints provided by the
extension.  Furthermore, users like Hrvoje might well want to provide
their own hints to supplement and override parts of the generic hints -
e.g. "I don't ever want to see non-European encodings decoded; treat
them as binary instead".]  [Then describe algorithmically how the
autodetection works.  First, you could describe it more generally, i.e.
presenting an algorithmic overview, then you could discuss in detail
exactly how autodetection of a particular type of external encoding
works - e.g. "for iso2022, we first look for an escape character,
followed by a byte in this range [. ... .] etc."]

Section 6
.........

This section describes the concept of a locale in XEmacs, and how it is
derived from the user's environment.  A locale in XEmacs is a pair, a
country and a language, together determining the handling of
locale-specific areas of XEmacs.  All locale-specific areas in XEmacs
make use of this XEmacs locale, and do not attempt to derive the locale
from any other sources.  The user is free to change the current locale
at any time; accessor and mutator functions are provided to do this so
that various locale-specific areas can optionally be changed together
with it.

   [Then you describe how the XEmacs locale is extracted from .emacs,
from `setlocale()', from the LANG environment variables, from -font, or
wherever else.  All other sections assume this dirty work is done and
never even mention it]

Section 7
.........

[Here you describe the default `autodetect()' hints value corresponding
to each possible locale.  You should probably use a schematic
description here, e.g.  an actual Lisp property list, liberally
commented.]

Section 8 etc.
..............

[Other sections cover anything I've missed.  By being very careful to
separate out the layers, you simultaneously introduce more rigor
(easier to catch bugs) and make it easier for someone else to
understand it completely.]

Better Algorithm, More Flexibility, Different Levels of Certainty
-----------------------------------------------------------------

Much More Flexible Coding System Priority List, per-Language Environment
------------------------------------------------------------------------

User Ability to Select Encoding when System Unsure or Encounters Errors
-----------------------------------------------------------------------

Another Autodetection Proposal
------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   however, in general the detection code has major problems and needs
lots of work:

   * instead of merely "yes" or "no" for particular categories, we need
     a more flexible system, with various levels of likelihood.
     Currently I've created a system with six levels, as follows:

     [see `file-coding.h']

     Let's consider what this might mean for an ASCII text detector.
     (In order to have accurate detection, especially given the
     iteration I proposed below, we need active detectors for *all*
     types of data we might reasonably encounter, such as ASCII text
     files, binary files, and possibly other sorts of ASCII files, and
     not assume that simply "falling back to no detection" will work at
     all well.)

     An ASCII text detector DOES NOT report ASCII text as level 0, since
     that's what the detector is looking for.  Such a detector ideally
     wants all bytes in the range 0x20 - 0x7E (no high bytes!), except
     for whitespace control chars and perhaps a few others; LF, CR, or
     CRLF sequences at regular intervals (where "regular" might mean an
     average < 100 chars and 99% < 300 for code and other stuff of the
     "text file w/line breaks" variety, but for the "text file w/o line
     breaks" variety, excluding blank lines, averages could easily be
     600 or more with 2000-3000 char "lines" not so uncommon); similar
     statistical variance between odds and evens (not Unicode);
     frequent occurrences of the space character; letters more common
     than non-letters; etc.  Also checking for too little variability
     between frequencies of characters and for exclusion of particular
     characters based on character ranges can catch ASCII encodings
     like base-64, UUEncode, UTF-7, etc.  Granted, this doesn't even
     apply to everything called "ASCII", and we could potentially
     distinguish off ASCII for code, ASCII for text, etc. as separate
     categories.  However, it does give us a lot to work off of, in
     deciding what likelihood to choose - and it shows there's in fact
     a lot of detectable patterns to look for even in something
     seemingly so generic as ASCII.  The detector would report most text
     files in level 1 or level 2.  EUC encodings, Shift-JIS, etc.
     probably go to level -1 because they also pass the EOL test and
     all other tests for the ASCII part of the text, but have lots of
     high bytes, which in essence turn them into binary.  Aberrant text
     files like something in BASE64 encoding might get placed in level
     0, because they pass most tests but fail dramatically the
     frequency test; but they should not be reported as any lower,
     because that would cause explicit prompting, and the user should
     be able any valid text file without prompting.  The escape
     sequences and the base-64-type checks might send 7-bit iso2022 to
     0, but probably not -1, for similar reasons.

   * The assumed algorithm for the above detection levels is to in
     essence sort categories first by detection level and then by
     priority.  Perhaps, however, we would want smarter algorithms, or
     at least something user-controllable - in particular, when (other
     than no category at level 0 or greater) do we prompt the user to
     pick a category?

   * Improvements in how the detection algorithm works: we want to
     handle lots of different ways something could be encoded,
     including multiple stacked encodings.  trying to specify a series
     of detection levels (check for base64 first, then check for gzip,
     then check for an i18n decoding, then for crlf) won't generally
     work.  for example, what about the same encoding appearing more
     than once? for example, take euc-jp, base64'd, then gzip'd, then
     base64'd again: this could well happen, and you could specify the
     encodings specifically as base64|gzip|base64|euc-jp, but we'd like
     to autodetect it without worrying about exactly what order these
     things appear in.  we should allow for iterating over
     detection/decoding cycles until we reach some maximum (we got
     stuck in a loop, due to incorrect category tables or detection
     algorithms), have no reported detection levels over -1, or we end
     up with no change after a decoding pass (i.e. the coding system
     associated with a chosen category was `no-conversion' or something
     equivalent).  it might make sense to divide things into two phases
     (internal and external), where the internal phase has a separate
     category list and would probably mostly end up handling EOL
     detection; but the i think about it, the more i disagree.  with
     properly written detectors, and properly organized tables (in
     general, those decodings that are more "distinctive" and thus
     detectable with greater certainty go lower on the list), we
     shouldn't need two phases.  for example, let's say the example
     above was also in CRLF format.  The EOL detector (which really
     detects *plain text* with a particular EOL type) would return at
     most level 0 for all results until the text file is reached,
     whereas the base64, gzip or euc-jp decoders will return higher.
     Once the text file is reached, the EOL detector will return 0 or
     higher for the CRLF encoding, and all other detectors will return
     0 or lower; thus, we will successfully proceed through CRLF
     decoding, or at worst prompt the user. (The only
     external-vs-internal distinction that might make sense here is to
     favor coding systems of the correct source type over those that
     require conversion between external and internal; if done right,
     this could allow the CRLF detector to return level 1 for all
     CRLF-encoded text files, even those that look like Base-64 or
     similar encoding, so that CRLF encoding will always get decoded
     without prompting, but not interfere with other decoders.  On the
     other hand, this external-vs-internal distinction may not matter
     at all - with automatic internal-external conversion, CRLF
     decoding can occur before or after decoding of euc-jp, base64,
     iso2022, or similar, without any difference in the final results.)

     #### What are we trying to say?  In base64, the CRLF decoding
     before base64 decoding is irrelevant, they will be thrown out as
     whitespace is not significant in base64.

     [sjt considers all of this to be rather bogus.  Ideas like "greater
     certainty" and "distinctive" can and should be quantified.  The
     issue of proper table organization should be a question of
     optimization.]

     [sjt wonders if it might not be a good idea to use Unicode's
     newline character as the internal representation so that (for
     non-Unicode coding systems) we can catch EOL bugs on Unix too.]

   * There need to be two priority lists and two
     category->coding-system lists.  Once is general, the other
     category->langenv-specific.  The user sets the former, the langenv
     category->the latter.  The langenv-specific entries take precedence
     category->over the others.  This works similarly to the
     category->category->Unicode charset priority list.

   * The simple list of coding categories per detectors is not enough.
     Instead of coding categories, we need parameters.  For example,
     Unicode might have separate detectors for UTF-8, UTF-7, UTF-16,
     and perhaps UCS-4; or UTF-16/UCS-4 would be one detection type.
     UTF-16 would have parameters such as "little-endian" and "needs
     BOM", and possibly another one like "collapse/expand/leave alone
     composite sequences" once we add this support.  Usually these
     parameters correspond directly to a coding system parameter.
     Different likelihood values can be specified for each parameter as
     well as for the detection type as a whole.  The user can specify
     particular coding systems for a particular combination of
     detection type and parameters, or can give "default parameters"
     associated with a detection type.  In the latter case, we create a
     new coding system as necessary that corresponds to the detected
     type and parameters.

   * a better means of presentation.  rather than just coming up with
     the new file decoded according to the detected coding system,
     allow the user to browse through the file and conveniently reject
     it if it looks wrong; then detection starts again, but with that
     possibility removed.  in cases where certainty is low and thus
     more than one possibility is presented, the user can browse each
     one and select one or reject them all.

   * fail-safe: even after the user has made a choice, if they later on
     realize they have the wrong coding system, they can go back, and
     we've squirreled away the original data so they can start the
     process over.  this may be tricky.

   * using a larger buffer for detection.  we use just a small piece,
     which can give quite random results.  we may need to buffer up all
     the data we look through because we can't necessarily rewind.  the
     idea is we proceed until we get a result that's at least at a
     certain level of certainty (e.g. "probable") or we reached a
     maximum limit of how much we want to buffer.

   * dealing with interactive systems.  we might need to go ahead and
     present the data before we've finished detection, and then
     re-decode it, perhaps multiple times, as we get better detection
     results.

   * Clearly some of these are more important than others.  at the very
     least, the "better means of presentation" should be implemented as
     soon as possible, along with a very simple means of fail-safe
     whenever the data is readibly available, e.g. it's coming from a
     file, which is the most common scenario.

   ben [at least that's what sjt thinks]

   *****

   Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   While this is clearly something of an improvement over earlier
designs, it doesn't deal with the most important issue: to do better
than categories (which in the medium term is mostly going to mean
"which flavor of Unicode is this?"), we need to look at statistical
behavior rather than ruling out categories via presence of specific
sequences.  This means the stream processor should

  1. keep octet distributions (octet, 2-, 3-, 4- octet sequences)

  2. in some kind of compressed form

  3. look for "skip features" (eg, characteristic behavior of leading
     bytes for UTF-7, UTF-8, UTF-16, Mule code)

  4. pick up certain "simple" regexps

  5. provide "triggers" to determine when statistical detectors should
     be invoked, such as octet count

  6. and "magic" like Unicode signatures or file(1) magic.

   -sjt


File: internals.info,  Node: Future Work -- Conversion Error Detection,  Next: Future Work -- Unicode,  Prev: Future Work -- Autodetection,  Up: Future Work -- Byte Code Snippets

40.16.2 Future Work - Conversion Error Detection
------------------------------------------------

"No Corruption" Scheme for Preserving External Encoding when Non-Invertible Transformation Applied
--------------------------------------------------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   A preliminary and simple implementation is:

     But you could implement it much more simply and usefully by just
     determining, for any text being decoded into mule-internal, can we
     go back and read the source again?  If not, remember the entire
     file (GNUS message, etc) in text properties.  Then, implement the
     UI interface (like Netscape's) on top of that.  This way, you have
     something that at least works, but it might be inefficient.  All we
     would need to do is work on making the underlying implementation
     more efficient.

   A more detailed proposal for avoiding binary file corruption is

     Basic idea: A coding system is a filter converting an entire input
     stream into an output stream. The resulting stream can be said to
     be "correspondent to" the input stream. Similarly, smaller units
     can correspond. These could potentially include zero width
     intervals on either side, but we avoid this.  Specifically, the
     coding system works like:

          loop (input) {

           Read bytes till we have enough to generate a translated character or a chars.

           This establishes a "correspondence" between the whole input and
           output more or less in minimal chunks.

          }

     We then do the following processing:

       1. Eliminate correspondences where one or the other of the I/O
          streams has a zero interval by combining with an adjacent
          interval;

       2. Group together all adjacent "identity" correspondences into as
          large groups as possible;

       3. Use text properties to store the non-identity correspondences
          on the characters. For identity correspondences, use a simple
          text property on all that contains no data but just indicates
          that the whole string of text is identity corresponded. (How
          do we define "identity"? Latin 1 or could it be something
          else? For example, Latin 2)?

       4. Figure out the procedures when text is inserted/deleted and
          copied or pasted.

       5. Figure out to save the file out making use of the
          correspondences. Allow ways of saving without
          correspondences, and doing a "save to buffer with and without
          correspondences."  Need to be clever when dealing with modal
          coding systems to parse the correspondences to get the
          internal state right.

Another Error-Catching Idea
---------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Nov 4, 1999

   Finally, I don't think "save the input" is as hard as you make it
out to be.  Conceptually, in fact, it's simple: for each minimal group
of bytes where you cannot absolutely guarantee that an
external->internal transformation is reversible, you put a text
property on the corresponding internal character indicating the bytes
that generated this character.  We also put a text property on every
character, indicating the coding system that caused the transformation.
This latter text property is extremely efficient (e.g. in a buffer
with no data pasted from elsewhere, it will map to a single extent over
all the buffer), and the former cases should not be prevalent enough to
cause a lot of inefficiency, esp. if we define what "reversible" means
for each coding system in such a way that it correctly handles the most
common cases.  The hardest part, in fact, is making all the string/text
handling in XEmacs be robust w.r.t. text properties.

Strategies for Error Annotation and Coding Orthogonalization
------------------------------------------------------------

Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   We really want to separate out a number of things.  Conceptually,
there is a nested syntax.

   At the top level is the ISO 2022 extension syntax, including charset
designation and invocation, and certain auxiliary controls such as the
ISO 6429 direction specification.  These are octet-oriented, with the
single exception (AFAIK) of the "exit Unicode" sequence which uses the
UTF's natural width (1 byte for UTF-7 and UTF-8, 2 bytes for UCS-2 and
UTF-16, and 4 bytes for UCS-4 and UTF-32).  This will be treated as a
(deprecated) special case in Unicode processing.

   The middle layer is ISO 2022 character interpretation.  This will
depend on the current state of the ISO 2022 registers, and assembles
octets into the character's internal representation.

   The lowest level is translating system control conventions.  At
present this is restricted to newline translation, but one could
imagine doing tab conversion or line wrapping here.  "Escape from
Unicode" processing would be done at this level.

   At each level the parser will verify the syntax.  In the case of a
syntax error or warning (such as a redundant escape sequence that
affects no characters), the parser will take some action, typically
inserting the erroneous octets directly into the output and creating an
annotation which can be used by higher level I/O to mark the affected
region.

   This should make it possible to do something sensible about
separating newline convention processing from character construction,
and about preventing ISO 2022 escape sequences from being recognized
inappropriately.

   The basic strategy will be to have octet classification tables, and
switch processing according to the table entry.

   It's possible that, by doing the processing with tables of functions
or the like, the parser can be used for both detection and translation.

Handling Writing a File Safely, Without Data Loss
-------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

     When writing a file, we need error detection; otherwise somebody
     will create a Unicode file without realizing the coding system of
     the buffer is Raw, and then lose all the non-ASCII/Latin-1 text
     when it's written out.  We need two levels

       1. first, a "safe-charset" level that checks before any actual
          encoding to see if all characters in the document can safely
          be represented using the given coding system.  FSF has a
          "safe-charset" property of coding systems, but it's stupid
          because this information can be automatically derived from
          the coding system, at least the vast majority of the time.
          What we need is some sort of
          alternative-coding-system-precedence-list, langenv-specific,
          where everything on it can be checked for safe charsets and
          then the user given a list of possibilities.  When the user
          does "save with specified encoding", they should see the same
          precedence list.  Again like with other precedence lists,
          there's also a global one, and presumably all coding systems
          not on other list get appended to the end (and perhaps not
          checked at all when doing safe-checking?).  safe-checking
          should work something like this: compile a list of all
          charsets used in the buffer, along with a count of chars
          used.  that way, "slightly unsafe" coding systems can perhaps
          be presented at the end, which will lose only a few characters
          and are perhaps what the users were looking for.

          [sjt sez this whole step is a crock.  If a universal coding
          system is unacceptable, the user had better know what he/she
          is doing, and explicitly specify a lossy encoding.  In
          principle, we can simply check for characters being writable
          as we go along.  Eg, via an "unrepresentable character
          handler."  We still have the buffer contents.  If we can't
          successfully save, then ask the user what to do.  (Do we ever
          simply destroy previous file version before completing a
          write?)]

       2. when actually writing out, we need error checking in case an
          individual char in a charset can't be written even though the
          charsets are safe.  again, the user gets the choice of other
          reasonable coding systems.

          [sjt - something is very confused, here; safe charsets should
          be defined as those charsets all of whose characters can be
          encoded.]

       3. same thing (error checking, list of alternatives, etc.) needs
          to happen when reading!  all of this will be a lot of work!

   Author: Stephen Turnbull (mailto:stephen@xemacs.org)

   I don't much like Ben's scheme.  First, this isn't an issue of I/O,
it's a coding issue.  It can happen in many places, not just on stream
I/O.  Error checking should take place on all translations.  Second,
the two-pass algorithm should be avoided if possible.  In some cases
(eg, output to a tty) we won't be able to go back and change the
previously output data.  Third, the whole idea of having a buffer full
of arbitrary characters which we're going to somehow shoehorn into a
file based on some twit user's less than informed idea of a coding
system is kind of laughable from the start.  If we're going to say that
a buffer has a coding system, shouldn't we enforce restrictions on what
you can put into it?  Fourth, what's the point of having safe charsets
if some of the characters in them are unsafe?  Fifth, what makes you
think we're going to have a list of charsets?  It seems to me that
there might be reasons to have user-defined charsets (eg, "German" vs
"French" subsets of ISO 8859/15).  Sixth, the idea of having language
environment determine precedence doesn't seem very useful to me.  Users
who are working with a language that corresponds to the language
environment are not going to run into safe charsets problems.  It's
users who are outside of their usual language environment who run into
trouble.  Also, the reason for specifying anything other than a
universal coding system is normally restrictions imposed by other users
or applications.  Seventh, the statistical feedback isn't terribly
useful.  Users rarely "want" a coding system, they want their file
saved in a useful way.  We could add a FORCE argument to conversions
for those who really want a specific coding system.  But mostly, a user
might want to edit out a few unsafe characters.  So (up to some
maximum) we should keep a list of unsafe text positions, and provide a
convenient function for traversing them.

   -sjt


File: internals.info,  Node: Future Work -- Unicode,  Next: Future Work -- BIDI Support,  Prev: Future Work -- Conversion Error Detection,  Up: Future Work -- Byte Code Snippets

40.16.3 Future Work - Unicode
-----------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Following is an old proposal.  Unicode has been implemented already,
in a different fashion; but there are some ideas here for more general
support, e.g. properties of Unicode characters other than their mappings
to particular charsets.

   We recognize 128, [256], 128x128, [256x256] for source charsets;

   for Unicode, 256x256 or 16x256x256.

   In all cases, use tables of tables and substitute a default subtable
if entire row is empty.

   If destination is Unicode, either 16 or 32 bits.

   If destination is charset, either 8 or 16 bits.

   For the moment, since we only do 94, 96, 94x94 or 96x96, only do 128
or 128x128 for source charsets and use the range 33-126 or 32-127.
(Except ASCII - we special case that and have no table because we can
algorithmically translate)

   Also have a 16x256x256 table -> 32 bits of Unicode char properties.

   A particular charset contains two associated mapping tables, for both
directions.

   API is set-unicode-mapping:

     (set-unicode-mapping
          unicode char
          unicode charset-code charset-offset
          unicode vector of char
          unicode list of char
          unicode string of char
          unicode vector or list of codes charset-offset

   Establishes a mapping between a unicode codepoint (an integer) and
one or more chars in a charset.  The mapping is automatically
established in both directions.  Chars in a charset can be specified
either with an actual character or a codepoint (i.e. an integer)   and
the charset it's within.  If a sequence of chars or charset   points is
given, multiple mappings are established for consecutive   unicode
codepoints starting with the given one.  Charset codepoints   are
specified as most-significant x 256 + least significant, with   both
bytes in the range 33-126 (for 94 or 94x94) or 32-127 (for 96   or
96x96), unless an offset is given, which will be subtracted from   each
byte.  (Most common values are 128, for codepoints given with   the
high bit set, or -32, for codepoints given as 1-94 or 0-95.)

   Other API's:

     (write-unicode-mapping file charset)

   Write the mapping table for a particular charset to the specified
file.  The tables are written in an internal format that allows for
efficient loading, for portability across platforms and XEmacs
invocations, for conserving space, for appending multiple tables one
directly after another with no need for a directory anywhere in the
file, and for reorganizing a file as in this format (with a magic
sequence at the beginning).  The data will be appended at the end of
a file, so that multiple tables can be written to a file; remove the
file first to avoid this.

     (write-unicode-properties file unicode-codepoint length)

   Write the Unicode properties (not including charset mappings) for
the specified range of contiguous Unicode codepoints to the end of
the file (i.e. append mode) in a binary format similar to what was
mentioned in the write-unicode-mapping description and with the same
features.

   Extension to set-unicode-mapping:

     (set-unicode-mapping
       list-or-vector-of-unicode-codepoints char
       ""                                   charset-code charset-offset
       ""                                   sequence of char
       ""                                   list-or-vector-of-codes
     charset-offset

   The first two forms are conceptually the inverse of the forms above
to specify characters for a contiguous range of Unicode codepoints.
These new forms let you specify the Unicode codepoints for a
contiguous range of chars in a charset.  "Contiguous" here means   that
if we run off the end of a row, we go to the first entry of the   next
row, rather than to an invalid code point.  For example, in a   94x94
charset, valid rows and columns are in the range 0x21-0x7e;   after
0x457c 0x457d 4x457e goes 0x4621, not something like 0x457f,   which is
invalid.

   The final two forms are the most general, letting you specify an
arbitrary set of both Unicode points and charset chars, and the two
are matched up just like a series of individual calls.  However, if
the lists or vectors do not have the same length, an error is
signaled.

     (load-unicode-mapping file &optional charset)

   If charset is omitted, loads all charset mapping tables found and
returns a list of the charsets found.  If charset is specified,
searches through the file for the appropriate mapping tables.  (This
is extremely fast because each entry in the file gives an offset to
the next one).  Returns t if found.

     (load-unicode-properties file unicode-codepoint)

     (list-unicode-entries file)

     (autoload-unicode-mapping charset)

   ...

   (unfinished)


File: internals.info,  Node: Future Work -- BIDI Support,  Next: Future Work -- Localized Text/Messages,  Prev: Future Work -- Unicode,  Up: Future Work -- Byte Code Snippets

40.16.4 Future Work - BIDI Support
----------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

  1. Use text properties to handle nesting levels, overrides
     BIDI-specific text properties (as per Unicode BIDI algorithm)
     computed at text insertion time.

  2. Lisp API for reordering a display line at redisplay time, possibly
     substitution of different glyphs (esp. mirroring of glyphs).

  3. Lisp API called after a display line is laid out, but only when
     reordering may be necessary (display engine checks for non-uniform
     BIDI text properties; can handle internally a line that's
     completely in one direction)

  4. Default direction is a buffer-local variable

  5. We concentrate on implementing Unicode BIDI algorithm.

  6. Display support for mirroring of entire window

  7. Display code keeps track of mirroring junctures so it can display
     double cursor.

  8. Entire layout of screen (on a per window basis) is exported as a
     Lisp API, for visual editing (also very useful for other purposes
     e.g. proper handling of word wrapping with proportional fonts,
     complex Lisp layout engines e.g. W3)

  9. Logical, visual, etc. cursor movement handled entirely in Lisp,
     using aforementioned API, plus a specifier for controlling how
     cursor is shown (e.g. split or not).


File: internals.info,  Node: Future Work -- Localized Text/Messages,  Prev: Future Work -- BIDI Support,  Up: Future Work -- Byte Code Snippets

40.16.5 Future Work - Localized Text/Messages
---------------------------------------------

NOTE: There is existing message translation in X Windows of menu names.
This is handled through X resources.  The files are in
`PACKAGES/mule-packages/locale/app-defaults/LOCALE/Emacs', where LOCALE
is `ja', `fr', etc.

   See lib-src/make-msgfile.lex.

   Long comment from jwz, some additions from ben marked "ben":

   (much of this comment is outdated, and a lot of it is actually
implemented)

40.16.6 Proposal for How This All Ought to Work
-----------------------------------------------

Author: Jamie Zawinski (mailto:jwz@jwz.org)

   this isn't implemented yet, but this is the plan-in-progress

   In general, it's accepted that the best way to internationalize is
for all messages to be referred to by a symbolic name (or number) and
come out of a table or tables, which are easy to change.

   However, with Emacs, we've got the task of internationalizing a huge
body of existing code, which already contains messages internally.

   For the C code we've got two options:

   * Use a Sun-like `gettext()' form, which takes an "english" string
     which appears literally in the source, and uses that as a hash key
     to find a translated string;

   * Rip all of the strings out and put them in a table.

   In this case, it's desirable to make as few changes as possible to
the C code, to make it easier to merge the code with the FSF version of
emacs which won't ever have these changes made to it.  So we should go
with the former option.

   The way it has been done (between 19.8 and 19.9) was to use
`gettext()', but *also* to make massive changes to the source code.
The goal now is to use `gettext()' at run-time and yet not require a
textual change to every line in the C code which contains a string
constant.  A possible way to do this is described below.

   (`gettext()' can be implemented in terms of `catgets()' for non-Sun
systems, so that in itself isn't a problem.)

   For the Lisp code, we've got basically the same options: put
everything in a table, or translate things implicitly.

   Another kink that lisp code introduces is that there are thousands
of third- party packages, so changing the source for all of those is
simply not an option.

   Is it a goal that if some third party package displays a message
which is one we know how to translate, then we translate it?  I think
this is a worthy goal.  It remains to be seen how well it will work in
practice.

   So, we should endeavor to minimize the impact on the lisp code.
Certain primitive lisp routines (the stuff in lisp/prim/, and
especially in `cmdloop.el' and `minibuf.el') may need to be changed to
know about translation, but that's an ideologically clean thing to do
because those are considered a part of the emacs substrate.

   However, if we find ourselves wanting to make changes to, say,
RMAIL, then something has gone wrong.  (Except to do things like remove
assumptions about the order of words within a sentence, or how
pluralization works.)

   There are two parts to the task of displaying translated strings to
the user: the first is to extract the strings which need to be
translated from the sources; and the second is to make some call which
will translate those strings before they are presented to the user.

   The old way was to use the same form to do both, that is,
`GETTEXT()' was both the tag that we searched for to build a catalog,
and was the form which did the translation.  The new plan is to
separate these two things more: the tags that we search for to build
the catalog will be stuff that was in there already, and the
translation will get done in some more centralized, lower level place.

   This program (`make-msgfile.c') addresses the first part, extracting
the strings.

   For the emacs C code, we need to recognize the following patterns:

       message ("string" ... )
       error ("string")
       report_file_error ("string" ... )
       signal_simple_error ("string" ... )
       signal_simple_error_2 ("string" ... )

       build_translated_string ("string")
       #### add this and use it instead of `build_string()' in some places.

       yes_or_no_p ("string" ... )
       #### add this instead of funcalling Qyes_or_no_p directly.

       barf_or_query_if_file_exists	#### restructure this
       check all callers of Fsignal	#### restructure these
       signal_error (Qerror ... )		#### change all of these to `error()'

       And we also parse out the `interactive' prompts from `DEFUN()' forms.

       #### When we've got a string which is a candidate for translation, we
       should ignore it if it contains only format directives, that is, if
       there are no alphabetic characters in it that are not a part of a `%'
       directive.  (Careful not to translate either "%s%s" or "%s: ".)

   For the emacs Lisp code, we need to recognize the following patterns:

       (message "string" ... )
       (error "string" ... )
       (format "string" ... )
       (read-from-minibuffer "string" ... )
       (read-shell-command "string" ... )
       (y-or-n-p "string" ... )
       (yes-or-no-p "string" ... )
       (read-file-name "string" ... )
       (temp-minibuffer-message "string")
       (query-replace-read-args "string" ... )

   I expect there will be a lot like the above; basically, any function
which is a commonly used wrapper around an eventual call to `message' or
`read-from-minibuffer' needs to be recognized by this program.

       (dgettext "domain-name" "string")		#### do we still need this?

       things that should probably be restructured:
         `princ' in `cmdloop.el'
         `insert' in `debug.el'
         face-interactive
         `help.el', `syntax.el' all messed up

   Author: Ben Wing (mailto:ben@xemacs.org)

   ben: (format) is a tricky case.  If I use format to create a string
that I then send to a file, I probably don't want the string translated.
On the other hand, If the string gets used as an argument to (y-or-n-p)
or some such function, I do want it translated, and it needs to be
translated before the %s and such are replaced.  The proper solution
here is for (format) and other functions that call gettext but don't
immediately output the string to the user to add the translated (and
formatted) string as a string property of the object, and have
functions that output potentially translated strings look for a
"translated string" property.  Of course, this will fail if someone
does something like

        (y-or-n-p (concat (if you-p "Do you " "Does he ")
          		(format "want to delete %s? " filename))))

   But you shouldn't be doing things like this anyway.

   ben: Also, to avoid excessive translating, strings should be marked
as translated once they get translated, and further calls to gettext
don't do any more translating.  Otherwise, a call like

        (y-or-n-p (format "Delete %s? " filename))

   would cause translation on both the pre-formatted and post-formatted
strings, which could lead to weird results in some cases (y-or-n-p has
to translate its argument because someone could pass a string to it
directly).  Note that the "translating too much" solution outlined
below could be implemented by just marking all strings that don't come
from a .el or .elc file as already translated.

   Menu descriptors: one way to extract the strings in menu labels
would be to teach this program about "^(defvar .*menu\n" forms; that's
probably kind of hard, though, so perhaps a better approach would be to
make this program recognize lines of the form

       "string" ... ;###translate

   where the magic token ";###translate" on a line means that the string
constant on this line should go into the message catalog.  This is
analogous to the magic ";###autoload" comments, and to the magic
comments used in the EPSF structuring conventions.

   ---- So this program manages to build up a catalog of strings to be
translated.  To address the second part of the problem, of actually
looking up the translations, there are hooks in a small number of low
level places in emacs.

   Assume the existence of a C function gettext(str) which returns the
translation of STR if there is one, otherwise returns STR.

   * `message()' takes a char* as its argument, and always filters it
     through `gettext()' before displaying it.

   * errors are printed by running the lisp function `display-error'
     which doesn't call `message' directly (it princ's to streams), so
     it must be carefully coded to translate its arguments.  This is
     only a few lines of code.

   * `Fread_minibuffer_internal()' is the lowest level interface to all
     minibuf interactions, so it is responsible for translating the
     value that will go into Vminibuf_prompt.

   * Fpopup_menu filters the menu titles through `gettext()'.

     The above take care of 99% of all messages the user ever sees.

   * The lisp function temp-minibuffer-message translates its arg.

   * query-replace-read-args is funny; it does   (setq from
     (read-from-minibuffer (format "%s: " string) ... ))   (setq to
     (read-from-minibuffer (format "%s %s with: " string from) ... ))

   What should we do about this?  We could hack query-replace-read-args
to translate its args, but might this be a more general problem?  I
don't think we ought to translate all calls to format.  We could just
change the calling sequence, since this is odd in that the first %s
wants to be translated but the second doesn't.

   Solving the "translating too much" problem:

   The concern has been raised that in this situation:

   * "Help" is a string for which we know a translation;

   * someone visits a file called Help, and someone does something
     contrived like (error buffer-file-name)

   then we would display the translation of Help, which would not be
correct.  We can solve this by adding a bit to Lisp_String objects
which identifies them as having been read as literal constants from a
.el or .elc file (as opposed to having been constructed at run time as
it would in the above case.)  To solve this:

   * `Fmessage()' takes a lisp string as its first argument.  If that
     string is a constant, that is, was read from a source file as a
     literal, then it calls `message()' with it, which translates.
     Otherwise, it calls `message_no_translate()', which does not
     translate.

   * `Ferror()' (actually, `Fsignal()' when condition is Qerror) works
     similarly.

   More specifically, we do:

      Scan specified C and Lisp files, extracting the following
     messages:

             C files:
                GETTEXT (...)
                DEFER_GETTEXT (...)
                DEFUN interactive prompts
             Lisp files:
                (gettext ...)
                (dgettext "domain-name" ...)
                (defer-gettext ...)
                (interactive ...)

     The arguments given to this program are all the C and Lisp source
     files of GNU Emacs.  .el and .c files are allowed.  There is no
     support for .elc files at this time, but they may be specified;
     the corresponding .el file will be used.  Similarly, .o files can
     also be specified, and the corresponding .c file will be used.
     This helps the makefile pass the correct list of files.

     The results, which go to standard output or to a file specified
     with -a or -o (-a to append, -o to start from nothing), are quoted
     strings wrapped in gettext(...).  The results can be passed to
     xgettext to produce a .po message file.

     However, we also need to do the following:

       1. Definition of Arg below won't handle a generalized argument
          as might appear in a function call.  This is fine for DEFUN
          and friends, because only simple arguments appear there; but
          it might run into problems if Arg is used for other sorts of
          functions.

       2. `snarf()' should be modified so that it doesn't output null
          strings and non-textual strings (see the comment at the top
          of `make-msgfile.c').

       3. parsing of (insert) should snarf all of the arguments.

       4. need to add set-keymap-prompt and deal with gettext of that.

       5. parsing of arguments should snarf all strings anywhere within
          the arguments, rather than just looking for a string as the
          argument.  This allows if statements as arguments to get
          parsed.

       6. `begin_paren_counting()' et al. should handle recursive entry.

       7. handle set-window-buffer and other such functions that take a
          buffer as the other-than-first argument.

       8. there is a fair amount of work to be done on the C code.
          Look through the code for #### comments associated with
          '#ifdef I18N3' or with an I18N3 nearby.

       9. Deal with `get-buffer-process' et al.

      10. Many of the changes in the Lisp code marked 'rewritten for
          I18N3 snarfing' should be undone once (5) is implemented.

      11. Go through the Lisp code in prim and make sure that all
          strings are gettexted as necessary.  This may reveal more
          things to implement.

      12. Do the equivalent of (8) for the Lisp code.

      13. Deal with parsing of menu specifications.


File: internals.info,  Node: Future Work -- Lisp Stream API,  Next: Future Work -- Multiple Values,  Prev: Future Work -- Byte Code Snippets,  Up: Future Work

40.17 Future Work - Lisp Stream API
===================================

Author: Ben Wing (mailto:ben@xemacs.org)

   Expose XEmacs internal lstreams to Lisp as stream objects.  (In
addition to the functions given below, each stream object has
properties that can be associated with it using the standard put, get
etc. API.  For GNU Emacs, where put and get have not been extended to
be general property functions, but work only on strings, we would have
to create functions set-stream-property, stream-property,
remove-stream-property, and stream-properties.  These provide the same
functionality as the generic get, put, remprop, and object-plist
functions under XEmacs)

   (Implement properties using a hash table, and *generalize* this so
that it is extremely easy to add a property interface onto any kind of
object)

     (write-stream STREAM STRING)

   Write the STRING to the STREAM.  This will signal an error if all the
bytes cannot be written.

     (read-stream STREAM &optional N SEQUENCE)

   Reads data from STREAM.  N specifies the number of bytes or
characters, depending on the stream.  SEQUENCE specifies where to write
the data into.  If N is not specified, data is read until end of file.
If SEQUENCE is not specified, the data is returned as a stream.  If
SEQUENCE is specified, the SEQUENCE must be large enough to hold the
data.

     (push-stream-marker STREAM)

   returns ID, probably a stream marker object

     (pop-stream-marker STREAM)

   backs up stream to last marker

     (unread-stream STREAM STRING)

   The only valid STREAM is an input stream in which case the data in
STRING is pushed back and will be read ahead of all other data.  In
general, there is no limit to the amount of data that can be unread or
the number of times that unread-stream can be called before another
read.

     (stream-available-chars STREAM)

   This returns the number of characters (or bytes) that can definitely
be read from the screen without an error.  This can be useful, for
example, when dealing with non-blocking streams when an attempt to read
too much data will result in a blocking error.

     (stream-seekable-p STREAM)

   Returns true if the stream is seekable.  If false, operations such as
seek-stream and stream-position will signal an error.  However, the
functions set-stream-marker and seek-stream-marker will still succeed
for an input stream.

     (stream-position STREAM)

   If STREAM is a seekable stream, returns a position which can be
passed to seek-stream.

     (seek-stream STREAM N)

   If STREAM is a seekable stream, move to the position indicated by N,
otherwise signal an error.

     (set-stream-marker STREAM)

   If STREAM is an input stream, create a marker at the current
position, which can later be moved back to.  The stream does not need
to be a seekable stream.  In this case, all successive data will be
buffered to simulate the effect of a seekable stream.  Therefore use
this function with care.

     (seek-stream-marker STREAM marker)

   Move the stream back to the position that was stored in the marker
object. (this is generally an opaque object of type stream-marker).

     (delete-stream-marker MARKER)

   Destroy the stream marker and if the stream is a non-seekable stream
and there are no other stream markers pointing to an earlier position,
frees up some buffering information.

     (delete-stream STREAM N)

     (delete-stream-marker STREAM ID)

     (close-stream stream)

   Writes any remaining data to the stream and closes it and the object
to which it's attached.  This also happens automatically when the
stream is garbage collected.

     (getchar-stream STREAM)

   Return a single character from the stream. (This may be a single byte
depending on the nature of the stream).  This is actually a macro with
an extremely efficient implementation (as efficient as you can get in
Emacs Lisp), so that this can be used without fear in a loop.  The
implementation works by reading a large amount of data into a vector
and then simply using the function AREF to read characters one by one
from the vector.  Because AREF is one of the primitives handled
specially by the byte interpreter, this will be very efficient.  The
actual implementation may in fact use the function
call-with-condition-handler to avoid the necessity of checking for
overflow.  Its typical implementation is to fetch the vector containing
the characters as a stream property, as well as the index into that
vector.  Then it retrieves the character and increments the value and
stores it back in the stream.  As a first implementation, we check to
see when we are reading the character whether the character would be
out of range.  If so, we read another 4096 characters, storing them
into the same vector, setting the index back to the beginning, and then
proceeding with the rest of the getchar algorithm.

     (putchar-stream STREAM CHAR)

   This is similar to getchar-stream but it writes data instead of
reading data.

     Function make-stream

   There are actually two stream-creation functions, which are:

     (make-input-stream TYPE PROPERTIES)
     (make-output-stream TYPE PROPERTIES)

   These can be used to create a stream that reads data, or writes data,
respectively.  PROPERTIES is a property list and the allowable
properties in it are defined by the type.  Possible types are:

  1. `file' (this reads data from a file or writes to a file)

     Allowable properties are:

    `:file-name'
          (the name of the file)

    `:create'
          (for output streams only, creates the file if it doesn't
          already exist)

    `:exclusive'
          (for output streams only, fails if the file already exists)

    `:append'
          (for output streams only; starts appending to the end of the
          file rather than overwriting the file)

    `:offset'
          (positions in bytes in the file where reading or writing
          should begin.  If unspecified, defaults to the beginning of
          the file or to the end of the file when :appended specified)

    `:count'
          (for input streams only, the number of bytes to read from the
          file before signaling "end of file".  If nil or omitted, the
          number of bytes is unlimited)

    `:non-blocking'
          (if true, reads or writes will fail if the operation would
          block.  This only makes sense for non-regular files).

  2. `process' (For output streams only, send data to a process.)

     Allowable properties are:

    `:process'
          (the process object)

  3. `buffer'  (Read from or write to a buffer.)

     Allowable properties are:

    `:buffer'
          (the name of the buffer or the buffer object.)

    `:start'
          (the position to start reading from or writing to.  If nil,
          use the buffer point.  If true, use the buffer's point and
          move point beyond the end of the data read or written.)

    `:end'
          (only for input streams, the position to stop reading at.  If
          nil, continue to the end of the buffer.)

    `:ignore-accessible'
          (if true, the default for :start and :end ignore any
          narrowing of the buffer.)

  4. `stream' (read from or write to a lisp stream)

     Allowable properties are:

    `:stream'
          (the stream object)

    `:offset'
          (the position to begin to be reading from or writing to)

    `:length'
          (For input streams only, the amount of data to read,
          defaulting to the rest of the data in the string.  Revise
          string for output streams only if true, the stream is resized
          as necessary to accommodate data written off the end,
          otherwise the writes will fail.

  5. `memory' (For output only, writes data to an internal memory
     buffer.  This is more lightweight than using a Lisp buffer.  The
     function memory-stream-string can be used to convert the memory
     into a string.)

  6. `debugging' (For output streams only, write data to the debugging
     output.)

  7. `stream-device' (During non-interactive invocations only, Read
     from or write to the initial stream terminal device.)

  8. `function' (For output streams only, send data by calling a
     function, exactly as with the STREAM argument to the print
     primitive.)

     Allowable Properties are:

    `:function'
          (the function to call.  The function is called with one
          argument, the stream.)

  9. `marker' (Write data to the location pointed to by a marker and
     move the marker past the data.)

     Allowable properties are:

    `:marker'
          (the marker object.)

 10. `decoding' (As an input stream, reads data from another stream and
     decodes it according to a coding system.  As an output stream
     decodes the data written to it according to a coding system and
     then writes results in another stream.)

     Properties are:

    `:coding-system'
          (the symbol of coding system object, which defines the
          decoding.)

    `:stream'
          (the stream on the other end.)

 11. `encoding' (As an input stream, reads data from another stream and
     encodes it according to a coding system.  As an output stream
     encodes the data written to it according to a coding system and
     then writes results in another stream.)

     Properties are:

    `:coding-system'
          (the symbol of coding system object, which defines the
          encoding.)

    `:stream'
          (the stream on the other end.)

   Consider

     (define-stream-type 'type
       :read-function
       :write-function
       :rewind-
       :seek-
       :tell-
       (?:buffer)

   Old Notes:

   Expose lstreams as hash (put get etc. properties) table.

       (write-stream stream string)
       (read-stream stream &optional n sequence)
       (make-stream ...)
       (push-stream-marker stream)
          returns ID prob a stream marker object
       (pop-stream-marker stream)
          backs up stream to last marker
       (unread-stream stream string)
       (stream-available-chars stream)
       (seek-stream stream n)
       (delete-stream stream n)
       (delete-stream-marker stream ic) can always be poe only nested if you
         have set stream marker

       (get-char-stream *generalizes* stream)

       a macro that tries to be efficient perhaps by reading the next
       e.g. 512 characters into a vector and arefing them.  Might check aref
       optimization for vectors in the byte interpreter.

       (make-stream 'process :process ... :type write)

       Consider

       (define-stream-type 'type
         :read-function
         :write-function
         :rewind-
         :seek-
         :tell-
         (?:buffer)


File: internals.info,  Node: Future Work -- Multiple Values,  Next: Future Work -- Macros,  Prev: Future Work -- Lisp Stream API,  Up: Future Work

40.18 Future Work - Multiple Values
===================================

Author: Ben Wing (mailto:ben@xemacs.org)

   On low level, all funs that can return multiple values are defined
with DEFUN_MULTIPLE_VALUES and have an extra parameter, a struct
mv_context *.

   It has to be this way to ensure that only the fun itself, and no
called funs, think they're called in an mv context.

   apply, funcall, eval might propagate their mv context to their
children?

   Might need eval-mv to implement calling a fun in an mv context.
Maybe also funcall_mv? apply_mv?

   Generally, just set up context appropriately.  Call fun (noticing
whether it's an mv-aware fun) and binding values on the way back or
passing them out.  (e.g. to multiple-value-bind)

Common Lisp multiple values, required for specifier improvements.
-----------------------------------------------------------------

The multiple return values from get-specifier should allow the
specifier value to be modified in the correct fashion (i.e.  should
interact correctly with all manner of changes from other callers) using
set-specifier.  We should check this and see if we need other return
values.  (how-to-add? inst-list?)

   In C, call multiple-values-context to get number of expected values,
and multiple-value-set (#, value) to get values other than the first.

   (Returns Qno_value, or something, if there are no values.

   #### Or should throw?  Probably not.  #### What happens if a fn
returns no values but the caller expects a #### value?

   Something like `funcall_with_multiple_values()' for setting up the
context.

   For efficiency, byte code could notice Ffuncall to m.v. functions and
sub in special opcodes during load in processing, if it mattered.


File: internals.info,  Node: Future Work -- Macros,  Next: Future Work -- Specifiers,  Prev: Future Work -- Multiple Values,  Up: Future Work

40.19 Future Work - Macros
==========================

Author: Ben Wing (mailto:ben@xemacs.org)

  1. Option to control whether beep really kills a macro execution.

  2. Recently defined macros are remembered on a stack, so accidentally
     defining another one doesn't fuck you up.  You can "rotate"
     anonymous macros or just pick one (numbered) to put on tags, so it
     works with execute macro - menu shows the anonymous macro, and
     lists some keystrokes.  Normally numbered but you can easily assign
     to named fun or to keyboard sequence or give it a number (or give
     it a letter accelerator?)


File: internals.info,  Node: Future Work -- Specifiers,  Next: Future Work -- Display Tables,  Prev: Future Work -- Macros,  Up: Future Work

40.20 Future Work - Specifiers
==============================

Author: Ben Wing (mailto:ben@xemacs.org)

Ideas To Work On When Their Time Has Come
-----------------------------------------

   * specifier-instance returns additional params (multiple-value) -
     the instantiator used, the associated tag set, the locale found
     in, a code that can be passed in as an additional param RESTART to
     restart an instantiation process, e.g. to allow an instantiator to
     "inherit" from another one higher up.  Also, domain can be 'global
     (look only in global specs) or "complex" - a list of the actual
     locales to look in (e.g. a buffer - frame - a device - 'global)

   * pragmatic-specifier-domain (locale) Converts a locale into a
     domain in a way that's "pragmatic" - does what most users expect
     will happen, but is not clean.  In particular, handling of
     "buffer" requires trickiness, as mentioned before.

   * ensure-instantiator-exists (specifier locale) Ensures an actual
     instantiator exists in a locale, so that it can later be futzed
     with.  If none exists, one is constructed by first calling
     pragmatic-specifier domain and then specifier-instance and
     fetching out the instantiator for this call.

   * map-modifying-instantiators (specifier fun &optional locale
     tag-set) Same args as map-specifier, but use the return value from
     the fun to replace the instantiator.  Called with three args
     (instantiator locale tag-set)

   * map-modifying-instantiators-force (specifier fun &optional locale
     tag-set) Same as previous, but calls ensure-instantiator-exists on
     each locale before processing.

   NOTE:  Can do preliminary implementation without Multiple Values -
instead create fun specifier-instance - that returns a list (and will
be deleted at some point)

specifier &c changes for glyphs
-------------------------------

  1.        * resizable vectors with funs to insert, delete elements
          (elements shift accordingly)

        * gap array vectors as an implementation of resizing vectors.

  2. You can `put' `get', etc. on vectors to modify properties within
     them.

  3. copy-over routines routines that carefully copy one complex item
     OVER another one, destroying the second in the process.  I wrote
     one for lists.  Need a general copy-over-tree.

  4. improvement to specifier mapping routines e.g.

     map-modifying-instantiator and its force versions below, so that we
     could implement in turns.

  5. put-specifier-property (specifier which finds the key, value
     instantiator in the locale, &opt locale possibly creating one
     tag-set) if necessary and goes into the vector, changes it, and
     puts it back into the specifier.

  6. Smarter add-spec-to-specifier

     If it notices that it's just replacing one instantiator with
     another, instead of just copy-tree the first one and throw away the
     other, use copy-over-tree to save lots of garbage when repeatedly
     called.

     ILLEGIBLE: GOTO LOO BUI BUGS LAST PNOTE

  7. When at image instantiate:
        * Some properties in the instantiators could be implemented
          through dynamically modifying an existing image instance
          (e.g. when the value of a slider or progress bar or text in a
          text field changes).  So when we hash, we only hash the part
          of the instantiator that cannot be dynamically modified (We
          might need to do something tricky here - allowing a :key
          property in hash tables or *ILLEGIBLE*).  Anyway, so we need
          to generate an image instance, and we mask off the dynamic
          properties and look up in our hash table, and we get
          something back!  But is it ours to modify?  (We already
          checked to see it wasn't exactly the same dynamic properties
          that it had)  Thus --

  8. Reference counting.  Somehow or other, each image instance in the
     cache needs to keep track of the instantiators that generated it.

   It might do this through some sort of special instantiator-reference
object.  This points to the instantiator, where in the hierarchy the
instantiator is etc.  When an instantiator gets removed, this
gu*ILLEGIBLE* values report not attached.  Somehow that gets
communicated back to the image instance in the cache.  So somehow or
other, the image instance in the cache knows who's using them and so
when you go and keep updating the slider value, by simply modifying an
instantiator, which efficiently changes the internal structure of this
specifier - eventually image instantiate notices that the image
instance it points has no other user and just modifiers it,  but in
complex situations, some optimizations get lost, but everything is
still correct.

   vs.

   Andy's set-image-instance-property, which achieves the same
optimizations much more easily, but

  1. falls apart in any more complicated system

  2. only works because of the way the caching system in XEmacs works.
     Any change (e.g. *ILLEGIBLE* more of making the caches GQ instead
     of GQ) is likely to make things stop working right in all but the
     simplest situation.

Specifier improvements for support of specifier inheritance (necessary for the new font mapping API)
----------------------------------------------------------------------------------------------------

'Fallback should be a locale/domain.

     (get-specifier specifier &optional locale)

     #### If locale is omitted, should it be (current-buffer) or 'global?
     #### Should argument not be optional?

   If a buffer is specified: find a window showing buffer by looking

   * at selected window

   * at other windows on selected frame

   * at selected windows on other frames in selected device

   * at other windows on ""

   * at selected windows on selected frames on other devices in selected
     console.

   * other windows sel from other devices sel con

   * ""       oth       ""           sel

   * sel win sel from sel dev oth con

   * oth win sel from sel dev oth con

   * sel win oth from sel dev oth con

   * oth win oth from sel dev oth con

   * sel win sel from oth dev oth con

   * oth win sel from oth dev oth con

   * oth win oth from oth dev oth con

   If none, use  buffer -> sel from -> etc.

     Returns multiple values
       second is instantiator
       third  is locale containing inst.
       fourth is tag set

     (restart-specifier-instance ...)

   like specifier-instance, but allows restarting the lookup, for
implementing inheritance, etc.  Obsoletes
specifier-matching-find-charset, or whatever it is.  The restart
argument is opaque, and is returned as a multiple value of
restart-specifier-instance.  (It's actually an integer with the low
bits holding the locale and the other bits count int to the list)
attached to the locale.)


File: internals.info,  Node: Future Work -- Display Tables,  Next: Future Work -- Making Elisp Function Calls Faster,  Prev: Future Work -- Specifiers,  Up: Future Work

40.21 Future Work - Display Tables
==================================

Author: Ben Wing (mailto:ben@xemacs.org)

   #### It would also be really nice if you could specify that the
characters come out in hex instead of in octal.  Mule does that by
adding a `ctl-hexa' variable similar to `ctl-arrow', but that's bogus -
we need a more general solution.  I think you need to extend the
concept of display tables into a more general conversion mechanism.
Ideally you could specify a Lisp function that converts characters, but
this violates the Second Golden Rule and besides would make things way
way way way slow.

   So instead, we extend the display-table concept, which was
historically limited to 256-byte vectors, to one of the following:

  1. A 256-entry vector, for backward compatibility;

  2. char-table, mapping characters to values;

  3. range-table, mapping ranges of characters to values;

  4. a list of the above.

   The fourth option allows you to specify multiple display tables
instead of just one.  Each display table can specify conversions for
some characters and leave others unchanged.  The way the character gets
displayed is determined by the first display table with a binding for
that character.  This way, you could call a function
`enable-hex-display' that adds a hex display-table to the list of
display tables for the current buffer.

   #### ...not yet implemented...  Also, we extend the concept of
"mapping" to include a printf-like spec.  Thus you can make all extended
characters show up as hex with a display table like this:

         #s(range-table data ((256 524288) (format "%x")))

   Since more than one display table is possible, you have great
flexibility in mapping ranges of characters.


File: internals.info,  Node: Future Work -- Making Elisp Function Calls Faster,  Next: Future Work -- Lisp Engine Replacement,  Prev: Future Work -- Display Tables,  Up: Future Work

40.22 Future Work - Making Elisp Function Calls Faster
======================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract: *This page describes many optimizations that can be made
to the existing Elisp function call mechanism without too much effort.
The most important optimizations can probably be implemented with only
a day or two of work.  I think it's important to do this work
regardless of whether we eventually decide to replace the Lisp engine.

   Many complaints have been made about the speed of Elisp, and in
particular about the slowness in executing function calls, and rightly
so.  If you look at the implementation of the `funcall' function,
you'll notice that it does an incredible amount of work.  Now logically,
it doesn't need to be so.  Let's look first from the theoretical
standpoint at what absolutely needs to be done to call a Lisp function.

   First, let's look at the situation that would exist if we were smart
enough to have made lexical scoping be the default language policy.  We
know at compile time exactly which code can reference the variables that
are the formal parameters for the function being called (specifically,
only the code that is part of that function's definition) and where
these references are.  As a result, we can simply push all the values of
the variables onto a stack, and convert all the variable references in
the function definition into stack references.  Therefore, binding
lexically-scoped parameters in preparation for a function call involves
nothing more than pushing the values of the parameters onto a stack and
then setting a new value for the frame pointer, at the same time
remembering the old one.  Because the byte-code interpreter has a
stack-based architecture, however, the parameter values have already
been pushed onto the stack at the time of the function call invocation.
Therefore, binding the variables involves doing nothing at all, other
than dealing with the frame pointer.

   With dynamic scoping, the situation is somewhat more complicated.
Because the parameters can be referenced anywhere, and these references
cannot be located at compile time, their values have to be stored into a
global table that maps the name of the parameter to its current value.
In Elisp, this table is called the "obarray".  Variable binding in
Elisp is done using the C function `specbind()'. (This stands for
"special variable binding" where "special" is the standard Lisp
terminology for a dynamically-scoped variable.)  What `specbind()'
does, essentially, is retrieve the old value of the variable out of the
obarray, remember the value by pushing it, along with the name of the
variable, onto what's called the "specpdl" stack, and then store the
new value into the obarray.  The term "specpdl" means "Special Variable
Pushdown List", where "Pushdown List" is an archaic computer science
term for a stack that used to be popular at MIT.  These binding
operations, however, should still not take very much time because of the
use of symbols, i.e. because the location in the obarray where the
variable's value is stored has already been determined (specifically, it
was determined at the time that the byte code was loaded and the symbol
created), so no expensive hash table lookups need to be performed.

   An actual function invocation in Elisp does a great deal more work,
however, than was just outlined above.  Let's just take a look at what
happens when one byte-compiled function invokes another byte-compiled
function, checking for places where unnecessary work is being done and
determining how to optimize these places.

  1.  The byte-compiled function's parameter list is stored in exactly
     the format that the programmer entered it in, which is to say as a
     Lisp list, complete with `&amp;optional' and `&amp;rest' keywords.
     This list has to be parsed for _every_ function invocation, which
     means that for every element in a list, the element is checked to
     see whether it's the `&amp;optional' or `&amp;rest' keywords, its
     surrounding cons cell is checked to make sure that it is indeed a
     cons cell, the `QUIT' macro is called, etc.  What should be
     happening here is that the argument list is parsed exactly once,
     at the time that the byte code is loaded, and converted into a C
     array.  The C array should be stored as part of the byte-code
     object.  The C array should also contain, in addition to the
     symbols themselves, the number of required and optional arguments.
     At function call time, the C array can be very quickly retrieved
     and processed.

  2.  For every variable that is to be bound, the `specbind()' function
     is called.  This actually does quite a lot of things, including:

       1.  Checking the symbol argument to the function to make sure
          it's actually a symbol.

       2.  Checking for specpdl stack overflow, and increasing its size
          as necessary.

       3.  Calling `symbol_value_buffer_local_info()' to retrieve
          buffer local information for the symbol, and then processing
          the return value from this function in a series of if
          statements.

       4.  Actually storing the old value onto the specpdl stack.

       5.  Calling `Fset()' to change the variable's value.



   The entire series of calls to `specbind()' should be inline and
merged into the argument processing code as a single tight loop, with no
function calls in the vast majority of cases.  The `specbind()' logic
should be streamlined as follows:

  1.  The symbol argument type checking is unnecessary.

  2.  The check for the specpdl stack overflow needs to be done only
     once, not once per argument.

  3.  All of the remaining logic should be boiled down as follows:

       1.  Retrieve the old value from the symbol's value cell.

       2.  If this value is a symbol-value-magic object, then call the
          real `specbind()' to do the work.

       3.  Otherwise, we know that nothing complicated needs to be
          done, so we simply push the symbol and its value onto the
          specpdl stack, and then replace the value in the symbol's
          value cell.

       4.  The only logic that we are omitting is the code in `Fset()'
          that checks to make sure a constant isn't being set.  These
          checks should be made at the time that the byte code for the
          function is loaded and the C array of parameters to the
          function is created.  (Whether a symbol is constant or not is
          generally known at XEmacs compile time.  The only issue here
          is with symbols whose names begin with a colon.  These
          symbols should simply be disallowed completely as parameter
          names.)



   Other optimizations that could be done are:

   *  At the beginning of the function that implements the byte-code
     interpreter (this is the Lisp primitive `byte-code'), the string
     containing the actual byte code is converted into an array of
     integers.  I added this code specifically for MULE so that the
     byte-code engine didn't have to deal with the complexities of the
     internal string format for text.  This conversion, however, is
     generally useful because on modern processors accessing 32-bit
     values out of an array is significantly faster than accessing
     unaligned 8-bit values.  This conversion takes time, though, and
     should be done once at load time rather than each time the byte
     code is executed.  This array should be stored in the byte-code
     object.  Currently, this is a bit tricky to do, because
     `byte-code' is not actually passed the byte-code object, but
     rather three of its elements.  We can't just change `byte-code' so
     that it is directly passed the byte-code object because this
     function, with its existing argument calling pattern, is called
     directly from compiled Elisp files.  What we can and should do,
     however, is create a subfunction that does take a byte-code object
     and actually implements the byte-code interpreter engine.
     Whenever the C code wants to execute byte code, it calls this
     subfunction.  `byte-code' itself also calls this subfunction after
     conjuring up an appropriate byte-code object and storing its
     arguments into this object.  With a small amount of work, it's
     possible to do this conjuring in such a way that it doesn't
     generate any garbage.

   *  At the end of a function call, the parameter bindings that have
     been done need to be undone.  This is standardly done by calling
     `unbind_to()'.  Just as for a `specbind()', this function does a
     lot of work that is unnecessary in the vast majority of cases, and
     it could also be inlined and streamlined.

   *  As part of each Elisp function call, a whole bunch of checks are
     done for a series of unlikely but possible conditions that may
     occur.  These include, for example,

        *  Calling the `QUIT' macro, which essentially involves
          checking a global volatile variable to see whether additional
          processing needs to be done.

        *  Checking whether a garbage collection needs to be done.

        *  Checking the variable `debug_on_next_call'.

        *  Checking for whether Elisp profiling is active.  (An
          additional optimization that's perhaps not worth the effort
          is to do some post-processing on the array of integers after
          it has been converted.  For example, whenever a 16-bit value
          occurs in the byte code, it has to be encoded as two separate
          8-bit values.  These values could be combined.  The tricky
          part here is that all of the places where a goto occurs
          across the place where this modification is made would have to
          have their offsets changed.  Other such optimizations can
          easily be imagined as well.)


   *  With a little bit smarter code, it should be possible to make a
     single trip variable that indicates whether any of these
     conditions is true.  This variable would be updated by any code
     that changes the actual variables whose values are checked in the
     various checks just mentioned.  (By the way, all of this is
     occurring in the C function `funcall_recording_as()'.)  There is a
     little bit of code between each of the checks.  This code would
     simply have to be duplicated between the two cases where this
     general trip variable is true and is false.  (Note: the
     optimization detailed in this item is probably not worth doing on
     the first pass.)



File: internals.info,  Node: Future Work -- Lisp Engine Replacement,  Next: Future Work -- Better Rendering Support,  Prev: Future Work -- Making Elisp Function Calls Faster,  Up: Future Work

40.23 Future Work - Lisp Engine Replacement
===========================================

* Menu:

* Future Work -- Lisp Engine Discussion::
* Future Work -- Lisp Engine Replacement -- Implementation::
* Future Work -- Startup File Modification by Packages::


File: internals.info,  Node: Future Work -- Lisp Engine Discussion,  Next: Future Work -- Lisp Engine Replacement -- Implementation,  Prev: Future Work -- Lisp Engine Replacement,  Up: Future Work -- Lisp Engine Replacement

40.23.1 Future Work - Lisp Engine Discussion
--------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract: *Recently there has been a great deal of talk on the
XEmacs mailing lists about potential changes to the XEmacs Lisp engine.
Usually the discussion has centered around the question which is better,
Common Lisp or Scheme?  This is certainly an interesting debate topic,
but it didn't seem to have much practical relevance to me, so I vowed to
stay out of the discussion.  Recently, however, it seems that people are
losing sight of the broader picture.  For example, nobody seems to be
asking the question, ""Would an extension language other than Lisp or
Scheme (perhaps not a Lisp variant at all) be more appropriate?"" Nor
does anybody seem to be addressing what I consider to be the most
fundamental question, is changing the extension language a good thing to
do?

   I think it would be a mistake at this point in XEmacs development to
begin any project involving fundamental changes to the Lisp engine or to
the XEmacs Lisp language itself.  It would take a huge amount of effort
to complete even part of this project, and would be a major drain on the
already-insufficient resources of the XEmacs development community.
Most of the gains that are purported to stem from a project such as this
could be obtained with far less effort by making more incremental
changes to the XEmacs core.  I think it would be an even bigger mistake
to change the actual XEmacs extension language (as opposed to just
changing the Lisp engine, making few, if any, externally visible
changes).  The only language change that I could possibly imagine
justifying would involve switching to some ubiquitous web language, such
as Java and JavaScript, or Perl.  (Even among those, I think Java would
be the only possibility that really makes sense).

   In the rest of this document I'll present the broader issues that
would be involved in changing the Lisp engine or extension language.
This should make clear why I've come to believe as I do.

Is everyone clear on the difference between interface and implementation?
-------------------------------------------------------------------------

There seems to be a great deal of confusion concerning the difference
between interface and implementation.  In the context of XEmacs,
changing the interface means switching to a different extension language
such as Common Lisp, Scheme, Java, etc.  Changing the implementation
means using a different Lisp engine.  There is obviously some relation
between these two issues, but there is no particular requirement that
one be changed if the other is changed.  It is quite possible, for
example, to imagine taking the underlying engine for any of the various
Lisp dialects in existence, and adapting it so that it implements the
same Elisp extension language that currently exists.  The vast majority
of the purported benefits that we would get from changing the extension
language could just as easily be obtained while making minimal changes
to the external Elisp interface.  This way nearly all existing Elisp
programs would continue to work, there would be no need to translate
Elisp programs into some other language or to simultaneously support two
incompatible Lisp variants, and there would be no need for users or
package authors to learn a new extension language that would be just as
unfamiliar to the vast majority of them as Elisp is.

Why should we change the Lisp engine?
-------------------------------------

Let's go over the possible reasons for changing the Lisp engine.

Speed.
......

Changing the Lisp engine might make XEmacs faster.  However, consider
the following.

  1.  XEmacs will get faster over time without any development effort
     at all because computers will get faster.

  2.  Perhaps the biggest causes of the slowness of XEmacs are not
     related to the Lisp engine at all.  It has been asserted, for
     example, that the slowness of XEmacs is primarily due to the
     redisplay mechanism, to the handling of insertion and deletion of
     text in a buffer, to the event loop, etc.  Nobody has done any
     real studies to determine what the actual cause of slowness is.

  3.  Emacs 18 seems plenty fast enough to most people.  However, Emacs
     18 also had a worse Lisp engine and a worse byte compiler than
     XEmacs.

  4.  Significant speed increases in the execution of Lisp code could be
     achieved without too much effort by working on the existing byte
     code interpreter and function call mechanism a bit.


Memory usage.
.............

A new Lisp engine with a better garbage collection mechanism might make
more efficient use of memory; for example, through the use of a
relocating garbage collector.  However, consider this:

  1.  A new Lisp engine would probably have a larger memory footprint,
     perhaps a significantly larger one.

  2.  The worst memory problems might not be due to Lisp object
     inefficiency at all.  The problems could simply be due mainly to
     the inefficient buffer representation.  Nobody has come up with
     any concrete numbers on where the real problem lies.


Robustness.
...........

A new Lisp engine might well be more robust.  (On the other hand, it
might not be.  It is not always easy to tell).  However, I think that
the biggest problems with robustness are in the part of the C code that
is not concerned with implementing the Lisp engine.  The redisplay
mechanism and the unexec mechanism are probably the biggest sources of
robustness problems.  I think the biggest robustness problems that are
related to the Lisp engine concern the use of GCPRO declarations.  The
entire GCPRO mechanism is ill-conceived and unsafe.  The only real way
to make this safe would be to do conservative garbage collection over
the C stack and to eliminate the GCPRO declarations entirely.  But how
many of the Lisp engines that are being considered have such a mechanism
built into them?

Maintainability.
................

A new Lisp engine might well improve the maintainability of XEmacs by
offloading the maintenance of the Lisp engine.  However, we need to make
very sure that this is, in fact, the case before embarking on a project
like this.  We would almost certainly have to make significant
modifications to any Lisp engine that we choose to integrate, and
without the active and committed support and cooperation of the
developers of that Lisp engine, the maintainability problem would
actually get worse.

Features.
.........

A new Lisp engine might have built in support for various features that
we would like to add to the XEmacs extension language, such as lexical
scoping and an object system.

Why would we want to change the extension language?
---------------------------------------------------

Possible reasons for changing the extension language include:

More standard.
..............

Switching to a language that is more standard and more commonly in use
would be beneficial for various reasons.  First of all, the language
that is more commonly used and more familiar would make it easier for
users to write their own extensions and in general, increase the
acceptance of XEmacs.  Also, an accepted standard probably has had a lot
more thought put into it than any language interface created by the
XEmacs developers themselves.  Furthermore, if our extension language is
being actively developed and supported, much of the work that we would
otherwise have to do ourselves is transferred elsewhere.

   However, both Scheme and Common Lisp flunk the familiarity test.
Neither language is being actively used for program development outside
of small research communities, and few prospective authors of XEmacs
extensions will be familiar with any Lisp variant for real world uses.
(I consider the argument that Scheme is often used in introductory
programming courses to be irrelevant.  Many existing programmers were
taught Pascal in their introductory programming courses.  How many of
them would actually be comfortable writing a program in Pascal?)
Furthermore, someone who wants to learn Lisp can't exactly go to their
neighborhood bookstore and pick up a book on this topic.

Ease of use.
............

There are endless arguments about which language is easiest to use.  In
practice, this largely boils down to which languages are most familiar.

Object oriented.
................

The object-oriented paradigm is the dominant one in use today for new
languages.  User interface concepts in particular are expressed very
naturally in an object-oriented system.  However, neither Scheme nor
Common Lisp has been designed with object orientation in mind.  There is
a standard object system for Common Lisp, but it is extremely complex
and difficult to understand.


File: internals.info,  Node: Future Work -- Lisp Engine Replacement -- Implementation,  Next: Future Work -- Startup File Modification by Packages,  Prev: Future Work -- Lisp Engine Discussion,  Up: Future Work -- Lisp Engine Replacement

40.23.2 Future Work - Lisp Engine Replacement - Implementation
--------------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Let's take a look at the sort of work that would be required if we
were to replace the existing Elisp engine in XEmacs with some other
engine, for example, the Clisp engine.  I'm assuming here, of course,
that we are not going to be changing the interface here at the same
time, which is to say that we will be keeping the same Elisp language
that we currently have as the extension language for XEmacs, except
perhaps for incremental changes that we will make, such as lexical
scoping and proper structure support in an attempt to gradually move
the language towards an upwardly-compatible goal, such as Common Lisp.
I am writing this page primarily as food for thought.  I feel fairly
strongly that actually doing this work would be a big waste of effort
that would inevitably become a huge time sink on the part of nearly
everyone involved in XEmacs development, and not only for the ones who
were supposed to be actually doing the engine change.  I feel that most
of the desired changes that we want for the language and/or the engine
can be achieved with much less effort and time through incremental
changes to the existing code base.

   First of all, in order to make a successful Lisp engine change in
XEmacs, it is vitally important that the work be done through a series
of incremental stages where at the end of each stage XEmacs can be
compiled and run, and it works.  It is tempting to try to make the
change all at once, but this would be disastrous.  If the resulting
product worked at all, it would inevitably contain a huge number of
subtle and extremely difficult to track down bugs, and it would be next
to impossible to determine which of the myriad changes made introduced
the bug.

   Now let's look at what the possible stages of implementation could
be.

An Extra C Preprocessing Stage
..............................

The first step would be to introduce another preprocessing stage for the
XEmacs C code, which is done before the C compiler itself is invoked on
the code, and before the standard C preprocessor runs.  The C
preprocessor is simply not powerful enough to do many of the things we
would like to do in the C code.  The existing results of this have been
a combination of a lot of hacked up and tricky-to-maintain stuff (such
as the `DEFUN' macro, and the associated `DEFSUBR'), as well as code
constructs that are difficult to write.  (Consider for example,
attempting to do structured exception handling, such as catch/throw and
unwind-protect constructs), as well as code that is potentially or
actually unsafe (such as the uses of `alloca'), which could easily
cause stack overflow with large amounts of memory allocated in this
fashion.)  The problem is that the C preprocessor does not allow macros
to have the power of an actual language, such as C or Lisp.  What our
own preprocessor should do is allow us to define macros, whose
definitions are simply functions written in some language which are
executed at compile time, and whose arguments are the actual argument
for the macro call, as well as an environment which should have a data
structure representation of the C code in the file and allow this
environment to be queried and modified.  It can be debated what the
language should be that these extensions are written in.  Whatever the
language chosen, it needs to be a very standard language and a language
whose compiler or interpreter is available on all of the platforms that
we could ever possibly consider putting XEmacs to, which is basically to
say all the platforms in existence.  One obvious choice is C, because
there will obviously be a C compiler available, because it is needed to
compile XEmacs itself.  Another possibility is Perl, which is already
installed on most systems, and is universally available on all others.
This language has powerful text processing facilities which would
probably make it possible to implement the macro definitions more
quickly and easily; however, this might also encourage bad coding
practices in the macros (often simple text processing is not
appropriate, and more sophisticated parsing or recursive data structure
processing needs to be done instead), and we'd have to make sure that
the nested data structure that comprises the environment could be
represented well in Perl.  Elisp would not be a good choice because it
would create a bootstrapping problem.  Other possible languages, such as
Python, are not appropriate, because most programmers are unfamiliar
with this language (creating a maintainability problem) and the Python
interpreter would have to be included and compiled as part of the XEmacs
compilation process (another maintainability problem).  Java is still
too much in flux to be considered at this point.

   The macro facility that we will provide needs to add two features to
the language: the ability to define a macro, and the ability to call a
macro.  One good way of doing this would be to make use of special
characters that have no meaning in the C language (or in C++ for that
matter), and thus can never appear in a C file outside of comments and
strings.  Two obvious characters are the @ sign and the $ sign.  We
could, for example, use `@' defined to define new macros, and the `$'
sign followed by the macro name to call a macro.  (Proponents of Perl
will note that both of these characters have a meaning in Perl.  This
should not be a problem, however, because the way that macros are
defined and called inside of another macro should not be through the use
of any special characters which would in effect be extending the macro
language, but through function calls made in the normal way for the
language.)

   The program that actually implements this extra preprocessing stage
needs to know a certain amount about how to parse C code.  In
particular, it needs to know how to recognize comments, strings,
character constants, and perhaps certain other kinds of C tokens, and
needs to be able to parse C code down to the statement level.  (This is
to say it needs to be able to parse function definitions and to separate
out the statements, `if' blocks, `while' blocks, etc. within these
definitions.  It probably doesn't, however need to parse the contents
of a C expression.)  The preprocessing program should work first by
parsing the entire file into a data structure (which may just contain
expressions in the form of literal strings rather than a data structure
representing the parsed expression).  This data structure should become
the environment parameter that is passed as an argument to macros as
mentioned above.  The implementation of the parsing could and probably
should be done using `lex' and `yacc'.  One good idea is simply to
steal some of the `lex' and `yacc' code that is part of GCC.

   Here are some possibilities that could be implemented as part of the
preprocessing:

  1.  A proper way of doing the `DEFUN' macros.  These could, for
     example, take an argument list in the form of a Lisp argument list
     (complete with keyword parameters and other complex features) and
     automatically generate the appropriate `subr' structure, the
     appropriate C function definition header, and the appropriate call
     to the `DEFSUBR' initialization function.

  2.  A truly safe and easy to use implementation of the `alloca'
     function.  This could allocate the memory in any fashion it chooses
     (calling `malloc' using a large global array, or a series of such
     arrays, etc.) an `insert' in the appropriate places to
     automatically free up this memory.  (Appropriate places here would
     be at the end of the function and before any return statements.
     Non-local exits can be handled in the function that actually
     implements the non-local exit.)

  3.  If we allow for the possibility of having an arbitrary Lisp
     engine, we can't necessarily assume that we can call Lisp
     primitives implemented in C from other C functions by simply
     making a function all.  Perhaps something special needs to happen
     when this is done.  This could be handled fairly easily by having
     our new and improved `DEFUN' macro define a new macro for use when
     calling a primitive.

Make the Existing Lisp Engine be Self-contained.
................................................

The goal of this stage is to gradually build up a self-contained Lisp
engine out of the existing XEmacs core, which has no dependencies on any
of the code elsewhere in the XEmacs core, and has a well-defined and
black box-style interface.  (This is to say that the rest of the C code
should not be able to access the implementation of the Lisp engine, and
should make as few assumptions as possible about how this implementation
works).  The Lisp engine could, and probably should, be built up as a
separate library which can be compiled on its own without any of the
rest of the XEmacs C code, and can be tested in this configuration as
well.

   The creation of this engine library should be done as a series of
subsets, each of which moves more code out of the XEmacs core and into
the engine library, and XEmacs should be compilable and runnable between
each sub-step.  One possible series of sub-steps would be to first
create an engine that does only object allocation and garbage
collection, then as a second sub-step, move in the code that handles
symbols, symbol values, and simple binding, and then finally move in the
code that handles control structures, function calling, `byte-code'
execution, exception handling, etc.  (It might well be possible to
further separate this last sub-step).

Removal of Assumptions About the Lisp Engine Implementation
...........................................................

Currently, the XEmacs C code makes all sorts of assumptions about the
implementation of the Lisp engine, particularly in the areas of object
allocation, object representation, and garbage collection.  A different
Lisp engine may well have different ways of doing these implementations,
and thus the XEmacs C code must be rid of any assumptions about these
implementations.  This is a tough and tedious job, but it needs to be
done.  Here are some examples:

  1.  `GCPRO' must go.  The `GCPRO' mechanism is tedious, error-prone,
     unmaintainable, and fundamentally unsafe.  As anyone who has
     worked on the C Core of XEmacs knows, figuring out where to insert
     the `GCPRO' calls is an exercise in black magic, and debugging
     crashes as a result of incorrect `GCPROing' is an absolute
     nightmare.  Furthermore, the entire mechanism is fundamentally
     unsafe.  Even if we were to use the extra preprocessing stage
     detailed above to automatically generate `GCPRO' and `UNGCPRO'
     calls for all Lisp object variables occurring anywhere in the C
     code, there are still places where we could be bitten.  Consider,
     for example, code which calls `cons' and where the two arguments
     to this functions are both calls to the `append' function.  Now
     the `append' function generates new Lisp objects, and it also
     calls `QUIT', which could potentially execute arbitrary Lisp code
     and cause a garbage collection before returning control to the
     `append' function.  Now in order to generate the arguments to the
     `cons' function, the `append' function is called twice in a row.
     When the first `append' call returns, new Lisp data has been
     created, but has no `GCPRO' pointers to it.  If the second
     `append' call causes a garbage collection, the Lisp data from the
     first `append' call will be collected and recycled, which is
     likely to lead to obscure and impossible-to-debug crashes.  The
     only way around this would be to rewrite all function calls whose
     parameters are Lisp objects in terms of temporary variables, so
     that no such function calls ever contain other function calls as
     arguments.  This would not only be annoying to implement, even in
     a smart preprocessor, but would make the C code become incredibly
     slow because of all the constant updating of the `GCPRO' lists.

  2.  The only proper solution here is to completely do away with the
     `GCPRO' mechanism and simply do conservative garbage collection
     over the C stack.  There are already portable implementations of
     conservative pointer marking over the C stack, and these could
     easily be adapted for use in the Elisp garbage collector.  If, as
     outlined above, we use an extra preprocessing stage to create a
     new version of `alloca' that allocates its memory elsewhere than
     actually on the C stack, and we ensure that we don't declare any
     large arrays as local variables, but instead use `alloca', then we
     can be guaranteed that the C stack is small and thus that the
     conservative pointer marking stage will be fast and not very
     likely to find false matches.

  3.  Removing the `GCPRO' declarations as just outlined would also
     remove the assumption currently made that garbage collection can
     occur only in certain places in the C code, rather than in any
     arbitrary spot.  (For example, any time an allocation of Lisp data
     happens).  In order to make things really safe, however, we also
     have to remove another assumption as detailed in the following
     item.

  4.  Lisp objects might be relocatable.  Currently, the C code assumes
     that Lisp objects other than string data are not relocatable and
     therefore it's safe to pass around and hold onto the actual
     pointers for the C structures that implement the Lisp objects.
     Current code, for example, assumes that a `Lisp_Object' of type
     buffer and a C pointer to a `struct buffer' mean basically the
     same thing, and indiscriminately passes the two kinds of buffer
     pointers around.  With relocatable Lisp objects, the pointers to
     the C structures might change at any time.  (Remember, we are now
     assuming that a garbage collection can happen at basically any
     point).  All of the C code needs to be changed so that Lisp
     objects are always passed around using a Lisp object type, and the
     underlying pointers are only retrieved at the time when a
     particular data element out of the structure is needed.  (As an
     aside, here's another reason why Lisp objects, instead of
     pointers, should always be passed around.  If pointers are passed
     around, it's conceivable that at the time a garbage collection
     occurs, the only reference to a Lisp object (for example, a
     deleted buffer) would be in the form of a C pointer rather than a
     Lisp object.  In such a case, the conservative pointer marking
     mechanism might not notice the reference, especially if, in an
     attempt to eliminate false matches and make the code generally
     more efficient, it will be written so that it will look for actual
     Lisp object references.)

  5.  I would go a step farther and completely eliminate the macros that
     convert a Lisp object reference into a C pointer.  This way the
     only way to access an element out of a Lisp object would be to use
     the macro for that element, which in one atomic operation
     de-references the Lisp object reference and retrieves the value
     contained in the element.  We probably do need the ability to
     retrieve actual C pointers, though.  For example, in the case
     where an array is stored in a Lisp object, or simply for
     efficiency purposes where we might want some code to retrieve the
     C pointer for a Lisp object, and work on that directly to avoid a
     whole bunch of extra indirections.  I think the way to do this
     would be through the use of a special locking construct
     implemented as part of the extra preprocessor stage mentioned
     above.  This would essentially be what you might call a "lock
     block", just like a `while' block.  You'd write the word `lock'
     followed by a parenthesized expression that retrieves the C
     pointer and stores it into a variable that is scoped only within
     the lock block and followed in turn by some code in braces, which
     is the actual code associated with the lock block, and which can
     make use of this pointer.  While the code inside the lock block is
     executing, that particular pointer and the object pointed to by it
     is guaranteed not to be relocated.

  6.  If all the XEmacs C code were converted according to these rules,
     there would be no restrictions on the sorts of implementations
     that can be used for the garbage collector.  It would be possible,
     for example, to have an incremental asynchronous relocating
     garbage collector that operated continuously in another thread
     while XEmacs was running.

  7.  The C implementation of Lisp objects might not, and probably
     should not, be visible to the rest of the XEmacs C code.  It
     should theoretically be possible, for example, to implement Lisp
     objects entirely in terms of association lists, rather than using
     C structures in the standard way.  (This may be an extreme
     example, but it's good to keep in mind an example such as this
     when cleaning up the XEmacs C code).  The changes mentioned in the
     previous item would go a long way towards removing this
     assumption.  The only places where this assumption might still be
     made would be inside of the lock blocks where an actual pointer is
     retrieved.  (Also, of course, we'd have to change the way that
     Lisp objects are defined in C so that this is done with some
     function calls and new and improved macros rather than by having
     the XEmacs C code actually define the structures.  This sort of
     thing would probably have to be done in any case once the
     allocation mechanism is moved into a separate library.)  With some
     thought it should be possible to define the lock block interface
     in such a way as to remove any assumptions about the
     implementation of Lisp objects.

  8.  C code may not be able to call Lisp primitives that are defined
     in C simply by making standard C function calls.  There might need
     to be some wrapper around all such calls.  This could be achieved
     cleanly through the extra preprocessing step mentioned above, in
     line with the example described there.


Actually Replacing the Engine.
..............................

Once we've done all of the work mentioned in the previous steps (and
admittedly, this is quite a lot of work), we should have an XEmacs that
still uses what is essentially the old and previously existing Lisp
engine, but which is ready to have its Lisp engine replaced.  The
replacement might proceed as follows:

  1.  Identify any further changes that need to be made to the engine
     interface that we have defined as a result of the previous steps
     so that features and idiosyncrasies of various Lisp engines that
     we examine could be properly supported.

  2.  Pick a Lisp engine and write an interface layer that sits on top
     of this Lisp engine and makes it adhere to what I'll now call the
     XEmacs Lisp engine interface.

  3.  Strongly consider creating, if we haven't already done so, a test
     suite that can test the XEmacs Lisp engine interface when used
     with a stand-alone Lisp engine.

  4.  Test the hell out of the Lisp engine that we've chosen when
     combined with its XEmacs Lisp engine interface layer as a
     stand-alone program.

  5.  Now finally attach this stand-alone program to XEmacs itself.
     Debug and fix any further problems that ensue (and there
     inevitably will be such problems), updating the test suite as we
     go along so that if it were run again on the old and buggy
     interfaced Lisp engine, it would note the bug.



File: internals.info,  Node: Future Work -- Startup File Modification by Packages,  Prev: Future Work -- Lisp Engine Replacement -- Implementation,  Up: Future Work -- Lisp Engine Replacement

40.23.3 Future Work - Startup File Modification by Packages
-----------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   OK, we need to create a design document for all of this, including:

   PRINCIPLE #1: Whenever you have auto-generated stuff, *CLEARLY*
indicate this in comments around the stuff.  These comments get
searched for, and used to locate the existing generated stuff to
replace.  Custom currently doesn't do this.

   PRINCIPLE #2: Currently, lots of functions want to add code to the
.emacs. (e.g. I get prompted for my mail address from
add-change-log-entry, and then prompted if I want to make this
permanent).  There needs to be a Lisp API for working with arbitrary
code to be added to a user's startup.  This API hides all the details
of which file to put the fragment in, where in it, how to mark it with
magical comments of the right kind so that previous fragments can be
replaced, etc.

   PRINCIPLE #3: *ALL* generated stuff should be loaded before any
user-written init stuff.  This way the user can override the generated
settings.  Although in the case of customize, it may work when the
custom stuff is at the end of the init file, it surely won't work for
arbitrary code fragments (which typically do `setq' or the like).

   PRINCIPLE #4: As much as possible, generated stuff should be place in
separate files from non-generated stuff.  Otherwise it's inevitable
that some corruption is going to result.

   PRINCIPLE #5: Packages are encouraged, as much as possible, to work
within the customize model and store all their customizations there.
However, if they really need to have their own init files, these files
should be placed in .xemacs/, given normal names (e.g.
`saved-abbrevs.el' not .abbrevs), and there should be some magic
comment at the top of the file that causes it to get automatically
loaded while loading a user's init file. (Alternatively, the
above-named API could specify a function that lets a package specify
that they want such-and-such file loaded from the init file, and have
the specifics of this get handled correctly.)

   OVERARCHING GOAL: The overarching goal is to provide a unified
mechanism for packages to store state and setting information about the
user and what they were doing when XEmacs exited, so that the same or a
similar environment can be automatically set up the next time.  In
general, we are working more and more towards being a truly GUI app
where users' settings are easy to change and get remembered correctly
and consistently from one session to the next, rather than requiring
nasty hacking in elisp.

   Hrvoje, do you have any interest in this?  How about you, Martin?
This seems like it might be up your alley.  This stuff has been
ad-hocked since kingdom come, and it's high time that we make this work
properly so that it could be relied upon, and a lot of things could
"just work".


File: internals.info,  Node: Future Work -- Better Rendering Support,  Prev: Future Work -- Lisp Engine Replacement,  Up: Future Work

40.24 Future Work - Better Rendering Support
============================================

This section was written by Stephen Turnbull <stephen@xemacs.org>, so
don't blame Ben (or Eric and Matthias, for that matter).  Feel free to
add, edit, and share the blame, guys!

   As of late November 2004, this principally means adding support for
the `Xft' library, which provides a more robust _font configuration_
mechanism via Keith Packard's `fontconfig' library improved glyph
rendering, including antialiasing, via the `freetype' library, and
client-side rendering (saving bandwidth and server memory) via the
`XRender extension'.  In fact, patches which provide Xft support have
been available for several years, but the authors have been unwilling
to deal with several important issues which block integration.  These
are _Mule_, and more generally, _face_ support; _widget_ support
(including the toolbar and menubar); and _redisplay refactoring_.

   However, in late 2003 Eric Knauel
<knauel@informatik.uni-tuebingen.de> and Matthias Neubauer
<neubauer@informatik.uni-freiburg.de> put forward a relatively complete
patch which was robust to daily use in ISO 8859-1 locales, and Stephen
Turnbull began work on the integration issues.  At this point a
(private) CVS branch is available for Stephen's patch (branch point tag
`sjt-xft-bp', branch tag `sjt-xft'), and one may be made available for
the Knauel-Matthias patch soon.

* Menu:

* Better Rendering Support -- Review Criteria::
* Better Rendering Support -- Implementation::
* Better Rendering Support -- Current Status::
* Better Rendering Support -- Configuration with the Interim Patches::
* Better Rendering Support -- Modern Font Support::


File: internals.info,  Node: Better Rendering Support -- Review Criteria,  Next: Better Rendering Support -- Implementation,  Up: Future Work -- Better Rendering Support

40.24.1 Better Rendering Support - Review Criteria
--------------------------------------------------

Of course it's "unfair" to demand that the implementers of a nice
feature like anti-aliasing support deal with accumulated cruft of the
last few years, but somebody must, sometime soon.  Even core developers
are complaining about how slow XEmacs is in some applications, and there
is reason to believe that some of the problem is in redisplay.  Adding
more _ad hoc_ features to redisplay will make the whole module more
complex and unintelligible.  Even if it doesn't inherently further
detract from efficiency, it will surely make reform and refactoring
harder.

   Similar considerations apply to Mule support.  If Xft support is not
carefully designed, or implemented with Mule support soon, it will
undoubtedly make later Mule implementation far more difficult than it
needs to be, and require redundant work be done (_e.g._, on `Options'
menu support).

   Besides the design issue--and many users are requesting more
flexibility, primarily face support, from the widgets--with widget
support there is also an aesthetic issue.  It is horribly unimpressive
to have clunky bitmapped fonts on the decorations when pleasant
antialiased fonts are available in the buffer.

   Finally, these issues interact.  Widgets and faces are inherently
heavyweight objects, requiring orders of magnitude more computation than
simply displaying a string in a fixed font.  This will have an
efficiency impact, of course.  And they interact with each other; Mule
was designed for use in buffers and display in Emacs windows--but a
widget's content is usually not a buffer, and widgets need not be
displayed in a window, but may appear in other contexts, especially in
the gutters.  So specifiers will probably have to be reworked, in order
to properly support display of different faces in non-buffer, non-window
contexts.


File: internals.info,  Node: Better Rendering Support -- Implementation,  Next: Better Rendering Support -- Current Status,  Prev: Better Rendering Support -- Review Criteria,  Up: Future Work -- Better Rendering Support

40.24.2 Better Rendering Support - Implementation
-------------------------------------------------

Stephen is thinking in terms of the following components of a
comprehensive proposal.

*Font configuration*
     In XEmacs, font configuration is handled via _faces_.  Currently
     XEmacs uses a special type of _font specifier_ to map XEmacs
     locales to font names.  Especially under X11, this can cause
     annoying problems because of the unreliability of X servers'
     mappings from `XLFD' names to X11 fonts, over which XEmacs has no
     influence whatsoever.  However, the `fontconfig' library which is
     used with `Xft' provides much more reliable mapping, along with a
     more reliably parsable naming scheme similar to that used by
     TrueType fonts on MS Windows and the Macintosh.  Since the
     capabilities of font specifiers and `fontconfig' overlap, we
     should consider using `fontconfig' instead of `XLFD' names.  This
     implies that use of `Xft''s rendering functionality should be
     separated from use of `fontconfig'.

*fontconfig*
     Fontconfig is dramatically different from the X model in several
     ways.  In particular, _fontconfig always returns a font_.
     However, the font returned need not be anything like the desired
     font.  This means that XEmacs must adopt a strategy of delegating
     the search to fontconfig, then sanity-checking the return, rather
     than trying to use the fontconfig API to search using techniques
     appropriate for the X11 core font API.

*Font menus*
     The `Options->Font' and `Options->Font Sizes' menus are broken, by
     design, not just by `Xft'.  Although they work better in Eric and
     Matthias's patch than in Stephen's, even their version has the
     problem that many fonts are unavailable because they don't match
     the current size--which is very strange, since `Xft' fonts are of
     course scalable.  But the whole idea of requiring that the font
     match the size is strange.  And the `Options->Font Weights' menu
     is just disabled, and has been for eons.

*X resources*
     Currently in Stephen's patch there are _four_ treatments of font
     resources.  There are the `XEmacs.FACE.attributeFont' resources
     used to set a single global font specification.  In the widgets,
     some (still) have a `font' resource using the automatic `Xt'
     resource conversion to `FontStruct', some have separate `font' and
     `xftFont' resources with the former automatically converted to
     `FontStruct' by `Xt' and the latter left as a string, to be
     converted by `XftParseName' later, and some have a single `font'
     resource which is converted to `FontStruct' by `Xt' or the latter
     left as a string, depending on whether `Xft' was enabled by
     `configure' or not.

     Stephen thinks that all of these should be converted to use the
     face approach, perhaps with some way to set specifications for
     individual widgets, frames, or buffers.  This will require some
     careful design work to incorporate face support in the widgets.
     We should just accept any or all of `font', `fontSet', and
     `fontList' resources, treat them all as lists of font names,
     either `XLFD'- or `fontconfig'-style, parse them ourselves (ie,
     not use the `Xt' resource manager), and add them to font
     specifiers as appropriate.  But this will require a bit of thought
     to obey POLA vis-a-vis usual `Xt' conventions.

*Rendering engine objects*
     With the introduction of the "Xft patch," the X11, Macintosh, and
     MS Windows platforms are all able to support multiple font
     rendering engines in the same binary.  Generically, there are
     several tasks that must be accomplished to render text on the
     display.  In both cases the code is rather disorganized, with
     substantial cross-platform duplication of similar routines.  While
     it may not be worthwhile to go the whole way to
     `RENDERER_HAS_METHOD' and `MAYBE_RENDMETH', refactoring these
     modules around the notion of interfacing a "generic rendering
     engine interface" to "text" seems like a plausible way to focus
     this work.

     Further evidence for this kind of approach is a bug recently fixed
     in the `xft-sjt' branch.  XEmacs was crashing because the Athena
     Label widget tried to access a nonexistent font in its
     initialization routine.  The font didn't exist because although no
     core X11 font corresponding to the spec existed, an Xft font was
     found.  So the XEmacs font instance existed but it did not specify
     an X11 core font, only the Xft font.  When this object was used to
     initialize the font for the Label widget, None (0) was passed to
     XtSetArgs, then XtCreateWidget was called, and the internal
     initialization routine attempted to access that (nonexistent) font
     while computing an X11 graphics context (GC).

     A similar issue applies to colors, but there Xft colors keep the
     pixel data internally, so (serendipitously) the X11 color (_i.e._,
     pixel) member does get updated.

*Colors, fonts, and faces*
     Besides the rendering engine itself, the XEmacs implementations of
     these objects are poorly supported by current widget
     implementations, including the traditional menubar and toolbar, as
     well as the more recent button, tab control, and progress bar
     widgets.  The refactoring suggested under "Rendering engine
     objects" should be conducted with an eye to making these widgets
     support faces, perhaps even to the extent of allowing rendering to
     X pixmaps (which some Athena widgets support, although they will
     not support rendering via Xft directly). Especially with `XRender'
     technology this should not be horribly inefficient.

*Specifiers, charsets, and languages*
     Traditionally Mule uses a rather rigid and low-level abstraction,
     the _charset_, to characterize font repertoires.  Unfortunately,
     support for a given charset is generally neither necessary nor
     sufficient to support a language.  Worse, although X11's only
     means for indicating font repertoires is the font's _registry_,
     the actual repertoire of many fonts is either deficient or
     font-dependent.  The only convenience is that the registry maps
     directly to a Mule charset in most cases, and vice versa.

     To date, XEmacs Mule has supported identification of appropriate
     fonts to support a language's repertoire of characters by
     identifying the repertoire as a subset of a union of charsets.  To
     each charset there is a regular expression matching the registry
     portion of a font name.  Then instantiation of a font proceeds by
     identifying the specifier domain, and then walking down the list
     of specifications, matching the regexp against font names until a
     match is found.  That font is requested from the system, and if
     not found, the process continues similarly until a font that can
     be loaded is found.

     This has several problems.  First, there's no guarantee that the
     union will be disjoint.  This problem manifests both in the case
     of display of Unicode representations of text in the `POSIX'
     default locale, where glyphs are typically drawn from several
     inappropriate fonts.  A similar problem often occurs, though for a
     different reason, in multilingual messages composed using `Gnus''s
     `message-mode' and MIME support.  This problem _cannot_ be avoided
     with the current design; it is quite possible that a font desired
     in one context will be shadowed by a font intended to get higher
     priority in a semantically different but syntactically similar (as
     far as Mule can tell) context.  (Of course, one could attach a
     different face as a text property, but that requires programming
     support; it can't be done by user configuration.)  The problem is
     only exacerbated as more and more Unicode fonts, supporting large
     repertoires with substantial overlap across fonts, are designed
     and published.

     A second problem is that registry names are often inaccurate.  For
     example, the Japanese JIS X 0208 standard was first published in
     1978 (as a relabelling of an older standard).  It was then revised
     in 1983, again in 1990, and once again in 2000, with slight
     changes to the repertoire and mapping in each revision.
     Technically, these standards can be distinguished in properly
     named fonts as `jisx0208.1978', `jisx0208.1983', `jisx0208.1990',
     `jisx0208.2000', but all of them are commonly simply labelled
     `jisx0208', and Western distributors, of course, generally lack
     the expertise to correctly relabel them.

     A third problem is that you generally can't tell if there are
     "holes" in the repertoire until you try to display the glyph.

     All of this tends to break standard idioms for handling Mule fonts
     in `init' files because they depend on charsets being disjoint
     repertoires.

     The TrueType fonts (and the later OpenType standard) provides for a
     proper character set query (as a Boolean vector indexed by Unicode
     code points), as well as providing a list of supported languages.

     I propose that we take advantage of these latter facilities by
     allowing a font to be specified either as a string (a font name),
     or as a list whose head is the font name and whose tail is a list
     of languages and Mule charsets (for backward compatibility) that
     user intends to use the font to display.  This will probably
     require a change to the specifier code.

     As mentioned above, specifiers will probably also have to be
     enhanced to recognize `widget' locales and domains, instead of the
     current hack where special `widget' and `gui-element' faces are
     created.

*Customize*
     Customize needs to deal with all this stuff!!


File: internals.info,  Node: Better Rendering Support -- Current Status,  Next: Better Rendering Support -- Configuration with the Interim Patches,  Prev: Better Rendering Support -- Implementation,  Up: Future Work -- Better Rendering Support

40.24.3 Better Rendering Support - Current Status
-------------------------------------------------

Stephen has a branch containing his stuff in XEmacs CVS.  The branch
point tag is `sjt-xft-bp', roughly corresponding to XEmacs 21.5.18, and
branch tag is `sjt-xft'.

40.24.3.1 Bugs Reported in sjt-xft
..................................

`ChangeLogs'
     A lot of these, especially for Eric and Matthias's work, are
     missing.  Mea culpa.

`Options->Font'
`Options->Font Size'
     These menus don't work.  All fonts are greyed out.  All sizes are
     available, but many (most?) faces don't change size, in particular,
     `default' does not.

`Antialiased text bleeding outside of reported extent'
     On my PowerBook G4 Titanium 15" screen, X.org server v6.8.1,
     dimensions:    1280x833 pixels (433x282 millimeters),
     resolution:    75x75 dots per inch,   depth of root window:    24
     planes (yes, those dimensions are broken), with font "Bitstream
     Vera Sans Mono-16:dpi=75" antialiased text may bleed out of the
     extent reported by XftTextExtents and other such facilities.  This
     is most obvious with the underscore character in that font.  The
     bottom of the underscore is antialiased, and insertions or
     deletions in the same line before the underscore leave a series of
     "phantom" underlines.  Except that it doesn't happen on the very
     first such insertion or deletion after a window refresh.  A
     similar effect sometimes occurs with deletions at the end of the
     line (no, I can't define "sometimes").  See also comments in
     `redisplay-x.c', functions `x_output_string' and
     `x_output_display_block'.  (Mostly duplicated here.)

     I think this is probably an Xft bug, but I'm not sure.


File: internals.info,  Node: Better Rendering Support -- Configuration with the Interim Patches,  Next: Better Rendering Support -- Modern Font Support,  Prev: Better Rendering Support -- Current Status,  Up: Future Work -- Better Rendering Support

40.24.4 Better Rendering Support - Configuration with the Interim Patches
-------------------------------------------------------------------------

For Stephen's `sjt-xft' branch, you should keep the following in mind
when configuring:

   * Although the menus don't work, it is possible to specify fonts for
     _faces_ using `set-face-font' (and other specifier-changing
     functions).

     There currently is no explicit way to specify that a particular
     font be used only for a given language.  However, since many fonts
     support only a limited repertoire such as ISO 8859/1, you can use
     the precedence of specifications for a given specifier locale to
     get something of this effect for non-Latin character sets.  This
     will normally work rather poorly for multiple Latin character
     sets, however, because the repertoires tend to have large amounts
     of overlap.  Support for specifying font by _language_ as well as
     by character set is planned.

     Because fonts supporting other languages tend to support English as
     well, if you want to use one font for English and another for the
     other language, you must use the `append' method when adding font
     specifications for the _other_ language.

     However, this leaves you with a problem if you want to change the
     other language's font: you have to remove the existing
     specification so it won't shadow the new one when you append.

     I use `define-specifier-tag' like this:

          (define-specifier-tag 'lang-ja)
          ;; No, I don't try to do real work with this font!  But it makes it
          ;; obvious that I got the requested font.  :-)
          (set-face-font 'default "AirCut-14")
          (set-face-font 'default "Kochi Mincho-14" nil '(lang-ja) 'append)
          ;; Oops, too sober.  Try something to match AirCut.
          (set-face-font 'default "Mikachan-14"
                         nil '(lang-ja) 'remove-tag-set-append)

   * The only way to configure widget fonts at the present time is to
     use X resources (or hack the source and rebuild).  Currently
     supported widgets are
        * menubars

        * tab controls

     Here are the resources I use.  *Warning:* _This interface will
     change._  The tab control and menubar have separate Font and
     XftFont resources, and use the X resource manager to instantiate a
     FontStruct from the Font resource.  There is no converter facility
     for XftFont yet, and creating one that handles both FontStruct and
     XftFont depending on XEmacs's configuration and the font name
     seems error-prone at best.  Probably we will should to a simple
     string representation for this resource, and convert to a face in
     XEmacs rather than a font in Xt/Xft.
          XEmacs*Tabs.xftFont: Bitstream Vera Sans-16
          XEmacs*menubar*xftFont: Bitstream Vera Sans-16
          XEmacs.modeline.attributeFont: Bitstream Charter-16
          XEmacs.default.attributeFont: Bitstream Vera Sans Mono-16
     I highly recommend use of a proportional font in the modeline
     because it allows a lot more text to fit there.  (Previously the
     font sizes were quite varied, and there was a comment that this
     weirdness gave good balance.  This isn't true on my main platform,
     Mac OS X, and needs to be rechecked on Linux, where it was
     observed.)


File: internals.info,  Node: Better Rendering Support -- Modern Font Support,  Prev: Better Rendering Support -- Configuration with the Interim Patches,  Up: Future Work -- Better Rendering Support

40.24.5 Better Rendering Support - Modern Font Support
------------------------------------------------------

*NB: This subtree eventually needs to be moved to the Lispref.*

   This chapter describes integration of the `Xft' font support library
into XEmacs.  This library is a layer over the separate `FreeType'
rendering engine and `fontconfig' font query and selection libraries.
`FreeType' provides rendering facilities for modern, good-looking
TrueType fonts with hinting and antialiasing, while `fontconfig'
provides a coherent interface to font query and selection which is
independent of the rendering engine, although currently it is only used
in `Xft' to interface to `FreeType'.

   From the user's point of view, `fontconfig' provides a naming
convention which is _precise_, _accurate_, and _convenient_.  Precision
means that all properties available in the programming API can be
individually specified.  Accuracy means that the truename of the font
is exactly the list of all properties specified by the font.  Thus, the
anomolies that occur with XLFDs on many servers (including modern Linux
distributions with XFree86 or X.org servers) cannot occur.  Convenience
is subjective, of course.  However, `fontconfig' provides a
configuration system which (1) explicitly specifies the defaults and
substitutions that will be made in processing user queries, and (2)
allows the user to specify search configuration, abbreviations,
substitutions, and defaults that override the system's, in the same
format as used by system files.  Further, a "standard minimal
configuration" is defined that ensures that at least _serif_,
_sans-serif_, and _monospace_ fonts are available on all `fontconfig'
systems.

* Menu:

* Modern Font Support -- Font Concepts:: GUI devices, fonts, glyphs, rendering.
* Modern Font Support -- fontconfig::    Querying and selecting fonts.
* Modern Font Support -- Xft::           Rendering fonts on X11.


File: internals.info,  Node: Modern Font Support -- Font Concepts,  Next: Modern Font Support -- fontconfig,  Up: Better Rendering Support -- Modern Font Support

40.24.5.1 Modern Font Support - Font Concepts
.............................................

In modern systems, displays are invariably "raster graphic devices",
which present an abstract interface of "pixel array" where each "pixel
value" is a color, and each pixel is individually mutable, and
(usually) readable.  In XEmacs, such devices are collectively called
"GUI devices", as opposed to "TTY devices" which are character stream
devices but may support control sequences for setting the color of
individual characters, and the insertion position in a rectangular
array.  Here we are concerned only with control of GUI devices but use
TTY devices as a standard for comparison.

   A "font" is an indexed collection of "glyphs", which are
specifications of character shapes.  On a TTY device, these shapes are
entirely abstract, and the index is the identity function.  Typically
fonts are embedded in TTY devices, the user has no control over the font
from within the application, and where choice is available, there is
limited selection, and no extensibility.  Simple, functional, and ...
ugly.

   On GUI devices, the situation is different in every respect.  Glyphs
may be provided by the device, the application, or the user.  Additional
glyphs may be added at will at any of those levels.  Arbitrary index
functions allow the same glyph to be used to display characters in
different languages or using application-specific codes.  Glyphs have
concrete APIs, allowing fine control of rendering parameters, even
user-specified shapes.  To provide convenient, consistent handling of
collections of glyphs, we need a well-defined font API.

   We can separate the necessary properties into two types: properties
which are common to all glyphs in the collection or a property of the
collection itself, and those which are glyph-specific.  Henceforth, the
former are called "font properties" and the latter "glyph properties".

   Font properties include identification like the font "family",
font-wide design parameters like "slant" and "weight", font "metrics"
like "size" (nominal height) and "average width" used for approximate
layout (such as sizing a popup dialog), and properties like the
"default glyph" that are associated with the font for convenient use by
APIs, but aren't really an intrinsic property of the font as a
collection of glyphs.  There may also be a "kerning table" (used to
improve spacing of adjacent glyphs).

   Glyph properties include the "index", glyph metrics such as
"ascent", "descent", "width", "offset" (the offset to the normal
position of the next glyph), "italic correction" (used to improve
spacing when slanted and unslanted glyphs are juxtaposed).  Most
important, of course, is the glyph's shape, which is provided in a
format specific to a rendering engine.  Common formats include bitmaps
(X11 BDF), Postcript programs (Type 1), and collections of spline curves
(TrueType).  When the shape is not itself a bitmap, it must be
"rendered" to a pixmap, either a region on the display or a separate
object which is copied to the display.  In that case, the shape may
include "multiple masters" or "hints" to allow context-specific
rendering which improves the appearance of the glyph on the display.

   Note that this use of "glyph" is mostly independent of the XEmacs
LISP glyph API.  *Note Glyphs::.  It is possible to extract a single
glyph from a font and encapsulate it in Lisp_Glyph object, but the LISP
glyph API allows access to only a very few glyph properties, none of
them related to the rendering process.

   XEmacs LISP does provide an API for selecting and querying fonts, in
the form of a fairly complete set of wrappers for `fontconfig' (*note
Modern Font Support -- fontconfig::).  It also provides some control of
rendering of text via wrappers for `Xft' APIs (*note Modern Font
Support -- Xft::), but this API is quite incomplete.  Also, since the
font selection and query facilities of `Xft' are provided by
`fontconfig', there is some confusion in the API.  For example, use of
antialiasing to improve the appearance of rendered glyphs can be
enabled or disabled.  The API for this is to set the `fontconfig' font
property `antialias' on the font.  However, from the point of view of
`fontconfig' this is merely a hint that the rendering engine may or may
not respect.  This property cannot be used to select only fonts
suitable for being antialiased, for example.  And `rgba' (subpixel
geometry) and `dpi' (pixel density) are conceptually properties of the
display, not of either the font.  They function as hints to the
rendering process.

   As a final confusing touch, `Xft' also provides some access to the
`XRender' extension provided by some modern X servers.  This is mostly
limited to colors, but rectangle APIs are also provided.  These are (of
course) completely independent of fonts, but `Xft' is designed for
client-side font rendering, and thus uses the `XRender' extension
heavily.


File: internals.info,  Node: Modern Font Support -- fontconfig,  Next: Modern Font Support -- Xft,  Prev: Modern Font Support -- Font Concepts,  Up: Better Rendering Support -- Modern Font Support

40.24.5.2 Modern Font Support - fontconfig
..........................................

_Implementation notes_: The functions which initialize the library and
handle memory management (_e.g._, `FcInit' and `FcPatternDestroy') are
intentionally not wrapped (in the latter case, `fc-pattern-destroy' was
provided, but this was ill-considered and will be removed; LISP code
should *never* call this function).  Thinking about some of the
auxiliary constructs used by `fontconfig' is in transition.  The
`FcObjectSet' API has been internalized; it is exposed to LISP as a
list of strings.  The `FcFontSet' API is still in use, but it also will
be internalized, probably as a list (alternatively, vector) of
`Lisp_fc_pattern' objects.  Changing the representation of `fontconfig'
objects (property names) from LISP strings to keywords is under
consideration.

   If `Xft' (including `fontconfig') support is integrated into the
XEmacs build, XEmacs provides the symbol `xft' at initialization.

   XEmacs provides the following functions wrapping the `fontconfig'
library API.

 -- Function: fc-fontset-p object
     Returns t if OBJECT is of type fc-fontset, nil otherwise.  _This
     API is likely to be removed in the near future._

 -- Function: fc-fontset-count fcfontset
     Counts the number of fc pattern objects stored in the fc fontset
     object FCFONTSET.  _This API is likely to be removed in the near
     future._

 -- Function: fc-fontset-ref fcfontset i
     Return the fc pattern object at index I in fc fontset object
     FCFONTSET.  Return nil if the index exceeds the bounds of
     FCFONTSET.  _This API is likely to be removed in the near future._

 -- Function: fc-fontset-destroy fcfontset
     Explicitly deallocate FCFONTSET.  _Do not call this function from
     LISP code.  You will crash.  This API will be removed in the near
     future._

 -- Function: fc-pattern-p object
     Returns t if OBJECT is of type fc-pattern, nil otherwise.

 -- Function: fc-pattern-create
     Return a fresh and empty fc-pattern object.

 -- Function: fc-name-parse name
     Parse string NAME as a fontconfig font name and return its
     representation as a fc pattern object.

 -- Function: fc-name-unparse pattern
     Unparse pattern object PATTERN to a string.

   `Xft''s similar function is actually a different API.  We provide
both for now.  (They probably invoke the same code from `fontconfig'
internally, but the `fontconfig' implementation is more conveniently
called from C.)

 -- Function: xft-name-unparse pattern
     Unparse pattern object PATTERN to a string (using the `Xft' API).

 -- Function: fc-pattern-duplicate pattern
     Make a copy of pattern object PATTERN and return it.

 -- Function: fc-pattern-add pattern property value
     Add attributes to the pattern object PATTERN.  PROPERTY is a
     string naming the attribute to add, VALUE the value for this
     attribute.

     VALUE may be a string, integer, float, or symbol, in which case
     the value will be added as an FcChar8[], int, double, or FcBool
     respectively.

 -- Function: fc-pattern-del pattern, property
     Remove attribute PROPERTY from pattern object PATTERN.

   This is the generic interface to `FcPatternGet'.  We don't support
the losing symbol-for-property interface.  However, it might be a very
good idea to use keywords for property names in LISP.

 -- Function: fc-pattern-get pattern property &optional id type
     From PATTERN, extract PROPERTY for the ID'th member, of type TYPE.

     PATTERN is an `Xft' (`fontconfig') pattern object.  PROPERTY is a
     string naming a `fontconfig' font property.  Optional ID is a
     nonnegative integer indexing the list of values for PROPERTY
     stored in PATTERN, defaulting to 0 (the first value).  Optional
     TYPE is a symbol, one of 'string, 'boolean, 'integer, 'float,
     'double, 'matrix, 'charset, or 'void, corresponding to the FcValue
     types.  ('float is an alias for 'double).

     Symbols with names of the form `fc-result-DESCRIPTION' are
     returned when the desired value is not available.  These are

          fc-result-type-mismatch   the value found has an unexpected type
          fc-result-no-match        there is no such attribute
          fc-result-no-id           there is no value for the requested ID

   The Lisp types returned will conform to TYPE:

     string          string
     boolean         `t' or `nil'
     integer         integer
     double (float)  float
     matrix          not implemented
     charset         not implemented
     void            not implemented

   The types of the following standard properties are predefined by
fontconfig.  The symbol 'fc-result-type-mismatch will be returned if the
object exists but TYPE does not match the predefined type.  It is best
not to specify a type for predefined properties, as a mistake here
ensures error returns on the correct type.

   Each standard property has a convenience accessor defined in
`fontconfig.el', named in the form `fc-pattern-get-PROPERTY'.  The
convenience functions are preferred to `fc-pattern-get' since a typo in
the string naming a property will result in a silent null return, while
a typo in a function name will usually result in a compiler or runtime
\"not fboundp\" error.  You may use `defsubst' to define convenience
functions for non-standard properties.

     family         String  Font family name
     style          String  Font style. Overrides weight and slant
     slant          Int     Italic, oblique or roman
     weight         Int     Light, medium, demibold, bold or black
     size           Double  Point size
     aspect         Double  Stretches glyphs horizontally before hinting
     pixelsize      Double  Pixel size
     spacing        Int     Proportional, monospace or charcell
     foundry        String  Font foundry name
     antialias      Bool    Whether glyphs can be antialiased
     hinting        Bool    Whether the rasterizer should use hinting
     verticallayout Bool    Use vertical layout
     autohint       Bool    Use autohinter instead of normal hinter
     globaladvance  Bool    Use font global advance data
     file           String  The filename holding the font
     index          Int     The index of the font within the file
     ftface         FT_Face Use the specified FreeType face object
     rasterizer     String  Which rasterizer is in use
     outline        Bool    Whether the glyphs are outlines
     scalable       Bool    Whether glyphs can be scaled
     scale          Double  Scale factor for point->pixel conversions
     dpi            Double  Target dots per inch
     rgba           Int     unknown, rgb, bgr, vrgb, vbgr, none - subpixel geometry
     minspace       Bool    Eliminate leading from line spacing
     charset        CharSet Unicode chars encoded by the font
     lang           String  List of RFC-3066-style languages this font supports

   The FT_Face, Matrix, CharSet types are unimplemented, so the
corresponding properties are not accessible from Lisp at this time.  If
the value of a property returned has type FT_Face, FcCharSet, or
FcMatrix, `fc-result-type-mismatch' is returned.

   The following properties which were standard in `Xft' v.1 are
obsolete in `Xft' v.2: `encoding', `charwidth', `charheight', `core',
and `render'.

 -- Function: fc-pattern-destroy pattern
     Explicitly deallocate pattern object PATTERN.  _Do not call this
     function from LISP code.  You will crash.  This API will be removed
     in the near future._

 -- Function: fc-font-match device pattern
     Return the font on DEVICE that most closely matches PATTERN.

     PATTERN is a `fontconfig' pattern object.  DEVICE is an X11
     device.  Returns a `fontconfig' pattern object representing the
     closest match to the given pattern, or an error code.  Possible
     error codes are `fc-result-no-match' and `fc-result-no-id'.

 -- Function: fc-list-fonts-pattern-objects device pattern properties
     List the fonts on DEVICE that match PATTERN for PROPERTIES.
     DEVICE is an X11 device.  PATTERN is a `fontconfig' pattern to be
     matched.  PROPERTIES is the list of property names (strings) that
     should be included in each returned pattern.  The result is a
     `fontconfig' fontset object containing the set of unique matching
     patterns.

   The PROPERTIES argument does not affect the matching.  So, for
example,

     (mapcar #'fc-name-unparse
       (let ((xfl (fc-list-fonts-pattern-objects nil
                   (fc-name-parse "FreeMono") '("style")))
             (i 0)
             (fl nil))
         (while (< i (fc-fontset-count xfl))
           (push (fc-fontset-ref xfl i) fl)
           (setq i (1+ i)))
         fl))

   will return something like `(":style=Bold" ":style=Medium"
":style=Oblique" ":style=BoldOblique")' if you have the FreeFont
package installed.  Note that the sets of objects in the target pattern
and the returned patterns don't even intersect.

   In using `fc-list-fonts-pattern-objects', be careful that only
intrinsic properties of fonts be included in the pattern.  Those
properties included in the pattern must be matched, or the candidate
font will be eliminated from the list.  When a font leaves a property
unspecified, it is considered to be a mismatch for any pattern with that
property specified.  Thus, inclusion of extraneous properties will
result in the list being empty.  Note that for scalable fonts (at
least), `size' is not an intrinsic property!  Thus a specification such
as `"Bitstream Vera Sans-12"' will return an empty list regardless of
whether the font is available or not--probably not what you (as
programmer or user) want.

   The list is unsorted.  In particular, the pattern
`":style=italic,oblique"' will not return italic fonts first, then
oblique ones.  The fonts will be returned in some arbitrary order.

   _Implementation notes_:  Fontset objects are slated for removal from
the API.  In the future `fc-list-fonts-pattern-objects' will return a
list.  The DEVICE argument is unused, ignored, and may be removed if
it's not needed to match other font-listing APIs.  This name will be
changed to correspond to Ben's new nomenclature, probably simply
`fc-font-list'.

 -- Function: fc-font-sort device pattern trim
     Return a fontset object listing all fonts sorted by proximity to
     PATTERN.  DEVICE is an X11 device.  PATTERN is a fontconfig
     pattern to be matched.  Optional argument TRIM, if non-nil, means
     to trim trailing fonts that do not contribute new characters to
     the union repertoire.

   _Implementation notes_:  Fontset objects are slated for removal from
the API.  In the future `fc-font-sort' will return a list (or perhaps a
vector) of FcPatterns.  The DEVICE argument is unused, ignored, and may
be removed if it's not needed to match other font-listing APIs.

 -- Function: fc-font-real-pattern fontname xdevice
     Temporarily open font FONTNAME (a string) on device XDEVICE and
     return the actual fc pattern matched by the Fc library.  _This
     function doesn't make much sense and will be removed from the API._

 -- Function: xlfd-font-name-p fontname
     Check whether string FONTNAME is a XLFD font name.

 -- Variable: xft-debug-level
     Level of debugging messages to issue to stderr for XFT.  A
     nonnegative integer.  Set to 0 to suppress all warnings.  Default
     is 1 to ensure a minimum of debugging output at initialization.
     Higher levels give more information.

 -- Variable: xft-version
     The major version number of the Xft library compiled with.

 -- Variable: xft-xlfd-font-regexp
     Regular expression matching XLFD font names.


File: internals.info,  Node: Modern Font Support -- Xft,  Prev: Modern Font Support -- fontconfig,  Up: Better Rendering Support -- Modern Font Support

40.24.5.3 Modern Font Support - fontconfig
..........................................

IIRC, we don't really provide any `Xft' APIs at the LISP level yet.


File: internals.info,  Node: Future Work Discussion,  Next: Old Future Work,  Prev: Future Work,  Up: Top

41 Future Work Discussion
*************************

This chapter includes (mostly) email discussions about particular design
issues, edited to include only relevant and useful stuff.  Ideally over
time these could be condensed down to a single design document to go
into the normal Future Work section.

* Menu:

* Discussion -- Garbage Collection::
* Discussion -- Glyphs::
* Discussion -- Dialog Boxes::
* Discussion -- Multilingual Issues::
* Discussion -- Instantiators and Generic Property Accessors::
* Discussion -- Switching to C++::
* Discussion -- Windows External Widget::
* Discussion -- Packages::
* Discussion -- Distribution Layout::


File: internals.info,  Node: Discussion -- Garbage Collection,  Next: Discussion -- Glyphs,  Prev: Future Work Discussion,  Up: Future Work Discussion

41.1 Discussion - Garbage Collection
====================================

* Menu:

* Discussion -- Pure Space::
* Discussion -- Hashtable-Based Marking and Cleanup::
* Discussion -- The Anti-Cons::


File: internals.info,  Node: Discussion -- Pure Space,  Next: Discussion -- Hashtable-Based Marking and Cleanup,  Prev: Discussion -- Garbage Collection,  Up: Discussion -- Garbage Collection

41.1.1 Discussion - Pure Space
------------------------------

On Tue, Oct 12, 1999 at 03:36:59AM -0700, Ben Wing wrote:

   So what am I missing here?

   In response, Olivier Galibert wrote:

   Two things:
  1. The purespace is gone

     I  mean  absolutely, completely and utterly  removed.   Fpurecopy
     is a no-op now (and  have been for  some time).  Readonly objects
     are gone too.  Having  less checks to  do in Fsetcar,  Fsetcdr,
     Faset and some others  is probably a  good thing, speedwise.  I
     have it removed some time ago because it  does not make  sense
     when using a portable dumper to copy data in a special area of the
     memory at dump time and I wanted to be  sure that supressing the
     copying  from Fpurecopy wouldn't break things.

     Now, we want to get the post-dumping data sharing back, of course.
     In today systems,  it is  quite   easy: you just   have  to map
     the  file MAP_PRIVATE and avoid writing to the subset of  pages
     you want to keep shared.   Copy-on-write does  the job for  you.
     It  has the nice side effect of  completely avoiding bus  errors
     due  to trying to  write to readonly memory zones.

     Avoiding writing to the "pure" objects themselves  is already
     done, of course.  Would lisp code  have written to the  purecopied
     parts of the dumped data that it would have exploded long ago.  So
     there is nothing to do in this area.  So the only remaining thing
     is  the markbit.  Two possible strategies:

        * have Fpurecopy mark somehow the lrecords it would have copied
          in the good old times.  Post-dump, use this mark as a "always
          marked, don't touch, don't look  into, don't free"  flag, the
          same way CHECK_PURE was used.

        * move the markbit outside of the lrecord.

     The second solution is more appealing to me for a bunch of reasons:
        * more things are shared  than only what  is purecopied (not
          yet  used functions come to mind)

        * no more "the only references to this  non-purecopied object
          are from purecopied objects, XEmacs will self-destruct  in
          ten seconds"  kind of bugs.

        * removing flags  goes   the right   way towards   implementing
          Jan's allocator ideas.

        * it becomes probably easier to experiment with the GC code

  2. Finding all the dumped objects in order to unmark them sucks

     Not  having to rebuild  a list of all the  dumped objects  in
     order to find  them all and ensure that  all are unmarked
     simplifies things for me.  Errr, ok, now that I really think of
     it, I can rebuild this list easily, in fact.  And I'm probably
     going to have to manage it, since I feel like the  lack of calls
     to  the finalizers for the dumped objects is going to someday
     turn over and bite me  in the face.  But anyways, it makes my life
     easier for now.

     So no,  it's  not a _necessity_.   But  it  helps.  And the
     automatic sharing of  all objects until  you write  to   them
     explicitely is,  I think, really cool.


File: internals.info,  Node: Discussion -- Hashtable-Based Marking and Cleanup,  Next: Discussion -- The Anti-Cons,  Prev: Discussion -- Pure Space,  Up: Discussion -- Garbage Collection

41.1.2 Discussion - Hashtable-Based Marking and Cleanup
-------------------------------------------------------

On 10/12/1999 5:49 PM Ben Wing wrote:

   OK, I can see the advantages.  But:

  1. There will be an inevitable loss of speed using a large hashtable.
     If it's large, I say that it's just not worth it.  There are
     things that are so much more important than futzing around with
     the garbage collector (e.g. fixing the god damn user interface),
     things which if not fixed will sooner or later cause XEmacs to die
     entirely.  If we are causing a major slowdown in the name of some
     not-so-important work that may or may not get done, we shouldn't
     do it.  (On the other hand, if the slowdown is negligible, I have
     no problems with this.)

  2. I think you should *expand* the concept of read-only objects so
     that *any* object (especially strings and cons cells) can get
     marked read-only by the C code if it wants. (Perhaps you could use
     the now-unused mark bit to hold a read-only flag.) This is
     important because it allows C code to directly return internal
     lists (e.g. from the specifiers and various object property lists)
     without having to do a copy, like is now done (and similarly,
     potentially to directly accept lists from a Lisp call without
     copying them for internal use, if the Lisp caller is made aware
     that the list might become read-only) - if the copy weren't done
     and some piece of Lisp code went and modified the list, XEmacs
     might very well crash.  Thus, this read-only flag would be a huge
     efficiency gain in terms of the garbage collection overhead saved
     as well as the speed of copying a large list.  The extra checks in
     `Fsetcar()', etc. for this that you mention are in fact negligible
     in their speed overhead - one or two instructions - and these
     functions are not used all that commonly, either.  With the
     changes I have proposed in Architecting XEmacs, the case of
     returning an internal list will become more and more common as the
     power of the user interface would be greatly increased and along
     with it are lots and lots of lists of info that need to be
     retrievable from Lisp.

   BTW there is a wonderful book all about garbage collection by Jones
and Lins.  Ever seen it?

     http://www.amazon.com/exec/obidos/ASIN/0471941484/qid=939775572/sr=1-1/002-3092633-2509405


File: internals.info,  Node: Discussion -- The Anti-Cons,  Prev: Discussion -- Hashtable-Based Marking and Cleanup,  Up: Discussion -- Garbage Collection

41.1.3 Discussion - The Anti-Cons
---------------------------------

From: "Ben Wing" <ben@666.com> Date: Tue, 14 May 2002 06:48:09 -0700

   i was thinking about the proliferating types of weak hash tables -
e.g. now we have "key-car-value weak" hash tables due to a need in the
glyphs code.  i realized there should be a general solution, that lets
you control exactly how the weakness of such hash tables work.

   and, assuming we implement a simple "reference" type, a simple
container whose object is a weak reference and thus gets converted to
nil (and a flag set on the reference) when the object is collected, it
would be useful for more precisely controlling the reference, too.

   it's called an "anti-cons".  it behaves somewhat like a cons in that
it boxes two items, but its marking properties are very different - in
fact, backwards.  normally, a cons, if marked, marks its children.  in
this case, if the children of an anti-cons are marked, it marks itself!
you'd need a few different kinds of anti-cons - probably the following:

     and [marks itself if both children marked]
     or [...]
     left [marks itself if left is marked, and then marks the right]
     right [...]
     not-left
     not-right

   by putting such an object inside of a weak reference - e.g. in a weak
hash table - we can set up a tree of arbitrary complexity which
implements any boolean formula of markedness over any number of
objects.  this would easily handle key-car, and key-cadr, and
key-car-or-cdr, and key-((caar or cadr) and cdr) etc. etc.

   implementing this in the current xemacs framework is mostly trivial.

   michael, would such an object get in the way of your new gc?

   From: sperber@informatik.uni-tuebingen.de (Michael Sperber [Mr.
Preprocessor]) Date: Tue, 14 May 2002 16:04:01 +0200

   You might want to look at

   http://research.microsoft.com/Users/simonpj/Papers/weak.htm

   for a pretty comprehensive survey of what you could want in terms of
weakness.  Its weak pointers are very similar to your anti-cons.
However, there are some problems in doing the same in a Lisp settings,
mainly because of symbols.  I intend to elaborate on this next week;
this week is full, unfortunately.

   Ben> implementing this in the current xemacs framework is mostly
Ben> trivial.

   Ben> michael, would such an object get in the way of your new gc?

   Well, our first commit will be an implementation of vanilla weak
boxes (ready within the next few days, I hope), and we'll then try to
replace most other instances of weakness with uses of those.  We'll
then try to find a more general solution for the rest.  (Richard
Reingruber has already done a comprehensive survey of the trouble spot.

   Can you wait until next week?  I'll try to come up with a battle plan
then.

   From: sperber@informatik.uni-tuebingen.de (Michael Sperber [Mr.
Preprocessor]) Date: Tue, 28 May 2002 16:14:20 +0200

   We've now started implementing ephemerons as a building block for the
more involved weakness-involving data structures:

   The relevant reference is

   Barry Hayes. Ephemerons: A New Finalization Mechanism.  OOPSLA 1997.
176-183

   The idea is this:

   an ephemeron consists of a key and a value.  Through the ephemeron,
the key is not reachable.  The value is only reachable if both the
ephemeron is reachable and the key is reachable.  If the ephemeron is
reachable and the key becomes unreachable, the value slot of the
ephemeron will be tombstoned, i.e. overwritten with NIL or something.

   This allows implementing, AFAICS, the other data structures involving
weakness, such as weak hash tables and their various mutants.

   We're also planning to come up with a more comprehensive solution for
finalization, but some design snags remain to be worked out.


File: internals.info,  Node: Discussion -- Glyphs,  Next: Discussion -- Dialog Boxes,  Prev: Discussion -- Garbage Collection,  Up: Future Work Discussion

41.2 Discussion - Glyphs
========================

Some comments (not always pretty!) by Ben:

   March 20, 2000

   Andy, I use the tab widgets but I've been having lots of problems.

   1] Sometimes clicking on them does nothing.

   2] There's a design flaw: I frequently use M-C-l to switch to the
previous buffer.  If I use this in conjunction with the tabs, things get
all screwed up because selecting a buffer with the tab does not bring it
to the front of the buffer list, like it should.  It looks like you're
doing this to avoid having the order of the tabs change, but this is
wrong: If you don't reorder the buffer list, everything else gets
screwed up.  If you want the order of the tabs not to change, you need
to decouple this order from the buffer list order.

   March 23, 2000

   I'm very confused.  The SIGIO timer is used *only* for C-g.  It has
nothing to do with any other events.  (sit-for 0) ought to

   (1) cause all pending non-command events to get executed, and (b) do
redisplay

   However, sit-for gets preempted by input coming in.

   What about (sit-for 0.1)?

   I suppose a solution along the lines of dispatch-non-command-events
might be OK if you've tried everything else and it doesn't work, but i'm
leery of introducing new Lisp functions to deal with specific problems.
Pretty soon we end up with a whole bevy of such ill-defined functions,
like we already have.  I think instead, you should introduce the
following primitive:

     (wait-for-event redisplay &rest event-specs)

   Waits for one of the event specifications specified to happen.
Returns something about what happened.

   REDISPLAY controls the behavior of redisplay during waiting.
Something like

   * nil (never redisplay),

   * t (redisplay when it seems appropriate), etc.

   EVENT-SPECS could be

     t                     -- drain all non-user events, and then return
     any-process           -- wait till input or state change on any process
     process               -- wait till input or state change on process
     time                  -- wait till such-and-such time has elapsed
     'user                 -- wait till user event has happened
     '(user predicate)     -- wait till user event matching the predicate has
                              happened
     'event                -- wait till any event has happened
     '(event predicate)    -- wait till event matching the predicate has happened

   The existing functions `next-event', `next-command-event',
`accept-process-output', `sit-for', `sleep-for', etc. could all be
written in terms of this new command.  You could use this command inside
of your glyph code to ensure that the events get processed that need do
in order for widget updates to happen.

   But you said something about need a magic event to invoke redisplay?
Why is that?

   April 2, 2000

   the internal distinction between "widget" and "layout" is bogus.
there exist widgets that do drawing and do layout of their children,
e.g. group-box widgets and proper tab widgets.  the only sensible
distinction is between widgets with children and those without children.

   April 5, 2000

   andy, i'm not sure i really believe that you need to cycle the event
code to get widgets to redisplay, but in any case you should

  1. hide the logic to do this in the c code; the lisp code should do
     nothing other than call (redisplay widget)

  2. make sure your event-cycling code processes *NO* events at all.
     this includes non-user events.  queue the events instead.

   in other words, dispatch-non-command-events must go, and i am
proposing a general function (redisplay OBJECT) to replace the existing
ad-hoc functions.

   April 6, 2000

   the tab widget code should simply be able to create a whole lot of
tabs without regard to the size of the gutter, and the surrounding
layout widget (please please make layouts be proper widgets!) should
automatically map and unmap them as necessary, to fill up the available
space.  perhaps this already works and what you're doing is just for
optimization?  but i get the feeling this is not the case.

   April 6, 2000

   the function make-gutter-only-dialog-frame is bogus.  the use of the
gutter here to hold widgets is an implementation detail and should not
be exposed in the interface.  similarly, make-search-dialog should not
have to do all the futzing that it does.  creating the frame unmapped,
creating an extent and messing with the gutter: all this stuff should be
hidden.  you should have a simple function make-dialog-frame that takes
a dialog specification, and that's all you need to do.

   also, these dialog boxes, and this function make-dialog-frame, should

  1. be in `dialog.el', not gutter-items.el.

  2. when possible, be placed in the interactive spec of standard lisp
     functions rather than accessed directly from `menubar-items.el'

  3. wrapped in calls to should-use-dialog-box-p, so the user has
     control over when dialog boxes appear.

   April 7, 2000

   hmmm ...  in that case, the whitespace absolutely needs to be
specified as properties of the layout widget (e.g. :border-width and
:border-height), rather than setting an overall size.  you have no idea
what the correct size should be if the user changes font size or uses
translations in a different language.

   Your modus operandi should be "hardcoded pixel sizes are *always*
bad."

   April 7, 2000

   you mean the number of tabs adjusts, or the size of each tab adjusts
(by making the font smaller or something)?  if the size of a single tab
is not related to the total space the tabs can fix into, then it should
be possible to simply specify as many tabs as exist for buffers, and
have the layout manager decide how many can fit into the available
space.  this does *not* mean the layout manager will resize the tabs,
because query-geometry on the tabs should find out that the tabs don't
want to be any size other than they are.

   the point here is that you should not *have* to worry about pixel
heights and widths *anywhere* in Lisp-level code.  The layout managers
should take care of everything for you.  The only exceptions may be in
some text fields, which will be blank by default and you want to specify
a maximum width (which should be done in 'n' sizes, not in pixels!).

   i won't stop complaining until i see nearly every one of those
pixel-width and pixel-height parameters gone, and the remaining ones
there for a very, very good reason.

   April 7, 2000

   Andy Piper wrote:

     > At 03:51 PM 4/6/00 -0700, Ben Wing wrote:
     > >[the function make-gutter-only-dialog-frame is bogus]
     >
     > The problem is that some of the callbacks and such need access to the
     > *created* frame, so you end up in a catch 22 unless you do what I've done.

   [Ben proposes other ways to avoid exposing all the guts, as in
`make-gutter-only-dialog-frame':]

  1. Instead of passing in the actual glyph spec or glyph, pass in a
     function of two args (the dialog frame and its parents), which when
     called, creates and returns the appropriate glyph.

  2. [Better] Provide a way for callbacks to determine where they were
     invoked at.  This is much more general and is what you should
     really do.  For example, have the code that calls the callbacks
     bind some global variables such as widget-callback-current-glyph
     and widget-callback-current-channel, which contain the glyph whose
     callback is being invoked, and the window or frame of the glyph
     (depending on where the glyph is) where the invocation actually
     happened.  That way, the callbacks can easily figure out the dialog
     box and its parent, and not have to worry about embedding it in at
     creation time.

   April 15, 2000 I don't understand when you say "the various types of
callback".  Are you using the callback for various different purposes?

   Your widget callbacks should work just like any other callback: they
take two arguments, one indicating the object to which the callback was
attached (an image instance, i think), and the event that caused the
callback to be invoked.

   April 17, 2000

   I am completely vetoing widget-callback-current-channel.  How about
you create a new keyword, :new-callback, that is a function of two args,
like i specified before.

   btw if you really are calling your callback using call-interactively,
why don't you declare a function (interactive "e") and then call
event-channel on the resulting event?  that should get you the same
result as widget-callback-current-channel.

   the problem with this and everything you've proposed is that there's
no way, of course, to get at the actual widget that you were invoked
from.  would you propose adding widget-callback-current-widget?


File: internals.info,  Node: Discussion -- Dialog Boxes,  Next: Discussion -- Multilingual Issues,  Prev: Discussion -- Glyphs,  Up: Future Work Discussion

41.3 Discussion - Dialog Boxes
==============================

     From:
             Ben Wing <ben@666.com>
                                                           10/7/1999 5:57 PM

      Subject:
             Re: Animated gif patch (2)
          To:
             Andy Piper <andy@xemacs.org>
         CC:
             xemacs-review@xemacs.org, xemacs-beta@xemacs.org




     The distinction between layouts and widgets makes no sense, so you should combine
     the different data required.  Consider a grouping widget.  Is this a layout or a
     widget?  It draws, like a widget, but has children, like a layout.  Same for a tab
     widget, properly implemented.  It draws, handles input, has children, and makes
     choices about how to lay them out.

     ben

     From:
             Ben Wing <ben@666.com>
                                                            9/7/1999 8:50 PM

      Subject:
             Re: Layouts done
          To:
             Andy Piper <andyp@beasys.com>




     this sounds great!  where can i see the code?

     as for user-defined layouts, you must certainly have some sort of abstraction
     layer for layouts, with DEFINE_LAYOUT_TYPE or something similar just like device
     types and such.  If not, you should certainly make one ...  it would have methods
     such as query-geometry and do-layout.  It should be easy to create a user-defined
     layout if you have such an abstraction.

     with a user-defined layout, complex built-in layouts such as grid should not be
     necessary because it's so easy to write snippets of lisp.

     as for the "redisplay too much" problem, perhaps you could put a dirty flag in
     each glyph indicating whether it needs to be redisplayed, recalculated, etc.?

     Andy Piper wrote:

     > You may want to check them out. I haven't done the user-defined layout
     > callback - I'm not sure what sort of API this could have.  Keywords I've done:
     >
     > :orientation - vertical or horizontal
     > :justify - left, center or right
     > :border - etch-in, etch-out, bevel-in, bevel -out or text (which gives you
     > etch-in with a title)
     >
     > You can embed any glyph type in a layout.
     >
     > There is probably room for improvements for justify to do grid-type layouts
     > as per java.
     >
     > The only annoying thing is that I've hacked up font-lock support to do a
     > progress gauge in the gutter area. I've used a layout to set things out
     > correctly. The problem is if you change one of the sub-widgets, the whole
     > layout gets redisplayed because it is treated as a single glyph by redisplay.
     >
     > Oh, and I've done line based scrolling so that glyphs scroll off the page
     > in units of the average display line height rather than the whole line at
     > once. This could easily be converted to pixel scrolling but would be very
     > slow I fear.
     >
     > andy
     > --------------------------------------------------------------
     > Dr Andy Piper
     > Senior Consultant Architect, BEA Systems Ltd




     From:
             Ben Wing <ben@666.com>
                                                          8/10/1999 11:11 PM

      Subject:
             Re: Widgets
          To:
             Andy Piper <andy@xemacs.org>




     I think you might have misinterpreted what i meant.  I meant to say that XEmacs should
     implement the *concept* of a hierarchy of nested child "widgets" or "gui items" or
     whatever we want to call them -- this includes container "widgets" such as grouping
     widgets (which draw a border around the children, like in Windows), tab widgets, simple
     layout widgets (invisible, but lay out their children appropriately), etc, plus leaf
     "widgets" (buttons, sliders, etc., also standard Emacs windows).  The layout calculations
     for these widgets would be handled entirely by XEmacs in a window-system-independent way.
     There is no need to create a corresponding hierarchy of window-system
     widgets/controls/whatever if it's not required, and certainly no need to try to use the
     window-system-supplied geometry management routines.  It's absolutely necessary to support
     this nesting concept in XEmacs, however, or it's impossible to have easily-designable
     dialog boxes.  On the other hand, I think it *is* required to create much of this
     hierarchy within the actual window system, at the very least for non-invisible container
     widgets (tab, grouping, etc.), otherwise we will have very bogus, non-native-looking
     containers like your current tab-widget implementation.  It's critical for XEmacs to be
     able to create dialog boxes in Windows or Motif that look just like those in any other
     standard application.  Otherwise people will continue to think that XEmacs is a
     backwards-looking, badly implemented piece of software, which in many ways it is,
     particularly in regards to its user interface.

     Perhaps we should talk on the phone?  This typing is quite hard for me still.  What hours
     are you at work?  My hours are approx. 2pm - 2am Pacific time (GMT - 7 hours currently).

     ben


     From:
             Ben Wing <ben@666.com>
                                                          7/21/1999 2:44 AM

      Subject:
             Re: Tabs 'n widgets screenshot
          To:
             Andy Piper <andy@xemacs.org>
         CC:
             xemacs-beta@xemacs.org, wmperry@aventail.com




     This is real cool, but looking at this, it's clear that it doesn't look the
     way tab widgets are supposed to work.  In particular, of course, they should
     have the proper borders around the stuff displayed.  I've attached a screen
     shot of a typical Windows dialog box with a tab widget in it.  The problem
     lies with this "expanded gutter" concept.  Tabs are *NOT* extra graphical junk
     placed in the gutters of a buffer but are GUI objects with *children* inside
     of them.  This is the right way to do things, and you would need no extra
     gutter functionality at all for this.  You just need to implement the concept
     of GUI objects containing other GUI objects within them.  One such GUI object
     needs to be a "Emacs-text" GUI object, which is an Emacs window and contains a
     buffer within it.  At this level, you need not be concerned with the
     complexities of geometry layout.  The only change that needs to be made in the
     overall strategy of frames, windows, etc. is that windows need not be exactly
     contiguous and tiled, as long as they are contained within a frame.  Or more
     specifically: Given that you could always split a window contained inside a
     GUI object, we just need to expand things so that each frame has *multiple*
     hierarchies of windows in it, rather than just one.  A hierarchy of windows
     can nest inside of another window -- e.g. I put a tab widget or a text widget
     inside of a buffer.  This should be easy to implement -- just change things so
     there are multiple hierarchies of windows where there are one, each (except
     the top-level one) being rooted inside some other window.

     Anyone willing to implement this? Andy?


     From:
             Ben Wing <ben@666.com>
                                                           6/30/1999 3:30 PM

      Subject:
             Re: Focus Help!
          To:
             Andy Piper <andy@xemacs.org>
         CC:
             Ben Wing <ben@xemacs.org>, martin@xemacs.org, andyp@beasys.com




     It sounds like you're doing very good work.  It also sounds like the approach
     you have followed is the correct one.  Now, it seems like there isn't really
     that much work left to get dialog boxes working.  What you really just need to
     do is implement container widgets, that is to say, subwindows that can contain
     other subwindows.  For example, the tab widget works this way. (It sounds like
     you have already implemented tab widgets, so I don't quite see how you've done
     this without the concept of container widgets.) So you might just try adding a
     framework for container widgets and then implementing very simple container
     widgets.  The basic container widgets are:

     1. A vertical-layout widget, which draws nothing itself and lays out its
     children one above the next.
     2. A horizontal-layout widget, which draws nothing itself and lays out its
     children side-to-side.
     3. A box (or "grouping") widget, which draws a rectangle around its single child
     and optionally draws some text on the top or bottom line of the rectangle.
     4. A tab widget, which displays a series of tabs horizontally at the top of its
     area, and then below it places one of its children,
     corresponding to the selected tab.
     5. A user widget, which draws nothing itself and does no layout at all on its
     children, except that it has a "layout callback"
     property, a Lisp function, so that the programmer can control the layout.

     The framework is as follows:

     1. Every widget has at least the following properties:
          a) a size, whose value can be "unspecified", which might be implemented
     using the value -1.  The default value should be "unspecified".
          b) whether it's mapped, i.e. whether it will be displayed. (Some container
     widgets, such as the tab widget, set the mapped
     property themselves on their children.  Others, such as the vertical and
     horizontal layout widgets, don't change this property but pay attention to it,
     and ignore completely all children marked as unmapped.) The default value should
     be "true".
          c) whether its size can be changed by another widget's layout routine. The
     default value should be "true".
          d) a layout procedure, which (potentially at least) determines the size of
     the widget as well as the position, size and mappedness of its child widgets.
     The layout procedure is inherent in the widget and is not an external property
     of the widget (except in the case of the "user widget"): it is instead more like
     the redisplay callback that each widget has.
     2. Every container widget contains a property which is a list of child widgets.
     3. Every child widget contains the following properties:
          a) a position indicating where the child is located relative to the top
     left corner of its parent.  The position's value can be "unspecified", which
     might be implemented using the value -1.  The default value should be
     "unspecified".
          b) whether its position can be changed by another widget's layout routine.
     The default value should be "true".
     4. All of the properties just listed (except possibly the layout procedure) can
     be modified directly by the programmer, and there are no proscriptions against
     doing so.  However, if the programmer wants to resize, reposition, map or unmap
     a widget in such a way that the layout of all the other widgets in the tree
     changes appropriately, he should use a special function to change the property,
     as described below.

     The redisplay mechanism pays attention to the position, size, and mappedness
     properties and to the hierarchy of widgets, mapping, resizing and repositioning
     the corresponding subwindows (the "real representation" of the widgets) as
     necessary.  It also pays attention to the hierarchy of the widgets, making sure
     that container subwindows get drawn before their child subwindows.  When it
     encounters widgets with an unspecified size, it should not draw them, and should
     issue a warning.  When it encounters widgets with an unspecified position, it
     should draw them at position (0, 0) and should issue a warning.

     The above framework should be fairly simple to implement and is basically
     universal across all high-level windowing system toolkits.  The stickyness comes
     with what procedures you follow for getting the layout done.

     Andy, I understand that implementing this may seem like a daunting task.
     Therefore, I propose that at first you implement the above framework but don't
     implement any of the layout procedures, or any of the functions that call them:
     Just make them stubs that do nothing.  This way, the Lisp programmer can still
     create any dialog boxes he wants, he just has to set the sizes and positions of
     all the widgets explicitly, and then recompute them whenever the widget tree is
     resized (once you get around to allowing this).  I have a lot more to write
     about exactly how the layout procedures work, but I'll send that to you later
     once you're ready.

     You should also think about making a way to have widget trees as top-level
     windows rather than just glyphs in a buffer.  There's already the concept of
     "popup" frames.  You could provide an easy way to create a popup frame with no
     menu, toolbars, scrollbars, modeline or minibuffer, and put a single glyph in
     the displayed buffer that takes up the whole Emacs window.

     Ben




     March 20, 2000

     You wrote to me awhile ago about this and asked about documentation, and I
     dictated a response but never got it sent, so here it is:

     I don't think there's any more documentation on how things work under Xt but it
     should be clear. The EmacsFrame widget is the widget corresponding to the X
     window that Emacs draws into and there is a handler for expose events called
     from Xt which arranges for the invalidated areas to get redrawn. I think this
     used to happen as part of the handler itself but now it is delayed until the
     next call to redisplay.

     However, one thing that you absolutely must not do is remove the Xt support.
     This would be an incredibly unfriendly thing to do as it would prevent people
     from using any widget set other than Qt or GTK. Keep in mind that people run
     XEmacs on all sorts of different versions of X in Unix, and Xt is the standard
     and the only toolkit that probably exists on all of these systems.

     Pardon me if I've misunderstood your intentions w.r.t. this.

     As for how you would implement GTK support, it will not be very hard to convert
     redisplay to draw into a GTK window instead of an Xt window. In fact redisplay
     basically doesn't know about Xt at all, except in the portion that handles
     updating menubars and scrollbars and stuff that's directly related to Xt.

     What you'd probably want to do is create a new set of event routines to replace
     the ones in event-Xt.c. On the display side you could conceivably create a new
     device type but you probably wouldn't want to do that because it would be an
     externally visible change at the Lisp level. You might simply want to put a
     flag on each frame indicating what sort of toolkit the frame was created under
     and put conditions in the redisplay code and the code to update toolbars and
     menubars and so forth to test this flag and do the appropriate thing.


     April 12, 2000

     This is way cool, buuuuutttttttt .............

     what we *really* need is the GUI interface on top of it.  I've taken a shot at
     it with generic-print-buffer
     (print-buffer is taken by lpr, which is such a total mess that it needs to be
     trashed; or at least, the generic
     stuff in this package needs to be taken out and properly genericized).  For
     the moment, generic-print-buffer
     just does something like what Kirill's been posting if we're running windows,
     and uses lpr otherwards.  However, what we absofuckinglutely need is a Lisp
     interface onto `EnumPrinters()' so that we can get the
     list of printers and have a nice menu listing the available printers, and you
     can check the one you want.  People in the Windows world don't normally even
     know the names of their local printers!

     Kirill, given what I've done in `simple.el' and `menubar-items.el', do you think
     you could add the `EnumPrinters()'
     support and fix up the GUI?  If you don't feel comfortable with the GUI, at
     least do the `EnumPrinters()'.

     But ...  Kirill, I tried your formula for printing and nothing happened.
     Perhaps I didn't call redisplay-frame or something?  You need to fix this up
     and make it work for multi-page documents. (Again, this is in
     generic-print-buffer.)  Nothing special, it just needs to fucking work!  There
     are zillions and zillions of postings every day on xemacs-nt about how to get
     printing working, and none seem to refer to the built-in support.

     ben


     April 19, 2000

     Kirill 'Big K' Katsnelson wrote:

     > Some time ago, Ben Wing wrote...
     > >kirill, the interface i created is more general, like this:
     >
     > [snip]
     >
     > >Unfortunately I haven't implemented much of this; just some of the file
     > >dialog box.  but i think
     > >this is better than creating new mswindows-specific primitives.  if you
     > >are interested in working on
     > >this, i'll send you the code i have.
     >
     > Sure. Can you just commit it for my starting point?
     >
     > >also, the dialogs shouldn't have anything directly to do with the printer
     > >device.  all they should
     > >do is return a set of values.  it's the caller's responsibility to
     > >interpret them and set device
     > >properties accordingly.  this way, there's a complete separation between
     > >the underlying
     > >functionality and the gui.
     >
     > Unfortunately. I thought about doing it this way, but we then lose a lot of
     > printer-specific setup in this case. The DEVMODE structure contains two
     > parts: printer independent, as defined by SDK typedef DEVMODE, and
     > some trailing bytes, of unknown structure, used by a driver. The driver
     > only returns the extra length it wants. Such options as PCL ReT resolution
     > enhancement options or PostScript negative output are not available
     > through the standard part of the devmode structure, and stored in the
     > driver part (printer dialogs are driver-specific).
     >
     > So we have total of three options:
     > - Not to implement options beyond standard DEVMODE
     > - Make DEVMODE a Lisp object.
     > - Hide DEVMODE inside the device object.
     >
     > First case looks cheesy. Letting DEVMODE fall off the printer is no good
     > either, since one needs both the device and the devmode to edit the
     > devmode, and they must match. I am still convinced that the devmode and
     > the printer should not be separated.

     hmm, i see ...  this completely breaks abstraction though.  it fails in various
     scenarios, e.g. a program wants to initialize the dialog box with certain
     non-driver-specific properties, without caring about the particular printer.

     i think you should create a new print-properties object that encapsulates all
     printer properties (which can be changed using get/put), including the printer
     name, and contains a DEVMODE in it.  if the printer name gets changed, the
     DEVMODE might change too, but the print-properties object itself stays the
     same.  you pass this object as a parameter to the dialog box, and it gets
     changed accordingly.  you can call something like set-device-print-properties to
     stick everything in this structure into the device. (you could imagine a case
     where someone wanted to keep multiple print configurations around ...)

     >
     >
     > Big K

     --
     Ben


File: internals.info,  Node: Discussion -- Multilingual Issues,  Next: Discussion -- Instantiators and Generic Property Accessors,  Prev: Discussion -- Dialog Boxes,  Up: Future Work Discussion

41.4 Discussion - Multilingual Issues
=====================================


                                                          4/10/2000 4:13 AM

     BTW I am planning on adding some more powerful font-mapping capabilities to
     XEmacs (i.e. how do we map particular characters to the proper fonts that can
     display them, and how do we map the character's codes to the indices into the
     font).  These will replace to hackish charset-registry/charset-ccl-program stuff
     we currently have, and be [a] much more powerful, [b] designed in a
     window-system-independent way, [c] works with specifiers so you can control the
     mapping of individual buffers, and [d] works on a character rather than charset
     level, to correctly handle Unicode.  One possible usage would be to declare that
     all latin1 in a particular buffer to be displayed with latin2 fonts; I bet
     Hrvoje would really appreciate that

     ---------------------------------------------------------------------------

     April 10, 2000

     [info from "creation of generic macros for accessing internally formatted data"]

     Hmm, so there I just wrote a detailed design for the macros.  I would be
     *THRILLED* and overjoyed if you went ahead and implemented this mechanism, or
     parts of it.

     I've just finished arranging for a new transcriptionist, and soon I should be
     able to send off and get back my dictation of my (a) exposing streams to lisp,
     and (b) allowing for proper lisp-created coding systems, which define their
     reading, writing, and detecting methods in lisp.


     BTW How's it going wrt your Unicode and decode-priority stuff?

     And ...  you sent me mail asking what it was you had promised me, and listed
     only one thing, which was
     profiling of vm and certain other operations you found showed tremendous
     slowdown with Japanese characters.  The other main thing I want from you is

     -- Your priorities, as an actual Japanese user and XEmacs developer,
     concerning what MULE work should be done, how it should be done, in what
     order, etc.

     I'm sure there's something else, but it's been awhile since I took my sleeping
     dose and my brain can barely function anymore.  Just let me know how you're
     going to proceed with the above macro changes.

     BTW there's some nice Perl scripts written by Martin and fixed by me to make
     global-search-and-replace
     much, much easier.  I've attached them.  The first one is a shell script that
     works like

     gr foo bar *.[ch]

     and replaces foo with bar in all of the files.  For each modified file, a
     backup is created in the backup/ directory, which is created as necessary.
     This shell script is a fairly trivial front end onto global-replace2, which is
     a perl script that takes one argument (a Perl expression such as s/foo/bar/g)
     and a list of files obtained by reading the stdin, and does the same global
     replacement.  This means that the regexp syntax used here has to be perl-style
     rather than standard emacs/grep style.

     ben

     ---------------------------------------------------------------------


     From:
             Ben Wing <ben@666.com>
                                                         12/23/1999 3:34 AM

      Subject:
             Re: check process state before accessing coding_stream (fix PR#1061)
          To:
             "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>
         CC:
             XEmacs Developers <xemacs-beta@xemacs.org>




     Thankfully, nearly all of this horridity you bring up is irrelevant.  In
     XEmacs, "gettext" does not refer to any standard API, but is merely a stand-in
     for a translation routine (presumably written by us).  We may as well call it
     something else.  We define our own concept of "current language".  We also
     allow for a function that needs a different version for each language, which
     handles all cases where simple translation isn't sufficient, e.g. when you
     have to pluralize some noun given to you or insert the correct form of the
     definite article.  No weird hacks needed.  No interaction problems with other
     pieces of software.

     What I wrote "awhile ago" is (unfortunately) not anywhere public currently,
     but it's on my list to put it on the web site.  "There you go again" is
     usually not true; most of what I quote was indeed put out publicly at some
     point, but I'll try to be more explicit about this in the future.

     ben

     "Stephen J. Turnbull" wrote:

     > >>>>> "Ben" == Ben Wing <ben@666.com> writes:
     >
     >     Ben> "Stephen J. Turnbull" wrote:
     >
     >     >> What I have in mind is not just gettext-izing everything in the
     >     >> XEmacs core sources.  I currently believe that to be
     >     >> unacceptable
     >
     >     Ben> I don't quite understand.  Could you elaborate and give some
     >     Ben> examples?
     >
     > Examples?  Hmm.
     >
     > First, there's the surface of Jan's y-or-n-p example.  You have to
     > coordinate the translation of the message string and the response
     > prompt.  This is handled by y-or-n-p itself (I see that we already do
     > have gettext for Emacs Lisp, that's nice to know).
     >
     > Except that it's not really handled by y-or-n-p.  There's no reason to
     > suppose that somebody writing a Lisp package would necessarily use the
     > XEmacs domain (in fact, due to the way gettext binds text domains---if
     > I understand that correctly---we don't want that to be the case,
     > because it means that every time a Lisp package is updated the whole
     > XEmacs catalog must also be updated).  So which domain gets used for
     > the message string?
     >
     > In the current implementation, it is the domain of y-or-n-p.  So
     > packages with their own domain won't get y-or-n-p prompts correctly
     > translated.  But that means that the package should do its own
     > translation.  But now you're applying gettext to the same string
     > twice; you just have to pray the that translator upstream doesn't
     > collide with an English string that's in the XEmacs domain.  (The
     > gettext docs mention the similar problem of English words with
     > multiple meanings that must map to different words in the target
     > language; this can be disambiguated by various trickeries in forming
     > the strings ... but only if you "own" them, which in the multi-domain,
     > interated gettext example you do not.)  AFAICT this means that you
     > must never pass untranslated strings across public APIs, but this may
     > or may not be reasonable, and certainly is inconvenient.
     >
     > Next, we have to translate the possible answer strings to match the
     > language being passed by the user.  This is presumably OK here,
     > because it's done by y-or-n-p.  But what if y-or-n-p returned a string
     > rather than a boolean?  Then we would need to coordinate the
     > presentation of the prompt (done by y-or-n-p) and the translation of
     > the possible answer strings (done by the caller).  This can in fact be
     > done using dgettext with the XEmacs domain, but you must know that
     > y-or-n-p is in the XEmacs domain.  This is not necessarily going to be
     > obvious, and it might very well be that sets of related packages might
     > have the same domain, so you wouldn't necessarily know which domain is
     > appropriate by looking at the requires.
     >
     > And what happens if one domain does supply translations for a language
     > and the other does not?  AFAIK, gettext has no way to find out if this
     > is the case.  But you might very will prefer a global fallback to
     > English if substantial phrases are drawn from both domains, while you
     > might prefer string-by-string fallback if the main text is translated
     > and only a few words are left to fallback to English.
     >
     > Aside from confusing users, this puts a great burden on programmers.
     > Programmers need to know about the status of the domains of packages
     > they use as well as the XEmacs domain; they need to program
     > defensively against the possibility that some package they use will
     > become gettext-ized, or the translation projects will be out of synch
     > (some teams will do the calling package first, others will do the
     > caller package first).
     >
     > I don't think anybody will use gettext in these circumstances.  At
     > least not after they get the first bug report that "XEmacs is stuck in
     > an infinite y-or-n-p loop and I can't get out."
     >
     >     Ben> I wrote this awhile ago:
     >
     > "There you go again."  Not anywhere I could see it!  (At least, it
     > doesn't look familiar and grepping the archives doesn't turn it up.)
     >
     > OK, you win.  Subscribe me to xemacs-review.  Or whatever seems
     > appropriate.
     >
     > --
     > University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
     > Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
     > _________________  _________________  _________________  _________________
     > What are those straight lines for?  "XEmacs rules."

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it is not apparent in your message,
     please say so.  Thanks for your understanding.



     --------------------------------------------------------------------


     From:
             Ben Wing <ben@666.com>
                                                         12/21/1999 2:22 AM

      Subject:
             Re: check process state before accessing coding_stream (fix PR#1061)
          To:
             "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>
         CC:
             XEmacs Developers <xemacs-beta@xemacs.org>





     "Stephen J. Turnbull" wrote:

     > >>>>> "Ben" == Ben Wing <ben@666.com> writes:
     >
     >     Ben> Implementing message translation is not that hard.
     >
     > What I have in mind is not just gettext-izing everything in the XEmacs
     > core sources.  I currently believe that to be unacceptable (see Jan's
     > message for the pitfalls in I18N; it's worse for M17N).  I think
     > really solving this problem needs a specifier-like fallback mechanism
     > (this would solve Jan's example because you could query the
     > text-specifier presenting the question for the affirmative and
     > negative responses, and the catalog-building mechanism would have
     > checks to make sure they were properly set, perhaps a locale
     > (language) argument), and gettext is just not sufficient for that.

     I don't quite understand.  Could you elaborate and give some examples?

     >
     >
     > At a minimum, we need to implement gettext for Lisp packages.
     > (Currently, gettext is only implemented for C AFAIK.)  But this could
     > potentially cuase more trouble than it's worth.
     >
     >     Ben> A lot depends on priority: How important do you think this
     >     Ben> issue is to your average Japanese/Chinese/etc. user?
     >
     > Which average Japanese (etc) user?  The English-skilled (relatively)
     > programmer in the free software movement, or my not-at-all-competent
     > undergrad students who I would love to have using an Emacs?  This is a
     > really important ease-of-use issue.
     >
     > Realistically, for Japanese, it's low priority.  The Japanese team in
     > the GNU Translation Project is doing very little AFAIK, so even if the
     > capability were there, I doubt the message catalog would soon be done.
     >
     > But I think that many non-English speakers would find it very
     > attractive, and for many languages there are well-organized and
     > productive translation teams.  I suspect that if the I18N facility
     > were well-designed, many Western European languages would have full
     > catalogs within a year (granted, they are the ones where it's least
     > needed :-( ).
     >
     > Personally, I think doing it well is hard, and of little benefit to
     > _current_ core XEmacs constituency.  I think doing a good job, with
     > catalogs, would be very attractive to many non-English-speaking
     > _potential_ users.
     >
     >     Ben> How does it compare to some of the other important Mule
     >     Ben> issues that Martin and I are (trying to work) on?
     >
     > I don't know what you guys are _trying_ to work on.  Everything in the
     > I18N section of "Architecting XEmacs" is red-flagged.  OTOH, it's
     > clear from your posts that you are overburdened, so I can't read
     > priority into the fact that you've responded to specific issues in the
     > past.

     I wrote this awhile ago:


     >
     >     Ben> The big question is, would you be willing to help do the
     >     Ben> actual implementation, to "be my hands"?
     >
     > Sure, subject to the usual caveat that I'd need to be convinced it's
     > worth doing and a secondary caveat that I am not an experienced coder.

     If you'll implement it, I'll design it.  It's more a case of will on your part
     than anything else.  I can give you instructions sufficient enough to match
     your level of expertise.

     ben

     >
     >
     > --
     > University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
     > Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
     > _________________  _________________  _________________  _________________
     > What are those straight lines for?  "XEmacs rules."

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it is not apparent in your message,
     please say so.  Thanks for your understanding.



     -----------------------------------------------------------------------------

     Dec 20, 1999


     Implementing message translation is not that hard.  I've already done a lot of
     preliminary work in places such as `make-msgfile.lex' in lib-src/.  Finishing up
     the work is not that big a task; I already know exactly how it should be
     done.  Perhaps I'll write up detailed design instructions for this, as I'm
     doing for other things.  A lot depends on priority: How important do you think
     this issue is to your average Japanese/Chinese/etc. user?  How does it compare
     to some of the other important Mule issues that Martin and I are (trying to
     work) on?  If I did the design document, would you be willing to do the
     necessary bit of C hackery to implement the document?  If the design document
     is not specific enough for you, I can give you an "implementation document"
     which will definitely be specific enough: i.e. I'll show you exactly where the
     code needs to be modified, and how.  The big question is, would you be willing
     to help do the actual implementation, to "be my hands"?

     ---------------------------------------------------------------------------

     From:
             Ben Wing <ben@666.com>
                                                         12/14/1999 11:00 PM

      Subject:
             Re: Mule UI disaster: displaying character tables
          To:
             Hrvoje Niksic <hniksic@iskon.hr>
         CC:
             XEmacs vs Mule <xemacs-mule@xemacs.org>




     What I mean is, please put my name in the header, as well as xemacs-mule.
     That way I'll see it in my personal box.

     I agree that Mule has problems, but:

     Brokenness can be fixed.
     Slowness can be fixed.
     Limitations can be fixed.

     The design limitation you mention below, for example, is not really very
     hard to change.

     Keep in mind that I pretty much rewrote Mule from scratch, and did it
     *all* in 6-7 months.  In comparison with that, the changes below are
     pretty minor, and each could be done by a good (and able-bodied!)
     programmer familiar with the Mule code in less than a week -- to the
     XEmacs code, at least.  The problem is, everyone who could do this work is
     instead spending their time complaining about Mule problems instead of
     doing things.

     I'll gladly help out anyone who wants to do Mule coding by explaining all
     the details; I'll even write a "Mule internals manual", if that will
     help.  I can also make international phone calls -- they're cheap here in
     the US due to the long distance wars.  But so far no one has asked me for
     help or shown any willingness to do any work on Mule.

     Perhaps people are daunted by the seeming vastness of the problems.  But I
     wager that if I had another 6 months to work on nothing but Mule, it would
     be nearly perfect.  The basic design of the XEmacs C code is good;
     incremental changes, without over-much concern for compatibility, could
     make huge strides in a short amount of time (as was the case the whole
     time I worked on it, esp. towards the end -- it didn't even *compile* for
     4 months!).  A "total rewrite" would be an incredible waste of time.

     Again, I'm completely willing to provide help, documentation, design
     improvement suggestions (ala Architecting XEmacs -- which seems to have
     been completely ignored, alas), etc.

     ben

     Hrvoje Niksic wrote:

     > Ben Wing <ben@666.com> writes:
     >
     > > I'm the one who did most of the Mule work in XEmacs, so if you have
     > > any questions about the core, please address them to me directly.  I
     > > can probably give you a very clear and detailed answer.
     >
     > Thanks.  I think it still makes sense to ask here, so that other
     > developer have a chance to chime in.
     >
     > > However, I need some explanation.  What's misdesigned that you're
     > > complaining about?  And what's the coding-system disaster?
     >
     > It's been spoken of a lot.  Basically:
     >
     > * Unlike XEmacs/no-Mule, XEmacs/Mule doesn't preserve binary files in
     >   Latin 2 locales by default.  This is annoying for users who are used
     >   to XEmacs/no-Mule.
     >
     > * XEmacs/Mule is much slower than XEmacs, and not only because of
     >   character/byte conversions.  It seems that font lookups etc. are
     >   slower.
     >
     > * The "coding-system disaster" refers to inherent limitations of the
     >   coding-system model.  If I understand things correctly,
     >   coding-systems convert streams of bytes to streams of Emchars.  It
     >   does not appear to be possible to create a "gzip" coding system for
     >   handling gzipped file.  Even EOL conversions look kludgish:
     >
     >     iso-2022-8
     >     iso-2022-8-dos
     >     iso-2022-8-mac
     >     iso-2022-8-unix
     >     iso-2022-8bit-ss2
     >     iso-2022-8bit-ss2-dos
     >     iso-2022-8bit-ss2-mac
     >     iso-2022-8bit-ss2-unix
     >     iso-2022-int-1
     >     iso-2022-int-1-dos
     >     iso-2022-int-1-mac
     >     iso-2022-int-1-unix
     >
     >   Ideally, it should be possible to specify a stream of
     >   coding-systems, where only the last one converts to actual Emchars.
     >
     > There are more problems I don't remember right now.  Many many usage
     > problems become apparent when I stand and look over the shoulders of
     > an XEmacs users who tries to use Mule.

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.

     If you need an immediate response and it is not apparent in your message,
     please say so.  Thanks for your understanding.



     -----------------------------------------------------------------------




     From:
             Ben Wing <ben@666.com>
                                                        12/14/1999 12:20 AM

      Subject:
             Re: Mule UI disaster: displaying character tables
          To:
             "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>
         CC:
             XEmacs vs Mule <xemacs-mule@xemacs.org>




     I think you should go ahead with your proposal, and assume it will get
     implemented.  I don't think Martin is really suggesting that API changes not
     be allowed, but just that they proceed in a somewhat orderly fashion; and in
     any case, I imagine I have final say in cases of Mule-related conflicts.

     ben

     "Stephen J. Turnbull" wrote:

     > >>>>> "Hrvoje" == Hrvoje Niksic <hniksic@iskon.hr> writes:
     >
     >     Hrvoje> So next I tried the "Mule" menu.  That's right, boys and
     >     Hrvoje> girls, I've never looked at it before.
     >
     > For quite a while, it didn't work at all, led to crashes and other
     > warm/fuzzy things.  IIRC there used to be a top level menu item
     > pointing to information about the current language environment but it
     > got removed.
     >
     >     Hrvoje> Wow.  Seeing shift_jis, iso-2022 variants and (above all
     >     Hrvoje> things) big5 makes me really warm and fuzzy.
     >
     > We've been through this recently---you were there.  We know what to do
     > about it, basically (Ben liked my proposal, and it would fix this
     > silliness as well as the binary file breakage).  But given that Ben
     > and Martin seem to have different ideas about where to go with Mule
     > (Ben seemed to be supporting API and implementation revisions, Martin
     > evidently wants to keep the current Mule), working on that proposal is
     > possibly a waste of time.  I've got other stuff on my plate and I'll
     > get back to it one of these days (not tomorrow but sooner than Real
     > Soon Now).
     >
     >     Hrvoje> The items it presents (leading to further submenus) are:
     >
     >     Hrvoje>     94 character set
     >     Hrvoje>     94 x 94 character set
     >     Hrvoje>     96 character set
     >
     > This _is_ bad UI, now that you point it out.  But it is quite natural
     > for a coding system lawyer (as all Japanese users have to be), I never
     > noticed it before.  Easy enough to fix ("raise my karma").
     >
     >     Hrvoje> But I do bear some Mule scars, so I happily select "96
     >     Hrvoje> character sets", then ISO8859-2.  And I get this:
     >
     > [Table omitted]
     >
     >     Hrvoje> So me wonders: what the hell is this?
     >
     > Huh?  That is the standard table that you see over and over again in
     > references.  I'll believe you if you say you've never seen one before,
     > but every Japanese users' manual has dozens of pages of those, using
     > exactly that format.
     >
     > The presentation in the range 00--7F is not unreasonable for Latin 2;
     > ISO-8859 is a version of ISO-2022, so the high bit should not be
     > interpreted as "+ x80" (technically speaking), it should be
     > interpreted as a character set shift.
     >
     > Of course, this doesn't make sense to anybody but a character set
     > lawyer, and so should be changed.  Especially since the header refers
     > to ISO-8859-2 which everybody these days thinks of as _one, 8-bit_
     > character set, not two 7-bit ones.
     >
     > As for the "Japanese" in the table, that's just a really stupid
     > "optimization": those happen to be line-drawing characters available
     > in JIS X 0208, to make pretty borders.  Substitute "-", "+", and "|"
     > in appropriate places to make ugly but portable borders.
     >
     >     Hrvoje> Mule is just broken.  Warn your friends.
     >
     > Hrvoje is on the rampage again.  Warn your friends ;-)
     >
     > --
     > University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
     > Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
     > _________________  _________________  _________________  _________________
     > What are those straight lines for?  "XEmacs rules."

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it is not apparent in your message,
     please say so.  Thanks for your understanding.



     ---------------------------------------------------------------------------

     From:
             Ben Wing <ben@666.com>
                                                         12/14/1999 10:28 PM

      Subject:
             Re: Autodetect proposal; specifer questions/suggestions
          To:
             "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>




     I've always thought the specifier API is too complicated (and too
     "write-only"), but I went back at one point well after I designed it and I
     couldn't figure out an obvious way to simplify it that still kept reasonable
     functionality.  Perhaps that's what Custom did, and why it turned out bad.

     Inefficiency is a stupid reason not to use them.  They seem efficient enough
     for redisplay.  Changing them might be inefficient, but Emacs Lisp is in
     general, right?

     Can you propose an API or functionality change that will make them more used?



     "Stephen J. Turnbull" wrote:

     > >>>>> "Ben" == Ben Wing <ben@666.com> writes:
     >
     >     Ben> I think you should go ahead with your proposal, and assume it
     >     Ben> will get implemented.
     >
     > OK.  "yas baas" ;-)
     >
     > On something totally different.  I'm really bothered by the fact that
     > specifiers are so little used (eg, Custom reimplements them badly),
     > and the fact that every package seems to define its own set of faces
     > (or whatever), rather than use the specifier mechanism to inherit from
     > existing ones, or add new specifications to existing ones.  API problem?
     >
     > Also, faces (maybe specifiers in general?) should have an autoload
     > mechanism, and a `<package>-faces.el' (or `<package>-specifiers.el')
     > convention.  There are a number of faces in (eg) Custom that I like to
     > use, but I have to load Custom to get them.  And Custom should be able
     > to somehow see all the faces in various packages available, even when
     > they are not loaded.
     >
     > I've seen claims that specifiers aren't very efficient.
     >
     > Opinions?
     >
     > --
     > University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
     > Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
     > _________________  _________________  _________________  _________________
     > What are those straight lines for?  "XEmacs rules."

     --
     In order to save my hands, I am cutting back on my responses, especially
     to XEmacs-related mail.  You _will_ get a response, but please be patient.
     If you need an immediate response and it is not apparent in your message,
     please say so.  Thanks for your understanding.


     -----------------------------------------------------------------------------
     From:
             Ben Wing <ben@666.com>
                                                          11/18/1999 9:02 PM

      Subject:
             Re: Char-related crashes (hopefully) fixed
          To:
             "Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp>
         CC:
             XEmacs Beta List <xemacs-beta@xemacs.org>




     OK, in summation:

     1. C-q is a user-level function and should do whatever makes the most sense.
     2. int-char is a low-level primitive and should never depend on high-level
     settings like language environment.
     3. Everything you can do with int-char can and should be done with make-char
     -- representation-independent, much less likelihood of bugs, etc.  Therefore
     int-char should be removed.
     4. Note that CLTL2 also removes int-char.
     5. Your statement

     > In one-byte buffers (either Olivier's 1/2/4 extension or `xemacs -font
     > *-iso8859-2') it implicitly will have dependence whatever you say.

     is confusing internal and external representations.

     ben

     "Stephen J. Turnbull" wrote:

     > Can somebody give a bunch of examples where using integers as
     > characters is useful?  For that matter, where they are actually used?
     > Ben said "backward compatibility," but I haven't seen this used, and I
     > don't really know how to grep for it.  I have grepped for int-char,
     > int-to-char, char-int, and char-to-int and they're pretty rare in the
     > core and package code (2/3 of it) that I have.
     >
     > The only one that I ever use is the C-q hack for inserting characters
     > by code value at the keyboard, and that could arguably (and in
     > Japanese invariably is) delegated to an input method which would know
     > about language environment (and return a true character).
     >
     > For iterating over a character set in "natural" order, only ASCII
     > satisfies the requirement of having one, and even that's shaky.  AFAIK
     > the Swedes and the Norwegians, or is it the Danes, disagree on
     > ordering the _letters_ in ISO-8859-1 character set.  This really
     > should be table-driven, and will have to be for everything except
     > ASCII and ISO-8859-1 if we go to a Unicode internal representation.
     >
     > We already have primitives for efficient case conversion and the like.
     >
     > The only example I can think of offhand where you would really really
     > want the facility is to iterate over a code space where you don't know
     > which points are legal characters.  Eg, to print out tables of fonts.
     > Pretty specialized.  And this can be done through make-char, anyway.
     >
     > According to CLtL1, the main portable use for char-int is for hashing.
     > But that doesn't square with the kind of usage we've been talking
     > about (in loops and the like).
     >
     > What else am I missing?
     >
     > Ben's desiderata have some problems.
     >
     > >>>>> "Ben" == Ben Wing <ben@666.com> writes:
     >
     >     Ben> Either int-char should be the mirror opposite of char-int
     >     Ben> (i.e. accept all legal char integers), or it should be
     >     Ben> removed entirely.
     >
     > OK.  I agree with this.
     >
     >     Ben> int-char should *never* have any dependence on the language
     >     Ben> environment.
     >
     > In one-byte buffers (either Olivier's 1/2/4 extension or `xemacs -font
     > *-iso8859-2') it implicitly will have dependence whatever you say.
     > Even without Mule, people can always use external encoders to change
     > raw ISO-8859-2 to ISO-2022 (not that anybody sane ever would, OK,
     > Hrvoje?).  Then the two files will be interpreted differently in a
     > Latin-1 locale Mule; the ISO-8859-2 file will be recognized as
     > ISO-8859-1, and the ISO-2022 file will be internally interpreted as
     > ISO-8859-2.
     >
     > The point is that people normally assume that int-char should accept
     > their "natural" integer to character map.  For Americans, that's
     > ASCII, for Germans, that's ISO-8859-1, for Croatians, that's
     > ISO-8859-2.  And it works "correctly" in a no-mule XEmacs with `-font
     > *-iso8859-2'!  Japanese usually use ku-ten or JIS, and there's a
     > "natural" map from byte-sized integer pairs to shorts, but it's full
     > of holes.  So language environments don't agree on what a legal char
     > integer is, and where they do (eg, ISO-8859-1 and ISO-8859-2), they
     > don't agree on the map.  To satisfy your dictum (with which I agree,
     > but I take to mean we should get rid of these functions) we can take
     > the intersection where they agree
     >
     > ==> legal char integers == ASCII
     >
     > which is what I prefer, or pick something arbitrary and efficient
     >
     > ==> char-int returns the internal representation
     >
     > which I really hate, or something else.  Suggestions?
     >
     >     Ben> I don't think C-q should either.  If Hrvoje wants to insert
     >     Ben> Latin-2 characters by number, then make C-u C-q work so that
     >     Ben> it also prompts for a character set, with a default chosen
     >     Ben> from the language environment.
     >
     > And restrict this to ASCII?  Or assume Latin-1 in GR if there is no
     > prefix argument?
     >
     > This is a useful feature.  C-q currently inserts Latin-2 characters
     > for Hrvoje in no-mule XEmacs (stretching the point only a little); I
     > think it should continue to do so in Mule.  This really is an input
     > method issue, not a keyboard issue.  In XEmacs, inserting an integer
     > into a buffer has no meaning.  Users insert characters.  So this is a
     > completely different issue from the programming API, and should not be
     > considered analogous.
     >
     > Maybe we could have C-q insert according to the Unicode standard, and
     > treat C-u C-q as part of the input method.  But I think most users
     > would prefer to have C-q insert according to their locale-standard
     > tables, and select Unicode explicitly using the C-u C-q idiom.  In
     > fact (again this points to the input method idea), Japanese users
     > would probably like to have the alternatives of using kuten (pairs
     > from 1--94 x 1--94) or JIS (pairs from 0x21--0x7E x 0x21--0x7E) as
     > options since both indexing systems are common in tables.
     >
     > --
     > University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
     > Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
     > __________________________________________________________________________
     > __________________________________________________________________________
     > What are those two straight lines for?  "Free software rules."

     --
     ben

     --
     In order to save my hands, I am cutting back on my responses, especially to
     XEmacs-related mail.  You
     _will_ get a response, but please be patient.  If you need an immediate
     response and it’s not apparent in
     your message, please say so.  Thanks for your understanding.



     -----------------------------------------------------------------------------

     From:
             Ben Wing <ben@666.com>
                                                         11/16/1999 11:03 PM

      Subject:
             Re: Char-related crashes (hopefully) fixed
          To:
             Yoshiki Hayashi <t90553@m.ecc.u-tokyo.ac.jp>
         CC:
             Hrvoje Niksic <hniksic@iskon.hr>,
             XEmacs Beta List <xemacs-beta@xemacs.org>




     Either int-char should be the mirror opposite of char-int (i.e. accept all
     legal char integers), or it should be removed entirely.

     int-char should *never* have any dependence on the language environment.

     I don't think C-q should either.  If Hrvoje wants to insert Latin-2
     characters by number, then make C-u C-q work so that it also prompts for a
     character set, with a default chosen from the language environment.

     ben

     Yoshiki Hayashi wrote:

     > Hrvoje Niksic <hniksic@iskon.hr> writes:
     >
     > > As Ben said, now that we've fixed the actual bugs, we can think about
     > > changing the behaviour for int-char conversions for 21.2.
     >
     > Following are proposed which integers should be accepted
     > where characters are expected:
     >
     > 1) Don't allow anything
     > 2) Accept 0-127
     > 3) Accept 0-256
     > 4) Accept everything
     >
     > Other things proposed are:
     >
     > a) When doing C-q, treat 128-256 as Latin-2 in Latin 2
     >    language environment.
     >
     > So far, most of the proposal is intended to apply to every
     > int-char conversions, I'd like to make some functions to
     > accept.
     >
     > My plan is:
     > Accept only 0-256 in every place except int-to-char.
     > int-to-char accepts every valid integers.
     > Make new function which does int-to-char conversion
     > correctly according to the language environment.
     >
     > This way, most of the code which does (insert (1+ ?a)) or
     > something continues working. Now internal representation is
     > changed a little bit, so disabling > 256 characters will
     > warn those who are dealing with internal representation
     > directly, which is bad. Still, you can do
     > (let ((i 1442))
     >   (while (i < 2000)
     >     (insert (int-to-char i))
     >     (setq i (+1 i))))
     > to achieve old behaviour.
     >
     > For C-q, I'm not for changing it's original definition,
     > since it might confuse people who are expecting Latin-1 in
     > other language environment and typing just 1 integer doesn't
     > make sense for multibyte world. It's cleaner to make new
     > function, which does make-char according to the charset of
     > language-info-alist so that people who use that often can
     > bind it to C-q or some other keys.
     >
     > --
     > Yoshiki Hayashi

     --
     ben

     --
     In order to save my hands, I am cutting back on my responses, especially to
     XEmacs-related mail.  You
     _will_ get a response, but please be patient.  If you need an immediate
     response and it’s not apparent in
     your message, please say so.  Thanks for your understanding.

