This is ../info/internals.info, produced by makeinfo version 4.7 from
internals/internals.texi.

INFO-DIR-SECTION XEmacs Editor
START-INFO-DIR-ENTRY
* Internals: (internals).       XEmacs Internals Manual.
END-INFO-DIR-ENTRY

   Edition History:

   Created November 1995 (?) by Ben Wing.  XEmacs Internals Manual
Version 1.0, March, 1996.  XEmacs Internals Manual Version 1.1, March,
1997.  XEmacs Internals Manual Version 1.4, March, 2001.  XEmacs
Internals Manual Version 21.5, October, 2004.

   Copyright (C) 1992 - 2004 Ben Wing.  Copyright (C) 1996, 1997 Sun
Microsystems.  Copyright (C) 1994 - 1998, 2002, 2003 Free Software
Foundation.  Copyright (C) 1994, 1995 Board of Trustees, University of
Illinois.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that the
entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the Foundation.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided also
that the section entitled "GNU General Public License" is included
exactly as in the original, and provided that the entire resulting
derived work is distributed under the terms of a permission notice
identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that the section entitled "GNU General Public License"
may be included in a translation approved by the Free Software
Foundation instead of in the original English.


File: internals.info,  Node: The Window Object,  Next: Modules for the Basic Displayable Lisp Objects,  Prev: Window Hierarchy,  Up: Consoles; Devices; Frames; Windows

26.4 The Window Object
======================

Windows have the following accessible fields:

`frame'
     The frame that this window is on.

`mini_p'
     Non-`nil' if this window is a minibuffer window.

`buffer'
     The buffer that the window is displaying.  This may change often
     during the life of the window.

`dedicated'
     Non-`nil' if this window is dedicated to its buffer.

`pointm'
     This is the value of point in the current buffer when this window
     is selected; when it is not selected, it retains its previous
     value.

`start'
     The position in the buffer that is the first character to be
     displayed in the window.

`force_start'
     If this flag is non-`nil', it says that the window has been
     scrolled explicitly by the Lisp program.  This affects what the
     next redisplay does if point is off the screen: instead of
     scrolling the window to show the text around point, it moves point
     to a location that is on the screen.

`last_modified'
     The `modified' field of the window's buffer, as of the last time a
     redisplay completed in this window.

`last_point'
     The buffer's value of point, as of the last time a redisplay
     completed in this window.

`left'
     This is the left-hand edge of the window, measured in columns.
     (The leftmost column on the screen is column 0.)

`top'
     This is the top edge of the window, measured in lines.  (The top
     line on the screen is line 0.)

`height'
     The height of the window, measured in lines.

`width'
     The width of the window, measured in columns.

`next'
     This is the window that is the next in the chain of siblings.  It
     is `nil' in a window that is the rightmost or bottommost of a
     group of siblings.

`prev'
     This is the window that is the previous in the chain of siblings.
     It is `nil' in a window that is the leftmost or topmost of a group
     of siblings.

`parent'
     Internally, XEmacs arranges windows in a tree; each group of
     siblings has a parent window whose area includes all the siblings.
     This field points to a window's parent.

     Parent windows do not display buffers, and play little role in
     display except to shape their child windows.  Emacs Lisp programs
     usually have no access to the parent windows; they operate on the
     windows at the leaves of the tree, which actually display buffers.

`hscroll'
     This is the number of columns that the display in the window is
     scrolled horizontally to the left.  Normally, this is 0.

`use_time'
     This is the last time that the window was selected.  The function
     `get-lru-window' uses this field.

`display_table'
     The window's display table, or `nil' if none is specified for it.

`update_mode_line'
     Non-`nil' means this window's mode line needs to be updated.

`base_line_number'
     The line number of a certain position in the buffer, or `nil'.
     This is used for displaying the line number of point in the mode
     line.

`base_line_pos'
     The position in the buffer for which the line number is known, or
     `nil' meaning none is known.

`region_showing'
     If the region (or part of it) is highlighted in this window, this
     field holds the mark position that made one end of that region.
     Otherwise, this field is `nil'.


File: internals.info,  Node: Modules for the Basic Displayable Lisp Objects,  Prev: The Window Object,  Up: Consoles; Devices; Frames; Windows

26.5 Modules for the Basic Displayable Lisp Objects
===================================================

     `console-msw.c'
     `console-msw.h'
     `console-stream.c'
     `console-stream.h'
     `console-tty.c'
     `console-tty.h'
     `console-x.c'
     `console-x.h'
     `console.c'
     `console.h'

   These modules implement the "console" Lisp object type.  A console
contains multiple display devices, but only one keyboard and mouse.
Most of the time, a console will contain exactly one device.

   *This model may no longer suffice.*  The X Window System (at least)
now supports a variety of input devices, including touchscreens and
tablets, as well as the traditional keyboard and mouse, and may even be
able to support multiple instances of a single type of input device
(especially pointing devices) on a single console.

   Consoles are the top of a lisp object inclusion hierarchy.  Consoles
contain devices, which contain frames, which contain windows.

     `device-msw.c'
     `device-tty.c'
     `device-x.c'
     `device.c'
     `device.h'

   These modules implement the "device" Lisp object type.  This
abstracts a particular screen or connection on which frames are
displayed.  As with Lisp objects, event interfaces, and other
subsystems, the device code is separated into a generic component that
contains a standardized interface (in the form of a set of methods) onto
particular device types.

   The device subsystem defines all the methods and provides method
services for not only device operations but also for the frame, window,
menubar, scrollbar, toolbar, and other displayable-object subsystems.
The reason for this is that all of these subsystems have the same
subtypes (X, TTY, NeXTstep, Microsoft Windows, etc.) as devices do.

   *This abstraction is probably broken* (as of late 2004), at least
for X consoles, with the advent of the *Xft* library.  Xft is a
complete break from the traditional approach to text rendering in the
X11 environment, since fonts are composed of glyphs rendered by
_client-side_ code.  These glyphs are then transmitted to the server as
sets of trapezoids, and displayed by the *XRender* extension (where
available; the X11 core protocol can also be used, but this is slow).
The XRender extension is especially attractive because it allows modern
image composition techniques to be used to render antialiased fonts.

   By contrast, the traditional approach renders fonts on the server
side as a collection of bitmaps.  It is also possible use a _font
server_ that knows how to render antialiased fonts, but for some reason
this approach has never caught on.

   The problem that this creates for XEmacs is that the traditional (and
still popular) widget sets, the various Athena variants and the Motif
widget set, are based on the server-side rendering model.  Thus, even if
XEmacs-specific widgets (such as the basic text window, the Lucid
menubar, and the recently added tab control) can be adapted to render
text via Xft, older widgets (such as buttons and labels) and "modern"
widgets derived from traditional widgets (the progress gauge) still
expect their font resources to be converted to server-side fonts.  Then
text is rendered by calls to the core protocol via Xlib, rather than by
calls to the XRender protocol via Xft.

   It's even possible to imagine a situation where a widget is composed
of components which draw their own text (_predefined widgets_) as well
as new components which can draw using more modern methods.  Handling
this will either require reworking the Emacs face mechanism to provide a
way to determine whether this widget can use that font, or ways to give
each of several different widgets, even different components of a given
widget, a different face property.  This is already an issue, as widgets
generally derive their font from the gui-element face, but this might
not be appropriate for widgets embedded in a buffer.

   There seem to be two overall ways to go.

  1. Add face information to the Lucid widget library, and extend the
     coverage of that library to device types that aren't supported by
     it yet (MS Windows, and maybe TTY).

  2. Create a lighter-weight interface, perhaps an extension of the
     device interface, that allows widgets to access face information
     from the device it is implemented on.

     `frame-msw.c'
     `frame-tty.c'
     `frame-x.c'
     `frame.c'
     `frame.h'

   Each device contains one or more frames in which objects (e.g. text)
are displayed.  A frame corresponds to a window in the window system;
usually this is a top-level window but it could potentially be one of a
number of overlapping child windows within a top-level window, using the
MDI (Multiple Document Interface) protocol in Microsoft Windows or a
similar scheme.

   The `frame-*' files implement the "frame" Lisp object type and
provide the generic and device-type-specific operations on frames (e.g.
raising, lowering, resizing, moving, etc.).

     `window.c'
     `window.h'

   Each frame consists of one or more non-overlapping "windows" (better
known as "panes" in standard window-system terminology) in which a
buffer's text can be displayed.  Windows can also have scrollbars
displayed around their edges.

   `window.c' and `window.h' implement the "window" Lisp object type
and provide code to manage windows.  Since windows have no associated
resources in the window system (the window system knows only about the
frame; no child windows or anything are used for XEmacs windows), there
is no device-type-specific code here; all of that code is part of the
redisplay mechanism or the code for particular object types such as
scrollbars.


File: internals.info,  Node: The Redisplay Mechanism,  Next: Extents,  Prev: Consoles; Devices; Frames; Windows,  Up: Top

27 The Redisplay Mechanism
**************************

The redisplay mechanism is one of the most complicated sections of
XEmacs, especially from a conceptual standpoint.  This is doubly so
because, unlike for the basic aspects of the Lisp interpreter, the
computer science theories of how to efficiently handle redisplay are not
well-developed.

   When working with the redisplay mechanism, remember the Golden Rules
of Redisplay:

  1. It Is Better To Be Correct Than Fast.

  2. Thou Shalt Not Run Elisp From Within Redisplay.

  3. It Is Better To Be Fast Than Not To Be.

* Menu:

* Critical Redisplay Sections::
* Line Start Cache::
* Redisplay Piece by Piece::
* Modules for the Redisplay Mechanism::
* Modules for other Display-Related Lisp Objects::


File: internals.info,  Node: Critical Redisplay Sections,  Next: Line Start Cache,  Prev: The Redisplay Mechanism,  Up: The Redisplay Mechanism

27.1 Critical Redisplay Sections
================================

*The following paragraphs are way out-of-date and inaccurate.*

   Within this section, we are defenseless and assume that the
following cannot happen:

  1. garbage collection

  2. Lisp code evaluation

  3. frame size changes

   We ensure (3) by calling `hold_frame_size_changes()', which will
cause any pending frame size changes to get put on hold till after the
end of the critical section.  (1) follows automatically if (2) is met.
#### Unfortunately, there are some places where Lisp code can be called
within this section.  We need to remove them.

   If `Fsignal()' is called during this critical section, we will
`abort()'.

   If garbage collection is called during this critical section, we
simply return. #### We should abort instead.

   #### If a frame-size change does occur we should probably actually
be preempting redisplay.

   *Begin up-to-date stuff*

27.1.1 Nasty Bugs due to Reentrancy in Redisplay Structures handling QUIT
-------------------------------------------------------------------------

These are now fixed as of November 10, 2004.

Crash - reentrant `regenerate_window()'
---------------------------------------

Here is a crash I (ben) just got - November 9, 2004: It can sort of be
reproduced by creating a bunch of frames, opening a bunch of large
files (which may be fontlocking for awhile). and immediately start
Alt-TAB-ing back and forth quickly and constantly scrolling up and down
using the scrolling dial on your mouse.


     Fatal error: assertion failed, file c:\xemacs\build\src\redisplay.c, line 5532,
     !dy->locked

     C backtrace:

     assert_failed(const char * 0x012a4ff0 `string', int 5532, const char * 0x0127bea4 `string') line 3839
     Dynarr_verify_mod_1(void * 0x023ad2b0, const char * 0x012a4ff0 `string', int 5532) line 1306 + 36 bytes
     regenerate_window(window * 0x02f2ca88, long 40372, long 40372, int 2) line 5532 + 25 bytes
     update_line_start_cache(window * 0x02f2ca88, long 40372, long 40401, long 40372, int 0) line 8543 + 19 bytes
     point_in_line_start_cache(window * 0x02f2ca88, long 40372, int 0) line 7850 + 23 bytes
     start_end_of_last_line(window * 0x02f2ca88, long 40372, int 1, int 1) line 8121 + 15 bytes
     end_of_last_line_may_error(window * 0x02f2ca88, long 40372) line 8203 + 17 bytes
     pixel_to_glyph_translation(frame * 0x02f2c900, int 291, int 317, int * 0x0082bb04, int * 0x0082bb00, int * 0x0082bafc, int * 0x0082baf8, window * * 0x0082bae8, long * 0x0082baf4, long * 0x0082baf0, long * 0x0082baec, long * 0x0082bb10, long * 0x0082bb0c) line 9336 + 32 bytes
     mswindows_handle_mousewheel_event(long 49465600, int 0, int -240, tagPOINTS {...}) line 360 + 82 bytes
     mswindows_wnd_proc(HWND__ * 0x00260a42, unsigned int 522, unsigned int 4279238656, long 29885130) line 3561 + 36 bytes
     intercepted_wnd_proc(HWND__ * 0x00260a42, unsigned int 522, unsigned int 4279238656, long 29885130) line 2376
     USER32! 77e11ef0()
     USER32! 77e1204c()
     USER32! 77e121af()
     mswindows_drain_windows_queue(int 0) line 1330 + 9 bytes
     emacs_mswindows_drain_queue() line 1339 + 7 bytes
     event_stream_drain_queue() line 1785
     event_stream_quit_p() line 1893
     check_quit() line 938
     check_what_happened() line 459
     internal_equal(long 22180468, long 22180468, int 0) line 2823 + 14 bytes
     update_image_instance(long 83498640, long 22180468) line 2121 + 18 bytes
     image_instantiate(long 21418616, long 20663624, long 54932896, long 22180468, long 3) line 3403 + 13 bytes
     va_call_trapping_problems_1(void * 0x0082cf94) line 5220 + 221 bytes
     call_trapping_problems_2(long 83160440) line 4867 + 13 bytes
     call_with_condition_handler(long (long, long, long)* 0x010cc4c0 flagged_a_squirmer(long, long, long), long 83160440, long (long)* 0x010cc440 call_trapping_problems_2(long), long 83160440) line 2129 + 7 bytes
     call_trapping_problems_1(long 83160440) line 4874 + 23 bytes
     internal_catch(long 21399864, long (long)* 0x010cc490 call_trapping_problems_1(long), long 83160440, int * volatile 0x0082ce4c, long * volatile 0x0082ce54) line 1527 + 7 bytes
     call_trapping_problems(long 20908160, const char * 0x00000000, int 98315, call_trapping_problems_result * 0x00000000, long (void *)* 0x010cca30 va_call_trapping_problems_1(void *), void * 0x0082cf94) line 5147 + 32 bytes
     call_with_suspended_errors(long (void)* 0x011448c0 image_instantiate(long, long, long, long, long), long 20663624, long 20908160, _error_behavior_struct_ {...}, int 5) line 5314 + 26 bytes
     specifier_instance_from_inst_list(long 21418616, long 20663624, long 54932896, long 21673760, _error_behavior_struct_ {...}, int 1, long 3) line 2501 + 54 bytes
     specifier_instance(long 21418616, long 20663624, long 54932896, _error_behavior_struct_ {...}, int 1, int 0, long 3) line 2614 + 64 bytes
     glyph_image_instance(long 22692176, long 54932896, _error_behavior_struct_ {...}, int 1) line 3955 + 31 bytes
     add_glyph_rune(position_redisplay_data_type * 0x0082d52c, glyph_block * 0x0082d454, int 0, int 0, glyph_cachel * 0x04f4e518) line 1972 + 26 bytes
     create_text_block(window * 0x034635a0, display_line * 0x033bfb28, long 29860, prop_block_dynarr * * 0x0082d7b8, int 2) line 2827 + 30 bytes
     generate_display_line(window * 0x034635a0, display_line * 0x033bfb28, int 1, long 29860, prop_block_dynarr * * 0x0082d7b8, int 2) line 979 + 38 bytes
     regenerate_window(window * 0x034635a0, long 29860, long 25012, int 2) line 5607 + 30 bytes
     update_line_start_cache(window * 0x034635a0, long 25012, long 28767, long 25012, int 0) line 8614 + 19 bytes
     point_in_line_start_cache(window * 0x034635a0, long 25012, int 0) line 7850 + 23 bytes
     start_end_of_last_line(window * 0x034635a0, long 25012, int 1, int 0) line 8121 + 15 bytes
     end_of_last_line(window * 0x034635a0, long 25012) line 8197 + 17 bytes
     Fwindow_end(long 54932896, long 20926544) line 1848 + 13 bytes
     Ffuncall(int 3, long * 0x0082dbb8) line 3841 + 93 bytes
     execute_optimized_program(const unsigned char * 0x032ceee8, int 7, long * 0x03289f40) line 823 + 16 bytes
     funcall_compiled_function(long 52991916, int 1, long * 0x0082dfb0) line 3454 + 85 bytes
     Ffuncall(int 2, long * 0x0082dfac) line 3880 + 17 bytes
     execute_optimized_program(const unsigned char * 0x02f667d8, int 6, long * 0x01558748) line 823 + 16 bytes
     funcall_compiled_function(long 22579576, int 3, long * 0x0082e3ac) line 3454 + 85 bytes
     Ffuncall(int 4, long * 0x0082e3a8) line 3880 + 17 bytes
     execute_optimized_program(const unsigned char * 0x03209c98, int 5, long * 0x03288c68) line 823 + 16 bytes
     funcall_compiled_function(long 51656320, int 1, long * 0x0082e7a4) line 3454 + 85 bytes
     Ffuncall(int 2, long * 0x0082e7a0) line 3880 + 17 bytes
     execute_optimized_program(const unsigned char * 0x0082e9ec, int 4, long * 0x03224990) line 823 + 16 bytes
     Fbyte_code(long 37927380, long 52578688, long 9) line 2564 + 70 bytes
     Feval(long 51505420) line 3601 + 187 bytes
     internal_catch(long 51959412, long (long)* 0x010c6f40 Feval(long), long 51505420, int * volatile 0x00000000, long * volatile 0x00000000) line 1527 + 7 bytes
     execute_rare_opcode(long * 0x0082eee8, const unsigned char * 0x03248365, Opcode Bcatch) line 1380 + 24 bytes
     execute_optimized_program(const unsigned char * 0x03248340, int 2, long * 0x02f3c0a0) line 715 + 17 bytes
     funcall_compiled_function(long 51656276, int 0, long * 0x0082f444) line 3454 + 85 bytes
     Ffuncall(int 1, long * 0x0082f440) line 3880 + 17 bytes
     run_hook_with_args_in_buffer(buffer * 0x04ee9060, int 1, long * 0x0082f440, run_hooks_condition RUN_HOOKS_TO_COMPLETION) line 4361 + 13 bytes
     run_hook_with_args(int 1, long * 0x0082f440, run_hooks_condition RUN_HOOKS_TO_COMPLETION) line 4374 + 23 bytes
     run_hook(long 51959028) line 4443 + 13 bytes
     safe_run_hook_trapping_problems_1(void * 0x013c73c0) line 5517 + 9 bytes
     call_trapping_problems_2(long 83157920) line 4867 + 13 bytes
     call_with_condition_handler(long (long, long, long)* 0x010cc4c0 flagged_a_squirmer(long, long, long), long 83157920, long (long)* 0x010cc440 call_trapping_problems_2(long), long 83157920) line 2129 + 7 bytes
     call_trapping_problems_1(long 83157920) line 4874 + 23 bytes
     internal_catch(long 21399864, long (long)* 0x010cc490 call_trapping_problems_1(long), long 83157920, int * volatile 0x0082f700, long * volatile 0x0082f708) line 1527 + 7 bytes
     call_trapping_problems(long 20925944, const char * 0x00000000, int 131235, call_trapping_problems_result * 0x0082f830, long (void *)* 0x010cd990 safe_run_hook_trapping_problems_1(void *), void * 0x013c73c0) line 5147 + 32 bytes
     safe_run_hook_trapping_problems(long 20741312, long 20739008, int 160) line 5543 + 36 bytes
     run_pre_idle_hook() line 2084 + 24 bytes
     redisplay() line 7224
     Fnext_event(long 37363732, long 20928056) line 2263
     Fcommand_loop_1() line 600 + 15 bytes
     command_loop_1(long 20928056) line 512
     condition_case_1(long 20925944, long (long)* 0x01096a80 command_loop_1(long), long 20928056, long (long, long)* 0x01096630 cmd_error(long, long), long 20928056) line 1918 + 7 bytes
     command_loop_3() line 262 + 35 bytes
     command_loop_2(long 20928056) line 277
     internal_catch(long 20683712, long (long)* 0x010967a0 command_loop_2(long), long 20928056, int * volatile 0x00000000, long * volatile 0x00000000) line 1527 + 7 bytes
     initial_command_loop(long 20928056) line 313 + 28 bytes
     xemacs_21_5_b18_i586_pc_win32(int 1, unsigned short * * 0x0082fed0, unsigned short * * 0x00000000, int 0) line 2551
     main(int 1, char * * 0x00e52610, char * * 0x00e52bb0) line 2992
     mainCRTStartup() line 338 + 17 bytes
     KERNEL32! 7c59893d()

     Lisp backtrace:

       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (catch #<INTERNAL OBJECT (XEmacs bug?) (opaque, size=0) 0x1468938> ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       window-end(#<window on "signal.c<2>" 0x5e4a> t)
       # (unwind-protect ...)
       # bind (buffer we-are-screwed check-text-props window)
       lazy-lock-fontify-window(#<window on "signal.c<2>" 0x5e4a>)
       # bind (walk-windows-current walk-windows-start arg which-devices which-frames
      minibuf function)
       walk-windows(lazy-lock-fontify-window no-minibuf #<mswindows-frame "emacs" 0x5
     e49>)
       # (unwind-protect ...)
       # bind (ssf65112 tick frame)
       lazy-lock-maybe-fontify-frame(#<mswindows-frame "emacs" 0x5e49>)
       # bind (frame starting-frame)
       byte-code("..." [starting-frame frame selected-frame frame-visible-p frame-min
     ibuffer-only-p next-frame visible-nomini throw lazy-lock-frame-loop-done t lazy-
     lock-maybe-fontify-frame] 4)
       # (catch lazy-lock-frame-loop-done ...)
       lazy-lock-pre-idle-fontify-windows()
       # (unwind-protect ...)
       # (catch #<INTERNAL OBJECT (XEmacs bug?) (opaque, size=0) 0x1468938> ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       (next-event "[internal]")
       # (condition-case ... . error)
       # (catch top-level ...)

Another Lisp trace of a similar situation (C stack trace not available):
........................................................................

     Fatal error: assertion failed, file c:\xemacs\build\src\redisplay.c, line 5532,
     !dy->locked

     Lisp backtrace follows:

       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       scrollbar-page-down((#<window on "*grep*" 0x1a5f9>))
       (dispatch-event "[internal]")
       # (unwind-protect ...)
       # (catch #<INTERNAL OBJECT (XEmacs bug?) (opaque, size=0) 0x1468270> ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # (catch #<INTERNAL OBJECT (XEmacs bug?) (opaque, size=0) 0x1468270> ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       window-end(#<window on "*grep*" 0x1a5f9> t)
       # (unwind-protect ...)
       # bind (buffer we-are-screwed check-text-props window)
       lazy-lock-fontify-window(#<window on "*grep*" 0x1a5f9>)
       # bind (walk-windows-current walk-windows-start arg which-devices which-frames
      minibuf function)
       walk-windows(lazy-lock-fontify-window no-minibuf #<mswindows-frame "emacs" 0x1
     9f64>)
       # (unwind-protect ...)
       # bind (ssf65112 tick frame)
       lazy-lock-maybe-fontify-frame(#<mswindows-frame "emacs" 0x19f64>)
       # bind (frame starting-frame)
       byte-code("..." [starting-frame frame selected-frame frame-visible-p frame-min
     ibuffer-only-p next-frame visible-nomini throw lazy-lock-frame-loop-done t lazy-
     lock-maybe-fontify-frame] 4)
       # (catch lazy-lock-frame-loop-done ...)
       lazy-lock-pre-idle-fontify-windows()
       # (unwind-protect ...)
       # (catch #<INTERNAL OBJECT (XEmacs bug?) (opaque, size=0) 0x1468270> ...)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       # (unwind-protect ...)
       # (unwind-protect ...)
       # bind (inhibit-quit)
       (next-event "[internal]")
       # (condition-case ... . error)
       # (catch top-level ...)

Crash - reentrant `generate_displayable_area()'
-----------------------------------------------

Original code said [Tricky tricky tricky.
`generate_displayable_area()' can (could) be called reentrantly, and
redisplay is not prepared to handle this:].

   assert_failed(const char * 0x0129c8c8 `string', int 5328, const char
* 0x01274068 `string') line 3620 Dynarr_verify_mod_1(void * 0x0250f228,
const char * 0x0129c8c8 `string', int 5328) line 1256 + 36 bytes
generate_displayable_area(window * 0x02480028, long 38776292, int 0,
int 0, int 265, int 169, display_line_dynarr * 0x0250f228, long 0, int
2) line 5328 + 25 bytes output_gutter(frame * 0x0228ad90, gutter_pos
TOP_GUTTER, int 1) line 409 + 69 bytes redraw_exposed_gutter(frame *
0x0228ad90, gutter_pos TOP_GUTTER, int 8, int 23, int 249, int 127)
line 687 + 15 bytes redraw_exposed_gutters(frame * 0x0228ad90, int 8,
int 23, int 249, int 127) line 703 + 29 bytes
mswindows_redraw_exposed_area(frame * 0x0228ad90, int 8, int 23, int
249, int 127) line 862 + 25 bytes mswindows_handle_paint(frame *
0x0228ad90) line 2176 + 25 bytes mswindows_wnd_proc(HWND__ *
0x001003e2, unsigned int 15, unsigned int 0, long 0) line 3233 + 45
bytes intercepted_wnd_proc(HWND__ * 0x001003e2, unsigned int 15,
unsigned int 0, long 0) line 2488 USER32! 77e3a244() USER32! 77e14730()
USER32! 77e1558a() NTDLL! KiUserCallbackDispatcher@12 + 19 bytes
USER32! 77e14680() USER32! 77e1a792() qxeIsDialogMessage(HWND__ *
0x001003e2, tagMSG * 0x0082a93c {msg=0x0000000f wp=0x00000000
lp=0x00000000}) line 2298 + 14 bytes mswindows_is_dialog_msg(tagMSG *
0x0082a93c {msg=0x0000000f wp=0x00000000 lp=0x00000000}) line 165 + 13
bytes mswindows_drain_windows_queue(int 0) line 1282 + 9 bytes
emacs_mswindows_drain_queue() line 1326 + 7 bytes
event_stream_drain_queue() line 1887 event_stream_quit_p() line 1992
check_quit() line 993 unbind_to_hairy(int 35) line 5963 unbind_to_1(int
35, long 20888208) line 5945 + 200 bytes
specifier_instance_from_inst_list(long 21379344, long 38135616, long
36220304, long 20888208, _error_behavior_struct_ {...}, int 1, long 3)
line 2522 + 16 bytes specifier_instance(long 21379344, long 38135616,
long 36220304, _error_behavior_struct_ {...}, int 1, int 0, long 3)
line 2625 + 65 bytes specifier_instance_no_quit(long 21379344, long
38135616, long 36220304, _error_behavior_struct_ {...}, int 0, long 1)
line 2658 + 31 bytes face_property_matching_instance(long 22612340,
long 20860632, long 22530956, long 36220304, _error_behavior_struct_
{...}, int 0, long 1) line 565 + 48 bytes
ensure_face_cachel_contains_charset(face_cachel * 0x0082b014, long
36220304, long 22530956) line 1104 + 35 bytes
update_face_cachel_data(face_cachel * 0x0082b014, long 36220304, long
22612340) line 1304 + 19 bytes query_string_geometry(long 21110576,
long 22612340, int * 0x00000000, int * 0x0082b5b4, int * 0x00000000,
long 38852960) line 2370 + 23 bytes
mswindows_widget_query_string_geometry(long 21110576, long 22612340,
int * 0x0082b5b8, int * 0x0082b5b4, long 38852960) line 2914 + 25 bytes
widget_query_string_geometry(long 21110576, long 22612340, int *
0x0082b5b8, int * 0x0082b5b4, long 38852960) line 514 + 32 bytes
edit_field_query_geometry(long 38857648, int * 0x0082b7b4, int *
0x0082b7b8, image_instance_geometry IMAGE_DESIRED_GEOMETRY, long
38852960) line 920 + 390 bytes widget_query_geometry(long 38857648, int
* 0x0082b7b4, int * 0x0082b7b8, image_instance_geometry
IMAGE_DESIRED_GEOMETRY, long 38852960) line 567 + 26 bytes
image_instance_query_geometry(long 38857648, int * 0x0082b7b4, int *
0x0082b7b8, image_instance_geometry IMAGE_DESIRED_GEOMETRY, long
38852960) line 2015 + 26 bytes glyph_query_geometry(long 38853384, int
* 0x0082b7b4, int * 0x0082b7b8, image_instance_geometry
IMAGE_DESIRED_GEOMETRY, long 38852960) line 4197 + 25 bytes
layout_query_geometry(long 38852960, int * 0x0082b9cc, int *
0x0082b9d0, image_instance_geometry IMAGE_DESIRED_GEOMETRY, long
38404624) line 1351 + 25 bytes widget_query_geometry(long 38852960, int
* 0x0082b9cc, int * 0x0082b9d0, image_instance_geometry
IMAGE_DESIRED_GEOMETRY, long 38404624) line 567 + 26 bytes
image_instance_query_geometry(long 38852960, int * 0x0082b9cc, int *
0x0082b9d0, image_instance_geometry IMAGE_DESIRED_GEOMETRY, long
38404624) line 2015 + 26 bytes glyph_query_geometry(long 38537976, int
* 0x0082b9cc, int * 0x0082b9d0, image_instance_geometry
IMAGE_DESIRED_GEOMETRY, long 38404624) line 4197 + 25 bytes
layout_layout(long 38404624, int 265, int 156, int -2, int -2, long
38273064) line 1468 + 23 bytes widget_layout(long 38404624, int 265,
int 156, int -2, int -2, long 38273064) line 626 + 30 bytes
image_instance_layout(long 38404624, int 265, int 156, int -2, int -2,
long 38273064) line 2102 + 51 bytes glyph_ascent(long 38404624, long
38273064) line 4009 + 21 bytes update_glyph_cachel_data(window *
0x02480028, long 36201168, glyph_cachel * 0x0248c3d8) line 4272 + 13
bytes get_glyph_cachel_index(window * 0x02480028, long 36201168) line
4306 + 17 bytes add_glyph_rune(position_redisplay_data_type *
0x0082bf2c, glyph_block * 0x024bd028, int 0, int 0, glyph_cachel *
0x00000000) line 1800 + 15 bytes
add_glyph_runes(position_redisplay_data_type * 0x0082bf2c, int 0) line
2085 + 31 bytes create_string_text_block(window * 0x02480028, long
38776292, display_line * 0x02514500, long 0, prop_block_dynarr * *
0x0082c13c, int 2) line 4907 + 14 bytes
generate_string_display_line(window * 0x02480028, long 38776292,
display_line * 0x02514500, long 0, prop_block_dynarr * * 0x0082c13c,
int 2) line 5293 + 29 bytes generate_displayable_area(window *
0x02480028, long 38776292, int 0, int 0, int 265, int 169,
display_line_dynarr * 0x0250f228, long 0, int 2) line 5372 + 29 bytes
output_gutter(frame * 0x0228ad90, gutter_pos TOP_GUTTER, int 0) line
409 + 69 bytes update_frame_gutters(frame * 0x0228ad90) line 639 + 15
bytes redisplay_frame(frame * 0x0228ad90, int 1) line 6792 + 9 bytes
redisplay_device(device * 0x0171df00, int 1) line 6911 + 11 bytes
redisplay_without_hooks() line 6957 + 11 bytes
redisplay_no_pre_idle_hook() line 7029 redisplay() line 7011
mswindows_wnd_proc(HWND__ * 0x001003e2, unsigned int 5, unsigned int 0,
long 10223881) line 3424 intercepted_wnd_proc(HWND__ * 0x001003e2,
unsigned int 5, unsigned int 0, long 10223881) line 2488 USER32!
77e3a244() USER32! 77e16362() USER32! 77e14c1a() USER32! 77e1dd30()
mswindows_wnd_proc(HWND__ * 0x001003e2, unsigned int 71, unsigned int
0, long 8578308) line 3926 + 21 bytes intercepted_wnd_proc(HWND__ *
0x001003e2, unsigned int 71, unsigned int 0, long 8578308) line 2488
USER32! 77e3a244() USER32! 77e14730() USER32! 77e174b4() NTDLL!
KiUserCallbackDispatcher@12 + 19 bytes mswindows_set_frame_size(frame *
0x0228ad90, int 265, int 156) line 355 internal_set_frame_size(frame *
0x0228ad90, int 265, int 156, int 0) line 2754 + 24 bytes
Fset_frame_displayable_pixel_size(long 36220304, long 531, long 313,
long 20888208) line 3004 + 32 bytes Ffuncall(int 4, long * 0x0082e778)
line 3844 + 168 bytes execute_optimized_program(const unsigned char *
0x02286e48, int 40, long * 0x01529b80) line 609 + 16 bytes
funcall_compiled_function(long 22433308, int 0, long * 0x0082ec08) line
3452 + 85 bytes Ffuncall(int 1, long * 0x0082ec04) line 3883 + 17 bytes
execute_optimized_program(const unsigned char * 0x02286d40, int 6, long
* 0x01548ddc) line 609 + 16 bytes funcall_compiled_function(long
22505864, int 11, long * 0x0082f00c) line 3452 + 85 bytes Ffuncall(int
12, long * 0x0082f008) line 3883 + 17 bytes
execute_optimized_program(const unsigned char * 0x02503e38, int 47,
long * 0x0152dc48) line 609 + 16 bytes funcall_compiled_function(long
22436784, int 0, long * 0x0082f534) line 3452 + 85 bytes Ffuncall(int
1, long * 0x0082f530) line 3883 + 17 bytes apply1(long 22436784, long
20888208) line 4458 + 11 bytes Fcall_interactively(long 20742816, long
20888208, long 20888208) line 460 + 13 bytes Ffuncall(int 2, long *
0x0082f8ec) line 3844 + 127 bytes call1(long 20854392, long 20742816)
line 4489 + 11 bytes execute_command_event(command_builder *
0x01798f98, long 24439276) line 4198 + 69 bytes Fdispatch_event(long
24439276) line 4569 + 13 bytes Fcommand_loop_1() line 569 + 9 bytes
command_loop_1(long 20888208) line 489 condition_case_1(long 20886024,
long (long)* 0x010955a0 command_loop_1(long), long 20888208, long
(long, long)* 0x01095150 cmd_error(long, long), long 20888208) line
1917 + 7 bytes command_loop_3() line 251 + 35 bytes command_loop_2(long
20888208) line 264 internal_catch(long 20650992, long (long)*
0x010952c0 command_loop_2(long), long 20888208, int * volatile
0x00000000, long * volatile 0x00000000) line 1527 + 7 bytes
initial_command_loop(long 20888208) line 300 + 28 bytes
xemacs_21_5_b10_i586_pc_win32(int 1, char * * 0x00e52620, char * *
0x00e52bb0, int 0) line 2356 main(int 1, char * * 0x00e52620, char * *
0x00e52bb0) line 2733 mainCRTStartup() line 338 + 17 bytes KERNEL32!
77ea847c()


File: internals.info,  Node: Line Start Cache,  Next: Redisplay Piece by Piece,  Prev: Critical Redisplay Sections,  Up: The Redisplay Mechanism

27.2 Line Start Cache
=====================

The traditional scrolling code in Emacs breaks in a variable height
world.  It depends on the key assumption that the number of lines that
can be displayed at any given time is fixed.  This led to a complete
separation of the scrolling code from the redisplay code.  In order to
fully support variable height lines, the scrolling code must actually be
tightly integrated with redisplay.  Only redisplay can determine how
many lines will be displayed on a screen for any given starting point.

   What is ideally wanted is a complete list of the starting buffer
position for every possible display line of a buffer along with the
height of that display line.  Maintaining such a full list would be very
expensive.  We settle for having it include information for all areas
which we happen to generate anyhow (i.e. the region currently being
displayed) and for those areas we need to work with.

   In order to ensure that the cache accurately represents what
redisplay would actually show, it is necessary to invalidate it in many
situations.  If the buffer changes, the starting positions may no longer
be correct.  If a face or an extent has changed then the line heights
may have altered.  These events happen frequently enough that the cache
can end up being constantly disabled.  With this potentially constant
invalidation when is the cache ever useful?

   Even if the cache is invalidated before every single usage, it is
necessary.  Scrolling often requires knowledge about display lines which
are actually above or below the visible region.  The cache provides a
convenient light-weight method of storing this information for multiple
display regions.  This knowledge is necessary for the scrolling code to
always obey the First Golden Rule of Redisplay.

   If the cache already contains all of the information that the
scrolling routines happen to need so that it doesn't have to go
generate it, then we are able to obey the Third Golden Rule of
Redisplay.  The first thing we do to help out the cache is to always
add the displayed region.  This region had to be generated anyway, so
the cache ends up getting the information basically for free.  In those
cases where a user is simply scrolling around viewing a buffer there is
a high probability that this is sufficient to always provide the needed
information.  The second thing we can do is be smart about invalidating
the cache.

   TODO--Be smart about invalidating the cache.  Potential places:

   * Insertions at end-of-line which don't cause line-wraps do not
     alter the starting positions of any display lines.  These types of
     buffer modifications should not invalidate the cache.  This is
     actually a large optimization for redisplay speed as well.

   * Buffer modifications frequently only affect the display of lines
     at and below where they occur.  In these situations we should only
     invalidate the part of the cache starting at where the
     modification occurs.

   In case you're wondering, the Second Golden Rule of Redisplay is not
applicable.


File: internals.info,  Node: Redisplay Piece by Piece,  Next: Modules for the Redisplay Mechanism,  Prev: Line Start Cache,  Up: The Redisplay Mechanism

27.3 Redisplay Piece by Piece
=============================

As you can begin to see redisplay is complex and also not well
documented. Chuck no longer works on XEmacs so this section is my take
on the workings of redisplay.

   Redisplay happens in three phases:

  1. Determine desired display in area that needs redisplay.
     Implemented by `redisplay.c'

  2. Compare desired display with current display Implemented by
     `redisplay-output.c'

  3. Output changes Implemented by `redisplay-output.c',
     `redisplay-x.c', `redisplay-msw.c' and `redisplay-tty.c'

   Steps 1 and 2 are device-independent and relatively complex.  Step 3
is mostly device-dependent.

   Determining the desired display

   Display attributes are stored in `display_line' structures. Each
`display_line' consists of a set of `display_block''s and each
`display_block' contains a number of `rune''s. Generally dynarr's of
`display_line''s are held by each window representing the current
display and the desired display.

   The `display_line' structures are tightly tied to buffers which
presents a problem for redisplay as this connection is bogus for the
modeline. Hence the `display_line' generation routines are duplicated
for generating the modeline. This means that the modeline display code
has many bugs that the standard redisplay code does not.

   The guts of `display_line' generation are in `create_text_block',
which creates a single display line for the desired locale. This
incrementally parses the characters on the current line and generates
redisplay structures for each.

   Gutter redisplay is different. Because the data to display is stored
in a string we cannot use `create_text_block'. Instead we use
`create_text_string_block' which performs the same function as
`create_text_block' but for strings. Many of the complexities of
`create_text_block' to do with cursor handling and selective display
have been removed.


File: internals.info,  Node: Modules for the Redisplay Mechanism,  Next: Modules for other Display-Related Lisp Objects,  Prev: Redisplay Piece by Piece,  Up: The Redisplay Mechanism

27.4 Modules for the Redisplay Mechanism
========================================

     `redisplay-output.c'
     `redisplay-msw.c'
     `redisplay-tty.c'
     `redisplay-x.c'
     `redisplay.c'
     `redisplay.h'

   These files provide the redisplay mechanism.  As with many other
subsystems in XEmacs, there is a clean separation between the general
and device-specific support.

   `redisplay.c' contains the bulk of the redisplay engine.  These
functions update the redisplay structures (which describe how the screen
is to appear) to reflect any changes made to the state of any
displayable objects (buffer, frame, window, etc.) since the last time
that redisplay was called.  These functions are highly optimized to
avoid doing more work than necessary (since redisplay is called
extremely often and is potentially a huge time sink), and depend heavily
on notifications from the objects themselves that changes have occurred,
so that redisplay doesn't explicitly have to check each possible object.
The redisplay mechanism also contains a great deal of caching to further
speed things up; some of this caching is contained within the various
displayable objects.

   `redisplay-output.c' goes through the redisplay structures and
converts them into calls to device-specific methods to actually output
the screen changes.

   `redisplay-x.c' and `redisplay-tty.c' are two implementations of
these redisplay output methods, for X frames and TTY frames,
respectively.

     `indent.c'

   This module contains various functions and Lisp primitives for
converting between buffer positions and screen positions.  These
functions call the redisplay mechanism to do most of the work, and then
examine the redisplay structures to get the necessary information.  This
module needs work.

     `termcap.c'
     `terminfo.c'
     `tparam.c'

   These files contain functions for working with the termcap
(BSD-style) and terminfo (System V style) databases of terminal
capabilities and escape sequences, used when XEmacs is displaying in a
TTY.

     `cm.c'
     `cm.h'

   These files provide some miscellaneous TTY-output functions and
should probably be merged into `redisplay-tty.c'.


File: internals.info,  Node: Modules for other Display-Related Lisp Objects,  Prev: Modules for the Redisplay Mechanism,  Up: The Redisplay Mechanism

27.5 Modules for other Display-Related Lisp Objects
===================================================

     `faces.c'
     `faces.h'

     `bitmaps.h'
     `glyphs-eimage.c'
     `glyphs-msw.c'
     `glyphs-msw.h'
     `glyphs-widget.c'
     `glyphs-x.c'
     `glyphs-x.h'
     `glyphs.c'
     `glyphs.h'

     `objects-msw.c'
     `objects-msw.h'
     `objects-tty.c'
     `objects-tty.h'
     `objects-x.c'
     `objects-x.h'
     `objects.c'
     `objects.h'

     `menubar-msw.c'
     `menubar-msw.h'
     `menubar-x.c'
     `menubar.c'
     `menubar.h'

     `scrollbar-msw.c'
     `scrollbar-msw.h'
     `scrollbar-x.c'
     `scrollbar-x.h'
     `scrollbar.c'
     `scrollbar.h'

     `toolbar-msw.c'
     `toolbar-x.c'
     `toolbar.c'
     `toolbar.h'

     `font-lock.c'

   This file provides C support for syntax highlighting--i.e.
highlighting different syntactic constructs of a source file in
different colors, for easy reading.  The C support is provided so that
this is fast.

     `dgif_lib.c'
     `gif_err.c'
     `gif_lib.h'
     `gifalloc.c'

   These modules decode GIF-format image files, for use with glyphs.
These files were removed due to Unisys patent infringement concerns.


File: internals.info,  Node: Extents,  Next: Faces,  Prev: The Redisplay Mechanism,  Up: Top

28 Extents
**********

* Menu:

* Introduction to Extents::     Extents are ranges over text, with properties.
* Extent Ordering::             How extents are ordered internally.
* Format of the Extent Info::   The extent information in a buffer or string.
* Zero-Length Extents::         A weird special case.
* Mathematics of Extent Ordering::  A rigorous foundation.
* Extent Fragments::            Cached information useful for redisplay.


File: internals.info,  Node: Introduction to Extents,  Next: Extent Ordering,  Prev: Extents,  Up: Extents

28.1 Introduction to Extents
============================

Extents are regions over a buffer, with a start and an end position
denoting the region of the buffer included in the extent.  In addition,
either end can be closed or open, meaning that the endpoint is or is
not logically included in the extent.  Insertion of a character at a
closed endpoint causes the character to go inside the extent; insertion
at an open endpoint causes the character to go outside.

   Extent endpoints are stored using memory indices (see `insdel.c'),
to minimize the amount of adjusting that needs to be done when
characters are inserted or deleted.

   (Formerly, extent endpoints at the gap could be either before or
after the gap, depending on the open/closedness of the endpoint.  The
intent of this was to make it so that insertions would automatically go
inside or out of extents as necessary with no further work needing to
be done.  It didn't work out that way, however, and just ended up
complexifying and buggifying all the rest of the code.)


File: internals.info,  Node: Extent Ordering,  Next: Format of the Extent Info,  Prev: Introduction to Extents,  Up: Extents

28.2 Extent Ordering
====================

Extents are compared using memory indices.  There are two orderings for
extents and both orders are kept current at all times.  The normal or
"display" order is as follows:

     Extent A is ``less than'' extent B,
     that is, earlier in the display order,
       if:    A-start < B-start,
       or if: A-start = B-start, and A-end > B-end

   So if two extents begin at the same position, the larger of them is
the earlier one in the display order (`EXTENT_LESS' is true).

   For the e-order, the same thing holds:

     Extent A is ``less than'' extent B in e-order,
     that is, later in the buffer,
       if:    A-end < B-end,
       or if: A-end = B-end, and A-start > B-start

   So if two extents end at the same position, the smaller of them is
the earlier one in the e-order (`EXTENT_E_LESS' is true).

   The display order and the e-order are complementary orders: any
theorem about the display order also applies to the e-order if you swap
all occurrences of "display order" and "e-order", "less than" and
"greater than", and "extent start" and "extent end".


File: internals.info,  Node: Format of the Extent Info,  Next: Zero-Length Extents,  Prev: Extent Ordering,  Up: Extents

28.3 Format of the Extent Info
==============================

An extent-info structure consists of a list of the buffer or string's
extents and a "stack of extents" that lists all of the extents over a
particular position.  The stack-of-extents info is used for
optimization purposes--it basically caches some info that might be
expensive to compute.  Certain otherwise hard computations are easy
given the stack of extents over a particular position, and if the stack
of extents over a nearby position is known (because it was calculated
at some prior point in time), it's easy to move the stack of extents to
the proper position.

   Given that the stack of extents is an optimization, and given that
it requires memory, a string's stack of extents is wiped out each time
a garbage collection occurs.  Therefore, any time you retrieve the
stack of extents, it might not be there.  If you need it to be there,
use the `_force' version.

   Similarly, a string may or may not have an extent_info structure.
(Generally it won't if there haven't been any extents added to the
string.) So use the `_force' version if you need the extent_info
structure to be there.

   A list of extents is maintained as a double gap array.  One gap array
is ordered by start index (the "display order") and the other is
ordered by end index (the "e-order").  Note that positions in an extent
list should logically be conceived of as referring _to_ a particular
extent (as is the norm in programs) rather than sitting between two
extents.  Note also that callers of these functions should not be aware
of the fact that the extent list is implemented as an array, except for
the fact that positions are integers (this should be generalized to
handle integers and linked list equally well).

   A gap array is the same structure used by buffer text: an array of
elements with a "gap" somewhere in the middle.  Insertion and deletion
happens by moving the gap to the insertion/deletion point, and then
expanding/contracting as necessary.  Gap arrays have a number of useful
properties:

  1. They are space efficient, as there is no need for next/previous
     pointers.

  2. If the items in them are sorted, locating an item is fast - O(log
     N).

  3. Insertion and deletion is very fast (constant time, essentially)
     if the gap is near (which favors localized operations, as will
     usually be the case).  Even if not, it requires only a block move
     of memory, which is generally a highly optimized operation on
     modern processors.

  4. Code to manipulate them is relatively simple to write.

   An alternative would be balanced binary trees, which have guaranteed
O(log N) time for all operations (although the constant factors are not
as good, and repeated localized operations will be slower than for a
gap array).  Such code is quite tricky to write, however.


File: internals.info,  Node: Zero-Length Extents,  Next: Mathematics of Extent Ordering,  Prev: Format of the Extent Info,  Up: Extents

28.4 Zero-Length Extents
========================

Extents can be zero-length, and will end up that way if their endpoints
are explicitly set that way or if their detachable property is `nil'
and all the text in the extent is deleted. (The exception is open-open
zero-length extents, which are barred from existing because there is no
sensible way to define their properties.  Deletion of the text in an
open-open extent causes it to be converted into a closed-open extent.)
Zero-length extents are primarily used to represent annotations, and
behave as follows:

  1. Insertion at the position of a zero-length extent expands the
     extent if both endpoints are closed; goes after the extent if it
     is closed-open; and goes before the extent if it is open-closed.

  2. Deletion of a character on a side of a zero-length extent whose
     corresponding endpoint is closed causes the extent to be detached
     if it is detachable; if the extent is not detachable or the
     corresponding endpoint is open, the extent remains in the buffer,
     moving as necessary.

   Note that closed-open, non-detachable zero-length extents behave
exactly like markers and that open-closed, non-detachable zero-length
extents behave like the "point-type" marker in Mule.


File: internals.info,  Node: Mathematics of Extent Ordering,  Next: Extent Fragments,  Prev: Zero-Length Extents,  Up: Extents

28.5 Mathematics of Extent Ordering
===================================

The extents in a buffer are ordered by "display order" because that is
that order that the redisplay mechanism needs to process them in.  The
e-order is an auxiliary ordering used to facilitate operations over
extents.  The operations that can be performed on the ordered list of
extents in a buffer are

  1. Locate where an extent would go if inserted into the list.

  2. Insert an extent into the list.

  3. Remove an extent from the list.

  4. Map over all the extents that overlap a range.

   (4) requires being able to determine the first and last extents that
overlap a range.

   NOTE: "overlap" is used as follows:

   * two ranges overlap if they have at least one point in common.
     Whether the endpoints are open or closed makes a difference here.

   * a point overlaps a range if the point is contained within the
     range; this is equivalent to treating a point P as the range [P,
     P].

   * In the case of an _extent_ overlapping a point or range, the extent
     is normally treated as having closed endpoints.  This applies
     consistently in the discussion of stacks of extents and such below.
     Note that this definition of overlap is not necessarily consistent
     with the extents that `map-extents' maps over, since `map-extents'
     sometimes pays attention to whether the endpoints of an extents
     are open or closed.  But for our purposes, it greatly simplifies
     things to treat all extents as having closed endpoints.

   First, define >, <, <=, etc. as applied to extents to mean
comparison according to the display order.  Comparison between an
extent E and an index I means comparison between E and the range [I, I].

   Also define e>, e<, e<=, etc. to mean comparison according to the
e-order.

   For any range R, define R(0) to be the starting index of the range
and R(1) to be the ending index of the range.

   For any extent E, define E(next) to be the extent directly following
E, and E(prev) to be the extent directly preceding E.  Assume E(next)
and E(prev) can be determined from E in constant time.  (This is
because we store the extent list as a doubly linked list.)

   Similarly, define E(e-next) and E(e-prev) to be the extents directly
following and preceding E in the e-order.

   Now:

   Let R be a range.  Let F be the first extent overlapping R.  Let L
be the last extent overlapping R.

   Theorem 1: R(1) lies between L and L(next), i.e. L <= R(1) < L(next).

   This follows easily from the definition of display order.  The basic
reason that this theorem applies is that the display order sorts by
increasing starting index.

   Therefore, we can determine L just by looking at where we would
insert R(1) into the list, and if we know F and are moving forward over
extents, we can easily determine when we've hit L by comparing the
extent we're at to R(1).

     Theorem 2: F(e-prev) e< [1, R(0)] e<= F.

   This is the analog of Theorem 1, and applies because the e-order
sorts by increasing ending index.

   Therefore, F can be found in the same amount of time as operation
(1), i.e. the time that it takes to locate where an extent would go if
inserted into the e-order list.  This is O(log N), since we are using
gap arrays to manage extents.

   Define a "stack of extents" (or "SOE") as the set of extents
(ordered in display order and e-order, just like for normal extent
lists) that overlap an index I.

   Now:

   Let I be an index, let S be the stack of extents on I and let F be
the first extent in S.

   Theorem 3: The first extent in S is the first extent that overlaps
any range [I, J].

   Proof: Any extent that overlaps [I, J] but does not include I must
have a start index > I, and thus be greater than any extent in S.

   Therefore, finding the first extent that overlaps a range R is the
same as finding the first extent that overlaps R(0).

   Theorem 4: Let I2 be an index such that I2 > I, and let F2 be the
first extent that overlaps I2.  Then, either F2 is in S or F2 is
greater than any extent in S.

   Proof: If F2 does not include I then its start index is greater than
I and thus it is greater than any extent in S, including F.  Otherwise,
F2 includes I and thus is in S, and thus F2 >= F.


File: internals.info,  Node: Extent Fragments,  Prev: Mathematics of Extent Ordering,  Up: Extents

28.6 Extent Fragments
=====================

Imagine that the buffer is divided up into contiguous, non-overlapping
"runs" of text such that no extent starts or ends within a run (extents
that abut the run don't count).

   An extent fragment is a structure that holds data about the run that
contains a particular buffer position (if the buffer position is at the
junction of two runs, the run after the position is used)--the
beginning and end of the run, a list of all of the extents in that run,
the "merged face" that results from merging all of the faces
corresponding to those extents, the begin and end glyphs at the
beginning of the run, etc.  This is the information that redisplay needs
in order to display this run.

   Extent fragments have to be very quick to update to a new buffer
position when moving linearly through the buffer.  They rely on the
stack-of-extents code, which does the heavy-duty algorithmic work of
determining which extents overly a particular position.


File: internals.info,  Node: Faces,  Next: Glyphs,  Prev: Extents,  Up: Top

29 Faces
********

Not yet documented.


File: internals.info,  Node: Glyphs,  Next: Specifiers,  Prev: Faces,  Up: Top

30 Glyphs
*********

Glyphs are graphical elements that can be displayed in XEmacs buffers or
gutters. We use the term graphical element here in the broadest possible
sense since glyphs can be as mundane as text or as arcane as a native
tab widget.

   In XEmacs, glyphs represent the uninstantiated state of graphical
elements, i.e. they hold all the information necessary to produce an
image on-screen but the image need not exist at this stage, and multiple
screen images can be instantiated from a single glyph.

   Glyphs are lazily instantiated by calling one of the glyph
functions. This usually occurs within redisplay when `Fglyph_height' is
called. Instantiation causes an image-instance to be created and
cached. This cache is on a per-device basis for all glyphs except
widget-glyphs, and on a per-window basis for widgets-glyphs.  The
caching is done by `image_instantiate' and is necessary because it is
generally possible to display an image-instance in multiple domains.
For instance if we create a Pixmap, we can actually display this on
multiple windows - even though we only need a single Pixmap instance to
do this. If caching wasn't done then it would be necessary to create
image-instances for every displayable occurrence of a glyph - and every
usage - and this would be extremely memory and cpu intensive.

   Widget-glyphs (a.k.a native widgets) are not cached in this way.
This is because widget-glyph image-instances on screen are toolkit
windows, and thus cannot be reused in multiple XEmacs domains. Thus
widget-glyphs are cached on an XEmacs window basis.

   Any action on a glyph first consults the cache before actually
instantiating a widget.

30.1 Glyph Instantiation
========================

Glyph instantiation is a hairy topic and requires some explanation. The
guts of glyph instantiation is contained within `image_instantiate'. A
glyph contains an image which is a specifier. When a glyph function -
for instance `Fglyph_height' - asks for a property of the glyph that
can only be determined from its instantiated state, then the glyph
image is instantiated and an image instance created. The instantiation
process is governed by the specifier code and goes through a series of
steps:

   * Validation. Instantiation of image instances happens dynamically -
     often within the guts of redisplay. Thus it is often not feasible
     to catch instantiator errors at instantiation time. Instead the
     instantiator is validated at the time it is added to the image
     specifier. This function is defined by `image_validate' and at a
     simple level validates keyword value pairs.

   * Duplication. The specifier code by default takes a copy of the
     instantiator. This is reasonable for most specifiers but in the
     case of widget-glyphs can be problematic, since some of the
     properties in the instantiator - for instance callbacks - could
     cause infinite recursion in the copying process. Thus the image
     code defines a function - `image_copy_instantiator' - which will
     selectively copy values.  This is controlled by the way that a
     keyword is defined either using `IIFORMAT_VALID_KEYWORD' or
     `IIFORMAT_VALID_NONCOPY_KEYWORD'. Note that the image caching and
     redisplay code relies on instantiator copying to ensure that
     current and new instantiators are actually different rather than
     referring to the same thing.

   * Normalization. Once the instantiator has been copied it must be
     converted into a form that is viable at instantiation time. This
     can involve no changes at all, but typically involves things like
     converting file names to the actual data. This function is defined
     by `image_going_to_add' and `normalize_image_instantiator'.

   * Instantiation. When an image instance is actually required for
     display it is instantiated using `image_instantiate'. This
     involves calling instantiate methods that are specific to the type
     of image being instantiated.

   The final instantiation phase also involves a number of steps. In
order to understand these we need to describe a number of concepts.

   An image is instantiated in a "domain", where a domain can be any
one of a device, frame, window or image-instance. The domain gives the
image-instance context and identity and properties that affect the
appearance of the image-instance may be different for the same glyph
instantiated in different domains. An example is the face used to
display the image-instance.

   Although an image is instantiated in a particular domain the
instantiation domain is not necessarily the domain in which the
image-instance is cached. For example a pixmap can be instantiated in a
window be actually be cached on a per-device basis. The domain in which
the image-instance is actually cached is called the "governing-domain".
A governing-domain is currently either a device or a window.
Widget-glyphs and text-glyphs have a window as a governing-domain, all
other image-instances have a device as the governing-domain. The
governing domain for an image-instance is determined using the
governing_domain image-instance method.

30.2 Widget-Glyphs
==================

30.3 Widget-Glyphs in the MS-Windows Environment
================================================

To Do

30.4 Widget-Glyphs in the X Environment
=======================================

Widget-glyphs under X make heavy use of lwlib (*note Lucid Widget
Library::) for manipulating the native toolkit objects. This is
primarily so that different toolkits can be supported for
widget-glyphs, just as they are supported for features such as menubars
etc.

   Lwlib is extremely poorly documented and quite hairy so here is my
understanding of what goes on.

   Lwlib maintains a set of widget_instances which mirror the
hierarchical state of Xt widgets. I think this is so that widgets can
be updated and manipulated generically by the lwlib library. For
instance update_one_widget_instance can cope with multiple types of
widget and multiple types of toolkit. Each element in the widget
hierarchy is updated from its corresponding widget_instance by walking
the widget_instance tree recursively.

   This has desirable properties such as lw_modify_all_widgets which is
called from `glyphs-x.c' and updates all the properties of a widget
without having to know what the widget is or what toolkit it is from.
Unfortunately this also has hairy properties such as making the lwlib
code quite complex. And of course lwlib has to know at some level what
the widget is and how to set its properties.


File: internals.info,  Node: Specifiers,  Next: Menus,  Prev: Glyphs,  Up: Top

31 Specifiers
*************

Not yet documented.

   Specifiers are documented in depth in the Lisp Reference manual.
*Note Specifiers: (lispref)Specifiers.  The code in `specifier.c' is
pretty straightforward.


File: internals.info,  Node: Menus,  Next: Events and the Event Loop,  Prev: Specifiers,  Up: Top

32 Menus
********

A menu is set by setting the value of the variable `current-menubar'
(which may be buffer-local) and then calling `set-menubar-dirty-flag'
to signal a change.  This will cause the menu to be redrawn at the next
redisplay.  The format of the data in `current-menubar' is described in
`menubar.c'.

   Internally the data in current-menubar is parsed into a tree of
`widget_value's' (defined in `lwlib.h'); this is accomplished by the
recursive function `menu_item_descriptor_to_widget_value()', called by
`compute_menubar_data()'.  Such a tree is deallocated using
`free_widget_value()'.

   `update_screen_menubars()' is one of the external entry points.
This checks to see, for each screen, if that screen's menubar needs to
be updated.  This is the case if

  1. `set-menubar-dirty-flag' was called since the last redisplay.
     (This function sets the C variable menubar_has_changed.)

  2. The buffer displayed in the screen has changed.

  3. The screen has no menubar currently displayed.

   `set_screen_menubar()' is called for each such screen.  This
function calls `compute_menubar_data()' to create the tree of
widget_value's, then calls `lw_create_widget()',
`lw_modify_all_widgets()', and/or `lw_destroy_all_widgets()' to create
the X-Toolkit widget associated with the menu.

   `update_psheets()', the other external entry point, actually changes
the menus being displayed.  It uses the widgets fixed by
`update_screen_menubars()' and calls various X functions to ensure that
the menus are displayed properly.

   The menubar widget is set up so that `pre_activate_callback()' is
called when the menu is first selected (i.e. mouse button goes down),
and `menubar_selection_callback()' is called when an item is selected.
`pre_activate_callback()' calls the function in activate-menubar-hook,
which can change the menubar (this is described in `menubar.c').  If
the menubar is changed, `set_screen_menubars()' is called.
`menubar_selection_callback()' enqueues a menu event, putting in it a
function to call (either `eval' or `call-interactively') and its
argument, which is the callback function or form given in the menu's
description.


File: internals.info,  Node: Events and the Event Loop,  Next: Asynchronous Events; Quit Checking,  Prev: Menus,  Up: Top

33 Events and the Event Loop
****************************

* Menu:

* Introduction to Events::
* Main Loop::
* Specifics of the Event Gathering Mechanism::
* Specifics About the Emacs Event::
* Event Queues::
* Event Stream Callback Routines::
* Other Event Loop Functions::
* Stream Pairs::
* Converting Events::
* Dispatching Events; The Command Builder::
* Focus Handling::
* Editor-Level Control Flow Modules::


File: internals.info,  Node: Introduction to Events,  Next: Main Loop,  Prev: Events and the Event Loop,  Up: Events and the Event Loop

33.1 Introduction to Events
===========================

An event is an object that encapsulates information about an
interesting occurrence in the operating system.  Events are generated
either by user action, direct (e.g. typing on the keyboard or moving
the mouse) or indirect (moving another window, thereby generating an
expose event on an Emacs frame), or as a result of some other typically
asynchronous action happening, such as output from a subprocess being
ready or a timer expiring.  Events come into the system in an
asynchronous fashion (typically through a callback being called) and
are converted into a synchronous event queue (first-in, first-out) in a
process that we will call "collection".

   Note that each application has its own event queue. (It is
immaterial whether the collection process directly puts the events in
the proper application's queue, or puts them into a single system
queue, which is later split up.)

   The most basic level of event collection is done by the operating
system or window system.  Typically, XEmacs does its own event
collection as well.  Often there are multiple layers of collection in
XEmacs, with events from various sources being collected into a queue,
which is then combined with other sources to go into another queue
(i.e. a second level of collection), with perhaps another level on top
of this, etc.

   XEmacs has its own types of events (called "Emacs events"), which
provides an abstract layer on top of the system-dependent nature of the
most basic events that are received.  Part of the complex nature of the
XEmacs event collection process involves converting from the
operating-system events into the proper Emacs events--there may not be
a one-to-one correspondence.

   Emacs events are documented in `events.h'; I'll discuss them later.


File: internals.info,  Node: Main Loop,  Next: Specifics of the Event Gathering Mechanism,  Prev: Introduction to Events,  Up: Events and the Event Loop

33.2 Main Loop
==============

The "command loop" is the top-level loop that the editor is always
running.  It loops endlessly, calling `next-event' to retrieve an event
and `dispatch-event' to execute it. `dispatch-event' does the
appropriate thing with non-user events (process, timeout, magic, eval,
mouse motion); this involves calling a Lisp handler function, redrawing
a newly-exposed part of a frame, reading subprocess output, etc.  For
user events, `dispatch-event' looks up the event in relevant keymaps or
menubars; when a full key sequence or menubar selection is reached, the
appropriate function is executed. `dispatch-event' may have to keep
state across calls; this is done in the "command-builder" structure
associated with each console (remember, there's usually only one
console), and the engine that looks up keystrokes and constructs full
key sequences is called the "command builder".  This is documented
elsewhere.

   The guts of the command loop are in `command_loop_1()'.  This
function doesn't catch errors, though--that's the job of
`command_loop_2()', which is a condition-case (i.e. error-trapping)
wrapper around `command_loop_1()'.  `command_loop_1()' never returns,
but may get thrown out of.

   When an error occurs, `cmd_error()' is called, which usually invokes
the Lisp error handler in `command-error'; however, a default error
handler is provided if `command-error' is `nil' (e.g. during startup).
The purpose of the error handler is simply to display the error message
and do associated cleanup; it does not need to throw anywhere.  When
the error handler finishes, the condition-case in `command_loop_2()'
will finish and `command_loop_2()' will reinvoke `command_loop_1()'.

   `command_loop_2()' is invoked from three places: from
`initial_command_loop()' (called from `main()' at the end of internal
initialization), from the Lisp function `recursive-edit', and from
`call_command_loop()'.

   `call_command_loop()' is called when a macro is started and when the
minibuffer is entered; normal termination of the macro or minibuffer
causes a throw out of the recursive command loop. (To
`execute-kbd-macro' for macros and `exit' for minibuffers.  Note also
that the low-level minibuffer-entering function,
`read-minibuffer-internal', provides its own error handling and does
not need `command_loop_2()''s error encapsulation; so it tells
`call_command_loop()' to invoke `command_loop_1()' directly.)

   Note that both read-minibuffer-internal and recursive-edit set up a
catch for `exit'; this is why `abort-recursive-edit', which throws to
this catch, exits out of either one.

   `initial_command_loop()', called from `main()', sets up a catch for
`top-level' when invoking `command_loop_2()', allowing functions to
throw all the way to the top level if they really need to.  Before
invoking `command_loop_2()', `initial_command_loop()' calls
`top_level_1()', which handles all of the startup stuff (creating the
initial frame, handling the command-line options, loading the user's
`.emacs' file, etc.).  The function that actually does this is in Lisp
and is pointed to by the variable `top-level'; normally this function is
`normal-top-level'.  `top_level_1()' is just an error-handling wrapper
similar to `command_loop_2()'.  Note also that `initial_command_loop()'
sets up a catch for `top-level' when invoking `top_level_1()', just
like when it invokes `command_loop_2()'.


File: internals.info,  Node: Specifics of the Event Gathering Mechanism,  Next: Specifics About the Emacs Event,  Prev: Main Loop,  Up: Events and the Event Loop

33.3 Specifics of the Event Gathering Mechanism
===============================================

Here is an approximate diagram of the collection processes at work in
XEmacs, under TTY's (TTY's are simpler than X so we'll look at this
first):


      asynch.      asynch.    asynch.   asynch.             [Collectors in
     kbd events  kbd events   process   process                the OS]
           |         |         output    output
           |         |           |         |
           |         |           |         |      SIGINT,   [signal handlers
           |         |           |         |      SIGQUIT,     in XEmacs]
           V         V           V         V      SIGWINCH,
          file      file        file      file    SIGALRM
          desc.     desc.       desc.     desc.     |
          (TTY)     (TTY)       (pipe)    (pipe)    |
           |          |          |         |      fake    timeouts
           |          |          |         |      file        |
           |          |          |         |      desc.       |
           |          |          |         |      (pipe)      |
           |          |          |         |        |         |
           |          |          |         |        |         |
           |          |          |         |        |         |
           V          V          V         V        V         V
           ------>-----------<----------------<----------------
                       |
                       |
                       | [collected using `select()' in `emacs_tty_next_event()'
                       |  and converted to the appropriate Emacs event]
                       |
                       |
                       V          (above this line is TTY-specific)
                     Emacs -----------------------------------------------
                     event (below this line is the generic event mechanism)
                       |
                       |
     was there     if not, call
     a SIGINT?  `emacs_tty_next_event()'
         |             |
         |             |
         |             |
         V             V
         --->------<----
                |
                |     [collected in `event_stream_next_event()';
                |      SIGINT is converted using `maybe_read_quit_event()']
                V
              Emacs
              event
                |
                \---->------>----- maybe_kbd_translate() ---->---\
                                                                 |
                                                                 |
                                                                 |
          command event queue                                    |
                                                    if not from command
       (contains events that were                   event queue, call
       read earlier but not processed,              `event_stream_next_event()'
       typically when waiting in a                               |
       sit-for, sleep-for, etc. for                              |
      a particular event to be received)                         |
                    |                                            |
                    |                                            |
                    V                                            V
                    ---->------------------------------------<----
                                                    |
                                                    | [collected in
                                                    |  `next_event_internal()']
                                                    |
      unread-     unread-       event from          |
      command-    command-       keyboard       else, call
      events      event           macro      `next_event_internal()'
        |           |               |               |
        |           |               |               |
        |           |               |               |
        V           V               V               V
        --------->----------------------<------------
                          |
                          |      [collected in `next-event', which may loop
                          |       more than once if the event it gets is on
                          |       a dead frame, device, etc.]
                          |
                          |
                          V
                 feed into top-level event loop,
                 which repeatedly calls `next-event'
                 and then dispatches the event
                 using `dispatch-event'

   Notice the separation between TTY-specific and generic event
mechanism.  When using the Xt-based event loop, the TTY-specific stuff
is replaced but the rest stays the same.

   It's also important to realize that only one different kind of
system-specific event loop can be operating at a time, and must be able
to receive all kinds of events simultaneously.  For the two existing
event loops (implemented in `event-tty.c' and `event-Xt.c',
respectively), the TTY event loop _only_ handles TTY consoles, while
the Xt event loop handles _both_ TTY and X consoles.  This situation is
different from all of the output handlers, where you simply have one
per console type.

   Here's the Xt Event Loop Diagram (notice that below a certain point,
it's the same as the above diagram):

     asynch. asynch. asynch. asynch.                 [Collectors in
      kbd     kbd    process process                    the OS]
     events  events  output  output
       |       |       |       |
       |       |       |       |     asynch. asynch. [Collectors in the
       |       |       |       |       X        X     OS and X Window System]
       |       |       |       |     events  events
       |       |       |       |       |        |
       |       |       |       |       |        |
       |       |       |       |       |        |    SIGINT, [signal handlers
       |       |       |       |       |        |    SIGQUIT,   in XEmacs]
       |       |       |       |       |        |    SIGWINCH,
       |       |       |       |       |        |    SIGALRM
       |       |       |       |       |        |       |
       |       |       |       |       |        |       |
       |       |       |       |       |        |       |      timeouts
       |       |       |       |       |        |       |          |
       |       |       |       |       |        |       |          |
       |       |       |       |       |        |       V          |
       V       V       V       V       V        V      fake        |
      file    file    file    file    file     file    file        |
      desc.   desc.   desc.   desc.   desc.    desc.   desc.       |
      (TTY)   (TTY)   (pipe)  (pipe) (socket) (socket) (pipe)      |
       |       |       |       |       |        |       |          |
       |       |       |       |       |        |       |          |
       |       |       |       |       |        |       |          |
       V       V       V       V       V        V       V          V
       --->----------------------------------------<---------<------
            |              |               |
            |              |               |[collected using `select()' in
            |              |               | `_XtWaitForSomething()', called
            |              |               | from `XtAppProcessEvent()', called
            |              |               | in `emacs_Xt_next_event()';
            |              |               | dispatched to various callbacks]
            |              |               |
            |              |               |
       emacs_Xt_        p_s_callback(),    | [popup_selection_callback]
       event_handler()  x_u_v_s_callback(),| [x_update_vertical_scrollbar_
            |           x_u_h_s_callback(),|  callback]
            |           search_callback()  | [x_update_horizontal_scrollbar_
            |              |               |  callback]
            |              |               |
            |              |               |
       enqueue_Xt_       signal_special_   |
       dispatch_event()  Xt_user_event()   |
       [maybe multiple     |               |
        times, maybe 0     |               |
        times]             |               |
            |            enqueue_Xt_       |
            |            dispatch_event()  |
            |              |               |
            |              |               |
            V              V               |
            -->----------<--               |
                   |                       |
                   |                       |
                dispatch             `Xt_what_callback()'
                event                  sets flags
                queue                      |
                   |                       |
                   |                       |
                   |                       |
                   |                       |
                   ---->-----------<--------
                        |
                        |
                        |     [collected and converted as appropriate in
                        |            `emacs_Xt_next_event()']
                        |
                        |
                        V          (above this line is Xt-specific)
                      Emacs ------------------------------------------------
                      event (below this line is the generic event mechanism)
                        |
                        |
     was there      if not, call
     a SIGINT?   `emacs_Xt_next_event()'
         |              |
         |              |
         |              |
         V              V
         --->-------<----
                |
                |        [collected in `event_stream_next_event()';
                |         SIGINT is converted using `maybe_read_quit_event()']
                V
              Emacs
              event
                |
                \---->------>----- maybe_kbd_translate() -->-----\
                                                                 |
                                                                 |
                                                                 |
          command event queue                                    |
                                                   if not from command
       (contains events that were                  event queue, call
       read earlier but not processed,             `event_stream_next_event()'
       typically when waiting in a                               |
       sit-for, sleep-for, etc. for                              |
      a particular event to be received)                         |
                    |                                            |
                    |                                            |
                    V                                            V
                    ---->----------------------------------<------
                                                    |
                                                    | [collected in
                                                    |  `next_event_internal()']
                                                    |
      unread-     unread-       event from          |
      command-    command-       keyboard       else, call
      events      event           macro      `next_event_internal()'
        |           |               |               |
        |           |               |               |
        |           |               |               |
        V           V               V               V
        --------->----------------------<------------
                          |
                          |      [collected in `next-event', which may loop
                          |       more than once if the event it gets is on
                          |       a dead frame, device, etc.]
                          |
                          |
                          V
                 feed into top-level event loop,
                 which repeatedly calls `next-event'
                 and then dispatches the event
                 using `dispatch-event'


File: internals.info,  Node: Specifics About the Emacs Event,  Next: Event Queues,  Prev: Specifics of the Event Gathering Mechanism,  Up: Events and the Event Loop

33.4 Specifics About the Emacs Event
====================================


File: internals.info,  Node: Event Queues,  Next: Event Stream Callback Routines,  Prev: Specifics About the Emacs Event,  Up: Events and the Event Loop

33.5 Event Queues
=================

There are two event queues here - the command event queue (#### which
should be called "deferred event queue" and is in my glyph ws) and the
dispatch event queue. (MS Windows actually has an extra dispatch queue
for non-user events and uses the generic one only for user events.  This
is because user and non-user events in Windows come through the same
place - the window procedure - but under X, it's possible to
selectively process events such that we take all the user events before
the non-user ones. #### In fact, given the way we now drain the queue,
we might need two separate queues, like under Windows.  Need to think
carefully exactly how this works, and should certainly generalize the
two different queues.

   The dispatch queue (which used to occur duplicated inside of each
event implementation) is used for events that have been read from the
window-system event queue(s) and not yet process by
`next_event_internal()'.  It exists for two reasons: (1) because in many
implementations, events often come from the window system by way of
callbacks, and need to push the event to be returned onto a queue; (2)
in order to handle QUIT in a guaranteed correct fashion without
resorting to weird implementation-specific hacks that may or may not
work well, we need to drain the window-system event queues and then look
through to see if there's an event matching quit-char (usually ^G).  the
drained events need to go onto a queue. (There are other, similar cases
where we need to drain the pending events so we can look ahead - for
example, checking for pending expose events under X to avoid excessive
server activity.)

   The command event queue is used *AFTER* an event has been read from
`next_event_internal()', when it needs to be pushed back.  This
includes, for example, `accept-process-output', `sleep-for' and
`wait_delaying_user_input()'.  Eval events and the like, generated by
`enqueue-eval-event', `enqueue_magic_eval_event()', etc. are also
pushed onto this queue.  Some events generated by callbacks are also
pushed onto this queue, #### although maybe shouldn't be.

   The command queue takes precedence over the dispatch queue.

   #### It is worth investigating to see whether both queues are really
needed, and how exactly they should be used.  `enqueue-eval-event', for
example, could certainly push onto the dispatch queue, and all
callbacks maybe should.  `wait_delaying_user_input()' seems to need
both queues, since it can take events from the dispatch queue and push
them onto the command queue; but it perhaps could be rewritten to avoid
this.  #### In general we need to review the handling of these two
queues, figure out exactly what ought to be happening, and document it.


File: internals.info,  Node: Event Stream Callback Routines,  Next: Other Event Loop Functions,  Prev: Event Queues,  Up: Events and the Event Loop

33.6 Event Stream Callback Routines
===================================

There is one object called an event_stream.  This object contains
callback functions for doing the window-system-dependent operations
that XEmacs requires.

   If XEmacs is compiled with support for X11 and the X Toolkit, then
this event_stream structure will contain functions that can cope with
input on XEmacs windows on multiple displays, as well as input from
dumb tty frames.

   If it is desired to have XEmacs able to open frames on the displays
of multiple heterogeneous machines, X11 and SunView, or X11 and NeXT,
for example, then it will be necessary to construct an event_stream
structure that can cope with the given types.  Currently, the only
implemented event_streams are for dumb-ttys, and for X11 plus dumb-ttys,
and for mswindows.

   To implement this for one window system is relatively simple.  To
implement this for multiple window systems is trickier and may not be
possible in all situations, but it's been done for X and TTY.

   Note that these callbacks are *NOT* console methods; that's because
the routines are not specific to a particular console type but must be
able to simultaneously cope with all allowable console types.

   The slots of the event_stream structure:

`next_event_cb'
     A function which fills in an XEmacs_event structure with the next
     event available.  If there is no event available, then this should
     block.

     IMPORTANT: timer events and especially process events *must not* be
     returned if there are events of other types available; otherwise
     you can end up with an infinite loop in `Fdiscard_input()'.

`event_pending_cb'
     A function which says whether there are events to be read.  If
     called with an argument of 0, then this should say whether calling
     the `next_event_cb' will block.  If called with a non-zero
     argument, then this should say whether there are that many
     user-generated events pending (that is, keypresses, mouse-clicks,
     dialog-box selection events, etc.). (This is used for redisplay
     optimization, among other things.)  The difference is that the
     former includes process events and timer events, but the latter
     doesn't.

     If this function is not sure whether there are events to be read,
     it *must* return 0.  Otherwise various undesirable effects will
     occur, such as redisplay not occurring until the next event occurs.

`handle_magic_event_cb'
     XEmacs calls this with an event structure which contains
     window-system dependent information that XEmacs doesn't need to
     know about, but which must happen in order.  If the
     `next_event_cb' never returns an event of type "magic", this will
     never be used.

`format_magic_event_cb'
     Called with a magic event; print a representation of the innards
     of the event to PSTREAM.

`compare_magic_event_cb'
     Called with two magic events; return non-zero if the innards of
     the two are equal, zero otherwise.

`hash_magic_event_cb'
     Called with a magic event; return a hash of the innards of the
     event.

`add_timeout_cb'
     Called with an EMACS_TIME, the absolute time at which a wakeup
     event should be generated; and a void *, which is an arbitrary
     value that will be returned in the timeout event.  The timeouts
     generated by this function should be one-shots: they fire once and
     then disappear.  This callback should return an int id-number
     which uniquely identifies this wakeup.  If an implementation
     doesn't have microseconds or millisecond granularity, it should
     round up to the closest value it can deal with.

`remove_timeout_cb'
     Called with an int, the id number of a wakeup to discard.  This id
     number must have been returned by the `add_timeout_cb'.  If the
     given wakeup has already expired, this should do nothing.

`select_process_cb'

`unselect_process_cb'
     These callbacks tell the underlying implementation to add or
     remove a file descriptor from the list of fds which are polled for
     inferior-process input.  When input becomes available on the given
     process connection, an event of type "process" should be generated.

`select_console_cb'

`unselect_console_cb'
     These callbacks tell the underlying implementation to add or
     remove a console from the list of consoles which are polled for
     user-input.

`select_device_cb'

`unselect_device_cb'
     These callbacks are used by Unixoid event loops (those that use
     `select()' and file descriptors and have a separate input fd per
     device).

`create_io_streams_cb'

`delete_io_streams_cb'
     These callbacks are called by process code to create the input and
     output lstreams which are used for subprocess I/O.

`quitp_cb'
     A handler function called from the `QUIT' macro which should check
     whether the quit character has been typed.  On systems with SIGIO,
     this will not be called unless the `sigio_happened' flag is true
     (it is set from the SIGIO handler).

   XEmacs has its own event structures, which are distinct from the
event structures used by X or any other window system.  It is the job
of the event_stream layer to translate to this format.


File: internals.info,  Node: Other Event Loop Functions,  Next: Stream Pairs,  Prev: Event Stream Callback Routines,  Up: Events and the Event Loop

33.7 Other Event Loop Functions
===============================

`detect_input_pending()' and `input-pending-p' look for input by
calling `event_stream->event_pending_p' and looking in
`[V]unread-command-event' and the `command_event_queue' (they do not
check for an executing keyboard macro, though).

   `discard-input' cancels any command events pending (and any keyboard
macros currently executing), and puts the others onto the
`command_event_queue'.  There is a comment about a "race condition",
which is not a good sign.

   `next-command-event' and `read-char' are higher-level interfaces to
`next-event'.  `next-command-event' gets the next "command" event (i.e.
keypress, mouse event, menu selection, or scrollbar action), calling
`dispatch-event' on any others.  `read-char' calls `next-command-event'
and uses `event_to_character()' to return the character equivalent.
With the right kind of input method support, it is possible for
(read-char) to return a Kanji character.


File: internals.info,  Node: Stream Pairs,  Next: Converting Events,  Prev: Other Event Loop Functions,  Up: Events and the Event Loop

33.8 Stream Pairs
=================

Since there are many possible processes/event loop combinations, the
event code is responsible for creating an appropriate lstream type. The
process implementation does not care about that implementation.

   The Create stream pair function is passed two void* values, which
identify process-dependent 'handles'. The process implementation uses
these handles to communicate with child processes. The function must be
prepared to receive handle types of any process implementation. Since
only one process implementation exists in a particular XEmacs
configuration, preprocessing is a means of compiling in the support for
the code which deals with particular handle types.

   For example, a unixoid type loop, which relies on file descriptors,
may be asked to create a pair of streams by a unix-style process
implementation.  In this case, the handles passed are unix file
descriptors, and the code may deal with these directly. Although, the
same code may be used on Win32 system with X-Windows. In this case,
Win32 process implementation passes handles of type HANDLE, and the
`create_io_streams' function must call appropriate function to get file
descriptors given HANDLEs, so that these descriptors may be passed to
`XtAddInput'.

   The handle given may have special denying value, in which case the
corresponding lstream should not be created.

   The return value of the function is a unique stream identifier. It
is used by processes implementation, in its  platform-independent part.
There is the get_process_from_usid function, which returns process
object given its USID. The event stream is responsible for converting
its internal handle type into USID.

   Example is the TTY event stream. When a file descriptor signals
input, the event loop must determine process to which the input is
destined. Thus, the implementation uses process input stream file
descriptor as USID, by simply casting the fd value to USID type.

   There are two special USID values. One, `USID_ERROR', indicates that
the stream pair cannot be created. The second, `USID_DONTHASH',
indicates that streams are created, but the event stream does not wish
to be able to find the process by its USID. Specifically, if an event
stream implementation never calls `get_process_from_usid', this value
should always be returned, to prevent accumulating useless information
on USID to process relationship.


File: internals.info,  Node: Converting Events,  Next: Dispatching Events; The Command Builder,  Prev: Stream Pairs,  Up: Events and the Event Loop

33.9 Converting Events
======================

`character_to_event()', `event_to_character()', `event-to-character',
and `character-to-event' convert between characters and keypress events
corresponding to the characters.  If the event was not a keypress,
`event_to_character()' returns -1 and `event-to-character' returns
`nil'.  These functions convert between character representation and
the split-up event representation (keysym plus mod keys).


File: internals.info,  Node: Dispatching Events; The Command Builder,  Next: Focus Handling,  Prev: Converting Events,  Up: Events and the Event Loop

33.10 Dispatching Events; The Command Builder
=============================================

Not yet documented.


File: internals.info,  Node: Focus Handling,  Next: Editor-Level Control Flow Modules,  Prev: Dispatching Events; The Command Builder,  Up: Events and the Event Loop

33.11 Focus Handling
====================

Ben's capsule lecture on focus:

   In GNU Emacs `select-frame' never changes the window-manager frame
focus.  All it does is change the "selected frame".  This is similar to
what happens when we call `select-device' or `select-console'.
Whenever an event comes in (including a keyboard event), its frame is
selected; therefore, evaluating `select-frame' in `*scratch*' won't
cause any effects because the next received event (in the same frame)
will cause a switch back to the frame displaying `*scratch*'.

   Whenever a focus-change event is received from the window manager, it
generates a `switch-frame' event, which causes the Lisp function
`handle-switch-frame' to get run.  This basically just runs
`select-frame' (see below, however).

   In GNU Emacs, if you want to have an operation run when a frame is
selected, you supply an event binding for `switch-frame' (and then
maybe call `handle-switch-frame', or something ...).

   In XEmacs, we *do* change the window-manager frame focus as a result
of `select-frame', but not until the next time an event is received, so
that a function that momentarily changes the selected frame won't cause
WM focus flashing. (#### There's something not quite right here; this
is causing the wrong-cursor-focus problems that you occasionally see.
But the general idea is correct.) This approach is winning for people
who use the explicit-focus model, but is trickier to implement.

   We also don't make the `switch-frame' event visible but instead have
`select-frame-hook', which is a better approach.

   There is the problem of surrogate minibuffers, where when we enter
the minibuffer, you essentially want to temporarily switch the WM focus
to the frame with the minibuffer, and switch it back when you exit the
minibuffer.

   GNU Emacs solves this with the crockish `redirect-frame-focus',
which says "for keyboard events received from FRAME, act like they're
coming from FOCUS-FRAME".  I think what this means is that, when a
keyboard event comes in and the event manager is about to select the
event's frame, if that frame has its focus redirected, the redirected-to
frame is selected instead.  That way, if you're in a minibufferless
frame and enter the minibuffer, then all Lisp functions that run see the
selected frame as the minibuffer's frame rather than the minibufferless
frame you came from, so that (e.g.) your typing actually appears in the
minibuffer's frame and things behave sanely.

   There's also some weird logic that switches the redirected frame
focus from one frame to another if Lisp code explicitly calls
`select-frame' (but not if `handle-switch-frame' is called), and saves
and restores the frame focus in window configurations, etc. etc.  All
of this logic is heavily `#if 0''d, with lots of comments saying "No,
this approach doesn't seem to work, so I'm trying this ...  is it
reasonable?  Well, I'm not sure ..." that are a red flag indicating
crockishness.

   Because of our way of doing things, we can avoid all this crock.
Keyboard events never cause a select-frame (who cares what frame they're
associated with?  They come from a console, only).  We change the actual
WM focus to a surrogate minibuffer frame, so we don't have to do any
internal redirection.  In order to get the focus back, I took the
approach in `minibuf.el' of just checking to see if the frame we moved
to is still the selected frame, and move back to the old one if so.
Conceivably we might have to do the weird "tracking" that GNU Emacs does
when `select-frame' is called, but I don't think so.  If the selected
frame moved from the minibuffer frame, then we just leave it there,
figuring that someone knows what they're doing.  Because we don't have
any redirection recorded anywhere, it's safe to do this, and we don't
end up with unwanted redirection.


File: internals.info,  Node: Editor-Level Control Flow Modules,  Prev: Focus Handling,  Up: Events and the Event Loop

33.12 Editor-Level Control Flow Modules
=======================================

     `event-Xt.c'
     `event-msw.c'
     `event-stream.c'
     `event-tty.c'
     `events-mod.h'
     `gpmevent.c'
     `gpmevent.h'
     `events.c'
     `events.h'

   These implement the handling of events (user input and other system
notifications).

   `events.c' and `events.h' define the "event" Lisp object type and
primitives for manipulating it.

   `event-stream.c' implements the basic functions for working with
event queues, dispatching an event by looking it up in relevant keymaps
and such, and handling timeouts; this includes the primitives
`next-event' and `dispatch-event', as well as related primitives such
as `sit-for', `sleep-for', and `accept-process-output'.
(`event-stream.c' is one of the hairiest and trickiest modules in
XEmacs.  Beware!  You can easily mess things up here.)

   `event-Xt.c' and `event-tty.c' implement the low-level interfaces
onto retrieving events from Xt (the X toolkit) and from TTY's (using
`read()' and `select()'), respectively.  The event interface enforces a
clean separation between the specific code for interfacing with the
operating system and the generic code for working with events, by
defining an API of basic, low-level event methods; `event-Xt.c' and
`event-tty.c' are two different implementations of this API.  To add
support for a new operating system (e.g. NeXTstep), one merely needs to
provide another implementation of those API functions.

   Note that the choice of whether to use `event-Xt.c' or `event-tty.c'
is made at compile time!  Or at the very latest, it is made at startup
time.  `event-Xt.c' handles events for _both_ X and TTY frames;
`event-tty.c' is only used when X support is not compiled into XEmacs.
The reason for this is that there is only one event loop in XEmacs:
thus, it needs to be able to receive events from all different kinds of
frames.

     `keymap.c'
     `keymap.h'

   `keymap.c' and `keymap.h' define the "keymap" Lisp object type and
associated methods and primitives. (Remember that keymaps are objects
that associate event descriptions with functions to be called to
"execute" those events; `dispatch-event' looks up events in the
relevant keymaps.)

     `cmdloop.c'

   `cmdloop.c' contains functions that implement the actual editor
command loop--i.e. the event loop that cyclically retrieves and
dispatches events.  This code is also rather tricky, just like
`event-stream.c'.

     `macros.c'
     `macros.h'

   These two modules contain the basic code for defining keyboard
macros.  These functions don't actually do much; most of the code that
handles keyboard macros is mixed in with the event-handling code in
`event-stream.c'.

     `minibuf.c'

   This contains some miscellaneous code related to the minibuffer
(most of the minibuffer code was moved into Lisp by Richard Mlynarik).
This includes the primitives for completion (although filename
completion is in `dired.c'), the lowest-level interface to the
minibuffer (if the command loop were cleaned up, this too could be in
Lisp), and code for dealing with the echo area (this, too, was mostly
moved into Lisp, and the only code remaining is code to call out to
Lisp or provide simple bootstrapping implementations early in temacs,
before the echo-area Lisp code is loaded).


File: internals.info,  Node: Asynchronous Events; Quit Checking,  Next: Lstreams,  Prev: Events and the Event Loop,  Up: Top

34 Asynchronous Events; Quit Checking
*************************************

* Menu:

* Signal Handling::
* Control-G (Quit) Checking::
* Profiling::
* Asynchronous Timeouts::
* Exiting::


File: internals.info,  Node: Signal Handling,  Next: Control-G (Quit) Checking,  Prev: Asynchronous Events; Quit Checking,  Up: Asynchronous Events; Quit Checking

34.1 Signal Handling
====================


File: internals.info,  Node: Control-G (Quit) Checking,  Next: Profiling,  Prev: Signal Handling,  Up: Asynchronous Events; Quit Checking

34.2 Control-G (Quit) Checking
==============================

_Note_: The code to handle QUIT is divided between `lisp.h' and
`signal.c'.  There is also some special-case code in the async timer
code in `event-stream.c' to notice when the poll-for-quit (and
poll-for-sigchld) timers have gone off.

   Here's an overview of how this convoluted stuff works:

  1.  Scattered throughout the XEmacs core code are calls to the macro
     QUIT; This macro checks to see whether a `C-g' has recently been
     pressed and not yet handled, and if so, it handles the `C-g' by
     calling `signal_quit()', which invokes the standard `Fsignal()'
     code, with the error being `Qquit'.  Lisp code can establish
     handlers for this (using `condition-case'), but normally there is
     no handler, and so execution is thrown back to the innermost
     enclosing event loop. (One of the things that happens when
     entering an event loop is that a `condition-case' is established
     that catches *all* calls to `signal', including this one.)

  2. How does the QUIT macro check to see whether `C-g' has been
     pressed; obviously this needs to be extremely fast.  Now for some
     history.  In early Lemacs as inherited from the FSF going back 15
     years or more, there was a great fondness for using SIGIO (which
     is sent whenever there is I/O available on a given socket, tty,
     etc.).  In fact, in GNU Emacs, perhaps even today, all reading of
     events from the X server occurs inside the SIGIO handler!  This is
     crazy, but not completely relevant.  What is relevant is that
     similar stuff happened inside the SIGIO handler for `C-g': it
     searched through all the pending (i.e. not yet delivered to XEmacs
     yet) X events for one that matched `C-g'.  When it saw a match, it
     set Vquit_flag to Qt.  On TTY's, `C-g' is actually mapped to be the
     interrupt character (i.e. it generates SIGINT), and XEmacs's
     handler for this signal sets Vquit_flag to Qt.  Then, sometime
     later after the signal handlers finished and a QUIT macro was
     called, the macro noticed the setting of `Vquit_flag' and used
     this as an indication to call `signal_quit()'.  What
     `signal_quit()' actually does is set `Vquit_flag' to Qnil (so that
     we won't get repeated interruptions from a single `C-g' press) and
     then calls the equivalent of (signal 'quit nil).

  3. Another complication is introduced in that Vquit_flag is actually
     exported to Lisp as `quit-flag'.  This allows users some level of
     control over whether and when `C-g' is processed as quit, esp. in
     combination with `inhibit-quit'.  This is another Lisp variable,
     and if set to non-nil, it inhibits `signal_quit()' from getting
     called, meaning that the `C-g' gets essentially ignored.  But not
     completely: Because the resetting of `quit-flag' happens only in
     `signal_quit()', which isn't getting called, the `C-g' press is
     still noticed, and as soon as `inhibit-quit' is set back to nil, a
     quit will be signalled at the next QUIT macro.  Thus, what
     `inhibit-quit' really does is defer quits until after the quit-
     inhibitted period.

  4. Another consideration, introduced by XEmacs, is critical quitting.
     If you press `Control-Shift-G' instead of just `C-g', `quit-flag'
     is set to `critical' instead of to t.  When QUIT processes this
     value, it *ignores* the value of `inhibit-quit'.  This allows you
     to quit even out of a quit-inhibitted section of code!
     Furthermore, when `signal_quit()' notices that it was invoked as a
     result of a critical quit, it automatically invokes the debugger
     (which otherwise would only happen when `debug-on-quit' is set to
     t).

  5. Well, I explained above about how `quit-flag' gets set correctly,
     but I began with a disclaimer stating that this was the old way of
     doing things.  What's done now?  Well, first of all, the SIGIO
     handler (which formerly checked all pending events to see if
     there's a `C-g') now does nothing but set a flag - or actually two
     flags, something_happened and quit_check_signal_happened.  There
     are two flags because the QUIT macro is now used for more than
     just handling QUIT; it's also used for running asynchronous
     timeout handlers that have recently expired, and perhaps other
     things.  The idea here is that the QUIT macros occur extremely
     often in the code, but only occur at places that are relatively
     safe - in particular, if an error occurs, nothing will get
     completely trashed.

  6. Now, let's look at QUIT again.

  7.  UNFINISHED.  Note, however, that as of the point when this
     comment got committed to CVS (mid-2001), the interaction between
     reading `C-g' as an event and processing it as QUIT was overhauled
     to (for the first time) be understandable and actually work
     correctly.  Now, the way things work is that if `C-g' is pressed
     while XEmacs is blocking at the top level, waiting for a user
     event, it will be read as an event; otherwise, it will cause QUIT.
     (This includes times when XEmacs is blocking, but not waiting for
     a user event, e.g. `accept-process-output' and
     `wait_delaying_user_events()'.)  Formerly, this was supposed to
     happen, but didn't always due to a bizarre and broken scheme,
     documented in `next_event_internal' like this:

          If we read a `C-g', then set `quit-flag' but do not discard
          the `C-g'.  The callers of `next_event_internal()' will do
          one of two things:

            1. set `Vquit_flag' to Qnil. (`next-event' does this.) This
               will cause the ^G to be treated as a normal keystroke.

            2. not change `Vquit_flag' but attempt to enqueue the ^G,
               at which point it will be discarded.  The next time QUIT
               is called, it will notice that `Vquit_flag' was set.

     This required weirdness in `enqueue_command_event_1' like this:

          put the event on the typeahead queue, unless the event is the
          quit char, in which case the `QUIT' which will occur on the
          next trip through this loop is all the processing we should
          do - leaving it on the queue would cause the quit to be
          processed twice.

     And further weirdness elsewhere, none of which made any sense, and
     didn't work, because (e.g.) it required that QUIT never happen
     anywhere inside `next_event_internal()' or any callers when `C-g'
     should be read as a user event, which was impossible to implement
     in practice.

     Now what we do is fairly simple.  Callers of
     `next_event_internal()' that want `C-g' read as a user event call
     `begin_dont_check_for_quit()'.  `next_event_internal()', when it
     gets a `C-g', simply sets `Vquit_flag' (just as when a `C-g' is
     detected during the operation of `QUIT' or `QUITP'), and then
     tries to `QUIT'.  This will fail if blocked by the previous call,
     at which point `next_event_internal()' will return the `C-g' as an
     event.  To unblock things, first set `Vquit_flag' to nil (it was
     set to t when the `C-g' was read, and if we don't reset it, the
     next call to `QUIT' will quit), and then `unbind_to()' the depth
     returned by `begin_dont_check_for_quit()'.  It makes no difference
     is `QUIT' is called a zillion times in `next_event_internal()' or
     anywhere else, because it's blocked and will never signal.

34.2.1 Reentrancy Problems due to QUIT Checking
-----------------------------------------------

Checking for QUIT can do quite a long of things - since it pumps the
event loop, this may cause arbitrary code to get executed, garbage
collection to happen. etc. (In fact, garbage collection cannot happen
because it is inhibited.) This has led to crashes when functions get
called reentrantly when not expecting it.  Example:

Crash - reentrant `re_match_2()'
--------------------------------

       /* dont_check_for_quit is set in three circumstances:

          (1) when we are in the process of changing the window
          configuration.  The frame might be in an inconsistent state,
          which will cause assertion failures if we check for QUIT.

          (2) when we are reading events, and want to read the C-g
          as an event.  The normal check for quit will discard the C-g,
          which would be bad.

          (3) when we're going down with a fatal error.  we're most likely
          in an inconsistent state, and we definitely don't want to be
          interrupted. */

       /* We should *not* conditionalize on Vinhibit_quit, or
          critical-quit (Control-Shift-G) won't work right. */

       /* WARNING: Even calling check_quit(), without actually dispatching
          a quit signal, can result in arbitrary Lisp code getting executed
          -- at least under Windows. (Not to mention obvious Lisp
          invocations like asynchronous timer callbacks.) Here's a sample
          stack trace to demonstrate:

      NTDLL! DbgBreakPoint@0 address 0x77f9eea9
     assert_failed(const char * 0x012d036c, int 4596, const char * 0x012d0354) line 3478
     re_match_2_internal(re_pattern_buffer * 0x012d6780, const unsigned char * 0x00000000, int 0, const unsigned char * 0x022f9328, int 34, int 0, re_registers * 0x012d53d0 search_regs, int 34) line 4596 + 41 bytes
     re_search_2(re_pattern_buffer * 0x012d6780, const char * 0x00000000, int 0, const char * 0x022f9328, int 34, int 0, int 34, re_registers * 0x012d53d0 search_regs, int 34) line 4269 + 37 bytes
     re_search(re_pattern_buffer * 0x012d6780, const char * 0x022f9328, int 34, int 0, int 34, re_registers * 0x012d53d0 search_regs) line 4031 + 37 bytes
     string_match_1(long 31222628, long 30282164, long 28377092, buffer * 0x022fde00, int 0) line 413 + 69 bytes
     Fstring_match(long 31222628, long 30282164, long 28377092, long 28377092) line 436 + 34 bytes
     Ffuncall(int 3, long * 0x008297f8) line 3488 + 168 bytes
     execute_optimized_program(const unsigned char * 0x020ddc50, int 6, long * 0x020ddf50) line 744 + 16 bytes
     funcall_compiled_function(long 34407748, int 1, long * 0x00829aec) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x00829ae8) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020ddc90, int 4, long * 0x020ddf90) line 744 + 16 bytes
     funcall_compiled_function(long 34407720, int 1, long * 0x00829e28) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x00829e24) line 3523 + 17 bytes
     mapcar1(long 15, long * 0x00829e48, long 34447820, long 34187868) line 2929 + 11 bytes
     Fmapcar(long 34447820, long 34187868) line 3035 + 21 bytes
     Ffuncall(int 3, long * 0x00829f20) line 3488 + 93 bytes
     execute_optimized_program(const unsigned char * 0x020c2b70, int 7, long * 0x020dd010) line 744 + 16 bytes
     funcall_compiled_function(long 34407580, int 2, long * 0x0082a210) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082a20c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020cf810, int 6, long * 0x020cfb10) line 744 + 16 bytes
     funcall_compiled_function(long 34407524, int 0, long * 0x0082a580) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082a57c) line 3523 + 17 bytes
     run_hook_with_args_in_buffer(buffer * 0x022fde00, int 1, long * 0x0082a57c, int 0) line 3980 + 13 bytes
     run_hook_with_args(int 1, long * 0x0082a57c, int 0) line 3993 + 23 bytes
     Frun_hooks(int 1, long * 0x0082a57c) line 3847 + 19 bytes
     run_hook(long 34447484) line 4094 + 11 bytes
     unsafe_handle_wm_initmenu_1(frame * 0x01dbb000) line 736 + 11 bytes
     unsafe_handle_wm_initmenu(long 28377092) line 807 + 11 bytes
     condition_case_1(long 28377116, long (long)* 0x0101c827 unsafe_handle_wm_initmenu(long), long 28377092, long (long, long)* 0x01005fa4 mswindows_modal_loop_error_handler(long, long), long 28377092) line 1692 + 7 bytes
     mswindows_protect_modal_loop(long (long)* 0x0101c827 unsafe_handle_wm_initmenu(long), long 28377092) line 1194 + 32 bytes
     mswindows_handle_wm_initmenu(HMENU__ * 0x00010199, frame * 0x01dbb000) line 826 + 17 bytes
     mswindows_wnd_proc(HWND__ * 0x000501da, unsigned int 278, unsigned int 65945, long 0) line 3089 + 31 bytes
     USER32! UserCallWinProc@20 + 24 bytes
     USER32! DispatchClientMessage@20 + 47 bytes
     USER32! __fnDWORD@4 + 34 bytes
     NTDLL! KiUserCallbackDispatcher@12 + 19 bytes
     USER32! DispatchClientMessage@20 address 0x77e163cc
     USER32! DefWindowProcW@16 + 34 bytes
     qxeDefWindowProc(HWND__ * 0x000501da, unsigned int 274, unsigned int 61696, long 98) line 1188 + 22 bytes
     mswindows_wnd_proc(HWND__ * 0x000501da, unsigned int 274, unsigned int 61696, long 98) line 3362 + 21 bytes
     USER32! UserCallWinProc@20 + 24 bytes
     USER32! DispatchClientMessage@20 + 47 bytes
     USER32! __fnDWORD@4 + 34 bytes
     NTDLL! KiUserCallbackDispatcher@12 + 19 bytes
     USER32! DispatchClientMessage@20 address 0x77e163cc
     USER32! DefWindowProcW@16 + 34 bytes
     qxeDefWindowProc(HWND__ * 0x000501da, unsigned int 262, unsigned int 98, long 540016641) line 1188 + 22 bytes
     mswindows_wnd_proc(HWND__ * 0x000501da, unsigned int 262, unsigned int 98, long 540016641) line 3362 + 21 bytes
     USER32! UserCallWinProc@20 + 24 bytes
     USER32! DispatchMessageWorker@8 + 244 bytes
     USER32! DispatchMessageW@4 + 11 bytes
     qxeDispatchMessage(const tagMSG * 0x0082c684 {msg=0x00000106 wp=0x00000062 lp=0x20300001}) line 989 + 10 bytes
     mswindows_drain_windows_queue() line 1345 + 9 bytes
     emacs_mswindows_quit_p() line 3947
     event_stream_quit_p() line 666
     check_quit() line 686
     check_what_happened() line 437
     re_match_2_internal(re_pattern_buffer * 0x012d5a18, const unsigned char * 0x00000000, int 0, const unsigned char * 0x02235000, int 23486, int 14645, re_registers * 0x012d53d0 search_regs, int 23486) line 4717 + 14 bytes
     re_search_2(re_pattern_buffer * 0x012d5a18, const char * 0x02235000, int 23486, const char * 0x0223b38e, int 0, int 14645, int 8841, re_registers * 0x012d53d0 search_regs, int 23486) line 4269 + 37 bytes
     search_buffer(buffer * 0x022fde00, long 29077572, long 13789, long 23487, long 1, int 1, long 28377092, long 28377092, int 0) line 1224 + 89 bytes
     search_command(long 29077572, long 46975, long 28377116, long 28377092, long 28377092, int 1, int 1, int 0) line 1054 + 151 bytes
     Fre_search_forward(long 29077572, long 46975, long 28377116, long 28377092, long 28377092) line 2147 + 31 bytes
     Ffuncall(int 4, long * 0x0082ceb0) line 3488 + 216 bytes
     execute_optimized_program(const unsigned char * 0x02047810, int 13, long * 0x02080c10) line 744 + 16 bytes
     funcall_compiled_function(long 34187208, int 3, long * 0x0082d1b8) line 516 + 53 bytes
     Ffuncall(int 4, long * 0x0082d1b4) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x01e96a10, int 6, long * 0x020ae510) line 744 + 16 bytes
     funcall_compiled_function(long 34186676, int 3, long * 0x0082d4a0) line 516 + 53 bytes
     Ffuncall(int 4, long * 0x0082d49c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x02156b50, int 4, long * 0x020c2db0) line 744 + 16 bytes
     funcall_compiled_function(long 34186564, int 2, long * 0x0082d780) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082d77c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x0082d964, int 3, long * 0x020c2d70) line 744 + 16 bytes
     Fbyte_code(long 29405156, long 34352480, long 7) line 2392 + 38 bytes
     Feval(long 34354440) line 3290 + 187 bytes
     condition_case_1(long 34354572, long (long)* 0x01087232 Feval(long), long 34354440, long (long, long)* 0x01084764 run_condition_case_handlers(long, long), long 28377092) line 1692 + 7 bytes
     condition_case_3(long 34354440, long 28377092, long 34354572) line 1779 + 27 bytes
     execute_rare_opcode(long * 0x0082dc7c, const unsigned char * 0x01b090af, int 143) line 1269 + 19 bytes
     execute_optimized_program(const unsigned char * 0x01b09090, int 6, long * 0x020ae590) line 654 + 17 bytes
     funcall_compiled_function(long 34186620, int 0, long * 0x0082df68) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082df64) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x02195470, int 1, long * 0x020c2df0) line 744 + 16 bytes
     funcall_compiled_function(long 34186508, int 0, long * 0x0082e23c) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082e238) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x01e5d410, int 6, long * 0x0207d410) line 744 + 16 bytes
     funcall_compiled_function(long 34186312, int 1, long * 0x0082e524) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x0082e520) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x02108fb0, int 2, long * 0x020c2e30) line 744 + 16 bytes
     funcall_compiled_function(long 34186340, int 0, long * 0x0082e7fc) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082e7f8) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020fe150, int 2, long * 0x01e6f510) line 744 + 16 bytes
     funcall_compiled_function(long 31008124, int 0, long * 0x0082ebd8) line 516 + 53 bytes
     Ffuncall(int 1, long * 0x0082ebd4) line 3523 + 17 bytes
     run_hook_with_args_in_buffer(buffer * 0x022fde00, int 1, long * 0x0082ebd4, int 0) line 3980 + 13 bytes
     run_hook_with_args(int 1, long * 0x0082ebd4, int 0) line 3993 + 23 bytes
     Frun_hooks(int 1, long * 0x0082ebd4) line 3847 + 19 bytes
     Ffuncall(int 2, long * 0x0082ebd0) line 3509 + 14 bytes
     execute_optimized_program(const unsigned char * 0x01ef2210, int 5, long * 0x01da8e10) line 744 + 16 bytes
     funcall_compiled_function(long 31020440, int 2, long * 0x0082eeb8) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082eeb4) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x0082f09c, int 3, long * 0x01d89390) line 744 + 16 bytes
     Fbyte_code(long 31102388, long 30970752, long 7) line 2392 + 38 bytes
     Feval(long 31087568) line 3290 + 187 bytes
     condition_case_1(long 30961240, long (long)* 0x01087232 Feval(long), long 31087568, long (long, long)* 0x01084764 run_condition_case_handlers(long, long), long 28510180) line 1692 + 7 bytes
     condition_case_3(long 31087568, long 28510180, long 30961240) line 1779 + 27 bytes
     execute_rare_opcode(long * 0x0082f450, const unsigned char * 0x01ef23ec, int 143) line 1269 + 19 bytes
     execute_optimized_program(const unsigned char * 0x01ef2310, int 6, long * 0x01da8f10) line 654 + 17 bytes
     funcall_compiled_function(long 31020412, int 1, long * 0x0082f740) line 516 + 53 bytes
     Ffuncall(int 2, long * 0x0082f73c) line 3523 + 17 bytes
     execute_optimized_program(const unsigned char * 0x020fe650, int 3, long * 0x01d8c490) line 744 + 16 bytes
     funcall_compiled_function(long 31020020, int 2, long * 0x0082fa14) line 516 + 53 bytes
     Ffuncall(int 3, long * 0x0082fa10) line 3523 + 17 bytes
     Fcall_interactively(long 29685180, long 28377092, long 28377092) line 1008 + 22 bytes
     Fcommand_execute(long 29685180, long 28377092, long 28377092) line 2929 + 17 bytes
     execute_command_event(command_builder * 0x01be1900, long 36626492) line 4048 + 25 bytes
     Fdispatch_event(long 36626492) line 4341 + 70 bytes
     Fcommand_loop_1() line 582 + 9 bytes
     command_loop_1(long 28377092) line 495
     condition_case_1(long 28377188, long (long)* 0x01064fb9 command_loop_1(long), long 28377092, long (long, long)* 0x010649d0 cmd_error(long, long), long 28377092) line 1692 + 7 bytes
     command_loop_3() line 256 + 35 bytes
     command_loop_2(long 28377092) line 269
     internal_catch(long 28457612, long (long)* 0x01064b20 command_loop_2(long), long 28377092, int * volatile 0x00000000) line 1317 + 7 bytes
     initial_command_loop(long 28377092) line 305 + 25 bytes
     STACK_TRACE_EYE_CATCHER(int 1, char * * 0x01b63ff0, char * * 0x01ca5300, int 0) line 2501
     main(int 1, char * * 0x01b63ff0, char * * 0x01ca5300) line 2938
     XEMACS! mainCRTStartup + 180 bytes
     _start() line 171
     KERNEL32! BaseProcessStart@4 + 115547 bytes

   [explain dont_check_for_quit() et al]


File: internals.info,  Node: Profiling,  Next: Asynchronous Timeouts,  Prev: Control-G (Quit) Checking,  Up: Asynchronous Events; Quit Checking

34.3 Profiling
==============

We implement our own profiling scheme so that we can determine things
like which Lisp functions are occupying the most time.  Any standard
OS-provided profiling works on C functions, which is not always that
useful - and inconvenient, since it requires compiling with profile
info and can't be retrieved dynamically, as XEmacs is running.

   The basic idea is simple.  We set a profiling timer using setitimer
(ITIMER_PROF), which generates a SIGPROF every so often.  (This runs not
in real time but rather when the process is executing or the system is
running on behalf of the process - at least, that is the case under
Unix.  Under MS Windows and Cygwin, there is no `setitimer()', so we
simulate it using multimedia timers, which run in real time.  To make
the results a bit more realistic, we ignore ticks that go off while
blocking on an event wait.  Note that Cygwin does provide a simulation
of `setitimer()', but it's in real time anyway, since Windows doesn't
provide a way to have process-time timers, and furthermore, it's broken,
so we don't use it.) When the signal goes off, we see what we're in, and
add 1 to the count associated with that function.

   It would be nice to use the Lisp allocation mechanism etc. to keep
track of the profiling information (i.e. to use Lisp hash tables), but
we can't because that's not safe - updating the timing information
happens inside of a signal handler, so we can't rely on not being in
the middle of Lisp allocation, garbage collection, `malloc()', etc.
Trying to make it work would be much more work than it's worth.
Instead we use a basic (non-Lisp) hash table, which will not conflict
with garbage collection or anything else as long as it doesn't try to
resize itself.  Resizing itself, however (which happens as a result of
a `puthash()'), could be deadly.  To avoid this, we make sure, at
points where it's safe (e.g. `profile_record_about_to_call()' -
recording the entry into a function call), that the table always has
some breathing room in it so that no resizes will occur until at least
that many items are added.  This is safe because any new item to be
added in the sigprof would likely have the
`profile_record_about_to_call()' called just before it, and the
breathing room is checked.

   In general: any entry that the sigprof handler puts into the table
comes from a backtrace frame (except "Processing Events at Top Level",
and there's only one of those).  Either that backtrace frame was added
when profiling was on (in which case `profile_record_about_to_call()'
was called and the breathing space updated), or when it was off - and in
this case, no such frames can have been added since the last time
`start-profile' was called, so when `start-profile' is called we make
sure there is sufficient breathing room to account for all entries
currently on the stack.

   Jan 1998: In addition to timing info, I have added code to remember
call counts of Lisp funcalls.  The `profile_increase_call_count()'
function is called from `Ffuncall()', and serves to add data to
Vcall_count_profile_table.  This mechanism is much simpler and
independent of the SIGPROF-driven one.  It uses the Lisp allocation
mechanism normally, since it is not called from a handler.  It may even
be useful to provide a way to turn on only one profiling mechanism, but
I haven't done so yet.  -hniksic

   Dec 2002: Total overhaul of the interface, making it sane and easier
to use. -ben

   Feb 2003: Lots of rewriting of the internal code.  Add
GC-consing-usage, total GC usage, and total timing to the information
tracked.  Track profiling overhead and allow the ability to have
internal sections (e.g. internal-external conversion, byte-char
conversion) that are treated like Lisp functions for the purpose of
profiling.  -ben

   BEWARE: If you are modifying this file, be *very* careful.  Correctly
implementing the "total" values is very tricky due to the possibility of
recursion and of functions already on the stack when starting to
profile/still on the stack when stopping.


File: internals.info,  Node: Asynchronous Timeouts,  Next: Exiting,  Prev: Profiling,  Up: Asynchronous Events; Quit Checking

34.4 Asynchronous Timeouts
==========================


File: internals.info,  Node: Exiting,  Prev: Asynchronous Timeouts,  Up: Asynchronous Events; Quit Checking

34.5 Exiting
============

Ben's capsule summary about expected and unexpected exits from XEmacs.

   Expected exits occur when the user directs XEmacs to exit, for
example by pressing the close button on the only frame in XEmacs, or by
typing `C-x C-c'.  This runs `save-buffers-kill-emacs', which saves any
necessary buffers, and then exits using the primitive `kill-emacs'.

   However, unexpected exits occur in a few different ways:

   * A memory access violation or other hardware-generated exception
     occurs.  This is the worst possible problem to deal with, because
     the fault can occur while XEmacs is in any state whatsoever, even
     quite unstable ones.  As a result, we need to be *extremely*
     careful what we do.

   * We are using one X display (or if we've used more, we've closed the
     others already), and some hardware or other problem happens and
     suddenly we've lost our connection to the display.  In this
     situation, things are not so dire as in the last one; our code
     itself isn't trashed, so we can continue execution as normal,
     after having set things up so that we can exit at the appropriate
     time.  Our exit still needs to be of the emergency nature; we have
     no displays, so any attempts to use them will fail.  We simply
     want to auto-save (the single most important thing to do during
     shut-down), do minimal cleanup of stuff that has an independent
     existence outside of XEmacs, and exit.

   Currently, both unexpected exit scenarios described above set
`preparing_for_armageddon' to indicate that nonessential and possibly
dangerous things should not be done, specifically:

   - no garbage collection.

   - no hooks are run.

   - no messages of any sort from autosaving.

   - autosaving tries harder, ignoring certain failures.

   - existing frames are not deleted.

   (Also, all places that set `preparing_for_armageddon' also set
`dont_check_for_quit'.  This happens separately because it's also
necessary to set other variables to make absolutely sure no quitting
happens.)

   In the first scenario above (the access violation), we also set
`fatal_error_in_progress'.  This causes more things to not happen:

   - assertion failures do not abort.

   - printing code does not do code conversion or gettext when printing
     to stdout/stderr.


File: internals.info,  Node: Lstreams,  Next: Subprocesses,  Prev: Asynchronous Events; Quit Checking,  Up: Top

35 Lstreams
***********

An "lstream" is an internal Lisp object that provides a generic
buffering stream implementation.  Conceptually, you send data to the
stream or read data from the stream, not caring what's on the other end
of the stream.  The other end could be another stream, a file
descriptor, a stdio stream, a fixed block of memory, a reallocating
block of memory, etc.  The main purpose of the stream is to provide a
standard interface and to do buffering.  Macros are defined to read or
write characters, so the calling functions do not have to worry about
blocking data together in order to achieve efficiency.

* Menu:

* Creating an Lstream::         Creating an lstream object.
* Lstream Types::               Different sorts of things that are streamed.
* Lstream Functions::           Functions for working with lstreams.
* Lstream Methods::             Creating new lstream types.


File: internals.info,  Node: Creating an Lstream,  Next: Lstream Types,  Prev: Lstreams,  Up: Lstreams

35.1 Creating an Lstream
========================

Lstreams come in different types, depending on what is being interfaced
to.  Although the primitive for creating new lstreams is
`Lstream_new()', generally you do not call this directly.  Instead, you
call some type-specific creation function, which creates the lstream
and initializes it as appropriate for the particular type.

   All lstream creation functions take a MODE argument, specifying what
mode the lstream should be opened as.  This controls whether the
lstream is for input and output, and optionally whether data should be
blocked up in units of MULE characters.  Note that some types of
lstreams can only be opened for input; others only for output; and
others can be opened either way.  #### Richard Mlynarik thinks that
there should be a strict separation between input and output streams,
and he's probably right.

   MODE is a string, one of

`"r"'
     Open for reading.

`"w"'
     Open for writing.

`"rc"'
     Open for reading, but "read" never returns partial MULE characters.

`"wc"'
     Open for writing, but never writes partial MULE characters.


File: internals.info,  Node: Lstream Types,  Next: Lstream Functions,  Prev: Creating an Lstream,  Up: Lstreams

35.2 Lstream Types
==================

stdio

filedesc

lisp-string

fixed-buffer

resizing-buffer

dynarr

lisp-buffer

print

decoding

encoding


File: internals.info,  Node: Lstream Functions,  Next: Lstream Methods,  Prev: Lstream Types,  Up: Lstreams

35.3 Lstream Functions
======================

 -- Function: Lstream * Lstream_new (Lstream_implementation *IMP, const
          char *MODE)
     Allocate and return a new Lstream.  This function is not really
     meant to be called directly; rather, each stream type should
     provide its own stream creation function, which creates the stream
     and does any other necessary creation stuff (e.g. opening a file).

 -- Function: void Lstream_set_buffering (Lstream *LSTR,
          Lstream_buffering BUFFERING, int BUFFERING_SIZE)
     Change the buffering of a stream.  See `lstream.h'.  By default the
     buffering is `STREAM_BLOCK_BUFFERED'.

 -- Function: int Lstream_flush (Lstream *LSTR)
     Flush out any pending unwritten data in the stream.  Clear any
     buffered input data.  Returns 0 on success, -1 on error.

 -- Macro: int Lstream_putc (Lstream *STREAM, int C)
     Write out one byte to the stream.  This is a macro and so it is
     very efficient.  The C argument is only evaluated once but the
     STREAM argument is evaluated more than once.  Returns 0 on
     success, -1 on error.

 -- Macro: int Lstream_getc (Lstream *STREAM)
     Read one byte from the stream.  This is a macro and so it is very
     efficient.  The STREAM argument is evaluated more than once.
     Return value is -1 for EOF or error.

 -- Macro: void Lstream_ungetc (Lstream *STREAM, int C)
     Push one byte back onto the input queue.  This will be the next
     byte read from the stream.  Any number of bytes can be pushed back
     and will be read in the reverse order they were pushed back--most
     recent first. (This is necessary for consistency--if there are a
     number of bytes that have been unread and I read and unread a
     byte, it needs to be the first to be read again.) This is a macro
     and so it is very efficient.  The C argument is only evaluated
     once but the STREAM argument is evaluated more than once.

 -- Function: int Lstream_fputc (Lstream *STREAM, int C)
 -- Function: int Lstream_fgetc (Lstream *STREAM)
 -- Function: void Lstream_fungetc (Lstream *STREAM, int C)
     Function equivalents of the above macros.

 -- Function: Bytecount Lstream_read (Lstream *STREAM, void *DATA,
          Bytecount SIZE)
     Read SIZE bytes of DATA from the stream.  Return the number of
     bytes read.  0 means EOF. -1 means an error occurred and no bytes
     were read.

 -- Function: Bytecount Lstream_write (Lstream *STREAM, void *DATA,
          Bytecount SIZE)
     Write SIZE bytes of DATA to the stream.  Return the number of
     bytes written.  -1 means an error occurred and no bytes were
     written.

 -- Function: void Lstream_unread (Lstream *STREAM, void *DATA,
          Bytecount SIZE)
     Push back SIZE bytes of DATA onto the input queue.  The next call
     to `Lstream_read()' with the same size will read the same bytes
     back.  Note that this will be the case even if there is other
     pending unread data.

 -- Function: int Lstream_close (Lstream *STREAM)
     Close the stream.  All data will be flushed out.

 -- Function: void Lstream_reopen (Lstream *STREAM)
     Reopen a closed stream.  This enables I/O on it again.  This is not
     meant to be called except from a wrapper routine that reinitializes
     variables and such--the close routine may well have freed some
     necessary storage structures, for example.

 -- Function: void Lstream_rewind (Lstream *STREAM)
     Rewind the stream to the beginning.


File: internals.info,  Node: Lstream Methods,  Prev: Lstream Functions,  Up: Lstreams

35.4 Lstream Methods
====================

 -- Lstream Method: Bytecount reader (Lstream *STREAM, unsigned char
          *DATA, Bytecount SIZE)
     Read some data from the stream's end and store it into DATA, which
     can hold SIZE bytes.  Return the number of bytes read.  A return
     value of 0 means no bytes can be read at this time.  This may be
     because of an EOF, or because there is a granularity greater than
     one byte that the stream imposes on the returned data, and SIZE is
     less than this granularity. (This will happen frequently for
     streams that need to return whole characters, because
     `Lstream_read()' calls the reader function repeatedly until it has
     the number of bytes it wants or until 0 is returned.)  The lstream
     functions do not treat a 0 return as EOF or do anything special;
     however, the calling function will interpret any 0 it gets back as
     EOF.  This will normally not happen unless the caller calls
     `Lstream_read()' with a very small size.

     This function can be `NULL' if the stream is output-only.

 -- Lstream Method: Bytecount writer (Lstream *STREAM, const unsigned
          char *DATA, Bytecount SIZE)
     Send some data to the stream's end.  Data to be sent is in DATA
     and is SIZE bytes.  Return the number of bytes sent.  This
     function can send and return fewer bytes than is passed in; in that
     case, the function will just be called again until there is no
     data left or 0 is returned.  A return value of 0 means that no
     more data can be currently stored, but there is no error; the data
     will be squirreled away until the writer can accept data. (This is
     useful, e.g., if you're dealing with a non-blocking file
     descriptor and are getting `EWOULDBLOCK' errors.)  This function
     can be `NULL' if the stream is input-only.

 -- Lstream Method: int rewinder (Lstream *STREAM)
     Rewind the stream.  If this is `NULL', the stream is not seekable.

 -- Lstream Method: int seekable_p (Lstream *STREAM)
     Indicate whether this stream is seekable--i.e. it can be rewound.
     This method is ignored if the stream does not have a rewind
     method.  If this method is not present, the result is determined
     by whether a rewind method is present.

 -- Lstream Method: int flusher (Lstream *STREAM)
     Perform any additional operations necessary to flush the data in
     this stream.

 -- Lstream Method: int pseudo_closer (Lstream *STREAM)

 -- Lstream Method: int closer (Lstream *STREAM)
     Perform any additional operations necessary to close this stream
     down.  May be `NULL'.  This function is called when
     `Lstream_close()' is called or when the stream is
     garbage-collected.  When this function is called, all pending data
     in the stream will already have been written out.

 -- Lstream Method: Lisp_Object marker (Lisp_Object LSTREAM, void
          (*MARKFUN) (Lisp_Object))
     Mark this object for garbage collection.  Same semantics as a
     standard `Lisp_Object' marker.  This function can be `NULL'.


File: internals.info,  Node: Subprocesses,  Next: Interface to MS Windows,  Prev: Lstreams,  Up: Top

36 Subprocesses
***************

The fields of a process are:

`name'
     A string, the name of the process.

`command'
     A list containing the command arguments that were used to start
     this process.

`filter'
     A function used to accept output from the process instead of a
     buffer, or `nil'.

`sentinel'
     A function called whenever the process receives a signal, or `nil'.

`buffer'
     The associated buffer of the process.

`pid'
     An integer, the Unix process ID.

`childp'
     A flag, non-`nil' if this is really a child process.  It is `nil'
     for a network connection.

`mark'
     A marker indicating the position of the end of the last output
     from this process inserted into the buffer.  This is often but not
     always the end of the buffer.

`kill_without_query'
     If this is non-`nil', killing XEmacs while this process is still
     running does not ask for confirmation about killing the process.

`raw_status_low'
`raw_status_high'
     These two fields record 16 bits each of the process status
     returned by the `wait' system call.

`status'
     The process status, as `process-status' should return it.

`tick'
`update_tick'
     If these two fields are not equal, a change in the status of the
     process needs to be reported, either by running the sentinel or by
     inserting a message in the process buffer.

`pty_flag'
     Non-`nil' if communication with the subprocess uses a PTY; `nil'
     if it uses a pipe.

`infd'
     The file descriptor for input from the process.

`outfd'
     The file descriptor for output to the process.

`subtty'
     The file descriptor for the terminal that the subprocess is using.
     (On some systems, there is no need to record this, so the value is
     `-1'.)

`tty_name'
     The name of the terminal that the subprocess is using, or `nil' if
     it is using pipes.


File: internals.info,  Node: Interface to MS Windows,  Next: Interface to the X Window System,  Prev: Subprocesses,  Up: Top

37 Interface to MS Windows
**************************

* Menu:

* Different kinds of Windows environments::
* Windows Build Flags::
* Windows I18N Introduction::
* Modules for Interfacing with MS Windows::


File: internals.info,  Node: Different kinds of Windows environments,  Next: Windows Build Flags,  Prev: Interface to MS Windows,  Up: Interface to MS Windows

37.1 Different kinds of Windows environments
============================================

(a) operating system (OS) vs. window system vs. Win32 API vs. C runtime library (CRT) vs. and compiler
......................................................................................................

There are various Windows operating systems (Windows NT, 2000, XP, 95,
98, ME, etc.), which come in two basic classes: Windows NT (NT, 2000,
XP, and all future versions) and 9x (95, 98, ME).  9x-class operating
systems are a kind of hodgepodge of a 32-bit upper layer on top of a
16-bit MS-DOS-compatible lower layer.  NT-class operating systems are
written from the ground up as 32-bit (there are also 64-bit versions
available now), and provide many more features and much greater
stability, since there is full memory protection between all processes
and the between processes and the system.  NT-class operating systems
also provide emulation for DOS programs inside of a "sandbox" (i.e. a
walled-off environment in which one DOS program can screw up another
one, but there is theoretically no way for a DOS program to screw up the
OS itself).  From the perspective of XEmacs, the different between NT
and 9x is very important in Unicode support (not really provided under
9x - see `intl-win32.c') and subprocess creation, among other things.

   The operating system provides the framework for accessing files and
devices and running programs.  From the perspective of a program, the
operating system provides a set of services.  At the lowest level, the
way to call these services is dependent on the processor the OS is
running on, but a portable interface is provided to C programs through
functions called "system calls".  Under Windows, this interface is
called the Win32 API, and includes file-manipulation calls such as
`CreateFile()' and `ReadFile()', process-creation calls such as
`CreateProcess()', etc.

   This concept of system calls goes back to Unix, where similar
services are available but through routines with different, simpler
names, such as `open()', `read()', `fork()', `execve()', etc.  In
addition, Unix provides a higher layer of routines, called the C
Runtime Library (CRT), which provide higher-level, more convenient
versions of the same services (e.g.  "stream-oriented" file routines
such as `fopen()' and `fread()') as well as various other utility
functions, such as string-manipulation routines (e.g. `strcpy()' and
`strcmp()').

   For compatibility, a C Runtime Library (CRT) is also provided under
Windows, which provides a partial implementation of both the Unix CRT
and the Unix system-call API, implemented using the Win32 API.  The CRT
sources come with Visual C++ (VC++).  For example, under VC++ 6, look in
the CRT/SRC directory, e.g. for me (ben): /Program Files/Microsoft
Visual Studio/VC98/CRT/SRC. The CRT is provided using either MSVCRT
(dynamically linked) or `LIBC.LIB' (statically linked).

   The window system provides the framework for creating overlapped
windows and unifying signals provided by various devices (input devices
such as the keyboard and mouse, timers, etc.) into a single event queue
(or "message queue", under Windows).  Like the operating system, the
window system can be viewed from the perspective of a program as a set
of services provided by an API of function calls.  Under Windows,
window-system services are also available through the Win32 API, while
under UNIX the window system is typically a separate component (e.g. the
X Windowing System, aka X Windows or X11).  The term "GUI" ("graphical
user interface") is often used to refer to the services provided by the
window system, or to a windowing interface provided by a program.

   The Win32 API is implemented by various dynamic libraries, or DLL's.
The most important are KERNEL32, USER32, and GDI32.  KERNEL32 implements
the basic file-system and process services.  USER32 implements the
fundamental window-system services such as creating windows and handling
messages.  GDI32 implements higher-level drawing capabilities - fonts,
colors, lines, etc.

   C programs are compiled into executables using a compiler.  Under
Unix, a compiler usually comes as part of the operating system, but not
under Windows, where the compiler is a separate product.  Even under
Unix, people often install their own compilers, such as gcc.  Under
Windows, the Microsoft-standard compiler is Visual C++ (VC++).

   It is possible to provide an emulation of any API using any other, as
long as the underlying API provides the suitable functionality.  This is
what Cygwin (www.cygwin.com) does.  It provides a fairly complete POSIX
emulation layer (POSIX is a government standard for Unix behavior) on
top of MS Windows - in particular, providing the file-system, process,
tty, and signal semantics that are part of a modern, standard Unix
operating system.  Cygwin does this using its own DLL, `cygwin1.dll',
which makes calls to the Win32 API services in `kernel32.dll'.  Cygwin
also provides its own implementation of the C runtime library, called
`newlib' (`libcygwin.a'; `libc.a' and `libm.a' are symlinked to it),
which is implemented on top of the Unix system calls provided in
`cygwin1.dll'.  In addition, Cygwin provides static import libraries
that give you direct access to the Win32 API - XEmacs uses this to
provide GUI support under Cygwin.  Cygwin provides a version of GCC
(the GNU Project C compiler) that is set up to automatically link with
the appropriate Cygwin libraries.  Cygwin also provides, as optional
components, pre-compiled binaries for a great number of open-source
programs compiled under the Cygwin environment.  This includes all of
the standard Unix file-system, text-manipulation, development,
networking, database, etc. utilities, a version of X Windows that uses
the Win32 API underlyingly (see below), and compilations of nearly all
other common open-source packages (Apache, TeX, [X]Emacs, Ghostscript,
GTK, ImageMagick, etc.).

   Similarly, you can emulate the functionality of X Windows using the
Win32 component of the Win32 API.  Cygwin provides a package to do this,
from the XFree86 project.  Other versions of X under Windows also exist,
such as the MicroImages MI/X server.  Each version potentially can come
comes with its own header and library files, allowing you to compile
X-Windows programs.

   All of these different operating system and emulation layers can make
for a fair amount of confusion, so:

(b) CRT is not the same as VC++
...............................

Note that the CRT is *NOT* (completely) part of VC++.  True, if you link
statically, the CRT (in the form of `LIBC.LIB', which comes with VC++)
will be inserted into the executable (.EXE), but otherwise the CRT will
be separate.  The dynamic version of the CRT is provided by `MSVCRT.DLL'
(or `MSVCRTD.DLL', for debugging), which comes with Windows.  Hence,
it's possible to use a different compiler and still link with MSVCRT -
which is exactly what MinGW does.

(c) CRT is not the same as the Win32 API
........................................

Note also that the CRT is totally separate from the Win32 API.  They
provide different functions and are implemented in different DLL's.
They are also different levels - the CRT is implemented on top of
Win32.  Sometimes the CRT and Win32 both have their own versions of
similar concepts, such as locales.  These are typically maintained
separately, and can get out of sync.  Do not assume that changing a
setting in the CRT will have any effect on Win32 API routines using a
similar concept unless the CRT docs specifically say so.  Do not assume
that behavior described for CRT functions applies to Win32 API or
vice-versa.  Note also that the CRT knows about and is implemented on
top of the Win32 API, while the Win32 API knows nothing about the CRT.

(d) MinGW is not the same as Cygwin
...................................

As described in (b), Microsoft's version of the CRT (`MSVCRT.DLL') is
provided as part of Windows, separate from VC++, which must be
purchased.  Hence, it is possible to write MSVCRT to provide CRT
services without using VC++.  This is what MinGW (www.mingw.org) does -
it is a port of GCC that will use MSVCRT.  The reason one might want to
do this is (a) it is free, and (b) it does not require a separately
installed DLL, as Cygwin does. (#### Maybe MinGW targets CRTDLL, not
MSVCRT?  If so, what is CRTDLL, and how does it differ from MSVCRT and
`LIBC.LIB'?) Primarily, what MinGW provides is patches to GCC (now
integrated into the standard distribution) and its own header files and
import libraries that are compatible with MSVCRT.  The best way to think
of MinGW is as simply another Windows compiler, like how there used to
be Microsoft and Borland compilers.  Because MinGW programs use all the
same libraries as VC++ programs, and hence the same services are
available, programs that compile under VC++ should compile under MinGW
with very little change, whereas programs that compile under Cygwin will
look quite different.

   The confusion between MinGW and Cygwin is the confusion between the
environment that a compiler runs under and the target environment of a
program, i.e. the environment that a program is compiled to run under.
It's theoretically possible, for example, to compile a program under
Windows and generate a binary that can only be run under Linux, or
vice-versa - or, for that matter, to use Windows, running on an Intel
machine to write and a compile a program that will run on the Mac OS,
running on a PowerPC machine.  This is called cross-compiling, and while
it may seem rather esoteric, it is quite normal when you want to
generate a program for a machine that you cannot develop on - for
example, a program that will run on a Palm Pilot.  Originally, this is
how MinGW worked - you needed to run GCC under a Cygwin environment and
give it appropriate flags, telling it to use the MinGW headers and
target `MSVCRT.DLL' rather than `CYGWIN1.DLL'. (In fact, Cygwin
standardly comes with MinGW's header files.) This was because GCC was
written with Unix in mind and relied on a large amount of Unix-specific
functionality.  To port GCC to Windows without using a POSIX emulation
layer would mean a lot of rewriting of GCC.  Eventually, however, this
was done, and it GCC was itself compiled using MinGW.  The result is
that currently you can develop MinGW applications either under Cygwin
or under native Windows.

(e) Operating system is not the same as window system
.....................................................

As per the above discussion, we can use either Native Windows (the OS
part of Win32 provided by `KERNEL32.DLL' and the Windows CRT as
provided by MSVCRT or CLL) or Cygwin to provide operating-system
functionality, and we can use either Native Windows (the windowing part
of Win32 as provided by `USER32.DLL' and `GDI32.DLL') or X11 to provide
window-system functionality.  This gives us four possible build
environments.  It's currently possible to build XEmacs with at least
three of these combinations - as far as I know native + X11 is no
longer supported, although it used to be (support used to exist in
`xemacs.mak' for linking with some X11 libraries available from
somewhere, but it was bit-rotting and you could always use Cygwin; ####
what happens if we try to compile with MinGW, native OS + X11?).  This
may still seem confusing, so:

Native OS + native windowing
     We call `CreateProcess()' to run subprocesses (`process-nt.c'),
     and `CreateWindowEx()' to create a top-level window
     (`frame-msw.c').  We use `nt/xemacs.mak' to compile with VC++,
     linking with the Windows CRT (`MSVCRT.DLL' or `LIBC.LIB') and with
     the various Win32 DLL's (`KERNEL32.DLL', `USER32.DLL',
     `GDI32.DLL'); or we use `src/Makefile[.in.in]' to compile with
     GCC, telling it (e.g. -mno-cygwin, see `s/mingw32.h') to use MinGW
     (which will end up linking with `MSVCRT.DLL'), and linking GCC
     with -lshell32 -lgdi32 -luser32 etc. (see `configure.in').

Cygwin + native windowing
     We call `fork()'/`execve()' to run subprocesses
     (`process-unix.c'), and `CreateWindowEx()' to create a top-level
     window (`frame-msw.c').  We use `src/Makefile[in.in]' to compile
     with GCC (it will end up linking with `CYGWIN1.DLL') and link GCC
     with -lshell32 -lgdi32 -luser32 etc. (see `configure.in').

Cygwin + X11
     We call `fork()'/`execve()' to run subprocesses
     (`process-unix.c'), and `XtCreatePopupShell()' to create a
     top-level window (`frame-x.c').  We use `src/Makefile[.in.in]' to
     compile with GCC (it will end up linking with `CYGWIN1.DLL') and
     link GCC with -lXt, -lX11, etc. (see `configure.in').

     Finally, if native OS + X11 were possible, it might look something
     like

[Native OS + X11]
     We call `CreateProcess()' to run subprocesses (`process-nt.c'),
     and `XtCreatePopupShell()' to create a top-level window
     (`frame-x.c').  We use `nt/xemacs.mak' to compile with VC++,
     linking with the Windows CRT (`MSVCRT.DLL' or `LIBC.LIB') and with
     the various X11 DLL's (`XT.DLL', `XLIB.DLL', etc.); or we use
     `src/Makefile[.in.in]' to compile with GCC, telling it (e.g.
     -mno-cygwin, see `s/mingw32.h') to use MinGW (which will end up
     linking with `MSVCRT.DLL'), and linking GCC with -lXt, -lX11, etc.
     (see `configure.in').

   One of the reasons that we maintain the ability to build under Cygwin
and X11 on Windows, when we have native support, is that it allows
Windows compilers to test under a Unix-like environment.


File: internals.info,  Node: Windows Build Flags,  Next: Windows I18N Introduction,  Prev: Different kinds of Windows environments,  Up: Interface to MS Windows

37.2 Windows Build Flags
========================

`CYGWIN'
     for Cygwin-only stuff.

`WIN32_NATIVE'
     Win32 native OS-level stuff (files, process, etc.).  Applies
     whenever linking against the native C libraries - i.e.  all
     compilations with VC++ and with MINGW, but never Cygwin.

`HAVE_X_WINDOWS'
     for X Windows (regardless of whether under MS Win)

`HAVE_MS_WINDOWS'
     MS Windows native windowing system (anything related to the
     appearance of the graphical screen).  May or may not apply to any
     of VC++, MINGW, Cygwin.

   Finally, there's also the MINGW build environment, which uses GCC
(similar to Cygwin), but native MS Windows libraries rather than a
POSIX emulation layer (the Cygwin approach).  This environment defines
WIN32_NATIVE, but also defines MINGW, which is used mostly because uses
its own include files (related to Cygwin), which have a few things
messed up.

   Formerly, we had a whole host of flags.  Here's the conversion, for
porting code from GNU Emacs and such:

Old Constant       New Constant
---------------------------------------------------------------
`WINDOWSNT'        `WIN32_NATIVE'
`WIN32'            `WIN32_NATIVE'
`_WIN32'           `WIN32_NATIVE'
`HAVE_WIN32'       `WIN32_NATIVE'
`DOS_NT'           `WIN32_NATIVE'
`HAVE_NTGUI'       `WIN32_NATIVE', unless it ends up already bracketed
                   by this
`HAVE_FACES'       always true
`MSDOS'            determine whether this code is really specific to
                   MS-DOS (and not Windows - e.g. DJGPP code); if so,
                   delete the code; otherwise, convert to `WIN32_NATIVE'
                   (we do not support MS-DOS w/DOS Extender under XEmacs)
`__CYGWIN__'       `CYGWIN'
`__CYGWIN32__'     `CYGWIN'
`__MINGW32__'      `MINGW'


File: internals.info,  Node: Windows I18N Introduction,  Next: Modules for Interfacing with MS Windows,  Prev: Windows Build Flags,  Up: Interface to MS Windows

37.3 Windows I18N Introduction
==============================

*Abstract:* This page provides an overview of the aspects of the Win32
internationalization API that are relevant to XEmacs, including the
basic distinction between multibyte and Unicode encodings.  Also
included are pointers to how XEmacs should make use of this API.

   The Win32 API is quite well-designed in its handling of strings
encoded for various character sets.  The API is geared around the idea
that two different methods of encoding strings should be supported.
These methods are called multibyte and Unicode, respectively.  The
multibyte encoding is compatible with ASCII strings and is a more
efficient representation when dealing with strings containing primarily
ASCII characters, but it has a great number of serious deficiencies and
limitations, including that it is very difficult and error-prone to work
with strings in this encoding, and any particular string in a multibyte
encoding can only contain characters from a very limited number of
character sets.  The Unicode encoding rectifies all of these
deficiencies, but it is not compatible with ASCII strings (in other
words, an existing program will not be able to handle the encoded
strings unless it is explicitly modified to do so), and it takes up
twice as much memory space as multibyte encodings when encoding a purely
ASCII string.

   Multibyte encodings use a variable number of bytes (either one or
two) to represent characters.  ASCII characters are also represented by
a single byte with its high bit not set, and non-ASCII characters are
represented by one or two bytes, the first of which always has its high
bit set.  (The second byte, when it exists, may or may not have its high
bit set.)  There is no single multibyte encoding.  Instead, there is
generally one encoding per non-ASCII character set.  Such an encoding is
capable of representing (besides ASCII characters, of course) only
characters from one (or possibly two) particular character sets.

   Multibyte encoding makes processing of strings very difficult.  For
example, given a pointer to the beginning of a character within a
string, finding the pointer to the beginning of the previous character
may require backing up all the way to the beginning of the string, and
then moving forward.  Also, an operation such as separating out the
components of a path by searching for backslashes will fail if it's
implemented in the simplest (but not multibyte-aware) fashion, because
it may find what appears to be a backslash, but which is actually the
second byte of a two-byte character.  Also, the limited number of
character sets that any particular multibyte encoding can represent
means that loss of data is likely if a string is converted from the
XEmacs internal format into a multibyte format.

   For these reasons, the C code in XEmacs should never do any sort of
work with multibyte encoded strings (or with strings in any external
encoding for that matter).  Strings should always be maintained in the
internal encoding, which is predictable, and converted to an external
encoding only at the point where the string moves from the XEmacs C
code and enters a system library function.  Similarly, when a string is
returned from a system library function, it should be immediately
converted into the internal coding before any operations are done on it.

   Unicode, unlike multibyte encodings, is a fixed-width encoding where
every character is represented using 16 bits.  It is also capable of
encoding all the characters from all the character sets in common use in
the world.  The predictability and completeness of the Unicode encoding
makes it a very good encoding for strings that may contain characters
from many character sets mixed up with each other.  At the same time, of
course, it is incompatible with routines that expect ASCII characters
and also incompatible with general string manipulation routines, which
will encounter a great number of what would appear to be embedded nulls
in the string.  It also takes twice as much room to encode strings
containing primarily ASCII characters.  This is why XEmacs does not use
Unicode or similar encoding internally for buffers.

   The Win32 API cleverly deals with the issue of 8 bit vs. 16 bit
characters by declaring a type called `"TCHAR"' which specifies a
generic character, either 8 bits or 16 bits.  Generally `TCHAR' is
defined to be the same as the simple C type `char', unless the
preprocessor constant `UNICODE' is defined, in which case `TCHAR' is
defined to be `WCHAR', which is a 16 bit type.  Nearly all functions in
the Win32 API that take strings are defined to take strings that are
actually arrays of `TCHAR's.  There is a type `LPTSTR' which is defined
to be a string of `TCHAR's and another type `LPCTSTR' which is a const
string of `TCHAR's.  The theory is that any program that uses `TCHAR's
exclusively to represent characters and does not make assumptions about
the size of a `TCHAR' or the way that the characters are encoded should
work transparently regardless of whether the `UNICODE' preprocessor
constant is defined, which is to say, regardless of whether 8 bit
multibyte or 16 bit Unicode characters are being used.  The way that
this is actually implemented is that every Win32 API function that takes
a string as an argument actually maps to one of two functions which are
suffixed with an `A' (which stands for ANSI, and means multibyte
strings) or `W' (which stands for wide, and means Unicode strings).
The mapping is, of course, controlled by the same `UNICODE'
preprocessor constant.  Generally all structures containing strings in
them actually map to one of two different kinds of structures, with
either an `A' or a `W' suffix after the structure name.

   Unfortunately, not all of the implementations of the Win32 API
implement all of the functionality described above.  In particular,
Windows 95 does not implement very much Unicode functionality.  It does
implement functions to convert multibyte-encoded strings to and from
Unicode strings, and provides Unicode versions of certain low-level
functions like `ExtTextOut()'.  In fact, all of the rest of the Unicode
versions of API functions are just stubs that return an error.
Conversely, all versions of Windows NT completely implement all the
Unicode functionality, but some versions (especially versions before
Windows NT 4.0) don't implement much of the multibyte functionality.
For this reason, as well as for general code cleanliness, XEmacs needs
to be written in such a way that it works with or without the `UNICODE'
preprocessor constant being defined.

   Getting XEmacs to run when all strings are Unicode primarily
involves removing any assumptions made about the size of characters.
Remember what I said earlier about how the point of conversion between
internally and externally encoded strings should occur at the point of
entry or exit into or out of a library function.  With this in mind, an
externally encoded string in XEmacs can be treated simply as an
arbitrary sequence of bytes of some length which has no particular
relationship to the length of the string in the internal encoding.

   #### The rest of this is *out-of-date* and needs to be written to
reference the actual coding systems or aliases that we currently use.

   [[ To facilitate this, the enum `external_data_format', which is
declared in `lisp.h', is expanded to contain three new formats, which
are `FORMAT_LOCALE', `FORMAT_UNICODE' and `FORMAT_TSTR'.
`FORMAT_LOCALE' always causes encoding into a multibyte string
consistent with the encoding of the current locale.  The functions to
handle locales are different under Unix and Windows and locales are a
process property under Unix and a thread property under Windows, but
the concepts are basically the same.  `FORMAT_UNICODE' of course causes
encoding into Unicode and `FORMAT_TSTR' logically maps to either
`FORMAT_LOCALE' or `FORMAT_UNICODE' depending on the `UNICODE'
preprocessor constant.

   Under Unix the behavior of `FORMAT_TSTR' is undefined and this
particular format should not be used.  Under Windows however
`FORMAT_TSTR' should be used for pretty much all of the Win32 API
calls.  The other two formats should only be used in particular APIs
that specifically call for a multibyte or Unicode encoded string
regardless of the `UNICODE' preprocessor constant.  String constants
that are to be passed directly to Win32 API functions, such as the
names of window classes, need to be bracketed in their definition with
a call to the macro `TEXT'.  This awfully named macro, which comes out
of the Win32 API, appropriately makes a string of either regular or
wide chars, which is to say this string may be prepended with an `L'
(causing it to be a wide string) depending on the `UNICODE'
preprocessor constant.

   By the way, if you're wondering what happened to `FORMAT_OS', I
think that this format should go away entirely because it is too vague
and should be replaced by more specific formats as they are defined.  ]]

   Use Qnative for Unix conversion, Qmswindows_tstr for Windows ...

   String constants that are to be passed directly to Win32 API
functions, such as the names of window classes, need to be bracketed in
their definition with a call to the macro XETEXT. This appropriately
makes a string of either regular or wide chars, which is to say this
string may be prepended with an L (causing it to be a wide string)
depending on XEUNICODE_P.


File: internals.info,  Node: Modules for Interfacing with MS Windows,  Prev: Windows I18N Introduction,  Up: Interface to MS Windows

37.4 Modules for Interfacing with MS Windows
============================================

There are two different general Windows-related include files in src.

   Uses are approximately:

`syswindows.h'
     Wrapper around `<windows.h>', including missing defines as
     necessary.  Includes stuff needed on both Cygwin and native
     Windows, regardless of window system chosen.  Includes definitions
     needed for Unicode conversion/encapsulation, and other
     Mule-related stuff, plus various other prototypes and
     Windows-specific, but not GUI-specific, stuff.

`console-msw.h'
     Used on both Cygwin and native Windows, but only when native window
     system (as opposed to X) chosen.  Includes `syswindows.h'.

   Summary of files:

`console-msw.h'
     include file for native windowing (otherwise, `console-x.h', etc.)

`console-msw.c, frame-msw.c, etc.'
     native windowing, as above

`process-nt.c'
     subprocess support for native OS (otherwise, `process-unix.c')

`nt.c'
     support routines used under native OS

`win32.c'
     support routines used under both OS environments

`syswindows.h'
     support header for both environments

`nt/xemacs.mak'
     Makefile for VC++ (otherwise, `src/Makefile.in.in')

`s/windowsnt.h'
     s header for basic native-OS defines, VC++ compiler

`s/mingw32.h'
     s header for basic native-OS defines, GCC/MinGW compiler

`s/cygwin.h'
     s header for basic Cygwin defines

`s/win32-native.h'
     s header for basic native-OS defines, all compilers

`s/win32-common.h'
     s header for defines for both OS environments

`intl-win32.c'
     internationalization functions for both OS environments

`intl-encap-win32.c'
     Unicode encapsulation functions for both OS environments

`intl-auto-encap-win32.c'
     Auto-generated Unicode encapsulation functions

`intl-auto-encap-win32.h'
     Auto-generated Unicode encapsulation headers


File: internals.info,  Node: Interface to the X Window System,  Next: Dumping,  Prev: Interface to MS Windows,  Up: Top

38 Interface to the X Window System
***********************************

Mostly undocumented.

* Menu:

* Lucid Widget Library::        An interface to various widget sets.
* Modules for Interfacing with X Windows::


File: internals.info,  Node: Lucid Widget Library,  Next: Modules for Interfacing with X Windows,  Prev: Interface to the X Window System,  Up: Interface to the X Window System

38.1 Lucid Widget Library
=========================

Lwlib is extremely poorly documented and quite hairy.  The author(s)
blame that on X, Xt, and Motif, with some justice, but also sufficient
hypocrisy to avoid drawing the obvious conclusion about their own work.

   The Lucid Widget Library is composed of two more or less independent
pieces.  The first, as the name suggests, is a set of widgets.  These
widgets are intended to resemble and improve on widgets provided in the
Motif toolkit but not in the Athena widgets, including menubars and
scrollbars.  Recent additions by Andy Piper integrate some "modern"
widgets by Edward Falk, including checkboxes, radio buttons, progress
gauges, and index tab controls (aka notebooks).

   The second piece of the Lucid widget library is a generic interface
to several toolkits for X (including Xt, the Athena widget set, and
Motif, as well as the Lucid widgets themselves) so that core XEmacs
code need not know which widget set has been used to build the
graphical user interface.

* Menu:

* Generic Widget Interface::    The lwlib generic widget interface.
* Scrollbars::
* Menubars::
* Checkboxes and Radio Buttons::
* Progress Bars::
* Tab Controls::


File: internals.info,  Node: Generic Widget Interface,  Next: Scrollbars,  Prev: Lucid Widget Library,  Up: Lucid Widget Library

38.1.1 Generic Widget Interface
-------------------------------

In general in any toolkit a widget may be a composite object.  In Xt,
all widgets have an X window that they manage, but typically a complex
widget will have widget children, each of which manages a subwindow of
the parent widget's X window.  These children may themselves be
composite widgets.  Thus a widget is actually a tree or hierarchy of
widgets.

   For each toolkit widget, lwlib maintains a tree of `widget_values'
which mirror the hierarchical state of Xt widgets (including Motif,
Athena, 3D Athena, and Falk's widget sets).  Each `widget_value' has
`contents' member, which points to the head of a linked list of its
children.  The linked list of siblings is chained through the `next'
member of `widget_value'.

                +-----------+
                | composite |
                +-----------+
                      |
                      | contents
                      V
                  +-------+ next +-------+ next +-------+
                  | child |----->| child |----->| child |
                  +-------+      +-------+      +-------+
                                     |
                                     | contents
                                     V
                              +-------------+ next +-------------+
                              | grand child |----->| grand child |
                              +-------------+      +-------------+

     The `widget_value' hierarchy of a composite widget with two simple
     children and one composite child.

   The `widget_instance' structure maintains the inverse view of the
tree.  As for the `widget_value', siblings are chained through the
`next' member.  However, rather than naming children, the
`widget_instance' tree links to parents.

                +-----------+
                | composite |
                +-----------+
                      A
                      | parent
                      |
                  +-------+ next +-------+ next +-------+
                  | child |----->| child |----->| child |
                  +-------+      +-------+      +-------+
                                     A
                                     | parent
                                     |
                              +-------------+ next +-------------+
                              | grand child |----->| grand child |
                              +-------------+      +-------------+

     The `widget_value' hierarchy of a composite widget with two simple
     children and one composite child.

   This permits widgets derived from different toolkits to be updated
and manipulated generically by the lwlib library. For instance
`update_one_widget_instance' can cope with multiple types of widget and
multiple types of toolkit. Each element in the widget hierarchy is
updated from its corresponding `widget_value' by walking the
`widget_value' tree.  This has desirable properties.  For example,
`lw_modify_all_widgets' is called from `glyphs-x.c' and updates all the
properties of a widget without having to know what the widget is or
what toolkit it is from.  Unfortunately this also has its hairy
properties; the lwlib code quite complex. And of course lwlib has to
know at some level what the widget is and how to set its properties.

   The `widget_instance' structure also contains a pointer to the root
of its tree.  Widget instances are further confi


File: internals.info,  Node: Scrollbars,  Next: Menubars,  Prev: Generic Widget Interface,  Up: Lucid Widget Library

38.1.2 Scrollbars
-----------------


File: internals.info,  Node: Menubars,  Next: Checkboxes and Radio Buttons,  Prev: Scrollbars,  Up: Lucid Widget Library

38.1.3 Menubars
---------------


File: internals.info,  Node: Checkboxes and Radio Buttons,  Next: Progress Bars,  Prev: Menubars,  Up: Lucid Widget Library

38.1.4 Checkboxes and Radio Buttons
-----------------------------------


File: internals.info,  Node: Progress Bars,  Next: Tab Controls,  Prev: Checkboxes and Radio Buttons,  Up: Lucid Widget Library

38.1.5 Progress Bars
--------------------


File: internals.info,  Node: Tab Controls,  Prev: Progress Bars,  Up: Lucid Widget Library

38.1.6 Tab Controls
-------------------


File: internals.info,  Node: Modules for Interfacing with X Windows,  Prev: Lucid Widget Library,  Up: Interface to the X Window System

38.2 Modules for Interfacing with X Windows
===========================================

     Emacs.ad.h

   A file generated from `Emacs.ad', which contains XEmacs-supplied
fallback resources (so that XEmacs has pretty defaults).

     EmacsFrame.c
     EmacsFrame.h
     EmacsFrameP.h

   These modules implement an Xt widget class that encapsulates a frame.
This is for ease in integrating with Xt.  The EmacsFrame widget covers
the entire X window except for the menubar; the scrollbars are
positioned on top of the EmacsFrame widget.

   *Warning:* Abandon hope, all ye who enter here.  This code took an
ungodly amount of time to get right, and is likely to fall apart
mercilessly at the slightest change.  Such is life under Xt.

     EmacsManager.c
     EmacsManager.h
     EmacsManagerP.h

   These modules implement a simple Xt manager (i.e. composite) widget
class that simply lets its children set whatever geometry they want.
It's amazing that Xt doesn't provide this standardly, but on second
thought, it makes sense, considering how amazingly broken Xt is.

     EmacsShell-sub.c
     EmacsShell.c
     EmacsShell.h
     EmacsShellP.h

   These modules implement two Xt widget classes that are subclasses of
the TopLevelShell and TransientShell classes.  This is necessary to deal
with more brokenness that Xt has sadistically thrust onto the backs of
developers.

     xgccache.c
     xgccache.h

   These modules provide functions for maintenance and caching of GC's
(graphics contexts) under the X Window System.  This code is junky and
needs to be rewritten.

     select-msw.c
     select-x.c
     select.c
     select.h

   This module provides an interface to the X Window System's concept of
"selections", the standard way for X applications to communicate with
each other.

     xintrinsic.h
     xintrinsicp.h
     xmmanagerp.h
     xmprimitivep.h

   These header files are similar in spirit to the `sys*.h' files and
buffer against different implementations of Xt and Motif.

   * `xintrinsic.h' should be included in place of `<Intrinsic.h>'.

   * `xintrinsicp.h' should be included in place of `<IntrinsicP.h>'.

   * `xmmanagerp.h' should be included in place of `<XmManagerP.h>'.

   * `xmprimitivep.h' should be included in place of `<XmPrimitiveP.h>'.

     xmu.c
     xmu.h

   These files provide an emulation of the Xmu library for those systems
(i.e. HPUX) that don't provide it as a standard part of X.

     ExternalClient-Xlib.c
     ExternalClient.c
     ExternalClient.h
     ExternalClientP.h
     ExternalShell.c
     ExternalShell.h
     ExternalShellP.h
     extw-Xlib.c
     extw-Xlib.h
     extw-Xt.c
     extw-Xt.h

   These files provide the "external widget" interface, which allows an
XEmacs frame to appear as a widget in another application.  To do this,
you have to configure with `--external-widget'.

   `ExternalShell*' provides the server (XEmacs) side of the connection.

   `ExternalClient*' provides the client (other application) side of
the connection.  These files are not compiled into XEmacs but are
compiled into libraries that are then linked into your application.

   `extw-*' is common code that is used for both the client and server.

   Don't touch this code; something is liable to break if you do.


File: internals.info,  Node: Dumping,  Next: Future Work,  Prev: Interface to the X Window System,  Up: Top

39 Dumping
**********

* Menu:

* Dumping Justification::
* Overview::
* Data descriptions::
* Dumping phase::
* Reloading phase::
* Remaining issues::


File: internals.info,  Node: Dumping Justification,  Next: Overview,  Prev: Dumping,  Up: Dumping

39.1 Dumping Justification
==========================

The C code of XEmacs is just a Lisp engine with a lot of built-in
primitives useful for writing an editor.  The editor itself is written
mostly in Lisp, and represents around 100K lines of code.  Loading and
executing the initialization of all this code takes a bit a time (five
to ten times the usual startup time of current xemacs) and requires
having all the lisp source files around.  Having to reload them each
time the editor is started would not be acceptable.

   The traditional solution to this problem is called dumping: the build
process first creates the lisp engine under the name `temacs', then
runs it until it has finished loading and initializing all the lisp
code, and eventually creates a new executable called `xemacs' including
both the object code in `temacs' and all the contents of the memory
after the initialization.

   This solution, while working, has a huge problem: the creation of the
new executable from the actual contents of memory is an extremely
system-specific process, quite error-prone, and which interferes with a
lot of system libraries (like malloc).  It is even getting worse
nowadays with libraries using constructors which are automatically
called when the program is started (even before `main()') which tend to
crash when they are called multiple times, once before dumping and once
after (IRIX 6.x `libz.so' pulls in some C++ image libraries thru
dependencies which have this problem).  Writing the dumper is also one
of the most difficult parts of porting XEmacs to a new operating system.
Basically, `dumping' is an operation that is just not officially
supported on many operating systems.

   The aim of the portable dumper is to solve the same problem as the
system-specific dumper, that is to be able to reload quickly, using only
a small number of files, the fully initialized lisp part of the editor,
without any system-specific hacks.


File: internals.info,  Node: Overview,  Next: Data descriptions,  Prev: Dumping Justification,  Up: Dumping

39.2 Overview
=============

The portable dumping system has to:

  1. At dump time, write all initialized, non-quickly-rebuildable data
     to a file [Note: currently named `xemacs.dmp', but the name will
     change], along with all information needed for the reloading.

  2. When starting xemacs, reload the dump file, relocate it to its new
     starting address if needed, and reinitialize all pointers to this
     data.  Also, rebuild all the quickly rebuildable data.

   Note: As of 21.5.18, the dump file has been moved inside of the
executable, although there are still problems with this on some systems.


File: internals.info,  Node: Data descriptions,  Next: Dumping phase,  Prev: Overview,  Up: Dumping

39.3 Data descriptions
======================

The more complex task of the dumper is to be able to write memory blocks
on the heap (lisp objects, i.e. lrecords, and C-allocated memory, such
as structs and arrays) to disk and reload them at a different address,
updating all the pointers they include in the process.  This is done by
using external data descriptions that give information about the layout
of the blocks in memory.

   The specification of these descriptions is in lrecord.h.  A
description of an lrecord is an array of struct memory_description.
Each of these structs include a type, an offset in the block and some
optional parameters depending on the type.  For instance, here is the
string description:

     static const struct memory_description string_description[] = {
       { XD_BYTECOUNT,         offsetof (Lisp_String, size) },
       { XD_OPAQUE_DATA_PTR,   offsetof (Lisp_String, data), XD_INDIRECT(0, 1) },
       { XD_LISP_OBJECT,       offsetof (Lisp_String, plist) },
       { XD_END }
     };

   The first line indicates a member of type Bytecount, which is used by
the next, indirect directive.  The second means "there is a pointer to
some opaque data in the field `data'".  The length of said data is
given by the expression `XD_INDIRECT(0, 1)', which means "the value in
the 0th line of the description (welcome to C) plus one".  The third
line means "there is a Lisp_Object member `plist' in the Lisp_String
structure".  `XD_END' then ends the description.

   This gives us all the information we need to move around what is
pointed to by a memory block (C or lrecord) and, by transitivity,
everything that it points to.  The only missing information for dumping
is the size of the block.  For lrecords, this is part of the
lrecord_implementation, so we don't need to duplicate it.  For C blocks
we use a struct sized_memory_description, which includes a size field
and a pointer to an associated array of memory_description.


File: internals.info,  Node: Dumping phase,  Next: Reloading phase,  Prev: Data descriptions,  Up: Dumping

39.4 Dumping phase
==================

Dumping is done by calling the function `pdump()' (in `dumper.c') which
is invoked from Fdump_emacs (in `emacs.c').  This function performs a
number of tasks.

* Menu:

* Object inventory::
* Address allocation::
* The header::
* Data dumping::
* Pointers dumping::


File: internals.info,  Node: Object inventory,  Next: Address allocation,  Prev: Dumping phase,  Up: Dumping phase

39.4.1 Object inventory
-----------------------

The first task is to build the list of the objects to dump.  This
includes:

   * lisp objects

   * other memory blocks (C structures, arrays. etc)

   We end up with one `pdump_block_list_elt' per object group (arrays
of C structs are kept together) which includes a pointer to the first
object of the group, the per-object size and the count of objects in the
group, along with some other information which is initialized later.

   These entries are linked together in `pdump_block_list' structures
and can be enumerated thru either:

  1. the `pdump_object_table', an array of `pdump_block_list', one per
     lrecord type, indexed by type number.

  2. the `pdump_opaque_data_list', used for the opaque data which does
     not include pointers, and hence does not need descriptions.

  3. the `pdump_desc_table', which is a vector of
     `memory_description'/`pdump_block_list' pairs, used for non-opaque
     C memory blocks.

   This uses a marking strategy similar to the garbage collector.  Some
differences though:

  1. We do not use the mark bit (which does not exist for generic
     memory blocks anyway); we use a big hash table instead.

  2. We do not use the mark function of lrecords but instead rely on the
     external descriptions.  This happens essentially because we need to
     follow pointers to generic memory blocks and opaque data in
     addition to Lisp_Object members.

   This is done by `pdump_register_object()', which handles Lisp_Object
variables, and `pdump_register_block()' which handles generic memory
blocks (C structures, arrays, etc.), which both delegate the
description management to `pdump_register_sub()'.

   The hash table doubles as a map object to pdump_block_list_elmt (i.e.
allows us to look up a pdump_block_list_elmt with the object it points
to).  Entries are added with `pdump_add_block()' and looked up with
`pdump_get_block()'.  There is no need for entry removal.  The hash
value is computed quite simply from the object pointer by
`pdump_make_hash()'.

   The roots for the marking are:

  1. the `staticpro''ed variables (there is a special
     `staticpro_nodump()' call for protected variables we do not want to
     dump).

  2. the Lisp_Object variables registered via
     `dump_add_root_lisp_object' (`staticpro()' is equivalent to
     `staticpro_nodump()' + `dump_add_root_lisp_object()').

  3. the data-segment memory blocks registered via `dump_add_root_block'
     (for blocks with relocatable pointers), or `dump_add_opaque' (for
     "opaque" blocks with no relocatable pointers; this is just a
     shortcut for calling `dump_add_root_block' with a NULL
     description).

  4. the pointer variables registered via `dump_add_root_block_ptr',
     each of which points to a block of heap memory (generally a C
     structure or array).  Note that `dump_add_root_block_ptr' is not
     technically necessary, as a pointer variable can be seen as a
     special case of a data-segment memory block and registered using
     `dump_add_root_block'.  Doing it this way, however, would require
     another level of static structures declared.  Since pointer
     variables are quite common, `dump_add_root_block_ptr' is provided
     for convenience.  Note also that internally we have to treat it
     separately from `dump_add_root_block' rather than writing the
     former as a call to the latter, since we don't have support for
     creating and using memory descriptions on the fly - they must all
     be statically declared in the data-segment.

   This does not include the GCPRO'ed variables, the specbinds, the
catchtags, the backlist, the redisplay or the profiling info, since we
do not want to rebuild the actual chain of lisp calls which end up to
the dump-emacs call, only the global variables.

   Weak lists and weak hash tables are dumped as if they were their
non-weak equivalent (without changing their type, of course).  This has
not yet been a problem.


File: internals.info,  Node: Address allocation,  Next: The header,  Prev: Object inventory,  Up: Dumping phase

39.4.2 Address allocation
-------------------------

The next step is to allocate the offsets of each of the objects in the
final dump file.  This is done by `pdump_allocate_offset()' which is
called indirectly by `pdump_scan_by_alignment()'.

   The strategy to deal with alignment problems uses these facts:

  1. real world alignment requirements are powers of two.

  2. the C compiler is required to adjust the size of a struct so that
     you can have an array of them next to each other.  This means you
     can have an upper bound of the alignment requirements of a given
     structure by looking at which power of two its size is a multiple.

  3. the non-variant part of variable size lrecords has an alignment
     requirement of 4.

   Hence, for each lrecord type, C struct type or opaque data block the
alignment requirement is computed as a power of two, with a minimum of
2^2 for lrecords.  `pdump_scan_by_alignment()' then scans all the
`pdump_block_list_elmt''s, the ones with the highest requirements
first.  This ensures the best packing.

   The maximum alignment requirement we take into account is 2^8.

   `pdump_allocate_offset()' only has to do a linear allocation,
starting at offset 256 (this leaves room for the header and keeps the
alignments happy).


File: internals.info,  Node: The header,  Next: Data dumping,  Prev: Address allocation,  Up: Dumping phase

39.4.3 The header
-----------------

The next step creates the file and writes a header with a signature and
some random information in it.  The `reloc_address' field, which
indicates at which address the file should be loaded if we want to avoid
post-reload relocation, is set to 0.  It then seeks to offset 256 (base
offset for the objects).


File: internals.info,  Node: Data dumping,  Next: Pointers dumping,  Prev: The header,  Up: Dumping phase

39.4.4 Data dumping
-------------------

The data is dumped in the same order as the addresses were allocated by
`pdump_dump_data()', called from `pdump_scan_by_alignment()'.  This
function copies the data to a temporary buffer, relocates all pointers
in the object to the addresses allocated in step Address Allocation,
and writes it to the file.  Using the same order means that, if we are
careful with lrecords whose size is not a multiple of 4, we are ensured
that the object is always written at the offset in the file allocated
in step Address Allocation.


File: internals.info,  Node: Pointers dumping,  Prev: Data dumping,  Up: Dumping phase

39.4.5 Pointers dumping
-----------------------

A bunch of tables needed to reassign properly the global pointers are
then written.  They are:

  1. the pdump_root_block_ptrs dynarr

  2. the pdump_opaques dynarr

  3. a vector of all the offsets to the objects in the file that
     include a description (for faster relocation at reload time)

  4. the pdump_root_objects and pdump_weak_object_chains dynarrs.

   For each of the dynarrs we write both the pointer to the variables
and the relocated offset of the object they point to.  Since these
variables are global, the pointers are still valid when restarting the
program and are used to regenerate the global pointers.

   The `pdump_weak_object_chains' dynarr is a special case.  The
variables it points to are the head of weak linked lists of lisp objects
of the same type.  Not all objects of this list are dumped so the
relocated pointer we associate with them points to the first dumped
object of the list, or Qnil if none is available.  This is also the
reason why they are not used as roots for the purpose of object
enumeration.

   Some very important information like the `staticpros' and
`lrecord_implementations_table' are handled indirectly using
`dump_add_opaque' or `dump_add_root_block_ptr'.

   This is the end of the dumping part.


File: internals.info,  Node: Reloading phase,  Next: Remaining issues,  Prev: Dumping phase,  Up: Dumping

39.5 Reloading phase
====================

39.5.1 File loading
-------------------

The file is mmap'ed in memory (which ensures a PAGESIZE alignment, at
least 4096), or if mmap is unavailable or fails, a 256-bytes aligned
malloc is done and the file is loaded.

   Some variables are reinitialized from the values found in the header.

   The difference between the actual loading address and the
reloc_address is computed and will be used for all the relocations.

39.5.2 Putting back the pdump_opaques
-------------------------------------

The memory contents are restored in the obvious and trivial way.

39.5.3 Putting back the pdump_root_block_ptrs
---------------------------------------------

The variables pointed to by pdump_root_block_ptrs in the dump phase are
reset to the right relocated object addresses.

39.5.4 Object relocation
------------------------

All the objects are relocated using their description and their offset
by `pdump_reloc_one'.  This step is unnecessary if the reloc_address is
equal to the file loading address.

39.5.5 Putting back the pdump_root_objects and pdump_weak_object_chains
-----------------------------------------------------------------------

Same as Putting back the pdump_root_block_ptrs.

39.5.6 Reorganize the hash tables
---------------------------------

Since some of the hash values in the lisp hash tables are
address-dependent, their layout is now wrong.  So we go through each of
them and have them resorted by calling `pdump_reorganize_hash_table'.


File: internals.info,  Node: Remaining issues,  Prev: Reloading phase,  Up: Dumping

39.6 Remaining issues
=====================

The build process will have to start a post-dump xemacs, ask it the
loading address (which will, hopefully, be always the same between
different xemacs invocations) [[unfortunately, not true on Linux with
the ExecShield feature]] and relocate the file to the new address.
This way the object relocation phase will not have to be done, which
means no writes in the objects and that, because of the use of mmap, the
dumped data will be shared between all the xemacs running on the
computer.

   Some executable signature will be necessary to ensure that a given
dump file is really associated with a given executable, or random
crashes will occur.  Maybe a random number set at compile or configure
time thru a define.  This will also allow for having
differently-compiled xemacsen on the same system (mule and no-mule
comes to mind).

   The DOC file contents should probably end up in the dump file.


File: internals.info,  Node: Future Work,  Next: Future Work Discussion,  Prev: Dumping,  Up: Top

40 Future Work
**************

* Menu:

* Future Work -- General Suggestions::
* Future Work -- Elisp Compatibility Package::
* Future Work -- Drag-n-Drop::
* Future Work -- Standard Interface for Enabling Extensions::
* Future Work -- Better Initialization File Scheme::
* Future Work -- Keyword Parameters::
* Future Work -- Property Interface Changes::
* Future Work -- Toolbars::
* Future Work -- Menu API Changes::
* Future Work -- Removal of Misc-User Event Type::
* Future Work -- Mouse Pointer::
* Future Work -- Extents::
* Future Work -- Version Number and Development Tree Organization::
* Future Work -- Improvements to the `xemacs.org' Website::
* Future Work -- Keybindings::
* Future Work -- Byte Code Snippets::
* Future Work -- Lisp Stream API::
* Future Work -- Multiple Values::
* Future Work -- Macros::
* Future Work -- Specifiers::
* Future Work -- Display Tables::
* Future Work -- Making Elisp Function Calls Faster::
* Future Work -- Lisp Engine Replacement::
* Future Work -- Better Rendering Support::


File: internals.info,  Node: Future Work -- General Suggestions,  Next: Future Work -- Elisp Compatibility Package,  Prev: Future Work,  Up: Future Work

40.1 Future Work - General Suggestions
======================================

Jamie Zawinski's XEmacs Wishlist
--------------------------------

This document is based on Jamie Zawinski's xemacs wishlist
(http://www.jwz.org/doc/xemacs-wishlist.html).        Throughout this
page, "I" refers to Jamie.

   The list has been substantially reformatted and edited to fit the
needs       of this site. If you have any soul at all, you'll go check
out the       original. OK? You should also check out some other
wishlists
(http://www.xemacs.org/Releases/Public-21.2/execution.html#wishlists).

About the List
..............

I've ranked these (roughly) from easiest to hardest; though of all of
them, I think the debugger improvements would be the most useful. I
think the combination of emacs+gdb is the best Unix development
environment currently available, but it's still lamentably primitive
and extremely frustrating (much like Unix itself), especially if you
know what kinds of features more modern integrated debuggers have.

XEmacs Wishlist
...............

*Improve the keyboard macro system.*
     Keyboard macros are one of the most useful concepts that emacs has
     to offer, but there's room for improvement.

    *Make it possible to embed one macro inside of another.*
          Often, I'll define a keyboard macro, and then realize that
          I've left something out, or that there's more that I need to
          do; for example, I may define a macro that does something to
          the current line, and then realize that I want to apply it to
          a lot of lines. So, I'd like this to work:

               C-x (
               ; start macro #1
               ...
               ; (do stuff)
               C-x )
               ; done with macro #1
               ...
               ; (do stuff)
               C-x (
               ; start macro #2
               C-x e
               ; execute macro #1 (splice it into macro #2)
               C-s foo
               ; move forward to the next spot
               C-x )
               ; done with macro #2
               C-u 1000 C-x e
               ; apply the new macro

          That is, simply, one should be able to wrap new text around an
          existing macro. I can't tell you how many times I've defined
          a complex macro but left out the "`C-n C-a'" at the end...

          Yes, you can accomplish this with M-x name-last-kbd-macro, but
          that's a pain. And it's also more permanent than I'd often
          like.

    *Make it possible to correct errors when defining a macro.*
          Right now, the act of defining a macro stops if you get an
          error while defining it, and all of the characters you've
          already typed into the macro are gone. It needn't be that
          way. I think that, when that first error occurs, the user
          should be given the option of taking the last command off of
          the macro and trying again.

          The macro-reader knows where the bounds of multi-character
          command sequences are, and it could even keep track of the
          corresponding undo records; rubbing out the previous entry on
          the macro could also undo any changes that command had made.
          (This should also work if the macro spans multiple buffers,
          and should restore window configurations as well.)

          You'd want multi-level undo for this as well, so maybe the
          way to go would be to add some new key sequence which was
          used only as the back-up-inside-a-keyboard-macro-definition
          command.

          I'm not totally sure that this would end up being very usable;
          maybe it would be too hard to deal with. Which brings us to:

    *Make it possible to edit a keyboard macro after it has been defined.*
          I only just discovered `edit-kbd-macro' (`C-x C-k').  It is
          very, very cool.

          The trick it does of showing the command which will be
          executed is somewhat error-prone, as it can only look up
          things in the current map or the global map; if the macro
          changed buffers, it wouldn't be displaying the right
          commands. (One of the things I often use macros for is
          operating on many files at once, by bringing up a dired buffer
          of those files, editing them, and then moving on to the next.)

          However, if the act of recording a macro also kept track of
          the actual commands that had gotten executed, it could make
          use of that info as well.

          Another way of editing a macro, other than as text in a
          buffer, would be to have a command which single-steps a
          macro: you would lean on the space bar to watch the macro
          execute one character (command?)  at a time, and then when
          you reached the point you wanted to change, you could do some
          gesture to either: insert some keystrokes into the middle of
          the macro and then continue; or to replace the rest of the
          macro from here to the end; or something.

          Another similar hack might be to convert a macro to the
          equivalent lisp code, so that one could tweak it later in
          ways that would be too hard to do from the keyboard (wrapping
          parts of it in `while' loops or something.) (`M-x
          insert-kbd-macro' isn't really what I'm talking about here: I
          mean insert the list of commands, not the list of keystrokes.)

*Save my wrists!*
     In the spirit of the ``teach-extended-commands-p'' variable, it
     would be interesting if emacs would keep track of what are the
     commands I use most often, perhaps grouped by proximity or mode -
     it would then be more obvious which commands were most likely
     candidates for placement on a toolbar, or popup menu, or just a
     more convenient key binding.

     Bonus points if it figures out that I type "`bt\n'" and
     "`ret\ny\n'" into my `*gdb*' buffer about a hundred thousand times
     a day.

*XmCreateFileSelectionBox*
     The thing that "File/Open..." pops up has excellent _hack_ value,
     but as a user interface, it's an abomination. Isn't it time
     someone added a real file selection dialog already? (For the
     Motifly-challenged, the Athena-based file selector that GhostView
     uses seems adequate.)

*Improve the toolbar system.*
     It's great that XEmacs has a toolbar, but it's damn near impossible
     to customize it.

    *Make it easy to define new toolbar buttons.*
          Currently, to define a toolbar button that has a text
          equivalent, one must edit a pixmap, and put the text there!
          That's prohibitive.  One should be able to add some kind of
          generic toolbar button, with a plain icon or none at all, but
          which has a text label, without having to use a paint program.

    *Make it easy to have customized, mode-local toolbars.*
          In my `c-mode-hook', for example, I can add a couple of new
          keybindings, and delete a few others, and to do that, I don't
          have to duplicate the entire definition of the `c-mode-map'.
          Making mode-local additions and subtractions to the toolbars
          should be as easy.

    *Make it easy to have customized, mode-local popup menus.*
          The same situation holds for the right-mouse-button popup
          menu; one should be able to add new commands to those menus
          without difficulty.  One problem is that each mode which does
          have a popup menu implements it in a different way...

*Make the External Widget work.*
     About half of the work is done to make a replacement for the
     `XmText' widget which offloads editing responsibility to an
     external Emacs process. Someone should finish that. The benefit
     here would be that then, any Motif program could be linked such
     that all editing happened with a real Emacs behind it. (If you're
     Athena-minded, flavor with `Text' instead of `XmText' - it's
     probably easy to make it work with both.)

     The part of this that is done already is the ability to run an
     Emacs screen on a Window object that has been created by another
     process (this is what the `ExternalClient.c' and `ExternalShell.c'
     stuff is.) What is left to be done is, adding the
     text-widget-editor aspects of this.

     First, the emacs screen being displayed on that window would have
     to be one without a modeline, and one which behaved sensibly in
     the context of "I am a small multi-line text area embedded in a
     dialog box" as opposed to "I am a full-on text editor and lord of
     all that I survey."

     Second, the API that the (non-emacs-aware) user of the `XmText'
     widget expects would need to be implemented: give the caller the
     ability to pull the edited text string back out, and so on.  The
     idea here being, hooking up emacs as the widget editor should be as
     transparent as possible.

*Bring the debugger interface into the eighties.*
     Some of you may have seen my `gdb-highlight.el' package, that I
     posted to gnu.emacs.sources last month. I think it's really cool,
     but there should be a lot more work in that direction.  For those
     of you who haven't seen it, what it does is watch text that gets
     inserted into the `*gdb*' buffer and make very nearly everything
     be clickable and have a context-sensitive menu. Generally, the
     types that are noticed are:

        * function names;

        * variable and parameter names;

        * structure slots;

        * source file names;

        * type names;

        * breakpoint numbers;

        * stack frame numbers.

     Any time one of those objects is presented in the `*gdb*' buffer,
     it is mousable. Clicking middle button on it takes some default
     action (edits the function, selects the stack frame, disables the
     breakpoint, ...) Clicking the right button pops up a menu of
     commands, including commands specific to the object under the
     mouse, and/or other objects on the same line.

     So that's all well and good, and I get far more joy out of what
     this code does for me than I expected, but there are still a bunch
     of limitations. The debugger interface needs to do much, much more.

    *Make gdbsrc-mode not suck.*
          The idea behind `gdbsrc-mode' is on the side of the angels:
          one should be able to focus on the source code and not on the
          debugger buffer, absolutely. But the implementation is just
          awful.

          First and foremost, it should not change "modes" (in the more
          general sense). Any commands that it defines should be on
          keys which are exclusively used for that purpose, not keys
          which are normally self-inserting. I can't be the only person
          who usually has occasion to actually _edit_ the sources which
          the debugger has chosen to display! Switching into and out of
          `gdbsrc-mode' is prohibitive.

          I want to be looking at my sources at all times, yet I don't
          want to have to give up my source-editing gestures. I think
          the right way to accomplish this is to put the gdbsrc
          commands on the toolbar and on popup menus; or to let the
          user define their own keys (I could see devoting my
          <kp_enter> key to "step", or something common like that.)

          Also it's extremely frustrating that one can't turn off
          gdbsrc mode once it has been loaded, without exiting and
          restarting emacs; that alone means that I'd probably never
          take the time to learn how to use it, without first having
          taken the time to repair it...

    *Make it easier access to variable values.*
          I want to be able to double-click on a variable name to
          highlight it, and then drag it to the debugger window to have
          its value printed.

          I want gestures that let me write as well as read: for
          example, to store value A into slot B.

    *Make all breakpoints visible.*
          Any time there is a running gdb which has breakpoints, the
          buffers holding the lines on which those breakpoints are set
          should have icons in them. These icons should be
          context-sensitive: I should be able to pop up a menu to
          enable or disable them, to delete them, to change their
          commands or conditions.

          I should also be able to _move_ them. It's annoying when you
          have a breakpoint with a complex condition or command on it,
          and then you realize that you really want it to be at a
          different location. I want to be able to drag-and-drop the
          icon to its new home.

    *Make a debugger status display window.*
             *  I want a window off to the side that shows persistent
               information - it should have a pane which is a
               drag-editable, drag-reorderable representation of the
               elements on gdb's "display" list; they should be
               displayed here instead of being just dumped in with the
               rest of the output in the `*gdb*' buffer.

             *  I want a pane that displays the current call-stack and
               nothing else. I want a pane that displays the arguments
               and locals of the currently-selected frame and nothing
               else. I want these both to update as I move around on
               the stack.

             *  Since the unfortunate reality is that excavating this
               information from gdb can be slow, it would be a good
               idea for these panes to have a toggle button on them
               which meant "stop updating", so that when I want to move
               fast, I can, but I can easily get the display back when
               I need it again.

          The reason for all of this is that I spend entirely too much
          time scrolling around in the `*gdb*' buffer; with
          gdb-highlight, I can just click on a line in the backtrace
          output to go to that frame, but I find that I spend a lot of
          time _looking_ for that backtrace: since it's mixed in with
          all the other random output, I waste time looking around for
          things (and usually just give up and type "`bt'" again, then
          thrash around as the buffer scrolls, and I try to find the
          lower frames that I'm interested in, as they have invariably
          scrolled off the window already...

    *Save and restore breakpoints across emacs/debugger sessions.*
          This would be especially handy given that gdb leaks like a
          sieve, and with a big program, I only get a few dozen
          relink-and-rerun attempts before gdb has blown my swap space.

    *Keep breakpoints in sync with source lines.*
          When a program is recompiled and then reloaded into gdb, the
          breakpoints often end up in less-than-useful places. For
          example, when I edit text which occurs in a file anywhere
          before a breakpoint, emacs is aware that the line of the bp
          hasn't changed, but just that it is in a different place
          relative to the top of the file. Gdb doesn't know this, so
          your breakpoints end up getting set in the wrong places
          (usually the maximally inconvenient places, like _after_ a
          loop instead of _inside_ it). But emacs knows, so emacs should
          inform the debugger, and move the breakpoints back to the
          places they were intended to be.

     (Possibly the OOBR stuff does some of this, but can't tell, because
     I've never been able to get it to do anything but beep at me and
     mumble about environments. I find it pretty funny that the manual
     keeps explaining to me how intuitive it is, without actually
     giving me a clue how to launch it...)

*Add better dialog box features.*
     It'd be nice to be able to create more complex dialog boxes from
     emacs-lisp: ones with checkboxes, radio button groups, text
     fields, and popup menus.

*Add embeddable dialog boxes.*
     One of the things that the now-defunct Energize code (the C side of
     it, that is) could do was embed a dialog box between the toolbar
     and the main text area - buffers could have control panels
     associated with them, that had all kinds of complex behavior.

*Make the mark-stack be visible.*
     You know, I've encountered people who have been using emacs for
     years, and never use the mark stack for navigation. I can't live
     without it; "`C-u C-SPC'" is among my most common gestures.

       1.  It would be a lot easier to realize what's going to happen
          if the marks on the mark stack were visible. They could be
          displayed as small "caret" glyphs, for example; something
          large enough to be visible, but not easily mistaken for a
          character or for the cursor.

       2.  The marks and the selected region should be visible in the
          scrollbar as well - I don't remember where I first saw this
          idea, but it's very cool: there's a second,
          less-strongly-rendered "thumb" in the scrollbar which
          indicates the position and size of the selection; and there
          are tiny tick-marks which indicate the positions of the saved
          points.

       3.  Markers which are in registers (`point-to-register', `C-x
          /') should be displayed differently (more prominent.)

       4.  It'd be cool if you could pick up markers and move them
          around, to adjust the points you'll be coming back to later.

*Write a new garbage collector.*
     The emacs GC is very primitive; it is also, fortunately, a rather
     well isolated module, and it would not be a very big task to swap
     it with a new one (once that new one was written, that is.) Someone
     should go bone up on modern GC techniques, and then just dive right
     in...

*Add support for lexical scope to the emacs-lisp runtime.*
     Yadda yadda, this list goes to eleven.

Subject: *Re: XEmacs wishlist* Date: Wed, 14 May 1997 16:18:23 -0700
From: Jamie Zawinski <jwz@netscape.com> Newsgroups: comp.emacs.xemacs,
comp.emacs

   Andreas Schwab wrote:

     _Use `C-u C-x (': _

     _start-kbd-macro:
     Non-nil arg (prefix arg) means append to last macro defined; This
     begins by re-executing that macro as if you typed it again. _

   Cool, I didn't know it did that...

   But it only lets you append. I often want to prepend, or embed the
macro multiple times (motion 1, C-x e, motion 2, C-x e, motion 3.)

21.2 Showstoppers
-----------------

Author: Ben Wing (mailto:ben@xemacs.org)

   DISTRIBUTION ISSUES

   A. Unified Source Tarball.

   Packages go under root/lib/xemacs/xemacs-packages and no one ever has
to mess with -package-path and the result can be moved from one
directory to another pre- or post-install.

   Unified Binary Tarballs with Packages.

   Same principles as above.

   If people complain, we can also provide split binary tarballs
(architecture dependent and independent) and place these files in a
subdirectory so as not to confuse the majority just looking for one
tarball.

   Under Windows, we need to provide a WISE-style GUI setup program.
It's already there but needs some work so you can select "all" packages
easily (should be the default).

   Parallel Root and Package Trees.

   If the user downloads separately, the main source and the packages,
he will naturally untar them into the same directory. This results in
the parallel root and package structure. We should support this as a
"last resort," i.e., if we find no packages anywhere and are about to
resign ourselves to not having packages, then look for a parallel
package tree. The user who sets things up like this should be able to
either run in place or "make install" and get a proper installed
XEmacs. Never should the user have to touch -package-path.

   II. WINDOWS PRINTING

   Looks like the internals are done but not the GUI. This must be
working in 21.2.

   III. WINDOWS MULE

   Basic support should be there. There's already a patch to get things
started and I'll be doing more work to make this real.

   IV. GUTTER ETC.

   This stuff needs to be "stable" and generally free from bugs. Any
API's we create need to be well-reviewed or marked clearly as
experimental.

   V. PORTABLE DUMPER

   Last bits need to be cleaned up. This should be made the "default"
for a while to flush-out problems. Under Microsoft Windows, Portable
Dumper must be the default in 21.2 because of the problems with the
existing dump process.

   COMMENT: I'd like to feature freeze this pretty soon and create a
21.3 tree where all of my major overhauls of Mule-related stuff will go
in. At the same time or around, we need to do the move-around in the
repository (or create a new one) and "upgrade" to the latest CVS server.


File: internals.info,  Node: Future Work -- Elisp Compatibility Package,  Next: Future Work -- Drag-n-Drop,  Prev: Future Work -- General Suggestions,  Up: Future Work

40.2 Future Work - Elisp Compatibility Package
==============================================

Author: Ben Wing (mailto:ben@xemacs.org)

   A while ago I created a package called Sysdep, which aimed to be a
forward compatibility package for Elisp.  The idea was that instead of
having to write your package using the oldest version of Emacs that you
wanted to support, you could use the newest XEmacs API, and then simply
load the Sysdep package, which would automatically define the new API in
terms of older APIs as necessary.  The idea of this package was good,
but its design wasn't perfect, and it wasn't widely adopted.  I propose
a new package called Compat that corrects the design flaws in Sysdep,
and hopefully will be adopted by most of the major packages.

   In addition, this package will provide macros that can be used to
bracket code as necessary to disable byte compiler warnings generated as
a result of supporting the APIs of different versions of Emacs; or
rather the Compat package strives to provide useful constructs to make
doing this support easier, and these constructs have the side effect of
not causing spurious byte compiler warnings.  The idea here is that it
should be possible to create well-written, clean, and understandable
Elisp that supports both older and newer APIs, and has no byte compiler
warnings.  Currently many warnings are unavoidable, and as a result,
they are simply ignored, which also causes a lot of legitimate warnings
to be ignored.

   The approach taken by the Sysdep package to make sure that the newest
API was always supported was fairly simple: when the Sysdep package was
loaded, it checked for the existence of new API functions, and if they
weren't defined, it defined them in terms of older API functions that
were defined.  This had the advantage that the checks for which API
functions were defined were done only once at load time rather than each
time the function was called.  However, the fact that the new APIs were
globally defined caused a lot of problems with unwanted interactions,
both with other versions of the Sysdep package provided as part of other
packages, and simply with compatibility code of other sorts in packages
that would determine whether an API existed by checking for the
existence of certain functions within that API.  In addition, the Sysdep
package did not scale well because it defined all of the functions that
it supported, regardless of whether or not they were used.

   The Compat package remedies the first problem by ensuring that the
new APIs are defined only within the lexical scope of the packages that
actually make use of the Compat package.  It remedies the second problem
by ensuring that only definitions of functions that are actually used
are loaded.  This all works roughly according to the following scheme:

  1.  Part of the Compat package is a module called the Compat
     generator.  This module is actually run as an additional step
     during byte compilation of a package that uses Compat.  This can
     happen either through the makefile or through the use of an
     `eval-when-compile' call within the package code itself.  What the
     generator does is scan all of the Lisp code in the package,
     determine which function calls are made that the Compat package
     knows about, and generates custom `compat' code that conditionally
     defines just these functions when the package is loaded.  The
     custom `compat' code can either be written to a separate Lisp file
     (for use with multi-file packages), or inserted into the beginning
     of the Lisp file of a single file package.  (In the latter case,
     the package indicates where this generated code should go through
     the use of magic comments that mark the beginning and end of the
     section.  Some will say that doing this trick is bad juju, but I
     have done this sort of thing before, and it works very well in
     practice).

  2.  The functions in the custom `compat' code have their names
     prefixed with both the name of the package and the word `compat',
     ensuring that there will be no name space conflicts with other
     functions in the same package, or with other packages that make
     use of the Compat package.

  3.  The actual definitions of the functions in the custom `compat'
     code are determined at run time.  When the equivalent API already
     exists, the wrapper functions are simply defined directly in terms
     of the actual functions, so that the only run time overhead from
     using the Compat package is one additional function call.
     (Alternatively, even this small overhead could be avoided by
     retrieving the definitions of the actual functions and supplying
     them as the definitions of the wrapper functions.  However, this
     appears to me to not be completely safe.  For example, it might
     have bad interactions with the advice package).

  4.  The code that wants to make use of the custom `compat' code is
     bracketed by a call to the construct `compat-execute'.  What this
     actually does is lexically bind all of the function names that are
     being redefined with macro functions by using the Common Lisp
     macro macrolet.  (The definition of this macro is in the CL
     package, but in order for things to work on all platforms, the
     definition of this macro will presumably have to be copied and
     inserted into the custom `compat' code).


   In addition, the Compat package should define the macro
`compat-if-fboundp'.  Similar macros such as `compile-when-fboundp' and
`compile-case-fboundp' could be defined using similar principles).  The
`compat-if-fboundp' macro behaves just like an `(if (fboundp ...) ...)'
clause when executed, but in addition, when it's compiled, it ensures
that the code inside the `if-true' sub-block will not cause any byte
compiler warnings about the function in question being unbound.  I
think that the way to implement this would be to make
`compat-if-fboundp' be a macro that does what it's supposed to do, but
which defines its own byte code handler, which ensures that the
particular warning in question will be suppressed.  (Actually ensuring
that just the warning in question is suppressed, and not any others,
might be rather tricky.  It certainly requires further thought).

   Note: An alternative way of avoiding both warnings about unbound
functions and warnings about obsolete functions is to just call the
function in question by using `funcall', instead of calling the
function directly.  This seems rather inelegant to me, though, and
doesn't make it obvious why the function is being called in such a
roundabout manner.  Perhaps the Compat package should also provide a
macro `compat-funcall', which works exactly like `funcall', but which
indicates to anyone reading the code why the code is expressed in such
a fashion.

   If you're wondering how to implement the part of the Compat generator
where it scans Lisp code to find function calls for functions that it
wants to do something about, I think the best way is to simply process
the code using the Lisp function `read' and recursively descend any
lists looking for function names as the first element of any list
encountered.  This might extract out a few more functions than are
actually called, but it is almost certainly safer than doing anything
trickier like byte compiling the code, and attempting to look for
function calls in the result.  (It could also be argued that the names
of the functions should be extracted, not only from the first element of
lists, but anywhere `symbol' occurs.  For example, to catch places
where a function is called using `funcall' or `apply'.  However, such
uses of functions would not be affected by the surrounding macrolet
call, and so there doesn't appear to be any point in extracting them).


File: internals.info,  Node: Future Work -- Drag-n-Drop,  Next: Future Work -- Standard Interface for Enabling Extensions,  Prev: Future Work -- Elisp Compatibility Package,  Up: Future Work

40.3 Future Work - Drag-n-Drop
==============================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* I propose completely redoing the drag-n-drop interface
to make it powerful and extensible enough to support such concepts as
drag over and drag under visuals and context menus invoked when a drag
is done with the right mouse button, to allow drop handlers to be
defined for all sorts of graphical elements including buffers, extents,
mode lines, toolbar items, menubar items, glyphs, etc., and to allow
different packages to add and remove drop handlers for the same drop
sites without interfering with each other.  The changes are extensive
enough that I think they can only be implemented in version 22, and the
drag-n-drop interface should remain experimental until then.

   The new drag-n-drop interface centers around the twin concepts of
"drop site" and "drop handler".  A "drop site" specifies a particular
graphical element where an object can be dropped onto, and a "drop
handler" encapsulates all of the behavior that happens when such an
object is dragged over and dropped onto a drop site.

   Each drop site has an object associated with it which is passed to
functions that are part of the drop handlers associated with that site.
The type of this object depends on the graphical element that comprises
the drop site.  The drop site object can be a buffer, an extent, a
glyph, a menu path, a toolbar item path, etc.  (These last two object
types are defined in Lisp Interface Changes (lisp-interface.html) in
the sections on menu and toolbar API changes.  If we wanted to allow
drops onto other kinds of drop sites, for example mode lines, we would
have to create corresponding path objects).  Each such object type
should be able to be accessed using the generalized property interface
defined above, and should have a property called `drop-handlers'
associated with it that specifies all of the drop handlers associated
with the drop site.  Normally, this property is not accessed directly,
but instead by using the drop handler API defined below, and Lisp
packages should not make any assumptions about the format of the data
contained in the `drop-handlers' property.

   Each drop handler has an object of type `drop-handler' associated
with it, whose primary purpose is to be a container for the various
properties associated with a particular drop handler.  These could
include, for example, a function invoked when the drop occurs, a context
menu invoked when a drop occurs as a result of a drag with the right
mouse button, functions invoked when a dragged object enters, leaves, or
moves within a drop site, the shape that the mouse pointer changes to
when an object is dragged over a drop site that allows this particular
object to be dropped onto it, the MIME types (actually a regular
expression matching the MIME types) of the allowable objects that can be
dropped onto the drop site, a "package tag" (a symbol specifying the
package that created the drop handler, used for identification
purposes), etc.  The drop handler object is passed to the functions that
are invoked as a result of a drag or a drop, most likely indirectly as
one of the properties of the drag or drop event passed to the function.
Properties of a drop handler object are accessed and modified in the
standard fashion using the generalized property interface.

   A drop handler is added to a drop site using the `add-drop-handler'
function.  The drop handler itself can either be created separately
using the `make-drop-handler' function and then passed in as one of the
parameters to `add-drop-handler', or it will be created automatically
by the `add-drop-handler' function, if the drop handler argument is
omitted, but keyword arguments corresponding to the valid keyword
properties for a drop handler are specified in the `add-drop-handler'
call.  Other functions, such as `find-drop-handler', `add-drop-handler'
(when specifying a drop handler before which the drop handler in
question is to be added), `remove-drop-handler' etc. should be defined
with obvious semantics.  All of these functions take or return a drop
site object which, as mentioned above, can be one of several object
types corresponding to graphical elements.  Defined drop handler
functions locate a particular drop handler using either the `MIME-type'
or `package-tag' property of the drop handler, as defined above.

   Logically, the drop handlers associated with a particular drop site
are an ordered list.  The first drop handler whose specified MIME type
matches the MIME type of the object being dragged or dropped controls
what happens to this object.  This is important particularly because the
specified MIME type of the drop handler can be a regular expression
that, for example, matches all audio objects with any sub-type.

   In the current drag-n-drop API, there is a distinction made between
objects with an associated MIME type and objects with an associated URL.
I think that this distinction is arbitrary, and should not exist.  All
objects should have a MIME type associated with them, and a new
XEmacs-specific MIME type should be defined for URLs, file names, etc.
as necessary.  I am not even sure that this is necessary, however, as
the MIME specification may specify a general concept of a pointer or
link to an object, which is exactly what we want.  Also in some cases
(for example, the name of a file that is locally available), the pointer
or link will have another MIME type associated with it, which is the
type of the object that is being pointed to.  I am not quite sure how we
should handle URL and file name objects being dragged, but I am positive
that it needs to be integrated with the mechanism used when an object
itself is being dragged or dropped.

   As is described in a separate page (misc-user-event.html), the
`misc-user-event' event type should be removed and split up into a
number of separate event types.  Two such event types would be
`drag-event' and `drop-event'.  A drop event is used when an object is
actually dropped, and a drag event is used if a function is invoked as
part of the dragging process.  (Such a function would typically be used
to control what are called "drag under visuals", which are changes to
the appearance of the drop site reflecting the fact that a compatible
object is being dragged over it).  The drag events and drop events
encapsulate all of the information that is pertinent to the drag or
drop action occurring, including such information as the actual MIME
type of the object in question, the drop handler that caused a function
to be invoked, the mouse event (or possibly even a keyboard event)
corresponding to the user's action that is causing the drag or drop,
etc.  This event is always passed to any function that is invoked as a
result of the drag or drop.  There should never be any need to refer to
the `current-mouse-event' variable, and in fact, this variable should
not be changed at all during a drag or a drop.


File: internals.info,  Node: Future Work -- Standard Interface for Enabling Extensions,  Next: Future Work -- Better Initialization File Scheme,  Prev: Future Work -- Drag-n-Drop,  Up: Future Work

40.4 Future Work - Standard Interface for Enabling Extensions
=============================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* Apparently, if you know the name of a package (for
example, `fusion'), you can load it using the `require' function, but
there's no standard way to turn it on or turn it off.  The only way to
figure out how to do that is to go read the source file, where
hopefully the comments at the start tell you the appropriate magic
incantations that you need to run in order to turn the extension on or
off.  There really needs to be standard functions, such as
`enable-extension' and `disable-extension', to do this sort of thing.
It seems like a glaring omission that this isn't currently present, and
it's really surprising to me that nobody has remarked on this.

   The easy part of this is defining the interface, and I think it
should be done as soon as possible.  When the package is loaded, it
simply calls some standard function in the package system, and passes
it the names of enable and disable functions, or perhaps just one
function that takes an argument specifying whether to enable or
disable.  In any case, this data is kept in a table which is used by the
`enable-extension' and `disable-extension' function.  There should also
be functions such as `extension-enabled-p' and
`enabled-extension-list', and so on with obvious semantics.  The hard
part is actually getting packages to obey this standard interface, but
this is mitigated by the fact that the changes needed to support this
interface are so simple.

   I have been conceiving of these enabling and disabling functions as
turning the feature on or off globally.  It's probably also useful to
have a standard interface returning a extension on or off in just the
particular buffer.  Perhaps then the appropriate interface would involve
registering a single function that takes an argument that specifies
various things, such as turn off globally, turn on globally, turn on or
off in the current buffer, etc.

   Part of this interface should specify the correct way to define
global key bindings.  The correct rule for this, of course, is that the
key bindings should not happen when the package is loaded, which is
often how things are currently done, but only when the extension is
actually enabled.  The key bindings should go away when the extension is
disabled.  I think that in order to support this properly, we should
expand the keymap interface slightly, so that in addition to other
properties associated with each key binding is a list of shadow
bindings.  Then there should be a function called
`define-key-shadowing', which is just like `define-key' but which also
remembers the previous key binding in a shadow list.  Then there can be
another function, something like `undefine-key', which restores the
binding to the most recently added item on the shadow list.  There are
already hash tables associated with each key binding, and it should be
easy to stuff additional values, such as a shadow list, into the hash
table.  Probably there should also be functions called
`global-set-key-shadowing' and `global-unset-key-shadowing' with
obvious semantics.

   Once this interface is defined, it should be easy to expand the
custom package so it knows about this interface.  Then it will be
possible to put all sorts of extensions on the options menu so that
they could be turned off and turned on very easily, and then when you
save the options out to a file, the design settings for whether these
extensions are enabled or not are saved out with it.  A whole lot of
custom junk that's been added to a lot of different packages could be
removed.  After doing this, we might want to think of a way to classify
extensions according to how likely we think the user will want to use
them.  This way we can avoid the problem of having a list of 100
extensions and the user not being able to figure out which ones might
be useful.  Perhaps the most useful extensions would appear immediately
on the extensions menu, and the less useful ones would appear in a
submenu of that, and another submenu might contain even less useful
extensions.  Of course the package authors might not be too happy with
this, but the users probably will be.  I think this at least deserves a
thought, although it's possible you might simply want to maintain a
list on the web site of extensions and a judgment on first of all, how
commonly a user might want this extension, and second of all, how well
written and bug-free the package is.  Both of these sorts of judgments
could be obtained by doing user surveys if need be.


File: internals.info,  Node: Future Work -- Better Initialization File Scheme,  Next: Future Work -- Keyword Parameters,  Prev: Future Work -- Standard Interface for Enabling Extensions,  Up: Future Work

40.5 Future Work - Better Initialization File Scheme
====================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* A proposal is outlined for converting XEmacs to use the
`.xemacs' subdirectory for its initialization files instead of putting
them in the user's home directory.  In the process, a general
pre-initialization scheme is created whereby all of the initialization
parameters, such as the location of the initialization files, whether
these files are loaded or not, where the initial frame is created, etc.
that are currently specified by command line arguments, by environment
variables, and other means, can be specified in a uniform way using
Lisp code.  Reasonable default behavior for everything will still be
provided, and the older, simpler means can be used if desired.
Compatibility with the current location and name of the initialization
file, and the current ill-chosen use for the `.xemacs' directory is
maintained, and the problem of how to gracefully migrate a user from the
old scheme into the new scheme while still allowing the user to use GNU
Emacs or older versions of XEmacs is solved.  A proposal for changing
the way that the initial frame is mapped is also outlined; this would
allow the user's initialization file to control the way that the initial
frame appears without resorting to hacks, while still making echo area
messages visible as they appear, and allowing the user to debug errors
in the initialization file.

Principles in the new scheme
----------------------------

  1.  XEmacs has a defined "pre-initialization process".  This process,
     whose purpose is to compute the values of the parameters that
     control how the initializiaton process proceeds, occurs as early
     as possible after the Lisp engine has been initialized, and in
     particular, it occurs before any devices have been opened, or
     before any initialization parameters are set that could reasonably
     be expected to be changed.  In fact, the pre-initialization
     process should take care of setting these parameters.  The code
     that implements the pre-initialization process should be written
     in Lisp and should be called from the Lisp function
     `normal-top-level', and the general way that the user customizes
     this process should also be done using Lisp code.

  2.  The pre-initialization process involves a number of properties,
     for example the directory containing the user initialization files
     (normally the `.xemacs' subdirectory), the name of the user init
     file, the name of the custom init file, where and what type the
     initial device is, whether and when the initial frame is mapped,
     etc.  A standard interface is provided for getting and setting the
     values of these properties using functions such as
     `set-pre-init-property', `pre-init-property', etc.  At various
     points during the pre-initialization process, the value of many of
     these properties can be undecided, which means that at the end of
     the process, the value of these properties will be derived from
     other properties in some fashion that is specific to each property.

  3.  The default values of these properties are set first from the
     registry under Windows, then from environment variables, then from
     command line switches, such as `-q' and `-nw'.

  4.  One of the command line switches is `-pre-init', whose value is a
     Lisp expression to be evaluated at pre-initialization time,
     similar to the `-eval' command line switch.  This allows any
     pre-initialization property to be set from the command line.

  5.  Let's define the term "to determine a pre-initialization
     property" to mean if the value of a property is undetermined, it
     is computed and set according to a rule that is specific to the
     property.  Then after the pre-init properties are initialized from
     the registry, from the environment variables, from command line
     arguments, two of the pre-init properties (specifically the init
     file directory and the location of the "pre-init file") are
     determined.  The purpose of the pre-init file is to contain Lisp
     code that is run at pre-initialization time, and to control how
     the initialization proceeds.  It is a bit similar to the standard
     init file, but the code in the pre-init file shouldn't do anything
     other than set pre-init properties.  Executing any code that does
     I/O might not produce expected results because the only device that
     will exist at the time is probably a stream device connected to the
     standard I/O of the XEmacs process.

  6.  After the pre-init file has been run, all of the rest of the
     pre-init properties are determined, and these values are then used
     to control the initialization process.  Some of the rules used in
     determining specific properties are:

       1.  If the `.xemacs' sub-directory exists, and it's not
          obviously a package root (which probably means that it
          contains a file like `init.el' or `pre-init.el', or if
          neither of those files is present, then it doesn't contain
          any sub-directories or files that look like what would be in
          a package root), then it becomes the value of the init file
          directory.  Otherwise the user's home directory is used.

       2.  If the init file directory is the user's home directory,
          then the init file is called `.emacs'.  Otherwise, it's
          called `init.el'.

       3.  If the init file directory is the user's home directory,
          then the pre-init file is called `.xemacs-pre-init.el'.
          Otherwise it's called `pre-init.el'. (One of the reasons for
          this rule has to do with the dialog box that might be
          displayed at startup.  This will be described below.)

       4.  If the init file directory is the user's home directory,
          then the custom init file is called `.xemacs-custom-init.el'.
          Otherwise, it's called `custom-init.el'.


  7.  After the first normal device is created, but before any frames
     are created on it, the XEmacs initialization code checks to see if
     the old init file scheme is being used, which is to say that the
     init file directory is the same as the user's home directory.  If
     that's the case, then normally a dialog box comes up (or a
     question is asked on the terminal if XEmacs is being run in a
     non-windowing mode) which asks if the user wants to migrate his
     initialization files to the new scheme.  The possible responses
     are *Yes*, *No*, and *No, and don't ask this again*.  If this last
     response is chosen, then the file `.xemacs-pre-init.el' in the
     user's home directory is created or appended to with a line of
     Lisp code that sets up a pre-init property indicating that this
     dialog box shouldn't come up again.  If the *Yes* option is
     chosen, then any package root files in `.xemacs' are moved into
     `.xemacs/packages', the file `.emacs' is moved into
     `.xemacs/init.el' and `.emacs' in the home directory becomes a
     symlink to this file.  This way some compatibility is still
     maintained with GNU Emacs and older versions of XEmacs.  The code
     that implements this has to be written very carefully to make sure
     that it doesn't accidentally delete or mess up any of the files
     that get moved around.


The custom init file
--------------------

The "custom init file" is where the custom package writes its options.
This obviously needs to be a separate file from the standard init file.
It should also be loaded before the init file rather than after, as is
usually done currently, so that the init file can override these
options if it wants to.

Frame mapping
-------------

In addition to the above scheme, the way that XEmacs handles mapping the
initial frame should be changed.  However, this change perhaps should be
delayed to a later version of XEmacs because of the user visible changes
that it entails and the possible breakage in people's init files that
might occur. (For example, if the rest of the scheme is implemented in
21.2, then this part of the scheme might want to be delayed until
version 22.)  The basic idea is that the initial frame is not created
before the initialization file is run, but instead a banner frame is
created containing the XEmacs logo, a button that allows the user to
cancel the execution of the init file and an area where messages that
are output in the process of running this file are displayed.  This area
should contain a number of lines, which makes it better than the current
scheme where only the last message is visible.  After the init file is
done, the initial frame is mapped.  This way the init file can make face
changes and other such modifications that affect initial frame and then
have the initial frame correctly come up with these changes and not see
any frame dancing or other problems that exist currently.

   There should be a function that allows the initialization file to
explicitly create and map the first frame if it wants to.  There should
also be a pre-init property that controls whether the banner frame
appears (of course it defaults to true) a property controlling when the
initial frame is created (before or after the init file, defaulting to
after), and a property controlling whether the initial frame is mapped
(normally true, but will be false if the `-unmapped' command line
argument is given).

   If an error occurs in the init file, then the initial frame should
always be created and mapped at that time so that the error is displayed
and the debugger has a place to be invoked.


File: internals.info,  Node: Future Work -- Keyword Parameters,  Next: Future Work -- Property Interface Changes,  Prev: Future Work -- Better Initialization File Scheme,  Up: Future Work

40.6 Future Work - Keyword Parameters
=====================================

Author: Ben Wing (mailto:ben@xemacs.org)

   NOTE: These changes are partly motivated by the various
user-interface changes elsewhere in this document, and partly for Mule
support.  In general the various API's in this document would benefit
greatly from built-in keywords.

   I would like to make keyword parameters an integral part of Elisp.
The idea here is that you use the `&amp;key' identifier in the
parameter list of a function and all of the following parameters
specified are keyword parameters.  This means that when these arguments
are specified in a function call, they are immediately preceded in the
argument list by a "keyword", which is a symbol beginning with the `:'
character.  This allows any argument to be specified independently of
any other argument with no need to place the arguments in any
particular order.  This is particularly useful for functions that take
many optional parameters; using keyword parameters makes the code much
cleaner and easier to understand.

   The `cl' package already provides keyword parameters of a sort, but
I would like to make this more integrated and useable in a standard
fashion.  The interface that I am proposing is essentially compatible
with the keyword interface in Common Lisp, but it may be a subset of the
Common Lisp functionality, especially in the first implementation.
There is one departure from the Common Lisp specification that I would
like to make in order to make it much easier to add keyword parameters
to existing functions with optional parameters, and in general, to make
optional and keyword parameters coexist more easily.  The Common Lisp
specification indicates that if a function has both optional and keyword
parameters, the optional parameters are always processed before the
keyword parameters.  This means, for example, that if a function has
three required parameters, two optional parameters, and some number of
keyword parameters following, and the program attempts to call this
function by passing in the three required arguments, and then some
keyword arguments, the first keyword specified and the argument
following it get assigned to the first and second optional parameters as
specified in the function definition.  This is certainly not what is
intended, and means that if a function defines both optional and keyword
parameters, any calls of this function must specify `nil' for all of
the optional arguments before using any keywords.  If the function
definition is later changed to add more optional parameters, all
existing calls to this function that use any keyword arguments will
break.  This problem goes away if we simply process keyword parameters
before the optional parameters.

   The primary changes needed to support the keyword syntax are:

  1.  The subr object type needs to be modified to contain additional
     slots for the number and names of any keyword parameters.

  2.  The implementation of the `funcall' function needs to be modified
     so that it knows how to process keyword parameters.  This is the
     only place that will require very much intricate coding, and much
     of the logic that would need to be added can be lifted directly
     from the `cl' code.

  3.  A new macro, similar to the `DEFUN' macro, and probably called
     `DEFUN_WITH_KEYWORDS', needs to be defined so that built-in Lisp
     primitives containing keywords can be created.  Now, the
     `DEFUN_WITH_KEYWORDS' macro should take an additional parameter
     which is a string, which consists of the part of the lambda list
     declaration for this primitive that begins with the `&amp;key'
     specifier.  This string is parsed in the `DEFSUBR' macro during
     XEmacs initialization, and is converted into the appropriate
     structure that needs to be stored into the subr object.  In
     addition, the MAX_ARGS parameter of the `DEFUN' macro needs to be
     incremented by the number of keyword parameters and these
     parameters are passed to the C function simply as extra parameters
     at the end.  The `DEFSUBR' macro can sort out the actual number of
     required, optional and keyword parameters that the function takes,
     once it has parsed the keyword parameter string.  (An alternative
     that might make the declaration of a primitive a little bit easier
     to understand would involve adding another parameter to the
     `DEFUN_WITH_KEYWORDS' macro that specifies the number of keyword
     parameters.  However, this would require some additional
     complexity in the preprocessor definition of the
     `DEFUN_WITH_KEYWORDS' macro, and probably isn't worth
     implementing).

  4.  The byte compiler would have to be modified slightly so that it
     knows about keyword parameters when it parses the parameter
     declaration of a function.  For example, so that it issues the
     correct warnings concerning calls to that function with incorrect
     arguments.

  5.  The `make-docfile' program would have to be modified so that it
     generates the correct parameter lists for primitives defined using
     the `DEFUN_WITH_KEYWORDS' macro.

  6.  Possibly other aspects of the help system that deal with function
     descriptions might have to be modified.

  7.  A helper function might need to be defined to make it easier for
     primitives that use both the `&amp;rest' and `&amp;key' specifiers
     to parse their argument lists.


Internal API for C primitives with keywords - necessary for many of the new Mule APIs being defined.
----------------------------------------------------------------------------------------------------

       DEFUN_WITH_KEYWORDS (Ffoo, "foo", 2, 5, 6, ALLOW_OTHER_KEYWORDS,
           (ichi, ARG_NIL), (ni, ARG_NIL), (san, ARG_UNBOUND), 0,
           (arg1, arg2, arg3, arg4, arg5)
           )
       {
         ...
       }

       -> C fun of 12 args:

       (arg1, ... arg5, ichi, ..., roku, other keywords)

       Circled in blue is actual example declaration

       DEFUN_WITH_KEYWORDS (Ffoo, "foo", 1,2,0 (bar, baz) <- arg list
       [ MIN ARGS, MAX ARGS, something that could be REST, SPECIFY_DEFAULT or
       REST_SPEC]

       [#KEYWORDS [ ALLOW_OTHER, SPECIFY_DEFAULT, ALLOW_OTHER_SPECIFY_DEFAULT
       6, ALLOW_OTHER_SPECIFY_DEFAULT,

       (ichi, 0) (ni, 0), (san, DEFAULT_UNBOUND), (shi, "t"), (go, "5"),
       (roku, "(current-buffer)")
       <- specifies arguments, default values (string to be read into Lisp
          data during init; then forms evalled at fn ref time.

       ,0 <- [INTERACTIVE SPEC] )

       LO = Lisp_Object

       -> LO Ffoo (LO bar, LO baz, LO ichi, LO ni, LO san, LO shi, LO go,
                   LO roku, int numkeywords, LO *other_keywords)

       #define DEFUN_WITH_KEYWORDS (fun, funstr, minargs, maxargs, argspec, \
                #args, num_keywords, keywordspec, keywords, intspec) \
       LO fun (DWK_ARGS (maxargs, args) \
               DWK_KEYWORDS (num_keywords, keywordspec, keywords))

       #define DWK_KEYWORDS (num_keywords, keywordspec, keywords) \
               DWK_KEYWORDS ## keywordspec (keywords)
               DWK_OTHER_KEYWORDS ## keywordspec)

       #define DWK_KEYWORDS_ALLOW_OTHER (x,y)
               DWK_KEYWORDS (x,y)

       #define DWK_KEYWORDS_ALLOW_OTHER_SPECIFICATIONS (x,y)
               DWK_KEYWORDS_SPECIFY_DEFAULT (x,y)

       #define DWK_KEYWORDS_SPECIFY_DEFAULT (numkey, key)
               ARGLIST_CAR ## numkey key

       #define ARGLT_GRZ (x,y) LO CAR x, LO CAR y


File: internals.info,  Node: Future Work -- Property Interface Changes,  Next: Future Work -- Toolbars,  Prev: Future Work -- Keyword Parameters,  Up: Future Work

40.7 Future Work - Property Interface Changes
=============================================

Author: Ben Wing (mailto:ben@xemacs.org)

   In my past work on XEmacs, I already expanded the standard property
functions of `get', `put', and `remprop' to work on objects other than
symbols and defined an additional function `object-plist' for this
interface.  I'd like to expand this interface further and advertise it
as the standard way to make property changes in objects, especially the
new objects that are going to be defined in order to support the added
user interface features of version 22.  My proposed changes are as
follows:

  1.  A new concept associated with each property called a "default
     value" is introduced.  (This concept already exists, but not in a
     well-defined way.) The default value is the value that the
     property assumes for certain value retrieval functions such as
     `get' when it is "unbound", which is to say that its value has not
     been explicitly specified. Note: the way to make a property
     unbound is to call `remprop'.  Note also that for some built-in
     properties, setting the property to its default value is
     equivalent to making it unbound.

  2.  The behavior of the `get' function is modified.  If the `get'
     function is called on a property that is unbound and the third,
     optional DEFAULT argument is `nil', then the default value of the
     property is returned.  If the DEFAULT argument is not `nil', then
     whatever was specified as the value of this argument is returned.
     For the most part, this is upwardly compatible with the existing
     definition of `get' because all user-defined properties have an
     initial default value of `nil'.  Code that calls the `get'
     function and specifies `nil' for the DEFAULT argument, and expects
     to get `nil' returned if the property is unbound, is almost
     certainly wrong anyway.

  3.  A new function, `get1' is defined.  This function does not take a
     default argument like the `get' function.  Instead, if the property
     is unbound, an error is signaled.  Note: `get' can be implemented
     in terms of `get1'.

  4.  New functions `property-default-value' and `property-bound-p' are
     defined with the obvious semantics.

  5.  An additional function `property-built-in-p' is defined which
     takes two arguments, the first one being a symbol naming an object
     type, and the second one specifying a property, and indicates
     whether the property name has a built-in meaning for objects of
     that type.

  6.  It is not necessary, or even desirable, for all object types to
     allow user-defined properties.  It is always possible to simulate
     user-defined properties for an object by using a weak hash table.
     Therefore, whether an object allows a user to define properties or
     not should depend on the meaning of the object.  If an object does
     not allow user-defined properties, the `put' function should
     signal an error, such as `undefined-property', when given any
     property other than those that are predefined.

  7.  A function called `user-defined-properties-allowed-p' should be
     defined with the obvious semantics.  (See the previous item.)

  8.  Three more functions should be defined, called
     `built-in-property-name-list', `property-name-list', and
     `user-defined-property-name-list'.


   Another idea:

     (define-property-method
       predicate object-type
       predicate cons :(KEYWORD)  (all lists beginning with KEYWORD)

       :put putfun
       :get
       :remprop
       :object-props
       :clear-properties
       :map-properties

       e.g. (define-property-method 'hash-table
              :put #'(lambda (obj key value) (puthash key obj value)))


File: internals.info,  Node: Future Work -- Toolbars,  Next: Future Work -- Menu API Changes,  Prev: Future Work -- Property Interface Changes,  Up: Future Work

40.8 Future Work - Toolbars
===========================

* Menu:

* Future Work -- Easier Toolbar Customization::
* Future Work -- Toolbar Interface Changes::


File: internals.info,  Node: Future Work -- Easier Toolbar Customization,  Next: Future Work -- Toolbar Interface Changes,  Prev: Future Work -- Toolbars,  Up: Future Work -- Toolbars

40.8.1 Future Work - Easier Toolbar Customization
-------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* One of XEmacs' greatest strengths is its ability to be
customized endlessly.  Unfortunately, it is often too difficult to
figure out how to do this.  There has been some recent work like the
Custom package, which helps in this regard, but I think there's a lot
more work that needs to be done.  Here are some ideas (which certainly
could use some more thought).

   Although there is currently an `edit-toolbar' package, it is not
well integrated with XEmacs, and in general it is much too hard to
customize the way toolbars look.  I would like to see an interface that
works a bit like the way things work under Windows, where you can
right-click on a toolbar to get a menu of options that allows you to
change aspects of the toolbar.  The general idea is that if you
right-click on an item itself, you can do things to that item, whereas
if you right-click on a blank part of a toolbar, you can change the
properties of the toolbar.  Some of the items on the right-click menu
for a particular toolbar button should be specified by the button
itself.  Others should be standard.  For example, there should be an
*Execute* item which simply does what would happen if you left-click on
a toolbar button.  There should probably be a *Delete* item to get rid
of the toolbar button and a *Properties* item, which brings up a
property sheet that allows you to do things like change the icon and
the command string that's associated with the toolbar button.

   The options to change the appearance of the toolbar itself should
probably appear both on the context menu for specific buttons, and on
the menu that appears when you click on a blank part of the toolbar.
That way, if there isn't a blank part of the toolbar, you can still
change the toolbar appearance.  As for what appears in these items, in
Outlook Express, for example, there are three different menu items, one
of which is called *Buttons*, which brings up, or pops up a window
which allows you to edit the toolbar, which for us could pop up a new
frame, which is running `edit-toolbar.el'.  The second item is called
*Align*, which contains a submenu that says *Top*, *Bottom*, *Left*,
and *Right*, which will be just like setting the default toolbar
position.  The third one says *Text Labels*, which would just let you
select whether there are captions or not.  I think all three of these
are useful and are easy to implement in XEmacs.  These things also need
to be integrated with custom so that a user can control whether these
options apply to all sessions, and in such a case can save the settings
out to an options file.  `edit-toolbar.el' in particular needs to
integrate with custom.  Currently it has some sort of hokey stuff of
its own, which it saves out to a `.toolbar' file.  Another useful
option to have, once we draw the captions dynamically rather than using
pre-generated ones, would be the ability to change the font size of the
captions.  I'm sure that Kyle, for one, would appreciate this.

   (This is incomplete.....)


File: internals.info,  Node: Future Work -- Toolbar Interface Changes,  Prev: Future Work -- Easier Toolbar Customization,  Up: Future Work -- Toolbars

40.8.2 Future Work - Toolbar Interface Changes
----------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   I propose changing the way that toolbars are specified to make them
more flexible.

  1.  A new format for the vector that specifies a toolbar item is
     allowed.  In this format, the first three items of the vector are
     required and are, respectively, a caption, a glyph list, and a
     callback.  The glyph list and callback arguments are the same as
     in the current toolbar item specification, and the caption is a
     string specifying the caption text placed below the toolbar glyph.
     The caption text is required so that toolbar items can be
     identified for the purpose of retrieving and changing their
     property values.  Putting the caption first also makes it easy to
     distinguish between the new and the old toolbar item vector
     formats.  In the old format, the first item, the glyph list, is
     either a list or a symbol.  In the new format, the first item is a
     string.  In the new format, following the three required items,
     are optional keyword items specified using keywords in the same
     format as the menu item vector format.  The keywords that should
     be predefined are: `:help-echo', `:context-menu',
     `:drop-handlers', and `:enabled-p'.  The `:enabled-p' and
     `:help-echo' keyword arguments are the same as the third and
     fourth items in the old toolbar item vector format.  The
     `:context-menu' keyword is a list in standard menu format that
     specifies additional items that will appear when the context menu
     for the toolbar item is popped up.  (Typically, this happens when
     the right mouse button is clicked on the toolbar item).  The
     `:drop-handlers' keyword is for use by the new drag-n-drop
     interface (see Drag-n-Drop Interface Changes (drag-n-drop.html) ),
     and is not normally specified or modified directly.

  2.  Conceivably, there could also be keywords that are associated
     with a toolbar itself, rather than with a particular toolbar item.
     These keyword properties would be specified using keywords and
     arguments that occur before any toolbar item vectors, similarly to
     how things are done in menu specifications.  Possible properties
     could include `:captioned-p' (whether the captions are visible
     under the toolbar), `:glyphs-visible-p' (whether the toolbar
     glyphs are visible), and `:context-menu' (additional items that
     will appear on the context menus for all toolbar items and
     additionally will appear on the context menu that is popped up
     when the right mouse button is clicked over a portion of the
     toolbar that does not have any toolbar buttons in it).  The
     current standard practice with regards to such properties seems to
     be to have separate specifiers, such as `left-toolbar-width',
     `right-toolbar-width', `left-toolbar-visible-p',
     `right-toolbar-visible-p', etc.  It could easily be argued that
     there should be no such toolbar specifiers and that all such
     properties should be part of the toolbar instantiator itself.  In
     this scheme, the only separate specifiers that would exist for
     individual properties would be default values.  There are a lot of
     reasons why an interface change like this makes sense.  For
     example, currently when VM sets its toolbar, it also sets the
     toolbar width and similar properties.  If you change which edge of
     the frame the VM toolbar occurs in, VM will also have to go and
     modify all of the position-specific toolbar specifiers for all of
     the other properties associated with a toolbar.  It doesn't really
     seem to make sense to me for the user to be specifying the width
     and visibility and such of specific toolbars that are attached to
     specific edges because the user should be free to move the
     toolbars around and expect that all of the toolbar properties
     automatically move with the toolbar. (It is also easy to imagine,
     for example, that a toolbar might not be attached to the edge of
     the frame at all, but might be floating somewhere on the user's
     screen).  With an interface where these properties are separate
     specifiers, this has to be done manually.  Currently, having the
     various toolbar properties be inside of toolbar instantiators
     makes them difficult to modify, but this will be different with
     the API that I propose below.

  3.  I propose an API for modifying toolbar and toolbar item
     properties, as well as making other changes to toolbar
     instantiators, such as inserting or deleting toolbar items.  This
     API is based around the concept of a path.  There are two kinds of
     paths here - "toolbar paths" and "toolbar item paths".  Each kind
     of path is an object (of type `toolbar-path' and
     `toolbar-item-path', respectively) whose properties specify the
     location in a toolbar instantiator where changes to the
     instantiator can be made.  A toolbar path, for example, would be
     created using the `make-toolbar-path' function, which takes a
     toolbar specifier (or optionally, a symbol, such as `left',
     `right', `default', or `nil', which refers to a particular
     toolbar), and optionally, parameters such as the locale and the
     tag set, which specify which actual instantiator inside of the
     toolbar specifier is to be modified.  A toolbar item path is
     created similarly using a function called
     `make-toolbar-item-path', which takes a toolbar specifier and a
     string naming the caption of the toolbar item to be modified, as
     well as, of course, optionally the locale and tag set parameters
     and such.

     The usefulness of these path objects is as arguments to functions
     that will use them as pointers to the place in a toolbar
     instantiator where the modification should be made.  Recall, for
     example, the generalized property interface described above.  If a
     function such as `get' or `put' is called on a toolbar path or
     toolbar item path, it will use the information contained in the
     path object to retrieve or modify a property located at the end of
     the path.  The toolbar path objects can also be passed to new
     functions that I propose defining, such as `add-toolbar-item',
     `delete-toolbar-item', and `find-toolbar-item'.  These functions
     should be parallel to the functions for inserting, deleting,
     finding, etc. items in a menu.  The toolbar item path objects can
     also be passed to the drop-handler functions defined in
     Drag-n-Drop Interface Changes (drag-n-drop.html) to retrieve or
     modify the drop handlers that are associated with a toolbar item.
     (The idea here is that you can drag an object and drop it onto a
     toolbar item, just as you could onto a buffer, an extent, a menu
     item, or any other graphical element).

  4.  We should at least think about allowing for separate default and
     buffer-local toolbars.  The user should either be able to position
     these toolbars one above the other, or side by side, occupying a
     single toolbar line.  In the latter case, the boundary between the
     toolbars should be draggable, and if a toolbar takes up more room
     than is allocated for it, there should be arrows that appear on
     one or both sides of the toolbar so that the items in the toolbar
     can be scrolled left or right.  (For that matter, this sort of
     interface should exist even when there is only one toolbar that is
     on a particular toolbar line, because the toolbar may very well
     have more items than can be displayed at once, and it's silly in
     such a case if it's impossible to access the items that are not
     currently visible).

  5.  The default context menu for toolbars (which should be specified
     using a specifier called `default-toolbar-context-menu' according
     to the rules defined above) should contain entries allowing the
     user to modify the appearance of a toolbar.  Entries would
     include, for example, whether the toolbar is captioned, whether
     the glyphs for the toolbar are visible (if the toolbar is
     captioned but its glyphs are not visible, the toolbar appears as
     nothing but text; you can set things up this way, for example, in
     Netscape), an option that brings up a package for editing the
     contents of a toolbar, an option to allow the caption face to be
     dchanged (perhaps thorough jan `edit-faces' or `custom'
     interface), etc.



File: internals.info,  Node: Future Work -- Menu API Changes,  Next: Future Work -- Removal of Misc-User Event Type,  Prev: Future Work -- Toolbars,  Up: Future Work

40.9 Future Work - Menu API Changes
===================================

Author: Ben Wing (mailto:ben@xemacs.org)

  1.  I propose making a specifier for the menubar associated with the
     frame.  The specifier should be called `default-menubar' and
     should replace the existing `current-menubar' variable.  This
     would increase the power of the menubar interface and bring it in
     line with the toolbar interface.  (In order to provide proper
     backward compatibility, we might have to complete the symbol value
     handler mechanism (symbol-value-handlers.html))

  2.  I propose an API for modifying menu instantiators similar to the
     API composed above for toolbar instantiators.  A new object called
     a "menu path" (of type `menu-path') can be created using the
     `make-menu-path' function, and specifies a location in a particular
     menu instantiator where changes can be made.  The first argument to
     `make-menu-path' specifies which menu to modify and can be a
     specifier, a value such as `nil' (which means to modify the default
     menubar associated with the selected frame), or perhaps some other
     kind of specification referring to some other menu, such as the
     context menus invoked by the right mouse button.  The second
     argument to `make-menu-path', also required, is a list of zero or
     more strings that specifies the particular menu or menu item in
     the instantiator that is being referred to.  The remaining
     arguments are optional and would be a locale, a tag set, etc.  The
     menu path object can be passed to `get', `put' or other standard
     property functions to access or modify particular properties of a
     menu or a menu item.  It can also be passed to expanded versions
     of the existing functions such as `find-menu-item',
     `delete-menu-item', `add-menu-button', etc.  (It is really a shame
     that `add-menu-item' is an obsolete function because it is a much
     better name than `add-menu-button').  Finally, the menu path
     object can be passed to the drop-handler functions described in
     Drag-n-Drop Interface Changes (drag-n-drop.html) to access or
     modify the drop handlers that are associated with a particular
     menu item.

  3.  New keyword properties should be added to the menu item vector.
     These include `:help-echo', `:context-menu' and `:drop-handlers',
     with similar semantics to the corresponding keywords for toolbar
     items.  (It may seem a bit strange at first to have a context menu
     associated with a particular menu item, but it is a user interface
     concept that exists both in Open Look and in Windows, and really
     makes a lot of sense if you give it a bit of thought).  These
     properties may not actually be implemented at first, but at least
     the keywords for them should be defined.



File: internals.info,  Node: Future Work -- Removal of Misc-User Event Type,  Next: Future Work -- Mouse Pointer,  Prev: Future Work -- Menu API Changes,  Up: Future Work

40.10 Future Work - Removal of Misc-User Event Type
===================================================

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* This page describes why the misc-user event type should
be split up into a number of different event types, and how to do this.

   The misc-user event should not exist as a single event type.  It
should be split up into a number of different event types: one for
scrollbar events, one for menu events, and one or two for drag-n-drop
events.  Possibly there will be other event types created in the
future.  The reason for this is that the misc-user event was a bad
design choice when I made it, and it has only gotten worse with
Oliver's attempts to add features to it to make it be used for
drag-n-drop.  I know that there was originally a separate drag-n-drop
event type, and it was folded into the misc-user event type on my
recommendation, but I have now realized the error of my ways.  I had
originally created a single event type in an attempt to prevent some
Lisp programs from breaking because they might have a case statement
over various event types, and would not be able to handle new event
types appearing.  I think now that these programs simply need to be
written in a way to handle new event types appearing.  It's not very
hard to do this.  You just use predicates instead of doing a case
statement over the event type.  If we preserve the existing predicate
called `misc-user-event-p', and just make sure that it evaluates to
true when given any user event type other than the standard simple
ones, then most existing code will not break either when we split the
event types up like this, or if we add any new event types in the
future.

   More specifically, the only clean way to design the misc-user event
type would be to add a sub-type field to it, and then have the nature
of all the other fields in the event type be dependent on this
sub-type.  But then in essence, we'd just be reimplementing the whole
event-type scheme inside of misc-user events, which would be rather
pointless.


File: internals.info,  Node: Future Work -- Mouse Pointer,  Next: Future Work -- Extents,  Prev: Future Work -- Removal of Misc-User Event Type,  Up: Future Work

40.11 Future Work - Mouse Pointer
=================================

* Menu:

* Future Work -- Abstracted Mouse Pointer Interface::
* Future Work -- Busy Pointer::


File: internals.info,  Node: Future Work -- Abstracted Mouse Pointer Interface,  Next: Future Work -- Busy Pointer,  Prev: Future Work -- Mouse Pointer,  Up: Future Work -- Mouse Pointer

40.11.1 Future Work - Abstracted Mouse Pointer Interface
--------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   *Abstract:* We need to create a new image format that allows
standard pointer shapes to be specified in a way that works on all
Windows systems.  I suggest that this be called `pointer', which has
one tag associated with it, named `:data', and whose value is a string.
The possible strings that can be specified here are predefined by
XEmacs, and are guaranteed to work across all Windows systems.  This
means that we may need to provide our own definition for pointer shapes
that are not standard on some systems.  In particular, there are a lot
more standard pointer shapes under X than under Windows, and most of
these pointer shapes are fairly useful.  There are also a few pointer
shapes (I think the hand, for example) on Windows, but not on X.
Converting the X pointer shapes to Windows should be easy because the
definitions of the pointer shapes are simply XBM files, which we can
read under Windows.  Going the other way might be a little bit more
difficult, but it should still not be that hard.

   While we're at it, we should change the image format currently called
`cursor-font' to `x-cursor-font', because it only works under X
Windows.  We also need to change the format called `resource' to be
`mswindows-resource'.  At least in the case of `cursor-font', the old
value should be maintained for compatibility as an obsolete alias.  The
`resource' format was added so recently that it's possible that we can
just change it.


File: internals.info,  Node: Future Work -- Busy Pointer,  Prev: Future Work -- Abstracted Mouse Pointer Interface,  Up: Future Work -- Mouse Pointer

40.11.2 Future Work - Busy Pointer
----------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   Automatically make the mouse pointer switch to a busy shape (watch
signal) when XEmacs has been "busy" for more than, e.g. 2 seconds.
Define the "busy time" as the time since the last time that XEmacs was
ready to receive input from the user.  An implementation might be:

  1. Set up an asynchronous timeout, to signal after the busy time;
     these are triggered through a call to QUIT so they will be
     triggered even when the code is busy doing something.

  2. We already have an "emacs_is_blocking" flag when we are waiting for
     input.  In the same place, when we are about to block and wait for
     input (regardless of whether input is already present), maybe call
     a hook, which in this case would remove the timer and put back the
     normal mouse shape.  Then when we exit the blocking stage (we got
     some input), call another hook, which in this case will start the
     timer.  Note that we don't want these "blocking" hooks to be
     triggered just because of an accept-process-output or some similar
     thing that retrieves events, only to put them back onto a queue
     for later processing.  Maybe we want some sort of flag that's
     bound by those routines saying that we aren't really waiting for
     input.  Making that flag Lisp-accessible allows it to be set by
     similar sorts of Lisp routines (if there are any?) that loop
     retrieving events but defer them, or only drain the queue, or
     whatnot.  #### Think about whether it would make some sense to try
     and be more clever in our determinations of what counts as "real
     waiting for user input", e.g.  whether the event gets dispatched
     (unfortunately this occurs way too late, we want to know to remove
     the busy cursor *before* getting an event), maybe whether there
     are any events waiting to be processed or we'll truly block, etc.
     (e.g. one possibility if there is input on the queue already when
     we "block" for input, don't remove the busy- wait pointer, but
     trigger the removal of it when we dispatch a user event).


File: internals.info,  Node: Future Work -- Extents,  Next: Future Work -- Version Number and Development Tree Organization,  Prev: Future Work -- Mouse Pointer,  Up: Future Work

40.12 Future Work - Extents
===========================

* Menu:

* Future Work -- Everything should obey duplicable extents::


File: internals.info,  Node: Future Work -- Everything should obey duplicable extents,  Prev: Future Work -- Extents,  Up: Future Work -- Extents

40.12.1 Future Work - Everything should obey duplicable extents
---------------------------------------------------------------

Author: Ben Wing (mailto:ben@xemacs.org)

   A lot of functions don't properly track duplicable extents.  For
example, the `concat' function does, but the `format' function does
not, and extents in keymap prompts are not displayed either.  All of
the functions that generate strings or string-like entities should
track the extents that are associated with the strings.  Currently this
is difficult because there is no general mechanism implemented for doing
this.  I propose such a general mechanism, which would not be hard to
implement, and would be easy to use in other functions that build up
strings.

   The basic idea is that we create a C structure that is analogous to a
Lisp string in that it contains string data and lists of extents for
that data.  Unlike standard Lisp strings, however, this structure (let's
call it `lisp_string_struct') can be incrementally updated and its
allocation is handled explicitly so that no garbage is generated.  (This
is important for example, in the event-handling code which would want to
use this structure, but needs to not generate any garbage for efficiency
reasons).  Both the string data and the list of extents in this string
are handled using dynarrs so that it is easy to incrementally update
this structure.  Functions should exist to create and destroy instances
of `lisp_string_struct' to generate a Lisp string from a
`lisp_string_struct' and vice-versa to append a sub-string of a Lisp
string to a `lisp_string_struct', to just append characters to a
`lisp_string_struct', etc.  The only thing possibly tricky about
implementing these functions is implementing the copying of extents from
a Lisp string into a `lisp_string_struct'.  However, there is already a
function `copy_string_extents()' that does basically this exact thing,
and it should be easy to create a modified version of this function.

