
Title:	How to generate X11 and Motif libraries

Author:	David N. Gray

Date:	October 28, 1998

Status:	describes work previously done

Version: $HopeName: D-X11-Xlib!how-to-generate.text(trunk.1) $
         $Date: 2004/03/12 00:10:33 $

---------------------------

This document explains how the Dylan FFI declarations were generated for
the "xlib", "xt", and "motif" libraries.

The ".dylan" files in the libraries are of three kinds:

  * interface files

    Each of these was automatically generated from one of the
    X Windows ".h" files.  These files can be easily recognized because
    they begin with a comment of the form:

      // This file is automatically generated from "foo.h"; do not edit.

    This is usually followed by a comment showing the RCS version
    identification from the ".h" file.

  * library and module definition files

    Each "library.dylan" file is automatically generated from a
    corresponding "library.src" file by inserting the `export' lists from
    "*.exp" files that are generated as a side-effect of translating the
    header files.

  * other

    There are a few ".dylan" files that were hand-coded; these include
    "extra.dylan", and "aux-xlib.dylan".  Generally these implement
    equivalents for C macros that couldn't be mechanically translated,
    define some pointer types that need to be explicitly declared for the
    FFI, and define any types that need special handling.

Since these libraries are needed before Collage will be available, a
different approach has been taken to converting the ".h" files to Dylan FFI
declarations.  This is a more ad-hoc approach that is sufficient to handle
particular patterns of usage, but is not intended to be as general purpose
as Collage will need to be.  

The converter consists of the following components (in directory
"~dylan/sources/x11/xlib/" unless otherwise indicated):

  * A general-purpose public-domain text processing utility named "gema".
    (Version 1.2 or later is required.)  Lacking a better place, it is
    currently installed as "/u/dylan/local/gema" for SparcStations.
    There is also a Windows NT/Windows 95 image on host "fog" in
    file "\users\gray\NTutil\gema.exe".  For documentation of this program,
    see URL: http://webhost.functionalobjects.com/~gray/gema-intro.html 
    If you really want hard-copy, there are PostScript versions in file:
    zephyr.menlo:/u/gray/ftp/gema-1-3/postscript.tar.gz
    The source code has been placed under Hope in directory
    "~dylan/tools/gema/".  The program is available for public distribution
    by anonymous FTP from "ftp://ftp.functionalobjects.com/pub/gema/".
    There is also a WinHelp version of the documentation available there.

  * Pattern files "patterns.pat" and "motif.pat", which define the text
    transformation rules to be applied by "gema" in order to read a ".h"
    file and emit corresponding Dylan FFI declarations and a "*.exp" file
    of exports.  The rules in "patterns.pat" are applicable to C in
    general, while "motif.pat" contains additional rules to handle special
    cases in the X and Motif headers.
 
  * The file "obsolete.text" is a list of C identifiers whose definitions
    are not to be translated because they are known to be obsolete.
    The file is read by "gema" under the direction of rules in the
    "patterns.pat" file (using the `-exclude' option).  Names may be
    separated by either commas or whitespace, comments may be included
    (using either "//" or "/*...*/"), and "*" may be used for wildcarded
    names.  This list was generated based on comments in the header files.

  * File "include.pat" is used in the generation of the library definition
    files.  It is a pattern file that tells "gema" how to merge the "*.src"
    and "*.exp" files.

  * File "make-sources" in each of the library directories is the make file
    that controls generation of the ".dylan" files.  (It isn't called
    "Makefile" here because a file with that name is generated by the C
    back-end compiler.)  For example:

       make -f make-sources

    will re-generate the sources for the library in the current directory.
    You may need to override the "make" variables "XHDIR" and "MHDIR" to
    specify the directories containing the header files.  For example:
 
       make XHDIR=/usr/include/X11 MHDIR=/usr/include/Xm -f make-sources ...

  * File "sources.mak" in "xlib" is included by each of the "make-sources"
    files; it is simply the portion of the make file that is the same for
    all of the libraries.

  * File "x-to-dylan" is a small shell script that is used to simplify the
    "make" files by not having to repeat the command line options that are
    always used.

The header files used to create the initial version of these libraries were
copied from Solaris host "lordosis.ed.functionalobjects.com" on Oct. 2, 1998,
directories "/usr/dt/share/include/Xm" and "/usr/openwin/include/X11".

A typical rule in the Makefile looks like:

  foo.dylan: $(XHDIR)/foo.h  $(PATTERN-DEP) $(COMSLOT)
	$(XTOD) -f $(COMSLOT) -module Xt  $(XHDIR)/foo.h $@

The command line expands to:

  /u/dylan/local/gema -f ../xlib/patterns.pat -f ../xlib/motif.pat \
	-f ../xlib/modcopyr.pat   -exclude ../xlib/obsolete.text \
	-f ../xlib/common-slots.pat -module Xt  $(XHDIR)/foo.h foo.dylan

The command line options "-module", "-only", "-exports", and "-exclude" are
actually defined in the "patterns.pat" file (see the lines beginning
"ARGV:"), so don't look for them in the "gema" documentation.  The "-f"
options direct "gema" to load rules from the files "patterns.pat",
"motif.pat", and "common-slots.pat"; the latter file is automatically
generated by the "make" file and simply specifies structure slot names that
are imported instead of exported.  The "-module" option specifies what to
put in the "module:" line at the beginning of the generated file.  This
command will read file "foo.h" and write files "foo.dylan" and "foo.exp";
the latter is the `export' lists that need to be inserted into the module
definition.  The "-exports" option may be used to specify a different name
for that file.  There are also "-D" and "-U" options which have the same
meaning as for the C pre-processor, namely to designate pre-processor flags
as being either defined or undefined.  Flags used in multiple files are
specified in the pattern files by "def-flag:" and "undef-flag:" rules.  If
a flag has not been specified as being either defined or undefined, the
conditional processing directive will be ignored, and both the "then" and
"else" parts will get translated.

The libraries do not currently include all of the header files from those
directories, only those that appear to be needed to support the Motif
back-end for DUIM.  If you want to extend one of these libraries by adding
a translation of an additional header file, you would need to do the
following: 

  * Add the file to the "make-sources" file; note that the generated
    ".dylan" file name should appear in three places.

  * Add the new source file to the ".lid" file.

  * Edit the "library.src" file to add a new line: "%include foo.exp".

  * Run "make -f make-sources" to both generate the new file and to update
    the "library.dylan" file.

Structure slot accessor names that are defined in more than one library
need special handling.  In "xlib/aux-xlib.dylan", `define open-accessor' is
used to make the generic function open.  If the name is not defined in the
"xlib" library itself, the name needs to be added to the export list in
"library.src".  In the other library, the name needs to be added to the
"export:" list of the "use xlib" clause in the "library.src" file.  When
"make" is run, it will automatically update the "common-slots.pat" file
which is used to suppress re-exporting the imported names.

If any changes are made to the pattern files, it is important to
re-generate all of the source files and compare against the previous
version to make sure that there are no surprise side-effects of the change.
Note that "gema" renames the original output file with a ".bak" suffix, so
you can compare to that for a quick check of what the last "make"
did; for example:
  diff foo.dylan.bak foo.dylan


Some quick hints for reading the pattern files:
 
  "!" introduces a comment.
 
  Lines beginning with "@" are used to set option switches and initialize
  variables. 

  The rest of the lines are transformation rules of the form
	pattern=replacement 
  or
	domain:pattern=replacement 
  Within the designated domain (the default is the top-level of the input
  file), input file text that matches the pattern is replaced in the output
  file by the result of evaluating what follows the "=".

  Angle brackets ("<...>") denote variable syntactic elements, like in BNF.

  @foo{...} is a function call.

  For example, the following rule handles integer constants:

    \L\#define <relevant-constant> <number>\W<endline>=\
	\Ndefine constant @export{\$$1} @tab{53}\=@wrap{\s@right{4;$2}\;}\n

  This says (glossing over minor details) that if the input contains
  `#define' followed by a suitable identifier (according to the "-only" or
  "-exclude" option) and a numeric constant, then a corresponding `define
  constant' is emitted.  `$1' is the value of the `<relevant-constant>' and
  `$2' is the converted value of the `<number>'.  Rules prefixed by
  `number:' define the recognition and conversion of a `<number>'.  The
  `@export' function converts the name to Dylan syntax (replacing "_" with
   "-") and has the side-effect of adding the name to the module's
  export list, and the other functions are just for aligning the spacing in
  the output.

  See the "gema" documentation for a complete explanation of the notation
  used.


