#
Chapter 4
Extensions and `csi -setup`

To simplify the management of extension libraries, a simple mechanism based on shared object files and dynamic loading is used to allow the loading of extension libraries at run-time. Currently this mechanism is only available on platforms that support dynamic loading of shared object files, at the time of writing this is Linux, FreeBSD, HP/UX, Windows (Cygwin) and Solaris. The macro `chicken-setup` is used to make packaging, building and installation as simple as possible for the user.

An extension is a compressed archive containing the source-code, compilation parameters and additional information. Technically an extension is a gzipped tar-archive, containing the source file (or a directory hierarchy with source-files, in the case extensions containing multiple modules) and a setup script with additional information.

The CHICKEN interpreter provides an option for performing basic operations on extensions. To build and install an extension, enter

    % csi -setup EXTENSION

(Note that you have to have write-access for the directory where system-wide extensions are installed. Normally this directory is `/usr/local/lib/chicken`. You can create this directory by entering `csi -setup` without additional parameters.)

The registry will be created automatically on its first use. Do create the registry explicitly, enter

    csi -setup -init

This will overwrite any existing registry (after asking the user for confirmation). This may also be needed should the extension-registry be damaged.

If you want to have your extensions installed at a different location than the default, you have two choices: either set the environment-variable `CHICKEN_REGISTRY` to the name of your desired registry-directory, or create a directory named `$HOME/.chicken-registry`.

To ``wrap'' up an extension, that is, to create a packaged archive ready for distribution, enter

    % csi -setup EXTENSION -wrap

The `-setup` option loads the setup-script `EXTENSION.setup` and passes any arguments that follow the extension-name as command-line arguments.

A setup-script is a normal Scheme-program, which should contain an invocation of the `chicken-setup` macro, like this:

    (chicken-setup (command-line-arguments) SPEC ...)

[syntax] (chicken-setup ARGUMENTS SPEC ...)
  Installs or wraps extension libraries with the specifications `SPEC ...`. `ARGUMENTS` should be a list of strings, where the string may be one of the following:

  - -b -build -build Builds the extension by compiling any provided source files.

  - -i -install -install Installs the extension. You might have to have root-privileges, if the registry of installed extensions is located in a system-directory. The extension will be built, if it isn't already.

  - -w -wrap -wrap Creates an archive from all source files and the setup-script. If the extension contains more than a single source file, then there must exist a directory with the same name as the extension, which all source files.

  If no argument is given, then the extension will be extracted, built and installed in one step.

  `SPEC` is a setup-specification that describes the structure and attributes of the extension:

      (EXTENSION-NAME
        (ATTRIBUTE ...)
        SUBMODULE ...) )

  - `EXTENSION-NAME` is a symbol naming the extension.

  - `ATTRIBUTE` is a list with one or more elements, where the first element must be a symbol naming an extension attribute.

  - `SUBMODULE` is again an setup-specification of a nested extension-module.

  The following attributes are allowed:

  (author STRING)
    The name of the author of the module.

  (license STRING)
    The type of license this module is distributed under.

  (comment STRING ...)
    Additional comments.

  (syntax)
  (source)
    Marks this module as source, that is, it is not compiled. Use this attribute for syntax/macro modules.

  (file STRING)
    If the module (or sub-module) has a different filename than the module-name, then the actual filename is declared using this property.

  (make STRING-OR-EXP)
    Defines the command used to build the module. This defaults to an invocation of the compiler-script `csc` with the appropriate arguments and options to build the shared library object. The argument may also be a Scheme expression that is evaluated to build the current module.

  (when EXP)
    If the evaulation of `EXP` returns `#f`, then this module will not be built.

  (options STRING ...)
    Sets the compiler options for Scheme files that are to be compiled using `csc`.

  (c-options STRING ...)
    Sets the C-compiler options for Scheme files that are to be compiled using `csc`.

  (link-options STRING ...)
    Sets the linker options for Scheme files that are to be compiled using `csc`.

  (require EXTENSION ...)
    Lists extensions that are required by the current module.

  (require-for-syntax EXTENSION ...)
    Lists extensions that are required at compile-time by the current module.

  (require-at-runtime EXTENSION ...)
    Lists extensions that are required at runtime to support a syntax-extension. This means that an expansion of

        (require-for-syntax EXT)

    where `EXT` is defined with a specification that contains an attribute like

        (require-at-runtime EXTR1 EXTR2)

    will result in this code

        (require 'EXTR1 'EXTR2)

  (provide EXTENSION)
    Declares the given extensions as aliases for the current extension.

As an example, we take two simple extensions: an implementation of red-black trees and a simple ``functor''-like syntax that encapsulates the operations on the rb-tree data-type. The files for an extension should reside in a separate directory, so in the case of our simple functor-implementations we have the following directory-structure:

    functor.scm
    functor/
      utils.scm

Here we put the macro-definition for `define-functor` in a separate file, this simplifies the use in compiled code. The code follows:

    ;;; functor.scm

    (require 'srfi-1)
    (require-for-syntax 'match)

    (define-macro (define-functor name imports . body)
      (let ([exports
             (filter-map
              (match-lambda
                [('define (name . llist) . body) name]
                [('define name val) name]
                [_ #f] )
              body) ] )
        `(define (,name ,@imports)
           ,@body
           (values ,@exports) ) ) )

and

    ;;; functor/utils.scm

    (define (instantiate-functor f . imports)
      (apply f imports) )

One more thing is needed: a setup specification that contains information about the files and settings of the extension. In this case we have:

    ;;; functor.setup

    (chicken-setup (command-line-arguments)
      (functor ((syntax)
                (require-at-runtime (functor utils)) )
        (utils ()) ) )

Now we create an extension package:

    % csi -setup functor -wrap
    wrapping extension `functor' ...
    functor.setup
    functor.scm
    functor/
    functor/utils.scm

This will create the compressed archive `functor.egg`.

A user of this package can now use `csi` in combination with the `-setup` option to extract the contents from the archive, to build its components and install it in his Chicken system for easy use.

    % ls
    functor.egg
    % csi -setup functor -build
    extracting files from extension `functor' ...
    functor.setup
    functor.scm
    functor/
    functor/utils.scm
    building extension `functor' ...
    backing up registry...
    adding entry for `/home/felix/chicken/lib/functor.setup' ...
    compiling: csc -s ./functor/utils.scm -o ./functor/utils.so -O1
    removing temporary extension `/home/felix/chicken/lib/functor.setup' ...
    restoring registry

Note that `functor.scm` is not compiled, because it is explicitly marked as `syntax`, which means it is only meant to be used at compile-time, in source-form.

The extension is now installed and ready for use.

    % csi -quiet
    >>> (require-for-syntax 'functor)
    ; loading /home/felix/chicken/lib/functor.scm ...
    ; loading /home/felix/chicken/lib/functor/utils.so ...
    >>> (define-functor adder (+)
          (define (make-adder init)
            (lambda (x)
              (set! init (+ init x))
              init) ) )
    >>> (define ma (instantiate-functor adder +))
    >>> (define a1 (ma 99))
    >>> (a1 1)
    100
    >>> (a1 22)
    122
    >>>

The extension for red/black trees is created in much the same way:

    ;;; rbtree.scm
    ;
    ; taken directly from Chris Okasaki's ``Purely functional Data Structures''

    (require-for-syntax 'match)
    (require-for-syntax 'functor)

    (define-functor rb-tree (rb<?)

      (define (member? x t)
        (match t
          [() #f]
          [(_ a y b)
           (cond [(rb<? x y) (member? x a)]
                 [(rb<? y x) (member? x b)]
                 [else #t] ) ] ))

      (define balance
        (match-lambda
          [(or ('black ('red ('red a x b) y c) z d)
               ('black ('red a x ('red b y c)) z d)
               ('black a x ('red ('red b y c) z d))
               ('black a x ('red b y ('red c z d))) )
           `(red (black ,a ,x ,b) ,y (black ,c ,z ,d)) ]
          [body body] ) )

      (define (insert x s)
        (define ins
          (match-lambda
            [() `(red () ,x ())]
            [(and s (color a y b))
             (cond [(rb<? x y) (balance (list color (ins a) y b))]
                   [(rb<? y x) (balance (list color a y (ins b)))]
                   [else s] ) ] ) )
        (match (ins s)
          [(_ a y b) `(black ,a ,y ,b)] ) ) )

The setup-specification looks like this. Since we only have a single file, we don't need to put it in a separate directory:

    (chicken-setup (command-line-arguments)
      (rb-tree ((options "-O2")
                (file "rbtree.scm")
                (require-for-syntax functor match) ) ) )

Note the properties used: `(options "-O2")` overrides the default compiler-options (`"-O1"`) and we use a different name for the source-file2. This extension requires the `define-functor` extension, we have to use `require-for-syntax` to make sure that the macro is included at compile-time.

The wrapping, unwrapping, building and installation works identical to the previously given example. Here is some code that uses both extensions:

    % cat testrb.scm
    (require 'rb-tree 'srfi-1 'extras)
    (require '(functor utils))

    (define-values (num-rb-member? num-rb-balance num-rb-insert)
      (instantiate-functor rb-tree <) )

    (randomize)
    (print "building list...")
    (define vals (list-tabulate 10000 (lambda (_) (random 100000))))

    (print "building rb-tree...")
    (define t1 '())
    (for-each (lambda (n) (set! t1 (num-rb-insert n t1))) vals)

    (print "building hash-table...")
    (define ht (make-hash-table))
    (for-each (lambda (n) (hash-table-set! ht n #t)) vals)

    (print "linear list:")
    (time (every (lambda (n) (memq n vals)) vals))

    (print "hash-table:")
    (time (every (lambda (n) (hash-table-ref ht n)) vals))

    (print "rb:")
    (time (every (lambda (n) (num-rb-member? n t1)) vals))
    % csc testrb.scm -O2
    % testrb
    building list...
    building rb-tree...
    building hash-table...
    linear list:
        1.74 seconds elapsed
           0 seconds in GC
           0 mutations
         161 minor GCs
           0 major GCs
    hash-table:
        0.03 seconds elapsed
           0 seconds in GC
           0 mutations
         779 minor GCs
           0 major GCs
    rb:
        0.27 seconds elapsed
        0.02 seconds in GC
           0 mutations
        4684 minor GCs
           0 major GCs

A special case of extensions is one that consists only of a single source file and which doesn't provide any additional syntax definitions. In this case, we can build and install the extensions simply by entering:

    % csi -setup foo

where `foo.scm` is the name of the source file containing the code.

It should be noted that the extension-mechanism does not address any namespace-issues. It is recommended to use the `export` declaration- specifier to export toplevel names that are to be used by client code. This will hide module-local definitions. Prefixing exported names with something like `<extension-name>:<definition>` avoids namespace-collisions with other libraries and names of client code.

2 just for the heck of it