This is ../../doc/qdoc.info, produced by makeinfo version 4.8 from
../../doc/qdoc.texi.

INFO-DIR-SECTION Programming Languages
START-INFO-DIR-ENTRY
* Q: (qdoc).                    The Q programming language and system.
END-INFO-DIR-ENTRY

   The Q Programming Language, Version 7.11, 23 February 2008.

   Copyright (c) 1992-2008 by Albert Graef.

   This manual has been published in the series `Musikinformatik und
Medientechnik', Bereich Musikinformatik, Musikwissenschaftliches
Institut, Johannes Gutenberg-Universitaet Mainz

55099 Mainz
Germany
ISSN 0941-0309

   The Q programming system is distributed under the terms of the GNU
General Public License. See the software license accompanying the
distribution for details.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that
the entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the author.


File: qdoc.info,  Node: Exception Handling,  Next: Miscellaneous Functions,  Prev: Lambda Abstractions,  Up: Built-In Functions

10.7 Exception Handling
=======================

As already mentioned, the Q language only knows a few "hard" runtime
error conditions a.k.a. "exceptions" such as stack or memory overflow.
However, "soft" exceptions can also be generated and handled in any
desired manner. The following functions are used to deal with all kinds
of exceptions during expression evaluation:

`halt'
     halt evaluation

`quit'
     exit the Q interpreter

`break'
     invoke the debugger

`fail'
     abort the current rule

`_FAIL_'
     abort the current reduction

`catch F X'
     handle an exception

`throw X'
     raise an exception

`trap ACT SIG'
     trap signals

   The `halt' function never returns a result, but raises an exception
which normally causes the evaluation process to be aborted immediately.
This last resort is commonly used in case of a fatal error condition.
The `quit' function is like `halt', but also causes exit from the
interpreter and returns you to the operating system shell; this
operation is often used interactively to terminate a session with the
interpreter. (Note that in a multithreaded script, see *Note POSIX
Threads::, `quit' only terminates the program when it is invoked from
the main thread. In other threads it just acts like `halt'.)

   If the `break' flag is `on' (see *Note Debugging::), the `break'
function interrupts an evaluation and invokes the symbolic debugger
built into the Q interpreter, just as if the user typed `Ctl-C'. This
operation allows you to set breakpoints in a script. For instance,

     foo X           = break || bar X;

causes the debugger to be invoked as soon as the rule is executed. If
the `break' flag is `off' then this operation has no effect. In any
case, the `break' function returns `()'.

   The `fail' function is used to abort the current rule, just like a
failing qualifier. The difference is that `fail' can be used anywhere
in the right-hand side or qualifier of a rule, and causes the rule to
be exited _immediately_. This allows you to handle complicated error
conditions which occur while a rule is already being executed, and also
provides an efficient method for implementing backtracking algorithms.
For instance, here is a quick solution for the famous "N queens"
problem:

     queens N        = search N 1 1 [];

     search N I J P  = write P || writes "\n" if I>N;
                     = search N (I+1) 1 (P++[(I,J)]) || fail if safe (I,J) P;
                     = search N I (J+1) P if J<N;
                     = () otherwise;

     safe (I1,J1) P  = not any (check (I1,J1)) P;

     check (I1,J1) (I2,J2)
                     = (I1=I2) or else (J1=J2) or else
                       (I1+J1=I2+J2) or else (I1-J1=I2-J2);

   This algorithm prints out all valid placements (i.e., lists of
(row,column) pairs) of N queens on an N times N board.  Note the use of
`fail' in the second equation for the `search' function, which causes
the rule to fail _after_ the current placement has been tried, after
which evaluation proceeds with the third rule which places the queen
into the next column. This turns the definition into a backtracking
algorithm. The `safe' function verifies that a placement `(I1,J1)' does
not put the queen in check with any of the other queens which have
already been placed.

   The `_FAIL_' function works in a fashion analogous to `fail', but
makes an entire _reduction_ fail (not just a single equation). This is
similar to throwing an exception (see below) in that the evaluation of
the current redex is aborted immediately, but does not actually raise
any exception; instead, it effectively forces the redex to be a normal
form, regardless of how many subsequent equations might still be
applicable. This operation is useful if you want to check for certain
error conditions _before_ trying any subsequent equations. For instance:

     foo X:Num       = _FAIL_ if X<=0;
                     = bar X if X>1;
                     = bar (1/X) otherwise;

   The `catch' function allows you to handle both "hard exceptions"
which are raised when the interpreter encounters one of the runtime
error conditions discussed in *Note Error Handling::, and "soft
exceptions" which are raised by the functions `halt' and `quit' already
explained above, and the `throw' function which is discussed below.
Moreover, the interpreter also handles certain signals sent to it,
e.g., via the `kill'(2) system call, by raising appropriate exceptions.
In the current implementation, by default only the signals `SIGINT'
("break"), `SIGTERM' ("terminate") and `SIGHUP' ("hangup") are handled;
the latter two are both treated as termination requests, like a call to
the `quit' function. Signal handlers can also be installed by the
running script using the `trap' function discussed below.

   Both arguments of `catch' are special. The `catch' function first
evaluates its second argument, the "target expression" and, if all is
well, returns the value of that expression. Otherwise, if any exception
was raised during the evaluation of the target expression, it evaluates
the first argument (the "exception handler"), applies it to the value
of the exception, and returns the result computed by the exception
handler.

   In the case of a runtime error condition and the exceptions raised
with the `halt' and `quit' functions, the "exception value" is of the
form `syserr N' where `N' is one of the integer error codes listed
below:

   * 1: "Break": break signal, user typed `Ctl-C'.

   * 2: "Halt": invocation of the `halt' function, or request to halt
     evaluation from the debugger.

   * 3: "Quit": termination signal, invocation of the `quit' function,
     or request to exit the interpreter from the debugger.

   * 4: "Memory overflow": the attempt to allocate memory for the stack
     or a new expression on the heap fails.

   * 5, 6: "Stack overflow": the attempt to push an expression on the
     stack, or the activation of a rule fails because the stack size
     limit has been reached (see the comments on memory management in
     *Note Running Compiler and Interpreter::). 5 signals an expression
     stack overflow, 6 an overflow of the rule stack.

   * 7: "Symbol table overflow": the attempt to create a new variable
     symbol fails.

   * 8: "Conditional error": the qualifying condition of a rule does not
     evaluate to a truth value.

   * 9: "External function error": an external function (see *Note C
     Language Interface::) signals an error condition.

   Let's try this by forcing a stack overflow condition:

     ==> iter 100000000 (+) 1
     ! Stack overflow
     >>> iter 100000000 (+) 1
                             ^

     ==> catch exception (iter 100000000 (+) 1)
     exception (syserr 5)

   The `syserr' constructor belongs to the built-in `SysException'
type, which in turn is a subtype of `Exception'. These types may be
thought of as being predefined as follows:

     public type Exception;
     public type SysException : Exception = const syserr N;

   User-defined exceptions can be generated with `throw'. The `throw'
function raises an exception whose value is given by its single
(non-special) argument. For instance:

     hd []           = throw '(hd []);
     exception X     = writes "Exception: " || write X || writes "\n";

   With these definitions we have:

     ==> catch exception (hd [1..3])
     1

     ==> catch exception (hd [1..0])
     Exception: '(hd [])
     ()

     ==> hd [1..0]
     ! Exception
     '(hd [])
     >>> hd [1..0]
                  ^

   Note that if an exception raised with `throw' is not handled with a
corresponding `catch', then the interpreter aborts the evaluation and
prints an error message (and it will also invoke the debugger to report
the rule which generated the exception, if `break' is `on').

   The `catch' and `throw' functions can also be used to implement
non-local value returns. For instance, here is a variation of the N
queens function which returns the first solution as soon as it has been
found during backtracking:

     queens1 N       = catch id (search1 N 1 1 []);

     search1 N I J P = throw P if I>N;
                     = search1 N (I+1) 1 (P++[(I,J)]) || fail if safe (I,J) P;
                     = search1 N I (J+1) P if J<N;
                     = () otherwise;

   Here, the "exception handler" is just the identity function `id', as
defined by the standard library, see *Note Standard Functions::.

   Another neat programming trick is the following which makes a rule
fail if an exception occurs during evaluation of the right-hand side:

     foo X           = catch fail (bar X);

   This works because the handler argument, which is special, is
evaluated only if an exception is actually encountered during
evaluation of the target expression. Otherwise, `foo X' will just
return the value of `bar X'. The `_FAIL_' function can be used in a
similar manner.

   Lisp and C++ programmers will probably miss an additional parameter
which restricts the kind of exceptions handled with `catch'. You can
implement this easily yourself by checking the exception value in your
handler function, and "throw on" an unknown exception to the next
enclosing `catch':

     type MyException : Exception = const empty_list;

     hd []           = throw empty_list;

     myexception X:MyException
                     = writes "myexception: " || write X ||
                       writes "\n" || halt;
     myexception X   = throw X otherwise;

   Note how we implemented the `empty_list' exception using our own
`Exception' subtype `MyException'. This method of structuring error
exceptions is recommended because it makes it easier to spot the source
of an exception. It also enables us to use a type guard (cf. *Note
Types::) in order to check for specific exception types, rather than
having to discriminate over different exception patterns.

   Finally, the `trap' function allows you to have exceptions be
generated when a signal arrives. The `trap' function takes two
arguments, `ACT' and `SIG', denoting the action to be performed and the
number of the signal to be trapped. It returns the previous action
associated with the signal. `ACT' must be an integer value. If it is
positive then the signal raises an exception of the form `syserr
(-SIG)'; if it is negative then the signal is completely ignored; and
if it is zero then the interpreter reverts to the default handling of
the signal. You can also use `trap' to redefine the default handling of
the break and termination signals. To allow signals to be caught
reliably, further signals are "blocked" (i.e., they are queued for
later delivery) while `catch' evaluates an exception handler. For
instance, here is a little script which sets up some signal handlers
and keeps on printing the trapped signals until it either gets
terminated with `kill -9' or the timed wait with the `sleep' function
times out. (The `sleep' function is described below. The symbolic
signal constants and the `getpid' and `printf' function are provided by
the `clib' module, see *Note Clib::.)

     test            = do (trap 1) [SIGINT,SIGTSTP,SIGTERM,SIGHUP,SIGQUIT] ||
                       printf "I'm running as pid %d, try to kill me!\n" getpid ||
                       loop;

     loop            = flush || catch sig (sleep 100);

     sig (syserr K)  = printf "Hey, I got signal %d.\n" (-K) || loop;


File: qdoc.info,  Node: Miscellaneous Functions,  Prev: Exception Handling,  Up: Built-In Functions

10.8 Miscellaneous Functions
============================

This group consists of some special operations which do not fit into any
of the other groups treated above.

`version, sysinfo'
     determine version/system information

`which S'
     determine absolute pathname of the executing script or of another
     file on Q library path

`time'
     get the system time

`sleep X'
     pause for some time

`isspecial X'
     predicate checking whether argument is a special form

`isconst X'
     predicate checking whether argument is a constant

`isfun X, isvar X'
     predicates checking whether argument is a function or a variable
     symbol, respectively

`isdef X'
     predicate checking whether the variable `X' has been assigned a
     value

`flip F'
     flip arguments of a binary function

   The `version' and `sysinfo' functions are useful when writing code
which depends on a particular version of the Q interpreter or a
machine/operating system type. They return a string describing the Q
interpreter version and the host system on which it is installed,
respectively.

   The `which' function performs a search for the given filename on the
Q library path. If the file is found then the absolute pathname is
returned; otherwise the function fails. The function can also be invoked
with the argument `()', in which case the full pathname of the
executing main script is returned. This function is most useful if a
script has to locate additional data files which are stored along with
it, see *Note Running Scripts from the Shell::.

   Two functions are provided for timing purposes. The `time' function
returns, as a floating point value, the time in seconds since the
"epoch" (00:00:00 UTC, January 1, 1970). The `sleep' function suspends
evaluation for the given time (integer or floating point value, again
in seconds). Be warned that the actual resolution of these functions
depends on the timing routines available on your system, and even if a
decent resolution is provided, the accuracy of results will depend on
various factors such as system load.

   The `isspecial', `isconst', `isfun', `isvar' and `isdef' predicates
are all special forms checking whether their single argument has a
certain property. The `isspecial' function allows you to determine
whether its argument is a special form which will receive its next
argument unevaluated. The `isconst' operation checks whether its
argument is a constant, i.e., a constant belonging to any of the
built-in types, a variable or function symbol declared with `const', or
an application whose head element is such a constant. The `isfun' and
`isvar' predicates are used to check whether an expression is a
function symbol (built-in or defined by the loaded script, can also be
a constant symbol), or a free variable symbol, respectively. The
`isdef' function can be used to check whether its argument (a variable)
has been assigned a value using `def'.

   The `flip' function exchanges arguments of a (binary) function, as
if it was defined by the following equation:

     flip F X Y              = F Y X;

This function is used internally to implement operator sections with
missing left argument. E.g., `(+1)' is nothing but syntactic sugar for
`flip (+) 1'. You can apply `flip' to other, user-defined functions as
well; thus, `flip foo X' denotes the function which maps `Y' to `foo Y
X'.

   In order to handle special forms like, e.g., `(or else)' correctly,
`flip' automagically adjusts to the argument pattern of its first
argument. That is, if the function given as the first argument (which is
always evaluated) has a special first (or second) argument, then
`flip''s second (or first) argument will be special as well.


File: qdoc.info,  Node: The Standard Library,  Next: Clib,  Prev: Built-In Functions,  Up: Top

11 The Standard Library
***********************

This chapter gives an overview of the data types and operations provided
by the standard library modules supplied with the Q programming system.
Most of these scripts are already "preloaded" when you run the
interpreter, because they are included in the "prelude" script
`prelude.q'. The exact collection of library scripts may depend on your
local setup, but a basic installation of the Q core package should at
least provide the modules listed below. In this list, (*) indicates
"external" modules which are (mostly) implemented in C, whereas (+)
denotes modules which have to be imported explicitly, as they are not
included in the prelude.

`assert.q'
     print diagnostics

`clib.q'
     basic C interface (*)

`complex.q'
     complex numbers

`cond.q'
     conditional expressions and tuple/list/stream comprehensions

`error.q'
     print error messages

`getopt.q'
     GNU getopt compatible option parser (+)

`graphics.q'
     PostScript graphics interface (+)

`math.q'
     mathematical functions

`prelude.q'
     standard prelude

`rational.q'
     rational numbers

`reftypes.q'
     tuples, lists and streams of references (+)

`sort.q'
     algorithms to sort a list

`stddecl.q'
     shared declarations of the container data structures, cf.
     `stdtypes.q' (+)

`stdlib.q'
     a collection of common standard functions

`stdtypes.q'
     a collection of efficient container data structures (+)

`stream.q'
     a lazy list data structure

`string.q'
     additional string functions

`system.q'
     POSIX system interface (*) (+)

`tuple.q'
     additional tuple functions

`typec.q'
     type-checking predicates

   Here is a quick rundown of the most important modules: The
`prelude.q' script implements the standard prelude, which loads the
entire collection (except the modules marked with (+) in the list above,
which must be imported explicitly). It adds some convenient operations
for comparing lists, streams and tuples, the syntactic inequality
operator `!=', as well as successor and predecessor functions on
integers (`succ' and `pred') and default implementations of various
functions for subtypes derived from the `Real' type. It also contains
definitions for some cases of list, tuple and stream enumerations which
are not provided as builtins.

   Specifically, the standard prelude overloads the `=', `<>', `<',
`>', `<=' and `>=' operators with rules for deciding equality and
inequality of lists, streams and tuples, and rules for ordering lists
and streams lexicographically. That is, tuples, lists and streams can
be compared with `=' and `<>', which is done by checking that the
operands are of the same size and recursively comparing all members of
the operands. Lists and streams can also be compared
"lexicographically" by recursively comparing the list or stream
members; the first unequal members decide the comparison. This works
just like the lexicographic comparison of strings. Thus, e.g., `[1,2]'
is considered to be less than both `[1,3]' and `[1,2,1]', but more than
`[1,1,3]' or `[0,5]'.

   Moreover, the prelude defines the syntactic inequality operator `!='
as the logical negation of the built-in `==' operator, cf. *Note
Non-Linear Equations::. Like `==', `!=' is implemented as a special
form.

   Last but not least, the standard prelude also provides default
definitions for arithmetic, relational, numeric and conversion functions
on `Real', so that a minimal implementation of a user-defined type
derived from `Real' (or any of its subtypes) just needs to define the
`float' function (which is needed to coerce values of the type to
`Float') and then add definitions for those operations which need
special treatment. The default definitions in `prelude.q' are at the
lowest priority level so that they can always be overridden if needed.

   The `stdlib.q' script has those operations which will probably be
used most frequently by the average programmer; it contains a lot of
additional list processing functions and other useful stuff mostly
adopted from [Bird/Wadler 1988]. This module is also included by many
other library scripts. Historically, it was the first script written
for the standard library.

   The `stream.q' script extends most list operations defined in
`stdlib.q' to streams, i.e., lazy lists. It also provides a number of
other useful stream operations.

   Similarly, the `string.q' and `tuple.q' scripts provide additional
operations on strings and tuples, and also extend some list operations
to these data types.

   The `stdtypes.q' script simply includes all modules which implement
common container data structures; currently these are `array.q',
`bag.q', `dict.q', `hdict.q', `heap.q' and `set.q', see *Note Standard
Types::. Some declarations shared by these modules are in the
`stddecls.q' script.

   The `complex.q' and `rational.q' scripts implement complex and
rational number types along with the usual operations on these types.
These have been designed to integrate seamlessly into Q's system of
built-in number types so that expressions involving arithmetic
operations with any combination of integer, floating point, complex and
rational arguments work as expected.

   The `clib.q' and `system.q' modules give access to various useful
system functions from the C library, including C-style formatted I/O,
binary file I/O, process and thread management, internationalization
support and regular expression routines. The `clib.q' module also
provides mutable expression cells ("references"), as well as a "byte
string" data type which lets you represent arbitrary C data, and it
overrides various important standard library functions with much faster
C versions. The operations of these modules are mostly written in C,
using the C language interface discussed in *Note C Language
Interface::. These modules are described in their own chapter, see
*Note Clib::.

   As of Q 7.8, the `reftypes.q' module provides additional convenience
operations to work with mutable expression sequences implemented as
tuples, lists or streams of references. These are also described in the
"Clib" chapter, see *Note Expression References::.

   The Q programming system comes bundled with some additional modules
for interfacing to various third-party libraries and software packages.
At the time of this writing, these are:

   * Curl, a library for transferring files with URL syntax

   * GNU dbm, a simple database library;

   * ODBC, the industry standard database interface;

   * Octave, a popular MATLAB-like open source software for doing
     numeric computations;

   * GGI, a portable raster graphics interface;

   * ImageMagick, a comprehensive image manipulation software;

   * Tcl/Tk, a portable user interface toolkit and scripting language;

   * IBM's Data Explorer, a scientific visualization software.

   Preliminary documentation for these modules can be found in the
`etc' subdirectory of your Q installation directory (usually in
`/usr/share/q' or `/usr/local/share/q'). Other useful add-on modules
are available as separate source packages. At the time of this writing
this comprises additional GUI libraries (including a complete interface
to Trolltech's very nice "Qt" toolkit), a bunch of graphics and
multimedia modules (including interfaces to OpenGL, OpenAL, the Xine
video library, and Grame's MidiShare and Faust) and more. Ready-made
plugins for using the Q interpreter from the "Chicken" Scheme compiler,
the Apache webserver and Miller Puckette's graphical computer music and
multimedia environment "PureData" are also available. Please check out
the Q website at `http://q-lang.sourceforge.net' for more information
about these. (The Q module for Chicken, which was written by John
Cowan, is available separately from the Chicken website, see
`http://www.call-with-current-continuation.org'.)

* Menu:

* Standard Functions::
* Tuple Functions::
* String Functions::
* Option Parsing::
* Type-Checking Predicates::
* Sorting Algorithms::
* Standard Types::
* Streams::
* Conditionals and Comprehensions::
* Mathematical Functions::
* Complex Numbers::
* Rational Numbers::
* Graphics::
* Diagnostics and Error Messages::


File: qdoc.info,  Node: Standard Functions,  Next: Tuple Functions,  Prev: The Standard Library,  Up: The Standard Library

11.1 Standard Functions
=======================

The `stdlib.q' script provides frequently used list operations and
other stuff mostly adopted from [Bird/Wadler 1988].

`abs X'
     absolute value of X

`all P Xs'
     verify that each element of the list `Xs' satisfies the predicate
     `P'

`any P Xs'
     verify that the list `Xs' contains an element satisfying predicate
     `P'

`append Xs X'
     append a value `X' to the list `Xs'

`cat Xs'
     concatenate a list of lists

`cons X Xs'
     prepend an element to a list

`cst X'
     constant-valued function: `cst X Y => X'

`curry F'
     curry a function: turn a function operating on pairs into a
     function with two arguments

`curry3 F'
     `curry' with three arguments

`do F Xs'
     apply a function `F' to each member of a list `Xs', return `()'

`dowith F Xs Ys'
     take two lists and apply a binary function to corresponding
     elements, return `()'

`dowith3 F Xs Ys Zs'
     `dowith' with three lists

`drop N Xs'
     remove the first `N' elements from the list `Xs'

`dropwhile P Xs'
     remove elements from the beginning of `Xs' while the predicate `P'
     is satisfied

`eq X Y'
     syntactic equality (cf. *Note Non-Linear Equations::)

`filter P Xs'
     filter a list with a predicate

`foldl F A Xs'
     fold-left

`foldl1 F Xs'
     fold-left over nonempty lists

`foldr F A Xs'
     fold-right

`foldr1 F Xs'
     fold-right over nonempty lists

`hd Xs'
     return the head element of a list

`hds Xs'
     return the list of all head elements in a list of lists

`id'
     the identity function: `id X => X'

`init Xs'
     return list `Xs' without its last element

`iter N F A'
     generate the list of the first `N' values `A', `F A', `F (F A)',
     ...

`last Xs'
     return the last element of a list

`map F Xs'
     apply function `F' to each member of a list

`max X Y'
     maximum of two values

`min X Y'
     minimum of two values

`mklist X N'
     create a list of `N' `X''s

`neg P'
     negate a predicate

`neq X Y'
     syntactic inequality

`null Xs'
     check whether a list is empty (`[]')

`nums N M'
     generate a list of numbers in a given range

`numsby K N M'
     generate a list of numbers with a given step size

`pop Xs'
     remove the head element from a list

`prd Xs'
     product of a list of numbers

`push Xs X'
     prepend an element to a list (`cons' with arguments reversed)

`reverse Xs'
     reverse a list

`scanl F A Xs'
     apply `foldl' to every initial part of a list

`scanl1 F Xs'
     apply `foldl1' to every nonempty initial part of a list

`scanr F A Xs'
     apply `foldr' to every final part of a list

`scanr1 F Xs'
     apply `foldr1' to every nonempty final part of a list

`sgn X'
     sign of a number

`sum Xs'
     sum of a list of numbers

`take N Xs'
     select the first `N' elements from the list `Xs'

`takewhile P Xs'
     select elements from the beginning of `Xs' while the predicate `P'
     is satisfied

`tl Xs'
     remove the head element from a list

`tls Xs'
     return a list of lists with all head elements removed

`top Xs'
     return the head element from a list

`transpose Xs'
     transpose a list of lists

`uncurry F'
     uncurry a function: turn a function with two arguments into a
     function operating on pairs

`uncurry3 F'
     `uncurry' with triples

`until P F X'
     repeat applying `F' to `X' until `P' is satisfied

`unzip Xs'
     transform a list of pairs into a pair of lists

`unzip3 Xs'
     `unzip' with triples

`while P F A'
     list repeated applications of `F' to `A' while `P' is satisfied

`zip Xs Ys'
     take two lists and return a list of corresponding pairs

`zip3 Xs Ys Zs'
     `zip' with three lists

`zipwith F Xs Ys'
     take two lists and map a binary function to corresponding elements

`zipwith3 F Xs Ys Zs'
     `zipwith' with three lists

   Note that some of these functions are actually overridden with much
faster C versions in the `clib.q' module; see *Note C Replacements for
Common Standard Library Functions::.


File: qdoc.info,  Node: Tuple Functions,  Next: String Functions,  Prev: Standard Functions,  Up: The Standard Library

11.2 Tuple Functions
====================

The `tuple.q' script provides some additional functions operating on
tuples:

`fst Xs'
     return first element of a tuple

`mktuple X N'
     create a tuple of `N' `X''s

`pair X Y'
     construct a pair

`snd Xs'
     return second element of a tuple

`trd Xs'
     return third element of a tuple

`triple X Y Z'
     construct a triple

`tuplecat Xs'
     concatenate a list of tuples

   Moreover, the following list functions from `stdlib.q' are
overloaded to also work on tuples: `append', `cat', `cons', `do',
`dowith', `dowith3', `map', `null', `pop', `push', `reverse', `top'.
Most of these return tuples when applied to such arguments, except
`null' (which returns a truth value), the `do'/`dowith' functions
(which return `()') and `cat' (which always returns a list; use
`tuplecat' if you want to concatenate a collection of tuples instead).

   Also note that the `reverse' and `tuplecat' operations are actually
overridden with much faster C versions in the `clib.q' module; see
*Note C Replacements for Common Standard Library Functions::.


File: qdoc.info,  Node: String Functions,  Next: Option Parsing,  Prev: Tuple Functions,  Up: The Standard Library

11.3 String Functions
=====================

The `string.q' script provides a collection of additional string
functions. Currently the following operations are implemented (most of
these operations are actually overridden with much more efficient C
implementations in the `clib.q' module; see *Note C Replacements for
Common Standard Library Functions::):

`chars S'
     return the list of individual characters in `S'

`join DELIM Xs'
     concatenate a list of strings, interpolating the given `DELIM'
     string between each pair of consecutive strings in the list

`mkstr S N'
     create a string consisting of `N' copies of the given string `S'

`split DELIM S'
     split a string into a list of substrings delimited by characters
     in the given `DELIM' string

`strcat Xs'
     concatenate a list of strings

   Moreover, as of Q 7.8 this module overloads all list operations in
`stdlib.q' so that they work on strings as expected. This lets you use
strings mostly as if they were just another kind of list value. For
instance:

     ==> take 3 "abcdef"
     "abc"

     ==> dropwhile (<="c") "abcdef"
     "def"

     ==> tl "abcdef"
     "bcdef"

     ==> zip "abcdef" "ABCDEF"
     [("a","A"),("b","B"),("c","C"),("d","D"),("e","E"),("f","F")]

     ==> map (+1) "HAL"
     "IBM"

     ==> map ord "HAL"
     [72,65,76]

     ==> [C-1 : C in "IBM"]
     ["H","A","L"]

   (As the last example shows, strings can also serve as sources in the
binding clauses of list comprehensions; cf. *Note Conditionals and
Comprehensions::.)

   Here's a somewhat more practical example which illustrates the use of
various list operations on strings to build a ROT13 translation table:

     import dict;

     def CHARS = strcat ["a".."z"], RCHARS = drop 13 CHARS ++ take 13 CHARS,
       ROT13 = dict $ zip (CHARS++toupper CHARS) (RCHARS++toupper RCHARS);

     rot13 C:Char    = C where C:Char = ROT13!C;
                     = C otherwise;
     rot13 S:String  = map rot13 S;

   (This employs the `dict' function to create a dictionary mapping
lower- and uppercase letters to their equivalents in the ROT13 encoding;
see *Note Dictionaries::.) Example:

     ==> CHARS; RCHARS
     "abcdefghijklmnopqrstuvwxyz"
     "nopqrstuvwxyzabcdefghijklm"

     ==> rot13 "This is an encoded string."
     "Guvf vf na rapbqrq fgevat."

     ==> rot13 _
     "This is an encoded string."


File: qdoc.info,  Node: Option Parsing,  Next: Type-Checking Predicates,  Prev: String Functions,  Up: The Standard Library

11.4 Option Parsing
===================

This module provides one function, `getopt', which takes two arguments:
`OPTS', a list of option descriptions in the format described below,
and `ARGS', a list of strings containing the command line parameters to
be parsed for options. The result is a pair `(OPTS,ARGS)' where `OPTS'
is a list of pairs of options and their arguments (if any; missing
arguments are returned as `()'), and `ARGS' is the list of remaining
(non-option) arguments. Options are parsed using the rules of GNU
`getopt'(1). If an invalid option is encountered (unrecognized option,
missing or extra argument, etc.), `getopt' throws the offending option
string as an exception.

   The `OPTS' argument of getopt is a list of triples
`(LONG,SHORT,FLAG)', where `LONG' denotes the long option, `SHORT' the
equivalent short option, and `FLAG' is one of the symbolic integer
values `NOARG', `OPTARG' and `REQARG' which specifies whether the
option has no argument, an optional argument or a required argument,
respectively. In the returned option-value list, all options will be
represented using their long option equivalents.

   Also note that both the long and short option values in the `OPTS'
argument may actually be of any type, so unneeded options may be
replaced with corresponding dummy values. You only have to make sure
that the values given for the long options allow you to identify which
option was actually specified.

   Finally, please note that, as already pointed out, this script does
_not_ belong to the prelude, and hence has to be imported explicitly if
you want to use the `getopt' operation in your program.

   Example (see *Note C-Style Formatted I/O::, for a description of
`printf' and `fprintf'):

     import getopt;

     def OPTS        = [("--help", "-h", NOARG),
                        ("--foo",  "-f", REQARG),
                        ("--bar",  "-b", OPTARG)];

     invalid_option PROG OPT
                     = fprintf ERROR "%s: invalid option %s\n" (PROG,OPT) ||
                       halt;

     test [PROG|ARGS]
                     = printf "OPTS = %s\nARGS = %s\n" (str OPTS, str ARGS)
                         where (OPTS,ARGS) = catch (invalid_option PROG) $
                           getopt OPTS ARGS;


File: qdoc.info,  Node: Type-Checking Predicates,  Next: Sorting Algorithms,  Prev: Option Parsing,  Up: The Standard Library

11.5 Type-Checking Predicates
=============================

The `typec.q' script contains a collection of predicates which check
whether an expression is of a given type. The following functions
perform a purely syntactic check; they are all equivalent to the
corresponding type guards:

`isbool X'
     check for truth values

`ischar X'
     check for single character strings

`iscomplex X'
     check for complex numbers (cf. *Note Complex Numbers::)

`isexcept X'
     check for `Exception' values (cf. *Note Exception Handling::)

`isfile X'
     check for file objects

`isfloat X'
     check for floating point numbers

`isfunction X'
     check for lambda functions

`isint X'
     check for integers

`islist X'
     check for lists

`isnum X'
     check for numbers

`isrational X'
     check for rational numbers (cf. *Note Rational Numbers::)

`isreal X'
     check for real numbers

`isstr X'
     check for strings

`issym X'
     check for function and variable symbols (special form)

`istuple X'
     check for tuples

   Note that the syntactic number predicates `isint', `isfloat' etc.
only check for a given representation. The `typec.q' module also
provides various predicates which classify numbers according to the
kind of abstract mathematical object they represent, regardless of the
concrete representation:

`iscompval X'
     check for complex values

`isintval X'
     check for integer values

`isratval X'
     check for rational values

`isrealval X'
     check for real values

   These predicates work mostly like their Scheme counterparts. E.g.,
`3' will be classified not only as an integer value, but also as a
rational, real and a complex value. A complex number with zero imaginary
part may also be classified as an integer, rational or real, depending
on its real part. A floating point number with zero fractional part and
a rational number with a denominator of `1' are both also classified as
an integer value.

   Moreover, the following predicates allow you to check whether a
number is exact or inexact (inexact numbers generally involve floating
point values), and whether a number represents an IEEE infinity or NaN
("not a number") value:

`isexact X'
     check for exact numbers

`isinexact X'
     check for inexact numbers

`isinf X'
     check for infinite floating point values

`isnan X'
     check for NaN floating point values

   Finally, two other special-purpose predicates provided in `typec.q'
are:

`isenum X'
     check for enumeration type members

`issym X'
     check for function and variable symbols (special form)


File: qdoc.info,  Node: Sorting Algorithms,  Next: Standard Types,  Prev: Type-Checking Predicates,  Up: The Standard Library

11.6 Sorting Algorithms
=======================

The `sort.q' script provides mergesort and quicksort algorithms for
sorting a list using a given order predicate:

`msort P Xs'
     mergesort algorithm

`qsort P Xs'
     quicksort algorithm

   The mergesort algorithm is more involved than quicksort, but may run
_much_ faster if input lists are large enough and are already partially
sorted. Both algorithms take an order predicate as their first
argument, which makes it possible to sort lists using different
criteria. The order predicate must be a function accepting two
arguments, and must return `true' iff the first argument is strictly
less than the second. For instance, to sort a list in ascending order,
you could say:

     ==> qsort (<) [1,5,3,2,4]
     [1,2,3,4,5]

   By reversing the order predicate, the list is sorted in descending
order:

     ==> qsort (>) [1,5,3,2,4]
     [5,4,3,2,1]

   Custom order predicates also allow you to sort a list according to
different sort keys. For instance, the following example shows how to
sort a list of pairs using the first component of each pair as the sort
key:

     ==> def L = [(1,2),(5,1),(3,3),(1,1),(4,5)]

     ==> var le1 = \X Y.fst X < fst Y

     ==> qsort le1 L
     [(1,2),(1,1),(3,3),(4,5),(5,1)]

   Both algorithms provided by this module are "stable", i.e., they
preserve the relative order of list elements with "equal" sort keys.
This is important when successive sorts are used to order elements
according to different criteria. For instance:

     ==> var le2 = \X Y.snd X < snd Y

     ==> qsort le2 L
     [(5,1),(1,1),(1,2),(3,3),(4,5)]

     ==> qsort le1 _
     [(1,1),(1,2),(3,3),(4,5),(5,1)]


File: qdoc.info,  Node: Standard Types,  Next: Streams,  Prev: Sorting Algorithms,  Up: The Standard Library

11.7 Standard Types
===================

Please note that, as of Q 7.8 this module is no longer included in the
prelude, so you have to import it explicitly if your script needs one of
the data types described here. It is also possible to just import the
individual data type modules (`array.q', `bag.q', etc.) if you do not
need the entire collection.

   The `stdtypes.q' script implements a collection of efficient
container data structures, which currently comprises arrays, heaps
(priority queues), ordered sets and bags, and (ordered as well as
hashed) dictionaries. The different data types are actually implemented
in the scripts `array.q', `bag.q', `dict.q', `hdict.q', `heap.q' and
`set.q'. Many operations of these modules are overloaded; the
declarations of these operations can be found in the `stddecl.q' script
which is included by the different modules.

   All data structures support equality checking with `=' and `<>', as
well as the operation `#' to determine the size of an object (number of
elements it contains). Furthermore, the set and bag data structures
overload the `<', `>', `<=' and `>=' operators to implement
subset/subbag comparisons, and the `+', `-' and `*' operators to
compute the union, difference and intersection of two sets or bags,
respectively.

   For convenience, default views are provided for all container data
structures, so that they will be printed in the form of a construction
function (which is implemented as a virtual constructor of the type,
cf. *Note Views::) applied to a member list (such as, e.g., `set
[1,2,3]' in the case of `Set', or `dict [("a",1),("b",2),("c",3)]' in
the case of `Dict'). These are at a low priority, so you can easily
override them when needed.

* Menu:

* Arrays::
* Heaps::
* Sets::
* Bags::
* Dictionaries::
* Hashed Dictionaries::


File: qdoc.info,  Node: Arrays,  Next: Heaps,  Prev: Standard Types,  Up: Standard Types

11.7.1 Arrays
-------------

The `array.q' script provides a zero-based array data structure `Array'
with logarithmic access times, implemented as size-balanced binary
trees. The following operations are provided:

`array Xs'
     create an array from list `Xs'

`array2 Xs'
     create a two-dimensional array from a list of lists

`emptyarray'
     return the empty array

`mkarray X N'
     create an array consisting of `N' `X''s

`mkarray2 X (N,M)'
     create a two-dimensional array with `N' rows and `M' columns

`isarray X'
     check whether `X' is an array

`null A'
     check whether `A' is the empty array

`A1 = A2, A1 <> A2'
     array equality/inequality

`#A'
     size of an array

`A!I'
     return `I'th member of an array

`A!(I,J)'
     two-dimensional subscript

`members A, list A'
     list the members of `A'

`members2 A, list2 A'
     list a two-dimensional array

`first A, last A'
     return the first and last element of an array

`rmfirst A, rmlast A'
     remove the first and last element from an array

`insert A X'
     insert `X' at the beginning of `A'

`append A X'
     append `X' at the end of `A'

`update A I X'
     replace the `I'th member of `A' by `X'

`update2 A (I,J) X'
     update two-dimensional array


File: qdoc.info,  Node: Heaps,  Next: Sets,  Prev: Arrays,  Up: Standard Types

11.7.2 Heaps
------------

The `heap.q' script provides an efficient heap (priority queue) data
structure `Heap' implemented as size-balanced binary trees. Heaps allow
quick (i.e., constant-time) access to the smallest element, and to
insert new elements in logarithmic time. The present implementation
does not allow fast random updates of heap members; if such
functionality is required, bags should be used instead (see *Note
Bags::).

   Heap members must be ordered by the `<=' predicate. Multiple
instances of the same element may be stored in a heap; however, the
order in which equal elements are retrieved is not specified.

   The following operations are provided:

`emptyheap'
     return the empty heap

`heap Xs'
     construct a heap from a list of its members

`isheap X'
     determine whether `X' is a heap

`null H'
     check whether `H' is the empty heap

`H1 = H2, H1 <> H2'
     heap equality/inequality

`#H'
     size of a heap

`members H, list H'
     list the members of `H' in ascending order

`first H'
     return the first element of `H'

`rmfirst H'
     remove the first element from `H'

`insert H X'
     insert `X' into `H'


File: qdoc.info,  Node: Sets,  Next: Bags,  Prev: Heaps,  Up: Standard Types

11.7.3 Sets
-----------

The `set.q' script provides an ordered set data structure `Set'
implemented as AVL trees, and thus guaranteeing logarithmic access
times.  The following operations are defined in `set.q':

`emptyset'
     return the empty set

`set Xs'
     create a set from a list of its members

`isset X'
     check whether `X' is a set

`null M'
     check whether `M' is the empty set

`member M X'
     check whether `M' contains element `X'

`M1 = M2, M1 <> M2'
     set equality/inequality

`M1 < M2, M1 > M2, M1 <= M2, M1 >= M2'
     set comparison

`M1 + M2, M1 - M2, M1 * M2'
     set union, difference and intersection

`#M'
     size of a set

`members M, list M'
     list the members of `M' in ascending order

`first M, last M'
     return the first and last member of `M'

`rmfirst M, rmlast M'
     remove the first and last member from `M'

`insert M X'
     insert `X' into `M'

`delete M X'
     delete `X' from `M'


File: qdoc.info,  Node: Bags,  Next: Dictionaries,  Prev: Sets,  Up: Standard Types

11.7.4 Bags
-----------

The `bag.q' script defines the type `Bag' as a variant of the set data
structure which may contain multiple instances of the same element. The
operations are analogous to those of the set data structure; see *Note
Sets::.  The `emptybag' function returns the empty bag, and `bag Xs'
constructs a bag from a list `Xs' of its members. The `isbag' predicate
checks whether its argument is a bag.


File: qdoc.info,  Node: Dictionaries,  Next: Hashed Dictionaries,  Prev: Bags,  Up: Standard Types

11.7.5 Dictionaries
-------------------

The `dict.q' script supplies an (ordered) dictionary data structure
`Dict' which maps "keys" from an ordered set to corresponding "values".
Like sets and bags, dictionaries are implemented as AVL trees, thus
guaranteeing logarithmic access times to individual members of the
dictionary. The following operations are defined in `dict.q':

`emptydict'
     return the empty dictionary

`dict XYs'
     create a dictionary from a list of key/value pairs

`mkdict Y Xs'
     create a dictionary from a list of keys and an initial value

`isdict X'
     check whether X is a dictionary

`null D'
     check whether `D' is the empty dictionary

`member D X'
     check whether `D' contains `X' as a key

`D1 = D2, D1 <> D2'
     dictionary equality/inequality

`#D'
     size of a dictionary

`D!X'
     return the value `Y' associated with `X' in `D'

`members D, list D'
     list the members (key/value pairs) of `D' in ascending order by key

`keys D'
     list the keys of `D' in ascending order

`vals D'
     list the corresponding values

`first D, last D'
     return the first and last member of `D'

`rmfirst D, rmlast D'
     remove the first and last member from `D'

`insert D (X,Y)'
     insert a key/value pair `(X,Y)' into `D'; update an existing entry
     for `X' if present

`delete D X'
     remove key `X' from `D'

`update D X Y'
     same as `insert D (X,Y)'


File: qdoc.info,  Node: Hashed Dictionaries,  Prev: Dictionaries,  Up: Standard Types

11.7.6 Hashed Dictionaries
--------------------------

The `hdict.q' script implements hashed dictionaries (`HDict' type), a
variation of the `Dict' type which uses hashed key values obtained with
the built-in `hash' function. The actual key-value pairs are stored in
"buckets" for each hash value. This kind of data structure is also
known as "hashes" or "associative arrays" in other programming
languages.

   The main advantage of the `HDict' type is that key values can be of
any type and do not have to belong to an ordered set. For instance,
`hdict [(0,1),(foo,2),("bar",3)]' is a legal `HDict' value which maps
the integer `0' to `1', the symbol `foo' to `2', and the string `"bar"'
to `3'.

   There are some other notable differences between the `Dict' and the
`HDict' type. First of all, a `HDict' stores it members in an
apparently random order which may depend on the order in which new
entries are added to the dictionary. Hence equality testing for
`HDict's is more involved than for `Dict's, as the member lists of two
"equal" `HDict's may be arbitrary permutations of each other. This also
means that the `first', `rmfirst', `last' and `rmlast' operations do
not make much sense with hashed dictionaries and are not supported by
the `HDict' type. Moreover, key values are always compared
syntactically when looking up and updating entries. Hence, e.g., `0'
and `0.0' are different key values in a `HDict' object, whereas they are
considered to be the same for the `Dict' type.

   Apart from these differences, the operations of the `HDict' and
`Dict' types work analogously. The `HDict' constructors are named
`emptyhdict', `hdict' and `mkhdict' which take the same arguments as
the corresponding `Dict' constructors, and the `ishdict' predicate
checks for `HDict' values.


File: qdoc.info,  Node: Streams,  Next: Conditionals and Comprehensions,  Prev: Standard Types,  Up: The Standard Library

11.8 Streams
============

The `stream.q' script provides operations on streams, Q's lazy list
data structure. Note that, as of Q 7.1, streams are now provided as a
built-in data type, which is declared as follows:

     public type Stream = special const nil_stream, cons_stream X Xs;

   As explained in *Note Lists Streams and Tuples::, the Q language now
also provides syntactic sugar for these structures, so that streams can
be denoted just like lists, using curly braces instead of brackets.

   The `stream.q' module overloads most list operations so that they
work on streams in a completely analogous fashion, and provides the
following additional operations:

`isstream X'
     check whether an object is a stream

`stream Xs'
     convert a tuple or a list to a stream

`list Xs'
     convert a stream to a list

`strict Xs'
     force the heads and tails of a stream (see comments below)

`lazy Xs'
     memoize the heads and tails of a stream (see comments below)

`numstream N'
     generate the stream of all numbers `>=N'

`numstreamby K N'
     generate a number stream with given step size

`mkstream X'
     generate an infinite stream of `X''s

`repeat X'
     generate an infinite stream of `X''s (works like `mkstream', but
     is implemented as a special form and hence doesn't evaluate `X')

`repeatn ~N X'
     generate a stream of `N' `X''s (special form, equivalent to `take
     N (repeat X)')

`cycle Xs'
     generate an infinite stream which repeatedly cycles through the
     members of the list or stream `Xs'

`iterate F A'
     generate the stream of all values `A', `F A', `F (F A)', ...

`streamcat Xs'
     concatenate a stream or list of streams and/or lists (see comments
     below)

   Operations like `#', `all', `foldl' will of course cause troubles
with infinite streams since it can take them an infinite time to
compute the result. The same holds for the `foldr' function unless the
second argument of the folded operation is special.

   The `stream.q' module also extends the list concatenation function
`cat' defined in `stdlib.q' to work with any mixture of streams and
lists. This operation will always return a list, thus it is not
suitable to work with infinite streams either.

   As a remedy, the `streamcat' function is provided which concatenates
a stream of streams in a fully lazy manner, i.e., you can concatenate a
(possibly infinite) stream of (possibly infinite) streams. The argument
of `streamcat' can actually be a list or stream, which consists of any
mixture of streams and lists. The result is always a stream.

   The `strict' function expands a stream by forcing all its heads and
tails to be evaluated (using `~', see *Note Special Forms::). This has
essentially the same effect as `list', but the result is still a stream
rather than a list.

   Conversely, the `lazy' function makes a stream even "lazier" than it
normally is, by causing the heads and tails of the result stream to be
memoized (using `&', see *Note Special Forms::). This reduces
computation times when a stream is to be traversed repeatedly.

   The `stream.q' module also provides some additional operations to
force or memoize only the heads (`hdstrict', `hdlazy') or the tails
(`tlstrict', `tllazy') of a stream, as well as the "mixed mode"
operations `strict_lazy' and `lazy_strict'. See the `streams.q' script
for a description of those.

   An example for the use of `lazy' can be found in *Note Memoization
and Lazy Evaluation::.


File: qdoc.info,  Node: Conditionals and Comprehensions,  Next: Mathematical Functions,  Prev: Streams,  Up: The Standard Library

11.9 Conditionals and Comprehensions
====================================

The `cond.q' script provides the special forms needed to implement both
various conditionals and tuple/list/stream comprehensions.

`ifelse ~P X Y, when ~P X, unless ~P X'
     simple conditional expressions

`cond CASES, case ~X CASES'
     multiway and pattern matching conditional expressions

`condfun CASES, casefun CASES'
     multiway and pattern matching conditional abstractions

`dowhile P X'
     simple looping construct

`for CLAUSES X'
     iterate over lists and streams

`tupleof X CLAUSES, listof X CLAUSES, streamof X CLAUSES'
     tuple, list and stream comprehensions

   All these functions are implemented as special forms.

   The `ifelse' function has already been mentioned in *Note
Conditional Expressions and Lambdas::. It returns the value of either
`X' or `Y', depending on whether the first argument is `true' or
`false'. For instance, here is how the factorial function can be
implemented using `ifelse':

     fac N                   = ifelse (N>0) (N*fac (N-1)) 1;

   Or, if you prefer to write this using the `if X then Y else Z'
syntax described in *Note Conditional Expressions and Lambdas:::

     fac N                   = if N>0 then N*fac (N-1) else 1;

   For cases in which you are only interested in executing one of the
branches, you can also use the `when' and `unless' functions which
return a `()' default value if the branch condition is _not_ met:

     ==> def X = 1

     ==> when (X>0) (writes "positive\n")
     positive
     ()

     ==> unless (X>=0) (writes "negative\n")
     ()

   As indicated, these constructs are most useful when used with
functions involving side-effects; `unless' works exactly like `when',
but reverses the branch condition `P'. The `if X then Y' construct is
syntactic sugar for `when':

     ==> if X>0 then writes "positive\n"
     positive
     ()

   A more general conditional expression is also provided, which allows
you to discriminate between an arbitrary number of cases. It has the
form:

     cond ((P1,X1),(P2,X2),...)

   Using the "grouping syntax" described in *Note Lists Streams and
Tuples::, this can also be written more succintly as:

     cond (P1,X1;P2,X2;...)

   This special form looks and behaves like Lisp's `cond'. That is, it
returns the first value `X' for which the corresponding condition `P'
evaluates to `true'. A branch condition of `true' can be used to
indicate the default case. If no case matches then `cond' raises a
`syserr 8' exception (note that the same error code is also thrown in
case of an invalid conditional in a rule, cf. *Note Exception
Handling::).

   There is also a more Haskell-like `case' conditional in which the
first expression of each case is a pattern `P' to be matched against
the given expression `X':

     case X (P1,Y1;P2,Y2;...)

   The expression `X', which is passed by value, is matched against
each of the patterns `P' in turn, and as soon as a matching pattern is
found, the corresponding value `Y' is returned, with the variables in
the pattern bound to the corresponding values using `lambda'. A pattern
of `_' can be used to denote the default case. If no case matches then
`case' raises a `syserr 8' exception.

   Some examples:

     fac N                   = cond (N>0, N*fac (N-1); true, 1);
     sign X                  = cond (X>0, 1; X<0, -1; true, 0);
     prod Xs                 = case Xs ([], 1; [Y|Ys], Y*prod Ys);
     reply X                 = case X
                               ("y" , true;
                                "n" , false;
                                _   , throw "bad reply");

   As of Q 7.7, there are also variations of `cond' and `case', named
`condfun' and `casefun', which take the data to be analyzed as the
second argument. These constructs can be used to create anonymous
conditional and pattern-matching functions, respectively:

     condfun (P1,F1;P2,F2;...) X
     casefun (P1,Y1;P2,Y2;...) X

   Note that `casefun' works exactly like `case', but the arguments are
reversed, so the above `reply' function could also be defined with an
implicit parameter, as follows:

     reply                   = casefun
                               ("y" , true;
                                "n" , false;
                                _   , throw "bad reply");

   In contrast, `condfun' interprets each branch `(P,F)' as a pair of a
predicate `P' and a function `F' which are to be applied to `condfun''s
second "data" parameter `X'. Going through the tuple of branches,
`condfun' returns the first value `F X' for which `P X' evaluates to
`true'. Thus `condfun' works like `cond' except that the constituents of
each branch are not constant values, but functions to be applied to the
data parameter. For instance, here's an alternative way to define the
`sign' function from above:

     sign                    = condfun ((>0), cst 1; (<0), cst (-1);
                                        cst true, cst 0);

   The `dowhile' function implements a simple looping construct which
keeps re-evaluating the expression in the second argument as long as the
condition in the first argument evaluates to `true'. Both arguments are
special. The result is always `()'. This construct obviously makes
sense only with expressions involving side-effects. For instance:

     ==> def F = fopen "/etc/passwd" "r"

     ==> dowhile (not feof F) (writes (freads F++"\n"))

   The `for' function provides a means to iterate an operation with
side-effects over tuples, lists and streams. It takes a tuple of clauses
in the first argument, which has the same format as the second argument
of a comprehension (see below). The second argument is then evaluated
for each binding, performing parameter binding using `lambda'. The
result is always `()'. Example:

     ==> for (I in [1..3],J in [1..I]) (write (I,J) || writes " ")
     (1,1) (2,1) (2,2) (3,1) (3,2) (3,3) ()

   The `listof' function allows to specify a list of values in a manner
similar to the way sets are described in mathematics:

     listof X (Y1, Y2, ...)

   This construct is also commonly called a "list comprehension". For
convenience, the Q language provides syntactic sugar for list
comprehensions so that they can also be written as follows:

     [X : Y1, Y2, ...]

   (Note that in scripts this alternative notation is only permitted on
the right-hand side of equations and variable definitions, as the colon
`:' is already used to denote type guards in left-hand side
expressions. So in the unlikely case that you have to extend the
definition of `listof' you will have to use the first syntax from
above.)

   The expressions `Y1, Y2, ...' may either be so-called binding
clauses or conditional clauses. A "binding clause" is an expression of
the form `Z in Zs', using the relational operator `in', which is
declared as follows:

     public const (in) X Y @ 2; // relational in operator

   A binding clause specifies that the pattern `Z' should be matched in
turn to each member of the list `Zs'; only values matching the pattern
will be extracted, and free variables in the pattern are bound to their
corresponding values using `lambda'. Binding clauses are considered
from left to right, which means that a clause may refer to any variable
introduced in an earlier binding clause.

   Any other expression specifies a "conditional clause" which must
evaluate to a truth value; only those elements will be listed for which
the conditional clause is satisfied.

   Note that, as of Q 7.8, binding clauses may actually draw values from
any combination of tuples, lists and streams. The result of the list
comprehension is always a list no matter which kind of sequences the
values come from. This also works analogously with the other type of
comprehensions discussed below.

   As an example, here is a function which computes the prime numbers
up to a given integer `N' using Erathosthenes' sieve. Note the list
comprehension `[Y:Y in Xs,Y mod X<>0]' in the second equation which is
used to extract all numbers which are not divisible by the first list
member (which is always a prime by virtue of the construction).

     primes N                = sieve [2..N];
     sieve [X|Xs]            = [X|sieve [Y:Y in Xs,Y mod X<>0]];
     sieve []                = [];

   The `tupleof' function works completely analogous, except that it
generates tuples instead of lists. It also uses the same kind of
syntactic sugar, but the expression is surrounded by ordinary
parentheses instead of brackets. These constructs are also known as
"tuple comprehensions".

   Similarly, the `streamof' function implements "stream
comprehensions". It works like `listof', but in a lazy fashion and
returns a stream instead of a list as the result. The Q language
provides syntactic sugar for stream comprehensions, so that they look
like list comprehensions, using curly braces instead of brackets.

   Here is the same prime sieve algorithm as above, formulated with
streams instead of lists. Instead of a list of all primes up to a given
limit it simply generates the infinite stream of _all_ prime numbers:

     primes                  = sieve {2..};
     sieve {X|Xs}            = {X|sieve {Y:Y in Xs,Y mod X<>0}};


File: qdoc.info,  Node: Mathematical Functions,  Next: Complex Numbers,  Prev: Conditionals and Comprehensions,  Up: The Standard Library

11.10 Mathematical Functions
============================

The `math.q' script defines the `const' variables `inf' and `nan',
which denote the special IEEE floating point values for (positive)
infinity and NaN ("not a number"), and the following additional
operations on floating point numbers:

`floor X, ceil X'
     round to the nearest integer below and above the given number

`asin X, acos X, tan X'
     additional trigonometric functions

`lg X, log X'
     base 2 and 10 logarithms

`sinh X, cosh X, tanh X'
     hyperbolic functions

`asinh X, acosh X, atanh X'
     inverse hyperbolic functions


File: qdoc.info,  Node: Complex Numbers,  Next: Rational Numbers,  Prev: Mathematical Functions,  Up: The Standard Library

11.11 Complex Numbers
=====================

The `complex.q' script implements complex numbers. As of Q 7.7,
`Complex' is now implemented as an abstract data type, very much like
Rational (cf. *Note Rational Numbers::). A complex value with real part
`X' and imaginary part `Y' is represented as `X+:Y', employing the
virtual constructor `+:', which is implemented as a binary infix
operator with the same precedence as `+'. (The alias `(:+)' for `(+:)'
is provided for Haskell compatibility.) Of course, to construct a
complex value you can also use the more customary notation `X+i*Y',
where the constant `i', which is implemented as a `const' variable,
denotes the imaginary unit `0+:1'.

   The script overloads the usual arithmetic operations (including
exponentiation), `sqrt', `exp', logarithms, trigonometric and
hyperbolic functions so that they also work with complex numbers. The
following basic operations are also provided:

`abs Z, arg Z'
     absolute value and argument

`re Z, im Z'
     real and imaginary part

`re_im Z'
     returns a pair with both the real and the imaginary part

`conj Z'
     complex conjugate

`cis X'
     the complex exponential `cis X = exp (i*X) = cos X + i*sin X'

`polar R X'
     convert polar coordinates to complex number, `polar R X = R*cis X'

   Note that `cis' and `polar' are only defined on `Real' arguments.
The other operations (including the virtual constructor `+:') work
consistently on both real and complex numbers. The `arg' function
returns the polar angle (in radians), thus `(abs Z,arg Z)' denotes the
polar coordinates of a complex number `Z'. Conversely, `polar R X'
converts the polar coordinates `(R,X)' back to the corresponding
complex number.

   Some examples:

     ==> 2^(1/i)
     0.769238901363972 +: -0.638961276313635

     ==> lg _
     0.0 +: -1.0

     ==> (abs _,arg _)
     (1.0,-1.5707963267949)

   Please note that the virtual constructor `+:', just like Haskell's
`:+' constructor, is really an operator symbol with the same precedence
as `+'. Thus an expression like `0+:1' is not "atomic", but has the
same precedence and associativity as all addition operators. This
yields results which might be somewhat surprising for the uninitiated.
For instance, consider:

     ==> 1+:2*3+:4
     1+:10

   This result is indeed correct, as the input expression is parsed as
`1+:(2*3)+:4 = (1+:6)+:4' and _not_ as `(1+:2)*(3+:4) = -5+:10'. (Note
that, unlike Haskell's `:+', Q's `+:' operator can be applied to _both_
real and complex operands, so that `Z1+:Z2' will yield the same as
`Z1+i*Z2' in any case.)


File: qdoc.info,  Node: Rational Numbers,  Next: Graphics,  Prev: Complex Numbers,  Up: The Standard Library

11.12 Rational Numbers
======================

Thanks to the work of Rob Hubbard, Q versions 7.2 and later also include
the `rational.q' script which implements the rational number type
`Rational' which is a subtype of `Real'. Please note that at present
this module only provides the data type and the basic arithmetic. More
advanced approximation and formatting operations for rational numbers
can be found in a separate add-on package available from the Q website
which also includes additional documentation [Hubbard 2006].

   The module overloads the usual arithmetic operations as well as the
functions `abs', `sgn', `round', `trunc', `int', `frac' and `pow' (the
latter comes from the `clib' module), and provides the following
additional operations:

`rational X'
     create a rational number from an integer, or a pair of integers
     (numerator, denominator)

`num Q, den Q'
     normalized numerator and denominator (numerator and denominator are
     coprime and the sign is always in the numerator)

`num_den Q'
     returns a pair of integers with the normalized numerator and
     denominator

   Last but not least, the module also implements the exact division
operator `%' (with the same precedence as `/') which returns a rational
or complex rational number for each combination of integer, rational
and complex integer/rational arguments; for any other arguments of type
`Num' `%' behaves like the `/' operator. The `%' operator is actually
implemented as a virtual constructor of the `Rational' type (cf. *Note
Views::), and `rational.q' provides a corresponding default view for
rational numbers in the format `N%D', where `N' and `D' are the
normalized numerator and denominator, respectively.

   Examples:

     ==> 5%7 + 2%3
     29%21

     ==> 3%8 - 1%3
     1%24

     ==> pow (11%10) 3
     1331%1000

     ==> pow 3 (-3)
     1%27

     ==> num_den _
     (1,27)

     ==> rational _
     1%27

   Since `Rational' is a subtype of `Real', operations on complex
numbers with rational components also work as expected:

     ==> (1+:2)%(3+:4)
     11%25+:2%25

     ==> var sqr = \X.X*X; sqr (5+:-5%18)
     8075%324 +: -25%9


File: qdoc.info,  Node: Graphics,  Next: Diagnostics and Error Messages,  Prev: Rational Numbers,  Up: The Standard Library

11.13 Graphics
==============

The `graphics.q' script implements an interface to Adobe's PostScript
language [Adobe 1990]. PostScript is a page description language which
has become one of the main standards in the desktop publishing world.
The nice thing about PostScript is that page descriptions are
_device-independent_ - the same page description can be used for
different output devices such as a screen previewer or a laser printer.
The `graphics.q' script allows you to produce graphics output on any
PostScript device. Note that, as already pointed out, this script does
_not_ belong to the set of "standard" scripts included by the prelude,
and hence has to be imported explicitly if you want to use the
operations described in the following.

   The graphics device is implemented by the `GRAPHICS' variable, which
by default is assigned to standard output. This is useful for taking a
look at the generated PostScript code, e.g., for debugging purposes,
but in a real application you will of course redirect the output to
some PostScript file or device, which can be done by assigning a
suitable value to the `GRAPHICS' variable. Any file object open for
writing will do, however the script also provides the following
convenience functions which each return an output file or pipe which can
be used as the value of the `GRAPHICS' variable:

`gsdev'
     pipe to the `ghostscript' program (see below)

`gvdev'
     pipe to `ghostview' (see below)

`lpdev'
     printer device (usually a pipe to `lpr'(1))

`filedev NAME'
     output file with name `NAME'

`nulldev'
     the null device (a synonym for `filedev "/dev/null"', or `filedev
     "nul"' under DOS/Windows)

   For instance, you define the graphics device as a pipe to
ghostscript as follows:

     def GRAPHICS = gsdev;

   You can either add this definition to your main script, or enter it
directly at the command prompt of the interpreter (see *Note Command
Language::).

   Ghostscript is a popular PostScript previewer available for a wide
range of different platforms. Ghostview is an improved X interface for
ghostscript. A similar program, GSView, is also available for the
Windows operating system. Note, however, that GSView cannot read its
input from a pipe, hence `gvdev' is not directly supported on Windows.
To preview your PostScript output under Windows, you can either use
`gsdev', or employ `filedev' to set up an output file and then invoke
GSView manually. For instance (assuming that the `gsview32' program is
on your path):

     ==> def GRAPHICS = filedev "graphics.ps"

     ==> // output your PostScript graphics here ...

     ==> !gsview32 graphics.ps

   Also note that on most systems output to a pipe created with `popen'
is buffered, thus you might have to flush buffered data when using
`gsdev' or `gvdev' as the graphics device. In addition, it might be
necessary to invoke `flushpage' to force an immediate display update,
see *Note Overview of Graphics Operations::. This can be done as
follows:

     ==> flushpage || fflush GRAPHICS

   If output goes to a printer or a file, you will probably need a
minimal header which identifies the file as a PostScript document. To
include such a header in the output file, use the `psheader' function
_before_ invoking any other graphics operation:

     ==> psheader

   You can also set up a custom header and include other DSC and EPSF
comments by means of the `ps' function; see *Note DSC and EPSF
Comments::, for details.

   In the following we give an overview of the graphics operations in
the PostScript language, as they are provided by this module, and
describe the implemented functions. For more details about PostScript
please refer to [Adobe 1990].

* Menu:

* Coordinate System::
* Overview of Graphics Operations::
* Path Construction::
* Painting::
* Clipping::
* Graphics State::
* Miscellaneous Operations::
* DSC and EPSF Comments::


File: qdoc.info,  Node: Coordinate System,  Next: Overview of Graphics Operations,  Prev: Graphics,  Up: Graphics

11.13.1 Coordinate System
-------------------------

The PostScript coordinate system has its origin `(0,0)' in the lower
left corner of the output page or display window, with the positive `X'
and `Y' axes extending horizontally to the right and vertically upward,
respectively. The default unit length is 1 "point" which is 1/72 of an
inch.

   The origin of the coordinate system, as well as the unit lengths and
orientation of the `X' and `Y' axes can be changed by means of the
`translate', `scale' and `rotate' operations, cf.  *Note Graphics
State::.


File: qdoc.info,  Node: Overview of Graphics Operations,  Next: Path Construction,  Prev: Coordinate System,  Up: Graphics

11.13.2 Overview of Graphics Operations
---------------------------------------

The process of painting a graphics object usually consists of the
following three steps:

   * Modify graphics parameters such as the text font, the color, or the
     translation and scaling of the coordinate axes.

   * Construct a "path" which outlines the shape of the object to be
     painted.

   * Execute the appropriate painting operation to display the object
     on the output page.

   A "path" is an ordered sequence of straight and curved line segments.
The individual segments may be connected to each other or they may be
disconnected. Thus a path may consist of several connected pieces which
are referred to as the "subpaths" of the path. In a subpath, each line
segment starts at the point where the previous segment ends. The
`newpath' function is used to begin a new path. A new subpath is
obtained by invoking `moveto' which specifies the first point in the
subpath. Various operations are provided to add straight and curved line
segments to the current subpath. For instance, a path consisting of
three straight line segments may be denoted as follows:

     newpath || moveto 0 0 || lineto 1 0 || lineto 1 1 || lineto 0 1

   The last point on the current subpath can be connected back to its
starting point (usually the last point specified with `moveto') by
"closing" the subpath with the `closepath' operation. For instance, a
rectangle is specified as follows:

     newpath || moveto 0 0 || lineto 1 0 || lineto 1 1 || lineto 0 1 ||
     closepath

   Having constructed a path, the `stroke' function draws the line
segments contained in the path. Alternatively, `fill' may be used to
fill the interior of the path (for this purpose the entire path should
consist of closed subpaths).

   The precise appearance of stroked and filled objects on the output
page is controlled by a collection of parameters referred to as the
"graphics state". Various operations are provided for changing these
parameters. For instance, you can set the linewidth and dash pattern
used by `stroke', the color used by the `stroke' and `fill' operations,
and the scale and translation of the coordinate axes. The current
settings can be saved on a stack using `gsave' and restored (popped
from the stack) with `grestore'.

   Another important parameter is the current "clipping path" which
specifies the regions on the page which can be affected by the painting
operations. By default, the paintable area is the whole page. In order
to restrict painting to a user-defined region, a path is constructed as
usual, and then the `clip' function is used to set this path as the
clipping path. Subsequent paint operations will only paint the interior
of the clipping path, i.e., the region which would have been filled had
we applied the `fill' operation instead of `clip' to the constructed
path.  Multiple applications of `clip' are accumulative. That is, `clip'
intersects the current clipping area (as defined by previous
invocations of `clip') with the interior of the current path.

   The treatment of textual output is somewhat special. It is possible
to define a path consisting of the outlines of the characters in a
given text string by means of the `charpath' function. More commonly,
however, text strings are simply displayed at a given position which is
accomplished by means of the `show' function. For instance, to display
a text string `S' at a position `(X,Y)' on the current page the
following expression is used:

     moveto X Y || show S

   The `graphics.q' script also provides several operations which deal
with a graphics page as a whole. First of all, `showpage' emits the
current page, and prepares for the next page by erasing the current
page. The `copypage' operation is like `showpage', but keeps the
contents of the current page. This allows you to accumulate the
contents of several pages. Both `showpage' and `copypage' are mainly
used when output goes to a printer.

   Two additional operations are provided for interactive use, when
output goes to a display window. The `erasepage' function causes the
contents of the current page to be erased. The `flushpage' operation
updates the display, like `showpage' or `copypage', but does not start
a new page. (To improve performance, graphics output under the X window
system is usually performed in larger chunks. The `flushpage' operation
is required to synchronize the display by flushing any unwritten data.)
Note that this operation is _not_ part of the PostScript standard, but
only works with Ghostscript and possibly some similar Postscript
viewers.

   If you want to achieve special effects which cannot be implemented in
terms of the operations provided by `graphics.q', you can directly
invoke PostScript commands by means of the `ps' function. Also, you can
copy a PostScript file to the graphics device with the `psfile'
operation. As an example for the `ps' function, operations to display a
string right-justified or centered at the current position can be
implemented as follows:

     showright S:String      = ps (psstr S++
                             " dup stringwidth pop neg 0 rmoveto show\n");
     showcenter S:String     = ps (psstr S++
                             " dup stringwidth pop 2 div neg 0 rmoveto show\n");

   (The `psstr' function converts a string to PostScript syntax; see
*Note Miscellaneous Operations::.) The `ps' function is also useful to
include DSC and EPSF comments in your graphics output if this is
necessary. See *Note DSC and EPSF Comments::, for details.


File: qdoc.info,  Node: Path Construction,  Next: Painting,  Prev: Overview of Graphics Operations,  Up: Graphics

11.13.3 Path Construction
-------------------------

The `graphics.q' script defines the following collection of operations
to define the current path used by the painting and clipping operations:

`newpath'
     start a new path

`closepath'
     close the current subpath

`clippath'
     set the current path to the current clipping path

`moveto X Y'
     absolute move to position `(X,Y)'

`rmoveto DX DY'
     move relatively by `DX' units in horizontal and `DY' units in
     vertical direction

`lineto X Y'
     straight line segment between the current point and absolute
     location `(X,Y)'

`rlineto DX DY'
     straight line segment specified by displacement `(DX,DY)' with
     respect to the current point

`curveto X1 Y1 X2 Y2 X3 Y3'
     Bezier cubic section between the current point and `(X3,Y3)', using
     `(X1,Y1)' and `(X2,Y2)' as control points

`rcurveto DX1 DY1 DX2 DY2 DX3 DY3'
     Bezier cubic section, with the points specified as displacements
     with respect to the current point

`arc X Y R A1 A2'
     arc of a circle with radius `R' centered at location `(X,Y)'
     starting and ending at angles `A1' and `A2' (`0<=A1,A2<=360'),
     respectively; if there is a current point, it is connected by a
     straight line segment to the first point on the arc

`narc X Y R A1 A2'
     negative arc; the arc is drawn in clockwise rather than in
     counter-clockwise direction

`arct X1 Y1 X2 Y2 R'
     arc specified by tangent lines; the center of the arc is located
     within the inner angle of the tangents, with the first tangent
     connecting the current point and `(X1,Y1)', and the second tangent
     connecting `(X1,Y1)' and `(X2,Y2)'

`charpath S T'
     path consisting of the character outlines that would result if the
     string S where shown at the current point using `show'; `T' is a
     truth value denoting whether the path should be used for stroking
     to draw the character outlines (`T = false') or whether the path
     should be adjusted for use with `fill' or `clip' (`T = true')


File: qdoc.info,  Node: Painting,  Next: Clipping,  Prev: Path Construction,  Up: Graphics

11.13.4 Painting
----------------

The following operations are provided for stroking and filling, and for
displaying text:

`stroke'
     draw the line segments in the current path

`fill, eofill'
     fill the interior of the current path (any unclosed subpaths of
     the current path are closed automatically)

`show S'
     paint string `S' at the current point

   The `fill' function uses the "nonzero winding number" rule for
determining which points lie "inside" the current path, while `eofill'
uses the "even-odd" rule. Please refer to [Adobe 1990] for the details.


File: qdoc.info,  Node: Clipping,  Next: Graphics State,  Prev: Painting,  Up: Graphics

11.13.5 Clipping
----------------

The following operations are used to determine the current clipping
path, as described in *Note Overview of Graphics Operations:::

`clip, eoclip'
     intersect the current clipping area with the interior of the
     current path

   The `clip' and `eoclip' operations use the same rules for insideness
testing as `fill' and `eofill', respectively, see *Note Painting::.


File: qdoc.info,  Node: Graphics State,  Next: Miscellaneous Operations,  Prev: Clipping,  Up: Graphics

11.13.6 Graphics State
----------------------

As already indicated in *Note Overview of Graphics Operations::, the
PostScript graphics state is a collection of parameters which control
the behavior of the graphics operations. The `graphics.q' script
provides the following operations to manipulate these parameters:

`gsave'
     save the current graphics state

`grestore'
     restore the previously saved graphics state

`savematrix'
     push the current transformation matrix (CTM) on the PostScript
     stack; the CTM is manipulated by the `translate', `scale' and
     `rotate' operations, see below

`restorematrix'
     restore the CTM from the stack

`translate TX TY'
     move the origin of the coordinate system `TX' units in horizontal
     and `TY' units in vertical direction

`scale SX SY'
     scale the unit lengths of the coordinate axes by `SX' in horizontal
     and `SY' in vertical direction

`rotate A'
     rotate the coordinate system by an angle of `0<=A<=360' degrees

`setlinewidth X'
     set the line width to `X' units

`setlinecap N'
     set the line cap style (`0' = butt caps, `1' = round caps, `2' =
     projecting square caps)

`setlinejoin N'
     set the line join style (`0' = miter joins, `1' = round joins, `2'
     = bevel joins)

`setdash Xs DX'
     set the dash pattern (see below)

`setgray X'
     set the gray shade (`0' = black, `1' = white)

`setrgbcolor R G B'
     set the color in the RGB model (`R' = red, `G' = green, `B' = blue)

`sethsbcolor H S B'
     set the color in the HSB model (`H' = hue, `S' = saturation, `B' =
     brightness)

`setcmykcolor C M Y K'
     set the color in the CMYK model (`C' = cyan, `M' = magenta, `Y' =
     yellow, `K' = black)

`setcolor C'
     set the color specified by symbolic color value `C' (see below)

`setfont S X'
     select font `S' scaled by `X' units (see below)

   The `setrgbcolor', `sethsbcolor' and `setcmykcolor' functions enable
you to select arbitrary colors in the RGB, HSB and CMYK model,
respectively. A more user-friendly, but less flexible, routine
`setcolor' is provided which allows colors to be selected from a fixed
set of symbolic constants implemented by the `Color' type. Please refer
to `graphics.q' for a list of the possible color values.

   The `setdash' function sets the dash pattern for straight and curved
line segments. The first argument of `setdash' is a list of length
values which alternately specify the lengths of the "on" and "off"
segments of the line (i.e., dashes and gaps between the dashes). This
list is cycled through by the `stroke' function. For instance, `setdash
[2,1] 0' specifies the dash pattern "2 on, 1 off, 2 on, 1 off, ...".
If the list is empty (`setdash [] 0') `stroke' produces solid lines.
The second argument of `setdash' denotes the "phase" of the dash
pattern, which is given by an offset into the pattern. E.g., `setdash
[2,3] 11' denotes the pattern "1 on, 3 off, 2 on, 3 off, 2 on, ...".

   The `setfont' function takes as its first argument a string denoting
a PostScript font name such as `"Times-Roman"' or
`"Helvetica-Oblique"'. The second argument denotes the size in units to
which the font should be scaled. For instance: `setfont "Helvetica" 10'.
(This function is implemented by a combination of the PostScript
operators `findfont', `scalefont' and `setfont'.)


File: qdoc.info,  Node: Miscellaneous Operations,  Next: DSC and EPSF Comments,  Prev: Graphics State,  Up: Graphics

11.13.7 Miscellaneous Operations
--------------------------------

`showpage'
     emit the current page

`copypage'
     like `showpage', but do not erase the contents of the current page

`flushpage'
     update the display (flush any buffered graphics)

`erasepage'
     erase the contents of the current page

`copies N'
     number of copies to be emitted with `showpage'

`psfile NAME'
     copy a PostScript file to the graphics device

`psstr S'
     convert a string to PostScript syntax

`psheader'
     output a minimal PostScript header

`ps CMD'
     output a PostScript command

   The `showpage', `copypage', `flushpage' and `erasepage' functions
have already been discussed in *Note Overview of Graphics Operations::.
The `copies' operation determines the number of copies which should be
printed when `showpage' is invoked:

     ==> copies 4 || showpage

   To submit a PostScript file to the graphics device the `psfile'
operation may be used. It takes one string argument, the name of the
file.  For instance:

     ==> psfile "foo.ps"

   The `ps' function is used to directly invoke a PostScript command.
Examples can be found in *Note Overview of Graphics Operations::. The
`psstr' function converts a string to PostScript format. It takes care
of embedded backslashes and parentheses. For instance:

     ==> writes (psstr "(silly\\) example") || writec "\n"
     (\(silly\\\) example)
     ()

   The `psheader' function is used to begin the output file with a
minimal header identifying the file as PostScript; see *Note DSC and
EPSF Comments::.


File: qdoc.info,  Node: DSC and EPSF Comments,  Prev: Miscellaneous Operations,  Up: Graphics

11.13.8 DSC and EPSF Comments
-----------------------------

Appendix G and H of [Adobe 1990] define a standard set of comments which
should be included in PostScript documents to make explicit the document
structure, and to allow inclusion of PostScript files in other
documents. These standards are known as the "document structuring
conventions" (DSC) and the "encapsulated PostScript file" (EPSF)
format. See [Adobe 1990] for a detailed discussion of the purpose of
these formats.

   The `graphics.q' script currently does not provide any specialized
operations for sending DSC and EPSF comments to the graphics device,
with the exception of the `psheader' function which outputs a minimum
header to make your printer recognize a graphics file as PostScript.
Invoke this function as follows, _before_ calling any other graphics
operation:

     ==> psheader

   You can also call the `psheader' function at initialization time
from within a script, using a line like the following:

     def INIT = psheader;

   To include other types of DSC and EPSF comments in the graphics
output, you have to specify these comments explicitly using the `ps'
function. For instance, a function writing out the necessary DSC header
comments of an EPS file may be implemented as follows:

     /* (X1,Y1) and (X2,Y2) denote the lower left and the upper right corner
        of the bounding box, respectively */
     epsf_header X1:Real Y1:Real X2:Real Y2:Real
                             = ps "%!PS-Adobe-3.0 EPSF-3.0\n" ||
                               ps ("%%BoundingBox: "++join " "
                                   [str X1, str Y1, str X2, str Y2]++"\n");

   As indicated, each comment _must_ be terminated by a newline
character. Use the `epsf_header' function in place of the `psheader'
function if you are composing an EPS file which is to be used in other
documents. E.g., when invoked as `epsf_header 5 5 105 105', the
following header information will be written to the graphics device:

     %!PS-Adobe-3.0 EPSF-3.0
     %%BoundingBox: 5 5 105 105


File: qdoc.info,  Node: Diagnostics and Error Messages,  Prev: Graphics,  Up: The Standard Library

11.14 Diagnostics and Error Messages
====================================

The `assert.q' script supplies the special form `assert' for printing
diagnostic messages:

     foo X                   = assert (X>0) || bar (1/X);

   The `assert' function verifies that the given expression evaluates
to `true', in which case it returns `()'. Otherwise it uses the `error'
function to abort evaluation after printing an error message of the
following form:

     ! Error: assertion (X) failed, value (VALUE OF X)

   Error messages are printed using the `error' operation of the
`error.q' script which prints an error message and then simply stops
evaluation using `halt'. For instance:

     hd []                   = error "hd: empty list";

   This will print an error message

     ! Error: hd: empty list

when executed, and halt evaluation.


File: qdoc.info,  Node: Clib,  Next: Q Language Grammar,  Prev: The Standard Library,  Up: Top

12 Clib
*******

`Clib' is the C interface and "system" module of the Q programming
language. As of Q 7.8, this component actually consists of two
different modules, `clib.q' which provides basic C data structures and
routines commonly used in most Q programs, and `system.q' which
contains most of the POSIX system interface. Only `clib.q' is part of
the prelude; for most system functions, you will have to explicitly
import `system.q' in your programs. In the sections below, we will
always indicate whether the `system.q' module is needed for the
described operations.

   In difference to the other standard library modules, `clib' and
`system' are "external" modules, i.e., most functions are actually
implemented in C (cf. *Note C Language Interface::). Together, `clib'
and `system' provide additional string operations, extended file
functions, C-style formatted I/O, low-level and binary I/O, an
interface to various system functions, POSIX thread functions,
expression references, time functions, internationalization support,
filename globbing and regular expression matching, additional integer
functions from the GMP library, and, last but not least, efficient C
replacements for some common standard library list and string processing
functions.

   Even if you do not use the extra functionality provided by these
modules, you will benefit from the replacement operations (which are in
`clib' and thus included in the prelude), which considerably speed up
basic list and string processing, sometimes by several orders of
magnitude.

   NOTE: Not all of the following operations are implemented on all
systems. The UNIX-specific operations are marked with the symbol `(U)'
in the `clib.q' and `system.q' scripts. Only a portable subset of the
UNIX system interface is provided, which encompasses the most essential
operations found on many recent UNIX (and other POSIX) systems, as
described by the ANSI C and POSIX standards as well as the Single UNIX
Specification (SUS). These operations are also available on Linux and
OSX systems.

* Menu:

* Manifest Constants::
* Additional String Functions::
* Byte Strings::
* Extended File Functions::
* C-Style Formatted I/O::
* File and Directory Functions::
* Process Control::
* Low-Level I/O::
* Terminal Operations::
* Readline Interface::
* System Information::
* Sockets::
* POSIX Threads::
* Expression References::
* Time Functions::
* Internationalization::
* Filename Globbing::
* Regular Expression Matching::
* Additional Integer Functions::
* C Replacements for Common Standard Library Functions::


File: qdoc.info,  Node: Manifest Constants,  Next: Additional String Functions,  Prev: Clib,  Up: Clib

12.1 Manifest Constants
=======================

`Clib' defines an abundance of symbolic values for use with various
system functions. Most of these can be found in `system.q', but a few
constants related to the memory sizes of various basic C data types and
the `fseek' and `setvbuf' operations can also be found in `clib.q'.

   System constants actually vary from system to system; only the most
common values are provided as global variables here. The variables are
declared `const' (read-only) and are initialized at startup time. Flag
values can be combined using bitwise logical operations as usual. A
complete list of the variables can be found at the beginning of the
`clib.q' and `system.q' scripts. Flag values which are unavailable on
the host system will be set to zero, other undefined values to -1. Thus
undefined values will generally have no effect or cause the
corresponding operations to fail.


File: qdoc.info,  Node: Additional String Functions,  Next: Byte Strings,  Prev: Manifest Constants,  Up: Clib

12.2 Additional String Functions
================================

These functions provide an interface to some familiar character routines
from the C library. They can all be found in `clib.q' and are thus in
the standard prelude.

*Character predicates:* These work exactly like the corresponding C
library routines, except that they work with arbitrary Unicode, not
just ASCII, characters, provided that the interpreter has been built
with Unicode support.

     public extern islower C, isupper C, isalpha C, isdigit C, isxdigit C,
       isalnum C, ispunct C, isspace C, isgraph C, isprint C, iscntrl C,
       isascii C;

*String conversion:* Convert a string to lower- or uppercase (like the
corresponding C functions, but work on arbitrary Unicode strings, not
just on single ASCII characters).

     public extern tolower S, toupper S;

Examples
--------

Count the number of alphanumeric characters in a text:

     ==> #filter isalnum (chars "The little brown fox.\n")
     17

   Convert a string to uppercase:

     ==> toupper "The little brown fox.\n"
     "THE LITTLE BROWN FOX.\n"


File: qdoc.info,  Node: Byte Strings,  Next: Extended File Functions,  Prev: Additional String Functions,  Up: Clib

12.3 Byte Strings
=================

The following type represents unstructured binary data implemented as C
byte vectors. This data structure is used by the low-level I/O functions
and other system functions which operate on binary data. The `ByteStr'
type itself and its operations are implemented in `clib.q' and thus
included in the prelude.

     public extern type ByteStr;

     public isbytestr B;                     // check for byte strings

   Byte strings are like ordinary character strings, but they do not
have a printable representation, and they may include zero bytes.
(Recall that a zero byte in a character string terminates the string.)
They can be used to encode arbitrary binary data such as C vectors and
structures. The `bytestr' function can be used to construct byte
strings from integers, floating point numbers, string values or lists
of unsigned byte values:

     public extern bytestr X;                // create a byte string

   The `X' argument denotes the data to be encoded and can be either a
list of byte values (unsigned integers in the range from 0 to 255), or
an atomic data object, i.e., an integer, floating point number or string
constant. In the latter case, the argument can also have the form
`(X,SIZE)' indicating the desired byte size of the object; otherwise a
reasonable default size is chosen. If the specified size differs from
the actual size of `X', the result is zero-padded or truncated
accordingly. Integer values are encoded in the host byte order, with
the least significant GMP limb first; negative integers are represented
in 2's complement. Floating point values are encoded using double
precision by default or if the byte count is sufficient (i.e., at least
8 on most systems), and using single precision otherwise. Strings are
by default encoded in the system encoding, but you can also specify the
desired target encoding as `(X,CODESET)' (or `(X,CODESET,SIZE)' if you
also need to specify a byte size), where `CODESET' is a string denoting
the target encoding.

   Like ordinary character strings, byte strings can be concatenated,
size-measured, indexed, sliced and compared lexicographically. Moreover,
a byte string can be converted back to a (multiprecision) integer,
floating point number, string value, or a list of byte values. (When
converting back to a string you can specify the source encoding as in
bstr `(B,CODESET)', otherwise the system encoding is assumed.) For
these purposes the following operations are provided.

     public extern bcat Bs;                  // concatenate list of byte strings
     public extern bsize B;                  // byte size of B
     public extern byte I B;                 // Ith byte of B
     public extern bsub B I J;               // slice of B (bytes I..J)
     public extern bcmp M1 M2;               // compare M1 and M2

     public extern bint B;                   // convert to unsigned integer
     public extern bfloat B;                 // convert to floating point number
     public extern bstr B;                   // convert to string
     public bytes B;                         // convert to list
     public ::list B;                        // dito

   You can use the `bytes' function to convert a byte string to a list
of byte values; the `list' function is overloaded to provide the same
functionality. These functions are defined as follows:

     bytes B:ByteStr                         = map (B!) [0..#B-1];
     list B:ByteStr                          = bytes B;

   For convenience, the common string operators and the `sub' function
are overloaded to work on byte strings as well. Thus `#B' returns the
size of `B' (the number of bytes it contains) and `B!I' the `I'th byte
of `B'. `B1++B2' concatenates `B1' and `B2', `sub B I J' returns the
slice from byte `I' to `J', and the relational operators `=', `<', `>'
etc. can be used to compare byte strings lexicographically. These
operations are all implemented in terms of the functions listed above.

Byte Strings as Mutable C Vectors
---------------------------------

As of Q 7.11, `clib' supports a number of additional operations which
allow you to treat byte strings as mutable C vectors of signed/unsigned
8/16/32 bit integers or single/double precision floating point numbers.
The following functions provide read/write access to elements and
slices of such C vectors:

     public extern get_int8 B I, get_int16 B I, get_int32 B I;
     public extern get_uint8 B I, get_uint16 B I, get_uint32 B I;
     public extern get_float B I, get_double B I;

     public extern put_int8 B I X, put_int16 B I X, put_int32 B I X;
     public extern put_uint8 B I X, put_uint16 B I X, put_uint32 B I X;
     public extern put_float B I X, put_double B I X;

   Note that the given index argument `I' is interpreted relative to
the corresponding element type. Thus, e.g., `get_int32 B I' returns the
`I'th 32 bit integer rather than the integer at byte offset `I'. Also
note that integer arguments must fit into machine integers, otherwise
these operations will fail. Integers passed for floating point
arguments will be coerced to floating point values automatically.

   For the `get_xxx' functions, the index parameter may also be a pair
`(I,J)' to return a slice of the given byte string instead of a single
element (this works like `sub'/`bsub', but interprets indices relative
to the element type). The `put_xxx' functions also accept a byte string
instead of an element as input, and will then overwrite the
corresponding slice of the target byte string `B' with the given source
byte string `X'. Similar to `sub'/`bsub', these variations of
`get_xxx'/`put_xxx' are "safe" in that they automatically adjust the
given indices to fit within the bounds of the target byte string.

   Moreover, the following convenience functions are provided to convert
between byte strings and lists of integer/floating point elements.

     public extern int8_list B, int16_list B, int32_list B;
     public extern uint8_list B, uint16_list B, uint32_list B;
     public extern float_list B, double_list B;

     public extern int8_vect Xs, int16_vect Xs, int32_vect Xs;
     public extern uint8_vect Xs, uint16_vect Xs, uint32_vect Xs;
     public extern float_vect Xs, double_vect Xs;

Examples
--------

Encode an integer as a byte string, take a look at its individual bytes,
and convert the byte string back to an integer:

     ==> hex

     ==> def B = bytestr 0x01020304; bytes B; bint B
     [0x4,0x3,0x2,0x1]
     0x1020304

   (Note that this result was obtained on a little-endian system, hence
the least significant byte `0x04' comes first in the byte list.)

   Negative integers are correctly encoded in 2's complement:

     ==> def B = bytestr (-2); bytes B; bint B
     [0xfe,0xff,0xff,0xff]
     0xfffffffe

   To work with these binary representations you must be aware of the
way GMP represents multiprecision integers. In particular, note that the
default size of an integer is always a multiple (at least one) of GMP's
limb size which is usually 4 or 8 bytes depending on the host system's
default `long' integer type. The actual limb size can be determined as
follows:

     ==> #bytes (bytestr 0)

   In order to get integers of arbitrary sizes, an explicit `SIZE'
argument may be used. For instance, here is how we encode small (1 or 2
byte) integers:

     ==> bytes (bytestr (0x01,1)); bytes (bytestr (0x0102,2))
     [0x1]
     [0x2,0x1]

   The host system's byte sizes of various atomic C types can be
determined with symbolic values declared at the beginning of `clib.q',
such as `SIZEOF_CHAR', `SIZEOF_SHORT', `SIZEOF_LONG', `SIZEOF_FLOAT'
and `SIZEOF_DOUBLE'.

   Another fact worth mentioning is that even on big-endian systems,
integers are always encoded with the "least significant limb" first.
So, for instance, given that the limb size is 4, as in the above
examples, the 2-limb integer `0x0102030405060708' consists of bytes
`0x8 0x7 0x6 0x5 0x4 0x3 0x2 0x1' on a little-endian system, in that
order, whereas the byte order on a big-endian system is `0x5 0x6 0x7
0x8 0x1 0x2 0x3 0x4'.

   Here is how we can quickly check the byte order of the host system:

     ==> hd (bytes (bytestr 1))

   This expression returns 1 on a little-endian system and zero
otherwise.

   As long as an integer does not exceed the machine's word size (which
usually matches the limb size), we can simply convert between big-endian
and little-endian representation by reversing the byte list:

     ==> bytestr (reverse (bytes B))

   Floating point values can be encoded either in double or single
precision, depending on the `SIZE' argument. The default size is double
precision (usually 8 bytes).

     ==> bfloat (bytestr (1/3)); bfloat (bytestr (1/3,SIZEOF_FLOAT))
     0.333333333333333
     0.333333343267441

   The default size of the encoding of a character string is the byte
size of the string in the target encoding (the system encoding by
default). If an explicit size is given, the string is zero-padded or
truncated if necessary. The following example will work with any system
encoding based on 7 bit ASCII (like Latin1, UTF-8 or ASCII itself):

     ==> dec

     ==> def S1 = bytestr "ABC", S2 = bytestr ("ABC",2), S3 = bytestr ("ABC",5)

     ==> bytes S1; bytes S2; bytes S3
     [65,66,67]
     [65,66]
     [65,66,67,0,0]

     ==> bstr S1; bstr S2; bstr S3
     "ABC"
     "AB"
     "ABC"

   By combining elements like the ones above, and including appropriate
"tagging" information, more complex data structures can be represented
as binary data as well. For this purpose, the byte strings of the tags
and the data elements can be concatenated with `bcat' or the `++'
operator. This is useful, in particular, for compact storage of objects
in files. Moreover, some system functions involve binary data which
might represent C structures and/or vectors. Such data can be assembled
from the constituent parts by simply concatenating them. For instance,
consider the following C struct:

     struct { char foo[108]; short bar; int baz; };

   A value of this type, say `{"Hello, world.", 4711, 123456}', can
then be encoded as follows:

     ==> bytestr ("Hello, world.",108) ++ bytestr (4711,SIZEOF_SHORT) ++ \
     bytestr (123456,SIZEOF_INT)

   Similarly, a list of integers can be converted to a corresponding C
vector as follows:

     ==> bcat (map bytestr [1..100])

   When encoding such C structures you must also consider alignment
issues. For instance, most C compilers will align non-byte data at even
addresses.

   In order to facilitate the handling of C vectors of integers and
floating point values, as of Q 7.11 `clib' offers a number of
specialized operations which provide direct read/write access to
elements and slices of numeric vectors, and allow you to convert between
C vectors and Q lists of integer or floating point values. These
operations are all implemented directly in C and will usually be much
more efficient for manipulating numeric C vectors than the basic
byte-oriented functions. Moreover, they allow you to modify the elements
of a C vector in a direct fashion, turning byte strings into a mutable
data structure.

   Different operations are provided to handle vectors of signed or
unsigned 8/16/32 bit (machine) integers, as well as single (32 bit) and
double precision (64 bit) floating point numbers. For instance:

     ==> def B = uint32_vect [100..110]

     ==> uint32_list B
     [100,101,102,103,104,105,106,107,108,109,110]

     ==> get_uint32 B 1
     101

     ==> put_uint32 B 1 0xffffffff
     ()

     ==> uint32_list B
     [100,4294967295,102,103,104,105,106,107,108,109,110]

   Note that, because these C vectors are just normal byte strings, you
can freely convert between different representations of the numeric
data. E.g.:

     ==> take 12 $ int8_list B
     [100,0,0,0,-1,-1,-1,-1,102,0,0,0]

   Entire slices of byte strings can be retrieved and overwritten as
well. Note that, as with `sub', the indices are adjusted automatically
to stay within the bounds of the target vector.

     ==> put_uint32 B (-2) (uint32_vect [90..94])
     ()

     ==> uint32_list B
     [92,93,94,103,104,105,106,107,108,109,110]

     ==> uint32_list $ get_uint32 B (-2,3)
     [92,93,94,103]

     ==> uint32_list $ get_uint32 B (8,100)
     [108,109,110]


File: qdoc.info,  Node: Extended File Functions,  Next: C-Style Formatted I/O,  Prev: Byte Strings,  Up: Clib

12.4 Extended File Functions
============================

`Clib' provides the following enhanced and additional file functions:

     public extern ::fopen NAME MODE, fdopen FD MODE, freopen NAME MODE F;
     public extern fileno F;
     public extern setvbuf F MODE;
     public extern fconv F CODESET;
     public extern tmpnam, tmpfile;
     public extern ftell F, fseek F POS WHENCE;
     public rewind F;
     public extern gets, fgets F;
     public extern fget F;
     public extern ungetc C, fungetc F C;

   These are all defined in `clib.q' and thus included in the prelude.

   The `fopen' version of `clib' handles the `+' flag in mode strings,
thus enabling you to open files for both reading and writing. The mode
`"r+"' opens an existing file for both reading and writing; the initial
file contents are unchanged, and both the input and output file
pointers are positioned at the beginning of the file. The `"w+"' mode
creates a new file, or truncates it to zero size if it already exists,
and positions the file pointers at the beginning of the file. The
`"a+"' mode appends to an existing file (or creates a new one); the
initial file pointer is set at the beginning of the file for reading,
and at the end of the file for writing. All these modes also work in
combination with the `b' (binary file) flag.

   The `freopen' function is like `fopen', but reopens an existing file
object on another file. Just as in C programming, the main purpose of
this operation is to enable the user to redirect the standard I/O
streams associated with the interpreter process (available in the
interpreter by means of the `INPUT', `OUTPUT' and `ERROR' variables).

   The `fdopen' function opens a new file object on a given file
descriptor, given that the mode is compatible. Conversely, the `fileno'
function returns the file descriptor of a file object. (See also the
functions for direct file descriptor manipulation in *Note Low-Level
I/O::.)

   The `setvbuf' function sets the buffering mode for a file (`IONBF' =
no buffering, `IOLBF' = line buffering, `IOFBF' = full buffering). This
operation should be invoked right after the file has been opened,
_before_ any I/O operations are performed.

   The `fconv' function sets the encoding of a file. By default, Q's
built-in I/O operations as well as `clib''s string I/O functions assume
the system encoding, and convert between this encoding and the internal
UTF-8 string representation as needed. If a text file uses an encoding
different from the system encoding, you can use the `fconv' function to
set the desired encoding. `CODESET' must be a string denoting a valid
encoding name for the `iconv' function (see also *Note
Internationalization::, below). This affects all subsequent text
read/write operations on the file. (This operation only works for
Unicode-capable systems which have `iconv' installed. Also note that
this function is only available for Q 7.0 and later.)

   The `tmpnam' and `tmpfile' functions work just like the
corresponding C routines: `tmpnam' returns a unique name for a temporary
file, and `tmpfile' constructs a temporary file opened in `"w+b"' mode,
which will be deleted automatically when it is closed. See the
`tmpnam'(3) and `tmpfile'(3) manual pages for details.

   The `ftell'/`fseek' functions are used for file positioning. The
`ftell' function returns the current file position, while `fseek'
function positions the file at the given position. The `rewind'
function provides a convenient shorthand for repositioning the file at
the beginning. These operations work just like the corresponding C
functions. The `WHENCE' argument of `fseek' determines how the `POS'
argument is to be interpreted; it can be either `SEEK_SET' (`POS' is
relative to the beginning of the file, i.e., an absolute position),
`SEEK_CUR' (`POS' is relative to the current position) or `SEEK_END'
(`POS' is relative to the end of the file). In the latter two cases
`POS' can also be negative.

   Portability Notes:

   * According to the ISO C specification, you should use `fflush' or
     `fseek' before switching between reading and writing on a file
     opened with the `+' flag.

   * On non-UNIX systems, `ftell' and `fseek' might only work reliably
     if the file is opened in binary mode (`b' flag).

   The `gets'/`fgets' functions work like the C `fgets' function, i.e.,
they read a line from standard input or the given file _including_ the
trailing newline, if any. The `fget' function reads an entire file at
once and returns it as a string. The `ungetc'/`fungetc' functions push
back a single character on standard input or the given input file, like
the C `ungetc' function. The C library only guarantees that pushing
back a single ASCII character will work, so the result of pushing back
multiple or multibyte characters is implementation-dependent.

   Moreover, the following additional aliases are provided for C
aficionados:

     public ::readc as getc, ::freadc F as fgetc;
     public ::writes S as puts, ::fwrites F S as fputs;
     public ::writec C as putc, ::fwritec F C as fputc;

Examples
--------

Open a new file for both reading and writing:

     ==> def F = fopen "test" "w+"

Write a string to the file:

     ==> fwrites F "The little brown fox.\n"
     ()

Current position is behind written string (at end-of-file):

     ==> ftell F
     22

Rewind (go to the beginning of the file):

     ==> rewind F
     ()

Read back the string we've written before:

     ==> fgets F
     "The little brown fox.\n"

Check that we're again at end-of-file:

     ==> feof F
     true

Output another string:

     ==> fwrites F "The second line.\n"
     ()

Position behind the first string:

     ==> fseek F 22 SEEK_SET
     ()

Reread the second string:

     ==> fgets F
     "The second line.\n"

And here's how to read an entire text file at once:

     ==> def T = fget (fopen "clib.q" "r")

To quickly compute a 32 bit checksum of the file:

     ==> sum (bytes (bytestr T)) mod 0x100000000
     3937166

Finally, let's split the text into lines and add line numbers using
`sprintf' (see *Note C-Style Formatted I/O::):

     ==> def L = split "\n" T

     ==> def L = map (sprintf "%3d: %s\n") (zip [1..#L] L)

     ==> do writes (take 5 L)
       1:
       2: /* clib.q: Q's system module */
       3:
       4: /* This file is part of the Q programming system.
       5:
     ()


File: qdoc.info,  Node: C-Style Formatted I/O,  Next: File and Directory Functions,  Prev: Extended File Functions,  Up: Clib

12.5 C-Style Formatted I/O
==========================

These functions provide an interface to the C `printf' and `scanf'
routines. They are all defined in `clib.q' and thus included in the
prelude.

     public extern printf FORMAT ARGS, fprintf F FORMAT ARGS,
       sprintf FORMAT ARGS;
     public extern scanf FORMAT, fscanf F FORMAT, sscanf S FORMAT;

   Arguments to the `printf' routines and the results of the `scanf'
routines are generally encoded as tuples or single non-tuple values (if
only one item is read/written).

   All the usual conversions and flags of C `printf'/`scanf' are
supported, except `%p' (pointer conversion). The basic `h' and `l'
length modifiers are also understood, but not the fancy ISO C99
extensions like `ll' or `hh', or `l' modifiers on characters and
strings. Two further unsupported features of the `printf' functions are
the `%n' (number of written characters) conversion and explicit
argument indexing (`m$'); thus all arguments have to be in the same
order as specified in the `printf' format string. The `%n' conversion
_is_ implemented for the `scanf' functions, though.

   As these functions are simply wrappers for the corresponding C
functions, integer conversions are generally limited to values which
fit into machine integers. To handle integers of arbitrary sizes, you
might treat them as strings (`%s') in the format string and do the
actual conversion manually with `val' or `str'.

   Seasoned C programmers will appreciate that the wrapper functions
provided here are _safe_ in that they check their arguments and prevent
buffer overflows, so they should never crash your program with a
segfault. To these ends, if a `%s' or `%[...]' conversion without
maximum field width is used with `scanf', the field width will
effectively be limited to some (large) value chosen by the
implementation.

Examples
--------

See the `printf'(3) and `scanf'(3) manual pages for a description of
the format string syntax. Some basic examples follow (`<CR>' indicates
that you hit the carriage return key to terminate a line):

     ==> printf "%d\n" 99
     99
     ()

     ==> printf "%d\n" (99)
     99
     ()

     ==> printf "%s %s %d\n" ("foo","bar",99)
     foo bar 99
     ()

     ==> scanf "%d"
     99<CR>
     99

     ==> scanf "%s %s %d"
     foo bar 99<CR>
     ("foo","bar",99)

   As indicated, multiple values are denoted as tuples, and the `printf'
function accepts both a single value or a one-tuple for a single
conversion. The `scanf' function always returns a single, non-tuple
value if only a single conversion is specified. Zero items are
represented using the empty tuple. Note that you always have to supply
the `ARGS' argument of `printf', thus you specify an empty tuple if
there are no output conversions:

     ==> printf "foo\n" ()
     foo
     ()

   The `scanf' function also returns an empty tuple if no input items
are converted. For instance (as usual, using the `*' flag with a `scanf'
conversion suppresses the corresponding input item):

     ==> scanf "%*s"
     foo<CR>
     ()

   Note that while `scanf' for most conversions skips an arbitrary
amount of leading whitespace, the trailing whitespace character at which
a conversion stops is _not_ discarded by `scanf'. You can notice this
if you invoke, e.g., `readc' afterwards:

     ==> scanf "%s %d"; readc
     foo 99<CR>
     ("foo",99)
     "\n"

   If you really have to skip the trailing whitespace character, you
can do this with a suppressed character conversion, e.g.:

     ==> scanf "%s %d%*c"; writes "input: "||reads
     foo 99<CR>
     ("foo",99)
     input: <reads function waiting for input here>

   The `fprintf'/`fscanf' functions work analogously, but are used when
writing or reading an arbitrary file instead of standard output or
input. For instance:

     ==> var msg = "You're not supposed to do that!"

     ==> fprintf ERROR "Error: %s\n" msg
     Error: You're not supposed to do that!
     ()

   The `sprintf' function returns the formatted text as a string
instead of writing it to a file:

     ==> sprintf "%s %s %d\n" ("foo","bar",99)
     "foo bar 99\n"

   Likewise, `sscanf' takes its input from a string:

     ==> sscanf "foo bar 99\n" "%s %s %d"
     ("foo","bar",99)

   The `%n' conversion is especially useful with `sscanf', since it
allows you to determine the number of characters which were actually
consumed:

     ==> sscanf "foo bar 99 *** extra text here ***\n" "%s %s %d%n"
     ("foo","bar",99,10)

   You might then use the character count, e.g., to check whether the
input format matched the entire string, or whether there remains some
text to be processed.

   Some remarks about the role of the length modifiers `h' and `l' are
in order. Just as with the C `scanf' routines, you need the `l'
modifier to read a double precision value; a simple `%f' will only read
single precision number:

     ==> scanf "%f"
     1e100<CR>
     inf

     ==> scanf "%lf"
     1e100<CR>
     1e+100

   The `printf' functions, however, always print double precision
numbers, so the `l' modifier is not needed:

     ==> sprintf "%g" 1e100
     "1e+100"

   For the integer conversions, the `h' and `l' modifiers denote short
(usually 2 byte) and long (usually 4 byte) integer values. If the
modifier is omitted, the default integer type is used (this usually is
the same as `long', but your mileage may vary).

   As already indicated, the `printf' and `scanf' routines are limited
to machine integer sizes. Thus a `scanf' integer conversion will always
return a short or long integer value, depending on the length modifier
used. If a `printf' integer conversion is applied to a "big" integer
value, only the least significant bytes of the value are printed, as if
the printed number (represented in 2's complement if negative) had been
cast to the corresponding integer type in C. Thus the printed result
will be consistent with C `printf' output under all circumstances. For
instance:

     ==> def N = 0xffff70008000 // big number

     ==> printf "%hu %lu\n" (N,N)
     32768 1879080960
     ()

     ==> printf "%hd %ld\n" (N,N)
     -32768 1879080960
     ()

   To correctly print a big integer value, you can convert it manually
with Q's built-in `str' function, then print the value using a `%s'
conversion:

     ==> printf "%s\n" (str 1234567812345678)
     1234567812345678
     ()

   Similarly, you can read a big integer value by converting it as a
string, and then apply the val builtin.

     ==> val (scanf "%s")
     1234567812345678<CR>
     1234567812345678

   Here you might use the `%[...]' conversion to ensure that the number
is in proper format (the initial blank is needed here to skip any
leading whitespace):

     ==> val (scanf " %[0-9-]")
     -1234567812345678
     -1234567812345678

   On output, the integer and floating point conversions can all be used
with either integer or floating point arguments; integers will be
converted to floating point values and vice versa if necessary:

     ==> printf "An integer: %d\n" 99.9
     An integer: 99
     ()

     ==> printf "A floating point value: %e\n" 99
     A floating point value: 9.900000e+01
     ()


File: qdoc.info,  Node: File and Directory Functions,  Next: Process Control,  Prev: C-Style Formatted I/O,  Up: Clib

12.6 File and Directory Functions
=================================

These functions provide the same functionality as their C counterparts.
They are all to be found in `system.q' and thus you have to explicitly
import the `system' module to use them.

     public extern rename OLD NEW;           // rename a file
     public extern unlink NAME;              // delete a file
     public extern truncate NAME LEN;        // truncate a file (U)
     public extern getcwd, chdir NAME;       // get/set the working directory
     public extern mkdir NAME MODE;          // create a new directory
     public extern rmdir NAME;               // remove a directory
     public extern readdir NAME;             // list the files in a directory
     public extern link OLD NEW;             // create a hard link (U)
     public extern symlink OLD NEW;          // create a symbolic link (U)
     public extern readlink NAME;            // read a symbolic link
     public extern mkfifo NAME MODE;         // create a named pipe (U)
     public extern access NAME MODE;         // test access mode
     public extern chmod NAME MODE;          // set the file mode
     public extern chown NAME MODE UID GID;  // set file ownership (U)
     public extern lchown NAME MODE UID GID; // set link ownership (U)
     public extern utime NAME TIMES;         // set the file times
     public extern umask N;                  // set/get file creation mask
     public extern stat NAME, lstat NAME;    // file and link information

   The `stat'/`lstat' functions return a tuple consisting of the
commonly available fields of the C `stat' struct, see `stat'(2). For
your convenience, the following mnemonic functions are provided for
accessing the different components:

     public st_dev STAT, st_ino STAT, st_mode STAT, st_nlink STAT,
       st_uid STAT, st_gid STAT, st_rdev STAT, st_size STAT, st_atime STAT,
       st_mtime STAT, st_ctime STAT;

Examples
--------

These all need the `system' module, so you have to import it in the
interpreter to make the following examples work. E.g.:

     ==> import system

   With that out of the way, let's play around with some of these
functions:

     ==> mkdir "tmp" 0777||chdir "tmp"||mkfifo "foo" 0666||\
     rename "foo" "bar"||unlink "bar"||chdir ".."||rmdir "tmp"
     ()

   (Create a `tmp' subdirectory, change to it, create a new FIFO
special file, rename that file, delete it, change back to the original
directory, and remove the `tmp' directory. All with a single expression
which realizes identity.)

   Now for something more useful. We can retrieve the current umask
while setting it to zero, and then reset it to the original value as
follows:

     ==> def U = umask 0; oct; umask U || U; dec
     022

   List the files in the current directory:

     ==> readdir "."
     [".","..","Makefile","givertcap","clib.c","clib.q","Makefile.am",
     "Makefile.in","README-Clib","examples","Makefile.mingw"]

   Get the size of a file:

     ==> st_size (stat "README-Clib")
     355


File: qdoc.info,  Node: Process Control,  Next: Low-Level I/O,  Prev: File and Directory Functions,  Up: Clib

12.7 Process Control
====================

With the notable exception of the `exit' function which is included in
`clib' (and thus in the prelude), all the following functions need an
explicit import of the `system' module.

   The `system' function returns the status code of the command if
execution was successful, and fails otherwise:

     public extern system CMD;               // exec command using the shell

   `Clib' also provides the usual UNIX process creation and management
routines. Most of these really require a UNIX system; no attempt is made
to emulate operations like `fork' on systems where they are not
implemented. Thus the only process operations which currently work under
Windows are `system', `exec', `spawn', `_spawn', `exit' and `getpid'.

     public extern fork;                     // fork a child process (U)
     public extern exec PROG ARGS;           // execute program
     public extern spawn PROG ARGS;          // execute program in child process
     public extern _spawn MODE PROG ARGS;    // execute child with options
     public extern nice INC;                 // change nice value (U)
     public extern exit N;                   // exit process with given exit code
     public extern pause;                    // pause until a signal occurs (U)
     public extern raise SIG;                // raise signal in current process
     public extern kill SIG PID;             // send signal to given process (U)
     public extern getpid;                   // current process id
     public extern getppid;                  // parent's process id (U)
     public extern wait;                     // wait for any child process (U)
     public extern waitpid PID OPTIONS;      // wait for given child process (U)

   All these operations are simply wrappers for the corresponding C
library routines. Note, however, that the `kill' function takes the
signal to send as its _first_ argument, which makes it easier to use
partial applications of the function, e.g., to iterate a kill operation
over a list of process numbers (as in `do (kill SIGTERM) PIDs').

   The `exec' function performs a path search like the C
`execlp'/`execvp' function; the parameters for the program are given as
a string list `ARGS', and as usual the first argument should repeat the
program file name. This function never returns unless it fails. The
`spawn' and `_spawn' operations are provided to accommodate Windows'
lack of `fork' and `wait'; these functions work on both UNIX and
Windows. The `spawn' function works like `exec', but runs the program
in a new child process. It returns the new process id (actually the
process handle under Windows). The `_spawn' function is like `spawn',
but accepts an additional `MODE' parameter which determines how the
child is to be executed, either `P_WAIT' (wait for the child, return
its exit status), `P_NOWAIT' (do not wait for the child, same as
`spawn'), `P_OVERLAY' (replace the current image with the new process,
same as `exec') and `P_DETACH' (run the new process in the background).
(Note that the `P_DETACH' option is ignored on UNIX systems; the
correct way to code a "daemon" on UNIX is shown in the examples section
below.)

   On UNIX, the following routines are provided to interpret the status
code returned by the `system', `_spawn', `wait' and `waitpid' functions:

     public extern isactive STATUS;
     // process is active
     public extern isexited STATUS, exitstatus STATUS;
     // process has exited normally, get its exit code
     public extern issignaled STATUS, termsig STATUS;
     // process was terminated by signal, get the signal number
     public extern isstopped STATUS, stopsig STATUS;
     // process was stopped by signal, get the signal number

   For more information about the process functions, we refer the
reader to the corresponding UNIX manual pages.

   Operations to access the process environment are also implemented.
The `getenv' function fails if the given variable is not set in the
environment; this lets you distinguish this error condition from a
defined variable with empty value. The `setenv' function overwrites an
existing definition of the given variable:

     public extern getenv NAME, setenv NAME VAL;
     // get/set environment variables

   On UNIX, the following operations provide access to process user and
group information, as well as process groups and sessions. Not all
operations may be implemented on all UNIX flavours. Please see the UNIX
manual for a description of these functions.

     /* User/group-related functions (U). */

     public extern setuid UID, setgid GID;   // set user/group id of process
     public extern seteuid UID, setegid GID; // set effective user/group id
     public extern setreuid RUID EUID, setregid RGID EGID;
                                             // set real and effective ids
     public extern getuid, geteuid;          // get real/effective user id
     public extern getgid, getegid;          // get real/effective group id
     public extern getlogin;                 // get real login name

     // get/set supplementary group ids of current process
     public extern getgroups, setgroups GIDS;

     /* Session-related routines (U). */

     public extern getpgid PID, setpgid PID PGID;    // get and set process group
     public extern getpgrp, setpgrp;                 // dito, for calling process
     public extern getsid PID;                       // get session id of process
     public extern setsid;                           // create a new session

Examples
--------

Invoke the `system' function to execute a shell command:

     ==> import system

     ==> system "ls -l"

   You can also run the program directly with the `spawn' function:

     ==> spawn "ls" ["ls","-l"]

   Get and set an environment variable:

     ==> getenv "HOME"
     "/home/ag"

     ==> getenv "MYVAR" // variable is undefined
     getenv "MYVAR"

     ==> setenv "MYVAR" "foo bar"
     ()

     ==> getenv "MYVAR"
     "foo bar"

   Here are some examples demonstrating the use of named pipes and the
process functions on UNIX systems. The `mkfifo' function allows the
creation of so-called "FIFO special files" a.k.a. "named pipes", which
provide a simple inter-process communication facility.

   For instance, create a named pipe as follows:

     ==> mkfifo "pipe" 0666
     ()

   You can then open the writeable end of the pipe:

     ==> def OUT = fopen "pipe" "w"

   Note that this call blocks until the input side of the pipe has been
opened. For this purpose, start another instance of the interpreter
(e.g., in another `xterm'), and from there open the pipe for reading:

     ==> def IN = fopen "pipe" "r"

   Both `fopen' calls should now have finished, and you can write
something to the output end of the pipe in the first instance of the
interpreter:

     ==> fwrites OUT "Hello, there!\n"

   Go to the other interpreter instance, and read back the string from
there:

     ==> freads IN
     "Hello, there!"

   As usual, each end of the pipe is closed as soon as the
corresponding file object is no longer accessible. When you close the
writeable end of the pipe using, e.g., `undef OUT' in the first
instance of the interpreter, the input side of the pipe will reach
end-of-file, and thus `feof IN' will become `true'. After closing the
pipe also on the input side, you can remove the FIFO special file with
the `unlink' function.

   You can also use named pipes to set up a communication channel to
child processes created with `fork'. For instance:

     import system;
     def NAME = tmpnam;
     def PIPE = mkfifo NAME 0666;
     def MSG  = "Hello there!\n";

     test     = printf "Parent writes: %s" MSG ||
                fwrites (fopen NAME "w") MSG ||
                writes "Parent waits for child ...\n" ||
                printf "Parent: child has exited with code %d\n" wait
                    if fork > 0;
              = printf "Child reads: %s\n" (freads (fopen NAME "r")) ||
                writes "Child exiting ...\n" || exit 0
                    otherwise;


     ==> test
     Parent writes: Hello there!
     Parent waits for child ...
     Child reads: Hello there!
     Child exiting ...
     Parent: child has exited with code 0
     ()

     ==> unlink NAME
     ()

   Another method for accomplishing this with anonymous pipes is
discussed in *Note Low-Level I/O::.

   On UNIX, it also possible to implement "daemons", i.e., processes
which place themselves in the background and continue to run even when
you log out. The following little script shows how to do this.

     import system;

     /* Becoming a daemon is easy: Just fork, have the parent exit, and call setsid
        in the child to start a new session. The new process becomes a child of the
        init process and has no controlling terminal. Thus it keeps running even if
        you log out, until it gets killed or the system shuts down. */

     daemon          = setsid || main if fork = 0;
                     = exit 0 otherwise;

     /* The main code of the daemon then closes file descriptors inherited by the
        parent and starts executing. In this example we just open a logfile and
        start logging messages in regular intervals. We also handle the condition
        that we are terminated by a signal. */

     main            = do close [0,1,2] || log F "daemon started" ||
                       do (trap 1) [SIGINT, SIGTERM, SIGHUP, SIGQUIT] ||
                       catch (sig F) (loop F) where F:File = fopen "log" "w";
                     = perror "daemon" || exit 1 otherwise;

     sig F (syserr SIG)
                     = log F (sprintf "daemon stopped by signal %d" (-SIG)) ||
                       exit 0;

     loop F          = sleep 5 || log F "daemon still alive" || loop F;

     log F MSG       = fprintf F "%s at %s" (MSG, ctime time) || fflush F;


File: qdoc.info,  Node: Low-Level I/O,  Next: Terminal Operations,  Prev: Process Control,  Up: Clib

12.8 Low-Level I/O
==================

These functions provide operations for direct manipulation of files on
the file descriptor level. They are all in the `system' module. For a
closer description of the following operations we refer the reader to
the corresponding UNIX manual pages.

     public extern open NAME FLAGS MODE;     // create a new descriptor
     public extern close FD;                 // close a descriptor
     public extern dup FD, dup2 OLDFD NEWFD; // duplicate a descriptor
     public extern pipe;                     // create an unnamed pipe
     public extern fstat FD;                 // stat descriptor
     public extern fchdir FD;                // change directory (U)
     public extern fchmod FD MODE;           // change file mode (U)
     public extern fchown FD UID GID;        // set file ownership (U)
     public extern ftruncate FD LEN;         // truncate a file (U)
     public extern fsync FD, fdatasync FD;   // sync the given file (U)

   The following operations can be used on both file descriptors and
file objects. They read and write binary data represented as byte
strings (see *Note Byte Strings::), providing an interface to the
system's `read'/`write'(2) and `fread'/`fwrite'(3) functions. The
`bread' function returns a byte string of the given size read from the
given file. Note that the returned byte string may actually be shorter
than `SIZE' bytes because, e.g., end of file has been reached or not
enough input was currently available on a pipe. The `bwrite' function
returns the number of bytes actually written which is usually the size
of the byte string unless an error occurs. The functions fail if an
error occurred before anything was read or written. It is the
application's responsibility to check these error conditions and handle
them in an appropriate manner.

     public extern bread FD SIZE;            // read a byte string
     public extern bwrite FD DATA;           // write a byte string

   For instance, the following function uses `bread' and `bwrite' to
copy an input to an output file, using chunks of 8192 bytes at a time:

     fcopy F G               = () if bwrite G (bread F 8192) < 8192;
                             = fcopy F G otherwise;

   The file pointer of a descriptor can be positioned with `lseek'. In
difference to `fseek', this function returns the new offset. To
determine the current position you can hence use an expression like
`lseek FD 0 SEEK_CUR'.

     public extern lseek FD POS WHENCE;

   Some terminal-related routines are also provided:

     public extern isatty FD;                // is descriptor a terminal?

   The following are UNIX-specific:

     public extern ttyname FD;               // terminal associated with descriptor
     public extern ctermid;                  // name of controlling terminal
     public extern openpty, forkpty;         // pseudo terminal operations

   The `openpty' function returns a pair `(MASTER, SLAVE)' of file
descriptors opened for both reading and writing on a "pseudo terminal".
`MASTER' is to be used in the controlling process, while `SLAVE' can be
used for the standard I/O streams in a child process. The `forkpty'
function combines `openpty' with `fork' and makes the slave device the
controlling terminal of the child process; it returns a pair `(PID,
MASTER)', where `PID' is zero in the child process and the process id
of the child in the parent, and `MASTER' is the master end of the
pseudo terminal to be used by the parent. These functions are commonly
used to implement applications which drive other programs through a
terminal emulation interface.

   On UNIX systems, `clib' also provides access to the following
`fcntl' operation (see Section 2 of the UNIX manual):

     public extern fcntl FD CMD ARG;

   The `ARG' parameter of `fcntl' depends on the type of command `CMD'
which is executed. The available command codes and other relevant
values are defined as global variables, as listed below. Flags are
bitwise disjunctions of the symbolic values listed below. (The
following are the values present on most systems. Specific
implementations may provide additional flags.)

     public var const
       // fcntl command codes
       F_DUPFD, F_GETFD, F_SETFD, F_GETFL, F_SETFL,
       F_GETLK, F_SETLK, F_SETLKW,

       // lock types
       F_RDLCK, F_WRLCK, F_UNLCK,

       // file access modes and access mode bitmask
       O_RDONLY, O_WRONLY, O_RDWR, O_ACCMODE,

       // file descriptor flags
       FD_CLOEXEC,

       // status flags
       O_CREAT, O_EXCL, O_TRUNC, O_APPEND, O_NONBLOCK, O_NDELAY, O_NOCTTY,
       O_BINARY;

   The following types of commands are implemented:

     fcntl FD F_DUPFD ARG                      // duplicate a file descriptor

     fcntl FD F_GETFD ()                       // get file descriptor flags
     fcntl FD F_SETFD FLAGS                    // set file descriptor flags

     fcntl FD F_SETFD ()                       // get status flags/access mode
     fcntl FD F_SETFD FLAGS                    // set status flags

     fcntl FD F_GETLK (TYPE,POS,LEN[,WHENCE])  // query file lock information
     fcntl FD F_SETLK (TYPE,POS,LEN[,WHENCE])  // set an advisory file lock
     fcntl FD F_SETLKW (TYPE,POS,LEN[,WHENCE]) // blocking variant of F_SETLK

   The first five commands serve to duplicate descriptors and to
retrieve and change the file descriptor and status flags. The remaining
commands are used for advisory file locking. A file lock is specified
as a triple `(TYPE, POS, LEN)' or quadruple `(TYPE, POS, LEN, WHENCE)',
where `TYPE' is the type of lock (`F_RDLCK', `F_WRLCK' or `F_UNLCK' for
read locks, write locks and unlocking, respectively), `POS' the
position in the file, `LEN' the number of bytes to be locked (0 means
up to the end of the file) and `WHENCE' specifies how the `POS'
argument is to be interpreted. (This parameter has the same meaning as
for the `fseek' and `lseek' functions, see above. If `WHENCE' is
omitted, it defaults to `SEEK_SET', i.e., absolute positions.) The
value returned by the `F_GETLK' command is the lock description with
`TYPE' set to `F_UNLCK' if the given lock would be accepted, and the
description of a current lock blocking the lock request otherwise. (In
the latter case the return value is actually a quadruple, with the id
of a process currently owning a conflicting lock in the last component.)

   Note that the standard I/O operations use buffered I/O by default
which might interfere with record locking. Therefore in applications
requiring individual record locking you should work with the low-level
operations (`open', `bwrite', etc.) instead.

   The following `select' function waits for a set of files to change
I/O status. Note that this operation is available on Windows as part of
the socket interface, but it only applies to sockets there.

     public extern select FILES;

   The input is a tuple `(IN, OUT, ERR, TIMEOUT)' consisting of three
lists of file descriptors and/or file objects to be watched, and an
optional integer or floating point value specifying a timeout in
seconds. The function returns as soon as either a member of `IN' or
`OUT' becomes ready for performing an I/O operation (without blocking),
or an error condition is signaled for a member of `ERR'. The returned
value is a triple `(IN, OUT, ERR)' with all the members of the original
lists which are now ready for I/O. If the timeout is exceeded before
any of the files has become ready, a triple of three empty lists is
returned. If no timeout is specified then the function may block
indefinitely.

Examples
--------

These examples are mostly UNIX-specific, thus Windows users might wish
to skip ahead.

Fcntl
.....

The following definitions show how the `fcntl' function can be used to
change a file's "non-blocking" flag. This is useful, e.g., if we want
to read from standard input or a pipe but do not want to be blocked
until input becomes available. Instead, having set the non-blocking
flag, input operations will fail immediately if there is no input to be
read right now.

     /* set and clear the O_NONBLOCK flag of a file */
     set_nonblock FD:Int     = fcntl FD F_SETFL (FLAGS or O_NONBLOCK)
                                 where FLAGS = fcntl FD F_GETFL ();
     clr_nonblock FD:Int     = fcntl FD F_SETFL (FLAGS and not O_NONBLOCK)
                                 where FLAGS = fcntl FD F_GETFL ();

   And here is how we can perform advisory locking on an entire file.

     /* place an advisory read or write lock on an entire file (fail if error) */

     rdlock FD:Int           = () where () = fcntl FD F_SETLK (F_RDLCK, 0, 0);
     wrlock FD:Int           = () where () = fcntl FD F_SETLK (F_WRLCK, 0, 0);

     /* remove the lock from the file */

     unlock FD:Int           = () where () = fcntl FD F_SETLK (F_UNLCK, 0, 0);

     /* predicates to check whether a read or write lock could be placed */

     rdlockp FD:Int          = (LOCK!0 = F_UNLCK)
                                 where LOCK:Tuple =
                                   fcntl FD F_GETLK (F_RDLCK, 0, 0);
     wrlockp FD:Int          = (LOCK!0 = F_UNLCK)
                                 where LOCK:Tuple =
                                   fcntl FD F_GETLK (F_WRLCK, 0, 0);

   Note that to apply these functions to standard file objects you can
use the `fileno' function (see *Note Extended File Functions::) as
follows:

     ==> rdlock (fileno F)

Select
......

The `select' function accepts both files and file descriptors as input.
Here is a way to test whether input is currently available from a
file/descriptor:

     avail F                 = not null (select ([F],[],[],0)!0);

   This is useful, in particular, if the file is actually a pipe. For
instance:

     ==> def F = popen "sleep 5; echo done" "r"

     ==> avail F   // no input to be read yet, wait ...
     false

     ==> avail F   // ... input available now
     true

     ==> fget F
     "done\n"

   However, most of the time `select' is used for multiplexing I/O
operations. For instance, the following loop processes input from a set
of files, one line at a time:

     loop FILES              = loop (proc FILES F)
                                 where ([F|_],_,_) = select (FILES,[],[])
                                 if not null FILES;
                             = () otherwise;

     proc FILES F            = // done with this file, get rid of it
                               filter (neq F) FILES
                                 if feof F;
                             = // process a line
                               writes (fgets F) || FILES;

Anonymous Pipes
...............

The `pipe' and `dup2' operations provide a quick way to reassign input
and output of a child process and connect it to corresponding file
objects in the parent. For instance, here's how we can implement a
`popen2' function which works like the built-in `popen' routine, but
allows to redirect _both_ the input and output side of a child process:

     import system;

     /* Create two unnamed pipes, one for the parent to read and the child to
        write, the other one for the child to read and the parent to write. */

     popen2 CMD      = spawn2 CMD (P_IN, P_OUT) (C_IN, C_OUT)
                         where (P_IN, C_OUT) = pipe, (C_IN, P_OUT) = pipe;

     /* Fork the child and redirect its standard input and output streams to the
        child's ends of the pipe. This is accomplished with dup2 SRC DEST which
        closes the file descriptor DEST and then makes DEST a copy of SRC. In the
        parent we use fdopen to open two new file objects for the parent's ends of
        the pipes. */

     spawn2 CMD (P_IN, P_OUT) (C_IN, C_OUT)
                     = close P_IN || close P_OUT ||
                       dup2 C_IN (fileno INPUT) || dup2 C_OUT (fileno OUTPUT) ||
                       exec "/bin/sh" ["/bin/sh", "-c", CMD]
                         if fork = 0;
                     = close C_IN || close C_OUT ||
                       (fdopen P_IN "r", fdopen P_OUT "w")
                         otherwise;

   The `popen2' function employs `fork' and `exec' to spawn a child
process which executes the given command using the shell, after
redirecting the child's input and output to two pipes. In the parent
process, the `popen2' function returns a pair of files opened on the
other ends of the child's descriptors.

   The following piece of Q code shows how to apply the `popen2'
function defined above in order to pipe a string list into the UNIX
`sort' program and construct the sorted list from the output:

     mysort STRL     = do (fprintf OUT "%s\n") STRL || fclose OUT || digest IN
                         where (IN, OUT) = popen2 "sort";

     digest IN       = [] if feof IN;
                     = [freads IN|digest IN] otherwise;

   Example:

     ==> mysort ["five","strings","to","be","sorted"]
     ["be","five","sorted","strings","to"]

     ==> wait // get exit code of child process (sort program)
     (15804,0)


File: qdoc.info,  Node: Terminal Operations,  Next: Readline Interface,  Prev: Low-Level I/O,  Up: Clib

12.9 Terminal Operations
========================

The following (UNIX-specific) operations from the `system' module
provide an interface to the POSIX termios interface. Terminal attributes
are stored in a "termios" structure, represented as a 7-tuple `(IFLAG,
OFLAG, CFLAG, LFLAG, ISPEED, OSPEED, CC)'. The control character set
`CC' is represented as a list of character numbers, indexed by the
symbolic constants `VEOF' etc. See `termios'(3) for further details.

     public extern tcgetattr FD;             // get terminal attributes
     public extern tcsetattr FD WHEN ATTR;   // set terminal attributes

     public extern tcsendbreak FD DURATION;  // send break
     public extern tcdrain FD;               // wait until all output finished
     public extern tcflush FD QUEUE;         // flush input or output queue
     public extern tcflow FD ACTION;         // control input/output flow

     public extern tcgetpgrp FD;             // get terminal process group
     public extern tcsetpgrp FD PGID;        // set terminal process group

     /* Access components of the termios structure. */

     public c_iflag ATTR, c_oflag ATTR, c_cflag ATTR, c_lflag ATTR,
       c_ispeed ATTR, c_ospeed ATTR, c_cc ATTR;

Example
-------

This example shows how to use the `termios' functions to read a
password from the terminal without echoing. This is an almost literal
translation of the C program described in Richard Stevens: Advanced
Programming in the UNIX Environment, Addison-Wesley, 1993, cf. p. 350.
The main difference is that we merely ignore the `SIGINT' and `SIGTSTP'
signals instead of blocking them (the latter is not supported by Q's
`trap' builtin).

     import system;

     getpass PROMPT  = fwritec F "\n" || unprep F SAVE || PW
                         where F:File = fopen ctermid "r+", SAVE = prep F,
                           PW = fwrites F PROMPT || fflush F || freads F;

     /* prep F: ignore SIGINT and SIGTSTP and prepare the terminal */

     prep F          = tcsetattr (fileno F) TCSAFLUSH NATTR || (ATTR,TRAPS)
                         where TRAPS = map (trap SIG_IGN) [SIGINT,SIGTSTP],
                           ATTR = tcgetattr (fileno F),
                           (IF,OF,CF,LF,IS,OS,CC) = ATTR,
                           LF = LF and not (ECHO or ECHOE or ECHOK or ECHONL),
                           NATTR = (IF,OF,CF,LF,IS,OS,CC);

     /* unprep F SAVE: revert to previous settings */

     unprep F (ATTR,TRAPS)
                     = tcsetattr (fileno F) TCSAFLUSH ATTR ||
                       zipwith trap TRAPS [SIGINT,SIGTSTP];


File: qdoc.info,  Node: Readline Interface,  Next: System Information,  Prev: Terminal Operations,  Up: Clib

12.10 Readline Interface
========================

The following functions from `system.q' provide a basic interface to
the GNU readline library. While readline doesn't really belong to the
POSIX interface, it is rather useful and the interpreter uses it anyway,
so it makes sense to also provide it as a part of the system interface.

     public extern readline PROMPT, rl_line_buffer;
     public extern add_history LINE, stifle_history MAX;
     public extern read_history FNAME, write_history FNAME;

   These functions work like their C counterparts. The `readline'
function prompts with the given string and reads an input line,
providing editing and history facilities. Basic bash-like filename
completion is also provided. The `rl_line_buffer' function returns the
text of the current input line, which is useful, e.g., if a custom
completion function (see below) needs to inspect surrounding context.

   The `add_history' routine adds a line to the history. The maximum
size of the history can be set with `stifle_history', which also
returns the size which was set previously. A negative `MAX' value makes
the size of the history unbounded (which is also the default). Last but
not least, the history can be read from and written to a file with the
`read_history' and `write_history' functions.

   For instance, the following commands read an input string, add it to
the history, and finally save the history to a file:

     ==> import system

     ==> readline "input> "
     input> foo bar
     "foo bar"

     ==> add_history _
     ()

     ==> write_history "myhistory"
     ()

   As another example, here is the definition of a little convenience
function which reads a line using `readline' and enters it into the
history if it is nonempty:

     get_line PROMPT = if not null LINE then add_history LINE || LINE
                         where LINE:String = readline PROMPT;

   Readline's standard filename completion facility can be augmented
with a custom _completion function_. This is achieved by simply setting
the following global variable to the desired function:

     public var RL_COMPLETION_FUNCTION;

   The completion function is invoked with two arguments, the string to
be completed and the position index of that string in the input line
(`rl_line_buffer', see above), and is expected to return a string list
with the possible completions. For instance, here is a simple example
of a function which checks a list of "command" words for possible
completions:

     def RL_COMPLETION_FUNCTION = complete;
     complete S _    = filter (is_prefix S) ["bar","foo","gnats","gnu"];
     is_prefix X Y   = (X=sub Y 0 (#X-1));

   The position index argument is useful if the completion depends on
the position inside the input line. For instance, the following function
only attempts completion at the beginning of the input line:

     complete S 0    = filter (is_prefix S) ["bar","foo","gnats","gnu"];

   Readline's default behaviour is to try a custom completion first, if
it is available, and to fall back to the standard filename completion
otherwise. The latter can be suppressed by ending the list of
completions with a `()' entry:

     complete S 0    = filter (is_prefix S) ["bar","foo","gnats","gnu"] ++ [()];
     complete _ _    = [()] otherwise;

   The text units for which readline attempts completion are the "words"
of the input line. There is a `RL_WORD_BREAK_CHARS' variable which
allows you to change readline's idea of what a word is, by setting the
variable to a string containing the word delimiter characters. The
default definition of this variable is as follows:

     public var RL_WORD_BREAK_CHARS = " \t\n\"\\'`@$><=;|&{(";

   Redefining `RL_WORD_BREAK_CHARS' affects both readline's cursor
movement commands and the behaviour of the completion routine. For
instance, to turn the comma into a word delimiter, you can change
`RL_WORD_BREAK_CHARS' as follows:

     def RL_WORD_BREAK_CHARS = RL_WORD_BREAK_CHARS++",";


File: qdoc.info,  Node: System Information,  Next: Sockets,  Prev: Readline Interface,  Up: Clib

12.11 System Information
========================

The functions described here are all in the `system' module. Error
codes from various system operations can be retrieved with the following
functions:

     public extern errno, seterrno N;        // get/set last error code
     public extern perror S, strerror N;     // print error message

   The `perror' function is commonly used to report error conditions in
system operations on the standard error file. For instance:

     ==> fopen "/etc/passw" "r"
     fopen "/etc/passw" "r"

     ==> perror "fopen"
     fopen: No such file or directory
     ()

   If more elaborate formatting is required then you can use `strerror'
on the `errno' value to obtain the error message as a string:

     ==> fprintf ERROR "fopen returned message `%s'\n" (strerror errno)
     fopen returned message `No such file or directory'
     ()

   Note that `errno' is only set when an error occurs in a system call.
You can use `seterrno' to reset the `errno' value before a system
operation to check whether there actually was an error while executing
the system call:

     ==> seterrno 0
     ()

     ==> fopen "/etc/passwd" "r"
     <<File>>

     ==> perror "fopen"
     fopen: Success
     ()

   The remaining operations are used to obtain various information about
the system and its information databases. For instance, the `uname'
operation returns a 5-tuple containing information identifying the
operating system (this operation is generally only available on
UNIX-like systems):

     public extern uname;

     /* Access components of uname result. */

     public un_sysname UNAME, un_nodename UNAME, un_release UNAME,
       un_version UNAME, un_machine UNAME;

   The hostname of the system can be retrieved with the `gethostname'
function.

     public extern gethostname;

   The password and group database can be accessed with the following
functions. Password entries are encoded as 7-tuples `(NAME, PASSWD,
UID, GID, GECOS, DIR, SHELL)', group entries as 4-tuples `(NAME,
PASSWD, GID, MEMBERS)'. This information is only available on UNIX-like
systems.

     public extern getpwuid UID, getpwnam NAME;      // look up a password entry
     public extern getpwent;                         // list of all pw entries
     public extern getgrgid GID, getgrnam NAME;      // look up group entry
     public extern getgrent;                         // list of all group entries

     /* Access components of password and group structures. */

     public pw_name PW, pw_passwd PW, pw_uid PW, pw_gid PW, pw_gecos PW,
       pw_dir PW, pw_shell PW;

     public gr_name GR, gr_passwd GR, gr_gid GR, gr_members GR;

   Moreover, the `crypt' function can be used to perform UNIX password
encryption (see `crypt'(3) for details).

     public extern crypt KEY SALT; // (U)

   The following functions can be used to query host information as
well as the network protocols and services available on your system.
This information is closely related to the socket interface described in
*Note Sockets::. For a closer description of these operations we refer
the reader to the corresponding manual pages. Note that the
`gethostent', `getprotoent' and `getservent' operations are not
available on Windows.

   The host database: Host entries are of the form `(NAME, ALIASES,
ADDR_TYPE, ADDR_LIST)', where `NAME' denotes the official hostname,
`ALIASES' its alternative names, `ADDR_TYPE' the address family and
`ADDR_LIST' the list of addresses.

     public extern gethostbyname HOST, gethostbyaddr ADDR;
     public extern gethostent; // (U)

     public h_name HENT, h_aliases HENT, h_addr_type HENT, h_addr_list HENT;

   Note that both hostnames and IP addresses are specified as strings.
Hostnames are symbolic names such as `"localhost"', and can also have a
domain name specified, as in `"www.gnu.org"'. IPv4 addresses use the
well-known "numbers-and-dots" notation, like the loopback address
`"127.0.0.1"'. IPv6 addresses are usually written as eight 16-bit
hexadecimal numbers that are separated by colons; two colons are used
to abbreviate strings of consecutive zeros. For example, the IPv6
loopback address `"0:0:0:0:0:0:0:1"' can be abbreviated as `"::1"'.

   The protocol database: Protocol entries are of the form `(NAME,
ALIASES, PROTO)' denoting official name, aliases and number of the
protocol.

     public extern getprotobyname NAME;
     public extern getprotobynumber PROTO;
     public extern getprotoent; // (U)

     public p_name PENT, p_aliases PENT, p_proto PENT;

   The service database: Service entries are of the form `(NAME,
ALIASES, PORT, PROTO)' denoting official name, aliases, port number and
protocol number of the port. The `NAME' argument of `getservbyname' can
also be a pair `(NAME, PROTO)' to restrict the search to services for
the given protocol (given by its name). Likewise, the `PORT' argument
can also be given as `(PORT, PROTO)'.

     public extern getservbyname NAME;
     public extern getservbyport PORT;
     public extern getservent; // (U)

     public s_name PENT, s_aliases PENT, s_port PENT, s_proto PENT;

   For instance, here is some information retrieved from a typical Linux
system:

     ==> import system

     ==> uname
     ("Linux","obelix","2.4.19-4GB","#2 Tue Mar 4 16:03:51 CET 2003","i686")

     ==> gethostname
     "obelix"

     ==> gethostbyname gethostname
     ("obelix.local",["obelix"],2,["127.0.0.2"])

     ==> gethostbyaddr "::1"
     ("localhost",["ipv6-localhost","ipv6-loopback"],10,["::1"])

     ==> getprotobyname "tcp"
     ("tcp",["TCP"],6)

     ==> getservbyname "ftp"
     ("ftp",[],5376,"tcp")

     ==> getpwuid getuid
     ("ag","x",500,100,"Albert Graef","/home/ag","/bin/bash")

     ==> getgrgid getgid
     ("users","x",100,[])


File: qdoc.info,  Node: Sockets,  Next: POSIX Threads,  Prev: System Information,  Up: Clib

12.12 Sockets
=============

The following functions from the `system' module are available on
systems providing a BSD-compatible socket layer, which, besides BSD,
includes BEOS, Linux, OSX, Windows and most recent System V flavours.
Sockets provide bidirectional communication channels on the local
machine as well as across the network. Sockets are represented by file
descriptors which can be written to and read from with the `send' and
`recv' functions. On most systems, socket descriptors are just ordinary
file descriptors which can also be used with low-level I/O functions
and `fdopen' as usual. However, on some systems (in particular, BEOS
and Windows), socket descriptors are "special" and all socket I/O must
be performed with the special socket operations.

   At creation time, a socket is described by the following attributes
(see `socket'(2) for more details):

   * Address family (sometimes also referred to as namespace, protocol
     family or domain). The adress families supported by this
     implementation are `AF_LOCAL' (a.k.a. `AF_UNIX' a.k.a. `AF_FILE'),
     `AF_INET' and `AF_INET6'. Not all protocol families may be
     available on all systems (e.g., the Windows socket library only
     supports `AF_INET').

   * Communication style (also called the socket type). One
     distinguishes between "connection-based" sockets which are used to
     establish client/server connections, and "connectionless" sockets
     which allow data to be received from and sent to arbitrary
     addresses. The socket types supported by this implementation are
     `SOCK_STREAM', `SOCK_DGRAM', `SOCK_SEQPACKET', `SOCK_RAW' and
     `SOCK_RDM'. Among these, `SOCK_STREAM', `SOCK_SEQPACKET' and
     `SOCK_RDM' are connection-based. Please note that not all socket
     types are supported for all protocol families, and some socket
     types may be entirely missing on non-UNIX systems.

   * Protocol: For each address a.k.a. protocol family and socket type
     there may be a number of different protocols available. For the
     `AF_LOCAL' namespace, which refers to the local filesystem, the
     protocol is always 0, the default protocol. The available
     protocols for the internet namespaces can be retrieved from the
     protocol database (see *Note System Information::).

   Before another process can connect to a socket it must also be bound
to an address. The address format depends on the address family of the
socket.  For the local namespace, the address is simply a filename on
the local filesystem. For the IPv4 namespace, it is a pair `(HOST,
PORT)' where `HOST' denotes the host name or IP address, specified as a
string, and `PORT' is a port number. For the IPv6 namespace, it is a
quadruple `(HOST, PORT, FLOWINFO, SCOPEID)'. Host names and known port
numbers can be retrieved from the host and service databases (see *Note
System Information::).

   The following operations are provided to create a socket, or a pair
of connected sockets, for the given address family, socket type and
protocol. They return the file descriptor of the socket (or a pair of
file descriptors).

     public extern socket FAMILY TYPE PROTO;
     public extern socketpair FAMILY TYPE PROTO; // (U)

   The `shutdown' function terminates data transmission on a socket.
You can stop reading, writing or both, depending on whether `HOW' is
`SHUT_RD', `SHUT_WR' or `SHUT_RDWR'. Note that this operation does not
close the socket's file descriptor; for this purpose `closesocket' is
used (see below).

     public extern shutdown SOCKET HOW;

   The `closesocket' function closes a socket. On most systems this is
just identical to `close' (see *Note Low-Level I/O::), but, as already
noted, on some systems socket descriptors are special and you must use
this function instead.

     public extern closesocket SOCKET;

   The `bind' function binds a socket to an address. This is also done
automatically when the socket is first used. However, if the socket has
to be found by another process you'll have to explicitly specify an
address for it. The `bind' function does just that.

     public extern bind SOCKET ADDR;

   The following operations are used to start listening for and accept
connection requests on a socket. These operations are used on the server
side of a connection-based socket. The argument `N' of `listen' denotes
the maximum number of pending connection requests for the server. After
the call to listen, the server can accept connections from a client
with the `accept' function, which returns a pair `(SOCKET, ADDR)',
where `SOCKET' is a new socket connected to the client, and `ADDR' is
the client's address.

     public extern listen SOCKET N;
     public extern accept SOCKET;

   The `connect' function is used to initiate a connection on a socket.
This function can be used on both connection-based and connectionless
sockets. In the former case, `connect' can only be invoked once. In the
latter case, it can be invoked multiple times, and sets the remote
socket for subsequent send and receive operations.

     public extern connect SOCKET ADDR;

   The following routines retrieve information about a socket. The local
address of a socket and the address of the remote socket it is connected
to can be retrieved with `getsockname' and `getpeername'. Socket
options, specified using a protocol level `LEVEL' and an option index
`OPT', can be queried and changed with `getsockopt' and `setsockopt'.
The option values are encoded as byte strings (cf. *Note Byte
Strings::). For a description of the available options see
`getsockopt'(2).

     public extern getsockname SOCKET;
     public extern getpeername SOCKET;
     public extern getsockopt SOCKET LEVEL OPT;
     public extern setsockopt SOCKET LEVEL OPT VAL;

   Finally, the following specialized I/O functions are used to transmit
data over a socket. All data is encoded as byte strings. The receive
operations return the received data (which may be shorter than the
requested size, if not enough data was currently available), the send
operations the number of bytes actually written. For
`recvfrom'/`sendto' the data is encoded as a pair `(ADDR, DATA)' which
includes the source/destination address; these operations are typically
used for connectionless sockets. The `FLAGS' argument is used to
specify special transmission options (see the `MSG_*' constants at the
beginning of `clib.q').

     public extern recv SOCKET FLAGS SIZE, send SOCKET FLAGS DATA;
     public extern recvfrom SOCKET FLAGS SIZE, sendto SOCKET FLAGS DATA;

Example
-------

The following script demonstrates how we can implement a connectionless
server in the IPv4 namespace which repeatedly accepts a request from a
client and sends back an answer. In this example the requests are
strings denoting Q expressions; the server evaluates each expression and
sends back the result as a string. The client reads input from the user,
transmits it to the server and prints the received answer. Note that the
transmitted strings are represented as byte strings, as required by the
`recvfrom' and `sendto' operations. The `bytestr' and `bstr' functions
are used to convert between character and byte strings, see *Note Byte
Strings::.

     import system;

     def BUFSZ = 500000; // buffer size

     /* the server: receive messages, evaluate them as Q expressions, and
        send back the results */

     def SERVER = ("localhost",5001); // the server address

     server          = server_loop FD
                         where FD:Int = socket AF_INET SOCK_DGRAM 0,
                           () = bind FD SERVER;
                     = perror "server" otherwise;

     server_loop FD  = sendto FD 0 (ADDR,eval MSG) || server_loop FD
                         where (ADDR,MSG) = recvfrom FD 0 BUFSZ;
                     = server_loop FD otherwise;

     /* evaluate an expression encoded as a byte string, catch syntax
        errors and exceptions, convert result back to a byte string */

     eval MSG        = catch exception (bytestr (str VAL))
                         where 'VAL = valq (bstr MSG);
                     = bytestr ">>> SYNTAX ERROR" otherwise;
     exception _     = bytestr ">>> ABORTED";

     /* the client: read input from user, send it to the server, print
        returned result */

     def CLIENT = ("localhost",5002); // the client address

     client          = client_loop FD
                         where FD:Int = socket AF_INET SOCK_DGRAM 0,
                           () = bind FD CLIENT;
                     = perror "client" otherwise;

     client_loop FD  = sendto FD 0 (SERVER,bytestr MSG) ||
                       printf "%s\n" (bstr (recv FD 0 BUFSZ)) || client_loop FD
                         if not null MSG
                         where MSG:String = writes "\nclient> " || flush || reads;
                     = () otherwise;

   For instance, we can invoke the server in a secondary thread and then
execute the client as follows:

     ==> def S = thread server

     ==> client

     client> prd [1..50]
     30414093201713378043612608166064768844377641568960512000000000000

     client> 1+)
     >>> SYNTAX ERROR

     client> quit
     >>> ABORTED

     client>


File: qdoc.info,  Node: POSIX Threads,  Next: Expression References,  Prev: Sockets,  Up: Clib

12.13 POSIX Threads
===================

On systems where the POSIX threads library or some compatible
replacement is available (this includes Windows and most modern UNIXes),
`clib' provides functions for handling multiple threads of control.
"Threads", a.k.a. "light-weighted processes", allow you to realize
"multithreaded scripts" consisting of different tasks which together
perform some computation in a distributed manner. All tasks are
executed concurrently. Thus you can, e.g., perform some lengthy
calculation in a background task while you go on evaluating other
expressions in the interpreter's command loop. You can also have tasks
communicate via "mutexes", "conditions" and "semaphores".

   The operations described in this section (which are all contained in
`clib' and thus included in the prelude) are in close correspondence
with POSIX 1003.1b. However, some operations are named differently, and
semaphores provide the extra functionality of sending data from one
thread to another. Mutexes are also supported, mostly for the purpose
of handling critical sections involving operations with side-effects
(I/O etc.). Mutexes are _not_ required to make conditions work since
these have their own internal mutex handling. For more information on
POSIX threads, please refer to the corresponding section in the UNIX
manual.

   Please note that these functions will only work as advertised if the
interpreter has been built with POSIX thread support. Moreover, in the
current implementation the interpreter effectively serializes
multithreaded scripts on the reduction level and thus user-level threads
cannot really take advantage of multi-processor machines.

* Menu:

* Thread Creation and Management::
* Realtime Scheduling::
* Mutexes::
* Conditions::
* Semaphores::
* Threads and Signals::
* Thread Examples::


File: qdoc.info,  Node: Thread Creation and Management,  Next: Realtime Scheduling,  Up: POSIX Threads

12.13.1 Thread Creation and Management
--------------------------------------

`Clib' threads are represented using "handles" (objects of type
`Thread'). Note that a thread is canceled automatically as soon as its
handle is garbage collected, thus you should keep the handle around as
long as the thread is needed. For convenience, thread handles are
numbered arbitrarily, starting at 0 which denotes the main thread, and
are ordered by the thread numbers.  This is handy, e.g., if you want to
use thread handles as indices in a dictionary.

     public extern type Thread;              // thread handle type

     public isthread THREAD;                 // check for thread objects

     public extern thread_no THREAD;         // thread number
     public extern this_thread;              // handle of the current thread

   The basic thread operations are listed below. The `thread' function
starts evaluating its special argument in a new thread, and returns its
handle. You can wait for a thread to terminate and obtain the evaluated
result with the `result' function. (If there is no result, because the
thread has been canceled, or was aborted with `halt', `quit' or a
runtime error, `result' fails.) Note that `halt' or `quit' in a thread
which is not the main thread only terminates the current thread;
however, the `exit' function, cf. *Note Process Control::, always exits
from the interpreter. You can also terminate the current thread
immediately and return a given value as its result with the `return'
function; in the main thread, this function is equivalent to `halt' and
the return value is ignored. Moreover, all threads except the main
thread can also be canceled from any other thread using the `cancel'
function. Finally, the `yield' function allows the interpreter to
switch threads at any given point (normally the interpreter will only
switch contexts in certain builtins and when a new rule is activated).

     public extern special thread X;         // start new thread
     public extern return X;                 // terminate thread with result X
     public extern cancel THREAD;            // cancel THREAD
     public extern result THREAD;            // wait for THREAD, return result
     public extern yield;                    // allow context switch

   `Clib' threads always use "deferred" cancellation, hence thread
cancellation requests are usually not honored immediately, but are
deferred until the thread reaches a "cancellation point" where it is
safe to do so. Cancellation points occur at certain C library calls
listed in the POSIX threads documentation, when a new equation is
activated in the Q interpreter, and when `yield' is called.

   You can also check whether a thread is still active or has been
canceled. If neither condition holds, then the thread has already been
terminated and you can obtain its result with the `result' operation.

     public extern active THREAD;            // check if THREAD is active
     public extern canceled THREAD;          // check if THREAD was canceled


File: qdoc.info,  Node: Realtime Scheduling,  Next: Mutexes,  Prev: Thread Creation and Management,  Up: POSIX Threads

12.13.2 Realtime Scheduling
---------------------------

Threads are always created with the default "non-realtime" scheduling
policy. On some systems it is also possible to increase a thread's
priority and have it scheduled in "realtime". This is useful for tasks
with strict timing and responsiveness requirements, such as a function
performing recording or playback in a multimedia application.

   The scheduling policy and the priority of a thread can be changed
with the following function which takes two extra arguments, the policy
`POL' (0 = default, 1 = realtime round-robin, 2 = realtime fifo
scheduling) and the priority `PRIO' (where 0 denotes the default
priority). Note that on most systems realtime scheduling will only be
granted to privileged processes.

     public extern setsched THREAD POL PRIO; // set scheduling parameters

   The current scheduling policy and priority of a thread can be
retrieved with the `getsched' function:

     public extern getsched THREAD;          // get scheduling parameters

   The actual range of priority values is system-specific. On a typical
UNIX system a non-realtime thread can only have priority 0, while
realtime threads always have positive priorities. Higher priority
threads always take precedence over lower priority ones. All else being
equal, threads using round-robin scheduling are each given their "fair"
timeslice, while fifo threads are handled on a "first come first
served" basis. The latter can only be interrupted by higher priority
processes (and signals) and thus should be used with utmost care. In
any case you must make sure that a high-priority thread does not run
unsuspended for extended periods of time, otherwise it might lock up
your system. Thus a realtime thread should typically spend most of its
time "in limbo" where it waits for input to arrive or a condition to be
signaled.

   Assuming that the priority value ranges are contiguous and contain
either 0 or 1, you can determine the ranges for your system with the
following little script:

     test POL PRIO   = setsched this_thread 0 0 || true
                         where () = setsched this_thread POL PRIO;
                     = false otherwise;

     priotest POL    = (hd L,last L) if not null L
                         where L = reverse (while (test POL) pred 0) ++
                           while (test POL) succ 1;

   Here is a sample result obtained on Linux:

     ==> map priotest [0,1,2]
     [(0,0),(1,99),(1,99)]

Realtime Scheduling under Windows
.................................

Under Windows, things are a bit different, since Windows does not really
have POSIX-compatible thread scheduling. Instead, processes have
"priority classes" while threads have "priorities" which are
interpreted in the context of the priority class of the process they
belong to. Therefore under Windows the `POL' argument of the `setsched'
operation is actually interpreted as a priority class for the _entire_
process while the `PRIO' argument specifies the priority of the
individual thread. `Clib' keeps track of all `setsched' calls and
always lets the process have the highest priority class specified for a
thread which is still active.

   To these ends, the policy values 0, 1 and 2 are mapped to the
priority classes "normal", "high" and "realtime". (The latter should be
used sparingly, if at all, in a Q script, because it makes Windows very
unresponsive.) Moreover, the `setsched' function also accepts a policy
value of -1 to denote "idle time" processes. (You'll rarely have any
use for this, unless you want to write a screensaver in Q.) For each
policy, the possible priority values have a range from -3 to 3, where
-3 denotes idle time threads, 0 is the normal priority, and 3 is the
highest priority to be used for "time-critical" threads. The remaining
priority levels provide some additional amount of control over which
thread gets the bone first.

   While some "special effects" can be achieved with Windows' more
exotic priority values, for typical usage a policy of 0 with zero
priority should be employed for ordinary threads, a policy of 1 with
some (small) positive priority value for a thread with moderate realtime
requirements, and a policy of 2 if time is very critical. If you follow
these conventions then your script will be able to run under Windows and
most UNIX systems unchanged.


File: qdoc.info,  Node: Mutexes,  Next: Conditions,  Prev: Realtime Scheduling,  Up: POSIX Threads

12.13.3 Mutexes
---------------

`Clib' mutexes come in three flavours: "fast" (no error checking),
"error checking" (fail if the current thread already holds a lock on
the mutex) and "recursive" (the same thread may lock the mutex multiple
times, and the same number of unlock operations is required to unlock
the mutex again). The supported operations are `lock' (wait for the
mutex to be unlocked, then lock it and return `()'), `unlock' (unlock
the mutex, return `()') and `try' (lock the mutex if it is available,
fail otherwise). As of Q 7.11, on systems which support this
functionality, the `try' operation can also be used with an argument of
the form `(MUTEX, TIME)' to denote an absolute timeout value. This
works analogously to the `await' operation on conditions, see *Note
Conditions::, for details.

   CAVEAT: These operations are _dangerous_, as you can easily create
deadlocks which might lock up the interpreter as well. Note that a
deadlock will also prevent the involved threads from being canceled
since, in accordance with the POSIX standard, the wait for a mutex lock
is _not_ a cancellation point. Thus these operations should be used
with care.

   Since Q has no mutable variables and the interpreter's builtins are
all thread-safe, mutexes are actually used much less in Q than in other,
procedural languages. They are most useful for protecting critical
sections in which a sequence of operations with side-effects (such as
I/O) is to be carried out in an atomic fashion. This can be done by
locking a mutex at the beginning and unlocking it at the end of the
sequence. Such usage is safe, provided that no operation in the sequence
may block for an extended or even indefinite period of time.

     public extern type Mutex;               // mutex type

     public ismutex MUTEX;                   // check for mutex objects

     public extern mutex;                    // standard (fast) mutex object
     public extern errorchecking_mutex;      // error checking mutex object
     public extern recursive_mutex;          // recursive mutex object

     public extern lock MUTEX;               // lock MUTEX
     public extern unlock MUTEX;             // unlock MUTEX
     public extern try MUTEX;                // try MUTEX, (MUTEX,TIME) for timeout


File: qdoc.info,  Node: Conditions,  Next: Semaphores,  Prev: Mutexes,  Up: POSIX Threads

12.13.4 Conditions
------------------

`Clib' conditions support the following operations: `signal' (wake up
one thread waiting for the condition), `broadcast' (wake up all threads
waiting for the condition) and `await' (suspend the current thread
until the condition is sent). Each of these operations returns `()'.
The `await' operation can also be invoked with a `(COND, TIME)' tuple
to denote a timed wait. If the wait times out or is interrupted by a
signal then the operation fails. Note that in accordance with the POSIX
standard the `TIME' value denotes an _absolute_ time (integer or
floating point value in seconds since the "epoch", see also the
description of the built-in `time' function in *Note Miscellaneous
Functions::).

     public extern type Condition;           // condition type

     public iscondition COND;                // check for condition objects

     public extern condition;                // new condition object

     public extern signal COND;              // signal COND
     public extern broadcast COND;           // broadcast COND
     public extern await COND;               // wait for COND, or (COND,TIME)


File: qdoc.info,  Node: Semaphores,  Next: Threads and Signals,  Prev: Conditions,  Up: POSIX Threads

12.13.5 Semaphores
------------------

`Clib' semaphores are in fact "semaphore queues" which can be used as a
a communication channel to pass values between different threads using
a FIFO discipline. In extension to the POSIX 1003.1b semaphore
operations, `clib' also provides support for "bounded" semaphores which
are created with a positive limit on the number of values the semaphore
queue may hold at any time. When a new value is posted to a bounded
semaphore, the current thread is suspended until the queue has room to
receive a new value, if necessary.

   The supported operations are `post' (enqueue a value), `get'
(dequeue a value and return it, as soon as one is available), `try'
(non-blocking version of `get' which fails if no value is currently
available), `get_size' or `#' which both return the current queue size,
and `get_bound' which returns the queue size limit of a bounded
semaphore (or zero if the semaphore is unbounded). As of Q 7.11, on
systems which support this functionality, the `try' operation can also
be used with an argument of the form `(SEM, TIME)' to denote an
absolute timeout value. This works analogously to the `await' operation
on conditions, see *Note Conditions::, for details.

   Note that even with unbounded semaphores the maximum semaphore size
is actually limited by the operating system, see your local POSIX
threads documentation for details. If this implementation-specific
limit is exceeded, the attempt to post a new value raises a `syserr 9'
exception.

     public extern type Semaphore;           // semaphore type

     public issemaphore SEM;                 // check for semaphore objects

     public extern semaphore;                // semaphore object
     public extern bounded_semaphore MAX;    // bounded semaphore object

     public extern post SEM X;               // enqueue a value
     public extern get SEM;                  // dequeue a value
     public extern try SEM;                  // try SEM, (SEM,TIME) for timeout

     public extern get_size SEM;             // get the current queue size
     public extern get_bound SEM;            // get the max queue size (0 if none)


File: qdoc.info,  Node: Threads and Signals,  Next: Thread Examples,  Prev: Semaphores,  Up: POSIX Threads

12.13.6 Threads and Signals
---------------------------

Some remarks about the interaction between `clib' threads and the
interpreter's signal handling are in order. It is a well-known fact that
threads and signals generally do not mix very well. Therefore, in the
current implementation, all signal handling is actually performed in the
interpreter's main thread. This is where you should set up your signal
handlers using the `trap' and `catch' builtins, as explained in *Note
Exception Handling::. If necessary, the main thread of your script can
inform other threads about received signals using the thread
synchronization functions described above.

   However, the interpreter still allows you to set up traps in
secondary threads and keeps track of the configured traps separately
for each thread. The traps in secondary threads will not have any
effect until such a thread forks (see *Note Process Control::). In this
case the forking thread will be the one and only thread in the child
process and becomes the main thread. At this point, it takes on the
signal handling, and any traps which have been set up before become
active. (Note that in order to make this work reliably, you should
configure the traps _before_ you call `fork' and protect the call to
`fork' with an enclosing `catch'. Otherwise a signal might arrive in the
time between the `fork' and the next function call, causing the new
process to exit before it had a chance of setting up its own signal
handling.)


File: qdoc.info,  Node: Thread Examples,  Prev: Threads and Signals,  Up: POSIX Threads

12.13.7 Thread Examples
-----------------------

As already mentioned, threads allow you to perform a lengthy calculation
as a background task. You can start such a task simply as follows:

     ==> def TASK = thread (sum [1..1000000])

     ==> // do some other work ...

     ==> result TASK // get the result
     500000500000

     ==> stats all
     thread #0: 0 secs, 1 reduction, 0 cells
     thread #1: 4.3 secs, 2000004 reductions, 2000007 cells

   As indicated, if the `stats' command is invoked with the parameter
`all' then it also shows the statistics of background threads once they
are finished. To release all resources associated with a thread (and
also remove it from the `stats all' list) you must undefine all
variables referencing the thread handle:

     ==> undef TASK

   Conditions are useful when a thread has to wait for some condition
before proceeding with a computation. E.g.:

     ==> def COND = condition, TASK = thread (await COND || writes "Got it!\n")

     ==> signal COND
     ()
     Got it!

   Semaphores are used when expression values have to be passed from one
thread to another. For instance, let us rewrite the backtracking
algorithm for the N queens problem (see *Note Exception Handling::) such
that it runs as a background task which returns results via a semaphore
instead of printing them directly on the terminal:

     def RES = semaphore; // semaphore used to transmit results

     queens N        = thread (search N 1 1 [] || post RES ());

     search N I J P  = post RES P if I>N;
                     = search N (I+1) 1 (P++[(I,J)]) || fail if safe (I,J) P;
                     = search N I (J+1) P if J<N;
                     = () otherwise;

   Note that we added a `post RES ()' to signal that all solutions have
been found. The `safe' function is as in *Note Exception Handling::. In
order to limit the size of the `RES' semaphore queue, we could also use
a bounded semaphore instead, e.g.:

     def RES = bounded_semaphore 10;

   We can use a second thread to process the semaphore queue and print
solutions as they become available:

     print           = thread (loop print1 RES);

     print1 ()       = return (); // no more results
     print1 P        = write P || writes "\n" otherwise;

     /* iterate a function over a semaphore */

     loop F SEM      = F (get SEM) || loop F SEM;

   To use this program, simply start the two background tasks as
follows:

     ==> def QUEENS = queens 8, PRINT = print

   Both threads will begin to execute immediately, and you will see the
results scroll by. Once the threads are finished, you can check the
resources used by each thread with the `stats all' command:

     ==> stats all
     thread #0: 0 secs, 2 reductions, 2 cells
     thread #1: 1.71 secs, 700451 reductions, 943 cells
     thread #2: 0 secs, 461 reductions, 3 cells

   Also note that here we used again a `def' statement to assign the
thread handles to corresponding variables. It is important that you keep
these variables as long as you want the threads to survive. The
interpreter automatically cancels a thread as soon as the corresponding
thread handle is garbage collected. Hence you can stop the threads at
any time by simply undefining the variables using, e.g., the `clear'
command.


File: qdoc.info,  Node: Expression References,  Next: Time Functions,  Prev: POSIX Threads,  Up: Clib

12.14 Expression References
===========================

These are in `clib' and thus included in the prelude. Expression
references work like pointers to expression values. The reference is
initialized with the `ref' function, giving the initial value as an
argument. The referenced value can then be changed with the `put'
function and retrieved with `get'. Also, as of Q 7.8 an "assignment"
operator `:=' is provided as syntactic sugar for `put'.

     public extern type Ref;                 // reference type

     public isref REF;                       // check for reference objects

     public extern ref X;                    // initialize a reference object

     public extern put REF X;                // store a new value
     public extern get REF;                  // retrieve the current value

     public (:=) X Y @ (=);                  // assignment operator

   NOTE: References can point to arbitrary values, in particular they
can also point to other references. However, cyclic chains of references
should be avoided since in the current implementation the interpreter
cannot garbage-collect such structures.

   References can be used to implement mutable data structures. This is
particularly useful if a function has to maintain some internal state to
perform its calculations. For instance, here is a function which
memoizes past function calls in a hash table so that each requested
value only has to be computed once:

     def HASH = ref emptyhdict;

     public hashed F X;
     hashed F X      = get HASH!(F,X) if member (get HASH) (F,X);
                     = put HASH (update (get HASH) (F,X) Y) || Y
                         where Y = F X otherwise;

   Using the `hashed' function, a hashed version of the recursive
Fibonacci function can be implemented as follows. Since no value of the
function is computed more than once, the definition works in linear time
(up to logarithmic factors for the table updates and lookups). In fact,
even an expression like `map fib [0..N]' will only require a linear
number of table manipulations and other operations.

     fib             = hashed hfib;

     hfib 0          = 0;
     hfib 1          = 1;
     hfib N          = fib (N-1)+fib (N-2) if N>1;

   References also provide a general means for communicating values in a
multithreaded script (see *Note POSIX Threads::). In this case they are
typically used together with a condition which is signaled when the
value changes:

     def VAL_CHANGED = condition, VAL = ref ();
     change X        = put VAL X || broadcast VAL_CHANGED;

   A loop like the following might then be executed by any number of
other threads which have to keep track of the value in question, and
perform some appropriate action when the value changes:

     watch           = await VAL_CHANGED || action (get VAL) || watch;

Sentinels
---------

As of Q 7.1, there is a a second, lazy kind of expression references
called _sentinels_. Sentinels defer the evaluation of the referenced
expression until the sentinel is garbage-collected. They are created
with the `sentinel' function which takes the expression to be evaluated
later as a special argument. There are no other access operations.

     public extern type Sentinel;            // sentinel type

     public issentinel S;                    // check for sentinel objects

     public extern special sentinel X;       // create a sentinel object

   Sentinels are a means to trigger automatic cleanup of an ordinary Q
data structure in the same fashion as the cleanup performed by some
built-in and external data types. Example:

     ==> def X = sentinel (puts "cleaning up...\n")

     ==> undef X
     cleaning up...

Mutable Sequences
-----------------

An important application of references are _mutable sequences_ which
can be implemented as tuples, lists or streams of references. As of
version 7.8, Q provides an auxiliary module `reftypes.q' which
implements some useful convenience functions for working with these
kinds of objects. Please note that this module is still experimental,
and has to be imported explicitly right now as it is not yet included in
the prelude.

   The following operations are provided to construct a reference tuple,
list or stream from a tuple, list or stream of its values, respectively.

     public reftuple Xs, reflist Xs, refstream Xs;

   The following functions work like the standard library functions
`mktuple', `mklist' and `mkstream', but construct a sequence of
references all initialized to the given value instead.

     public mkreftuple X N, mkreflist X N, mkrefstream X;

   All of these are also provided as "2D" variations to deal with
matrix-like sequences:

     public reftuple2 Xs, reflist2 Xs, refstream2 Xs;
     public mkreftuple2 X NM, mkreflist2 X NM, mkrefstream2 X;

   Note that `mkreftuple2' and `mkreflist2' take a pair of dimensions
`(N,M)' as the second parameter to denote the desired number of rows
`N' and columns `M'.

   The module also provides the following pseudo type-checking
functions. Please note that, for efficiency, these only look at the
first component to see whether it is a reference (or reference
sequence).

     public isreftuple Xs, isreflist Xs, isrefstream Xs;
     public isreftuple2 Xs, isreflist2 Xs, isrefstream2 Xs;

   Moreover, the module overloads the `get' and `put' functions so that
they work with an entire sequence at once. You can apply `get' to a
tuple, list or stream of references to obtain the tuple, list or stream
of its values, respectively, and you can invoke `put Xs Ys' to set the
references in `Xs' to the corresponding values in the sequence `Ys'.
This also works for nested sequences such as a list of lists of
references.

   A dereferencing index operator `Xs!!I' is provided as an
abbreviation for `get (Xs!I)':

     public (!!) Xs I @ (!);

   The `fill' function fills references with a given value, and
`putmap' implements a destructive variation of the standard library
`map' function:

     public fill Y X, putmap F X;

   These both work with either individual references or nested
sequences of references. Note that `putmap F X' applied to a reference
`X' updates `X' with `F (get X)', whereas `fill' just fills references
with a constant value. (Thus `fill X' does exactly the same as `putmap
(cst X)', but is implemented directly for efficiency.) CAVEAT: Please
note that `fill' takes the fill value as the _first_ parameter. This
allows for easier currying. For instance, you can quickly define
yourself a function to zero out a sequence as follows:

     zero = fill 0;

   The following examples show most of the operations in action, taking
a reference list as an example.

     ==> import reftypes             // needs to be imported explicitly

     ==> def Xs = reflist [1..10]    // create a new reference list

     ==> get (Xs!5)                  // get a member
     6

     ==> Xs!!5                       // syntactic sugar for the above
     6

     ==> put (Xs!5) 77               // change a member
     ()

     ==> Xs!5 := 77                  // syntactic sugar for the above
     ()

     ==> Xs!!5
     77

     ==> get Xs                      // get all members
     [1,2,3,4,5,77,7,8,9,10]

     ==> putmap (*2) Xs              // destructive map
     ()

     ==> get Xs
     [2,4,6,8,10,154,14,16,18,20]

     ==> fill 0 Xs                   // fill with given value
     ()

     ==> get Xs
     [0,0,0,0,0,0,0,0,0,0]

     ==> Xs := [1..10]               // update all values at once
     ()

     ==> get Xs
     [1,2,3,4,5,6,7,8,9,10]

   Tuples and streams work in an analogous fashion. Note that reference
streams are always fully memoized so that the reference values are
properly remembered when the stream is traversed for inspection and
updates. Infinite streams work, too, since only the finite part of the
stream which is accessed in the program ever gets evaluated. You must
take care, however, that you don't apply `fill', `put' or `putmap'
(which are strict operations) to the entire stream, but only to a
finite subsection of the stream. For instance:

     ==> def Xs = refstream {1..}

     ==> Xs!!5
     6

     ==> Xs!5 := 77
     ()

     ==> Xs!!5
     77

     ==> def Ys = take 10 Xs // apply putmap only to this finite subsequence

     ==> strict $ get Ys
     {1,2,3,4,5,77,7,8,9,10}

     ==> putmap (*2) Ys
     ()

     ==> strict $ get Ys
     {2,4,6,8,10,154,14,16,18,20}

     ==> Xs!!5
     154

   Note that at this point only the first 10 members of the stream have
actually been evaluated, the rest of the stream is still "thunked":

     ==> Xs
     {<<Ref>>,<<Ref>>,<<Ref>>,<<Ref>>,<<Ref>>,<<Ref>>,<<Ref>>,<<Ref>>,<<Ref>>,
     <<Ref>>,ref 11|lazy (map ref (iterate (+1) ((+1) 11)))}

   The `reftypes.q' module also contains some magic to make the
overloaded `get', `put', `fill' and `putmap' functions work with any
"indexed" reference container type providing a `vals', `list' or
`list2' function which gives access to the container's reference
components as a list. Examples of suitable container types are the
`Array', `Dict' and `HDict' types discussed in *Note Standard Types::.
For instance (taking the `Array' type, which provides a kind of
vector/matrix data structure with logarithmic lookup and modification
times, as an example):

     ==> import array reftypes

     ==> def A = array $ reflist [1..10]

     ==> get A
     [1,2,3,4,5,6,7,8,9,10]

     ==> A!5 := 99; get A
     ()
     [1,2,3,4,5,99,7,8,9,10]

     ==> putmap (*2) A; get A
     ()
     [2,4,6,8,10,198,14,16,18,20]

     ==> fill 0 A; get A
     ()
     [0,0,0,0,0,0,0,0,0,0]

     ==> A := [0.1,0.2..1.0]; get A
     ()
     [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0]

   2D containers a.k.a. matrices can be used in the same fashion:

     ==> def A = array2 $ reflist2 [1,2,3; 4,5,6]

     ==> get A
     [[1,2,3],[4,5,6]]

     ==> A!(1,1) := 55
     ()

     ==> A!!(1,1)
     55

     ==> get A
     [[1,2,3],[4,55,6]]

     ==> fill 0 A; get A
     ()
     [[0,0,0],[0,0,0]]

     ==> A := [0.1,0.2,0.3; 0.4,0.5,0.6]; get A
     ()
     [[0.1,0.2,0.3],[0.4,0.5,0.6]]

     ==> putmap (*10) A; get A
     ()
     [[1.0,2.0,3.0],[4.0,5.0,6.0]]

     ==> A!1 := [3,2,1]; get A
     ()
     [[1.0,2.0,3.0],[3,2,1]]


File: qdoc.info,  Node: Time Functions,  Next: Internationalization,  Prev: Expression References,  Up: Clib

12.15 Time Functions
====================

The functions described in this section (which need to be imported from
the `system' module) are used to return information about the active
timezone and the current time, and convert time values to various
formats. Also available are functions to measure cpu time and
high-resolution timers. These functions are in close correspondence with
the date and time functions of the C library.

   The calendar date and time functions use two different
representations for time values:

   * Simple time (denoted `T' in the sequel) is represented as the
     number of seconds elapsed since the "epoch", 00:00:00 on January 1,
     1970, UTC.

   * Broken-down time (denoted `TM') is encoded as a 9-tuple `(YEAR,
     MONTH, DAY, HOUR, MIN, SEC, WDAY, YDAY, ISDST)'. See the
     description of the `tm' struct in the `ctime'(3) manual page for
     more information.

   Three functions (`tzname', `timezone', `daylight') are provided to
return information about the current timezone and daylight savings
settings. The `gmtime' and `localtime' functions convert simple time to
broken-down time using UTC or the local timezone, respectively. The
functions `mktime' and `asctime' are used to convert a broken-down time
value into a simple time value or the standardized string
representation used by the `date'(1) program, respectively. The `ctime'
function combines `localtime' and `asctime'. For more flexible
formatting of time values, the `strftime' function can be used, which
takes as its first argument a format string as described on the
`strftime'(3) manual page.

     public extern tzname, timezone, daylight;
     public extern ctime T, gmtime T, localtime T;
     public extern mktime TM, asctime TM, strftime FORMAT TM;

   The components of broken-down time. can be accessed with the
following functions:

     public tm_year TM, tm_month TM, tm_day TM, tm_hour TM, tm_min TM,
       tm_sec TM, tm_wday TM, tm_yday TM, tm_isdst TM;

   On the current 32 bit systems time values must be representable as 4
byte integer values, which means that the available range usually goes
from `"Fri Dec 13 20:45:52 1901"' to `"Tue Jan 19 03:14:07 2038"'.
(This will probably be fixed before "time ends" on current UNIX systems
in January 2038, though.)

   Some examples:

     ==> tzname; timezone; daylight
     ("CET","CEST")
     -3600
     1

     ==> ctime time
     "Mon Mar 17 04:08:29 2003\n"

     ==> localtime time
     (103,2,17,4,8,41,1,75,0)

     ==> strftime "Hey it's %c" _
     "Hey it's Mon Mar 17 04:08:41 2003"

     ==> ctime (st_mtime (stat "README-Clib"))
     "Wed May  1 16:59:00 2002\n"

Measuring CPU Time
------------------

Two additional functions are provided for measuring cpu time. Note that
`clock' and `times' measure cpu time in different units, given by the
constants `CLOCKS_PER_SEC' and `CLK_TICK', respectively. The `times'
function returns a 5-tuple `(TOTAL, UTIME, STIME, CHILD_UTIME,
CHILD_STIME)'. See `times'(2) for details.

     public extern clock, times;

   For instance, you can calculate the cpu time in seconds it takes to
evaluate an expression by computing the difference between the `clock'
time at the end and at the beginning, divided by `CLOCKS_PER_SEC':

     ==> -(clock - (prd [1..10000] || clock)) / CLOCKS_PER_SEC
     0.14

   Note that the initial value and the resolution of the `clock' timer
is system-dependent. Usually the timer is initialized at process
creation time, but that is not guaranteed. Moreover, the value of
`clock' typically is a machine integer which wraps around after a
finite time interval; on 32 bit systems with `CLOCKS_PER_SEC =
1000000', as recommended by the POSIX standard, this happens
approximately every 72 minutes.

High-Resolution Timers
----------------------

On systems where the POSIX 1003.1-2001 timer extension is available (at
the time of this writing, this comprises most recent Linux and Unix
systems, but not Windows), the `system' module provides a number of
operations to deal with high-resolution timers. These will be useful for
realtime applications and other programs with strict timing
requirements.

     public extern nanotime ID, nanores ID;
     public extern nanosleep ID T, nanosleep_until ID T;

   Note that these operations differ from the builtin `time' and
`sleep' functions (see *Note Miscellaneous Functions::) in that they
use integer (unsigned 64 bit) time values specified in nanoseconds. The
`nanotime' function returns the current time in nanosecs, the `nanores'
function the resolution of the given timer (i.e., the smallest
measurable amount of time in nanosecs). The `nanosleep' function sleeps
for the specified amount of nanoseconds, while the `nanosleep_until'
function sleeps until the given (absolute) time arrives. In case of any
internal error condition, these functions will set `errno' accordingly.

   Just like the `sleep' function, the `nanosleep' and
`nanosleep_until' functions may wake up early if a signal is delivered
to the process. In this case `errno' will be set to `EINTR', and
`nanosleep' will return the remaining time until the planned wakeup,
while `nanosleep_until' will simply fail. If the sleep terminates
normally, `nanosleep' will return zero and `nanosleep_until' `()',
respectively.

   For each of the functions you have to specify an integer-valued timer
id. In the current implementation, the following timers are supported:

   * `CLOCK_REALTIME': The system clock, measured in nanosecs since the
     epoch. This will be the same time as returned by the builtin `time'
     function (up to rounding issues with the latter).

   * `CLOCK_MONOTONIC': Monotonic (non-decreasing) time since some
     unspecified starting point. This clock cannot be reset by the user
     and is thus to be preferred in realtime applications.

   * `CLOCK_PROCESS_CPUTIME', `CLOCK_THREAD_CPUTIME': Highres
     per-process and per-thread CPU timers. (Normally these are for
     timing purposes only and might be unreliable on SMP systems, see
     `clock_gettime'(3) for details.)

   Only `CLOCK_REALTIME' is guaranteed to exist (on systems which
implement the highres timers at all, that is). If a given clock is
unavailable on the host system, the corresponding id will be `()'.  (If
`CLOCK_REALTIME' is `()' then the highres timers are not supported at
all.)

   Moreover, if CPU timers are available, then the following
`process_cpu_clockid' and `thread_cpu_clockid' functions can be used to
return the clock id for the given process id (0 denotes the current
process) and the given thread, respectively:

     public extern process_cpu_clockid PID, thread_cpu_clockid THREAD;

   CAVEAT: While the highres timers nominally support nanoseconds
resolution, the actual resolutions depend on your system setup and will
typically be much coarser (for the system and monotonic clocks, they are
usually in the milliseconds range on current PCs). Also, be aware that
there are system latencies which might cause calls to `nanosleep' and
`nanosleep_until' to wake up late. You can use realtime scheduling
priorities to mitigate these effects to some extent, see *Note Realtime
Scheduling::, but some sources of latency will still remain.


File: qdoc.info,  Node: Internationalization,  Next: Filename Globbing,  Prev: Time Functions,  Up: Clib

12.16 Internationalization
==========================

As of Q 7.0, the following functions give access to the locale and
internationalization functions provided by the C library on most modern
operating systems, which let you deal with different character sets and
language-specific conventions. If you have `iconv' and `gettext' on
your system, interfaces to these functions will be provided as well. As
of Q 7.8, all of these need to be imported from the `system' module.

     public extern setlocale CATEGORY LOCALE;
     public extern localeconv;
     public extern nl_langinfo ITEM;

   The `setlocale' function sets the current locale of the program.
This function is usually invoked near the beginning of an application
to initialize locale-dependent processing. (The Q interpreter also
initializes the default locale at startup time, so you only have to
invoke this function yourself if you want to set a locale which is
different from the user's default locale, or if you have to change the
locale during execution of the program.) The `CATEGORY' parameter
determines which part of the locale is to be modified, and can be any
of the `LC_XXX' constants defined in the manifest constants section at
the beginning of this module. The `LOCALE' string must be a valid
locale value for the given category, or `""' to specify that the locale
is to be set from the corresponding user defaults. `LOCALE' can also be
`()' if you only want to query the current locale. In any case the
current locale value for the given category is then returned as a
string. For a description of the different locale categories please
refer to the `setlocale'(3) manpage.

   The `localeconv' function returns various numeric formatting
information about the current locale. The result is returned as a
tuple. Please refer to `clib.q' and the `localeconv'(3) manpage for
details about this operation. You'll rarely have to use the information
provided by this function directly, as the `strfmon' function (see
below) allows you to format numbers and monetary values in a
locale-dependent way.

   On SUSv2-compatible Unix systems, additional information about the
current locale is available using the `nl_langinfo' function. The
`ITEM' parameter is one of the symbolic constants defined in the
manifest constants section at the beginning of `clib.q' (see
`nl_langinfo'(3) for a description of these). The result of
`nl_langinfo' is always a string.

   The following functions help with the formatting of various
locale-dependent items:

     public extern strfmon FORMAT ARGS;
     public extern strcoll S1 S2, strxfrm S;
     public extern wcswidth S;
     public wcwidth C;

   The `strfmon' function provides locale-dependent formatting of
numbers and monetary values. See `strfmon'(3) for a description of the
syntax of the `FORMAT' string. `ARGS' is either a singleton value or a
tuple of floating point values, as required for the given format
string. Integers in the `ARGS' parameter are automatically converted to
floating point values.

   The `strcoll' function compares two strings in a locale-dependent
way; it returns <0, 0, >0 iff `S1' is, respectively, less than, equal
to, or greater than `S2'. The `strxfrm' function converts the given
string into a form which can be used for locale-based comparisons; by
comparing the transformed strings you always get the same results as by
checking the result of `strcoll'.

   The `wcswidth' function returns the actual number of columns
required to print a string, which may be different from the length of
the string if the string contains non-ASCII Unicode characters. If any
of the characters in the given string is non-printable, -1 is returned.
The `wcwidth' function does the same for single characters.

Iconv Interface
---------------

     public extern type IConv;
     public extern iconv_open TO FROM, iconv_close IC, iconv IC B;

   The `IConv' type represents conversion state objects for the `iconv'
function. Objects of this type are created with `iconv_open' which
takes two strings describing the desired target and source encodings as
arguments, and destroyed with the `iconv_close' function (this is also
done automatically when an `IConv' object is garbage-collected).

   The `iconv' function does the actual conversion. It takes as
arguments the current conversion state `IC' (an `IConv' object) and the
byte string `B' to be converted, and returns the converted byte string.
The second argument can also be `()' in which case iconv resets the
conversion to the initial state and returns the "shift" sequence
necessary to return to the initial state, if any (this will always be
the empty byte string in the case of a stateless encoding).

   If anything goes wrong during the conversion, instead of a single
byte string `iconv' returns a pair of byte strings `(C,D)' where `C' is
the successfully converted initial part of the sequence and `D' is the
remaining data, starting from the position in the source buffer `B'
where the error occurred. It is then the responsibility of the
application to check the `errno' value in order to determine what
exactly went wrong, see `iconv'(3) for details.

   Please note that the supported combinations of source and target
encodings depend on the system and particular `iconv' implementation.
Using GNU `iconv', you can obtain the list of all supported encodings
with the `iconv --list' command.

Gettext Interface
-----------------

     public extern textdomain DOMAIN, bindtextdomain DOMAIN DIR;
     public extern gettext MSGID, dgettext DOMAIN MSGID,
       dcgettext DOMAIN MSGID CATEGORY;
     public extern ngettext MSGID1 MSGID2 N, dngettext DOMAIN MSGID1 MSGID2 N,
       dcngettext DOMAIN MSGID1 MSGID2 N CATEGORY;

   These are simple wrappers for the corresponding C functions from the
GNU `gettext' library.

   The `textdomain' function selects the default domain to be used with
the `gettext'/`ngettext' functions. The `bindtextdomain' function
determines the base directory where the message files for the given
domain are to be found. Both functions return the current setting,
i.e., the current domain for `textdomain', and the current directory
for the given domain for `bindtextdomain'. To only query the current
setting without changing it, use `()' as the `DOMAIN' argument to
textdomain and the `DIR' argument to `bindtextdomain'.

   The `gettext' function retrieves a message from a message catalog
(determined with the `textdomain' and `bindtextdomain' functions) and
returns the translated string. The message to be translated is
specified as a string `MSGID'. If no translation can be found then
`gettext' returns `MSGID' unchanged.

   The other variations of `gettext' allow you to explicitly specify a
domain (usually the application name, which defaults to the value set
with the `textdomain' function) and a locale category (`LC_MESSAGES' by
default). The `ngettext' functions are similar, but provide a way to
translate plural forms. See the `gettext'(3) and `ngettext'(3) manual
pages for details.

   Note that the GNU `gettext' family of functions uses message
catalogs in a binary format, which can be constructed from a readable
description using the `msgfmt'(1) utility. Currently no special support
is provided for the automatic extraction of messages from Q scripts
using tools like `xgettext'(1). This will likely be fixed in future
releases, but for the time being you will have to prepare the `msgfmt'
input files manually, or use a custom script for that purpose.


File: qdoc.info,  Node: Filename Globbing,  Next: Regular Expression Matching,  Prev: Internationalization,  Up: Clib

12.17 Filename Globbing
=======================

Matching filenames against patterns with shell wildcards (`*', `?'
etc.), also known as filename "globbing", is implemented by the
following functions from `clib.q':

     public extern fnmatch PATTERN S; // check whether S matches PATTERN
     public extern glob PATTERN; // list of all filenames matching PATTERN

Examples
--------

     ==> map (fnmatch "*.q") ["clib.q","clib.c"]
     [true,false]

     ==> map (fnmatch "*.[cq]") ["clib.q","clib.c"]
     [true,true]

     ==> glob "*.[qc]"
     ["clib.c","clib.q","factor.q","globexamp.q","regexamp.q"]

     ==> glob "/h*"
     ["/home"]

   Only a single pattern is accepted by both `fnmatch' and `glob'.
However, it is a simple matter to provide your own rules which extend
the `clib' operations to lists of patterns:

     fnmatch [] S:String      = false;
     fnmatch [P:String|Ps] S:String
                              = fnmatch P S or else fnmatch Ps S;

     glob Ps:List             = rmdups (sort (<) (cat (map glob Ps)));

     rmdups []                = [];
     rmdups [X,X|Xs]          = rmdups [X|Xs];
     rmdups [X|Xs]            = [X|rmdups Xs] otherwise;

   Note that in our extension of the definition of `glob' we employ the
`sort' function (see *Note C Replacements for Common Standard Library
Functions::) to sort the resulting list, and then remove adjacent
duplicates (which could occur because a file may match more than one
pattern).

   Now let's see these definitions in action:

     ==> map (fnmatch ["*.q","*.c"]) ["clib.q","clib.c"]
     [true,true]

     ==> glob ["*.q","*.c","clib.*"]
     ["clib.c","clib.q","clib.so","factor.q","globexamp.q","regexamp.q"]


File: qdoc.info,  Node: Regular Expression Matching,  Next: Additional Integer Functions,  Prev: Filename Globbing,  Up: Clib

12.18 Regular Expression Matching
=================================

The `clib' module implements regular expression matching using the
POSIX `regcomp' and `regexec' functions. Regular expressions are
generally specified using the "extended" (`egrep'-like) syntax. `Clib'
divides the matching functions into two interfaces, the "high-level"
interface (`regex' function) and the "low-level" interface (`regmatch'
and friends). Both interfaces are used with a third group of "match
state" functions (`reg' and friends) which provide access to
information about the current match.

   For most purposes, the high-level `regex' function should be all
that is needed.  However, the low-level functions are provided so that
you can create your own specialized regular expression engines, should
you ever need to do so. In this case you might wish to take a look at
the definition of the `regex' function in `clib.q' and modify it to
suit your needs.

   Note that in accordance with POSIX matching semantics, all matching
functions search for an occurrence of the pattern in the target string,
rather than simply checking whether the whole string matches the given
pattern. If the latter functionality is required, you can use `^' and
`$' to tie the match to the beginning and the end of the string.

* Menu:

* High-Level Interface::
* Low-Level Interface::
* Match State Information::
* Basic Examples::
* Empty and Overlapping Matches::
* Splitting::
* Performing Replacements::
* Submatches::
* Nested Searches::


File: qdoc.info,  Node: High-Level Interface,  Next: Low-Level Interface,  Prev: Regular Expression Matching,  Up: Regular Expression Matching

12.18.1 High-Level Interface
----------------------------

The following function is implemented as a special form, with the
`EXPR' argument being passed unevaluated.

     public special regex ~OPTS ~REGEX ~S EXPR;

   The `regex' function evaluates, for each match of the given regular
expression (also called the "pattern") in the given string, the special
`EXPR' argument, and returns the collection of all results as a list,
in the order in which the matches were found. If no match is found, the
result list will be empty. The `EXPR' argument typically uses the match
state functions (see *Note Match State Information::) to retrieve the
current match information. You can also invoke the `regdone' function
(see *Note Low-Level Interface::) to escape from the search at any time.

   Matching generally proceeds from left to right, and if several
different substrings match at a given position, the longest match is
preferred. The matching process is controlled by means of the `OPTS'
string argument, which may contain zero or more of the following option
characters (the corresponding flags of the POSIX `regcomp'/`regexec'
functions are given in parentheses, where applicable):

`g'
     Match globally, i.e., find all non-overlapping occurrences.
     Otherwise only the first match is reported (if any).

`G'
     Like `g', but report overlapping matches as well.

`i'
     Do case-insensitive matches (`REG_ICASE').

`n'
     Do multi-line matches. This makes `^' and `$' match line beginnings
     and ends (besides beginning and end of the string), and makes `.'
     and lists `[...]' _not_ match the newline character, unless it is
     explicitly included in a list (`REG_NEWLINE').

`^'
     Do _not_ match `^' at the beginning of the string (`REG_NOTBOL').

`$'
     Do _not_ match `$' at the end of the string (`REG_NOTEOL').

   Abnormal error conditions such as bad regular expression syntax are
handled by returning an expression of the form `regerr MSG' where `MSG'
is a string describing the error. You may give an appropriate
definition of `regerr', or check for literal `regerr MSG' values, to
handle such error conditions in any way you like.

   The `regex' function is `clib''s primary interface to perform
regular expression matching, and should cover most usual applications.
You can find a number of examples showing how to use this function
below (see *Note Basic Examples::). The `regex' function itself is
implemented in Q using the low-level functions discussed in the
following section. You might wish to take a look at these if you need
to do something special which cannot be done with the `regex' function
(or cannot be done in an efficient manner).


File: qdoc.info,  Node: Low-Level Interface,  Next: Match State Information,  Prev: High-Level Interface,  Up: Regular Expression Matching

12.18.2 Low-Level Interface
---------------------------

The following functions are implemented in C using the POSIX regex
functions.

     public extern regmatch OPTS REGEX S, regnext, regdone;

   The `regmatch' function searches for the first (i.e., leftmost)
occurrence of the regular expression pattern in the given string, and
the parameterless `regnext' function searches for the next occurrence.
Both functions normally return a truth value indicating success or
failure (but see the remarks concerning error handling below). In case
of success, you can retrieve the current match using the match state
functions (see *Note Match State Information::).

   *IMPORTANT:* These functions have side-effects; they change the
internal match state which is accessed using the match state functions
(see *Note Match State Information::). This hidden state is provided
for your convenience, to spare you the trouble of having to pass
explicit parameters between the matching operations and the match state
functions. All match state information is maintained on an internal
stack, so that you can start a new search with the `regmatch' function
while another one is still in progress. (This also implies that you can
recursively invoke `regex' inside the `EXPR' argument of a `regex'
call.)

   The `OPTS' argument has the same meaning as for the `regex'
function, and error handling also works in the same fashion. That is,
in case of an abnormal error condition the `regmatch'/`regnext'
functions return a `regerr MSG' expression instead of a truth value.
See *Note High-Level Interface::.

   Note that the `regnext' function will always return `false', unless
one of the `g'/`G' (global search) options is specified in the
preceding `regmatch' call.

   To terminate a (global) search which is still in progress (i.e., has
not failed yet), you can call the `regdone' function, which always
returns `()'. Subsequent invocations of the match state functions will
then behave just like after a failing `regmatch'/`regnext' call. (After
a failed match, `regdone' is not needed; it will be invoked
automatically.)

   Some care is needed to ensure proper operation of the low-level
routines with multiple nested searches. As a general rule, each search
successfully started with `regmatch' _must_ be terminated either with a
failing `regnext' call, or by an invocation of the `regdone' function.
In both cases, after termination of the nested search, a subsequent
`regnext' will continue where `regmatch'/`regnext' left off when the
nested search was started.

   It is important to note that these rules also apply if the search is
_non-global_. The main rationale behind this behaviour is that, even if
one is only interested in the first match, one must still be able to
obtain information about any trailing unmatched text using the
`regstart'/`regskip' functions (see *Note Match State Information::),
and this information is only available _after_ a match has failed (or
was aborted using `regdone'). Thus the interface functions should always
behave consistently, no matter whether the `g'/`G' option was specified
or not. (In fact, the only difference `g'/`G' makes is that, if it is
omitted, then `regnext' will pretend that no next match is available
even if there is one.)

   If this sounds confusing to you, just stick with the high-level
`regex' function which takes care of all those messy details. A nested
matching example using `regex' is discussed in *Note Nested Searches::.


File: qdoc.info,  Node: Match State Information,  Next: Basic Examples,  Prev: Low-Level Interface,  Up: Regular Expression Matching

12.18.3 Match State Information
-------------------------------

These functions are typically invoked after `regmatch', `regnext', or
in the `EXPR' argument of `regex', to report information about the
current match.

     public extern regstart, regskip, reg N, regpos N, regend N, regs;

   The `reg' function returns the text of a match, `regpos' its
beginning in the target string (first character position), and `regend'
its end (first position _behind_ the match). These functions take an
integer argument denoting the "group" (a.k.a. parenthesized
subexpression) of the regular expression being matched. An argument of
`0' always refers to the whole match, `N > 0' to the text matched by
the `N'th group (counting opening parentheses from the left). The match
reported for a given group will always extend to the right as far as
possible ("longest match"), subject to the constraint that enclosing
groups will also prefer the longest match, and they always take
priority. Thus `reg 0' will extend to the right as far as possible,
`reg 1' will be the longest possible match for group 1 inside this
match, etc.

   If a group belongs to a repetitive pattern (patterns involving the
`*', `+' operators, etc.), then it may be matched more than once. In
this case, the text _last_ matched by the group is reported by the
`reg'/`regpos'/`regend' functions.

   If the group belongs to an optional part of the pattern (`*', `?',
different alternatives with `|'), then the group might not be matched at
all. For such unmatched groups, `regpos' and `regend' both return `-1'
and `reg' returns the empty string. These values are also returned for
_all_ groups (including `0') after a match has failed or has been
aborted using the `regdone' function.

   The `regs' function returns a list with the indices of all matched
groups (except `0').

   The parts of the target string which were _not_ matched can also be
accessed. This is done by means of the `regstart' and `regskip'
functions, which return, respectively, the position where the last
search for a match started, and the text from this position up to the
following match (or to the end of the string if the match failed).
These values can also be accessed after a failing or aborted match
(e.g., after `regex' has finished), which is useful to determine a
trailing unmatched portion in the input string.


File: qdoc.info,  Node: Basic Examples,  Next: Empty and Overlapping Matches,  Prev: Match State Information,  Up: Regular Expression Matching

12.18.4 Basic Examples
----------------------

Please refer to your UNIX manual for a description of the regular
expression syntax. (To get a start, take a look at the `egrep'(1)
manual page; most of what it has to say about (extended) regular
expressions should be applicable here too. There are also some good
books on the subject.)

   So let's consider some elementary examples first. Here's how we can
find all occurrences of "identifiers" (defined here as sequences of
letters and digits, starting with a letter) in a string:

     ==> regex "g" "[A-Za-z][A-Za-z0-9]*" "1var foo 99 BAR $%&" (reg 0)
     ["var","foo","BAR"]

   The first argument to `regex' is always the option string; the `"g"'
in our example indicates a "global" search, i.e., a search for all
(non-overlapping) matches of the given pattern. The second argument is
the regular expression pattern to be matched, and the third argument is
the string to be searched for matches against the pattern. The fourth
argument is the expression to be evaluated for each match; in our case,
`reg 0' is used to simply retrieve the match. Actually, we could
provide any appropriate expression here, e.g., we might use `sprintf'
to add some fancy formatting:

     ==> regex "g" "[A-Za-z][A-Za-z0-9]*" "1var foo 99 BAR $%&" \
     (sprintf "MATCH: %s" (reg 0))
     ["MATCH: var","MATCH: foo","MATCH: BAR"]

   What happens if the pattern contains an error, such as an unmatched
bracket?  Let's see:

     ==> regex "g" "[A-Za-z][A-Za-z0-9*" "1var foo 99 BAR $%&" (reg 0)
     regerr "Unmatched [ or [^"

   Reasonably enough, `regex' returns an error message.

   The `regex' function always enumerates matches from left to right,
and only reports at most one match for each position in the input
string. If more than one substring matches at a given position, the
longest match is preferred. Thus in the above example, e.g., `foo' is
matched, and not the individual characters `f' and `o' which by
themselves also match the given pattern.

   Note that with the `g' option only non-overlapping matches are
reported; if we want _all_ occurrences (even overlapping ones) then we
specify the `G' option instead:

     ==> regex "G" "[A-Za-z][A-Za-z0-9]*" "1var foo 99 BAR $%&" (reg 0)
     ["var","ar","r","foo","oo","o","BAR","AR","R"]

   And if we are only interested in the first match, we simply omit the
`g'/`G' option character:

     ==> regex "" "[A-Za-z][A-Za-z0-9]*" "1var foo 99 BAR $%&" (reg 0)
     ["var"]

   So far, so good. After these basics, let's reconsider our original
search expression:

     ==> regex "g" "[A-Za-z][A-Za-z0-9]*" "1var foo 99 BAR $%&" (reg 0)
     ["var","foo","BAR"]

   We can simplify the pattern somewhat if we use the `i' option to
specify a case-independent search:

     ==> regex "gi" "[a-z][a-z0-9]*" "1var foo 99 BAR $%&" (reg 0)
     ["var","foo","BAR"]

   Another way to write this pattern uses the predefined "character
classes" `[:alpha:]' and `[:alnum:]'; these constructs have the
advantage that they are portable, i.e., they work in different
character sets and locales.

     ==> regex "g" "[[:alpha:]][[:alnum:]]*" "1var foo 99 BAR $%&" \
     (reg 0)
     ["var","foo","BAR"]

   So now we know how we can find identifiers in a string. But what if
we have to check whether a given string _is_ an identifier? For this
purpose, we must match the string _as a whole_ against the pattern. We
can do this by tying our pattern to the beginning and end of the string
using the `^' and `$' "anchors". (The `g' option is not required here,
since we are only looking for a single match.)

     ==> regex "" "^[[:alpha:]][[:alnum:]]*$" "foo" (reg 0)
     ["foo"]

     ==> regex "" "^[[:alpha:]][[:alnum:]]*$" "1var" (reg 0)
     []

   If we only want a truth value, we can simply test the size of the
result list; in this case, any expression argument will do:

     ==> 1=#regex "" "^[[:alpha:]][[:alnum:]]*$" "foo" ()
     true

     ==> 1=#regex "" "^[[:alpha:]][[:alnum:]]*$" "1var" ()
     false


File: qdoc.info,  Node: Empty and Overlapping Matches,  Next: Splitting,  Prev: Basic Examples,  Up: Regular Expression Matching

12.18.5 Empty and Overlapping Matches
-------------------------------------

Empty matches are permitted as well, subject to the constraint that at
most one match is reported for each position (which also prevents
looping). And of course an empty match will only be reported if nothing
else matches. For instance:

     ==> regex "g" "" "foo" (regpos 0)
     [0,1,2,3]

     ==> regex "g" "o*" "foo" (regpos 0)
     [0,1,3]

   (The usefulness of such constructs might be questionable, but see
*Note Performing Replacements::, for a reasonable example.)

   As already mentioned, only non-overlapping matches are reported with
the `g' option. However, occasionally it might be useful to also
determine overlapping matches, and for this purpose the `G' option is
provided. For instance, suppose that we want to produce a table showing
which letters follow another given letter in a text. The first step is
to produce a list of all pairs of adjacent letters, and since we really
need _all_ pairs here, we employ the `G' option:

     ==> regex "G" "[[:alpha:]]{2}" "silly" (reg 0!0,reg 0!1)
     [("s","i"),("i","l"),("l","l"),("l","y")]

   Now it is an easy matter to collect the desired information using,
e.g., a dictionary and a little bit of "list voodoo:"

     ==> var add = \D (X,Y).update D X (insert (D!X) Y)

     ==> def D = foldl add (mkdict emptyset (map fst _)) _

     ==> zip (keys D) (map list (vals D))
     [("i",["l"]),("l",["l","y"]),("s",["i"])]


File: qdoc.info,  Node: Splitting,  Next: Performing Replacements,  Prev: Empty and Overlapping Matches,  Up: Regular Expression Matching

12.18.6 Splitting
-----------------

In previous examples we saw how we can tokenize a string by matching its
constituents (see *Note Basic Examples::). Sometimes one also has to go
the other way round, i.e., split a string into arbitrary tokens
separated by "non-tokens". E.g., we might want to split a string into
tokens delimited with whitespace, which can be described by an
expression like `"[ \t\n]+"', or, in a locale-independent fashion, using
`"[[:space:]]+"'. To do this with the `regex' function, we must find
all text that is _between_ matches, and any trailing text after the
last matched delimiter. The `regskip' function lets us do this as
follows:

     ==> regex "g" "[[:space:]]+" "The little\t brown\n fox." regskip \
     ++ [regskip]
     ["The","little","brown","fox."]

   Here the `regskip' expression argument to `regex' is used to access
the intervening tokens, and the final `++ [regskip]' appends the last
token.

   We can use this method to define our own little regular expression
tokenizing function as follows:

     regsplit OPTS REGEX S = regex OPTS REGEX S regskip ++ [regskip];

   With this definition, the above example works as follows:

     ==> regsplit "g" "[[:space:]]+" "The little\t brown\n fox."
     ["The","little","brown","fox."]

   We could also omit the `g' option, in which case only the first
delimiter match is used as a splitting point:

     ==> regsplit "" "[[:space:]]+" "The little\t brown\n fox."
     ["The","little\t brown\n fox."]

   Splitting with an empty or optional pattern also works as expected:

     ==> regsplit "g" "" "some text"
     ["","s","o","m","e"," ","t","e","x","t",""]

     ==> regsplit "g" " ?" "some text"
     ["","s","o","m","e","","t","e","x","t",""]


File: qdoc.info,  Node: Performing Replacements,  Next: Submatches,  Prev: Splitting,  Up: Regular Expression Matching

12.18.7 Performing Replacements
-------------------------------

A similar idea is also used to replace matches with new text. You can
define a substitution function, which replaces each match with the
result of an expression, as follows:

     special regsub ~OPTS ~REGEX ~S EXPR;
     regsub OPTS REGEX S EXPR = strcat (regex OPTS REGEX S (regskip++EXPR))
                                ++ regskip;

   For instance:

     ==> regsub "g" "[[:alpha:]][[:alnum:]]*" "1var foo 99 BAR $%&" \
     (sprintf "-*-%s-*-" (reg 0))
     "1-*-var-*- -*-foo-*- 99 -*-BAR-*- $%&"

   As usual, if we only want to replace the first match, we can omit
the `g' option:

     ==> regsub "" "[[:alpha:]][[:alnum:]]*" "1var foo 99 BAR $%&" \
     (sprintf "-*-%s-*-" (reg 0))
     "1-*-var-*- foo 99 BAR $%&"

   And here's an example showing empty and optional patterns at work:

     ==> regsub "g" "" "some text" " "
     " s o m e   t e x t "

     ==> regsub "g" " ?" "some text" ":"
     ":s:o:m:e::t:e:x:t:"


File: qdoc.info,  Node: Submatches,  Next: Nested Searches,  Prev: Performing Replacements,  Up: Regular Expression Matching

12.18.8 Submatches
------------------

It is also possible to access the parts of matches, called "groups",
which are defined by parenthesized subexpressions of the regular
expression. Groups are numbered starting at 1, counting opening
parentheses from the left. The text matched by a group can be accessed
using `reg N' where `N' is the group number.

   For instance, suppose we want to parse environment lines, such as
those returned by the shell's `set' command. We assume that each line
with a definition looks like `VARIABLE=VALUE', which can be described
by the pattern `"^([^=]+)=(.*)$"', in which group 1 denotes the
variable and group 2 the value. Thus we can obtain the parts of a
definition using a `regex' search like the following:

     ==> regex "" "^([^=]+)=(.*)$" "VARIABLE=VALUE" (reg 1,reg 2)
     [("VARIABLE","VALUE")]

   We can turn this into an `env' function which returns the current
environment as a list of (variable,value) pairs as follows. Here, we use
the `fget' function to read in the whole environment as a single string
from a pipe opened on the `set' command. The environment string is then
parsed using the `n' option of regex, which makes the `^' and `$'
anchors match the beginning and end of each line, respectively.

     env         = regex "gn" "^([^=]+)=(.*)$" envget (reg 1,reg 2);

     envget      = fget (popen "set" "r");

   Given these definitions, we can now get hold of the whole
environment and, e.g., turn it into a dictionary which can be accessed
in a convenient fashion using the functions from the `dict.q' standard
library module:

     ==> var envdict = dict env

     ==> envdict!"SHELL"
     "/bin/bash"

   When a pattern consists of multiple alternatives, such as
`"(foo)|(bar)"', then some of the groups might participate in a match,
while others don't. You can check whether a group matched by testing the
corresponding `regpos' value. If the value is nonnegative, then the
group matched; otherwise (the value is `-1') it didn't. You can also
use the `regs' function to determine the list of all groups that
matched. For instance:

     ==> regex "" "(foo)|(bar)" "foo" (regpos 1,regpos 2)
     [(0,-1)]

     ==> regex "" "(foo)|(bar)" "bar" regs
     [[2]]

   So here's a simple way to implement a lexical analyzer which
distinguishes between different kinds of tokens. For example, let us
tokenize identifiers and integers, skip whitespace, and return all
remaining characters as literals.

     def SYN = "([[:alpha:]][[:alnum:]]*)|(-?[[:digit:]]+)|([^[:space:]])";
     def TOK = (none,ident,num,id);

     toks S  = regex "g" SYN S ((TOK!(hd regs)) (reg (hd regs)));

   Note that the literal characters are mapped to the standard library
`id' function, the identity operation. Hence these characters are
represented by themselves, whereas other tokens are denoted by
appropriate constructor terms, `ident S' and `num S' in our example,
which contain the actual text of the token as arguments. Let's see this
in action:

     ==> toks "foo -99; bar 99;"
     [ident "foo",num "-99",";",ident "bar",num "99",";"]

   The token sequence is now ready to be processed by a higher-level
routine such as a parser, but this is another story which will be told
another time ...

   Yet another way to employ submatches are the "back references". You
can specify that a group is to be repeated by using the notation `\N',
where `N' is a single digit between 1 and 9, in the regular expression
argument. For instance, `regex "g" "([ab]+)\\1"' will match all
substrings which have two consecutive identical sequences of `a''s and
`b''s.

   Back references add considerably to the matching complexity (some
matches may take exponential time) and hence should be avoided. In
fact, it can be shown that regular expression matching with back
references is NP-complete.


File: qdoc.info,  Node: Nested Searches,  Prev: Submatches,  Up: Regular Expression Matching

12.18.9 Nested Searches
-----------------------

Regular expression searches can also be nested, by invoking the `regex'
function recursively in the expression argument of another `regex'
call. This is useful, e.g., if `regex' is used to tokenize a string, and
we want to further analyze the individual tokens. We could do this by
mapping a `regex' call to the finished token list, but it may be more
efficient to perform the second search right away on each individual
token. Here's an abstract example, which finds all substrings delimited
by `c''s, and counts the number of consecutive `a''s and `b''s in them:

     ==> regex "g" "[^c]+" "abbacbaaca" \
     (regex "g" "a+|b+" (reg 0) (reg 0!0,#reg 0))
     [[("a",1),("b",2),("a",1)],[("b",1),("a",2)],[("a",1)]]

   There's one caveat with nested searches: The current `regex'
implementation has the somewhat annoying but hardly avoidable limitation
that it is _not_ possible to "return" to the match state of an
enclosing `regex' call after a recursive call to `regex'. Therefore all
match state information must be retrieved _before_ the recursive call,
and must then either be processed right away or saved in local
variables for later use.


File: qdoc.info,  Node: Additional Integer Functions,  Next: C Replacements for Common Standard Library Functions,  Prev: Regular Expression Matching,  Up: Clib

12.19 Additional Integer Functions
==================================

The Q interpreter implements basic integer arithmetic using the GMP (GNU
multiprecision) library. With the `clib' module you also get some of the
more advanced GMP routines, namely integer powers and roots, and various
number-theoretic functions. (These are all included in the prelude.)

12.19.1 Powers and Roots
------------------------

     public extern pow M N, root M N, intsqrt M;

   These functions compute (exact) powers and (integer parts of) roots
of integers. The results are always integers; roots are truncated to
the largest integer below the exact root value. The `pow' function
computes the `N'th power of `M' (`N >= 0'), `root' the integer part of
the `N'th root of `M' (`M >= 0', `N > 0'), and `intsqrt' the integer
part of the square root of `M >= 0'.

     public extern powmod K M N, invmod K M;

   The `powmod' function returns the (smallest nonnegative) `N'th power
of `M' modulo `K', where `K' must be nonzero. This function is much
more efficient than `pow M N mod K' if `N' is large. The `invmod'
function computes the (smallest positive) multiplicative inverse of `M'
modulo `K' (if it exists, i.e. if `M' is nonzero and relatively prime
to `K'). In other words, `invmod K M' returns a number `N' in the range
`1..K-1' such that `M*N = N*M = 1' modulo `K'.

12.19.2 Prime Test
------------------

     public extern isprime N;

   The `isprime' function implements the Miller-Rabin algorithm for
probabilistic prime testing. If `isprime N' (where `N > 1' is an
integer) returns `true' or `false', then `N' is surely prime or
composite, respectively. If the function fails, then the number is
composite with a probability of only `(1/4)^ISPRIME_REP', where the
`ISPRIME_REP' variable denotes the number of repetitions of the
probabilistic prime test.

   To avoid the overhead of checking `ISPRIME_REP' each time the
`isprime' function is invoked, `isprime' determines the value of the
variable once, at the time of its first invocation. Thus, if you set
this value, make sure to set it _before_ you call `isprime' for the
first time. Reasonable `ISPRIME_REP' values vary from 5 to 10; if the
variable is not set when `isprime' is first called, a default value of
5 is used, which means that the probability of `isprime' failing on a
composite will be 1/1024.

   When using this function, your script should provide an appropriate
default definition which handles the case that the `isprime' operation
of this module fails. There are basically three different ways to cope
with a failing `isprime' call. First, you can make your program abort,
e.g., using an "error rule" like the following:

     isprime N:Int = error "Prime test failed!";

   Given that the test will probably not fail very often, this might be
a viable option - you can always run your script again and hope that it
finishes successfully, since the probabilistic test will choose
different random parameters each time it is invoked.

   Second, you can pretend that the prime test succeeded by providing
the definition:

     isprime N:Int = true;

   With this definition there will be a small probability that you
mistake a composite for a prime, but for some algorithms this might be
tolerable.

   Third, you can play safe by providing your own primality test which
catches those few cases when the probabilistic test fails. For instance,
employing the usual trial division method:

     isprime N:Int = true if N = 2;
                   = false if N and 1 = 0; // even number
                   = try_div 3 (intsqrt N) N otherwise;

     try_div L M N = true if L > M;
                   = false if N mod L = 0;
                   = try_div (L+2) M N otherwise;

   Of course such an exhaustive search will take a _long_ time for big
numbers, but this might be acceptable if the default rule is not
invoked very often.

12.19.3 Other Number-Theoretic Functions
----------------------------------------

     public extern gcd M N, lcm M N;

   These are the usual greatest-common-divisor and least-common-multiple
functions. Both `gcd' (which is only defined if at least one of the
arguments is nonzero) and `lcm' (which returns zero if either argument
is zero) always return a nonnegative integer (note that older GMP
versions returned the sign of the product of `N' and `M' in the lcm
value, but this is no longer the case).

     public extern remove_factor M N;

   The `remove_factor' function is used to help in factoring. It
returns a pair `(K,R)' consisting of the multiplicity `K' of `N' in `M'
(i.e., the maximum `L' such that `pow N L' divides `M') and the
remainder `R' which has all `N' factors removed, i.e., `R = M div pow N
K'. This function requires that `M' is nonzero and `N' positive.

     public extern jacobi M N;

   This function computes the "Jacobi symbol" `M' over `N', which can
be used to find quadratic residues of an odd prime. `M' can be an
arbitrary integer, but `N' must be positive; the value of the function
is always 1, 0 or -1, where the value 0 indicates that `gcd M N > 1'.
In the case of an odd prime `N', we have that `jacobi M N = 1' iff `M'
is a nonzero quadratic residue modulo `N'.

12.19.4 Examples
----------------

Determine all invertible numbers modulo 9 and their corresponding
inverses:

     ==> filter (\(X,Y).isnum Y) (zip [1..8] (map (invmod 9) [1..8]))
     [(1,1),(2,5),(4,7),(5,2),(7,4),(8,8)]

   Check the result:

     ==> map (\(X,Y).X*Y mod 9) _
     [1,1,1,1,1,1]

   Find all (nonzero) quadratic residues modulo 7:

     ==> filter (\X.jacobi X 7 = 1) [1..6]
     [1,2,4]

   Decompose 858330 into prime factors (brute force):

     ==> def N = 858330, P = filter isprime [2..N]

     ==> filter (\(X,Y).Y>0) (zip P (map (fst.remove_factor N) P))
     [(2,1),(3,3),(5,1),(11,1),(17,2)]

   Well, that's probably not the right way to do it. So here's a
reasonable factorization algorithm using `remove_factor'. This method
is a _lot_ faster - but still not fast enough for code breaking.

     factor 0          = [];
     factor N:Int      = factor (-N) if N<0;
                       = [(2,K)|factor_from 3 M]
                             where (K,M) = remove_factor N 2
                             if N and 1 = 0; // even number
                       = factor_from 3 N
                             otherwise;

     factor_from P N   = [] if P > N;
                       = [(P,K)|factor_from (P+2) M]
                             where (K,M) = remove_factor N P
                             if N mod P = 0; // P divides N (must be prime)
                       = factor_from (P+2) N
                             otherwise; // not a factor, try the next one

   For instance, try the following:

     ==> factor 807699854836875
     [(3,1),(5,4),(7,2),(11,5),(13,2),(17,1),(19,1)]

   Check the result:

     ==> prd (map (\(X,Y).pow X Y) _)
     807699854836875


File: qdoc.info,  Node: C Replacements for Common Standard Library Functions,  Prev: Additional Integer Functions,  Up: Clib

12.20 C Replacements for Common Standard Library Functions
==========================================================

Last but not least, the `clib' module also provides "built-in"
replacements for various standard library functions:

     public extern stdlib::append Xs Y, stdlib::cat Xs, stdlib::mklist X N,
       stdlib::nums N M, stdlib::numsby K N M, stdlib::reverse Xs,
       tuple::tuplecat Xs;

     public extern string::chars S, string::join DELIM Xs,
       string::split DELIM Xs, string::strcat Xs;

   These functions are _much_ more efficient, both in running time and
memory requirements, than the "standard" definitions in `stdlib.q' and
`string.q', and improve the performance of many common list and string
processing tasks. In some cases, the provided operations are even
several orders of magnitude faster. In particular, the standard
definitions of the `strcat' and `tuplecat' operations are _very_ slow
in comparison, because they are implemented using repeated
concatenation and hence take quadratic running time. In contrast, all
functions provided here are guaranteed to run in linear time.

   A faster sorting routine is provided as well:

     public extern sort P Xs;

   This function works like the `msort' and `qsort' operations provided
by the `sort.q' module, but is implemented using the quicksort routine
from the C library. Note that, in difference to `msort'/`qsort', the
`sort' function does not perform stable sorting, so you still have to
use `msort' or `qsort' if this feature is required.


File: qdoc.info,  Node: Q Language Grammar,  Next: Using Q,  Prev: Clib,  Up: Top

Appendix A Q Language Grammar
*****************************

This appendix summarizes the syntactical rules of the Q language. Please
refer to *Note Lexical Matters::, for a detailed discussion of lexical
matters.  The syntax rules are given in BNF with the following
extensions:

   * `{ }' denotes repetition; the enclosed elements may be repeated
     zero or more times.

   * `[ ]' denotes optional parts; the enclosed elements may be omitted.

   The left-hand and right-hand sides of syntax rules are separated
with a colon, and the `|' symbol denotes different alternatives.
Identifiers stand for nonterminal grammar symbols, and terminal symbols
are enclosed in single quotes.

   In order to keep the grammar to a reasonable size, it does _not_
specify the precedence of operator symbols. Expressions are parsed
according to the precedence and associativity of operator symbols
specified in *Note Built-In Operators::. A `=' symbol occuring
unparenthesized in an equation separates left- and right-hand side.

     script                  : {declaration|definition}

     declaration             : unqualified-import
                             | qualified-import

                             | prefix headers ';'
                             | [scope] 'type' unqualified-identifier
                               [':' identifier] ['=' sections] ';'
                             | [scope] 'extern' 'type' unqualified-identifier
                               [':' identifier] ['=' sections] ';'
                             | [scope] 'type' qualified-identifier
                               ['as' unqualified-identifier] ';'
                             | [scope] 'type' unqualified-identifier
                               '==' identifier ';'

                             | '@' ['+'|'-'] unsigned-number

     unqualified-import      : 'import' module-spec {',' module-spec} ';'
                             | 'include' module-spec {',' module-spec} ';'

     qualified-import        : 'from' module-spec 'import' [symbol-specs] ';'
                             | 'from' module-spec 'include' [symbol-specs] ';'

     module-spec             : module-name ['as' unqualified-identifier]
     module-name             : unqualified-identifier
                             | string

     symbol-specs            : symbol-spec {',' symbol-spec}

     symbol-spec             : unqualified-identifier ['as' unqualified-identifier]
                             | unqualified-opsym ['as' unqualified-opsym]

     prefix                  : scope
                             | [scope] modifier {modifier}

     scope                   : 'private'|'public'

     modifier                : 'const'|'special'|'extern'|'var'|'virtual'

     headers                 : header {',' header}

     header                  : unqualified-identifier '=' expression
                             | unqualified-identifier
                               {['~'] variable-identifier}
                             | qualified-identifier
                               {['~'] variable-identifier}
                               ['as' unqualified-identifier]
                             | '(' unqualified-opsym ')'
                               {['~'] variable-identifier}
                               ['@' precedence]
                             | '(' qualified-opsym ')'
                               {['~'] variable-identifier}
                               ['@' precedence]
                               ['as' unqualified-opsym]

     precedence              : unsigned-number
                             | '(' operator ')'

     sections                : section {'|' section}

     section                 : [prefix] headers

     definition              : expression
                               lqualifiers '=' expression qualifiers ';'
                               {lqualifiers '=' expression qualifiers ';'}
                             | 'def' expression '=' expression
                               {',' expression '=' expression} ';'
                             | 'undef' identifier {',' identifier} ';'

     lqualifiers             : [qualifier {qualifier} ':']

     qualifiers              : {qualifier}

     qualifier               : condition
                             | where-clause

     condition               : 'if' expression
                             | 'otherwise'

     where-clause            : 'where' expression '=' expression
                               {',' expression '=' expression}

     expression              : identifier
                             | 'var' unqualified-identifier
                             | variable-identifier ':' identifier
                             | number
                             | string
                             | expression expression
                             | unary-op expression
                             | expression binary-op expression
                             | 'if' expression 'then' expression
                               ['else' expression]
                             | '\' expression { expression } '.' expression
                             | '(' [element-list|enumeration|comprehension] ')'
                             | '[' [element-list|enumeration|comprehension] ']'
                             | '{' [element-list|enumeration|comprehension] '}'
                             | '(' op ')'
                             | '(' expression binary-op ')'
                             | '(' binary-op expression ')'

     element-list            : expression-list [',' | ';' | '|' expression]

     enumeration             : expression-list '..' [expression]

     comprehension           : expression ':' expression-list

     expression-list         : expression {',' expression}
                             | expression {';' expression}

     identifier              : unqualified-identifier
                             | qualified-identifier

     qualified-identifier    : module-identifier '::'
                               unqualified-identifier
     unqualified-identifier  : variable-identifier
                             | function-identifier
                             | type-identifier

     module-identifier       : letter {letter|digitsym}
     type-identifier         : letter {letter|digitsym}
     variable-identifier     : uppercase-letter {letter|digitsym}
                             | '_'
     function-identifier     : lowercase-letter {letter|digitsym}

     op                      : unary-op|binary-op

     unary-op                : opsym
     binary-op               : opsym|'and' 'then'|'or' 'else'

     opsym                   : unqualified-opsym
                             | qualified-opsym

     qualified-opsym         : module-identifier '::'
                               unqualified-opsym
     unqualified-opsym       : function-identifier
                             | punctsym {punctsym}

     number                  : ['-'] unsigned-number
     unsigned-number         : '0' octdigitseq
                             | '0x' hexdigitseq
                             | '0X' hexdigitseq
                             | digitseq ['.' [digitseq]] [scalefact]
                             | [digitseq] '.' digitseq [scalefact]

     digitseq                : digit {digit}
     octdigitseq             : octdigit {octdigit}
     hexdigitseq             : hexdigit {hexdigit}

     scalefact               : 'E' ['-'] digitseq
                             | 'e' ['-'] digitseq

     string                  : '"' {char} '"'

     letter                  : uppercase-letter|lowercase-letter
     uppercase-letter        : 'A'|...|'Z'|UPPERCASE UNICODE LETTER
     lowercase-letter        : 'a'|...|'z'|'_'|LOWERCASE UNICODE LETTER
     digitsym                : '0'|...|'9'|UNICODE DIGIT
     punctsym                : UNICODE PUNCTUATION CHARACTER

     digit                   : '0'|...|'9'
     octdigit                : '0'|...|'7'
     hexdigit                : '0'|...|'9'|'a'|...|'f'|'A'|...|'F'

     char                    : ANY CHARACTER BUT NEWLINE AND `"'


File: qdoc.info,  Node: Using Q,  Next: C Language Interface,  Prev: Q Language Grammar,  Up: Top

Appendix B Using Q
******************

The following discussion assumes that you have already installed the Q
programming system on your machine. In particular, you should make sure
that the programs `q' and `qc' have been copied to a directory which is
searched for executables, and that the standard library scripts have
been installed properly.

   You may also refer to the UNIX manual page q(1) for a brief
description of the Q programs.

   The Q programming system consists of two main parts: the compiler and
the interpreter. The compiler is used to translate Q scripts into a
binary format, a so-called "bytecode" file, which is then executed by
the interpreter. Since the interpreter invokes the compiler
automatically when you invoke it with a source script, you usually do
not have to care about the compiler yourself. But if you wish you can
also run compiler and interpreter separately.

* Menu:

* Running Compiler and Interpreter::
* Command Language::
* Setting up your Environment::
* Running Scripts from the Shell::
* Translating Scripts to C::


File: qdoc.info,  Node: Running Compiler and Interpreter,  Next: Command Language,  Prev: Using Q,  Up: Using Q

B.1 Running Compiler and Interpreter
====================================

Here is how you normally run a script with the interpreter:

     q [OPTIONS] [ FILE | - ] [ARGUMENT ...]

   The interpreter will then start up, display its prompt, and wait for
you to type some expressions or other commands (see *Note Command
Language::). You can exit the interpreter by typing `quit' (which
invokes the built-in `quit' function), or by entering the end-of-file
character at the beginning of the input line.

   The first non-option argument denotes either the main script or a
bytecode file. You can also specify `-' here to indicate that the main
script is to be read from standard input. At most one source or code
file may be specified; the remaining command line arguments can be
accessed in the interpreter by means of the built-in `ARGS' variable.
You can also use the `--' option at any point the argument list to
indicate that the interpreter should stop its option processing and
pass on the remaining arguments unchanged, which is useful if the Q
script takes option parameters itself.

   If the given file is in bytecode format, it is loaded immediately.
Otherwise, the interpreter invokes the compiler to translate the source
script to a temporary code file. Then, if the source script was
translated successfully, the interpreter loads the generated code file
and you can start typing in expressions and commands. (After the code
file has been loaded by the interpreter, it is no longer needed and is
discarded immediately. If you want to keep the code file, you must
invoke the compiler separately, see below.)

   Scripts and code files which have no absolute path specified are
searched for on Q's library path (as given by the `QPATH' shell
environment variable or with the `--path' option, see below). As with
the `PATH' environment variable, the individual directories are
separated with a colon `:' (semicolon `;' on DOS/Windows systems). The
directories are searched in the indicated order; if the file cannot be
found on the search path, an attempt is made to open it by its name.
You should include the `.'  directory in the search path if you wish to
search the current directory prior to other locations. The default
search path is usually something like `.:/usr/share/q/lib:/usr/lib/q'
which causes compiler and interpreter to first search the current
directory, and then other (system-dependent) locations where "library
scripts" are kept.

   Note that in difference to shell path searches, the Q interpreter
will also search for relative pathnames involving a directory prefix
(other than `~', `.' and `..') in subdirectories. Thus, e.g., given the
search path `.:/usr/share/q/lib:/usr/lib/q', the module `foo/bar.q'
will be searched for under `./foo/bar.q', `/usr/share/q/lib/foo/bar.q'
and finally `/usr/lib/q/foo/bar.q'.

   By default, the interpreter automatically includes the `prelude.q'
script (which will be searched for on the library path as usual) as the
first module in your program. The prelude provides the definitions which
are available without explicit import or include; normally it is used to
include most of the standard library, see *Note Scripts and Modules::.
You can also suppress the automatic inclusion of the prelude with the
`--no-prelude' compiler option, see below, or provide your own, see
*Note Setting up your Environment::.

   The interpreter can also be invoked with no arguments at all, in
which case an empty script is assumed. This means that only the built-in
operations of the Q language will be available, as well as the
definitions in the `prelude.q' script (unless the `--no-prelude' option
has been specified). (If you want to specify an empty script while
still supplying arguments, specify `""' as the script name.)

   If you want to avoid recompilations of large scripts, you can also
run compiler and interpreter separately. For this purpose, you first
invoke the compiler as described below to translate your script(s) to a
bytecode file. Then you can invoke the interpreter, specifying the name
of the bytecode file (normally `q.out', unless another code file name
has been specified with the `-o' compiler option). The interpreter can
then load the bytecode file and start execution immediately.

   The compiler is executed as follows:

     qc [OPTIONS] [ FILE | - ] ...

   It compiles the specified files and writes the resulting bytecode to
the code file, normally `q.out'. Note that multiple source files may be
specified. The first file denotes the main script, whose namespace
defines the global scope; the name of the main script can also be `-'
to indicate that the script is to be read from standard input, or it
can be missing or be specified as the empty string `""', to indicate an
empty main script. The remaining source files are additional scripts to
be imported into the global scope.

   The compiler recognizes the following options:

`--debug, -d'
     Print a listing of the output code on standard output. The listing
     includes both the bytecode for the right-hand sides of equations
     and the pattern-matching automaton generated for the left-hand
     sides. This option is mainly useful for debugging the compiler
     itself. You may wish to combine this with the `-n' option below if
     you only want to see the listing without actually creating a code
     file.

`--encoding=ENCODING'
     Specify the default encoding of script files (Q 7.0 and later). If
     this option is not used, the system encoding (as specified by the
     user's locale) is assumed. You only need this option if your
     script files contain non-ASCII text and the encoding of the files
     differs from the system encoding. (By including this option in the
     script files themselves, you can also specify the encoding on a
     per-file basis, see *Note Running Scripts from the Shell::.)

`--help, -h'
     Print a short help message.

`--list=LIST-FILE, -l LIST-FILE'
     When this option is used, error messages are written to the
     specified file instead of the standard error device.

`--no-code, -n'
     This option suppresses code file generation, which is useful if
     you only want to produce a listing or check the given scripts for
     syntax errors. (Please note that in the current implementation the
     code file will be created anyway, so that appropriate information
     can be given with the `-d' and `-v' options, but will finally be
     removed if the `-n' option is used.)

`--no-prelude'
     Suppresses automatic inclusion of the prelude.

`--output=OUTPUT-FILE, -o OUTPUT-FILE'
     This option allows you to change the default name of the generated
     code file.

`--path=PATH, -p PATH'
     This option sets the library path used to search for source and
     code files (default: system-dependent hardcoded default or value
     of the `QPATH' shell variable). If the PATH argument starts or ends
     with the path delimiter (`:' on UNIX) then the given path is
     appended or prepended to the current path, respectively.

`--prelude=FILE'
     Specify the name of the prelude script (default: `prelude.q').

`--tablesize=SIZE, -t SIZE'
     This option allows you to determine the size of the runtime hash
     table used for the hashing of identifiers. For best performance
     this should be a prime number; the default size is 5711. With this
     option, you can change the size of the hash table if you are
     compiling a script with a very large number of function symbols,
     or a script which dynamically creates a lot of symbols.  Inreasing
     the size of the hash table will reduce the "load" of the table and
     thereby improve the performance of symbol hashing at runtime.

`--verbose, -v'
     Display processed source files and statistics.

`--version, -V'
     This option causes version number and copyright information to be
     displayed, after which the program terminates.

`--warnings[=LEVEL], -w[LEVEL], --pedantic, --paranoid'
     Print warnings about undeclared function (and, optionally,
     variable) symbols. If just `-w' or `-w1' is specified then function
     symbols will be reported as undeclared if they are neither
     imported nor declared anywhere in the file, where all appearances
     of a function symbol on the left-hand side of equations and
     variable definitions count as implicit declarations.

     To get more warnings, use the `-w2' option (or, equivalently,
     `--pedantic'), which will warn you about _all_ function symbols
     which are used without a prior _explicit_ declaration.  I.e., when
     using that option, you really have to declare each and every
     function symbol explicitly, to make the compiler happy. Moreover,
     as of Q 7.8 `-w2' also warns about non-builtin and non-prelude
     symbols from unqualified imports which are used without proper
     qualification. This is useful to report a common source of errors,
     namely the accidental "reuse" of an imported symbol due to a
     missing local symbol declaration. (But note that in the current
     implementation you will never be warned about unqualified operator
     symbols, as these are treated as special tokens which are handled
     already during lexical analysis.)

     If this still isn't enough, then you can use the `-w3' option (or,
     equivalently, `--paranoid'), which will also warn you about
     undeclared free variables and unqualified prelude symbols. Thus
     you'll also have to declare each and every free variable, and use
     qualified symbols or qualified import even for prelude symbols, if
     you want to avoid all warnings from the compiler. Please note that
     this option will produce an excessive amount of warnings even for
     most perfectly legal scripts. But it may occasionally be useful to
     check your script for missing declarations or mistyped
     identifiers. If you find yourself using this option regularly,
     then Q probably is not for you. ;-)

     NOTE: As of Q 7.8, the default warning level can also be set with
     the `QWARN' environment variable. A note for library writers: If
     you publish libraries of Q modules for public consumption, they
     should pass at least `-w2' without any warnings, since many Q
     programmers will want to use that option at least occasionally.

   Options are generally parsed using `getopt' which means that
multiple single-letter options may be combined (e.g., `-vn') and a
parameterized option can be followed immediately by the corresponding
argument (e.g., `-ofoo.out'); see `getopt' in Section 3 of the UNIX
manual for details. All programs also accept long option names
following the GNU conventions, e.g., you can use `--output' instead of
`-o'. Note that in the case of parameterized long options the parameter
must either follow in the next command line argument, or it must be
separated from the option name by a `=' character. Long options may be
abbreviated by an unambigious prefix of the option name. For instance,
you may use `--out' instead of `--output'. Option parsing ends as soon
as the `--' option is found; the remaining command line arguments are
taken as normal parameters, even if they start with `-'.

   After a script has been compiled, you can invoke the interpreter on
the generated code file as already discussed above (in this case, the
code file is _not_ discarded). Below we describe the options which are
accepted by the interpreter. Note that the interpreter also recognizes
all of the compilation options discussed above, and will pass them on
when it invokes the compiler.

`--break'
     This option causes invocation of the debugger in case of an
     exception like `Ctl-C', invalid rule qualifier, user-raised
     exception (`throw'), or a call to the built-in `break' function.
     See *Note Debugging::, for details.

`--command=CMD, -c CMD'
     This option instructs the interpreter to execute the given command
     in "batch mode" (see below).

`--cstacksize=SIZE'
     Specify the maximum amount of C stack in KB (default: 256) that
     can be used for the recursive evaluation of special form
     arguments; see the comments on memory management below.

`--debug, -d'
     This option causes activation of the debugger. See *Note
     Debugging::, for details.

`--debug-options'
     Set various options which control how much detail the debugger
     prints when showing a reduction or rule. See *Note Debugging::,
     for details.

`--dec, --hex, --oct'
     Sets the integer output format to decimal (default), hexadecimal or
     octal, respectively.

`--echo, -e'
     Enable the echoing of commands when running in batch mode.

`--exitrc=FILE'
     This option allows you to specify the name of the file which is
     executed when the interpreter is exited; the default is usually
     `~/.qexitrc'.

`--fix[=PREC], --sci[=PREC], --std[=PREC]'
     Sets the floating point output format to fixed point, scientific
     (always using exponents) or standard (only using exponents if it
     results in a shorter notation; this is the default). For the
     scientific and standard formats, the given precision denotes the
     number of significant digits to be printed, and defaults to `15'.
     For the fixed point format, the precision specifies the number of
     digits following the decimal point (`2' by default).

`--gnuclient'
     Lets the interpreter run as a client of gnuserv(1). This allows the
     interpreter to interact with an emacs process driving the
     interpreter, e.g., in "comint" mode. In particular, the `edit' and
     `help' commands of the interpreter (see *Note Command Language::)
     will then be handled by sending corresponding Emacs Lisp commands
     via the gnuclient command instead of processing them directly. The
     actual name of the gnuclient program can be set using the
     `GNUCLIENT_PROGRAM' environment variable. Note that this option
     also disables the interpreter's own history processing as it is
     assumed that emacs takes care of it.

`--help, -h'
     Print a short help message.

`--histfile=FILE'
     Specify the name of the file in which the command history is stored
     (default: `~/.q_history').

`--histsize=SIZE'
     Specify the size of the command history (default: 500).

`--initrc=FILE'
     Specify the name of the file which is executed at startup (default:
     `~/.qinitrc').

`--interactive, -i'
     Indicates that the interpreter is running interactively.

`--memsize=SIZE'
     Specify the maximum size of the expression heap (default: 4096000;
     see comments on memory management below).

`--no-editing'
     Disable command line editing using readline.

`--no-exitrc'
     Suppress execution of the exitrc file.

`--no-initrc'
     Suppress execution of the initrc file.

`--path=PATH, -p PATH'
     As with `qc', sets the library path used to search for source and
     code files.

`--prompt=STR'
     Set the prompt string (default: `\n==> ').

`--quiet, -q'
     This option causes a quiet startup; it suppresses the sign-on
     message.

`--source=FILE, -s FILE'
     Source file with interpreter commands (batch mode).

`--stacksize=SIZE'
     Specify the maximum size of the internal stacks (default: 1024000;
     see comments on memory management below).

`--version, -V'
     As with `qc', display version number and copyright information.

Options for Interactive and Batch Usage
---------------------------------------

Unless one of the `-c' and `-s' options is specified, the interpreter
starts up in "interactive mode", in which the user is repeatedly
prompted to enter an expression, and the interpreter answers with the
corresponding normal form. The interpreter also understands some other
special commands, which are discussed in *Note Command Language::. With
the `-c' and `-s' options, the interpreter runs in "batch mode", in
which it executes the commands and command files specified on the
command line, prints the results on the standard output device, and
then exits immediately. The `-d' option invokes the symbolic debugger
built into the interpreter which allows you to trace evaluations, cf.
*Note Debugging::.

   When the interpreter starts up in interactive mode, and is connected
to a terminal, it also provides a command history through the GNU
readline library. (*Note Readline: (readline)Command Line Editing.)
That is, the lines you type are memorized, and you can search for and
edit previous commands. The command history is also saved to a file
when the interpreter is exited, and completion of filenames, keywords,
command names and defined function and variable symbols is supported as
well. You can disable this feature with the `--no-editing' option.

   You can redirect the interpreter's input and output streams to a
file or a pipe as usual. The Q interpreter checks whether it is
connected to a terminal in order to prevent garbled output when input
or output is redirected. If you're feeling adventurous, try something
like the following:

     q -d <<EOF
     def Xs = [1..50]
     msort (>) Xs
     EOF

   The `-i' option can be used to force the interpreter to interactive
mode even when input or output is redirected. This causes sign-on and
prompt to be printed, and the standard output buffers to be flushed
after each evaluation, which is useful when another program is driving
the interpreter. (If you still want to suppress the sign-on message, use
`-i' in combination with `-q'.) Furthermore, the `-i' option also
suppresses the batch mode execution of `-c' and `-s' options.

   The `--initrc' and `--exitrc' options allow you to specify command
files which are executed when the interpreter starts and exits, which
is commonly used to customize the environment when running the
interpreter interactively, see *Note Setting up your Environment::. The
execution of these files will be suppressed when the interpreter is run
in batch mode, or when using the `--no-initrc' and `--no-exitrc'
options.

Memory Management
-----------------

The Q interpreter has a fully dynamic memory management system. In
particular, it automatically resizes both the expression heap and the
evaluation stack as required during the course of a computation. Thus
you normally do not have to worry about memory management issues.

   However, the `--stacksize' option allows you to specify a maximum
size for the evaluation stack (more precisely, it sets the limit for
both the expression and the rule stack), which keeps the stack from
growing to infinity. This is useful, e.g., to catch infinite recursion.
Note that as the stack is also used internally for the purpose of
parsing expressions, a reasonable minimum size is required for the
interpreter to function properly. Therefore the interpreter enforces
that the given size is not less than a certain minimum value (usually
100); otherwise the option is ignored and the stack size is reset to
its default value.

   Similarly, the `--memsize' option limits the number of expression
nodes on the heap and thereby prevents the interpreter from using up all
available main memory on your system. You can disable both limits by
specifying `0' as the argument to the corresponding option.

   In the current implementation, the interpreter also needs a variable
amount of C stack space for evaluating "deferred" values like special
form arguments. This might cause the interpreter to exit unexpectedly
due to C stack overflows. To prevent this, as of Q 7.7 the interpreter
poses a limit on the amount of stack space used during evaluation of
deferred values and will gracefully terminate the computation with a
stack overflow error when the allocated C stack space is exhausted. The
default stack size is 256 kilobytes; you can change this with the
`--cstacksize' option. Setting this option to zero disables the C stack
checks. (Please note that this option does _not_ change the interpreter
program's C stack size, it only sets the limit used in the stack test;
your operating system should supply you with a command, such as
`ulimit' on UNIX-like systems, to actually set the program stack size.)

   The interpreter manages expression memory using a reference counting
scheme. This has the considerable advantage that no disruptive "garbage
collection" process has to be invoked during an evaluation, and that
expression objects are reclaimed and other automatic cleanup (closing
files, etc.) is performed on them as soon as they become inaccessible.
The downside of this scheme is that cyclic expression structures (which
can be created, e.g., using expression references, cf. *Note Expression
References::) cannot be collected and thus should be avoided.


File: qdoc.info,  Node: Command Language,  Next: Setting up your Environment,  Prev: Running Compiler and Interpreter,  Up: Using Q

B.2 Command Language
====================

The commands understood by the interpreter make up a simple "command
language" which is described in the following. Commands can also be
passed to the interpreter by means of the `-c' option, and you can put
a sequence of commands into a file and execute them using the `-s'
option (see *Note Running Compiler and Interpreter::). Command files
can also be executed in the interpreter using the `source' command (see
below). Moreover, when the interpreter starts up and is exited in
interactive mode, it reads and executes commands from its
initialization and termination files, which allows you to customize your
environment according to your taste (see *Note Setting up your
Environment::).

   The command language is line-oriented, i.e., commands are terminated
by the newline character. As usual, line ends can be escaped by putting
the `\' character immediately before the end of the line; this also
works inside of string literals. A line may contain multiple expressions
and commands, which are separated by the `;' delimiter:

     ==> def X = foo Y; X; X/2

(You may also terminate a command line with an extra `;' delimiter, but
this is not required.)

   Here are the most common commands which allow you to evaluate
expressions and bind global variables:

`EXPRESSION'
     Evaluate the given expression and print the corresponding result.

`def EXPRESSION = EXPRESSION, ...'
     Define free variable values.

`undef VARIABLE, ...'
     Undefine the given variables.

`var VARIABLE [ = EXPRESSION ], ...'
     Declare the given variables and optionally initialize their values.

   The syntax of expressions and variable declarations and definitions
is the same as that of the corresponding constructs in scripts. After
evaluating an expression, the interpreter usually prints the
corresponding normal form as the result. (Note, however, that some
scripts may also provide special "views" to implement custom
pretty-printing of certain types of objects, see *Note Views::.) No
response is printed for variable declarations and definitions, unless an
error occurs.

   As of Q 7.7, the interpreter allows you to declare variables
interactively with the `var' command, which works pretty much like
`var' declarations in scripts (see *Note Declarations::). The declared
variables are created in the global scope (i.e., the scope of the main
script). In particular, this command allows you to declare
uncapitalized identifiers (which would otherwise be interpreted as
function symbols) as variables.

   After a variable has been assigned a value, it will be replaced by
its associated value whenever it is evaluated. This allows you to store
intermediate results in a computation, set the values of free variables
occuring in a script, and to define new functions interactively in the
interpreter. For instance:

     ==> var double = (*) 2, X = double 4; X
     8

   A subsequent `undef' removes a variable definition:

     ==> undef X; X
     X

   As explained in *Note Free Variables::, the left-hand side of a
variable definition may actually be an arbitrary pattern to be matched
against the right-hand side value:

     ==> def [X|Xs] = [a,b,c]

   When starting up, the interpreter always defines the variables
`INPUT', `OUTPUT', `ERROR' and `ARGS'. `INPUT' and `OUTPUT' are set to
the standard input/output devices used by the terminal I/O functions
(see *Note Terminal I/O::), while `ERROR' denotes the standard error
device normally used to display error messages. You can use these
symbols when passing a file to one of the built-in file I/O functions
(see *Note File I/O::). As already mentioned, the `ARGS' variable is
assigned a string list which contains the additional command line
arguments the Q interpreter was invoked with (starting with the script
name). These variable symbols are "read-only"; they cannot be re- or
undefined.

   There is one other special built-in variable, the "anonymous"
variable `_', which, when used in an expression or the right-hand side
of a variable definition, refers to the result of the most recent
expression evaluation:

     ==> 1/3
     0.333333333333333

     ==> 1/_
     3.0

   The interpreter also provides some other special commands which
extend the expression and definition syntax of the Q language. They
usually consist of a command name and possibly some parameters. Unlike
the `def', `undef' and `var' keywords, which are also reserved words in
the Q language itself, the other command names are only treated
specially when occuring at the beginning of a command; anywhere else
they are parsed as ordinary Q identifiers.

   The parameters of special commands are simply string literals which
may be single-quoted, double-quoted or unquoted, and which are delimited
with whitespace, very much like in the UNIX command shell. The
interpreter will not try to evaluate these parameters. However, the
usual escapes for special characters are supported. The following table
lists the available commands, where `[...]' is used to denote optional
arguments.

`! COMMAND'
     Shell escape. The given command is executed using the shell. Unlike
     other types of commands, the shell escape must be on a line by
     itself.

`? EXPRESSION'
     Escape an expression. This just evaluates EXPRESSION and prints
     the result as usual, but inhibits the interpretation of special
     command names at the beginning of the expression. E.g.,

          ==> ? echo "some string"

     will evaluate an expression involving some `echo' function instead
     of executing the `echo' command.

`copying'
     Show the GNU license conditions.

`help [S]'
     Start the GNU info reader (as specified by the `INFO_PROGRAM'
     environment variable, `info' by default) with the Q manual; if a
     keyword S is given, perform an index search. When the interpreter
     has been started with the `--gnuclient' option (see *Note Running
     Compiler and Interpreter::), then the command submits a request to
     invoke the info reader of a driving emacs process using the
     gnuclient program.

`path [S]'
     Set the library search path (`QPATH') to S (print the current
     setting if S is omitted). If the S argument starts or ends with
     the path delimiter (`:' on UNIX) then the given path is appended
     or prepended to the current path, respectively.

`prompt [S [S2 [S3]]]'
     Set the prompt string (print the current setting if S is omitted).
     The `\\v', `\\s', `\\w', `\\W', `\\m' and `\\M' escape sequences
     may be used in the prompt string to denote the Q version number,
     host system information, current working directory, basename of
     the current directory, full pathname of the current script, and
     the script's basename, respectively. (Note the double escapes;
     these are required to make one `\' get through to the command.)
     The optional second and third argument allow you to change the
     default continuation (`> ') and debugger prompt (`: '); no special
     escape sequences are substituted in these strings.

`dec, hex, oct'
     Sets the integer output format to decimal, hexadecimal and octal,
     respectively (see also the corresponding interpreter command line
     options in *Note Running Compiler and Interpreter::). Note that
     these commands only affect the display of expressions which are
     printed in response to interpreter and debugger commands,
     otherwise integer values will always be unparsed in decimal
     notation.

`fix [PREC], sci [PREC], std [PREC]'
     Sets the floating point output format to fixed point, scientific
     and standard, respectively (see also the corresponding interpreter
     command line options in *Note Running Compiler and Interpreter::).
     If the precision is omitted, it defaults to `2' for the fixed point
     format, and to `15' otherwise. Note that these commands only affect
     the display of expressions which are printed in response to
     interpreter and debugger commands, otherwise floating point values
     will always be unparsed with the full internal precision.

`histfile [S], histsize [N]'
     Change the history filename and set the size of the history (print
     the current setting if no argument).

`cstacksize [N]'
     Set the maximum C stack size in kilobytes (print the current
     setting if no argument); see the discussion of the `--cstacksize'
     option in *Note Running Compiler and Interpreter::.

`stacksize [N]'
     Set the maximum stack size (print the current setting if no
     argument); see the discussion of the `--stacksize' option in *Note
     Running Compiler and Interpreter::.

`memsize [N]'
     Set the maximum number of expression nodes on the heap (print the
     current setting if no argument); see the discussion of the
     `--memsize' option in *Note Running Compiler and Interpreter::.

`stats [all]'
     Print statistics about the most recent expression evaluation in
     the main thread. If the optional `all' parameter is given then the
     command also lists finished background threads which have not been
     recycled yet. The given figures are the cpu time needed to
     complete the evaluation, the total number of reductions performed
     by the interpreter, and the number of expression nodes (heap
     cells) created during the course of the computation. The
     application of each built-in rule counts as a single reduction.
     Note that since different built-in rules may have different
     execution times (in fact, the time needed by many of the built-in
     operations varies with the size of the input), the number of
     reductions usually only provides a rough estimate of the actual
     running time. But this value is still quite useful to perform
     asymptotic analysis (up to linear factors, if one wants to be
     picky), and can also be used to compare profiling results obtained
     on different machines with different processors. The cell count is
     the _maximum_ number of cells required at any time during the
     evaluation, which does _not_ include the size of the input
     expression. This value is used to measure space requirements, and
     is proportional to the "real" memory requirements of the
     interpreter (each expression node requires some 24 bytes in the
     current implementation). But note that subexpressions may be
     "shared" between different expressions, and thus the cell count
     will usually be less than the tree size of the constructed
     expressions.

`debug [on|off|OPTIONS]'
     When invoked with arguments `on' or `off', activate or deactivate
     debugging; print the current setting if no argument is specified.
     Alternatively, you can also use this command to set various
     options which control how much detail the debugger prints when
     showing a reduction or rule. See *Note Debugging::, for details.

`break [on|off|FUNCTION ...]'
     When invoked with arguments `on' or `off', this command sets the
     interpreter's break flag which controls how the interpreter
     responds to an exception like `Ctl-C', invalid rule qualifier,
     user-raised exception (`throw'), or a call to the built-in `break'
     function. When invoked with a list of function symbols, it sets
     breakpoints on the given symbols, and also enables the break flag
     if necessary. If no argument is given, the current setting of the
     break flag is printed, along with a list of the current
     breakpoints. See *Note Debugging::, for details.

`tbreak FUNCTION ...'
     This commands works like `break' when invoked with a list of
     function symbols, but sets temporary breakpoints which are removed
     when hit.

`profile [FUNCTION ...]'
     Sets a flag on the given function symbols, so that reductions are
     counted whenever an application of the function is reduced. The
     rules for counting reductions are the same as for the `stats'
     command, i.e., each invocation of a built-in or external function
     counts as a single reduction. When invoked without arguments, the
     command prints a list of all flagged function symbols along with
     the corresponding reduction numbers (sorted by decreasing counts)
     and resets the reduction counters. This command is useful for
     profiling purposes, e.g., if you want to find "bottlenecks" among
     the functions of your script, whose optimization will most likely
     lead to noticeable improvements of running times.

`clear [VARIABLE ...]'
     Undefine the given variables. If no argument is specified, clear
     all user-defined variables (including the value of the `_'
     variable).

`clear break [FUNCTION ...]'
     Remove breakpoints on the given function symbols (all breakpoints
     if no argument is specified).

`clear profile [FUNCTION ...]'
     Disable profiling on the given function symbols (all functions if
     no argument is specified).

`unparse [on|off]'
     Enable or disable custom pretty-printing of expressions (print
     current setting if no option is given), as defined by rules for
     the built-in `view' function, cf. *Note Views::. This is enabled
     by default.

`echo [on|off], echo S'
     Enable or disable command echoing (print current setting if no
     option is given), or echo the given string to standard output
     (with a newline character appended). When echoing is enabled, all
     command lines executed in batch mode (using the `-s' or `-c'
     option, or the `source' command) are echoed to standard output
     before they are executed. (You can inhibit command echoing by
     prefixing a command line with the `@' character.)

`chdir [S], cd [S]'
     Change to the given directory (your home directory if none given).

`pwd'
     Print the current working directory.

`ls [S ...]'
     List the contents of the current directory, or the files matching
     the given patterns. This command simply invokes the corresponding
     UNIX command with the given arguments.

`which [S]'
     Print the full name of the script or command file (as given by a
     search on the library path) that would be run when the relative
     pathname S is given to the `run' or `source' command; if invoked
     without arguments, print the full pathname of the running script.

`edit [S]'
     Edit the given file (current main script if none specified), using
     the editor specified by the `EDITOR' environment variable, or `vi'
     by default. When the interpreter has been started with the
     `--gnuclient' option (see *Note Running Compiler and
     Interpreter::), then the command causes an edit request to be
     submitted to a driving `emacs' process using the `gnuclient'
     program.

`run [S [ARGS ...]]'
     Run the given script with the given arguments. If no script is
     specified, rerun the current script with the current arguments.
     (Note that to force the interpreter back to an empty script, you
     can specify `""' for the script name. You can also use `-' as the
     script name to indicate that the script is to be read from
     standard input.)  When in interactive mode, the `exitrc' and
     `initrc' files are sourced as usual. Note that in the current
     implementation this operation also clears the variable memory, so
     you might wish to save your variables _before_ invoking this
     command, unless your `exitrc' file already takes care of this.

`import [[+|-]S ...]'
     Rerun the current script with the current arguments, as with `run',
     but also add or remove the given scripts to/from the current
     "import list". A `+' prefix (or no prefix at all) causes the
     module to be added to the import list and `-' removes the module,
     if possible. If no arguments are given then the import list is
     cleared. Modules on the import list are imported into the global
     scope (i.e., the namespace of the main script), in addition to the
     prelude and the imports of the main script. The current import
     list can be shown with the `imports' command, see below.

     Note that you can also add modules which are already included in
     the running script, but which are not currently visible in the
     global scope. On the other hand, you can only remove those modules
     which have actually been added to the import list, not any modules
     which are only included in the running script. The main script can
     be removed as well, which has the same effect as doing `run' with
     an empty script name.

     *IMPORTANT:* This command will only work as advertised when
     running a source script. If a bytecode file is run, the import
     list will be cleared, and subsequent `import' commands will have
     no effect.

`source S, . S'
     Source a file with interpreter commands (may be nested). As
     indicated, this command may be abbreviated as `.'. The given
     command file is searched on the Q library path.

`save [S], load [S]'
     Save and restore variable values to/from the given file (if no file
     specified, use the file last specified with `save' or `load', or
     `.q_vars' by default).

     The `save' command stores all user-defined variables in the form of
     variable definitions, i.e., `var' commands. Floating point values
     are always stored with maximum internal precision so that they can
     be reconstructed exactly, at least on machines with IEEE
     compatible 64 bit doubles. The built-in variables are _not_ saved
     with this command; thus, if you want to also save the value of the
     `_' variable, you must explicitly assign it to a user variable.

     The `load' command is very much like `source', so the file may
     actually contain arbitrary interpreter commands, but no library
     path search is performed, so a path must be given unless the file
     is contained in the current directory.

     Putting a pair of `load' and `save' commands in your
     `initrc'/`exitrc' files (see *Note Setting up your Environment::)
     is a convenient method to have the interpreter automatically
     remember the variable environment from a previous session. Note,
     however, that the `load' command reevaluates all values in the
     context of the current script, and hence the results may be quite
     different from the original values if the script has been changed
     during invocations of the interpreter. Another potential obstacle
     is that the `load' command cannot reconstruct "external" objects
     (`<<TYPEID>>', see *Note File I/O::, and *Note C Language
     Interface::), since these objects do not have a parseable
     representation at all.

`modules'
     List all loaded modules, i.e., all scripts imported or included in
     the running script (including the prelude). External modules are
     indicated by a trailing `*'.

`imports'
     Like the `modules' command, but only list the modules visible in
     the global scope. This includes the main script itself, the
     prelude and all the modules it includes (normally the standard
     library), the imports and includes of the main script, and the
     extra imports specified with the `import' command (see above). The
     latter modules and the main script are indicated by an initial `+'
     character. (These are the modules which can be removed with the
     `import' command.)

`who [all]'
     List all user-defined variable symbols. This only lists variables
     which have been defined in the interpreter. If the optional `all'
     parameter is specified then all variables are listed.

`whos SYMBOL ...'
     Describe the given (function, variable or operator) symbols. Prints
     useful information about each symbol, such as whether it is a
     function symbol or a variable, name of the script file and line
     number where the symbol is first declared, defined or used,
     `const' and `special' attributes, etc.

`whois SYMBOL ...'
     As of Q 7.8, this is a new and improved version of the `whos'
     command which also lists, for each given symbol, the fully
     qualified names under which the symbol is available in the global
     namespace.

`completion_matches S'
     List the possible completions of a token prefix. This command
     allows programs like `emacs' (see *Note Running Scripts in
     Emacs::) to implement completion when driving the interpreter as
     an inferior process.


