| Standard Text Processing Module

Text scanning

This chapter is dedicated to the AFNIX text scanning, a subpart of the standard text processing module. Text scanning refers to the ability to constructs lexemes from a text stream. The lexemes are built from the stream with the help of regular expressions. Eventually some special operations like text transliteration is also possible.

All AFNIX text processing objects are located in the `afnix-txt` module. This module must be loaded prior any operation. Multiple calls to the module initialization routine are harmless. The interpreter method `module` loads a specific module by name. When the module has been loaded, the object are available in the `afnix:txt` nameset.

Scanning concepts

Text scanning is the ability to extract lexical elements or lexemes from a stream. A scanner or lexical analyzer is the principal object used to perform this task. A scanner is created by adding special object that acts as a pattern matcher. When a pattern is matched, a special object called a lexeme is returned.

Pattern object

A `Pattern` object is a special object that acts as model for the string to match. There are several ways to build a pattern. The simplest way to build it is with a regular expression. Another type of pattern is a balanced pattern. In its first form, a pattern object can be created with a regular expression object.

In this example, the pattern object is built to detect integer objects.

The `check` method return true if the input string matches the pattern. The `match` method returns the string that matches the pattern. Since the pattern object can also operates with stream object, the `match` method is appropriate to match a particular string. The pattern object is, as usual, available with the appropriate predicate.

Another form of pattern object is the balanced pattern. A balanced pattern is determined by a starting string and an ending string. There are two types of balanced pattern. One is a single balanced pattern and the other one is the recursive balanced pattern. The single balanced pattern is appropriate for those lexical element that are defined by a character. For example, the classical C-string is a single balanced pattern with the double quote character.

In the case of the C-string, the pattern might be more appropriately defined with an additional escape character. Such character is used by the pattern matcher to grab characters that might be part of the pattern definition.

In this form, a balanced pattern with an escape character is created. The same string is used for both the starting and ending string. Another constructor that takes two strings can be used if the starting and ending strings are different. The last pattern form is the balanced recursive form. In this form, a starting and ending string are used to delimit the pattern. However, in this mode, a recursive use of the starting and ending strings is allowed. In order to have an exact match, the number of starting string must equal the number of ending string. For example, the C-comment pattern can be viewed as recursive balanced pattern.

Lexeme object

The `Lexeme` object is the object built by a scanner that contains the matched string. A lexeme is therefore a tagged string. Additionally, a lexeme can carry additional information like a source name and index.

The default lexeme is created with any value. A value can be set with the `set-value` method and retrieved with the `get-value` methods.

Similar are the `set-tag` and `get-tag` methods which operate with an integer. The source name and index are defined as well with the same methods.

Text scanning

Text scanning is the ability to extract lexical elements or lexemes from an input stream. Generally, the lexemes are the results of a matching operation which is defined by a pattern object. As a result, the definition of a scanner object is the object itself plus one or several pattern object.

Scanner construction

By default, a scanner is created without pattern objects. The `length` method returns the number of pattern objects. As usual, a predicate is associated with the scanner object.

The scanner construction proceeds by adding pattern objects. Each pattern can be created independently, and later added to the scanner. For example, a scanner that reads real, integer and string can be defined as follow:

The order of pattern integration defines the priority at which a token is recognized. The symbol name for each pattern is optional since the functional programming permits the creation of patterns directly. This writing style makes the scanner definition easier to read.

Using the scanner

Once constructed, the scanner can be used as is. A stream is generally the best way to operate. If the scanner reaches the end-of-stream or cannot recognize a lexeme, the nil object is returned. With a loop, it is easy to get all lexemes.

In this loop, it is necessary first to check for the end of the stream. This is done with the help of the special loop construct that initialize the `valid` symbol. As soon as the the lexeme is built, it can be used. The lexeme holds the value as well as it tag.

Text sorting and transformation

This chapter is dedicated to the AFNIX text sorting and transformation, which is a subpart of the standard text processing module.

Text sorting

Sorting is one the primary function implemented inside the text processing module. There are three sorting functions available in the module.

Ascending and descending order sorting

The `sort-ascent` function operates with a vector object and sorts the elements in ascending order. Any kind of objects can be sorted as long as they support a comparison method. The elements are sorted in placed by using a quick sort algorithm.

The `sort-descent` function is similar to the `sort-ascent` function except that the object are sorted in descending order.

Lexical sorting

The `sort-lexical` function operates with a vector object and sorts the elements in ascending order using a lexicographic ordering relation. Objects in the vector must be literal objects or an exception is raised.

Transliteration

Transliteration is the process of changing characters my mapping one to another one. The transliteration process operates with a character source and produces a target character with the help of a mapping table. The transliteration process is not necessarily reversible as often indicated in the literature.

Literate object

The `Literate` object is a transliteration object that is bound by default with the identity function mapping. As usual, a predicate is associate with the object.

The transliteration process can also operate with an escape character in order to map double character sequence into a single one, as usually found inside programming language.

Transliteration configuration

The `set-map` configures the transliteration mapping table while the `set-escape-map` configure the escape mapping table. The mapping is done by setting the source character and the target character. For instance, if one want to map the tabulation character to a white space, the mapping table is set as follow:

The escape mapping table operates the same way. It should be noted that the mapping algorithm translate first the input character, eventually yielding to an escape character and then the escape mapping takes place. Note also that the `set-escape` method can be used to set the escape character.

Transliteration process

The transliteration process is done either with a string or an input stream. In the first case, the `translate` method operates with a string and returns a translated string. On the other hand, the `read` method returns a character when operating with a stream.

Text hashing and ciphering

This chapter is dedicated to the AFNIX text hashing and ciphering, which is a subpart of the standard text processing module.

Text hashing

Text hashing is the ability to generate an almost unique id from a string. Although, there is no guarantee that two different string will not produce the same result -- known as collision -- the sophistication of the hashing function attempt to minimize such eventuality. The hashing process is not reversible. There are several hashing functions available in the public domain. To name a few, MD5 is the message digest 5, and SHA is the secure hash algorithm. The following table illustrates the size of the result with different hashing functions.

Function | Result size
--- | ---
MD-5 | 128 bits
SHA-1 | 160 bits
SHA-256 | 256 bits
SHA-384 | 384 bits
SHA-512 | 512 bits

Hasher object

The `Hasher` class is a text hashing computation class. The class computes a hash value from a literal object, a buffer or an input stream. Once computed, the hash value is stored as an array of bytes that can be retrieved one by one or at all in the form of a string representation.

Creating a MD5 hasher

The `Md5` object is the hasher object that implements the MD-5 algorithm. The constructor does not take any argument.

The `compute` method computes the hash value. For example, the string "abc" returns the value "900150983CD24FB0D6963F7D28E17F72" which is 16 bytes long.

Creating a SHA hasher

There are several SHA objects that produces results of different size as indicated in the next table.

Hasher | Size | Constructor
--- | --- | ---
SHA-1 | 160 bits | Sha1
SHA-256 | 256 bits | Sha256
SHA-384 | 384 bits | Sha384
SHA-512 | 512 bits | Sha512

The `compute` method computes the hash value. For example, the string "abc" returns with SHA-1 the value "A9993E364706816ABA3E25717850C26C9CD0D89D" which is here 20 bytes long.

Cipher key

Cipher key management is an important concept in the land of text ciphering. In a simple mode, a key is used by a text cipher to encode some data. Although the key can be any sequence of bytes, it is preferable to have it built with some special algorithms like a text hasher. Most of the time, it is also desirable to have a key that follow simple constraints like size.

Creating a cipher key

The `Key` class is a cipher key suitable for text cipher. The key can be 128 bits, 192 bits or 256 bits long. By default, a random key is generated, but the key can be also generated from a string message. Internally, the key is generated with the SHA-256 algorithm. By default, the key is 128 bits long.

A random typed key can be created by specifying the key type in the constructor. The type `K128`, `K192` and `K256` are currently supported.

The constructor also supports the use of a string message for the generation of the key. The message can also be combined with a key type

Key functions

The basic operations associated with a key are the byte extraction as well as the key formating. In normal use, the key object is used directly by the text cipher. However, it might become necessary to access the key value directly. The `get-type` and `get-size` methods returns respectively the key type and key size. The `format` method returns a string representation of the key. The `get` method returns a key byte by index.

Stream cipher

A stream cipher is a class that encodes an input stream with a particular algorithm. The stream cipher is symmetric. This means that the data can be coded and later decoded to their original form. Most of the time. a stream cipher requires a key to operate. It is the key, if kept secret, that can ensure the confidentiality of the information being coded.

Cipher base class

The `Cipher` class is a base class for the stream cipher engine. The class provides a `stream` method that reads from an input stream and write into an output stream. The `Cipher` class is an abstract class and cannot be instantiated by itself. The object is actually created by using a cipher algorithm class such like the `Aes` class.

The `stream` method returns the number of character that have been encoded. Care should be taken that most of the stream cipher operates by block and therefore, will block until a complete block has been read from the input stream, unless the end of stream is read. Furthermore, the coded results does precise how the padding should be made for a particular cipher. It is therefore not recommended to use the `stream` stream method, but rather use the `InputCipher` class.

Creating a cipher

A `Cipher` object can be created with a cipher constructor. As of today, only the advanced encryption standard or AES is supported. The `Aes` class creates a new cipher that conform to the AES standard.

A cipher can be created with a key and eventually a reverse flag. With one argument, the cipher key is associated with the cipher. Such key can be created as indicated in the previous section. The reverse flag is used to determine if the cipher operate in encoding or decoding mode. By default, the cipher operates in coding mode.

Cipher information

The `Cipher` class has several methods that provide information about the cipher. This include the cipher name with the `get-name` method and the cipher block size with the `get-size` method.

The cipher operating mode can be found with the `get-reverse` method. If the `get-reverse` method returns true, the cipher is operating in decoding mode. Note that a `set-reverse` method exists also.

Input cipher

In the presence of a `Cipher` object, it is difficult to read an input stream and encode the character of a block basis. Furthermore, the existence of various method for block padding makes the coding operation even more difficult. For this reason, the `InputCipher` class provides the necessary method to code or decode an input stream in various mode of operations, including the management of the block padding.

Input cipher mode

The `InputCipher` class is an input stream that binds an input stream with a cipher. The class acts like an input stream, read the character from the binded input stream and encode or decode them from the binded cipher. The `InputCipher` defines two modes of operations. In electronic codebook mode or ECB, the character are encoded in a block basis. In cipher block chaining mode, the block are encoded by doing an XOR operation with the previous block. In all cases, a block padding is added, following the recommendation of RFC 2630 and NIST 800-38A.

Creating an input cipher

By default an input cipher is created with a cipher object. Eventually, an input stream and/or the input mode can be specified at the object construction.

In this example, the input cipher is created in ECB mode. The input stream is later associated with the `set-is` method.

Input cipher operation

The `InputCipher` class operates with one or several input streams. The `set-is` method sets the input stream. Read operation can be made with the help of the `valid-p` predicate.

Since the `InputCipher` operates like an input stream, the stream can be read as long as the `valid-p` predicate returns true. Note that the `InputCipher` manages automatically the padding operation by maintaining internally a buffer with the encoded padded characters. During the decoding phase, the class make sure that the padded characters are removed correctly.

Standard text processing reference

This appendix is a reference of the AFNIX standard text processing module.

Symbol | Description
--- | ---
afnix-txt | module
afnix:txt | nameset

Pattern

The `Pattern` class is a pattern matching class based either on regular expression or balanced string. In the regex mode, the pattern is defined with a regex and a matching is said to occur when a regex match is achieved. In the balanced string mode, the pattern is defined with a start pattern and end pattern strings. The balanced mode can be a single or recursive. Additionally, an escape character can be associated with the class. A name and a tag is also bound to the pattern object as a mean to ease the integration within a scanner.

Predicate

    pattern-p

Inheritance

    Object

Constructors

Pattern (none)

The Pattern constructor creates an empty pattern.

Pattern (String|Regex)

The Pattern constructor creates a pattern object associated with a regular expression. The argument can be either a string or a regular expression object. If the argument is a string, it is converted into a regular expression object.

Pattern (String String)

The Pattern constructor creates a balanced pattern. The first argument is the start pattern string. The second argument is the end balanced string.

Pattern (String String Character)

The Pattern constructor creates a balanced pattern with an escape character. The first argument is the start pattern string. The second argument is the end balanced string. The third character is the escape character.

Pattern (String String Boolean)

The Pattern constructor creates a recursive balanced pattern. The first argument is the start pattern string. The second argument is the end balanced string.

Constants

REGEX

The REGEX constant indicates that the pattern is a regular expression.

BALANCED

The BALANCED constant indicates that the pattern is a balanced pattern.

RECURSIVE

The RECURSIVE constant indicates that the pattern is a recursive balanced pattern.

Methods

check (→ Boolean) (String)

The check method checks the pattern against the input string. If the verification is successful, the method returns true, false otherwise.

match (→ String) (String|Input)

The match method attempts to match an input string or an input stream. If the matching occurs, the matching string is returned. If the input is a string, the end of string is used as an end condition. If the input stream is used, the end of stream is used as an end condition.

set-tag (→ none) (Integer)

The set-tag method sets the pattern tag. The tag can be further used inside a scanner.

get-tag (→ Integer) (none)

The get-tag method returns the pattern tag.

set-name (→ none) (String)

The set-name method sets the pattern name. The name is symbol identifier for that pattern.

get-name (→ String) (none)

The get-name method returns the pattern name.

set-regex (→ none) (String|Regex)

The set-regex method sets the pattern regex either with a string or with a regex object. If the method is successfully completed, the pattern type is switched to the REGEX type.

set-escape (→ none) (Character)

The set-escape method sets the pattern escape character. The escape character is used only in balanced mode.

get-escape (→ Character) (none)

The get-escape method returns the escape character.

set-balanced (→ none) (String| String String)

The set-balanced method sets the pattern balanced string. With one argument, the same balanced string is used for starting and ending. With two arguments, the first argument is the starting string and the second is the ending string.

Lexeme

The `Lexeme` class is a literal object that is designed to hold a matching pattern. A lexeme consists in string (i.e. the lexeme value), a tag and eventually a source name (i.e. file name) and a source index (line number).

Predicate

    lexeme-p

Inheritance

    Literal

Constructors

Lexeme (none)

The Lexeme constructor creates an empty lexeme.

Lexeme (String)

The Lexeme constructor creates a lexeme by value. The string argument is the lexeme value.

Methods

set-tag (→ none) (Integer)

The set-tag method sets the lexeme tag. The tag can be further used inside a scanner.

get-tag (→ Integer) (none)

The get-tag method returns the lexeme tag.

set-value (→ none) (String)

The set-value method sets the lexeme value. The lexeme value is generally the result of a matching operation.

get-value (→ String) (none)

The get-value method returns the lexeme value.

set-index (→ none) (Integer)

The set-index method sets the lexeme source index. The lexeme source index can be for instance the source line number.

get-index (→ Integer) (none)

The get-index method returns the lexeme source index.

set-source (→ none) (String)

The set-source method sets the lexeme source name. The lexeme source name can be for instance the source file name.

get-source (→ String) (none)

The get-source method returns the lexeme source name.

Scanner

The `Scanner` class is a text scanner or lexical analyzer that operates on an input stream and permits to match one or several patterns. The scanner is built by adding patterns to the scanner object. With an input stream, the scanner object attempts to build a buffer that match at least one pattern. When such matching occurs, a lexeme is built. When building a lexeme, the pattern tag is used to mark the lexeme.

Predicate

    scanner-p

Inheritance

    Object

Constructors

Scanner (none)

The Scanner constructor creates an empty scanner.

Methods

add (→ none) (Pattern*)

The add method adds 0 or more pattern objects to the scanner. The priority of the pattern is determined by the order in which the patterns are added.

length (→ Integer) (none)

The length method returns the number of pattern objects in this scanner.

get (→ Pattern) (Integer)

The get method returns a pattern object by index.

check (→ Lexeme) (String)

The check method checks that a string is matched by the scanner and returns the associated lexeme.

scan (→ Lexeme) (InputStream)

The scan method scans an input stream until a pattern is matched. When a matching occurs, the associated lexeme is returned.

Literate

The `Literate` class is transliteration mapping class. Transliteration is the process of changing characters my mapping one to another one. The transliteration process operates with a character source and produces a target character with the help of a mapping table. This transliteration object can also operate with an escape table. In the presence of an escape character, an escape mapping table is used instead of the regular one.

Predicate

    literate-p

Inheritance

    Object

Constructors

Literate (none)

The Literate constructor creates a default transliteration object.

Literate (Character)

The Literate constructor creates a default transliteration object with an escape character. The argument is the escape character.

Methods

read (→ Character) (Input)

The read method reads a character from the input stream and translate it with the help of the mapping table. A second character might be consumed from the stream if the first character is an escape character.

reset (→ none) (none)

The reset method resets all the mapping table and install a default identity one.

set-map (→ none) (Character Character)

The set-map method set the mapping table by using a source and target character. The first character is the source character. The second character is the target character.

get-map (→ Character) (Character)

The get-map method returns the mapping character by character. The source character is the argument.

translate (→ String) (String)

The translate method translate a string by transliteration and returns a new string.

set-escape (→ none) (Character)

The set-escape method set the escape character.

get-escape (→ Character) (none)

The get-escape method returns the escape character.

set-escape-map (→ none) (Character Character)

The set-escape-map method set the escape mapping table by using a source and target character. The first character is the source character. The second character is the target character.

get-escape-map (→ Character) (Character)

The get-escape-map method returns the escape mapping character by character. The source character is the argument.

Functions

sort-ascent (→ none) (Vector)

The sort-ascent function sorts in ascending order the vector argument. The vector is sorted in place.

sort-descent (→ none) (Vector)

The sort-descent function sorts in descending order the vector argument. The vector is sorted in place.

sort-lexical (→ none) (Vector)

The sort-lexical function sorts in lexicographic order the vector argument. The vector is sorted in place.

Text hashing and ciphering reference

Hasher

The `Hasher` class is a base class that is used to build a message hash. The hash result is stored in an array of bytes and can be retrieved byte by byte or as a formatted printable string. This class does not have a constructor.

Predicate

    hasher-p

Inheritance

    Object

Methods

get-name (→ String) (none)

The get-name method return the hash algorithm name.

get-hash (→ Character) (Integer)

The get-hash method returns the hash byte value by index. The argument is the byte index which must be in the range of the hash algorithm result value.

format (→ String) (none)

The format method return a string representation of the hash value.

compute (→ String) (Literal|Buffer|Input)

The compute method computes the hash value from a string, a buffer or an input stream. The method returns a string representation of the result hash value.

Md5

The `Md5` class is a hashing class that implements the MD-5 algorithm.

Predicate

    md5-p

Inheritance

    Hasher

Constructors

Md5 (none)

The Md5 constructor creates a default hashing object that implements the MD-5 algorithm.

Sha1

The `Sha1` class is a hashing class that implements the SHA-1 algorithm.

Predicate

    sha1-p

Inheritance

    Hasher

Constructors

Sha1 (none)

The Sha1 constructor creates a default hashing object that implements the SHA-1 algorithm.

Sha256

The `Sha256` class is a hashing class that implements the SHA-256 algorithm.

Predicate

    sha256-p

Inheritance

    Hasher

Constructors

Sha256 (none)

The Sha256 constructor creates a default hashing object that implements the SHA-256 algorithm.

Sha384

The `Sha384` class is a hashing class that implements the SHA-384 algorithm.

Predicate

    sha384-p

Inheritance

    Hasher

Constructors

Sha384 (none)

The Sha384 constructor creates a default hashing object that implements the SHA-384 algorithm.

Sha512

The `Sha512` class is a hashing class that implements the SHA-512 algorithm.

Predicate

    sha512-p

Inheritance

    Hasher

Constructors

Sha512 (none)

The Sha512 constructor creates a default hashing object that implements the SHA-512 algorithm.

Key

The `Key` class is a cipher key class that generates random key or message key. The key generation algorithm is based on the SHA-256 algorithm and accept different size. The default key size is 128 bits.

Predicate

    key-p

Inheritance

    Object

Constructors

Key (none)

The Key constructor creates a default cipher key. The key is generated with random bytes and is 128 bits long.

Key (K128|K192|K256)

The Key constructor creates a random cipher key by type. The argument is the key type which specify the key size.

Key (String)

The Key constructor creates a message cipher key those size is 128 bits long. The first argument is the message to be used for the key generation.

Key (K128|K192|K256 String)

The Key constructor creates a message cipher key by type. The first argument is the key type which specifies the key size. The second argument is the message to be used for the key generation.

Constants

K128

The K128 constant indicates that the key is a 128 bits key.

K192

The K192 constant indicates that the key is a 192 bits key.

K256

The K256 constant indicates that the key is a 256 bits key.

Methods

get (→ Character) (Integer)

The get method returns a key character by index. The index must be in the key range or an exception is raised.

format (→ String) (none)

The format method returns a string representation of the key. Each byte is represented with its equivalent string representation.

get-type (→ Item) (none)

The get-type method returns the key type in the form of an item object. The item object is associated with the key object and the key size constant. The item can be used in key constructor.

get-size (→ Integer) (none)

The get-size method returns the key size in bytes.

Cipher

The `Cipher` class is a base class that is used to implement a symmetric block cipher. A reverse flag controls the cipher operation. When false, the cipher operates in encrypt mode. When true, the cipher operates in decrypt mode, that is the reverse mode. This class cannot be instantiated by itself.

Predicate

    cipher-p

Inheritance

    Object

Methods

reset (→ none) (none)

The reset method reset the cipher internal state.

stream (→ Integer) (Output Input)

The stream method process an input stream and write into an output stream. The method returns the number of character processed. The first argument is the output stream used to write the coded characters. The second argument is the input stream used to read the characters.

set-key (→ none) (Key)

The set-key method sets the cipher key. The first argument is the key to set.

get-key (→ Key) (none)

The get-key method returns the cipher key.

get-name (→ String) (none)

The get-name method returns the cipher standard name.

get-size (→ Integer) (none)

The get-size method returns the cipher block size.

set-reverse (→ none) (Boolean)

The set-reverse method sets the cipher reverse flag. The first argument is the flag to set. If the flag is true, the cipher operates in reverse mode. If the flag is false, the cipher operates in direct mode.

get-reverse (→ Boolean) (none)

The get-reverse method returns the cipher reverse flag. If the flag is true, the cipher operates in reverse mode. If the flag is false, the cipher operates in direct mode.

Aes

The `Aes` class is a cipher class that implements the advanced encryption standard or AES block cipher. The AES cipher is a symmetric block cipher.

Predicate

    aes-p

Inheritance

    Cipher

Constructors

Aes (Key)

The Aes constructor creates a direct cipher with a key. The first argument is the key used by the cipher.

Aes (Key Boolean)

The Aes constructor creates a cipher with a key and a reverse flag. The first argument is the key used by the cipher. The second argument is the reverse flag.

InputCipher

The `InputCipher` class is an stream interface that can stream out an input stream from a cipher. In other word, an input stream is read and block are encoded as long as the input stream read characters. If the cipher is nil, the input cipher simply read the input stream and is therefore transparent. The class acts like an input stream, read the character from the bounded input stream and encode or decode them from the bounded cipher. The `InputCipher` defines two modes of operations. In electronic codebook mode or ECB, the character are encoded in a block basis. In cipher block chaining mode, the block are encoded by doing an XOR operation with the previous block. In all cases, a block padding is added, following the recommendation of RFC 2630 and NIST 800-38A.

Predicate

    input-cipher-p

Inheritance

    Input

Constructors

InputCipher (Cipher)

The InputCipher constructor creates an input cipher with a cipher object. The first argument is the cipher to used for processing.

InputCipher (Cipher Input)

The InputCipher constructor creates an input cipher with a cipher object and an input stream. The first argument is the cipher to used for processing. The second argument is the input stream object used for the character reading.

InputCipher (Cipher Input ECB|CBC)

The InputCipher constructor creates an input cipher with a cipher object, an input stream and a mode. The first argument is the cipher to used for processing. The second argument is the input stream object used for the character reading. The third argument is the input cipher mode which can be either ECB or CBC.

Constants

ECB

The ECB constant indicates that the input cipher is to operate in electronic codebook mode. This mode is the default mode.

CBC

The CBC constant indicates that the input cipher is to operate in cipher chaining block mode.

Methods

set-is (→ none) (Input)

The set-is method sets the input cipher input stream. This method can be used to chain multiple input streams in a unique coding session.

dup (→ InputCipher) (Input)

The dup method duplicate the input cipher by binding the same input stream parameter and attaching the input stream argument. This method is particularly useful with InputMapped stream.