# XPath and XSLT extensions

Sometimes the built-in facilities of XPath and XSLT aren't quite enough to meet your processing needs. Luckily it's easy to extend the function of these libraries using user extension functions and elements, which are written in Python.

## Extension functions (XPath and XSLT)

To define your own extension functions for XPath and XSLT, you write corresponding Python function in a module, and provide a mapping from the desired XPath function names to Python function objects (or any callables). Start with a simple example. The following is a complete module which defines a single XPath function, `unichr(s)` a simple example that takes a string and returns the Unicode code point number for the first character in that string.

    #ord.py
    from Ft.Xml.XPath import Conversions

    def Ord(context, s):
        '''
        Available in XPath as ord() as defined by ExtFunctions mapping below
        Takes an object, which is coerced to string
        Returns the Unicode code point number for the first character in that string    Or returns -1 if it's an empty string
        '''
        s = Conversions.StringValue(s)  #Coerce the passed object to string
        if s:
            return ord(s[0])
        else:
            return -1

    ExtFunctions = {
        (u'urn:x-4suite:x', u'ord'): Ord,
    }

As this simple example illustrates, The basic way to map XPath function names to Python function objects is in dictionary named "ExtFunctions", global to the module in which the extension function is defined. The XPath/XSLT extension names are expressed as a Python tuple of two Unicode objects. If you're familiar with XPath, this is just a Python representation of an expanded name. The first item in the expanded name tuple is the namespace URI for the element, and the second is the local name. The namespace URI cannot be an empty string.

You have to actually tell the processor to load your extension modules. There are several ways to do so.

1. From Python code you can register them in a context object used for XPath processing by using the optional `extModuleList` to pass in a list of module objects.

    from Ft.Xml import Parse
    from Ft.Xml.XPath.Context import Context
    from Ft.Xml.XPath import Evaluate
    import ord #The code listed above

    doc = Parse('<doc>abc</doc>')
    ctx = Context(doc, extModuleList=[ord],
                  processorNss={u'ext': u'urn:x-4suite:x'})
    nodes = Evaluate(u'ext:ord(.)', ctx)

2. You can also register particular functions rather than a complete module in a XPath context object using the optional `extFunctionMap` argument. It takes a mapping dictionary similar to the `ExtFunctions` dictionary shown in the above sample module.

    from Ft.Xml import Parse
    from Ft.Xml.XPath.Context import Context
    from Ft.Xml.XPath import Evaluate
    from ord import Ord #The code listed above

    doc = Parse('<doc>abc</doc>')
    ctx = Context(doc, extFunctionMap={(u'urn:x-4suite:x', u'ord'): Ord},
                  processorNss={u'ext': u'urn:x-4suite:x'})
    nodes = Evaluate(u'ext:ord(.)', ctx)

3. If you are using the XSLT processor you can register extension functions on a processor object using the `registerExtensionModules()` method.

    from Ft.Xml.Xslt import Processor
    import ord #The code listed above

    processor = Processor.Processor()
    processor.registerExtensionModules([ord])
    #Now continue to append stylesheets and run against source docs.  The
    #extensions i the module ord will be available

4. When using the XSLT processor you can also register individual extension functions on a processor object using `registerExtentionFunction()` method. It takes the namespace and localName for the extension function and the callable object that implements it).

    from Ft.Xml.Xslt import Processor
    from ord import Ord #The code listed above

    processor = Processor.Processor()
    processor.registerExtensionFunction(u'urn:x-4suite:x', u'ord', Ord)
    #Now continue to append stylesheets and run against source docs.  The
    #ext:ord() function will be available

5. In some cases the user can list extension modules using the environment variable "EXTMODULES". "EXTMODULES" is a colon-separated list of Python modules names. This works for the 4xslt command line and for `Ft.Xml.XPath.Evaluate`. For other APIs, use one of the other two methods, which can easily be extended to read the "EXTMODULES" variable. In general the other methods for registering extensions are preferable.

Note that extension modules will automatically be searched for XSLT extension elements as well as functions.

The following is a longer example, a module that implements two functions are. One returns the current time and the other creates a hash of the context node name:

    # demo.py
    import time, urlparse
    from Ft.Xml.XPath import Conversions

    def GetCurrentTime(context):
        '''available in XPath as get-current-time()'''
        return time.asctime(time.localtime())

    def HashContextName(context, maxkey):
        '''
        available in XPath as hash-context-name(maxkey),
        where maxkey is an object converted to number
        '''
        # It is a good idea to use the appropriate core function to coerce
        # arguments to the expected type
        maxkey = Conversions.NumberValue(maxkey)
        key = reduce(lambda a, b: a + b, context.node.nodeName)
        return key % maxkey

    ExtFunctions = {
        ('urn:x-4suite:x', 'get-current-time'): GetCurrentTime,
        ('urn:x-4suite:x', 'hash-context-name'): HashContextName
    }

You can use this in plain XPath as follows:

    from Ft.Xml.XPath.Context import Context
    from Ft.Xml.XPath import Compile, Evaluate
    from Ft.Xml.Domlette import NonvalidatingReader

    DOC = "<spam xmlns='http://spam.com'>eggs</spam>"

    ctx = Context(None, extFunctionMap=demo.ExtFunctions,
                  processorNss={"a": "http://spam.com"})
    expr = Compile("get-current-time()")

    doc = NonvalidatingReader.parseString(DOC, "http://spam.com/base")
    print Evaluate(expr, doc, ctx)

Notice that you might choose to use None for the extension function namespaces. If so, you don't need to specify the processorNss context attribute, but you might want to watch out for clashes with other extenstion function names, including the built-in library. Again, if you plan to use an extension function from within XSLT, its namespace URI must not be None.

You can use this in XSLT just as easily:

    # useextfunc.py

    TRANSFORM = """<?xml version="1.0"?>
    <xsl:stylesheet
      xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
      xmlns:s="http://spam.com"
      version="1.0">

      <xsl:template match="/">
        <xsl:value-of select="s:get-current-time()"/>
      </xsl:template>

    </xsl:stylesheet>
    """

    SOURCE = """<dummy/>"""

    from Ft.Xml.Xslt import Processor
    processor = Processor.Processor()

    # Register the extension function using method (3)
    processor.registerExtensionModules(['demo'])
    from Ft.Xml import InputSource
    transform = InputSource.DefaultFactory.fromString(TRANSFORM, "http://foo.com")
    source = InputSource.DefaultFactory.fromString(SOURCE, "http://foo.com")
    processor.appendStylesheet(transform)
    result = processor.run(source)
    print result

For good examples of modules with extension elements, see the source code for the modules `Ft.Xml.XPath.BuiltInExtFunctions`, `Ft.Xml.Xslt.BuiltInExtFunctions` and the modules in Ft.Xml.Xslt.Exslt. The latter are especially good examples given their diversity and detailed specifications at exslt.org.

## Extension elements (XSLT)

To define your own extension elements, define a class derived from `Ft.Xml.Xslt.XsltElement`. The module in which it is defined should have a global dictionary named "ExtElements" mapping element expanded names to element class objects.

Finally, modules containing any extension elements used must be indicated as such to the processor in one of several ways.

1. You can register all extension functions and elements in a module by using a processor object's `registerExtensionModules()` method.

    from Ft.Xml.Xslt import Processor
    import ord #The code listed above

    processor = Processor.Processor()
    processor.registerExtensionModules([ord])
    #Now continue to append stylesheets and run against source docs.  The
    #extensions i the module ord will be available

2. You can also register individual extension elements on a processor object using `registerExtensionElement()` method. It takes the namespace and localName for the extension function and the callable object that implements it).

3. In some cases the user can list extension modules using the environment variable "EXTMODULES". "EXTMODULES" is a colon-separated list of Python modules names. This works for the 4xslt command line and for `Ft.Xml.XPath.Evaluate`. For other APIs, use one of the other two methods, which can easily be extended to read the "EXTMODULES" variable. In general the other methods for registering extensions are preferable.

Note that extension modules will automatically be searched for XPath extension functions as well as Extension elements.

## Extension element API

There are several aspects of the extension element API worth discussing in more detail.

The class-level "content" variable specifies a content model to be enforced by the XSLT processor. If the element is used in a way that doesn't meet the specified content model, the user will get an error message. The content model is a structure that uses certain special classes, including:

- ContentInfo.Empty - matches no content at all (empty element)

- ContentInfo.Text - matches plain text content

- ContentInfo.Seq - matches the given sequence of sub-patterns

- ContentInfo.Alt - matches one of the given choice of sub-patterns

- ContentInfo.Rep - matches 0 or more repeated instances of the given sub-pattern

- ContentInfo.Rep1 - matches 0 or more repeated instances of the given sub-pattern

- ContentInfo.Opt - matches zero or one of the given sub-pattern

- ContentInfo.ResultElements - matches elements not in the XSL namespace

- ContentInfo.Instructions - matches any sequence of XSLT elements categorized as instructions in the spec

- ContentInfo.Template - matches an XSLT template body according to the spec

- ContentInfo.TopLevelElements - matches any sequence of XSLT elements categorized as top level in the spec

- ContentInfo.QName - matches a particular element by giving its namespace and node name (the prefix in the node name is only used for documentation and error messages)

So, for instance, the xsl:choose element would be described as

    content = ContentInfo.Seq(
        ContentInfo.Rep1(ContentInfo.QName(XSL_NAMESPACE, 'xsl:when')),
        ContentInfo.Opt(ContentInfo.QName(XSL_NAMESPACE, 'xsl:otherwise')),
        )

The class-level "legalAttrs" variable specifies the attributes allowed or required on the element. It is a Python dictionary mapping attribute name to its specification. The specification is a class according o the type of attribute.

The following are the supported attribute classes. The parameters specified are for the initializer. Note that most general patterns have a plain variant and an attribute value template (AVT) variant:

- AttributeInfo.String - any XPath string

- AttributeInfo.StringAvt - an AVT yielding any string

- AttributeInfo.Char - any XPath string of length 1

- AttributeInfo.CharAvt - AVT version of Char

- AttributeInfo.Choice - a string which must be one of a number of given values. The values are given by a list of strings with is the first parameter

- AttributeInfo.ChoiceAvt - AVT version of Choice

- AttributeInfo.YesNo - Abbreviation for AttributeInfo.Choice ( See Oasis web site)

- AttributeInfo.YesNoAvt - AVT version of YesNo

- AttributeInfo.Number - any XPath number

- AttributeInfo.NumberAvt - AVT version of Number

- AttributeInfo.UriReference - XPath string that is syntactically a URI reference

- AttributeInfo.UriReferenceAvt - AVT version of UriReference

- AttributeInfo.Id - XPath string that is syntactically an XML ID

- AttributeInfo.IdAvt - AVT version of Id

- AttributeInfo.QName - XPath string that is syntactically an XML namespaces qualified name

- AttributeInfo.QNameAvt - AVT version of QName

- AttributeInfo.NCName - XPath string that is syntactically an XML namespaces "no colon" name

- AttributeInfo.NCNameAvt - AVT version of NCName

- AttributeInfo.Prefix - Same as NCName

- AttributeInfo.PrefixAvt - Same as NCNameAvt

- AttributeInfo.NMToken - XPath string that is syntactically an XML Name token

- AttributeInfo.NMTokenAvt - AVT version of NMToken

- AttributeInfo.QNameButNotNCName - A QName that contains a colon

- AttributeInfo.QNameButNotNCNameAvt - AVT version of QNameButNotNCName

- AttributeInfo.Token - XPath string that is syntactically an XPath name test (i.e. "foo", "ns:foo", ns:" or "")

- AttributeInfo.TokenAvt - AVT version of Token

- AttributeInfo.Expression - XPath string that is syntactically an XPath expression

- AttributeInfo.ExpressionAvt - AVT version of Expression

- AttributeInfo.StringExpression - XPath string that is syntactically an XPath expression, which would be expected to return a string value

- AttributeInfo.StringExpressionAvt - AVT version of StringExpression

- AttributeInfo.NodeSetExpression - XPath string that is syntactically an XPath expression, which would be expected to return a node set value

- AttributeInfo.NodeSetExpressionAvt - AVT version of NodeSetExpression

- AttributeInfo.NumberExpression - XPath string that is syntactically an XPath expression, which would be expected to return a number value

- AttributeInfo.NumberExpressionAvt - AVT version of NumberExpression

- AttributeInfo.BooleanExpression - XPath string that is syntactically an XPath expression, which would be expected to return a boolean value

- AttributeInfo.BooleanExpressionAvt - AVT version of BooleanExpression

- AttributeInfo.Pattern - XPath string that is syntactically an XSLY pattern

- AttributeInfo.PatternAvt - AVT version of Pattern

- AttributeInfo.Tokens - XPath string that is syntactically a space-delimited series of tokens

- AttributeInfo.TokensAvt - AVT version of Tokens

- AttributeInfo.QNames - XPath string that is syntactically a space-delimited series of QNames

- AttributeInfo.QNamesAvt - AVT version of QNames

- AttributeInfo.Prefixes - XPath string that is syntactically a space-delimited series of NCNames

- AttributeInfo.PrefixesAvt - AVT version of Prefixes

All of these classes take the following optional keyword parameters:

- description - for documentation

- default - the default value of the attribute to be used if omitted

Some examples from the XSLT spec:

xsl:output

    content = ContentInfo.Empty
    legalAttrs = {
        'method' : AttributeInfo.QName(),
        'version' : AttributeInfo.NMToken(),
        'encoding' : AttributeInfo.String(),
        'omit-xml-declaration' : AttributeInfo.YesNo(),
        'standalone' : AttributeInfo.YesNo(),
        'doctype-public' : AttributeInfo.String(),
        'doctype-system' : AttributeInfo.String(),
        'cdata-section-elements' : AttributeInfo.QNames(),
        'indent' : AttributeInfo.YesNo(),
        'media-type' : AttributeInfo.String(),
        }

xsl:sort

    content = ContentInfo.Empty
    legalAttrs = {
        'select' : AttributeInfo.StringExpression(default='.'),
        'lang' : AttributeInfo.NMTokenAvt(),
        # We don't support any additional data-types, hence no
        # AttributeInfo.QNameButNotNCName()
        'data-type' : AttributeInfo.ChoiceAvt(['text', 'number'],
                                              default='text'),
        'order' : AttributeInfo.ChoiceAvt(['ascending', 'descending'],
                                          default='ascending'),
        'case-order' : AttributeInfo.ChoiceAvt(['upper-first', 'lower-first']),
        }

xsl:number

    content = ContentInfo.Empty
    legalAttrs = {
        'level' : AttributeInfo.Choice(['single', 'multiple', 'any'],
                                       default='single'),
        'count' : AttributeInfo.Pattern(),
        'from' : AttributeInfo.Pattern(),
        'value' : AttributeInfo.Expression(),
        'format' : AttributeInfo.StringAvt(default='1'),
        'lang' : AttributeInfo.NMToken(),
        'letter-value' : AttributeInfo.ChoiceAvt(['alphabetic', 'traditional']),
        'grouping-separator' : AttributeInfo.CharAvt(),
        'grouping-size' : AttributeInfo.NumberAvt(default=0),
        }

Of course, it's always a good idea to use descriptions, which the above do not.

For good examples of modules with extension elements, see the source code for the modules Ft.Xml.Xslt.BuiltInExtElements and Ft.Xml.Xslt.Exslt.Common . The various modules in Ft.Xml.Xslt.Exslt have a strong diversity and make good examples, especially given their detailed specifications at exslt.org

### Controlling output from XSLT extensions

The most common special need for XSLT extensions is to generate XSLT output. For extension elements this is easy enough to do using the API on the procssor instance that is passed to the instantiate() method of extension element classes. For example

    class SpamElement(XsltElement):
        legalAttrs = {}
        def instantiate(self, context, processor):
            processor.output().startElement('title')
            processor.output().text('Life of Brian'))
            processor.output().endElement('title')
            return (context,)

Extension functions are not passed a processor instance directly, but context objects hold a reference to the processor in effect, so the following example works:

    def Spam(context):
        context.processor.output().startElement('title')
        context.processor.output().text('Life of Brian'))
        context.processor.output().endElement('title')
        return

However, it is probably better design to reserve such side effects as output for extension elements rather than functions.

In the above examples the elements and text out out just use the current output parameters. In order to change output parameters or change the output stream, you can stack a new output handler:

    stream = cStringIO.StringIO()

    # Clone the current outputparameters
    op = processor.writers[-1]._outputParams.clone()

    # Force XML output method with XML declaration
    # Output method is a qualified name, so must flag null ns
    # to use standard xml method
    op.method = (EMPTY_NAMESPACE, 'xml')
    op.omitXmlDeclaration = "yes"

    # Push the new handler to the top of the writer stack
    processor.addHandler(op, stream)
    processor.output().startElement('title')
    processor.output().text('Life of Brian'))
    processor.output().endElement('title')

    # Pop back to the previous handler stream.getvalue()
    # now contains the new  output
    processor.removeHandler()

### Creating result tree fragments

Another common need is to treat the body of an extension element as a template so that something can be done with the RTF that results from it. The following example demonstrates this:

    try:
        # Set the output to an RTF riter, which wll create an RTF for us
        processor.pushResultTree(self.baseUri)

        # The template is manifested as children of the extension element
        # node.  Instantiate each in turn
        for child in self.children:
            child.instantiate(context, processor)
    # You want to be sure you re-balance the stack even in case of error
    finally:
        # Retrieve the resulting RTF
        result_rtf = processor.popResult()

### Comunicating with the external code that invokes XSLT

You can set and communicate state information with external code by using the processor.extensionParams attribute. For example, the following sents a time stamp of precisely when the extension was instantiated, which can later be retrieved from the processor after the XSLT process, or even by later extensions. In a similar way, state can be set up by calling functions and retrieved by extensions.

    # Extension parameters have fully qualified names, so you must come up
    # with a namespace to set them
    processor.extensionParams[(SPAM_NAMESPACE, 'tstamp')] = time.time()