- lxml

- Developing with lxml

- Extending lxml

- Developing lxml

# Benchmarks and Speed

Author: | Stefan Behnel

lxml.etree is a very fast XML library. Most of this is due to the speed of libxml2, e.g. the parser and serialiser, or the XPath engine. Other areas of lxml were specifically written for high performance in high-level operations, such as the tree iterators.

On the other hand, the simplicity of lxml sometimes hides internal operations that are more costly than the API suggests. If you are not aware of these cases, lxml may not always perform as you expect. A common example in the Python world is the Python list type. New users often expect it to be a linked list, while it actually is implemented as an array, which results in a completely different complexity for common operations.

Similarly, the tree model of libxml2 is more complex than what lxml's ElementTree API projects into Python space, so some operations may show unexpected performance. Rest assured that most lxml users will not notice this in real life, as lxml is very fast in absolute numbers. It is definitely fast enough for most applications, so lxml is probably somewhere between 'fast enough' and 'the best choice' for yours. Read some messages from happy users to see what we mean.

This text describes where lxml.etree (abbreviated to 'lxe') excels, gives hints on some performance traps and compares the overall performance to the original ElementTree (ET) and cElementTree (cET) libraries by Fredrik Lundh. The cElementTree library is a fast C-implementation of the original ElementTree.

Contents

- General notes
- How to read the timings
- Parsing and Serialising
- The ElementTree API
  - Child access
  - Element creation
  - Merging different sources
  - deepcopy
  - Tree traversal
- XPath
- A longer example
- lxml.objectify
  - ObjectPath
  - Caching Elements
  - Further optimisations

# General notes

First thing to say: there is an overhead involved in having a DOM-like C library mimic the ElementTree API. As opposed to ElementTree, lxml has to generate Python representations of tree nodes on the fly when asked for them, and the internal tree structure of libxml2 results in a higher maintenance overhead than the simpler top-down structure of ElementTree. What this means is: the more of your code runs in Python, the less you can benefit from the speed of lxml and libxml2. Note, however, that this is true for most performance critical Python applications. No one would implement fourier transformations in pure Python when you can use NumPy.

The up side then is that lxml provides powerful tools like tree iterators, XPath and XSLT, that can handle complex operations at the speed of C. Their pythonic API in lxml makes them so flexible that most applications can easily benefit from them.

# How to read the timings

The statements made here are backed by the (micro-)benchmark scripts bench_etree.py, bench_xpath.py and bench_objectify.py that come with the lxml source distribution. They are distributed under the same BSD license as lxml itself, and the lxml project would like to promote them as a general benchmarking suite for all ElementTree implementations. New benchmarks are very easy to add as tiny test methods, so if you write a performance test for a specific part of the API yourself, please consider sending it to the lxml mailing list.

The timings cited below compare lxml 2.1 (with libxml2 2.6.33) to the April 2008 SVN trunk versions of ElementTree (1.3alpha) and cElementTree (1.2.7). They were run single-threaded on a 1.8GHz Intel Core Duo machine under Ubuntu Linux 7.10 (Gutsy). The C libraries were compiled with the same platform specific optimisation flags. The Python interpreter (2.5.1) was used as provided by the distribution.

The scripts run a number of simple tests on the different libraries, using different XML tree configurations: different tree sizes (T1-4), with or without attributes (-/A), with or without ASCII string or unicode text (-/S/U), and either against a tree or its serialised XML form (T/X). In the result extracts cited below, T1 refers to a 3-level tree with many children at the third level, T2 is swapped around to have many children below the root element, T3 is a deep tree with few children at each level and T4 is a small tree, slightly broader than deep. If repetition is involved, this usually means running the benchmark in a loop over all children of the tree root, otherwise, the operation is run on the root node (C/R).

As an example, the character code `(SATR T1)` states that the benchmark was running for tree T1, with plain string text (S) and attributes (A). It was run against the root element (R) in the tree structure of the data (T).

Note that very small operations are repeated in integer loops to make them measurable. It is therefore not always possible to compare the absolute timings of, say, a single access benchmark (which usually loops) and a 'get all in one step' benchmark, which already takes enough time to be measurable and is therefore measured as is. An example is the index access to a single child, which cannot be compared to the timings for `getchildren()`. Take a look at the concrete benchmarks in the scripts to understand how the numbers compare.

# Parsing and Serialising

Serialisation is an area where lxml excels. The reason is that it executes entirely at the C level, without any interaction with Python code. The results are rather impressive, especially for UTF-8, which is native to libxml2. While 20 to 40 times faster than (c)ElementTree 1.2 (which is part of the standard library in Python 2.5), lxml is still more than 7 times as fast as the much improved ElementTree 1.3:

    lxe: tostring_utf16  (SATR T1)   25.7590 msec/pass
    cET: tostring_utf16  (SATR T1)  179.6291 msec/pass
    ET : tostring_utf16  (SATR T1)  188.5638 msec/pass

    lxe: tostring_utf16  (UATR T1)   26.0060 msec/pass
    cET: tostring_utf16  (UATR T1)  176.9981 msec/pass
    ET : tostring_utf16  (UATR T1)  188.2110 msec/pass

    lxe: tostring_utf16  (S-TR T2)   26.9201 msec/pass
    cET: tostring_utf16  (S-TR T2)  182.5061 msec/pass
    ET : tostring_utf16  (S-TR T2)  190.2061 msec/pass

    lxe: tostring_utf8   (S-TR T2)   19.5830 msec/pass
    cET: tostring_utf8   (S-TR T2)  183.0020 msec/pass
    ET : tostring_utf8   (S-TR T2)  187.7251 msec/pass

    lxe: tostring_utf8   (U-TR T3)    5.5292 msec/pass
    cET: tostring_utf8   (U-TR T3)   56.1349 msec/pass
    ET : tostring_utf8   (U-TR T3)   56.6628 msec/pass

The same applies to plain text serialisation. Note that cElementTree does not currently support this, as it is new in ET 1.3:

    lxe: tostring_text_ascii     (S-TR T1)    3.8729 msec/pass
    ET : tostring_text_ascii     (S-TR T1)   90.7841 msec/pass

    lxe: tostring_text_ascii     (S-TR T3)    1.1508 msec/pass
    ET : tostring_text_ascii     (S-TR T3)   28.0581 msec/pass

    lxe: tostring_text_utf16     (S-TR T1)    5.6219 msec/pass
    ET : tostring_text_utf16     (S-TR T1)   87.4891 msec/pass

    lxe: tostring_text_utf16     (U-TR T1)    7.0660 msec/pass
    ET : tostring_text_utf16     (U-TR T1)   82.1049 msec/pass

Unlike ElementTree, the `tostring()` function in lxml also supports serialisation to a Python unicode string object:

    lxe: tostring_text_unicode   (S-TR T1)    4.2419 msec/pass
    lxe: tostring_text_unicode   (U-TR T1)    5.2760 msec/pass
    lxe: tostring_text_unicode   (S-TR T3)    1.3049 msec/pass
    lxe: tostring_text_unicode   (U-TR T3)    1.4210 msec/pass

For parsing, on the other hand, the advantage is clearly with cElementTree. The (c)ET libraries use a very thin layer on top of the expat parser, which is known to be extremely fast:

    lxe: parse_stringIO  (SAXR T1)   40.6771 msec/pass
    cET: parse_stringIO  (SAXR T1)   19.3741 msec/pass
    ET : parse_stringIO  (SAXR T1)  355.7711 msec/pass

    lxe: parse_stringIO  (S-XR T3)    5.9960 msec/pass
    cET: parse_stringIO  (S-XR T3)    5.8751 msec/pass
    ET : parse_stringIO  (S-XR T3)   93.7259 msec/pass

    lxe: parse_stringIO  (UAXR T3)   26.2671 msec/pass
    cET: parse_stringIO  (UAXR T3)   30.6449 msec/pass
    ET : parse_stringIO  (UAXR T3)  178.8890 msec/pass

While about as fast for smaller documents, the expat parser allows cET to be up to 2 times faster than lxml on plain parser performance for large input documents. Similar timings can be observed for the `iterparse()` function:

    lxe: iterparse_stringIO  (SAXR T1)   50.8120 msec/pass
    cET: iterparse_stringIO  (SAXR T1)   24.9379 msec/pass
    ET : iterparse_stringIO  (SAXR T1)  388.9420 msec/pass

    lxe: iterparse_stringIO  (UAXR T3)   29.0790 msec/pass
    cET: iterparse_stringIO  (UAXR T3)   32.1240 msec/pass
    ET : iterparse_stringIO  (UAXR T3)  189.1720 msec/pass

However, if you benchmark the complete round-trip of a serialise-parse cycle, the numbers will look similar to these:

    lxe: write_utf8_parse_stringIO  (S-TR T1)   63.7550 msec/pass
    cET: write_utf8_parse_stringIO  (S-TR T1)  292.0721 msec/pass
    ET : write_utf8_parse_stringIO  (S-TR T1)  635.2799 msec/pass

    lxe: write_utf8_parse_stringIO  (UATR T2)   75.0258 msec/pass
    cET: write_utf8_parse_stringIO  (UATR T2)  341.7251 msec/pass
    ET : write_utf8_parse_stringIO  (UATR T2)  713.1951 msec/pass

    lxe: write_utf8_parse_stringIO  (S-TR T3)   11.4899 msec/pass
    cET: write_utf8_parse_stringIO  (S-TR T3)   96.8502 msec/pass
    ET : write_utf8_parse_stringIO  (S-TR T3)  185.6079 msec/pass

    lxe: write_utf8_parse_stringIO  (SATR T4)    1.2081 msec/pass
    cET: write_utf8_parse_stringIO  (SATR T4)    6.8581 msec/pass
    ET : write_utf8_parse_stringIO  (SATR T4)   10.6261 msec/pass

For applications that require a high parser throughput of large files, and that do little to no serialization, cET is the best choice. Also for iterparse applications that extract small amounts of data from large XML data sets that do not fit into the memory. If it comes to round-trip performance, however, lxml tends to be multiple times faster in total. So, whenever the input documents are not considerably larger than the output, lxml is the clear winner.

Regarding HTML parsing, Ian Bicking has done some benchmarking on lxml's HTML parser, comparing it to a number of other famous HTML parser tools for Python. lxml wins this contest by quite a length. To give an idea, the numbers suggest that lxml.html can run a couple of parse-serialise cycles in the time that other tools need for parsing alone. The comparison even shows some very favourable results regarding memory consumption.

# The ElementTree API

Since all three libraries implement the same API, their performance is easy to compare in this area. A major disadvantage for lxml's performance is the different tree model that underlies libxml2. It allows lxml to provide parent pointers for elements, but also increases the overhead of tree building and restructuring. This can be seen from the tree setup times of the benchmark (given in seconds):

    lxe:       --     S-     U-     -A     SA     UA
         T1: 0.0437 0.0498 0.0516 0.0430 0.0498 0.0519
         T2: 0.0550 0.0643 0.0677 0.0612 0.0685 0.0721
         T3: 0.0168 0.0142 0.0159 0.0338 0.0350 0.0359
         T4: 0.0003 0.0002 0.0003 0.0007 0.0007 0.0007
    cET:       --     S-     U-     -A     SA     UA
         T1: 0.0093 0.0093 0.0093 0.0097 0.0094 0.0094
         T2: 0.0153 0.0155 0.0152 0.0157 0.0154 0.0154
         T3: 0.0076 0.0076 0.0076 0.0099 0.0122 0.0100
         T4: 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001
    ET :       --     S-     U-     -A     SA     UA
         T1: 0.1074 0.1669 0.1050 0.2054 0.2401 0.1047
         T2: 0.2920 0.1172 0.3393 0.3830 0.1184 0.4215
         T3: 0.0347 0.0331 0.0316 0.0368 0.3944 0.0377
         T4: 0.0006 0.0005 0.0007 0.0006 0.0007 0.0006

While lxml is still faster than ET in most cases (10-70%), cET can be up to five times faster than lxml here. One of the reasons is that lxml must additionally discard the created Python elements after their use, when they are no longer referenced. ET and cET represent the tree itself through these objects, which reduces the overhead in creating them.

## Child access

The same reason makes operations like collecting children as in `list(element)` more costly in lxml. Where ET and cET can quickly create a shallow copy of their list of children, lxml has to create a Python object for each child and collect them in a list:

    lxe: root_list_children        (--TR T1)    0.0160 msec/pass
    cET: root_list_children        (--TR T1)    0.0081 msec/pass
    ET : root_list_children        (--TR T1)    0.0541 msec/pass

    lxe: root_list_children        (--TR T2)    0.2100 msec/pass
    cET: root_list_children        (--TR T2)    0.0319 msec/pass
    ET : root_list_children        (--TR T2)    0.4420 msec/pass

This handicap is also visible when accessing single children:

    lxe: first_child               (--TR T2)    0.2341 msec/pass
    cET: first_child               (--TR T2)    0.2198 msec/pass
    ET : first_child               (--TR T2)    0.8960 msec/pass

    lxe: last_child                (--TR T1   )    0.2549 msec/pass
    cET: last_child                (--TR T1   )    0.2251 msec/pass
    ET : last_child                (--TR T1   )    0.8969 msec/pass

... unless you also add the time to find a child index in a bigger list. ET and cET use Python lists here, which are based on arrays. The data structure used by libxml2 is a linked tree, and thus, a linked list of children:

    lxe: middle_child              (--TR T1)    0.2699 msec/pass
    cET: middle_child              (--TR T1)    0.2089 msec/pass
    ET : middle_child              (--TR T1)    0.8910 msec/pass

    lxe: middle_child              (--TR T2)    1.9410 msec/pass
    cET: middle_child              (--TR T2)    0.2151 msec/pass
    ET : middle_child              (--TR T2)    0.8960 msec/pass

## Element creation

As opposed to ET, libxml2 has a notion of documents that each element must be in. This results in a major performance difference for creating independent Elements that end up in independently created documents:

    lxe: create_elements           (--TC T2)    1.7340 msec/pass
    cET: create_elements           (--TC T2)    0.1929 msec/pass
    ET : create_elements           (--TC T2)    1.3809 msec/pass

Therefore, it is always preferable to create Elements for the document they are supposed to end up in, either as SubElements of an Element or using the explicit `Element.makeelement()` call:

    lxe: makeelement               (--TC T2)    1.6100 msec/pass
    cET: makeelement               (--TC T2)    0.3171 msec/pass
    ET : makeelement               (--TC T2)    1.6270 msec/pass

    lxe: create_subelements        (--TC T2)    1.3542 msec/pass
    cET: create_subelements        (--TC T2)    0.2329 msec/pass
    ET : create_subelements        (--TC T2)    3.3019 msec/pass

So, if the main performance bottleneck of an application is creating large XML trees in memory through calls to Element and SubElement, cET is the best choice. Note, however, that the serialisation performance may even out this advantage, especially for smaller trees and trees with many attributes.

## Merging different sources

A critical action for lxml is moving elements between document contexts. It requires lxml to do recursive adaptations throughout the moved tree structure.

The following benchmark appends all root children of the second tree to the root of the first tree:

    lxe: append_from_document      (--TR T1,T2)    3.0038 msec/pass
    cET: append_from_document      (--TR T1,T2)    0.2639 msec/pass
    ET : append_from_document      (--TR T1,T2)    1.2522 msec/pass

    lxe: append_from_document      (--TR T3,T4)    0.0398 msec/pass
    cET: append_from_document      (--TR T3,T4)    0.0160 msec/pass
    ET : append_from_document      (--TR T3,T4)    0.0811 msec/pass

Although these are fairly small numbers compared to parsing, this easily shows the different performance classes for lxml and (c)ET. Where the latter do not have to care about parent pointers and tree structures, lxml has to deep traverse the appended tree. The performance difference therefore increases with the size of the tree that is moved.

This difference is not always as visible, but applies to most parts of the API, like inserting newly created elements:

    lxe: insert_from_document      (--TR T1,T2)    4.9140 msec/pass
    cET: insert_from_document      (--TR T1,T2)    0.4108 msec/pass
    ET : insert_from_document      (--TR T1,T2)    1.4670 msec/pass

or replacing the child slice by a newly created element:

    lxe: replace_children_element  (--TC T1)    0.1500 msec/pass
    cET: replace_children_element  (--TC T1)    0.0238 msec/pass
    ET : replace_children_element  (--TC T1)    0.1600 msec/pass

as opposed to replacing the slice with an existing element from the same document:

    lxe: replace_children          (--TC T1)    0.0160 msec/pass
    cET: replace_children          (--TC T1)    0.0119 msec/pass
    ET : replace_children          (--TC T1)    0.0741 msec/pass

While these numbers are too small to provide a major performance impact in practice, you should keep this difference in mind when you merge very large trees.

## deepcopy

Deep copying a tree is fast in lxml:

    lxe: deepcopy_all              (--TR T1)    9.4090 msec/pass
    cET: deepcopy_all              (--TR T1)  120.1589 msec/pass
    ET : deepcopy_all              (--TR T1)  901.3789 msec/pass

    lxe: deepcopy_all              (-ATR T2)   12.4569 msec/pass
    cET: deepcopy_all              (-ATR T2)  135.8809 msec/pass
    ET : deepcopy_all              (-ATR T2)  940.7840 msec/pass

    lxe: deepcopy_all              (S-TR T3)    2.7640 msec/pass
    cET: deepcopy_all              (S-TR T3)   30.1108 msec/pass
    ET : deepcopy_all              (S-TR T3)  228.4350 msec/pass

So, for example, if you have a database-like scenario where you parse in a large tree and then search and copy independent subtrees from it for further processing, lxml is by far the best choice here.

## Tree traversal

Another area where lxml is very fast is iteration for tree traversal. If your algorithms can benefit from step-by-step traversal of the XML tree and especially if few elements are of interest or the target element tag name is known, lxml is a good choice:

    lxe: getiterator_all      (--TR T1)    5.0449 msec/pass
    cET: getiterator_all      (--TR T1)   42.0539 msec/pass
    ET : getiterator_all      (--TR T1)   22.9158 msec/pass

    lxe: getiterator_islice   (--TR T2)    0.0789 msec/pass
    cET: getiterator_islice   (--TR T2)    0.3579 msec/pass
    ET : getiterator_islice   (--TR T2)    0.2351 msec/pass

    lxe: getiterator_tag      (--TR T2)    0.0651 msec/pass
    cET: getiterator_tag      (--TR T2)    0.7648 msec/pass
    ET : getiterator_tag      (--TR T2)    0.4380 msec/pass

    lxe: getiterator_tag_all  (--TR T2)    0.8650 msec/pass
    cET: getiterator_tag_all  (--TR T2)   42.7120 msec/pass
    ET : getiterator_tag_all  (--TR T2)   21.5559 msec/pass

This translates directly into similar timings for `Element.findall()`:

    lxe: findall              (--TR T2)    6.8750 msec/pass
    cET: findall              (--TR T2)   46.8600 msec/pass
    ET : findall              (--TR T2)   27.0121 msec/pass

    lxe: findall              (--TR T3)    1.5690 msec/pass
    cET: findall              (--TR T3)   13.6340 msec/pass
    ET : findall              (--TR T3)    8.8100 msec/pass

    lxe: findall_tag          (--TR T2)    1.0221 msec/pass
    cET: findall_tag          (--TR T2)   42.8400 msec/pass
    ET : findall_tag          (--TR T2)   21.4801 msec/pass

    lxe: findall_tag          (--TR T3)    0.4241 msec/pass
    cET: findall_tag          (--TR T3)   10.7069 msec/pass
    ET : findall_tag          (--TR T3)    5.8560 msec/pass

Note that all three libraries currently use the same Python implementation for `findall()`, except for their native tree iterator (`element.iter()`).

# XPath

The following timings are based on the benchmark script bench_xpath.py.

This part of lxml does not have an equivalent in ElementTree. However, lxml provides more than one way of accessing it and you should take care which part of the lxml API you use. The most straight forward way is to call the `xpath()` method on an Element or ElementTree:

    lxe: xpath_method         (--TC T1)    1.5969 msec/pass
    lxe: xpath_method         (--TC T2)   21.3680 msec/pass
    lxe: xpath_method         (--TC T3)    0.1218 msec/pass
    lxe: xpath_method         (--TC T4)    1.0300 msec/pass

This is well suited for testing and when the XPath expressions are as diverse as the trees they are called on. However, if you have a single XPath expression that you want to apply to a larger number of different elements, the `XPath` class is the most efficient way to do it:

    lxe: xpath_class          (--TC T1)    0.6590 msec/pass
    lxe: xpath_class          (--TC T2)    2.9969 msec/pass
    lxe: xpath_class          (--TC T3)    0.0520 msec/pass
    lxe: xpath_class          (--TC T4)    0.1619 msec/pass

Note that this still allows you to use variables in the expression, so you can parse it once and then adapt it through variables at call time. In other cases, where you have a fixed Element or ElementTree and want to run different expressions on it, you should consider the `XPathEvaluator`:

    lxe: xpath_element        (--TR T1)    0.4120 msec/pass
    lxe: xpath_element        (--TR T2)   11.5321 msec/pass
    lxe: xpath_element        (--TR T3)    0.1152 msec/pass
    lxe: xpath_element        (--TR T4)    0.3202 msec/pass

While it looks slightly slower, creating an XPath object for each of the expressions generates a much higher overhead here:

    lxe: xpath_class_repeat   (--TC T1)    1.5409 msec/pass
    lxe: xpath_class_repeat   (--TC T2)   20.2711 msec/pass
    lxe: xpath_class_repeat   (--TC T3)    0.1161 msec/pass
    lxe: xpath_class_repeat   (--TC T4)    0.9799 msec/pass

# A longer example

... based on lxml 1.3.

A while ago, Uche Ogbuji posted a benchmark proposal that would read in a 3MB XML version of the Old Testament of the Bible and look for the word begat in all verses. Apparently, it is contained in 120 out of almost 24000 verses. This is easy to implement in ElementTree using `findall()`. However, the fastest and most memory friendly way to do this is obviously `iterparse()`, as most of the data is not of any interest.

Now, Uche's original proposal was more or less the following:

    def bench_ET():
        tree = ElementTree.parse("ot.xml")
        result = []
        for v in tree.findall("//v"):
            text = v.text
            if 'begat' in text:
                result.append(text)
        return len(result)

which takes about one second on my machine today. The faster `iterparse()` variant looks like this:

    def bench_ET_iterparse():
        result = []
        for event, v in ElementTree.iterparse("ot.xml"):
            if v.tag == 'v':
                text = v.text
                if 'begat' in text:
                    result.append(text)
            v.clear()
        return len(result)

The improvement is about 10%. At the time I first tried (early 2006), lxml didn't have `iterparse()` support, but the `findall()` variant was already faster than ElementTree. This changes immediately when you switch to cElementTree. The latter only needs 0.17 seconds to do the trick today and only some impressive 0.10 seconds when running the iterparse version. And even back then, it was quite a bit faster than what lxml could achieve.

Since then, lxml has matured a lot and has gotten much faster. The iterparse variant now runs in 0.14 seconds, and if you remove the `v.clear()`, it is even a little faster (which isn't the case for cElementTree).

One of the many great tools in lxml is XPath, a swiss army knife for finding things in XML documents. It is possible to move the whole thing to a pure XPath implementation, which looks like this:

    def bench_lxml_xpath_all():
        tree = etree.parse("ot.xml")
        result = tree.xpath("//v[contains(., 'begat')]/text()")
        return len(result)

This runs in about 0.13 seconds and is about the shortest possible implementation (in lines of Python code) that I could come up with. Now, this is already a rather complex XPath expression compared to the simple "//v" ElementPath expression we started with. Since this is also valid XPath, let's try this instead:

    def bench_lxml_xpath():
        tree = etree.parse("ot.xml")
        result = []
        for v in tree.xpath("//v"):
            text = v.text
            if 'begat' in text:
                result.append(text)
        return len(result)

This gets us down to 0.12 seconds, thus showing that a generic XPath evaluation engine cannot always compete with a simpler, tailored solution. However, since this is not much different from the original findall variant, we can remove the complexity of the XPath call completely and just go with what we had in the beginning. Under lxml, this runs in the same 0.12 seconds.

But there is one thing left to try. We can replace the simple ElementPath expression with a native tree iterator:

    def bench_lxml_getiterator():
        tree = etree.parse("ot.xml")
        result = []
        for v in tree.getiterator("v"):
            text = v.text
            if 'begat' in text:
                result.append(text)
        return len(result)

This implements the same thing, just without the overhead of parsing and evaluating a path expression. And this makes it another bit faster, down to 0.11 seconds. For comparison, cElementTree runs this version in 0.17 seconds.

So, what have we learned?

- Python code is not slow. The pure XPath solution was not even as fast as the first shot Python implementation. In general, a few more lines in Python make things more readable, which is much more important than the last 5% of performance.
- It's important to know the available options - and it's worth starting with the most simple one. In this case, a programmer would then probably have started with `getiterator("v")` or `iterparse()`. Either of them would already have been the most efficient, depending on which library is used.
- It's important to know your tool. lxml and cElementTree are both very fast libraries, but they do not have the same performance characteristics. The fastest solution in one library can be comparatively slow in the other. If you optimise, optimise for the specific target platform.
- It's not always worth optimising. After all that hassle we got from 0.12 seconds for the initial implementation to 0.11 seconds. Switching over to cElementTree and writing an `iterparse()` based version would have given us 0.10 seconds - not a big difference for 3MB of XML.
- Take care what operation is really dominating in your use case. If we split up the operations, we can see that lxml is slightly slower than cElementTree on `parse()` (both about 0.06 seconds), but more visibly slower on `iterparse()`: 0.07 versus 0.10 seconds. However, tree iteration in lxml is increadibly fast, so it can be better to parse the whole tree and then iterate over it rather than using `iterparse()` to do both in one step. Or, you can just wait for the lxml authors to optimise iterparse in one of the next releases...

# lxml.objectify

The following timings are based on the benchmark script bench_objectify.py.

Objectify is a data-binding API for XML based on lxml.etree, that was added in version 1.1. It uses standard Python attribute access to traverse the XML tree. It also features ObjectPath, a fast path language based on the same meme.

Just like lxml.etree, lxml.objectify creates Python representations of elements on the fly. To save memory, the normal Python garbage collection mechanisms will discard them when their last reference is gone. In cases where deeply nested elements are frequently accessed through the objectify API, the create-discard cycles can become a bottleneck, as elements have to be instantiated over and over again.

## ObjectPath

ObjectPath can be used to speed up the access to elements that are deep in the tree. It avoids step-by-step Python element instantiations along the path, which can substantially improve the access time:

    lxe: attribute                  (--TR T1)    8.4081 msec/pass
    lxe: attribute                  (--TR T2)   51.3301 msec/pass
    lxe: attribute                  (--TR T4)    8.2269 msec/pass

    lxe: objectpath                 (--TR T1)    4.6120 msec/pass
    lxe: objectpath                 (--TR T2)   47.0440 msec/pass
    lxe: objectpath                 (--TR T4)    4.4930 msec/pass

    lxe: attributes_deep            (--TR T1)   12.6550 msec/pass
    lxe: attributes_deep            (--TR T2)   56.0241 msec/pass
    lxe: attributes_deep            (--TR T4)   12.5690 msec/pass

    lxe: objectpath_deep            (--TR T1)    5.9190 msec/pass
    lxe: objectpath_deep            (--TR T2)   49.6972 msec/pass
    lxe: objectpath_deep            (--TR T4)    5.7530 msec/pass

Note, however, that parsing ObjectPath expressions is not for free either, so this is most effective for frequently accessing the same element.

## Caching Elements

A way to improve the normal attribute access time is static instantiation of the Python objects, thus trading memory for speed. Just create a cache dictionary and run:

    cache[root] = list(root.iter())

after parsing and:

    del cache[root]

when you are done with the tree. This will keep the Python element representations of all elements alive and thus avoid the overhead of repeated Python object creation. You can also consider using filters or generator expressions to be more selective. By choosing the right trees (or even subtrees and elements) to cache, you can trade memory usage against access speed:

    lxe: attribute_cached           (--TR T1)    6.4209 msec/pass
    lxe: attribute_cached           (--TR T2)   48.0378 msec/pass
    lxe: attribute_cached           (--TR T4)    6.3779 msec/pass

    lxe: attributes_deep_cached     (--TR T1)    7.8559 msec/pass
    lxe: attributes_deep_cached     (--TR T2)   51.0719 msec/pass
    lxe: attributes_deep_cached     (--TR T4)    7.7350 msec/pass

    lxe: objectpath_deep_cached     (--TR T1)    3.2761 msec/pass
    lxe: objectpath_deep_cached     (--TR T2)   45.7590 msec/pass
    lxe: objectpath_deep_cached     (--TR T4)    3.1459 msec/pass

Things to note: you cannot currently use `weakref.WeakKeyDictionary` objects for this as lxml's element objects do not support weak references (which are costly in terms of memory). Also note that new element objects that you add to these trees will not turn up in the cache automatically and will therefore still be garbage collected when all their Python references are gone, so this is most effective for largely immutable trees. You should consider using a set instead of a list in this case and add new elements by hand.

## Further optimisations

Here are some more things to try if optimisation is required:

- A lot of time is usually spent in tree traversal to find the addressed elements in the tree. If you often work in subtrees, do what you would also do with deep Python objects: assign the parent of the subtree to a variable or pass it into functions instead of starting at the root. This allows accessing its descendents more directly.
- Try assigning data values directly to attributes instead of passing them through DataElement.
- If you use custom data types that are costly to parse, try running `objectify.annotate()` over read-only trees to speed up the attribute type inference on read access.

Note that none of these measures is guaranteed to speed up your application. As usual, you should prefer readable code over premature optimisations and profile your expected use cases before bothering to apply optimisations at random.