====================================
Python extensions for XPath and XSLT
====================================

This document describes how to use Python extension functions in XPath
and XSLT like this:

.. sourcecode:: xml

  <xsl:value-of select="f:myPythonFunction(.//sometag)" />

and extension elements in XSLT as in the following example:

.. sourcecode:: xml

  <xsl:template match="*">
      <my:python-extension>
          <some-content />
      </my:python-extension>
  </xsl:template>


.. contents::
.. 
   1  XPath Extension functions
     1.1  The FunctionNamespace
     1.2  Global prefix assignment
     1.3  The XPath context
     1.4  Evaluators and XSLT
     1.5  Evaluator-local extensions
     1.6  What to return from a function
   2  XSLT extension elements
     2.1  Declaring extension elements
     2.2  Applying XSL templates
     2.3  Working with read-only elements

..
  >>> try: from StringIO import StringIO
  ... except ImportError:
  ...    from io import BytesIO
  ...    def StringIO(s):
  ...        if isinstance(s, str): s = s.encode("UTF-8")
  ...        return BytesIO(s)


XPath Extension functions
=========================

Here is how an extension function looks like.  As the first argument,
it always receives a context object (see below).  The other arguments
are provided by the respective call in the XPath expression, one in
the following examples.  Any number of arguments is allowed:

.. sourcecode:: pycon

  >>> def hello(context, a):
  ...    return "Hello %s" % a
  >>> def ola(context, a):
  ...    return "Ola %s" % a
  >>> def loadsofargs(context, *args):
  ...    return "Got %d arguments." % len(args)


The FunctionNamespace
---------------------

In order to use a function in XPath or XSLT, it needs to have a
(namespaced) name by which it can be called during evaluation.  This
is done using the FunctionNamespace class.  For simplicity, we choose
the empty namespace (None):

.. sourcecode:: pycon

  >>> from lxml import etree
  >>> ns = etree.FunctionNamespace(None)
  >>> ns['hello'] = hello
  >>> ns['countargs'] = loadsofargs

This registers the function `hello` with the name `hello` in the default
namespace (None), and the function `loadsofargs` with the name `countargs`.

Since lxml 4.1, it is preferred to use the ``FunctionNamespace`` as a decorator.
Either pass an explicit function name (``@ns("countargs")``), or just use the
bare decorator to register the function under its own name:

.. sourcecode:: pycon

  >>> @ns
  ... def hello(context, a):
  ...    return "Hello %s" % a

Now we're going to create a document that we can run XPath expressions
against:

.. sourcecode:: pycon

  >>> root = etree.XML('<a><b>Haegar</b></a>')
  >>> doc = etree.ElementTree(root)

Done. Now we can have XPath expressions call our new function:

.. sourcecode:: pycon

  >>> print(root.xpath("hello('Dr. Falken')"))
  Hello Dr. Falken
  >>> print(root.xpath('hello(local-name(*))'))
  Hello b
  >>> print(root.xpath('hello(string(b))'))
  Hello Haegar
  >>> print(root.xpath('countargs(., b, ./*)'))
  Got 3 arguments.

Note how we call both a Python function (``hello()``) and an XPath built-in
function (``string()``) in exactly the same way.  Normally, however, you would
want to separate the two in different namespaces.  The FunctionNamespace class
allows you to do this:

.. sourcecode:: pycon

  >>> ns = etree.FunctionNamespace('http://mydomain.org/myfunctions')
  >>> ns['hello'] = hello

  >>> prefixmap = {'f' : 'http://mydomain.org/myfunctions'}
  >>> print(root.xpath('f:hello(local-name(*))', namespaces=prefixmap))
  Hello b


Global prefix assignment
------------------------

In the last example, you had to specify a prefix for the function namespace.
If you always use the same prefix for a function namespace, you can also
register it with the namespace:

.. sourcecode:: pycon

  >>> ns = etree.FunctionNamespace('http://mydomain.org/myother/functions')
  >>> ns.prefix = 'es'
  >>> ns['hello'] = ola

  >>> print(root.xpath('es:hello(local-name(*))'))
  Ola b

This is a global assignment, so take care not to assign the same prefix to
more than one namespace.  The resulting behaviour in that case is completely
undefined.  It is always a good idea to consistently use the same meaningful
prefix for each namespace throughout your application.

The prefix assignment only works with functions and FunctionNamespace objects,
not with the general Namespace object that registers element classes.  The
reasoning is that elements in lxml do not care about prefixes anyway, so it
would rather complicate things than be of any help.


The XPath context
-----------------

Functions get a context object as first parameter.  In lxml 1.x, this value
was None, but since lxml 2.0 it provides two properties: ``eval_context`` and
``context_node``.  The context node is the Element where the current function
is called:

.. sourcecode:: pycon

  >>> def print_tag(context, nodes):
  ...     print("%s: %s" % (context.context_node.tag, [ n.tag for n in nodes ]))

  >>> ns = etree.FunctionNamespace('http://mydomain.org/printtag')
  >>> ns.prefix = "pt"
  >>> ns["print_tag"] = print_tag

  >>> ignore = root.xpath("//*[pt:print_tag(.//*)]")
  a: ['b']
  b: []

The ``eval_context`` is a dictionary that is local to the evaluation.  It
allows functions to keep state:

.. sourcecode:: pycon

  >>> def print_context(context):
  ...     context.eval_context[context.context_node.tag] = "done"
  ...     print(sorted(context.eval_context.items()))
  >>> ns["print_context"] = print_context

  >>> ignore = root.xpath("//*[pt:print_context()]")
  [('a', 'done')]
  [('a', 'done'), ('b', 'done')]


Evaluators and XSLT
-------------------

Extension functions work for all ways of evaluating XPath expressions and for
XSL transformations:

.. sourcecode:: pycon

  >>> e = etree.XPathEvaluator(doc)
  >>> print(e('es:hello(local-name(/a))'))
  Ola a

  >>> namespaces = {'f' : 'http://mydomain.org/myfunctions'}
  >>> e = etree.XPathEvaluator(doc, namespaces=namespaces)
  >>> print(e('f:hello(local-name(/a))'))
  Hello a

  >>> xslt = etree.XSLT(etree.XML('''
  ...   <stylesheet version="1.0"
  ...          xmlns="http://www.w3.org/1999/XSL/Transform"
  ...          xmlns:es="http://mydomain.org/myother/functions">
  ...     <output method="text" encoding="ASCII"/>
  ...     <template match="/">
  ...       <value-of select="es:hello(string(//b))"/>
  ...     </template>
  ...   </stylesheet>
  ... '''))
  >>> print(xslt(doc))
  Ola Haegar

It is also possible to register namespaces with a single evaluator after its
creation.  While the following example involves no functions, the idea should
still be clear:

.. sourcecode:: pycon
  
  >>> f = StringIO('<a xmlns="http://mydomain.org/myfunctions" />')
  >>> ns_doc = etree.parse(f)
  >>> e = etree.XPathEvaluator(ns_doc)
  >>> e('/a')
  []

This returns nothing, as we did not ask for the right namespace.  When we
register the namespace with the evaluator, however, we can access it via a
prefix:

.. sourcecode:: pycon

  >>> e.register_namespace('foo', 'http://mydomain.org/myfunctions')
  >>> e('/foo:a')[0].tag
  '{http://mydomain.org/myfunctions}a'

Note that this prefix mapping is only known to this evaluator, as opposed to
the global mapping of the FunctionNamespace objects:

.. sourcecode:: pycon

  >>> e2 = etree.XPathEvaluator(ns_doc)
  >>> e2('/foo:a')
  Traceback (most recent call last):
  ...
  lxml.etree.XPathEvalError: Undefined namespace prefix


Evaluator-local extensions
--------------------------

Apart from the global registration of extension functions, there is also a way
of making extensions known to a single Evaluator or XSLT.  All evaluators and
the XSLT object accept a keyword argument ``extensions`` in their constructor.
The value is a dictionary mapping (namespace, name) tuples to functions:

.. sourcecode:: pycon

  >>> extensions = {('local-ns', 'local-hello') : hello}
  >>> namespaces = {'l' : 'local-ns'}

  >>> e = etree.XPathEvaluator(doc, namespaces=namespaces, extensions=extensions)
  >>> print(e('l:local-hello(string(b))'))
  Hello Haegar

For larger numbers of extension functions, you can define classes or modules
and use the ``Extension`` helper:

.. sourcecode:: pycon

  >>> class MyExt:
  ...     def function1(self, _, arg):
  ...         return '1'+arg
  ...     def function2(self, _, arg):
  ...         return '2'+arg
  ...     def function3(self, _, arg):
  ...         return '3'+arg

  >>> ext_module = MyExt()
  >>> functions = ('function1', 'function2')
  >>> extensions = etree.Extension( ext_module, functions, ns='local-ns' )

  >>> e = etree.XPathEvaluator(doc, namespaces=namespaces, extensions=extensions)
  >>> print(e('l:function1(string(b))'))
  1Haegar

The optional second argument to ``Extension`` can either be a
sequence of names to select from the module, a dictionary that
explicitly maps function names to their XPath alter-ego or ``None``
(explicitly passed) to take all available functions under their
original name (if their name does not start with '_').

The additional ``ns`` keyword argument takes a namespace URI or
``None`` (also if left out) for the default namespace.  The following
examples will therefore all do the same thing:

.. sourcecode:: pycon

  >>> functions = ('function1', 'function2', 'function3')
  >>> extensions = etree.Extension( ext_module, functions )
  >>> e = etree.XPathEvaluator(doc, extensions=extensions)
  >>> print(e('function1(function2(function3(string(b))))'))
  123Haegar

  >>> extensions = etree.Extension( ext_module, functions, ns=None )
  >>> e = etree.XPathEvaluator(doc, extensions=extensions)
  >>> print(e('function1(function2(function3(string(b))))'))
  123Haegar

  >>> extensions = etree.Extension(ext_module)
  >>> e = etree.XPathEvaluator(doc, extensions=extensions)
  >>> print(e('function1(function2(function3(string(b))))'))
  123Haegar

  >>> functions = {
  ...     'function1' : 'function1',
  ...     'function2' : 'function2',
  ...     'function3' : 'function3'
  ...     }
  >>> extensions = etree.Extension(ext_module, functions)
  >>> e = etree.XPathEvaluator(doc, extensions=extensions)
  >>> print(e('function1(function2(function3(string(b))))'))
  123Haegar

For convenience, you can also pass a sequence of extensions:

.. sourcecode:: pycon

  >>> extensions1 = etree.Extension(ext_module)
  >>> extensions2 = etree.Extension(ext_module, ns='local-ns')
  >>> e = etree.XPathEvaluator(doc, extensions=[extensions1, extensions2],
  ...                          namespaces=namespaces)
  >>> print(e('function1(l:function2(function3(string(b))))'))
  123Haegar


What to return from a function
------------------------------

.. _`XPath return values`: xpathxslt.html#xpath-return-values

Extension functions can return any data type for which there is an XPath
equivalent (see the documentation on `XPath return values`).  This includes
numbers, boolean values, elements and lists of elements.  Note that integers
will also be returned as floats:

.. sourcecode:: pycon

  >>> def returnsFloat(_):
  ...    return 1.7
  >>> def returnsInteger(_):
  ...    return 1
  >>> def returnsBool(_):
  ...    return True
  >>> def returnFirstNode(_, nodes):
  ...    return nodes[0]

  >>> ns = etree.FunctionNamespace(None)
  >>> ns['float'] = returnsFloat
  >>> ns['int']   = returnsInteger
  >>> ns['bool']  = returnsBool
  >>> ns['first'] = returnFirstNode

  >>> e = etree.XPathEvaluator(doc)
  >>> e("float()")
  1.7
  >>> e("int()")
  1.0
  >>> int( e("int()") )
  1
  >>> e("bool()")
  True
  >>> e("count(first(//b))")
  1.0

As the last example shows, you can pass the results of functions back into
the XPath expression.  Elements and sequences of elements are treated as
XPath node-sets:

.. sourcecode:: pycon

  >>> def returnsNodeSet(_):
  ...     results1 = etree.Element('results1')
  ...     etree.SubElement(results1, 'result').text = "Alpha"
  ...     etree.SubElement(results1, 'result').text = "Beta"
  ...
  ...     results2 = etree.Element('results2')
  ...     etree.SubElement(results2, 'result').text = "Gamma"
  ...     etree.SubElement(results2, 'result').text = "Delta"
  ...
  ...     results3 = etree.SubElement(results2, 'subresult')
  ...     return [results1, results2, results3]

  >>> ns['new-node-set'] = returnsNodeSet

  >>> e = etree.XPathEvaluator(doc)

  >>> r = e("new-node-set()/result")
  >>> print([ t.text for t in r ])
  ['Alpha', 'Beta', 'Gamma', 'Delta']

  >>> r = e("new-node-set()")
  >>> print([ t.tag for t in r ])
  ['results1', 'results2', 'subresult']
  >>> print([ len(t) for t in r ])
  [2, 3, 0]
  >>> r[0][0].text
  'Alpha'

  >>> etree.tostring(r[0])
  b'<results1><result>Alpha</result><result>Beta</result></results1>'

  >>> etree.tostring(r[1])
  b'<results2><result>Gamma</result><result>Delta</result><subresult/></results2>'

  >>> etree.tostring(r[2])
  b'<subresult/>'

The current implementation deep-copies newly created elements in node-sets.
Only the elements and their children are passed on, no outlying parents or
tail texts will be available in the result.  This also means that in the above
example, the `subresult` elements in `results2` and `results3` are no longer
identical within the node-set, they belong to independent trees:

.. sourcecode:: pycon

  >>> print("%s - %s" % (r[1][-1].tag, r[2].tag))
  subresult - subresult
  >>> print(r[1][-1] == r[2])
  False
  >>> print(r[1][-1].getparent().tag)
  results2
  >>> print(r[2].getparent())
  None

This is an implementation detail that you should be aware of, but you should
avoid relying on it in your code.  Note that elements taken from the source
document (the most common case) do not suffer from this restriction.  They
will always be passed unchanged.


XSLT extension elements
=======================

Just like the XPath extension functions described above, lxml supports
custom extension *elements* in XSLT.  This means, you can write XSLT
code like this:

.. sourcecode:: xml

  <xsl:template match="*">
      <my:python-extension>
          <some-content />
      </my:python-extension>
  </xsl:template>

And then you can implement the element in Python like this:

.. sourcecode:: pycon

  >>> class MyExtElement(etree.XSLTExtension):
  ...     def execute(self, context, self_node, input_node, output_parent):
  ...         print("Hello from XSLT!")
  ...         output_parent.text = "I did it!"
  ...         # just copy own content input to output
  ...         output_parent.extend( list(self_node) )

The arguments passed to the ``.execute()`` method  are

context
    The opaque evaluation context.  You need this when calling back
    into the XSLT processor.

self_node
    A read-only Element object that represents the extension element
    in the stylesheet.

input_node
    The current context Element in the input document (also read-only).

output_parent
    The current insertion point in the output document.  You can
    append elements or set the text value (not the tail).  Apart from
    that, the Element is read-only.


Declaring extension elements
----------------------------

In XSLT, extension elements can be used like any other XSLT element,
except that they must be declared as extensions using the standard
XSLT ``extension-element-prefixes`` option:

.. sourcecode:: pycon

  >>> xslt_ext_tree = etree.XML('''
  ... <xsl:stylesheet version="1.0"
  ...     xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  ...     xmlns:my="testns"
  ...     extension-element-prefixes="my">
  ...     <xsl:template match="/">
  ...         <foo><my:ext><child>XYZ</child></my:ext></foo>
  ...     </xsl:template>
  ...     <xsl:template match="child">
  ...         <CHILD>--xyz--</CHILD>
  ...     </xsl:template>
  ... </xsl:stylesheet>''')

To register the extension, add its namespace and name to the extension
mapping of the XSLT object:

.. sourcecode:: pycon

  >>> my_extension = MyExtElement()
  >>> extensions = { ('testns', 'ext') : my_extension }
  >>> transform = etree.XSLT(xslt_ext_tree, extensions = extensions)

Note how we pass an instance here, not the class of the extension.
Now we can run the transformation and see how our extension is
called:

.. sourcecode:: pycon

  >>> root = etree.XML('<dummy/>')
  >>> result = transform(root)
  Hello from XSLT!
  >>> str(result)
  '<?xml version="1.0"?>\n<foo>I did it!<child>XYZ</child></foo>\n'


Applying XSL templates
----------------------

XSLT extensions are a very powerful feature that allows you to
interact directly with the XSLT processor.  You have full read-only
access to the input document and the stylesheet, and you can even call
back into the XSLT processor to process templates.  Here is an example
that passes an Element into the ``.apply_templates()`` method of the
``XSLTExtension`` instance:

.. sourcecode:: pycon

  >>> class MyExtElement(etree.XSLTExtension):
  ...     def execute(self, context, self_node, input_node, output_parent):
  ...         child = self_node[0]
  ...         results = self.apply_templates(context, child)
  ...         output_parent.append(results[0])

  >>> my_extension = MyExtElement()
  >>> extensions = { ('testns', 'ext') : my_extension }
  >>> transform = etree.XSLT(xslt_ext_tree, extensions = extensions)

  >>> root = etree.XML('<dummy/>')
  >>> result = transform(root)
  >>> str(result)
  '<?xml version="1.0"?>\n<foo><CHILD>--xyz--</CHILD></foo>\n'

Here, we applied the templates to a child of the extension element
itself, i.e. to an element inside the stylesheet instead of an element
of the input document.

The return value of ``.apply_templates()`` is always a list.  It may
contain a mix of elements and strings, collected from the XSLT processing
result.  If you want to append these values to the output parent, be aware
that you cannot use the ``.append()`` method to add strings.  In many
cases, you would only be interested in elements anyway, so you can discard
strings (e.g. formatting whitespace) and append the rest.

If you want to include string results in the output, you can either build
an appropriate tree yourself and append that, or you can manually add the
string values to the current output tree, e.g. by concatenating them with
the ``.tail`` of the last element that was appended.

Note that you can also let lxml build the result tree for you by passing
the ``output_parent`` into the ``.apply_templates()`` method.  In this
case, the result will be None and all content found by applying templates
will be appended to the output parent.

If you do not care about string results at all, e.g. because you already
know that they will only contain whitespace, you can pass the option
``elements_only=True`` to the ``.apply_templates()`` method, or pass
``remove_blank_text=True`` to remove only those strings that consist
entirely of whitespace.


Working with read-only elements
-------------------------------

There is one important thing to keep in mind: all Elements that the
``execute()`` method gets to deal with are read-only Elements, so you
cannot modify them.  They also will not easily work in the API.  For
example, you cannot pass them to the ``tostring()`` function or wrap
them in an ``ElementTree``.

What you can do, however, is to deepcopy them to make them normal
Elements, and then modify them using the normal etree API.  So this
will work:

.. sourcecode:: pycon

  >>> from copy import deepcopy
  >>> class MyExtElement(etree.XSLTExtension):
  ...     def execute(self, context, self_node, input_node, output_parent):
  ...         child = deepcopy(self_node[0])
  ...         child.text = "NEW TEXT"
  ...         output_parent.append(child)

  >>> my_extension = MyExtElement()
  >>> extensions = { ('testns', 'ext') : my_extension }
  >>> transform = etree.XSLT(xslt_ext_tree, extensions = extensions)

  >>> root = etree.XML('<dummy/>')
  >>> result = transform(root)
  >>> str(result)
  '<?xml version="1.0"?>\n<foo><child>NEW TEXT</child></foo>\n'