<<if: ZXIDBOOK>> <<else: >>ZXID Low Level ("Raw") API ########################## <<author: Sampo Kellomäki (sampo@iki.fi)>> <<cvsid: $Id: zxid-raw.pd,v 1.5 2010-01-08 02:10:09 sampo Exp $>> <<class: article!a4paper!!ZXID-RAW 01>> <<define: ZXDOC=ZXID Raw API>> <<abstract: ZXID.org Identity Management toolkit implements standalone SAML 2.0 and Liberty ID-WSF 2.0 stacks. This document describes the low level API. >> <<maketoc: 1>> 1 Introduction ============== Here we describe the general philosophy of the ZXID low level APIs. Some function level documentation is available from <<link:../ref/html/index.html: Function reference>>. Before you barge head first to use the raw API, you should check if the +easy+ and simple API in <<link:zxid-simple.html: zxid_simple()>> meets your needs. Or you may be able to use <<link:../mod_auth_saml/mod_auth_saml.html: mod_auth_saml>> and not have to program at all. Happy hacking! 1.1 Other documents ------------------- <> <<htmlpreamble:

# ZXID Low Level ("Raw") API

>> 12 Full Native C API ==================== <<fi: >> The generated aspects of the native C API are in c/*-data.h, for example c/zx-sa-data.h Studying this file is very instructive.<<footnote: emacs tip: run `make tags' and then try hitting M-. while cursor is over a struct or function name in c/zx-sa-data.h - this makes navigation painless.>> 12.1 C Data Structures ---------------------- From .sg a header (NN-data.h) is generated. This header contains structs that represent the data of the elements. Each element and attribute generates its own node. Even trivial nodes like strings have to be kept this way because the nodes form basis of remembering the ordering of data. This ordering is needed for exclusive XML canonicalization, and thus for signature verification.<<footnote: It's unfortunate that the XML standards do not make this any easier. Without order maintenance requirement, it would be possible to represent trivial child elements directly as struct fields. An approach that tried to do just this is available from CVS tag GEN_LALR (ca. 29.5.2006).>> Any missing data is represented by NULL pointer. Any repeating data is kept as a linked list, in reverse order of being seen in the data stream.<<footnote: Reverse order is just an optimization - or an artifact of simply adding latest element to the head of the list. If this bothers you, it's easy enough to reverse the list afterwards. Linked list is simple and works well for data whose order does not matter much (we use separate pointer for remembering the canonicalization order) and where random access is not needed, or cardinality is low enough so that simple pointer chasing is efficient enough.>> <<ignore: *** Problem here: how to preserve ordering of elements. We need to * do SO canonicalization as there are new elements, yet we would like * to maintain WO as much as possible, especially for elements for which * we do not have schema ("any" elements). Always reverse any elem list? >> Simple elements and all attributes are represented by simple string node (even if they are booleans or integers). *Example* Consider following XML lNIzVMrp8CwTE= GeMp7LS...vnjn8= Decoding would produce the data structure in Fig-<<see: fig:decode-data>>. You should also look at c/zx-sa-data.h to see the structs involved in this example. <<dot: decode-data: Typical data structure produced by decode. // This graph crashes dot 1.12, but works in dot 2.8, seems to crash 2.20.2 size="11.0,6.0" margin=0 rankdir=LR { rank=same; siginfo; sigval; } { rank=same; canonmeth; sigmeth; ref; } //{ rank=same; canonmeth; sigmeth; ref; digmeth; digval; } //{ rank=same; xforms; xform_env; xform_c14n; } //{ rank=same; xform_env; xform_c14n; digmeth; digval; } { rank=same; xforms; digmeth; digval; } { rank=same; xform_c14n; xform_env; } sig [shape=record,label="zx_ds_Signature_s|{|{<f_kids>gg.kids|<f_siginfo>SignedInfo|<f_sigval>SignatureValue|KeyInfo (0)|Object (0)|Id (0)}}"]; siginfo [shape=record,label="zx_ds_SignedInfo_s|{|{<f_kids>gg.kids|<f_wo>gg.g.wo|<f_canonmeth>CanonicalizationMethod|<f_sigmeth>SignatureMethod|<f_ref>Reference|Id (0)}}"]; canonmeth [shape=record,label="zx_ds_CanonicalizationMethod_s|{|{<f_wo>gg.g.wo|Algorithm\n\"http://w3.org/xml-exc-c14n#\"}}"]; sigmeth [shape=record,label="zx_ds_SignatureMethod_s|{|{<f_wo>gg.g.wo|Algorithm\n\"http://w3.org/xmldsig#rsa-sha1\"}}"]; ref [shape=record,label="zx_ds_Reference_s|{|{<f_kids>gg.kids|gg.g.wo (0)|<f_xforms>Transforms|<f_digmeth>DigestMethod|<f_digval>DigestValue|Id (0)|Type (0)|URI\n\"#RrcrNwFIw6n\"}}"]; xforms [shape=record,label="zx_ds_Transforms_s|{|{<f_kids>gg.kids|<f_wo>gg.g.wo|gg.g.n (0)|<f_xform>Transform}}"]; xform_c14n [shape=record,label="zx_ds_Transform_s|{|{<f_wo>gg.g.wo|gg.g.n (0)|XPath (0)|<f_c14n_algo>Algorithm\n\"http://w3.org/xml-exc-c14n#\"}}"]; xform_env [shape=record,label="zx_ds_Transform_s|{|{gg.g.wo (0)|<f_n>gg.g.n|XPath (0)|Algorithm\n\"http://w3.org/xmldsig#env-sig\"}}"]; xforms:f_xform -> xform_env xform_env:f_n -> xform_c14n digmeth [shape=record,label="zx_ds_DigestMethod_s|{|{<f_wo>gg.g.wo|Algorithm\n\"http://w3.org/xmldsig#sha1\"}}"]; digval [shape=record,label="zx_elem_s|{|{gg.g.wo (0)|content\n\"lNIzVMrp8CwTE=\"}}"]; sigval [shape=record,label="zx_ds_SignatureValue_s|{|{gg.g.wo (0)|gg.content\n\"GeMp7LS...vnjn8=\"|Id (0)}}"]; sig:f_siginfo -> siginfo sig:f_sigval -> sigval siginfo:f_canonmeth -> canonmeth siginfo:f_sigmeth -> sigmeth siginfo:f_ref -> ref ref:f_xforms -> xforms ref:f_digmeth -> digmeth ref:f_digval -> digval sig:f_kids ->siginfo [weight=0,arrowhead=empty,color=red] siginfo:f_wo ->sigval [weight=0,arrowhead=empty,color=red] siginfo:f_kids -> canonmeth [weight=0,arrowhead=empty,color=red] canonmeth:f_wo -> sigmeth [weight=0,arrowhead=empty,color=red] sigmeth:f_wo -> ref [weight=0,arrowhead=empty,color=red] ref:f_kids -> xforms [weight=0,arrowhead=empty,color=red] xforms:f_wo -> digmeth [weight=0,arrowhead=empty,color=red] digmeth:f_wo -> digval [weight=0,arrowhead=empty,color=red] xforms:f_kids -> xform_c14n [weight=0,arrowhead=empty,color=red] xform_c14n:f_wo -> xform_env [weight=0,arrowhead=empty,color=red] >> There are two pointer systems at play here. The black solid arrows depict the logical structure of the XML document. For each child element there is a struct field that simply points to the child. If there are multiple occurrences of the child, as in ~sig->SignedInfo->Reference->Transforms->Transform~, the children are kept in a linked list connected by gg.g.n (next) fields.<<footnote: This linked list may be in inverted order depending on the phase of the moon and position of the trams in Helsinki. Until implementation matures, its better not to depend on the ordering.>> The +wire order+ structure, depicted by red hollow arrows, is maintained using gg.kids and gg.g.wo fields. For example ~sig->SignedInfo->Reference->Transforms~ keeps its kids, the ~zx_ds_Transform~ objects, in the original order hanging from the kids and linked with the ~wo~ field. As can be seen, the order kept with ~wo~ fields can be different than the one kept using <<tt: n>> (next) fields. What's more, the kids list can contain dissimilar objects, witness ~sig->SignedInfo->Reference->gg.kids~. The wire order representation is only captured when decoding the document and is mainly useful for correctly canonicalizing the document for signature verification. If you are building a data structure in your own program, you typically will not set the gg.kids and gg.g.wo fields. In the diagram, the objects of type ~zx_str~ were collapsed to double quoted strings. Superfluous gg.kids, gg.g.wo, and gg.g.n fields were omitted: they exist in all structures, but are not shown when they are ~NULL~. The ~NULL~ is depicted as zero (0).<<footnote: All this gg.g business is just C's way of referencing the fields of a common base type of element objects.>> <<notacountry: so wo>> 12.1.1 Handling XML Namespaces ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ An annoying feature of XML documents is that they have variable namespace prefixes. The namespace prefix for the unqualified elements is taken to be the one specified in target() directive of the .sg input. Name of an element in C code is formed by prefixing the element by the namespace prefix and an underscore. Attributes will only have namespace prefix if such was expressly specified in .sg input. When decoding, the actual namespace prefixes are recorded. The wire order encoder knows to use these recorded prefixes so that accurate canonicalization for XMLDSIG can be produced. If the message on wire uses wrong namespaces, the wrong ones are remembered so that canonicalization for signature validation will work irrespective. The ability to accept wrong namespaces only works as long as there is no ambiguity as to which tag was meant - there are some tags that need namespace information to distinguish. If you hit one of these then either you get lucky and the one that is arbitrarily picked by the decoder happens to be the correct one, or you are stuck with no easy way to make it right. Of course the XML document was wrong to start with so theoretically this is not a concern. Generally the more schemata that are simultaneously generated to one package, the greater the risk of collisions between tags. The schema order encoder always uses the prefixes defined using target() directives in .sg files. The runtime notion of namespaces is handled by ~ns_tab~ field of the decoding and encoding context. It is initialized to contain all namespaces known by virtue of .sg declarations. The runtime assigned prefixes are held in a linked list hanging from <<tt: n>> (next) field of ~struct zx_ns_s~. (*** more work needed here) The code generation creates a file, such as c/zx-ns.c, which contains initialization for the table. The main program should point the ~ns_tab~ field of context as follows: main { struct zx_ctx* ctx; ... ctx->ns_tab = zx_ns_tab; /* Here zx_ is the prefix chosen in code generation */ } Consider the following evil contortion Assuming the ~ns_tab~ assigns prefix <<tt: y>> to the namespace URI, we would have following data structure as a result of a decode <<dot: ns-data,,: Decode of XML and resulting namespace structures. margin=0 //rankdir=LR { rank=same; ns_tab; e; h; b; } { rank=same; H; B; } { rank=same; C; D; } ns_tab [shape=record,label="{ns_tab|{y|uri|<uri_n>}|{z|iru|<iru_n>}}"] e [shape=record,label="e|uri|<n>"] h [shape=record,label="h|uri|<n>"] b [shape=record,label="b|uri|0"] i [shape=record,label="e|iru|0"] ns_tab:uri_n -> e ns_tab:iru_n -> i e:n -> h h:n -> b E -> H [style=bold] E -> B [style=bold] B -> C [style=bold] B -> D [style=bold] D -> F [style=bold] E -> e [color=red,arrowhead=empty] H -> h [color=red,arrowhead=empty] B -> b [color=red,arrowhead=empty] C -> e [color=red,arrowhead=empty] D -> i [color=red,arrowhead=empty] F -> e [color=red,arrowhead=empty] >> The red hollow arrows indicate how the elements reference the namespaces. Since none of the elements used the prefix originally specified in the schema grammar target() directive, we ended up allocating "alias" nodes for the uri. However, since E and C use the same prefix, they share the alias node. Things get interesting with D: it redefines the prefix e to mean different namespace URI, "iru", which happens to be an alias of prefix z. Later, when wire order canonical encode is done, the red thin arrows are chased to determine the namespaces. However, we need to keep a separate "seen" stack to track whether parent has already declared the prefix and URI. E would declare xmlns:e="uri", but C would not because it had already been "seen". However, F would have to declare it again because the xmlns:e="iru" in D masks the declaration. The ~zx_ctx~ structure is used to track the namespaces and "seen" status through out decoders and encoders. <<dot: seen-data,,: Seen data structure (blue dotted and green dashed arrows) in the end of decoding F. S=seen, SN=seen_n. margin=0 //rankdir=LR { rank=same; ns_tab; ee; e; h; b; } { rank=same; H; B; } { rank=same; C; D; } ns_tab [shape=record,label="{ns_tab|{P|URI|S|SN|N}|{y|uri|0|0|<uri_n>}|{z|iru|0|0|<iru_n>}}"] e [shape=record,label="e|uri|0|0|<n>"] ee [shape=record,label="e|uri||0|<n>"] h [shape=record,label="h|uri|0|<sn>|<n>"] b [shape=record,label="b|uri|0|<sn>|0"] i [shape=record,label="e|iru||0|0"] ctx [shape=record,label="{ctx|{|{<ns>ns_tab|<sn>seen_n}}}"] ns_tab:uri_n -> ee ns_tab:iru_n -> i ee:n -> e e:n -> h h:n -> b E -> H [style=bold] E -> B [style=bold] B -> C [style=bold] B -> D [style=bold] D -> F [style=bold] E -> e [color=red,arrowhead=empty] H -> h [color=red,arrowhead=empty] B -> b [color=red,arrowhead=empty] C -> e [color=red,arrowhead=empty] D -> i [color=red,arrowhead=empty] F -> ee [color=red,arrowhead=empty] ns_tab -> ctx:ns [arrowhead=none,arrowtail=normal] b -> ctx:sn [color=blue,style=dotted,arrowhead=none,arrowtail=normal] b:sn -> h [color=blue,style=dotted] h:sn -> ee [color=blue,style=dotted] ee:s -> i [color=green,style=dashed] i:s -> e [color=green,style=dashed] >> Here we can see how the ~seen_n~ list, represented by the blue dotted arrows, was built: at the head of the list, ~ctx->seen_n~, is the last seen prefix, namely b (because, although the meaning of e at F was different, e as a prefix had already been seen earlier at E), followed by other prefixes in inverse order of first occurrence.<<footnote: This is a mere artifact of implementation: it's cheapest to add to the head of the list. This may change in future.>> The green dashed arrows from e:uri to e:iru and then on to second e:uri reflect the fact that e:uri (second) was put to the list first (when we were at E), but later, at D, a different meaning, iru, was given to prefix e. Finally at F we give again a different meaning for e, thus pushing to the "seen stack" another node. Although e at E and at F have namespace URI, "uri", we are not able to use the same node because we need to keep the stack order. Thus we are forced to allocate two identical nodes. 12.1.2 Handling any and anyAttribute ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since our aim is to be lax in what we accept, every element can handle unexpected additional attributes as well as unexpected elements. Thus whether the schema specifies any or anyAttribute or not, we handle everything as if they were there. However, when attributes and elements are received outside of their expected context, they are simply treated as strings with string names. This is true even for those attributes and elements that would be recognizable in their proper context. The any extension points, as well as some bookkeeping data are hidden inside ~ZX_ELEM_EXT~ macro. If you tinker with this macro, be sure you know what you are doing. If you want to add your own specific fields to all structs, redefining ~ZX_ELEM_EXT~ may be appropriate, but if you want to add more fields only to some specific structures, you can define a macro of form TPF_EEE_EXT and put in it whatever fields you want. These fields will be initialized to zero when the structure is created, but are not touched in any other way by the generated code. In particular, if some of your fields are pointers, it will be your responsibility to free them. The standard free functions will not understand to free them. See the data structure walking functions, below for one way to accomplish this. 12.1.3 Root data structure ~~~~~~~~~~~~~~~~~~~~~~~~~~ The root data structure struct zx_root_s; is a special structure that has a field for every top level recognizable element. 12.1.4 Per element data structures ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *** TBW 12.1.5 Memory Allocation ~~~~~~~~~~~~~~~~~~~~~~~~ After decoding all string data points directly into the input buffer, i.e. strings are NOT copied. Be sure to not free the input buffer until you are done processing the data structure. If you need to take a copy of the strings, you will need to walk the data structure as a post processing step and do your copies. This can be done using void TPF_dup_strs_len_NS_EEE(struct zx_dec_ctx* c, struct TPF_NS_EEE_s* x); The structures are allocated via ZX_ZALLOC() macro, which by default calls zx_zalloc() function, which in turn uses system malloc(3). However, you can redefine the macro to use whatever other allocation scheme you desire. The generated libraries never free(3) memory. In many programming patterns, this is actually desirable: for example a CGI program can count on dying - the process exit(2) will free all the memory. If you need to free(3) the data structure, you will need to walk it using void TPF_free_len_NS_EEE(struct zx_dec_ctx* c, struct TPF_NS_EEE_s* x, int free_strings); void zx_free_any(struct zx_dec_ctx* c, struct zx_note_s* n, int free_strs); The zx_free_any() works by having a gigantic switch statement that calls the appropriate specific free function. You can deep clone the data structure with void TPF_deep_clone_NS_EEE(struct zx_dec_ctx* c, struct TPF_NS_EEE_s* x, int dup_strings); struct zx_note_s* zx_clone_any(struct zx_dec_ctx* c, struct zx_note_s* n, int dup_strs); The zx_clone_any() works by having a gigantic switch statement that calls the appropriate specific free function. 12.2 Decoder as Recursive Descent Parser ---------------------------------------- The entry point to the decoder is struct zx_root_s* zx_DEC_root(struct zx_dec_ctx* c, struct zx_ns_s* dummy, int n_decode); The decoding context holds pointer to the raw data and must be initialized prior to calling the decoder. The third argument specifies how many recognized elements are decoded before returning. Usually you would specify 1 to consume one top level element from the stream.<<footnote: The second argument, the dummy namespace, is meaningless for root node, but makes sense for element decoders. For root you can simply supply 0 (NULL).>> The returned data structure, ~struct zx_root_s~, contains one pointer for each type of top level element that can be recognized. The ~tok~ field of the returned value identifies the last top level element recognized and can be used to dispatch to correct request handler: zx_prepare_dec_ctx(c, TPF_ns_tab, start_ptr, end_ptr); struct TPF_root_s* x = TPF_DEC_root(c, 0, 1); switch (x->gg.g.tok) { case TPF_NS_EEE_ELEM: return process_EEE_req(x->NN_EEE); } When processing responses, it is generally already known which type of response you are expecting, so you can simply check for NULLness of the respective pointer in the returned data structure. Internally zx_DEC_root() works much the same way: it scans a beginning of an element from the stream, looks up the token number corresponding to the element name, and switches on that, calling element specific decoder functions (see next section) to do the detailed processing. In the above code fragment, you should note the call to zx_prepare_dec_ctx() which initializes the decoder machinery. It takes +ns_tab+ argument, which specifies which namespaces will be recognized. This table MUST match the TPF_DEC_root() function you call (i.e. both must have been generated as part of the same xsd2sg.pl invocation). The other arguments are the start of the buffer to decode and pointer one past the end of the buffer to decode. 12.2.1 Element Decoders ~~~~~~~~~~~~~~~~~~~~~~~ For each recognizable element there is a function of form struct TPF_NS_EEE_s* zx_DEC_NS_EEE(struct zx_dec_ctx* c); where TPF is the prefix, NS is the namespace prefix, and EEE is the element name. For example: struct zx_se_Envelope_s* zx_DEC_se_Envelope(struct zx_ctx* c); These functions work much the same way as the root decoder. You should consult dec-templ.c for the skeleton of the decoder. Generally you should not be calling element specific decoders: they exist so that zx_DEC_root() can call them. They have somewhat nonintuitive requirements, for example the opening <, the namespace prefix, and the element name must have already been scanned from the input stream by the time you call element specific decoder. 12.2.2 Decoder Extension Points ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The generated code is instrumented with following macros ZX_ATTR_DEC_EXT(ss):: Extension point called just after decoding known attribute ZX_XMLNS_DEC_EXT(ss):: Extension point called just after decoding xmlns attribute ZX_UNKNOWN_ATTR_DEC_EXT(ss):: Extension point called just after decoding unknown attr ZX_START_DEC_EXT(x):: Extension point called just after decoding element name and allocating struct, but before decoding any of the attributes. ZX_END_DEC_EXT(x):: Extension point called just after decoding the entire element. ZX_START_BODY_DEC_EXT(x):: Extension point called just after decoding element tag, including attributes, but before decoding the body of the element. ZX_PI_DEC_EXT(pi):: Extension point called just after decoding processing instruction ZX_COMMENT_DEC_EXT(comment):: Extension point called just after decoding comment ZX_CONTENT_DEC(ss):: Extension point called just after decoding string content ZX_UNKNOWN_ELEM_DEC_EXT(elem):: Extension point called just after decoding unknown element Following macros are available to the extension points TPF:: Type prefix (as specified by -p during code generation) EL_NAME:: Namespaceful element name (NS_EEE) EL_STRUCT:: Name of the struct that describes the element EL_NS:: Namespace prefix of the element (as seen in input schema) EL_TAG:: Name of the element without any namespace qualification. 12.3 Exclusive Canonical Encoder (Serializer) --------------------------------------------- The encoder receives a C data structure and generates a gigantic string containing an XML document corresponding to the data structure and the input schemata. The XML document conforms to the rules of exclusive XML canonicalization and hence is useful as input to XMLDSIG. One encoder is generated for each root node specified at the code generation. Often these encoders share code for interior nodes. The encoders allow two pass rendering. You can first use the length computation method to calculate the amount of storage needed and then call one of the rendering functions to actually render. Or if you simply have large enough buffer, you can just render directly. The encoders take as argument next free position in buffer and return a char pointer one past the last byte used. Thus you can discover the length after rendering by subtracting the pointers. This is guaranteed to result same length as returned by the length computation method.<<footnote: This is a useful sanity check. If the two ever disagree, please report a bug.>> You can also call the next encoder with the return value of the previous encoder to render back-to-back elements. The XML namespace and XML attribute handling of the encoders is novel in that the specified sort is done already at code generation time, i.e. the renderers are already in the order that the sort mandates. For attributes we know the sort order directly from the schema because [XML-C14N], sec 2.2, p.7, specifies that they sort first by namespace URI and then by name, both of which we know from the schema. For ~xmlns~ specifications the situation is similarly easy in the schema order encoder case because we know the namespace prefixes already at code generation time. However, for the wire order encoder we actually need a runtime sort because we can not control which namespace prefixes get used. However, for both cases we can make a pretty good guess about which namespaces might need to be declared at any given element: the element's own namespace and namespaces of each of its attributes. That's all, and it's all known at code generation time. At runtime we only need to check if the namespace has already been seen at outer layer. 12.3.1 Length computation ~~~~~~~~~~~~~~~~~~~~~~~~~ Compute length of an element (and its subelements). The XML attributes and elements are processed in schema order. int TPF_LEN_SO_NS_EEE(struct zx_ctx* c, struct TPF_NS_EEE_s* x); For example: int zx_LEN_SO_se_Envelope(struct zx_ctx* c, struct zx_se_Envelope_s* x); Compute length of an element (and its subelements). The XML namespaces and elements are processed in wire order. int TPF_LEN_WO_NS_EEE(struct zx_ctx* c, struct TPF_NS_EEE_s* x); For example: int zx_LEN_WO_se_Envelope(struct zx_ctx* c, struct zx_se_Envelope_s* x); 12.3.2 Encoding in schema order ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Render an element into string. The XML elements are processed in schema order. The xmlns declarations and XML attributes are always sorted per [XML-EXC-C14N] rules.<<footnote: The sort is actually done already at code generation time by xsd2sg.pl.>> This is what you generally want for rendering new data structure to a string. The wo pointers are not used. char* TPF_ENC_SO_NS_EEE(struct zx_ctx* c, struct TPF_NS_EEE_s* x, char* p); For example: char* zx_ENC_SO_se_Envelope(struct zx_ctx* c, struct zx_se_Envelope_s* x, char* p); Since it is a very common requirement to allocate correct sized buffer and then render an element, a helper function is provided to do this in one step. struct zx_str* zx_EASY_ENC_SO_se_Envelope(struct zx_ctx* c, struct zx_se_Envelope_s* x); The returned string is allocated from allocation arena described by ~zx_ctx~. 12.3.3 Encoding in wire order ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Render element into string. The XML elements are processed in wire order by chasing wo pointers. This is what you want for validating signatures on other people's XML documents. If the wire representation was schema invalid, e.g. elements were in wrong order, the wire representation is still respected, except for xmlns declarations and XML attributes, which are always sorted, per exc-c14n rules. For each element a function is generated as follows char* TPF_ENC_WO_NS_EEE(struct zx_ctx* c, struct TPF_NS_EEE_s* x, char* p); For example char* zx_ENC_WO_se_Envelope(struct zx_ctx* c, struct zx_se_Envelope_s* x, char* p); A helper function is also available struct zx_str* zx_EASY_ENC_WO_se_Envelope(struct zx_ctx* c, struct zx_se_Envelope_s* x); 12.4 Signatures (XMLDSIG) ------------------------- 12.4.1 Signature Generation ~~~~~~~~~~~~~~~~~~~~~~~~~~~ *** TBW 12.4.2 Signature Validation ~~~~~~~~~~~~~~~~~~~~~~~~~~~ For signature validation you need to walk the decoded data structure to locate the signature as well as the references and pass them to zxsig_validate(). The validation involves wire order exclusive canonical encoding of the referenced XML blobs, computation of SHA1 or MD5 checksums over them, and finally computation of SHA1 check sum over the <SignedInfo> element and validation of the actual <SignatureValue> against that. The validation involves public key decryption using the signer's certificate. A nasty problem in exclusive canonicalization is that the namespaces that are needed in the blob may actually appear in the containing XML structures, thus in order to know the correct meaning of a namespace prefix, we need to perform the +seen+ computation for all elements outside and above the blob of interest.<<footnote: This is yet another indication of how botched the XML namespace concept is. Or this could have been fixed in the exclusive canonicalization spec by not using namespace prefixes at all.>> To verify signature, you have to do certain amount of preparatory work to locate the signature and the data that was signed. Generally what should be signed will be evident from protocol specifications or from the security requirements of your application environment. Conversely, if there is a signature, but it does not reference the appropriate elements, its worthless and you might as well reject the document without even verifying the signature. *Example* struct zxsig_ref refs[1]; cf = zxid_new_conf("/var/zxid/"); ent = zxid_get_ent_from_file(cf, "YV7HPtu3bfqW3I4W_DZr-_DKMP4."); refs[0].ref = r->Envelope->Body->ArtifactResolve ->Signature->SignedInfo->Reference; refs[0].blob = (struct zx_elem_s*)r->Envelope->Body->ArtifactResolve; res = zxsig_validate(cf->ctx, ent->sign_cert, r->Envelope->Body->ArtifactResolve->Signature, 1, refs); if (res == ZXSIG_OK) { D("sig vfy ok %d", res); } else { ERR("sig vfy failed due to(%d)", res); } This code illustrates 1. You have to determine who signed and provide the entity object that corresponds to the signer. Often you would determine the entity from <Issuer> element somewhere inside the message. The entity is used for retrieving the signing certificate. Another alternative is that the signature itself contains a <KeyInfo> element and you extract the certificate from there. You would still need to have a way to know if you trust the certificate. 2. You have to prepare the refs array. It contains pairs of <SignedInfo><Reference> specifications combined with the actual elements that are signed. Generally the URI XML attribute of the <Reference> element points to the data that was signed. However, it is application dependent what type of ID XML attribute the URI actually references or the URI could even reference something outside the document. It would be way too unreliable for the zxsig_validate() to attempt guessing how to locate the signed data: therefore we push the responsibility to you. Your code will have to walk the data to locate all referenced bits and pieces. In the above example, locating the one signed bit was very easy: the specification says where it is (and this location is fixed so there really is no need to check the URI either). You pass the length of the refs array and the array itself as two last arguments to zxsig_validate(). 3. You need to locate the <Signature> element in the document and pass it as argument to zxsig_validate(). Usually a protocol specification will say where the <Signature> element is to be found, so locating it is not difficult. 4. The return value will indicate validation status. ZXSIG_OK, which has numerical value of 0, indicates success. Other nonzero values indicate various kinds of failure. 12.4.3 Certificate Validation and Trust Model ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trust models for TLS and signature validation are separate. TLS layer is handled mainly by libcurl or in case of ClientTLS, by the https web server (which is not part of zxid). In signature validation the primary trust mechanism is that entity's metadata specifies the signing certificate and there is no Certification Authority check at all.<<footnote: If you develop CA check, please submit patches to ZXID project.>> This model works well if you control the admission to your CoT. However, ZXID ships by default with the automatic CoT feature turned on, thus anyone can get added to the CoT and therefore signature with any certificate they declare is "valid". This hardly is acceptable for anything involving money. 12.5 Data Accessor Functions ---------------------------- Simple read access to data should, in C, be done by simply referencing the fields of the struct, e.g. if (!r->EntitiesDescriptor->EntityDescriptor) goto bad_md; *** TBW 12.6 Memory Allocation and Free ------------------------------- *** TBW 12.7 Walking the data structure ------------------------------- *** TBW 12.9 Thread Safety ------------------ All generated libraries are designed to be thread safe, provided that the underlying libc APIs, such as malloc(3) are thread safe. 15 Creating New Interfaces Using ZXID Methodology ================================================= The ZXID code generation methodology can be used to create interfaces to any XML document or protocol that can be described as a Schema Grammar (which includes any document that can be expressed as XML Schema - XSD). The general steps are 1. Convert .xsd file to .sg, or write the .sg directly. For conversion, you would typically use a command like ~/pd/xsd2sg.pl <foo.xsd >foo.sg 2. Tweak and rationalize the resulting .sg file. In ideal world any construct expressible as .xsd should be nicely representable, but in practise some work better than others, thus you can create a much nicer interface if you invest in some manual tweaking. Note that the tweaked .sg still is able to represent the same document as the original .xsd described, though often the tweaking causes some relaxation. Most common tweaks a. If the .xsd is written so that the targeted namespace is also the default namespace, you should introduce a namespace prefix because this is needed during code generation to keep different C identifiers from clashing with each other. Ideally you should coordinate the namespace prefixes globally so that even two different projects will not clash. b. Where the choice construct is used, indicated by pipey symbol (|) in the .sg file, you should refactor these into sequences of zero-or-one occurrence (?) instances of the alternatives of the choice. This is needed because for the foreseeable future xsd2sg.pl has a limitation in code generation feature. If the choice has maxOccurs="unbounded" you should use (*) instead. c. xml:lang and other similar attributes may need to be factored open to be just of type %xs:string. This is a bug in xsd2sg.pl 3. "Connect" the schema to bigger framework. Usually this means adding your schema grammar to the ZX_SG variable in zxid/Makefile and supplying additional -r flags in ZX_ROOT variable. This allows your new schema to be visible at top level. If your schema is meant to extend leafs or interior nodes of the parse tree, such as SOAP Body, you would edit the SOAP schema to accept your new protocol elements in the Body. Or that the generic SOAP header can accept your specific header schemata, or that the SAML attribute definitions accept your kind of attributes - whatever makes sense in your context. Alternative to this is to create an entirely new monolithic encoder decoder, i.e. instead of extending the existing ZXID project to accommodate your new protocol, you just start a new project that uses the same methodology. You should see how the SAML protocol part is separated from the SAML metadata parsing and from the WSF parsing in the existing project. 17 Code Generation Tools ======================== Main work horse of code generation is xsd2sg.pl, which serves multiple purposes 1. Build hashes of all declarations in .sg input. Each hash element consists of array of elements and attributes, as well as groups and attribute groups. The type of array element sis determined from prefix, per .sg rules. 2. Expand groups and attribute groups 3. Evaluate each element wrt its type and generate a. C data structures b. Decoder grammar c. Token descriptions for perfect hash and lexical analyzer d. Encoder C code The code to build hashes is interwoven in the code that generates .xsd from .sg. The rest of the generation happens in a function called generate(). Typical command line (to generate SAML 2.0 protocol engine) ~/plaindoc/xsd2sg.pl -d -gen saml2 -p zx_ \ -r saml:Assertion -r se:Envelope \ -S \ sg/saml-schema-assertion-2.0.sg \ sg/saml-schema-protocol-2.0.sg \ sg/xmldsig-core.sg \ sg/xenc-schema.sg \ sg/soap11.sg \ >/dev/null <<ignore: ~/plaindoc/xsd2sg.pl -d -gen saml2 -p zx_ -r saml:Assertion -r se:Envelope -S sg/saml-schema-assertion-2.0.sg sg/saml-schema-protocol-2.0.sg sg/xmldsig-core.sg sg/xenc-schema.sg sg/soap11.sg >/dev/null >> To generate SAML 2.0 Metadata engine you would issue ~/plaindoc/xsd2sg.pl -d -gen saml2md -p zx_ \ -r md:EntityDescriptor -r md:EntitiesDescriptor \ -S \ sg/saml-schema-assertion-2.0.sg \ sg/saml-schema-metadata-2.0.sg \ sg/xmldsig-core.sg \ sg/xenc-schema.sg \ >/dev/null <<ignore: ~/plaindoc/xsd2sg.pl -d -gen saml2md -p zx_ -r md:EntityDescriptor -r md:EntitiesDescriptor -S sg/saml-schema-assertion-2.0.sg sg/saml-schema-metadata-2.0.sg sg/xmldsig-core.sg sg/xenc-schema.sg >/dev/null >> 17.1 Special Support for Specific Programming Languages ------------------------------------------------------- While C code generation is the main output, and this can always be converted to other languages using SWIG, sometimes a more natural language interface can be built by directly generating it. We plan to enhance the code generation to do something like this. At least direct hash-of-hashes-of-arrays-of-hashes type data-structure generation for benefit of some scripting languages is planned. <<if: ZXIDBOOK>> <<else: >> 18 ZXID SP ========== *** warning: not checked lately, may be wrong! <<table: ZXID SP URLs URL Description ============ ======================================================= /zxid Same as o=M. Main convenience entry point /zxid?o=M SSO with CDC; or management if already logged in /zxid?o=C Common Domain Cookie (CDC) reader, usually under common domain host name. /zxid?o=E SSO after CDC read; or management if already logged in. /zxid?o=P HTTP POST end point. Used for forms and last part of POST profile SSO. /zxid?o=Q HTTP binding (POST or redirect) request end point (e.g. SLO, MNI). /zxid?o=S SOAP end point (HTTP POST) /zxid?o=B Get SP metadata (or combined SP and IdP metadata if proxying). >> 96 License ========== Copyright (c) 2006-2009 Symlabs (symlabs@symlabs.com), All Rights Reserved. Author: Sampo Kellomäki (sampo@iki.fi) Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. 96.2 Specification IPR ---------------------- ZXID is based on open SAML and Liberty specifications. The parties that have developed these specifications, including Symlabs, have made Royalty Free (RF) licensing commitment. Please ask OASIS and Liberty Alliance for the specifics of their IPR policies and IPR disclosures. Some protocols, such as WS-Trust and WS-Federation enjoy Microsoft's pledge<<footnote: If you have a reference to where this pledge can be found, please let me know so it can be included here.>> that they will not sue you even if you implement these specifications. You should evaluate yourself whether this is good enough for your situation. <> <> <<notapath: TCP/IP a.k.a xBSD/Unix n/a Perl/mod_perl PHP/mod_php Java/Tomcat>> <<EOF: >> <<fi: >>