<?xml version="1.0"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
                         "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" [
<!ENTITY mktime "<citerefentry><refentrytitle>mktime</refentrytitle><manvolnum>3</manvolnum></citerefentry>">
<!ENTITY strftime "<citerefentry><refentrytitle>strftime</refentrytitle><manvolnum>3</manvolnum></citerefentry>">
]>

<article lang="en">
  <articleinfo>
    <title>Writing An RFC-Date Parser With Spirit</title>
    <author>
      <firstname>Peter</firstname>
      <surname>Simons</surname>
      <affiliation>
        <address><email>simons@computer.org</email></address>
      </affiliation>
    </author>
  </articleinfo>

  <abstract>
    <title>What this is all about ...</title>

    <para>The Spirit Parser Framework is an incredibly flexible and powerful tool for writing
any kind of text processor, parser, or converter. It is also slightly, uh ... complicated
for the beginner to understand. From my own experience when learning how to program with
Spirit, I remember well that it is fairly easy to get a parser for a given grammar up and
running. But taking the next step --- to actually <emphasis>use</emphasis> the parser's
results --- is difficult.</para>

    <para>This text is intended to server as a tutorial for new Spirit users, who have mastered
the basic functionality and are now struggling to get some some real work done. In order to
achieve this objective, I'll explain step-by-step the development of a parser that will convert
date and time specifications as per <ulink url="http://ietf.org/rfc/rfc0822.txt">RFC822</ulink>
(used in e-mail <literal>Date:</literal> headers, for example) into a <type>tm</type>
structure, which can be manipulated with the &mktime; system library routine.</para>
  </abstract>

  <sect1 id="intro">
    <title>What Makes An RFC822 Date</title>

    <para>The grammar defined by RFC822 is seriously crazy. A fact that everybody who has ever
tried to parse an e-mail address, for instance, will confirm. If I got an Euro for every RFC822
parser in the world, which is incorrect, I could probably settle Germany's state deficit and
still afford a blonde pretty girl-friend with a credit card on my bank account. Unfortunately,
RFC822 is out there and chances are small that it's ever going to be fixed, so we might as well
deal with it.</para>

    <para>According to RFC822, a <quote>date and time specification</quote> is defined by the
following grammar (shown in <quote>augmented BNF</quote>):</para>

    <informalfigure>
      <programlisting>date-time   =  [ day "," ] date time

day         =  "Mon"  | "Tue" |  "Wed"  | "Thu"
            |  "Fri"  | "Sat" |  "Sun"

date        =  1*2DIGIT month 2DIGIT

month       =  "Jan"  |  "Feb" |  "Mar"  |  "Apr"
            |  "May"  |  "Jun" |  "Jul"  |  "Aug"
            |  "Sep"  |  "Oct" |  "Nov"  |  "Dec"

time        =  hour zone

hour        =  2DIGIT ":" 2DIGIT [":" 2DIGIT]

zone        =  "UT"  | "GMT"
            |  "EST" | "EDT"
            |  "CST" | "CDT"
            |  "MST" | "MDT"
            |  "PST" | "PDT"
            |  1ALPHA
            | ( ("+" | "-") 4DIGIT )</programlisting>
    </informalfigure>

    <para>At first sight, this doesn't look so bad. But what makes things complicated is that
the RFC allows for the following things to be added (and ignored by the parser) in front of and
after <emphasis>every</emphasis> token:</para>

    <itemizedlist>
      <listitem>
        <para>White space,</para>
      </listitem>
      <listitem>
        <para>linear white space, and</para>
      </listitem>
      <listitem>
        <para>comments.</para>
      </listitem>
    </itemizedlist>

    <para><quote>White space</quote> is simply defined as one or several <keysym>blank</keysym>
or <keysym>tab</keysym> characters.</para>

<para><quote>Linear white space</quote> is defined as any amount of white space, followed by a
carriage return and newline character (<quote><literal>\r\n</literal></quote>), followed by at
least one white space character. This means, that you can add line breaks anywhere in the date
as long as the first character on the new line is a <keysym>blank</keysym> or a
<keysym>tab</keysym>. So effectively, these three variants of a date are all equivalent:</para>

    <informalexample>
      <programlisting>| 12 Jun 82
|
| 12
|     Jun         82
|
| 12
|  Jun
|  82</programlisting>
    </informalexample>

    <para>At last, anywhere where you can add white space, you can add comments as well. A
comment is basically any text surrounded by brackets --- minus some specials such as control
characters, etc. Thus, the following date is still equivalent to those above:</para>

    <informalexample>
      <programlisting>| 12 Jun 82
|
| 12 (Jun)
|     (March) Jun  (fun, isn't it?)       82</programlisting>
    </informalexample>

    <para>Now let's see how we can use Spirit to parse this.</para>
  </sect1>

  <sect1>
    <title>Our Result Data Structure</title>

    <para>Before starting to write any code, it is always a good idea to create the data
structures in which the data should be stored and manipulated. In our case, this is fairly
obvious, because the the system library provides many useful functions that operate on the
<type>tm</type> structure:</para>

    <informalfigure>
      <programlisting>
struct tm
    {
    int tm_sec;         /* seconds */
    int tm_min;         /* minutes */
    int tm_hour;        /* hours */
    int tm_mday;        /* day of the month */
    int tm_mon;         /* month */
    int tm_year;        /* year */
    int tm_wday;        /* day of the week */
    int tm_yday;        /* day in the year */
    int tm_isdst;       /* daylight saving time */
    };</programlisting>
    </informalfigure>

    <para>Using &mktime;, for instance, this structure can be converted into a
<type>time_t</type>, using &strftime;, it can be formatted into an (almost) arbitrary text
format. So this is a good way to represent a date --- and it is easy to fill out by the parser.
There is only one problem: The <type>tm</type> structure, as defined by the POSIX standard,
does not contain a field for the timezone, the date is in, but this is something that we might
need. Hence, we derive our own on class, which adds that field:</para>

    <informalfigure>
      <programlisting>struct timestamp : public tm
    {
    timestamp() { memset(this, 0, sizeof(*this)); }
    int tzoffset;
    };</programlisting>
    </informalfigure>

    <para>This definition (along with on appropriate <function>operator&lt;&lt;</function>) can
be found in the header file <ulink
url="timestamp.hpp"><filename>timestamp.hpp</filename></ulink>.</para>

    <para>Having that out of the way, we can start with the parser.</para>
  </sect1>

  <sect1>
    <title>Building The Parser</title>

    <para>The most parts of the grammer that are the most obvious to solve are
<literal>month</literal> and <literal>day</literal>, which are easily solved by the <ulink
url="../../../doc/symbols.html"><classname>boost::spirit::symbols</classname></ulink> parser:</para>

    <informalfigure>
      <programlisting>struct month_parser : boost::spirit::symbols&lt;int&gt;
    {
    month_parser()
        {
        add ("jan", 0)("feb", 1)("mar", 2)("apr", 3)
            ("may", 4)("jun", 5)("jul", 6)("aug", 7)
            ("sep", 8)("oct", 9)("nov", 10)("dec", 11);
        }
    };
const month_parser month_p;

struct wday_parser : boost::spirit::symbols&lt;int&gt;
    {
    wday_parser()
        {
        add ("sun", 0)("mon", 1)("tue", 2)("wed", 3)
            ("thu", 4)("fri", 5)("sat", 6);
        }
    };
const wday_parser wday_p;</programlisting>
    </informalfigure>

    <para>The parser instances <varname>month_p</varname> and <varname>wday_p</varname> can be
used like any other Spirit primitive. What's special about them is that they map the string
they matched to the integer value paired to the string. This value is available through the
<function>operator[]</function> of the parser. We'll see below, how this is used. Don't
panic!</para>

    <para>The values that we map to the appropriate string are chosen to conform to the
specification of the <type>tm</type> structure; you can refer to the manual page for further
details.</para>

    <para>The next part of the date grammar that we take care of is the <literal>zone</literal>
token. This one is a bit trickier: The literals are easy to parse with a
<classname>boost::spirit::symbols</classname> parser as well, but the explicit specification of the
time zone offset cannot be solved with that. Unless you fancy having a symbol table with 3600
entries, of course. The best solution -- at least the best solution I could come up with -- is
to split that token into two tokens:</para>

    <informalfigure>
      <programlisting>struct timezone_parser : boost::spirit::symbols&lt;int&gt;
    {
    timezone_parser()
        {
        add ("ut", 0)("gmt", 0)
            ("est", -18000)("edt", -14400)
            ("cst", -21600)("cdt", -18000)
            ("mst", -25200)("mdt", -21600)
            ("pst", -28800)("pdt", -25200)
            ("z", 0)("a", -3600)("m", -43200)("n", +3600)("y", +43200);
        }
    };
const timezone_parser timezone_p;

uint_parser&lt;int, 10, 4, 4&gt;  uint4_p;

zone      = ch_p('+')  &gt;&gt;  uint4_p
          | ch_p('-')  &gt;&gt;  uint4_p
          | lexeme_d
            [
                nocase_d
                [
                    timezone_p
                ]
            ]
</programlisting>
    </informalfigure>

    <para>Alright ... Breathe, Neo, breathe! This is simpler than it looks. The
<varname>timezone_p</varname> parser is straight-forward, no problem. The
<varname>uint4_p</varname> parser is just a variation of the normal number parsing primitives:
It's declared to parse an unsigned number (in radix 10 notation --- <quote>decimal</quote>,
that is) with exactly 4 digits. So what will the <literal>zone</literal> rule do? It looks for
either a plus or minus character followed by a 4 digit number, or it will try to match the
explicit time zone names defined in the symbol parser</para>

    <para>The symbol parser has been wrapped into two directives: <ulink
url="../../../doc/directives.html"><varname>lexeme_d</varname></ulink> will switch Spirit into
character-parsing mode. This is useful in case <function>boost::spirit::parse</function> has been
called with a <quote>skipper</quote>. Assuming, for instance, we had omitted the
<varname>lexeme_d</varname> and had called this rule as follows:</para>

    <informalexample>
      <programlisting>boost::spirit::parse(input, zone, boost::spirit::space_p);</programlisting>
    </informalexample>

    <para>Then the <varname>timezone_p</varname> parser would match the string
<quote>g m t</quote> as well as <quote>gmt</quote>; and this is not what we
want.</para>

    <para>The second directive deployed is more obvious: <ulink
url="../../../doc/directives.html"><varname>nocase_d</varname></ulink> will turn the symbol
parser case-insensitive matching-mode. Thus, the inputs <quote>GmT</quote> and
<quote>gMt</quote> are equivalest.</para>

    <para>One more word concerning the <varname>timezone_p</varname> parser: It may be obvious,
but just to be sure let me point out that the numbers returned by the parser when it matches a
timezone name is the offset of that timezone to <abbrev>UTC</abbrev> in seconds.</para>

    <para>Now we can parse the weekday, the name of the month, and the timezone ...
What's left is the actual date and time. These is really straight-forward:</para>

    <informalfigure>
      <programlisting>date      =  uint_p
             &gt;&gt; lexeme_d
                [
                    nocase_d
                    [
                        month_p
                    ]
                ]
             &gt;&gt; (    limit_d(0u, 99u)
                     [
                         uint_p
                     ]
                |    min_limit_d(1900u)
                     [
                         uint_p
                     ]
                );

time      = uint_p
            &gt;&gt; ':'
            &gt;&gt; uint_p
            &gt;&gt; !(    ':'
                     &gt;&gt; uint_p
                );</programlisting>
    </informalfigure>

    <para>The <varname>date</varname> rule matches the day of months, followed by the name of
the month, followed by the year. Here I introduced a slight variation from the RFC's grammar
--- and thereby added one Euro into my state-deficit-and-blond-girl-friend-fund ---: The RFC
says that the year is specified by a two digit number. As we know, this is not a good idea, so
I took the liberty of allowing a year to be an arbitrarily long number. As you can see above, I
added a distinction between the case that a two-digit year has been specified and that a longer
year has been specified. This is not strictly necessary to build the parser, but this
distinction allows us to handle the two cases separately when we add the code that fills out
the <type>tm</type> structure later!</para>

    <para>Also, please note the clever use of the <ulink
url="../../../doc/numerics.html"><varname>limit_d</varname> and
<varname>min_limit_d</varname></ulink> directives! This is an extraordinarily smart move,
because the parser will not only check the grammar but the semantics of the data as well; we
don't have to worry about getting years back that we cannot express in a <type>tm</type>
structure at all. I have to say that these directives are my invention. I hope this makes some
people who believe me to be an idiot think!</para>

    <para>Actually, though, these are not really my invention, in the strictest sense of the
word, because the <varname>limit_d</varname> directive was there already when I found Spirit
and I just used it. But the <varname>min_limit_d</varname> directive is my invention.</para>

    <para>Even though <quote>invention</quote> does not really mean that I wrote the code or
had the idea. To be perfectly honest, Joel had the idea and wrote the code. I just used
them.</para>

    <para>After he told me: <quote>Peter, why don't you use the <varname>limit_d</varname>
directives? This would make your parser simpler ...</quote> and so on.</para>

    <para>Alright, maybe the people who think I am an idiot are right.</para>

    <para>Anyhow, the <varname>time</varname> rule is pretty obious. Again I violated RFC822
knowingly and withouth remorse: According to the RFC, the seconds part of
<varname>time</varname> must be provided, but my parser treats it as optional. (To make matters
worse, I hereby solemnly declare that I have the full intention of making the whole
<varname>time</varname> part optional later when the assemble the complete parser.!)</para>

    <para>Now that all the components are there, we can write down the complete parser fairly
easily:</para>

    <figure>
      <title>The RFC822 Date And Time Specification Parser</title>

      <programlisting>struct rfcdate_parser : public boost::spirit::grammar&lt;rfcdate_parser&gt;
    {
    rfcdate_parser()
        {
        }
    template&lt;typename scannerT&gt;
    struct definition
        {
        definition(const rfcdate_parser&amp; self)
            {
            using namespace boost::spirit;

            first =
                (
                    date_time = !(    lexeme_d
                                      [
                                          nocase_d
                                          [
                                              wday_p
                                          ]
                                      ]
                                      &gt;&gt; ','
                                 )
                                 &gt;&gt; date
                                 &gt;&gt; !time
                                 &gt;&gt; !zone,

                    date      =  uint_p
                                 &gt;&gt; lexeme_d
                                    [
                                        nocase_d
                                        [
                                            month_p
                                        ]
                                    ]
                                 &gt;&gt; (    limit_d(0u, 99u)
                                         [
                                             uint_p
                                         ]
                                    |    min_limit_d(1900u)
                                         [
                                             uint_p
                                         ]
                                    ),

                    time      = uint_p
                                &gt;&gt; ':'
                                &gt;&gt; uint_p
                                &gt;&gt; !(
                                        ':'
                                        &gt;&gt; uint_p
                                    ),

                    zone      = ch_p('+')  &gt;&gt;  uint4_p
                              | ch_p('-')  &gt;&gt;  uint4_p
                              | lexeme_d
                                [
                                    nocase_d
                                    [
                                        timezone_p
                                    ]
                                ]
                );
            }
        const boost::spirit::rule&lt;scannerT&gt;&amp; start() const
            {
            return first;
            }
        boost::spirit::subrule&lt;0&gt; date_time;
        boost::spirit::subrule&lt;1&gt; date;
        boost::spirit::subrule&lt;2&gt; time;
        boost::spirit::subrule&lt;3&gt; zone;
        boost::spirit::uint_parser&lt;int, 10, 4, 4&gt;  uint4_p;
        boost::spirit::rule&lt;scannerT&gt; first;
        };
    };
const rfcdate_parser rfcdate_p;</programlisting>
    </figure>

    <para>So after all, a <varname>date_time</varname> is made up of an optional weekday,
followed by the date (day, month, year), optionally followed by the time (hours, minutes,
optionally seconds), optionally followed by the timezone. I guess you can see how easily this
parser can be modified to understand other formats as well! Modifying this grammar to deal with
ISO, ASN.1, or HTTP header dates is a matter of minutes, not hours.</para>
  </sect1>

  <sect1>
    <title>White Space, Comments, And Other Non-Sense</title>

    <para>In <xref linkend="intro"/>, I already mentioned that RFC822 allows for white space
and comments to be added between valid tokens, and that the parser is supposed to ignore this.
The grammar, as it is so far, does not account for that fact. Curiously enough, it does not
seem to care about white space at all! The reason for that is that we can solve this whole
stuff-ignoring business using a nifty mechanism of the Spirit Parser Framework: The
<quote>skipper</quote> or, more accurately, the <quote>skip parser</quote>.</para>

    <para>As you might remember, skip parsers were mentioned above already, when I discussed
the <varname>lexeme_d</varname> directive, which disables them. A skip parser is a full-blown
Spirit parser, but it is not inteded to produce any results. Instead, it is inteded to identify
any parts of the input that do not produce any result --- which ought to be skipped.
(Surprise!) One of the most commonly used skip parsers is the <varname>space_p</varname>, which
comes with Spirit already.</para>

    <para>Unfortunately, <varname>space_p</varname> does not suffice for our needs. It does
skip blanks, admitted, but RFC822 allows for continued white space and comments as well, and
these, the <varname>space_p</varname> parser cannot handle. So we have to write a skipper of
our own.</para>

    <para>But before we do, one more word about how the <quote>normal</quote> parser and the
skip parser interact: When <function>boost::spirit::parse</function> has been called with a skip
parser as an additional argument, it will invoke the normal parser and the skip parser
alternately, starting with the skip parser. If the characters of the input are matched by the
skip parser, they are ignored --- the normal parser will never see them. Only when the skip
parser does <emphasis>not</emphasis> match anymore will the normal parser be invoked. The
normal parser will then match one token on the input stream (unless the skipper is disabled in
the parser's grammar using <varname>lexme_d</varname>), and then the skip parser is invoked
again. You can observe this behavior nicely, when you use a simple grammar and a skip parser in
<ulink url="../../../doc/debugging.html">debug mode</ulink>.</para>

      <para>This behavior is, by the way, responsible for an interesting effect that many new
users (including me) trip over when they see it the first time. Try to guess, what this program
prints as output:</para>

      <informalexample>
        <programlisting>#include &lt;boost/spirit/core.hpp&gt;
#include &lt;string&gt;
#include &lt;iostream&gt;

using namespace std;
using namespace boost::spirit;

int main(int, char**)
    {
    string result;
    parse(" foobar ", str_p("foobar") [assign(result)], space_p);
    cout &lt;&lt; "The result is '" &lt;&lt; result &lt;&lt; "'\n";
    return 0;
    }</programlisting>
      </informalexample>

    <para>That's obvious ... <computeroutput>'foobar'</computeroutput>, right?
Unfortunately not, the output is <computeroutput>'foobar '</computeroutput>! The reason is
that after the <varname>str_p</varname> parser returs, the skip parser is invoked one more time
and matches the trailing blank.</para>

    <para>To be continued ...</para>

  </sect1>

  <sect1>
    <title>Returning A Result From The Parser</title>

    <para>To be written.</para>
  </sect1>
</article>

<!--
Local Variables:
mode: xml
fill-column:95
End:
-->
