        Mail System Design
        ------------------

        1.  The mail store

            The mail store is a complex problem.  On the one hand, we
            could simply store RFC822 messages in raw form.
            Unfortunately, this could lead to serious problems when trying
            to manipulate messages.  (For example, the code to parse out
            part 2.1.5 of a message would have to actuall parse MIME.)  It
            might be better to use a view in which MIME messages are
            pre-separated into parts.  Or perhaps a single message is
            stored and a "part" table describes the ranges covered by
            various MIME parts.

            More research into this needs to be done.  For the short term,
            using a non-MIME mechanism for storing information we wish to
            send out (attachments in bboard, the like) is sufficient.  The
            conversion can be done at mail send time.  Received email can
            be treated as text/plain for the moment, even when it is
            richer.

            Here's a plan for the table design:

            message_body
            ------------
            rfc822_id
            header_from
            header_date
            header_message_id
            header_content_type
            ... others

            message
            -------
            message_id
            rfc822_id
            env_from
            env_to

            With each message_body being associated with a content value
            from the acs_contents table (or CR revisions table.)

            Notice that each message body belongs to one *or more*
            messages.  This is because the same message might be addressed
            to more than one recipient, but still be the same message.
            Because of this it is recommended that editing messages take
            place via "copy on write" rather than "modify in place".  This
            will allow messages to be edited, but for old versions in
            other locations not be be lost.  (Keep the message_id the
            same, but point it at a new rfc822_id for the body.)

            Separating messages from message_bodies also makes it more
            clear how to determine what access control and URL any message
            belongs to.  Each message belongs in only one place, but the
            message_body may be shared.

            The above design for message_body should include whatever
            header fields are determined to be necessary and useful.
            Difficulties arise, since for email addresses it would be nice
            for this version to point at parties, but the "from" header of
            messages may actually contain more than one address, and so
            on.  Some thought needs to be applied here.

        2.  Sending email

            The first iteration should send email using ns_sendmail or the
            Oracle UTL_SMTP stuff.  ns_sendmail is preferred, since it's
            much higher level than UTL_SMTP.

            A simple queue should be sufficient for sending.  Each message
            may be queued to be sent to various people, like so:

            outgoing_message_queue
            ----------------------
            message_id

            Notice that the pointer is to a message, not a message body.
            The expectation is that a new message is created (to contain
            the new envelope information), and that that message is
            recorded in this queue.

            Once the message is sent, the message is removed from this
            queue (and possibly added to an audit trail?  Kept in the
            queue marked as done?)

            The requirement to be able to delete messages from the system
            immediately after deletion means that there are some other
            issues here.  See "open issues" below.

        3.  Receiving email

            The simplest design for receiving email into various queues
            assumes that each message actually belongs to one and only one
            queue:

            incoming_message_queue
            ----------------------
            message_id
            queue_name

            A table can describe a set of rules (with like patterns?) to
            use to determine what queue_name should be used for initial
            insertion.  This gives any application the capability to
            periodically find all new messages that it's interested in
            processing.  Removal from the queue is per application
            request.

            An alternative method is for messages to go in all queues with
            matching requirements.  The same as above, only unique on the
            pair instead of just message_id.  This would allow messages to
            be placed into multiple queues for processing by more than one
            application.  I suspect this is unnecessary.

            A final thought is that it would be possible to simply record
            message ids in this table (as in the outgoing queue.)  Then
            individual applications do a select over the incoming message
            queue, looking for interesting messages.  When they are
            finished, they delete the message.  Again, this is based on
            the "no message will belong to more than one application"
            model.

        4.  Manipulating messages

            This is a difficult task in Tcl, since MIME is fairly
            complicated.  There is already a good API for Java.  I suggest
            that for the Tcl API, simple abilities to deal with encoding
            (and later possibly decoding) messages with multipart/mixed
            and multipart/alternative content types.  This will require
            good base64 encoding tools in Tcl.

        5.  Open questions

            The biggest open question is garbage collection.
            Paradoxically, the split into messages and message_bodys may
            help with this.  Since we can imagine that each application
            keeps its own message_body object, the application is allowed
            to delete that message_body in a reasonable way.  At that
            point, it's sufficient to periodically clear the message store
            of any message bodies that have no associated messages!

            Extension of the message class also becomes more possible.
            When a new message comes in to be entered into a bboard forum,
            the bboard application can create a new message object
            pointing at the same message body.  The new message object can
            be a "bboard_message".  Previously, acs_messaging would create
            the message object, and bboard would have to deal with it even
            if it were a "normal" message.

            It is unclear at this point whether this system should
            actually provide mechanisms to automatically build digests.
            On the one hand, it already has scheduled tasks to do such
            things.  On the other hand, putting the generic mechanisms to
            handle collation and the like into the system is messy.  And
            on top of that, there are optimizations that applications can
            do.  (For example, bboard could build a single message_body
            for a daily digest and send it out to all subscribers, instead
            of messaging building a customized message for each
            subscriber, because it can't determine that they would all be
            the same.)

        6.  Data model in short:

                            outgoing_message_queue
                             | (message_id)
                             |
            message_body -- message --<|------------ [ extended by application ]
               (rfc822_id)   | (message_id)
                             |
                            incoming_message_queue
                               (message_id, queue_name)

            A scheduled process periodically sends any messages in the
            outgoing message_queue and deletes the message entry.

            When messages arrive, some MTA mechanism inserts them as a
            message_body, with a message containing the envelope info, and
            puts that message into the appropriate
            incoming_message_queues.

            A scheduled process periodically deletes any message_body for
            which there exists no referencing message.

            Each message belongs to an individual application.  That
            application is responsible for removing it once it is no
            longer needed (and then GC can be accomplished.)