              Cyrus IMAP Server:  Overview and Concepts

    This document gives an overview of the Cyrus IMAP server. The Cyrus
    IMAP (Internet Message Access Protocol) server provides access to
    personal mail and system-wide bulletin boards through the IMAP
    protocol. The Cyrus IMAP server is a scaleable enterprise mail system
    designed for use from small to large enterprise environments using
    standards-based technologies. 

    A full Cyrus IMAP implementation allows a seamless mail and bulletin
    board environment to be set up across multiple servers. It differs from
    other IMAP server implementations in that it is run on "sealed"
    servers, where users are not normally permitted to log in. The mailbox
    database is stored in parts of the filesystem that are private to the
    Cyrus IMAP system. All user access to mail is through software using
    the IMAP, POP3, or KPOP protocols. 

    The private mailbox database design gives the server large advantages
    in efficiency, scalability, and administratability. Multiple concurrent
    read/write connections to the same mailbox are permitted. The server
    supports access control lists on mailboxes and storage quotas on
    mailbox hierarchies. 

    This document is organized into the following areas: 

      * Mailbox Namespace 
      * Access Control Lists 

          * Access Rights 
          * Identifiers 
          * "anonymous" and "anyone" 
          * Kerberos vs. Unix Authorization 
          * Negative Rights 
          * Calculating Users' Rights 
          * Implicit Rights for Administrators on Personal Mailboxes 
          * Initial ACLs for Newly Created Mailboxes 

      * Login Authentication 

          * Anonymous Logins 
          * Kerberos Logins 
          * Unix Logins 

      * Quotas 

          * Supports Quotas on Storage 
          * Quota Roots 
          * Mail Delivery Behavior 
          * Quota Warnings Upon Select When User Has "d" Rights 
          * Quotas and Partitions 

      * New Mail Notification 
      * Partitions 

          * Specifying Partitions with "create" 
          * Changing Partitions with "rename" 

      * News 

          * "news" Partition 
          * Netnews 
          * collectnews 
          * rmnews 
          * syncnews 

      * IMSP Integration 
      * POP3 Server 
      * The syslog facility 
      * Mail Directory Recovery 

          * Reconstructing Mailbox Directories 
          * Reconstructing the Mailboxes File 
          * Reconstructing Quota Roots 
          * Removing Quota Roots 
          * Subscriptions 

      * Configuration Directory 

          * "log" Directory 
          * "proc" Directory 

Mailbox Namespace

    The Cyrus IMAP server presents mailboxes using the netnews namespace
    convention. Mailbox names are case-sensitive. A mailbox name may not
    start or end with a "." character, nor may it contain two "."
    characters in a row. 

    In the initial implementation, non-ASCII characters, shell
    metacharacters, and "/" are not permitted in mailbox names. Later
    versions of the server may make the set of permissible characters
    configurable at each site. 

    All personal mailboxes for user "bovik" begin with the string
    "user.bovik.". For example, if user "bovik" had a personal "work"
    mailbox, it would be called "user.bovik.work". To user "bovik",
    however, the prefix "user.bovik." normally appears as "INBOX.". The
    mailbox "user.bovik.work" would therefore appear as "INBOX.work". If
    the access control list of the mailbox permitted other users to see
    that mailbox, it would appear to them as "user.bovik.work". 

    The mailbox "user.bovik" is where the user "bovik" normally receives
    new mail, and normally appears to user "bovik" as "INBOX". The mailbox
    "user.bovik" is referred to in this document as user "bovik"'s INBOX. 

    Administrators create and delete users by creating and deleting the
    users' INBOX. If a user has an INBOX, then they are allowed to
    subscribe to mailboxes. Only users without dots in their userid are
    permitted to have an INBOX. (A user with a dot in their userid would be
    able to login but would not be able to receive mail.) 

    When an administrator deletes a user's INBOX, all of the user's
    personal mailboxes are deleted as well. 

    With the one notable exception of INBOX, all mailbox names are
    system-wide--they refer to the same mailbox regardless of the user.
    Access control lists determine which users can access or see which
    mailboxes. 

    In contexts which permit relative mailbox names, the mailbox namespace
    works as follows: 

      * Names that do not start with "." are fully qualified. 
      * Names that start with "." are relative to the current context. 

    Thus, if you are working with folder names and the top of the hierarchy
    is named "cmu.", the name "comp.infosystems.www" resolves to
    "comp.infosystems.www" and the name ".comp.infosystems.www" resolves to
    "cmu.comp.infosystems.www". 

Access Control Lists

    Access to each mailbox is controlled by each mailbox's access control
    list. Access Control Lists (ACLs) provide a powerful mechanism for
    specifying the users or groups of users who have permission to access
    the mailboxes. 

    An ACL is a list of zero or more entries. Each entry has an identifier
    and a set of rights. The identifier specifies the user or group of
    users for which the entry applies. The set of rights is one or more
    letters or digits, each letter or digit conferring a particular
    privilege. 

Access Rights

    The defined rights are: 

    l       lookup - The user may see that the mailbox exists. 
    r       read - The user may read the mailbox. The user may select the
            mailbox, fetch data, perform searches, and copy messages from
            the mailbox. 
    s       seen - Keep per-user seen state. The "Seen" and "Recent" flags
            are preserved for the user. 
    w       write - The user may modify flags and keywords other than
            "Seen" and "Deleted" (which are controlled by other sets of
            rights). 
    i       insert - The user may insert new messages into the mailbox. 
    p       post - The user may send mail to the submission address for the
            mailbox. This right differs from the "i" right in that the
            delivery system inserts trace information into submitted
            messages. 
    c       create - The user may create new sub-mailboxes of the mailbox. 
    d       delete - The user may store the "Deleted" flag, perform
            expunges, and delete or rename the mailbox. 
    a       administer - The user may change the ACL on the mailbox. 

    You can combine access rights in different ways. For example: 

    lrs     The user can read the mailbox. 

    lrsp    The user can read the mailbox, and can post to it through the
            delivery system. Most delivery systems do not provide
            authentication, so the "p" right usually has meaning only for
            the "anonymous" user. 

    lr      The user can see the mailbox and can read it, but the server
            does not preserve the "Seen" and "Recent" flags. This set of
            rights is primarily useful for "anonymous IMAP." 

    rs      The user can read the mailbox and the server preserves the
            "Seen" and "Recent" flags, but the mailbox is not visible to
            the user through the various mailbox listing commands. The user
            must know the name of the mailbox to be able to access it. 

    lrsip   The user can read and append to the mailbox, either through
            IMAP, or through the delivery system. 

Identifiers

    The identifier part of an ACL entry specifies the user or group for
    which the entry applies. The meaning of an identifier usually depends
    on the authorization mechanism being used. 

"anonymous" and "anyone"

    With any authorization mechanism, two special identifiers are defined.
    The identifier "anonymous" refers to the anonymous, or unauthenticated
    user. The identifier "anyone" refers to all users, including the
    anonymous user. 

Kerberos vs. Unix Authorization

    The Cyrus IMAP server comes with two authorization mechanisms, one for
    use with Unix "/etc/passwd" files, one for use with Kerberos. 

    In the Unix authorization mechanism, identifiers are either a userid
    listed in "/etc/passwd" or the string "group": followed by a group
    listed in "/etc/group". Thus: 

        root                Refers to the user root
        group:staff         Refers to the group staff

    Using the Kerberos authorization mechanism, identifiers are of the
    form: 

        principal.instance@realm

    If ".instance" is omitted, it defaults to the null string. If "@realm"
    is omitted, it defaults to the local realm. 

    "Principal", "instance", or "realm" may be replaced by the "*"
    wildcard. 

      * A "*" in an identifier represents all principals with the null
        instance at the local realm. 
      * A "*.*@FOO.COM" in an identifier represents all principals with all
        instances at FOO.COM. 
      * A "*.*@*" in an identifier represents all Kerberos-authenticated
        identities. 

    None of the above refer to the anonymous user. 

    The file "/etc/krb.equiv" contains mappings between Kerberos
    principals. The file contains zero or more lines, each containing two
    fields. Any identity matching the first field of a line is changed to
    the second identity during canonicalization. For example, a line in
    "/etc/krb.equiv" of: 

        bovik@REMOTE.COM bovik

    will cause the identity "bovik@REMOTE.COM" to be treated as if it were
    the local identity "bovik". 

    A site may wish to write their own authorization mechanism, perhaps to
    implement a local group mechanism. 

Negative Rights

    Any of the above defined identifiers may be prefixed with a "-"
    character. The associated rights are then removed from that identifier.
    These are referred to as "negative rights". 

Calculating Users' Rights

    To calculate the set of rights granted to a user, the server first
    calculates the union of all of the rights granted to the user and to
    all groups the user is a member of. The server then calculates and
    removes the union of all the negative rights granted to the user and to
    all groups the user is a member of. 

    For example, given the ACL: 

       anyone       lrsp
       fred         lwi
       -anonymous   s

    The user "fred" will be granted the rights "lrswip" and the anonymous
    user will be granted the rights "lrp". 

Implicit Rights for Administrators on Personal Mailboxes

    Regardless of the ACL on a mailbox, users who are listed in the
    "admins" configuration option in "/etc/imapd.conf" implicitly have the
    "l" and "a" rights on all mailboxes. Users also implicitly have the "l"
    and "a" rights on their INBOX and all of their personal mailboxes. 

Initial ACLs for Newly Created Mailboxes

    When a mailbox is created, its ACL starts off with a copy of the ACL of
    its closest parent mailbox. When a user is created, the ACL on the
    user's INBOX starts off with a single entry granting all rights to the
    user. When a non-user mailbox is created and does not have a parent,
    its ACL is initialized to the value of the "defaultacl" option in
    "/etc/imapd.conf" 

Login Authentication

    This section discusses different types of logins that can be used with
    IMAP. 

    The Cyrus IMAP server uses the Cyrus SASL library for authentication.
    This section describes how to configure SASL with use with Cyrus imapd.
    Please consult the Cyrus SASL System Administrator's Guide for more
    detailed, up-to-date information. 

Anonymous Logins

    Regardless of the SASL mechanism used by an individual connection, the
    server can support anonymous login. If the "allowanonymouslogin" option
    in "/etc/imapd.conf" is turned on, then the server will permit
    plaintext password logins using the user "anonymous" and any password.
    Additionally, the server will enable any SASL mechanisms that allow
    anonymous logins. 

Plaintext Authentication

    The SASL library has several ways of verifying plaintext passwords
    Plaintext passwords are passed either by the IMAP LOGIN command or by
    the SASL PLAIN mechanism. 

      * PAM 
      * Kerberos v4 Plaintext passwords are verified by obtaining a ticket
        for the server's Kerberos identity, to protect against Kerberos
        server spoofing attacks. 
      * /etc/passwd 
      * /etc/shadow 

    sasl_auto_transition automatically creates secrets for shared secret
    authentication when given a password 

Kerberos Logins

    The Kerberos SASL mechanism supports the KERBEROS_V4 authentication
    mechanism. The mechanism requires that a srvtab file exist in the
    location given in the "srvtab" configuration option. The srvtab file
    must be readable by the Cyrus server and must contain a
    "imap.<host>@<realm>" service key, where <host> is the first component
    of the server's host name and <realm> is the server's Kerberos realm. 

    The server will permit logins by identities in the local realm and
    identities in the realms listed in the "loginrealms" option in
    "/etc/imapd.conf". 

    The file "/etc/krb.equiv" contains mappings between Kerberos
    principals. The file contains zero or more lines, each containing two
    fields. Any identity matching the first field of a line is permitted to
    log in as the identity in the second field. 

    If the "loginuseacl" configuration option is turned on, than any
    Kerberos identity that is granted the "a" right on the user's INBOX is
    permitted to log in as that user. 

Shared Secrets Logins

Quotas

    Quotas allow server administrators to limit resources used by
    hierarchies of mailboxes on the server. 

Supports Quotas on Storage

    The Cyrus IMAP server supports quotas on storage, which is defined as
    the number of bytes of the relevant RFC-822 messages, in kilobytes.
    Each copy of a message is counted independently, even when the server
    can conserve disk space use by making hard links to message files. The
    additional disk space overhead used by mailbox index and cache files is
    not charged against a quota. 

Quota Roots

    Quotas are applied to quota roots, which can be at any level of the
    mailbox hierarchy. Quota roots need not also be mailboxes. 

    Quotas on a quota root apply to the sum of the usage of any mailbox at
    that level and any sub-mailboxes of that level that are not underneath
    a quota root on a sub-hierarchy. This means that each mailbox is
    limited by at most one quota root. 

    For example, if the mailboxes 

       user.bovik
       user.bovik.list.imap
       user.bovik.list.info-cyrus
       user.bovik.saved
       user.bovik.todo

    exist and the quota roots 

       user.bovik 
       user.bovik.list
       user.bovik.saved

    exist, then the quota root "user.bovik" applies to the mailboxes
    "user.bovik" and "user.bovik.todo"; the quota root "user.bovik.list"
    applies to the mailboxes "user.bovik.list.imap" and
    "user.bovik.list.info-cyrus"; and the quota root "user.bovik.saved"
    applies to the mailbox "user.bovik.saved". 

    Quota roots are created automatically when they are mentioned in the
    "setquota" command. Quota roots may not be deleted through the
    protocol, see Removing Quota Roots for instructions on how to delete
    them. 

Mail Delivery Behavior

    Normally, in order for a message to be inserted into a mailbox, the
    quota root for the mailbox must have enough unused storage so that
    inserting the message will not cause the block quota to go over the
    limit. 

    Mail delivery is a special case. In order for a message to be delivered
    to a mailbox, the quota root for the mailbox must not have usage that
    is over the limit. If the usage is not over the limit, then one message
    may be delivered regardless of its size. This puts the mailbox's usage
    over the quota, causing a user to be informed of the problem and
    permitting them to correct it. If delivery were not permitted in this
    case, the user would have no practical way of knowing that there was
    mail that could not be delivered. 

    If the usage is over the limit, then the mail delivery will fail with a
    temporary error. This will cause the delivery system to re-attempt
    delivery for a couple of days (permitting the user time to notice and
    correct the problem) and then return the mail to the sender. 

Quota Warnings Upon Select When User Has "d" Rights

    When a user selects a mailbox whose quota root has usage that is close
    to or over the limit and the user has "d" rights on the mailbox, the
    server will issue an alert notifying the user that usage is close to or
    over the limit. The threshold of usage at which the server will issue
    quota warnings is set by the "quotawarn" configuration option. 

    The server only issues warnings when the user has "d" rights because
    only users with "d" rights are capable of correcting the problem. 

Quotas and Partitions

    Quota roots are independent of partitions. A single quota root can
    apply to mailboxes in different partitions. 

New Mail Notification

    The Cyrus IMAP server comes with an optional notification mechanism,
    for use with the Zephyr notification service. 

    If the server software is compiled with the Zephyr notification
    mechanism, then the server sends a Zephyrgram to a user whenever mail
    is delivered to one of that user's mailboxes. The Zephyrgram is sent
    with a class of "MAIL" and an instance of the mailbox name. 

Partitions

    Partitions allow administrators to store different mailboxes in
    different parts of the Unix filesystem. This is intended to be used to
    allow hierarchies of mailboxes to be spread across multiple disks. 

Specifying Partitions with "create"

    When an administrator creates a new mailbox, the name of the partition
    for the mailbox may be specified using an optional second argument to
    the "create" command. Non-administrators are not permitted to specify
    the partition of a mailbox. If the partition is not specified, then the
    mailbox inherits the partition of its most immediate parent mailbox. If
    the mailbox has no parent, it gets the partition specified in the
    "defaultpartition" configuration option. 

    The optional second argument to the "create" command can usually be
    given only when using a specialized administrative client such as
    cyradm. 

Changing Partitions with "rename"

    An administrator may change the partition of a mailbox by using the
    rename command with an optional third argument. When a third argument
    to rename is given, the first and second arguments can be the
    same--this changes the partition of a mailbox without changing its
    name. If a third argument to rename is not given and the first argument
    is not "INBOX", the partition of a mailbox does not change. If a third
    argument to rename is not given and the first argument is "INBOX", the
    newly created mailbox gets the same partition it would get from the
    "create" command. 

News

    This section contains an overview of concepts related to exporting news
    groups through the IMAP server. 

"news" Partition

    The partition named "news" is reserved for netnews and cannot be used
    for any other purpose. 

Netnews

    The IMAP server can export netnews newsgroups as IMAP mailboxes. This
    is done by creating a partition named "news" which points to the news
    spool directory and arranging for the IMAP server's auxiliary databases
    to be maintained. The server software includes programs for integrating
    with the INN netnews server. The following information assumes working
    knowledge of the INN netnews server. 

collectnews

    The "collectnews" program takes as input a list of article file path
    names relative to the news spool directory. It then adds those messages
    to the auxiliary databases. If collectnews encounters a newsgroup that
    does not have a corresponding IMAP mailbox listed in the mailboxes file
    (described later), it adds it there. 

    The "collectnews" program is normally run from cron as the cyrus user.
    It is given as input a news overview feed. Since collectnews needs to
    write the auxiliary databases in the news spool area, the cyrus user
    must be in the news group and the news group must be able to write to
    the news spool area. 

rmnews

    The "rmnews" program takes a list of article pathnames relative to the
    news spool directory. It removes (expunges) any mention of those
    messages from the auxiliary databases and unlinks the article files.
    The "rmnews" program is used to remove both expired and canceled
    articles. 

    "Rmnews" takes the same input as the "fastrm" program. It is normally
    run to clean up after expire by invoking news.daily with the "delayrm"
    argument and having RMPROC in expirerm changed from "fastrm" to
    "rmnews". 

    Rmnews is also normally run from cron as the cyrus user to remove
    canceled and superseded articles. It is given as input a special cancel
    feed. 

    As "rmnews" is invoked at expire time by the news user and must be able
    to write to the auxiliary databases, it must be installed with set-uid
    to the cyrus user and set-gid to the news group. 

syncnews

    The "syncnews" program takes as an argument the name of the news active
    file. It compares the list of IMAP news mailboxes with the news active
    file. News mailboxes which are not listed in the active file are
    removed. Newsgroups listed in the active file but not in the IMAP
    mailboxes file have IMAP mailboxes created. 

    A newsgroup must have a status of ``y'', ``m'', or ``n'' to be
    considered listed in the active file. The "syncnews" program should be
    invoked by the cyrus user at least daily from cron. 

IMSP Integration

    Integration with the Cyrus IMSP server, to allow transparent location
    of mailboxes on multiple servers, is not completed as of version 1.5. 

POP3 Server

    The Cyrus IMAP server software comes with a compatibility POP3 server.
    Due to limitations in the POP3 protocol, the server can only access a
    user's INBOX and only one instance of a POP3 server may exist for any
    one user at any time. While a POP3 server has a user's INBOX open,
    expunge operations from any concurrent IMAP session will fail. 

    When Kerberos login authentication is being used, the POP3 server uses
    the server identity "pop.host@realm" instead of "imap.host@realm",
    where "host" is the first component of the server's host name and
    "realm" is the server's Kerberos realm. When the POP3 server is invoked
    with the "-k" switch, the server exports MIT's KPOP protocol instead of
    generic POP3. 

The syslog facility

    The Cyrus IMAP server software sends log messages to the "local6"
    syslog facility. The severity levels used are: 

      * CRIT - Critical errors which probably require prompt administrator
        action 
      * ERR - I/O errors, including failure to update quota usage. The
        syslog message includes the specific file and Unix error. 
      * WARNING - Protection mechanism failures, client inactivity timeouts 
      * NOTICE - Authentications, both successful and unsuccessful 
      * INFO - Mailbox openings, duplicate delivery suppression 

Mail Directory Recovery

    This section describes the various databases used by the Cyrus IMAP
    server software and what can be done to recover from various
    inconsistencies in these databases. 

Reconstructing Mailbox Directories

    The largest database is the mailbox directories. Each mailbox directory
    contains the following files: 

    message files 
            There is one file per message, containing the message in RFC
            822 format. Lines in the message are separated by CRLF, not
            just LF. The file name of each message is the message's UID
            followed by a dot (.). 

            In netnews newsgroups, the message files instead follow the
            format and naming conventions imposed by the netnews software. 

    cyrus.header 
            This file contains a magic number and variable-length
            information about the mailbox itself. 

    cyrus.index 
            This file contains fixed-length information about the mailbox
            itself and each message in the mailbox. 

    cyrus.cache 
            This file contans variable-length information about each
            message in the mailbox. 

    cyrus.seen 
            This file contains variable-length state information about each
            reader of the mailbox who has "s" permissions. 

    The "reconstruct" program can be used to recover from corruption in
    mailbox directories. If "reconstruct" can find existing header and
    index files, it attempts to preserve any data in them that is not
    derivable from the message files themselves. The state "reconstruct"
    attempts to preserve includes the flag names, flag state, and internal
    date. "Reconstruct" derives all other information from the message
    files. 

    An administrator may recover from a damaged disk by restoring message
    files from a backup and then running reconstruct to regenerate what it
    can of the other files. 

    The "reconstruct" program does not adjust the quota usage recorded in
    any quota root files. After running reconstruct, it is advisable to run
    "quota -f" (described below) in order to fix the quota root files. 

Reconstructing the Mailboxes File

    The mailboxes file in the configuration directory is the most critical
    file in the entire Cyrus IMAP system. It contains a sorted list of each
    mailbox on the server, along with the mailboxes quota root and ACL. To
    reconstruct a corrupted mailboxes file, run the "reconstruct -m"
    command. The "reconstruct" program, when invoked with the "-m" switch,
    scavenges and corrects whatever data it can find in the existing
    mailboxes file. It then scans all partitions listed in the imapd.conf
    file for additional mailbox directories to put in the mailboxes file. 

    The cyrus.header file in each mailbox directory stores a redundant copy
    of the mailbox ACL, to be used as a backup when rebuilding the
    mailboxes file. 

Reconstructing Quota Roots

    The subdirectory "quota" of the configuration directory (specified in
    the "configdirectory" configuration option) contains one file per quota
    root, with the file name being the name of the quota root. These files
    store the quota usage and limits of each of the quota roots. 

    The "quota" program, when invoked with the "-f" switch, recalculates
    the quota root of each mailbox and the quota usage of each quota root. 

Removing Quota Roots

    To remove a quota root, remove the quota root's file. Then run "quota
    -f" to make the quota files consistent again. 

Subscriptions

    The subdirectory "user" of the configuration directory contains user
    subscriptions. There is one file per user, with a filename of the
    userid followed by ".sub". Each file contains a sorted list of
    subscribed mailboxes. 

    There is no program to recover from damaged subscription files. A site
    may recover from lost subscription files by restoring from backups. 

Configuration Directory

    Many objects in the configuration directory are discussed in the
    Database Recovery section. This section documents two other directories
    that reside in the configuration directory. 

"log" Directory

    The subdirectory "log" under the configuration directory permits
    administrators to keep protocol telemetry logs on a per-user basis. 

    If a subdirectory of "log" exists with the same name as a user, the
    IMAP and POP3 servers will keep a telemetry log of protocol sessions
    authenticating as that user. The telemetry log is stored in the
    subdirectory with a filename of the server process-id and starts with
    the first command following authentication. 

"proc" Directory

    The subdirectory "proc" under the configuration directory contains one
    file per active server process. The file name is the ASCII
    representation of the process id and the file contains the following
    tab-separated fields: 

      * hostname of the client 
      * login name of the user, if logged in 
      * selected mailbox, if a mailbox is selected 

    The file may contain arbitrary characters after the first newline
    character. 

    The "proc" subdirectory is normally be cleaned out on server reboot. 

