# Maildir filter implementation

Implementing mail filtering requires a couple of dominos to fall in the right order. This is not difficult to do, but is a bit tricky. Here's how it works, in general. Specifics follow.

Some people would probably have a difficult time setting it up. That's to be expected. The implementation allows only selected accounts to be set up for mail filtering, so the suggested way is to attempt to set up mail filtering for one account only, test it to make sure it works, then roll it out globally.

## The big picture

The maildrop mail filter is used to do the actual mail filter. The first thing you need to do is to download and install maildrop. Download maildrop from http://www.courier-mta.org/maildrop/. Follow the instruction in the maildrop package to configure it and install it. maildrop must be installed as your mail server's mail delivery agent. maildrop must be used to deliver mail to the maildir, obviously, since that's how filtering takes place. You need to follow the instructions for your mail server in order to install maildrop as the mail delivery agent. maildrop must be configured to locate the recipient's maildir, and dump mail there by default. Review maildrop's and the mail server's documentation. This can't be any more specific. Different mail servers are configured in different way.

For example, with Qmail the usual thing to do is to initialize $HOME/.qmail with "| /usr/local/bin/maildrop". maildrop is also included as part of the Courier mail server. If you're running Courier already, you don't need to do actually download and install maildrop. You already have everything you need, all you need to do is to flip the switch that uses maildrop for local mail delivery. This is done by editing the `courierd` configuration file (usually installed in the `/usr/lib/courier/etc` directory), and setting `DEFAULTDELIVERY` according to the instructions in that configuration file.

The next thing to do is to actually learn how maildrop works, and learn its filtering language. Although the mail filter will be automaticaly generated here, you still need to become familiar with the filtering language in order to troubleshoot any installation problems. maildrop comes with manual pages documenting the filtering language, as well as some example. Read them.

## The little picture

Here's what's going to happen. The webmail server will automatically generate a maildrop filtering recipe. maildrop reads the recipe, and does its thing. Sounds simple enough, right?

Well, it's not. There are a few little details that need to be resolved.

For starters, the default maildrop filtering recipe is `$HOME/.mailfilter`. That's how things usually are when actual physical system accounts are used. When virtual mailboxes are installed, things are less murky. There is no standard way to do virtual mail arrangement, so the actual implementation varies from system to system. Somehow or other all or some portion of virtual mailboxes share the same physical account, and have the same $HOME. In that case the usual thing to do is for each virtual mailbox, the corresponding mail filtering recipe is manually specified to maildrop, as an extra command line argument. Review the maildrop documentation for more information.

Now, on the other hand, the webmail server doesn't really know anything about physical and virtual accounts. The mapping between a login ID and the maildir is completely encapsulated in a black box-type authentication layer. The login ID and password are validated by the authentication layer, which obtains the maildir corresponding to the login ID, and the webmail server is started for that maildir. Whether or not a login ID corresponds to an actual system account or to virtual account, that's something the webmail server doesn't really know, or care. All it cares about is the maildir where all the mail is, and that's the end of the story.

So, the issue is that the webmail server needs to find the corresponding maildrop filtering recipe, so it knows where to write the mail delivery instructions. This is how it works.

In order for mail filtering to be enabled, it is necessary to initialize the file named `maildirfilterconfig` in the maildir itself. This is where the webmail server runs, so it will simply look for the file of this name. The contents of this file should be as follows:

    MAILDIRFILTER=pathtomailfilter
    MAILDIR=pathtomaildir

The pathtomailfilter specifies the location of the maildrop filtering recipe for this maildir, relative to the maildir itself. Set the current directory to the maildir directory, now ask yourself, where's my maildrop filtering recipe.

In most cases, where virtual mailboxes are not used, everyone's maildir is `$HOME/Maildir`, and maildrop uses `$HOME/.mailfilter` by default. In this case, pathtomailfilter must be set to `../.mailfilter`.

When virtual mail accounts are used, this will obviously be something else. The only restriction is that the maildrop filtering recipe and the maildir must be on the same filesystem or partition.

pathtomaildir is the other half of the story. When maildrop gets a message to deliver, maildrop needs to know where the mailboxes and folders are. Maildrop begins running in what it considers to be the recipient's home directory, reading either `.mailfilter`, by default, or the file specified on the command line.

The webmail server needs to generate filtering instruction that deliver messages to its maildir. By default, the maildir for non-virtual accounts is $HOME/Maildir, so pathtomaildir needs to be set to `./Maildir`.

## Summary for virtual accounts

Basically, 99% of the time `MAILDIRFILTER` will be `../.mailfilter` and `MAILDIR` will be `./Maildir`. When virtual mail accounts are used, maildrop still must be used as a mail delivery agent, somethow, specifying the correct maildir that corresponds to the recipient's mail account. Usually all or most virtual accounts are set up inside a single physical account. In that case it is necessary to set up a different maildrop filtering recipe file for each virtual mail account (since everyone's `$HOME/.mailfilter` will be the same file), and in each maildir specify the relative path to its corresponding filtering recipe, and the relative path to the maildir from the default home directory. Then, for each virtual mail account, the mail server must run maildrop to do the actual mail delivery, explicitly specifying the filtering recipe to be used.

## The global `maildirfilterconfig` file

In most cases where virtual mail accounts are not used, every maildir's `maildirfilterconfig` should be the same. As an alternative to installing the same `maildirfilterconfig` in each maildir, it is possible to install a single `maildirfilterconfig` systemwide.

For SqWebMail, install `maildirfilterconfig` in `/usr/local/share/sqwebmail` (the default installation directory). For Courier, install `maildirfilterconfig` in Courier's configuration directory (`/usr/lib/courier/etc` by default).

The global `maildirfilterconfig` will be used unless `maildirfilterconfig` exists in the maildir directory. Therefore, the global `maildirfilterconfig` can be used to provide a default for the majority of the mail accounts, and any exceptions are handled by installing `maildirfilterconfig` in the maildir directory, whose contents will override the global file.

## Happy filtering.

If everything is set up correctly, SqWebMail will now display a new link to a screen where mail filtering rules are defined and installed. A mail filter consists of a condition, and an action. A condition is usually based on the contents of some header, or the message body. Regular expressions are allowed. Which means that certain special characters must be quoted. For example, to search for the string "[announce]" verbatim in the subject header, it must be entered as "\[announce\]". Pattern matching, for now, is case-insensitive. The regular expression syntax uses pretty much the same syntax as Perl. See the maildropfilter manual page for more information.

Multiple mail filtering rules can be installed. Their relative order can be rearranged by selecting a filtering rule, then selecting either the "Up" or the "Down" button. It is necessary to select "Save all changes" for any changes to the filtering rules to take effect. Leaving that page in any other way will throw away all changes made.

## BUGS

Known bug:

- Create a filter that delivers mail to folder X.
- Delete folder X.
- What's going to happen now with mail grabbed by this filter? In the current implementation this results with a deferred error. The message is not bounced immediately, but will stay in the queue until the mail server times it out. If this condition is caught quickly enough, and the folder is recreated, all pent up mail will be redelivered.
- There's some school of thought that says that this should result in mail bouncing immediately. No decision has been made at this point.