fetchmail

fetchmail acts as a gateway between an external message store such as an IMAP or POP3 server and James. Mail is fetched from the external message store and injected into the James input spool.

fetchmail is useful when delivery via standard SMTP is not an option, as a means of consolidating mail delivered to several external accounts into a single James account, or to apply the mail processing capabilities of James to mail stored in an external message store.

fetchmail has several configuration options that control the fetching and filtering of mail injected into the James input spool. Once there, James' flexible mail processing engine can be used to further process the mail, just as if it had been delivered via standard SMTP.

How fetchmail Works
fetchmail Configuration Parameters
fetchmail Examples
fetchmail Caveats

How fetchmail Works

Mail is delivered by periodically running fetch tasks that read messages from an external message store and injects them into the James input spool. Fetch tasks run concurrently.

A set of filters applies to each fetch task. Each filter provides the ability to reject a message that matches the filter criteria. Rejected messages are not injected into the James input spool; they are either marked as seen or deleted. When a filter is configured to accept a message that matches its criteria, messages are marked with a MailAttribute. This MailAttribute can be detected within the James matcher/mailet chain, allowing further processing as required.

Each fetch task is associated with a single host server. Accounts are defined to the fetch task for each mailbox on the server from which mail is to be fetched. Accounts run consecutively.

Optionally, the fetch task can be configured with an <alllocal> Account that generates an Account entry for each user defined in the James user repository. This removes the requirement to manually add or remove Account entries to the fetchmail configuration each time a James user is added or removed. Currently this is only useful if the server supports virtual mailboxes that allow the same password to apply to all users within a domain.

Accounts can be configured to deliver all mail for an Account to a specified recipient or to deduce the intended recipient from the mail headers.

Accounts are normally configured to deliver all mail for an Account to a specified recipient, ignoring the recipient in the mail headers. This works well in the majority of cases where a mailbox is guaranteed to contain mail for a sole mailbox recipient.

Accounts are configured to deduce the intended recipient from the mail headers when a mailbox contains mail for several users, typically all users in a domain. Used alone, this is not foolproof as there are circumstances when a single unique recipient cannot be deduced from the mail headers alone. Used in conjunction with an appropriately configured <alllocal> account, it is always possible to deduce the intended recipient when the recipient is a James user.

fetchmail Configuration Parameters

The fetchmail configuration parameters are part of the James configuration, whose base file is `config.xml`. For clarity and flexibility, the fetchmail configuration parameters are stored in the file `james-fetchmail.xml`, which is referenced within `config.xml`.

The configuration parameters are described below.

fetchmail

The configuration block delimited by the fetchmail tag controls fetchmail.

The tag has these attributes:

enabled
  A boolean. If "true", the fetch tasks will be run periodically. If "false", no fetch tasks will be run. The default is "false".

The tag has these child tags (minimum cardinality, maximum cardinality):

- fetch (0, *)

<fetchmail enabled="true"> ... </fetchmail>
fetch

The fetch tag defines a fetch task to be run periodically. Fetch tasks run concurrently.

The tag has these attributes:

name
  A string uniquely identifying the fetch task.

The tag has these child tags (minimum cardinality, maximum cardinality):

- accounts (1, 1)
- blacklist (1, 1)
- defaultdomain (0, 1)
- fetchall (1, 1)
- fetched (1, 1)
- host (1, 1)
- interval (1, 1)
- javaMailFolderName (1, 1)
- javaMailProperties (0, 1)
- javaMailProviderName (1, 1)
- maxmessagesize (0, 1)
- recipientnotfound (1, 1)
- recursesubfolders (1, 1)
- remoteReceivedHeader (0, 1)
- remoterecipient (1, 1)
- undeliverable (1, 1)
- userundefined (1, 1)

<fetch name="mydomain.com"> ... </fetch>
accounts

The accounts tag declares the accounts from which mail will be fetched by the fetch task. Accounts run concurrently.

The tag has these child tags (minimum cardinality, maximum cardinality):

- account (0, *)
- alllocal (0, 1)

<accounts> ... </accounts>
blacklist

The blacklist tag declares a list of recipient addresses for whom mail will be rejected and what happens to the rejected mail.

The tag value is a tab, comma or space delimited list of recipient addresses, eg: `wibble@mydomain.com, flobble@mydomain.com`.

The tag has these attributes:

reject
  A boolean. If "true", mail for recipients in the blacklist will not be injected into the James input spool. If "false", mail for recipients in the blacklist will be injected into the James input spool with the Mail Attribute `org.apache.james.fetchmail.isBlacklistedRecipient` added to the mail.
leaveonserver
  A boolean. If "true", mail for recipients in the blacklist will be left on the server. If "false", mail for recipients in the blacklist will be marked for deletion.
markseen
  A boolean. If "true", mail for recipients in the blacklist will be marked as seen on the server. If "false", mail for recipients in the blacklist will not be marked as seen.

<blacklist reject="true" leaveonserver="true" markseen="true"> wibble@mydomain.com, flobble@mydomain.com </blacklist>
defaultdomain

The defaultdomain tag declares the domain name to be appended to the `From:` header of a mail that has a valid user part but is missing the domain part.

If not specified, the default behaviour is to append the canonical host name of the James server.

The tag value is the name of the server to append. The name must be a server declared in the servernames tag of the James block in the configuration or the name `localhost`.

<defaultdomain> mydomain.com </defaultdomain>
fetchall

The fetchall tag declares if all mail should be fetched from the server, or just unseen mail.

The tag value is a boolean. If true, all mail is fetched. If false, only unseen mail is fetched.

<fetchall>false</fetchall>
fetched

The fetched tag declares what will happen to mail on the external server that is successfully injected into the James input spool.

The tag has these attributes:

leaveonserver
  A boolean. If "true", mail injected into the James input spool will be left on the server. If "false", mail injected into the James input spool will be marked for deletion.
markseen
  A boolean. If "true", mail injected into the James input spool will be marked as seen on the server. If "false", mail injected into the James input spool will not be marked as seen.

<fetched leaveonserver="true" markseen="true"/>
host

The host tag declares the IP address of the external server from which mail is fetched.

The tag value is the DNS name or IP address literal of the external server.

<host>pop3.server.com</host>
interval

The interval tag declares the period between invocations of the fetch tasks. If a fetch task is still active from a previous invocation when the period expires, the new invocation is skipped over.

The tag value is an integer representing the number of milliseconds to elapse between invocations of the fetch tasks.

<interval>60000</interval>
javaMailFolderName

The javaMailFolderName tag declares the name of the root folder on the external server from which mail is fetched.

The tag value is the cAsE-sEnSiTiVe name of the root folder on the external server from which mail is fetched. For POP3 servers this is always `INBOX`.

<javaMailFolderName>INBOX</javaMailFolderName>
javaMailProperties

The javaMailProperties tag declares the properties to be applied to the JavaMail Session used by the fetch task. These override the properties answered by `System.getProperties()`. Many JavaMail properties are specific to the JavaMail Provider selected by the javaMailProviderName tag.

Relying on the default values selected by the Provider can be inappropriate. For instance, the default connection and I/O timeout values of infinite for the default IMAP and POP3 Providers is rarely what is required. Consult the documentation of the Provider for details and options.

Documentation for the default Provider for IMAP is located here.

Documentation for the default Provider for POP3 is located here.

Details of how to change a Provider are located here.

The tag has these child tags (minimum cardinality, maximum cardinality):

- property (0, *)

<javaMailProperties> ... </javaMailProperties>
javaMailProviderName

The javaMailProviderName tag selects the JavaMail protocol Provider used to interact with the external server.

The tag value is the name of a JavaMail supported protocol, such as `pop3` or `imap`. The name is used to select the default Provider for the protocol.

<javaMailProviderName>pop3</javaMailProviderName>
maxmessagesize

The maxmessagesize tag declares the maximum permitted message size for messages injected into the James input spool and what happens to fetched messages that exceed this size.

The tag has these attributes:

limit
  An integer. The maximum message size expressed in Kilobytes. If 0, there is no limit.
reject
  A boolean. If "true", mail whose message size exceeds the maximum permitted size will not be injected into the James input spool. If "false", mail whose message size exceeds the maximum permitted size will have its contents removed, an explanatory error message and the Mail Attribute `org.apache.james.fetchmail.isMaxMessageSizeExceeded` added prior to injection into the James input spool, (see below for the location of an example).
leaveonserver
  A boolean. If "true", mail whose message size exceeds the maximum permitted size will be left on the server. If "false", mail whose message size exceeds the maximum permitted size will be marked for deletion.
markseen
  A boolean. If "true", mail whose message size exceeds the maximum permitted size will be marked as seen on the server. If "false", mail whose message size exceeds the maximum permitted size will not be marked as seen.

<maxmessagesize limit="4096" reject="false" leaveonserver="false" markseen="false"/>

An example configuration using James mailet processing to bounce fetched messages that exceed the maximum permitted size can be found in the file `$PHOENIX_HOME/apps/james/conf/samples/fetchmail/maxMessageSize.xml`.

recipientnotfound

The recipientnotfound tag declares what happens to mail for which a sole intended recipient cannot be found when attempting to determine the recipient from the mail headers.

In configurations with more than one account per fetch task, processing of matched mail can be deferred to the next run of the fetch task. This gives other accounts that may be able to determine a sole intended recipient an opportunity to do so before recipientnotfound processing is invoked.

The tag has these attributes:

defer
  A boolean. If "true", mail for which a sole intended recipient cannot be determined is left unprocessed until the next run of the fetch task. If "false", mail for which a sole intended recipient cannot be determined is processed immediately.
reject
  A boolean. If "true", mail for which a sole intended recipient cannot be determined will not be injected into the James input spool. If "false", mail for which a sole intended recipient cannot be determined will be injected into the James input spool using the recipient attribute of the current account and with the Mail Attribute `org.apache.james.fetchmail.isRecipientNotFound` added to the mail.
leaveonserver
  A boolean. If "true", mail for which a sole intended recipient cannot be determined will be left on the server. If "false", mail for which a sole intended recipient cannot be determined will be marked for deletion.
markseen
  A boolean. If "true", mail for which a sole intended recipient cannot be determined will be marked as seen on the server. If "false", mail for which a sole intended recipient cannot be determined will not be marked as seen.

<recipientnotfound defer="true" reject="true" leaveonserver="true" markseen="true"/>
recursesubfolders

The recursesubfolders tag declares if mail should be fetched from sub-folders of the root folder, or just the root folder.

The tag value is a boolean. If true, mail is fetched from the root folder and its subfolders. If false, mail is fetched from just the root folder.

<recursesubfolders>false</recursesubfolders>
remoteReceivedHeader

The remoteReceivedHeader tag declares the zero based index of the RFC2822 compliant RECEIVED header used to determine the address and host name of the remote MTA that sent a fetched message and what happens to messages when the specified header is invalid.

Typically, the first (index = 0) RECEIVED header is for the local MTA that delivered the message to the message store and the second (index = 1) RECEIVED header is for the remote MTA that delivered the message to the local MTA. When this configuration applies, the remoteReceivedHeaderIndex should be set to 1.

To verify the correct setting, examine the RECEIVED headers for messages delivered to the configured message store and locate the first one containing a remote domain in the'from' field. Remembering that zero based indexing is used, if this the second header, use an index of 1, if this is the third header, use an index of 2, and so forth.

Matchers such as InSpammerBlacklist use the remote address and/or remote host name to identify illegitimate remote MTAs. If you do not use such matchers, the remoteReceivedHeaderIndex tag may be omitted or the default index value of -1 can be specified. This causes the remote address to be set to `127.0.0.1` and the remote host name to be set to `localhost`. Matchers almost always considered these values to be legitimate.

The tag has these attributes:

index
  An integer whose meaning is described above.
reject
  A boolean. If "true", mail whose specified recieved header is invalid will not be injected into the James input spool. If "false", mail whose specified recieved header is invalid will be injected into the James input spool with the Mail Attribute `org.apache.james.fetchmail.isInvalidReceivedHeader` added to the mail, the remote address set to `127.0.0.1` and the remote host name set to `localhost`.
leaveonserver
  A boolean. If "true", mail whose specified recieved header is invalid will be left on the server. If "false", mail whose specified recieved header is invalid will be marked for deletion.
markseen
  A boolean. If "true", mail whose specified recieved header is invalid will be marked as seen on the server. If "false", mail whose specified recieved header is invalid will not be marked as seen.

<remoteReceivedHeader index="1" reject="true" leaveonserver="true" markseen="true"/>

An example configuration using James mailet processing to notify the postmaster of fetched messages that contain an invalid Received header can be found in the file `$PHOENIX_HOME/apps/james/conf/samples/fetchmail/remoteReceivedHeader.xml`.

remoterecipient

The remoterecipient tag declares what happens to mail for which the domain part of the recipient is remote. A domain is remote if it is not a server declared in the servernames tag of the James block in the configuration.

The tag has these attributes:

reject
  A boolean. If "true", mail for remote recipients will not be injected into the James input spool. If "false", mail for remote recipients will be injected into the James input spool with the Mail Attribute `org.apache.james.fetchmail.isRemoteRecipient` added to the mail.
leaveonserver
  A boolean. If "true", mail for remote recipients will be left on the server. If "false", mail for remote recipients will be marked for deletion.
markseen
  A boolean. If "true", mail for remote recipients will be marked as seen on the server. If "false", mail for remote recipients will not be marked as seen.

<remoterecipient reject="true" leaveonserver="true" markseen="true"/>
undeliverable

The undeliverable tag declares what happens to mail that cannot be delivered.

The tag has these attributes:

leaveonserver
  A boolean. If "true", mail that cannot be delivered will be left on the server. If "false", mail that cannot be delivered will be marked for deletion.
markseen
  A boolean. If "true", mail for that cannot be delivered will be marked as seen on the server. If "false", mail that cannot be delivered will not be marked as seen.

<undeliverable leaveonserver="true" markseen="true"/>
userundefined

The userundefined tag declares what happens to mail for which the recipient is not defined as a James user.

The tag has these attributes:

reject
  A boolean. If "true", mail for recipients who are not defined as James users will not be injected into the James input spool. If "false", mail for recipients who are not defined as James users will be injected into the James input spool with the Mail Attribute `org.apache.james.fetchmail.isUserUndefined` added to the mail.
leaveonserver
  A boolean. If "true", mail for recipients who are not defined as James users will be left on the server. If "false", mail for recipients who are not defined as James users will be marked for deletion.
markseen
  A boolean. If "true", mail for recipients who are not defined as James users will be marked as seen on the server. If "false", mail for recipients who are not defined as James users will not be marked as seen.

<userundefined reject="true" leaveonserver="true" markseen="true"/>
account

The account tag declares an account on the external server from which mail should be fetched.

The tag has these attributes:

user
  The string to be passed as the user when connecting to the external server.
password
  The string to be passed as the password when connecting to the external server.
recipient
  The recipient to whom messages will be delivered when the intended recipient cannot be determined or when the intended recipient is to be ignored.
ignorercpt-header
  A boolean. If "true", mail is always delivered to the recipient declared in the recipient attribute above. If "false", the intended recipient is determined from the mail headers or the process declared by the recipientnotfound tag.

<account user="myaccount" password="mypassword" recipient="user@localhost" ignorercpt-header="true"/>
alllocal

The alllocal tag declares the parameters to be applied to dynamic accounts. The set of dynamic accounts is refreshed each time the fetch task runs by combining the alllocal tag attributes with each of the currently defined James users to create an account for every James user.

The tag has these attributes:

userprefix
  The string to be added before the James user when constructing the string passed as the user when connecting to the external server.
usersuffix
  The string to be added after the James user when constructing the string passed as the user when connecting to the external server.
password
  The string to be passed as the password when connecting to the external server.
recipientprefix
  The string to be added before the James user when constructing the recipient to whom messages will be delivered when the intended recipient cannot be determined or when the intended recipient is to be ignored.
recipientsuffix
  The string to be added after the James user when constructing the recipient to whom messages will be delivered when the intended recipient cannot be determined or when the intended recipient is to be ignored.
ignorercpt-header
  A boolean. If "true", mail is always delivered to the recipient constructed from the recipientprefix and recipientsuffix attributes above and the James user. If "false", the intended recipient is determined from the mail headers or the process declared by the recipientnotfound tag.

<alllocal userprefix="" usersuffix="@external.domain.com" password="mypassword" recipientprefix="" recipientsuffix="@mydomain.com" ignorercpt-header="true"/>
property

The property tag declares a name/value pair.

The tag has these attributes:

name
  The name of the property.
value
  The value of the property.

<property name="mail.pop3.connectiontimeout" value="180000"/>
fetchmail Examples

Full sources to the examples discussed below can be found in the directory `$PHOENIX_HOME/apps/james/conf/samples/fetchmail`.

One Account, One User

When all mail for an account is to be delivered to a single user, configure each account to ignore the recipient in the mail headers and deliver to the specified recipient. The accounts block looks like this:

<accounts> <account user="user1@external.domain.com" password="password1" recipient="user1@localhost" ignorercpt-header="true"/> <account user="user2@external.domain.com" password="password2" recipient="user2@localhost" ignorercpt-header="true"/> <account user="user3@external.domain.com" password="password3" recipient="user3@localhost" ignorercpt-header="true"/> </accounts>
One Account, Many Users

When an account contains mail to be delivered to many users, configure each account to determine the recipient from the mail headers and deliver to that user. The accounts block looks like this:

<accounts> <account user="global@external.domain.com" password="password" recipient="fetchmail@localhost" ignorercpt-header="false"/> </accounts>

The recipientnotfound tag is used to declare what happens when the recipient cannot be determined from the mail headers. In the example below, mail is injected into the spool using the recipient declared in the account tag:

<recipientnotfound defer="false" reject="false" leaveonserver="false" markseen="false"/>
One Account, One User - Dynamic

When an external server supports virtual mailboxes, fetchmail's dynamic account facility can be used. This greatly simplifies user configuration as the fetchmail accounts for users are automatically synchronized with those defined in the James user repository. This guarantees that mail for all local users will be fetched and delivered.

Currently, there is a limitation that all virtual accounts and the global account must share the same password.

The alllocal tag declares the parameters for the dynamic accounts. The accounts block below will deliver mail for `user1@external.domain.com` to `user1@localhost`, `user2@external.domain.com` to `user2@localhost`, `userZ@external.domain.com` to `userZ@localhost` etc.:

<accounts> <alllocal userprefix="" usersuffix="@external.domain.com" password="mypassword" recipientprefix="" recipientsuffix="@localhost" ignorercpt-header="true"/> </accounts>
One Account, Many Users - Dynamic

The One Account, One User - Dynamic example guarantees delivery of mail for all local users, but leaves other mail on the external server unprocessed. The One Account, Many Users example processes all mail on the external server, but cannot guarantee delivery to the intended recipient. By combining the two, it is possible to guarantee the delivery of mail for all local users and process all mail.

In the snippet below, the alllocal tag declares dynamic accounts for all local users and the account tag configures an account to fetch all mail.

The recipientnotfound tag rejects mail for which a recipient cannot be determined. By the time this processing is activated, the dynamic accounts will have processed mail for all local users, so the mail can only be mail for non-local users or newly arrived mail for local users. It is not possible to know which, but we want to leave mail for local users to be dealt with by the dynamic accounts. The next time the dynamic accounts run any newly arrived mail for local users will be processed. The remainder will be for non-local users and can now be safely dealt with.

The `<recipientnotfound defer="true"` attribute enables deferal of the processing of messages for which the recipient cannot be determined to the next iteration of the fetch task, and is used here. The relevant tags are:

<accounts> <alllocal userprefix="" usersuffix="@external.domain.com" password="mypassword" recipientprefix="" recipientsuffix="@localhost" ignorercpt-header="true"/> <account user="global@external.domain.com" password="password" recipient="fetchmail@localhost" ignorercpt-header="false"/> </accounts> <recipientnotfound defer="true" reject="true" leaveonserver="true" markseen="true"/>
fetchmail Caveats

These are some things to be aware of when using fetchmail:

- As noted in the One Account, One User - Dynamic example, all virtual accounts and the global account must share the same password. A future version might associate each James user to a set of account credentials.
- When using dynamic accounts, an account is generated and an attempt made to fetch mail for all James users defined to James even if there is no such mailbox on the server. This is inefficient but not fatal. The solution is the same as described above.
- When using dynamic accounts, as described in the One Account, Many Users - Dynamic example, the user name used to fetch the mail for all accounts must not be defined as a James user. If it is, a dynamic account will be generated for it and fetch all the mail before the account declared to process mail for all users has an opportunity to run!
- The now deprecated fetchPOP interacted with the `FetchedFrom` matcher to detect mail injected by fetchPOP. This will not work with fetchmail. Compared to fetchPOP, there are far fewer occasions when mail injected by fetchmail requires special processing. When it does, use the HasMailAttribute matcher to match the attribute named `org.apache.james.fetchmail.taskName` to detect all mail injected by fetchmail. To detect mail injected by a specific fetch task, use one of the HasMailAttributeWithValue matchers to match on the attribute name and the attribute value. The attribute value is the name of the fetch task that injected the mail.
- The POP3 protocol does not enforce support of any of the Flags associated with messages other than DELETED. This means that `markseen="true"` will most likely have no effect and therefore, the fetchall tag will be inoperative. In this situation, the only way to avoid repeatedly fetching the same mail is to delete it from the server using `leaveonserver="false"/>`.