PPoossttffiixx PPoossttssccrreeeenn HHoowwttoo

-------------------------------------------------------------------------------

IInnttrroodduuccttiioonn

The Postfix postscreen(8) server performs triage on multiple inbound SMTP
connections in parallel. While a single postscreen(8) process keeps zombies
away from Postfix SMTP server processes, more Postfix SMTP server processes
remain available for legitimate clients.

By doing these checks in a single postscreen(8) process, Postfix can avoid
wasting one SMTP server process per zombie. A side benefit of postscreen(8)'s
DNSBL lookups is that DNS records will already be cached before the Postfix
SMTP server looks them up later.

Topics in this document:

  * Introduction
  * The basic idea behind postscreen(8)
  * General operation
  * Quick tests before everything else
  * Tests before the 220 SMTP server greeting
  * Tests after the 220 SMTP server greeting
  * Other errors
  * When all tests succeed
  * Configuring the postscreen(8) service
  * Historical notes and credits

TThhee bbaassiicc iiddeeaa bbeehhiinndd ppoossttssccrreeeenn((88))

Most email is spam, and most spam is sent out by zombies (malware on
compromised end-user computers). Wietse expects that the zombie problem will
get worse before things improve, if ever. Without a tool like postscreen(8)
that keeps the zombies away, Postfix would be spending most of its resources
not receiving email.

The main challenge for postscreen(8) is to make an is-it-a-zombie decision
based on a single measurement. This is necessary because many zombies avoid
spamming the same site repeatedly, in an attempt to fly under the radar. Once
postscreen(8) decides that a client is not-a-zombie, it whitelists the client
temporarily to avoid further delays for legitimate mail.

Zombies have challenges too: they have only a limited amount of time to deliver
spam before their IP address becomes blacklisted. To speed up spam deliveries,
zombies make compromises in their SMTP protocol implementation. For example,
they speak before their turn, or they ignore responses from SMTP servers and
continue sending mail even when the server tells them to go away.

postscreen(8) uses a variety of measurements to recognize zombies. First,
postscreen(8) determines if the remote SMTP client IP address is blacklisted.
Second, postscreen(8) looks for protocol compromises that are made to speed up
delivery. The results of such measurements don't change with each delivery
attempt, and are therefore good for making an is-it-a-zombie decision based on
a single measurement.

postscreen(8) does not inspect message content. Message content can vary widely
with each delivery attempt, especially with clients that (also) send legitimate
email. Content is therefore not good for making an is-it-a-zombie decision
based on a single measurement, and that is the problem that postscreen(8) is
focused on.

GGeenneerraall ooppeerraattiioonn

The postscreen(8) triage process involves a number of tests, in the order as
described below. Some tests introduce a delay of a few seconds. Once a client
passes all tests, its IP address is temporarily excluded from any tests,
typically 24 hours for simple tests or 1 week for complex tests. This minimizes
the impact of the tests on legitimate mail clients.

After logging its findings, postscreen(8) by default hands off all connections
to a Postfix SMTP server process. This mode is useful for non-destructive
testing.

In a typical production setting, postscreen(8) is configured to reject mail
from clients that fail one or more tests, after logging the helo, sender and
recipient information.

Note: postscreen(8) is not an SMTP proxy; this is intentional. The purpose is
to keep zombies away from Postfix, with minimal overhead for legitimate
clients.

QQuuiicckk tteessttss bbeeffoorree eevveerryytthhiinngg eellssee

Before engaging in SMTP-level tests. postscreen(8) queries a number of local
black and whitelists. These tests speed up the handling of known clients.

  * Permanent whitelist test
  * Permanent blacklist test
  * Temporary whitelist test

PPeerrmmaanneenntt wwhhiitteelliisstt tteesstt

The postscreen_whitelist_networks parameter (default: $mynetworks) specifies a
permanent whitelist for SMTP client IP addresses. When the SMTP client address
matches the permanent whitelist, this is logged as:

    WWHHIITTEELLIISSTTEEDD address

The action is not configurable: immediately hand off the connection to a
Postfix SMTP server process.

PPeerrmmaanneenntt bbllaacckklliisstt tteesstt

The postscreen_blacklist_networks parameter (default: empty) specifies a
permanent blacklist for SMTP client IP addresses. The address syntax is as with
mynetworks. When the SMTP client address matches the permanent blacklist,
postscreen(8) logs this as:

    BBLLAACCKKLLIISSTTEEDD address

The postscreen_blacklist_action parameter specifies the action that is taken
next. See "When tests fail before the 220 SMTP server greeting" below.

TTeemmppoorraarryy wwhhiitteelliisstt tteesstt

The postscreen(8) daemon maintains a temporary whitelist for SMTP client IP
addresses that have passed all the tests described below. The
postscreen_cache_map parameter specifies the location of the temporary
whitelist. The temporary whitelist is not used for SMTP client addresses that
appear on the permanent blacklist or whitelist.

When the SMTP client address appears on the temporary whitelist, postscreen(8)
logs this as:

    PPAASSSS OOLLDD address

The action is not configurable: immediately hand off the connection to a
Postfix SMTP server process. The client is excluded from further tests until
its temporary whitelist entry expires, as controlled with the postscreen_*_ttl
parameters. Expired entries are silently renewed if possible.

TTeessttss bbeeffoorree tthhee 222200 SSMMTTPP sseerrvveerr ggrreeeettiinngg

The postscreen_greet_wait parameter specifies a short time interval before the
"220 text..." server greeting, where postscreen(8) can run a number of tests in
parallel.

When a good client passes these tests, and no "deep protocol tests" are
configured, postscreen(8) adds the client to the temporary whitelist and hands
off the "live" connection to a Postfix SMTP server process. The client can then
continue as if postscreen(8) never even existed (except of course for the short
postscreen_greet_wait delay).

  * Pregreet test
  * DNS Blocklist test
  * When tests fail before the 220 SMTP server greeting

PPrreeggrreeeett tteesstt

The SMTP protocol is a classic example of a protocol where the server speaks
before the client. postscreen(8) detects zombies that are in a hurry and that
speak before their turn. This test is enabled by default.

The postscreen_greet_banner parameter specifies the text portion of a "220-
text..." teaser banner (default: $smtpd_banner). Note that this becomes the
first part of a multi-line server greeting. The postscreen(8) daemon sends this
before the postscreen_greet_wait timer is started. The purpose of the teaser
banner is to confuse zombies so that they speak before their turn. It has no
effect on SMTP clients that correctly implement the protocol.

To avoid problems with poorly-implemented SMTP engines in network appliances or
network testing tools, either exclude them from all tests with the
postscreen_whitelist_networks feature or else specify an empty teaser banner:

/usr/local/etc/postfix/main.cf:
    # Exclude broken clients by whitelisting. $mynetworks is default.
    postscreen_whitelist_networks = $mynetworks, 192.168.254.0/24

/usr/local/etc/postfix/main.cf:
    # Disable the teaser banner (try whitelisting first if you can).
    postscreen_greet_banner =

When an SMTP client sends a command before the postscreen_greet_wait time has
elapsed, postscreen(8) logs this as:

    PPRREEGGRREEEETT count aafftteerr time ffrroomm address text...

Translation: the client at address sent count bytes before its turn to speak.
This happened time seconds after the postscreen_greet_wait timer was started.
The text is what the client sent (truncated to 100 bytes, and with non-
printable characters replaced with "?").

The postscreen_greet_action parameter specifies the action that is taken next.
See "When tests fail before the 220 SMTP server greeting" below.

DDNNSS BBlloocckklliisstt tteesstt

The postscreen_dnsbl_sites parameter (default: empty) specifies a list of DNS
blocklist servers with optional filters and weight factors. These servers will
be queried in parallel with the reverse client IP address. This test is
disabled by default.

    CAUTION: when postscreen rejects mail, it's SMTP reply contains the DNSBL
    domain name. Use the postscreen_dnsbl_reply_map feature to hide "password"
    information in DNSBL domain names.

When the postscreen_greet_wait time has elapsed, and the combined DNSBL score
is equal to or greater than the postscreen_dnsbl_threshold parameter value,
postscreen(8) logs this as:

    DDNNSSBBLL rraannkk count ffoorr address

Translation: the SMTP client at address has a combined DNSBL score of count.

The postscreen_dnsbl_action parameter specifies the action that is taken when
the combined DNSBL score is equal to or greater than the threshold. See "When
tests fail before the 220 SMTP server greeting" below.

WWhheenn tteessttss ffaaiill bbeeffoorree tthhee 222200 SSMMTTPP sseerrvveerr ggrreeeettiinngg

When the client address matches the permanent blacklist, or when the client
fails the pregreet or DNSBL tests, the action is specified with
postscreen_blacklist_action, postscreen_greet_action, or
postscreen_dnsbl_action, respectively.

iiggnnoorree (default)
    Ignore the failure of this test. Allow other tests to complete. Repeat this
    test the next time the client connects. This option is useful for testing
    and collecting statistics without blocking mail.
eennffoorrccee
    Allow other tests to complete. Reject attempts to deliver mail with a 550
    SMTP reply, and log the helo/sender/recipient information. Repeat this test
    the next time the client connects.
ddrroopp
    Drop the connection immediately with a 521 SMTP reply. Repeat this test the
    next time the client connects.

TTeessttss aafftteerr tthhee 222200 SSMMTTPP sseerrvveerr ggrreeeettiinngg

In this phase of the protocol, postscreen(8) implements a number of "deep
protocol" tests. These tests use an SMTP protocol engine that is built into the
postscreen(8) server.

Important note: deep protocol tests are disabled by default. They are more
intrusive than the pregreet and DNSBL tests, and they have limitations as
discussed next.

  * When a good client passes the deep protocol tests, postscreen(8) adds the
    client to the temporary whitelist but it cannot hand off the "live"
    connection to a Postfix SMTP server process in the middle of the session.
    Instead, postscreen(8) defers mail delivery attempts with a 4XX status,
    logs the helo/sender/recipient information, and waits for the client to
    disconnect.

    The next time the client connects it will be allowed to talk to a Postfix
    SMTP server process to deliver its mail. To minimize the impact of this
    limitation, postscreen(8) gives deep protocol tests a relatively long
    expiration time.

  * postscreen(8)'s built-in SMTP engine does not implement the AUTH, STARTTLS,
    XCLIENT, and XFORWARD features. STARTTLS and AUTH support may be added in a
    future version.

End-user clients should connect directly to the submission service. Other
systems that require the above features should directly connect to a Postfix
SMTP server, or they should be placed on the postscreen(8) whitelist.

  * Command pipelining test
  * Non-SMTP command test
  * Bare newline test
  * When tests fail after the 220 SMTP server greeting

CCoommmmaanndd ppiippeelliinniinngg tteesstt

By default, SMTP is a half-duplex protocol: the sender and receiver send one
command and one response at a time. Unlike the Postfix SMTP server, postscreen
(8) does not announce support for ESMTP command pipelining. Therefore, clients
are not allowed to send multiple commands. postscreen(8)'s deep protocol test
for this is disabled by default.

With "postscreen_pipelining_enable = yes", postscreen(8) detects zombies that
send multiple commands, instead of sending one command and waiting for the
server to reply.

This test is opportunistically enabled when postscreen(8) has to use the built-
in SMTP engine anyway. This is to make postscreen(8) logging more informative.

When a client sends multiple commands, postscreen(8) logs this as:

    CCOOMMMMAANNDD PPIIPPEELLIINNIINNGG aafftteerr time ffrroomm address

Translation: the SMTP client at address sent multiple SMTP commands, instead of
sending one command and then waiting for the server to reply. This happened
time seconds after the "220 " server greeting was sent.

The postscreen_pipelining_action parameter specifies the action that is taken
next. See "When tests fail after the 220 SMTP server greeting" below.

NNoonn--SSMMTTPP ccoommmmaanndd tteesstt

Some spambots send their mail through open proxies. A symptom of this is the
usage of commands such as CONNECT and other non-SMTP commands. Just like the
Postfix SMTP server's smtpd_forbidden_commands feature, postscreen(8) has an
equivalent postscreen_forbidden_commands feature to block these clients.
postscreen(8)'s deep protocol test for this is disabled by default.

With "postscreen_non_smtp_command_enable = yes", postscreen(8) detects zombies
that send commands specified with the postscreen_forbidden_commands parameter.
This also detects commands with the syntax of a message header label. The
latter is a symptom that the client is sending message content after ignoring
all the responses from postscreen(8) that reject mail.

This test is opportunistically enabled when postscreen(8) has to use the built-
in SMTP engine anyway. This is to make postscreen(8) logging more informative.

When a client sends non-SMTP commands, postscreen(8) logs this as:

    NNOONN--SSMMTTPP CCOOMMMMAANNDD ffrroomm address command

Translation: the SMTP client at address sent a command that matches the
postscreen_forbidden_commands parameter, or that has the syntax of a message
header label.

The postscreen_non_smtp_command_action parameter specifies the action that is
taken next. See "When tests fail after the 220 SMTP server greeting" below.

BBaarree nneewwlliinnee tteesstt

SMTP is a line-oriented protocol: lines have a limited length, and are
terminated with <CR><LF>. Lines ending in a "bare" <LF>, that is newline not
preceded by carriage return, are not allowed in SMTP. postscreen(8)'s deep
protocol test for this is disabled by default.

With "postscreen_bare_newline_enable = yes", postscreen(8) detects clients that
send lines ending in bare newline characters.

This test is opportunistically enabled when postscreen(8) has to use the built-
in SMTP engine anyway. This is to make postscreen(8) logging more informative.

When a client sends bare newline characters, postscreen(8) logs this as:

    BBAARREE NNEEWWLLIINNEE ffrroomm address

Translation: the SMTP client at address sent a bare newline character, that is
newline not preceded by carriage return.

The postscreen_bare_newline_action parameter specifies the action that is taken
next. See "When tests fail after the 220 SMTP server greeting" below.

WWhheenn tteessttss ffaaiill aafftteerr tthhee 222200 SSMMTTPP sseerrvveerr ggrreeeettiinngg

When the client fails the pipelining, non-SMTP command or bare newline tests,
the action is specified with postscreen_pipelining_action,
postscreen_non_smtp_command_action or postscreen_bare_newline_action,
respectively.

iiggnnoorree (default for bare newline)
    Ignore the failure of this test. Allow other tests to complete. Do NOT
    repeat this test before the result from some other test expires. This
    option is useful for testing and collecting statistics without blocking
    mail permanently.
eennffoorrccee (default for pipelining)
    Allow other tests to complete. Reject attempts to deliver mail with a 550
    SMTP reply, and log the helo/sender/recipient information. Repeat this test
    the next time the client connects.
ddrroopp (default for non-SMTP commands)
    Drop the connection immediately with a 521 SMTP reply. Repeat this test the
    next time the client connects. This action is compatible with the Postfix
    SMTP server's smtpd_forbidden_commands feature.

OOtthheerr eerrrroorrss

When an SMTP client hangs up unexpectedly during any tests, postscreen(8) logs
this as:

    HHAANNGGUUPP aafftteerr time ffrroomm address iinn test name

Translation: the SMTP client at address disconnected unexpectedly, time seconds
after the start of the test named test name.

The following errors are reported by the built-in SMTP engine. This engine
never accepts mail, therefore it has per-session limits on the number of
commands and on the session length.

    CCOOMMMMAANNDD TTIIMMEE LLIIMMIITT ffrroomm address

Translation: the SMTP client at address reached the per-command time limit as
specified with the postscreen_command_time_limit parameter. The session is
terminated immediately.

    CCOOMMMMAANNDD CCOOUUNNTT LLIIMMIITT ffrroomm address

Translation: the SMTP client at address reached the per-session command count
limit as specified with the postscreen_command_count_limit parameter. The
session is terminated immediately.

    CCOOMMMMAANNDD LLEENNGGTTHH LLIIMMIITT ffrroomm address

Translation: the SMTP client at address reached the per-command length limit,
as specified with the line_length_limit parameter. The session is terminated
immediately.

WWhheenn aallll tteessttss ssuucccceeeedd

When a new SMTP client passes all tests (i.e. it is not whitelisted via some
mechanism), postscreen(8) logs this as:

    PPAASSSS NNEEWW address

Where address is the client IP address. Then, postscreen(8) creates a temporary
whitelist entry that excludes the client IP address from further tests until
the temporary whitelist entry expires, as controlled with the postscreen_*_ttl
parameters.

When no "deep protocol tests" are configured, postscreen(8) hands off the
"live" connection to a Postfix SMTP server process. The client can then
continue as if postscreen(8) never even existed (except for the short
postscreen_greet_wait delay).

When any "deep protocol tests" are configured, postscreen(8) cannot hand off
the "live" connection to a Postfix SMTP server process in the middle of the
session. Instead, postscreen(8) defers mail delivery attempts with a 4XX
status, logs the helo/sender/recipient information, and waits for the client to
disconnect. The next time the client connects it will be allowed to talk to a
Postfix SMTP server process to deliver its mail. postscreen(8) mitigates the
impact of this limitation by giving deep protocol tests a long expiration time.

CCoonnffiigguurriinngg tthhee ppoossttssccrreeeenn((88)) sseerrvviiccee

postscreen(8) has been tested on FreeBSD [4-8] and Linux 2.[4-6] systems. It
probably needs additional work before it can be used on Solaris.

  * Turning on postscreen(8) without blocking mail
  * Blocking mail with postscreen(8)
  * Turning off postscreen(8)

TTuurrnniinngg oonn ppoossttssccrreeeenn((88)) wwiitthhoouutt bblloocckkiinngg mmaaiill

To enable the postscreen(8) service and log client information without blocking
mail:

 1. Comment out the "smtp inet ... smtpd" service in master.cf, including any
    "-o parameter=value" entries that follow.

    /usr/local/etc/postfix/master.cf:
        #smtp      inet  n       -       n       -       -       smtpd
        #    -o parameter=value ...

 2. Uncomment the new "smtpd pass ... smtpd" service in master.cf, and
    duplicate any "-o parameter=value" entries from the smtpd service that was
    commented out in step 1.

    /usr/local/etc/postfix/master.cf:
        smtpd     pass  -       -       n       -       -       smtpd
            -o parameter=value ...

 3. Uncomment the new "smtp inet ... postscreen" service in master.cf.

    /usr/local/etc/postfix/master.cf:
        smtp      inet  n       -       n       -       1       postscreen

 4. Uncomment the new "dnsblog unix ... dnsblog" service in master.cf. This
    service does DNSBL lookups for postscreen(8) and logs results.

    /usr/local/etc/postfix/master.cf:
        dnsblog   unix  -       -       n       -       0       dnsblog

 5. To enable DNSBL lookups, list some DNS blocklist sites in main.cf,
    separated by whitespace. Different sites can have different weights. For
    example:

    /usr/local/etc/postfix/main.cf:
        postscreen_dnsbl_threshold = 2
        postscreen_dnsbl_sites = zen.spamhaus.org*2
            bl.spamcop.net*1 b.barracudacentral.org*1

    Note: if your DNSBL queries have a "secret" in the domain name, you must
    censor this information from the postscreen(8) SMTP replies. For example:

    /usr/local/etc/postfix/main.cf:
        postscreen_dnsbl_reply_map = texthash:$config_directory/dnsbl_reply

    /usr/local/etc/postfix/dnsbl_reply:
        # Secret DNSBL name        Name in postscreen(8) replies
        secret.zen.spamhaus.org    zen.spamhaus.org

    The texthash: format is similar to hash: except that there is no need to
    run postmap(1) before the file can be used, and that it does not detect
    changes after the file is read. It is new with Postfix version 2.8.

 6. Read the new configuration with "postfix reload".

Notes:

  * Some postscreen(8) configuration parameters implement stress-dependent
    behavior. This is supported only when the default value is stress-dependent
    (that is, it looks like ${stress?X}${stress:Y}). Other parameters always
    evaluate as if the stress value is the empty string.

  * See "Tests before the 220 SMTP server greeting" for details about the
    logging from these postscreen(8) tests.

  * By default, postscreen(8) whitelists all clients in mynetworks. This is a
    safety feature to avoid you from getting into trouble with local users.

  * If you run Postfix 2.6 or earlier you must stop and start the master daemon
    ("postfix stop; postfix start"). This is needed because the Postfix "pass"
    master service type did not work reliably on all systems.

BBlloocckkiinngg mmaaiill wwiitthh ppoossttssccrreeeenn((88))

To use the postscreen(8) service to block mail, edit main.cf and specify one or
more of:

  * "postscreen_dnsbl_action = enforce", to reject clients that are on DNS
    blocklists, and to log the helo/sender/recipient information. With good
    DNSBLs this reduces the amount of load on Postfix SMTP servers
    dramatically.

  * "postscreen_greet_action = enforce", to reject clients that talk before
    their turn, and to log the helo/sender/recipient information. This stops
    over half of all known-to-be illegitimate connections to Wietse's mail
    server. It is backup protection for zombies that haven't yet been
    blacklisted.

  * You can also enable "deep protocol tests", but these are more intrusive
    than the pregreet or DNSBL tests.

    When a good client passes the "deep protocol tests", postscreen(8) adds the
    client to the temporary whitelist but it cannot hand off the "live"
    connection to a Postfix SMTP server process in the middle of the session.
    Instead, postscreen(8) defers mail delivery attempts with a 4XX status,
    logs the helo/sender/recipient information, and waits for the client to
    disconnect.

    When the good client comes back in a later session, it is allowed to talk
    directly to a Postfix SMTP server. See "after_220 Tests after the 220 SMTP
    server greeting above for limitations with STARTTLS, AUTH and other
    features that clients may need.

    An unexpected benefit from "deep protocol tests" is that some "good"
    clients don't return after the 4XX reply; these clients were not so good
    after all. Wietse enables "deep protocol tests" on his own internet-facing
    mail server.

  * There is also support for permanent blacklists and whitelists; see the
    description of the postscreen_whitelist_networks and
    postscreen_blacklist_networks parameters for details.

TTuurrnniinngg ooffff ppoossttssccrreeeenn((88))

To turn off postscreen(8) and handle mail directly with Postfix SMTP server
processes:

 1. Comment out the "smtp inet ... postscreen" service in master.cf, including
    any "-o parameter=value" entries that follow.

    /usr/local/etc/postfix/master.cf:
        #smtp      inet  n       -       n       -       1       postscreen
        #    -o parameter=value ...

 2. Comment out the "dnsblog unix ... dnsblog" service in master.cf.

    /usr/local/etc/postfix/master.cf:
        #dnsblog   unix  -       -       n       -       0       dnsblog

 3. Comment out the "smtpd pass ... smtpd" service in master.cf, including any
    "-o parameter=value" entries that follow.

    /usr/local/etc/postfix/master.cf:
        #smtpd     pass  -       -       n       -       -       smtpd
        #    -o parameter=value ...

 4. Uncomment the "smtp inet ... smtpd" service in master.cf, including any "-
    o parameter=value" entries that follow.

    /usr/local/etc/postfix/master.cf:
        smtp      inet  n       -       n       -       -       smtpd
            -o parameter=value ...

 5. Read the new configuration with "postfix reload".

HHiissttoorriiccaall nnootteess aanndd ccrreeddiittss

Many ideas in postscreen(8) were explored in earlier work by Michael Tokarev,
in OpenBSD spamd, and in MailChannels Traffic Control.

Wietse threw together a crude prototype with pregreet and dnsbl support in June
2009, because he needed something new for a Mailserver conference presentation
in July. Ralf Hildebrandt ran this code on several servers to collect real-
world statistics. This version used the dnsblog(8) ad-hoc DNS client program.

Wietse needed new material for a LISA conference presentation in November 2010,
so he added support for DNSBL weights and filters in August, followed by a
major code rewrite, deep protocol tests, helo/sender/recipient logging, and
stress-adaptive behavior in September. Ralf Hildebrandt ran this code on
several servers to collect real-world statistics. This version still used the
embarrassing dnsblog(8) ad-hoc DNS client program.

