Poor man's greylisting with Exim and SQLite

Jakob Hirsch, <grey@plonk.de>
created 2006-03-22, last update on 2007-07-09


*** Introduction ***

Greylisting is temporarily rejecting hosts with the assumption and hope that
spam hosts will not try again, but legit senders will, as this is a standard
mechanism of the smtp protocol. There are lots of discussions you can find
via google, so I'll skip that. It works fairly well right now, some senders
(like yahoo groups) seem to dislike temp errors, some spam senders seem to
adopt, so this is surely not the FUSSP as some people might like to see it.
Right now it's an easy and low cost measure to enhance the SNR (or lower the
load SpamAssassin puts on your system). I'd dissuade from using greylisting
blindly on every hosts, because it puts unnecessary load on innocent systems, 
delays your mail and even might cause mail loss (from broken hosts). Combining
it with DNS blacklists works very fine, though (and you probably shouldn't use
those on their own, too).

I also use greylisting to defer mail that triggers other checks, e.g.:
- hosts with dynamic IP addresses
- recipient looks like a msgid (e.g. 46912499.4060108@plonk.de, I started to
  get a lot of that lately)
- hosts without reverse DNS
- hosts where HELO/EHLO hostname does not match reverse DNS

To minimize the delay of legit mail, I have my MX running with two IP
addresses. Mail on the second one is only accepted, if the sending host 
tried to send mail to the primary address and was greylisted.

I'm using SQLite to store records, which is quite fast and effective without
the need of a full blown RDBMS server. If you have more then one MX you'll be
better off using something like MySQL or PostgreSQL (and probably using a more
sophisticated config).


*** Database setup ***

A single table is used to store the sending host, sender and recipient address
and the time of first and latest contact.

To create the database structure, run "sqlite3 /etc/exim/var/grey.sqlt" and
pipe this in:

CREATE TABLE greylist (
 host text,
 sender text,
 rcpt text,
 created integer,
 updated integer,
 PRIMARY KEY (host, sender, rcpt)
);

This scheme is far from being good design, but simple and quite sufficient for
the intended use. Note that only the first recipient for a specific
host/sender tuple is stored (for non-null senders), but the recipient is not
relevant in this case, anyway.


*** Exim config ***

I use host and sender address (recipient address is sender is <>) to
distinguish, you could argue that using only the host, some part of the host
address (e.g. /24) or the full tuple (host, sender, recipient) is
better/saner/whatever. This will obviously change the false positive/negative
ratio, but as we are doing "soft rejects" and a proper spam filter
(SpamAssassin, bogofilter, dspam etc.) should be used anyway, the effect is
not that big.


# main config

CFG=/etc/exim

# greylisting
GREY_DB = $spool_directory/var/grey.sqlt
# greylist for 14 minutes
GREY_DELAY = (14*60)

...

# acl config

# check if sending host is greylisted

acl_grey_check:

  accept
    condition = ${if def:acl_m_grey}
    condition = ${if < {0} {$acl_m_grey}}

  deny
    log_message = Greylisting delay expired
    condition = ${if def:acl_m_grey}
    condition = ${if >= {0} {$acl_m_grey}}

  # check if tuple is in database
  accept
    set acl_m_grey = ${lookup sqlite {GREY_DB \
        select created from greylist \
        where host='${quote_sqlite:$sender_host_address}' \
        and sender='${quote_sqlite:$sender_address}' \
        and (sender!='' or rcpt='${quote_sqlite:${local_part}@${domain}}') \
        limit 1} {${eval10:0${value} + GREY_DELAY - ${tod_epoch}}} }
    #condition = ${if def:acl_m_grey}
    condition = ${if < {0} {$acl_m_grey}}

  deny

# check if sending host is greylisted
# add it to greylist db if not
acl_greylist:

  accept
    acl = acl_grey_check

  # no entry -> insert
  accept
      condition = ${if !def:acl_m_grey}
      set acl_m_grey = ${eval10:GREY_DELAY}
      condition = ${lookup sqlite {GREY_DB \
          insert into greylist (host, sender, rcpt, created, updated) \
          values ('${quote_sqlite:$sender_host_address}', \
                          '${quote_sqlite:$sender_address}', \
                          '${quote_sqlite:${local_part}@${domain}}', \
                           $tod_epoch, $tod_epoch)} {1}{1}}

  # delay not reached, update entry
  accept
      condition = ${if >= {$acl_m_grey} {0}}
      condition = ${lookup sqlite {GREY_DB \
          update greylist set updated = $tod_epoch \
          where host='${quote_sqlite:$sender_host_address}' \
          and sender='${quote_sqlite:$sender_address}' \
          and rcpt='${quote_sqlite:${local_part}@${domain}}' \
      } {1}{1}}

  deny


# ...

acl_check_rcpt:

...

accept
  authenticated = *
  control       = submission/sender_retain

accept
  hosts         = +relay_from_hosts

# be very loose on the use of dnslists because we don't block.
# there are senders that don't like deferrals (like yahoo groups), so
# a manual or automatic (e.g. mxrate.com, dnswl.org) whitelist could be useful.
defer
  dnslists = ix.dnsbl.manitu.net : relays.ordb.org : sbl-xbl.spamhaus.org : \
    list.dsbl.org : multihop.dsbl.org : combined.njabl.org : bl.spamcop.net : \
    bogons.dnsiplists.completewhois.com : hijacked.dnsiplists.completewhois.com : \
    dnsbl.sorbs.net : dnsbl-1.uceprotect.net : l1.spews.dnsbl.sorbs.net
  acl = acl_greylist
  message = Delayed because $sender_host_address is listed in ${dnslist_domain}:\n\
            $dnslist_text\nIf you are not a spammer, come back in $ACL_GREY seconds
  log_message = greylisted for ${ACL_GREY}s, in $dnslist_domain
  delay = 10s


*** Maintenance ***

Run this regularly (e.g. daily) to expire hosts that didn't come back:

sqlite3 /etc/exim/var/grey.sqlt "DELETE FROM greylist WHERE created=updated AND created<strftime('%s', 'now', '-36 hours');"

It may also be a good idea to expire hosts that didn't send mail for some time
(e.g. 6 weeks):

sqlite3 /etc/exim/var/grey.sqlt "DELETE FROM greylist WHERE updated<strftime('%s', 'now', '-42 days');"