Amavis’ cool features

I guess that most people use amavisd-new together with spamassassin and for example ClamAV. Probably a few more use features like DKIM verification and signing with amavis. However, there are some features which aren’t found in the usual howtos. Here are some of them.

#1 Reading Postfix configuration files

Did you know, that Amavis can read and include Postfix hash files as well as CIDR files? If you manage postfix using files like I do you might like this because you don’t need to configure it twice.

Similar configuration snippets

My postfix configuration contains:

mynetworks = 84.200.7.144/28 127.0.0.0/8
virtual_mailbox_domains = no-uce.de, jeanbruenn.info

My amavis configuration contains:

@mynetworks = qw( 127.0.0.0/8 84.200.7.144/28 );
@local_domains_maps = ( [ "no-uce.de", "jeanbruenn.info" ] );

So whenever I change mynetworks (might happen) I will need to change it both in Amavis and Postfix) and whenever I add or remove domains (this will happen) I will again need to change it in both amavis as well as postfix.

Special considerations with mynetworks

mynetworks is a special case because in Postfix mynetworks defines who is a trusted sender. For amavis mynetworks defines who is an internal sender. At a first glance this looks very similar. Amavis’ release notes also state that both settings will likely be identical. This however, might not always be the case.

read_cidr example

# Postfix main.cf
mynetworks = cidr:/etc/postfix/mynetworks

# Amavis 50-user
@mynetworks = @{ read_cidr('/etc/postfix/mynetworks') };

# /etc/postfix/mynetworks
127.0.0.0/8		OK
84.200.7.144/28		OK

read_hash example

# Postfix main.cf
virtual_mailbox_domains = hash:/etc/postfix/virtual/domains

# Amavis 50-user.conf
@local_domains_maps = ( read_hash("/etc/postfix/virtual/domains") );

# /etc/postfix/virtual/domains
# this is a hash file so you have to execute postmap
no-uce.de		1
jeanbruenn.info		1

#2 Redis

Redis alone won’t do much. However, the next three features may use Redis. Using Redis with Amavis is pretty easy. You just need to install redis (by default it only listens to 127.0.0.1, you should however make sure that this is the case) and configure it for example like this in amavis:

#
# redis
#

@storage_redis_dsn = ( { server => '127.0.0.1:6377' } );
# 30 days. Default is 16 days.
$storage_redis_ttl = 30*24*60*60;

You may choose to use db_id to select a database in Redis, like this:

@storage_redis_dsn = ( { 
  server => '127.0.0.1:6377',
  db_id => 1
} );

You may however also work with multiple redis instances, which is my preferred way.

#3 Pen Pals

The idea of this feature is that you may reduce the spamscore if you receive an email from someone you wrote a mail to beforehand. This soft-whitelisting feature has been first implemented in 2.4.2. Redis-support has made it into Amavis in 2.8.1 (probably better support in 2.9.0).

Pre-requisites

I will cite the RELEASE-NOTES with a few useful information in the following:

both the outgoing and the incoming mail must pass through amavisd (although outgoing mail may have checks disabled or made more permissive if desired);

June 27, 2006 amavisd-new-2.4.2 release notes

Unless you’re running a relay / smarthost / gateway you just need to make sure that mails going through smtp as well as submission will be processed by amavis. I wrote a guide which uses a milter (before-queue) for inbound mail and a content_filter (after-queue) for outbound mail before.

SQL logging must be enabled (@storage_sql_dsn) and records should be kept for at least several days (some statistics (2006-11 update): 90% of replied mail (or followups) is sent within 2 weeks since previous correspondence, 40% within 24 hours, 20% within 3 hours, 10% within 30 minutes, 5% within 12 minutes);

June 27, 2006 amavisd-new-2.4.2 release notes

You may ignore this pre-requisite because we will use Redis here instead of SQL.

@mynetworks and @local_domains_maps must reflect reality, allowing amavisd to distinguish between outgoing, incoming and internal-to-internal mail;

June 27, 2006 amavisd-new-2.4.2 release notes

This one is important. So seriously make sure, that mynetworks contains the IPs which are local to amavis and local_domains_maps contains all your domains. It is important that Amavis knows what mails are in- and outbound.

the information about client IP address must be available to amavisd, i.e. Postfix XFORWARD protocol extension must be enabled, or AM.PDP+milter;

June 27, 2006 amavisd-new-2.4.2 release notes

In my Postfix setup I do use a Milter for inbound mails (so I don’t need to change something here). I do use an after-queue filter for my clients, so I need to make sure the XFORWARD extension is enabled in my master.cf:

submission inet n       -       n       -       -       smtpd
  -o content_filter=amavisfeed:[127.0.0.1]:10023
  -o syslog_name=postfix/submission
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

[..]

amavisfeed unix    -       -       n        -      2     lmtp
     -o lmtp_data_done_timeout=1200
     -o lmtp_send_xforward_command=yes
     -o disable_dns_lookups=yes
     -o max_use=20

The lmtp_send_xforward_command is set to yes, which means everything is fine. If you’re using smtp instead of lmtp there is smtp_send_xforward_command.

configuration variable $penpals_bonus_score must be set to a positive value (such as 1.0, increase to perhaps 5 or 8 after seeing that it works), zero disables the feature and is a default;

June 27, 2006 amavisd-new-2.4.2 release notes

I use the following settings for testing purposes in my 50-user.conf:

# 
# penpals
#

$penpals_bonus_score = 1;
$penpals_threshold_low = undef;
$penpals_threshold_high = undef;

$sql_clause{‘sel_penpals’} must contain a SELECT clause (which by default it does, unless overridden by an old assignment to %sql_clause in amavisd.conf);

June 27, 2006 amavisd-new-2.4.2 release notes

Not relevant due to our use of Redis.

sender/recipient address pair must exactly match recipient/sender pair of previous correspondence (except for allowed case-changes in domain part), which means that care must be taken when canonical and/or virtual mapping is performed by MTA (such as mapping between internal and external address forms) – if external address forms of local addresses are to be seen by a content filter then canonical mapping (int->ext) must be done *before* filtering and virtual mapping (ext->int) *after*; alternatively, if internal address forms are to be seen by a content filter, then canonical mapping should be done after filtering, and virtual mapping before; see README.postfix, section “TO DO ‘VIRTUAL ALIAS’ MAPPING AND OTHER POSTFIX CLEANUP PROCESSING BEFORE OR AFTER CONTENT FILTERING?” (P.S. later renamed to ‘Advanced Postfix and amavisd-new configuration’);

June 27, 2006 amavisd-new-2.4.2 release notes

The referenced documentation explains how to set up two cleanup services to achieve this behavior. Please read that document it’s really good. Exactly like in that guide I added a second cleanup service to Postfix master.cf. I do add -o syslog_name=postfix/pre-queue to it because without that I won’t notice if it works.

pre-cleanup unix    n       -       n       -       0       cleanup
  -o syslog_name=postfix/pre-cleanup
  -o virtual_alias_maps=

then I modified the existing cleanup service:

cleanup unix    n       -       n       -       0       cleanup
  -o mime_header_checks= 
  -o nested_header_checks= 
  -o body_checks= 
  -o header_checks=

Finally I modified pickup, submission and smtp to contain -o cleanup_service_name=pre-cleanup. Mind that I add -o cleanup_service_name to my smtpd service due to my use of postscreen (instead of to smtp)

smtp      inet  n       -       n       -       1       postscreen
smtpd     pass  -       -       n       -       -       smtpd
  -o smtpd_milters=${milter_amavis}
  -o cleanup_service_name=pre-cleanup
[..]

submission inet n       -       n       -       -       smtpd
  -o content_filter=amavisfeed:[127.0.0.1]:10023
  -o syslog_name=postfix/submission
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o cleanup_service_name=pre-cleanup
pickup    unix  n       -       n       60      1       pickup
  -o cleanup_service_name=pre-cleanup

Check the logs (hence my usage of syslog_name) to make sure that this works and you can both receive as well as send emails 🙂

Settings

I will cite the RELEASE-NOTES again.

mail with (unadjusted) SA score below $penpals_threshold_low (1 by default) is exempted from pen pals check to save time and lighten the load on SQL; similarly for high score spam which would not have a chance of being ‘saved’ even by a maximal pen pals bonus score;

June 27, 2006 amavisd-new-2.4.2 release notes

Due to the use of Redis instead of a heavy sql database, I believe settings this lower would be fine. Depending on your load and available resources you might still want to limit this. I keep $penpals_threshold_low at undef; so that there is no bottom limit.

$penpals_threshold_high … when (SA_score – $penpals_bonus_score > $penpals_threshold_high) pen pals lookup will not be performed to save time, as it could not influence blocking of spam even at maximal penpals bonus (age=0); usual choice for value would be kill level or tag2 level, or other reasonably high value; undef lets the threshold be ignored and is a default (useful for testing and statistics gathering);

June 27, 2006 amavisd-new-2.4.2 release notes

I set this to $sa_kill_level_deflt:

$penpals_threshold_high = $sa_kill_level_deflt;

configuration variable $penpals_bonus_score must be set to a positive value (such as 1.0, increase to perhaps 5 or 8 after seeing that it works), zero disables the feature and is a default;

[..]

$penpals_bonus_score … a maximal (positive) score value by which spam score is lowered when sender is known to have previously received mail from our local user from this mail system. Zero or undef disables pen pals lookups, and is a default.

$penpals_halflife … exponential decay time constant in seconds, defaults to 7 days; pen pal bonus is halved for each halflife period since the last mail sent by a local user to the current message’s sender;

June 27, 2006 amavisd-new-2.4.2 release notes

The value fo $penpals_halflife is fine in my opinion. The value for $penpals_bonus_score could be $sa_kill_level_deflt – 1:

$penpals_bonus_score = $sa_kill_level_deflt - 1;

So all in all I do use:

# 
# penpals
#

$penpals_bonus_score = $sa_kill_level_deflt - 1;
$penpals_threshold_low = undef;
$penpals_threshold_high = $sa_kill_level_deflt;

If you set your log_level somewhat higher (I believe it was 2, I’m using 3, though). You can check the logs to see if it’s working:

Aug 22 12:52:54 mail amavis[3555]: (03555-16) penpals: (redis) found (3,2) age 0 12:23:51, refs: weRr6tCck-FQ (44631 s) rid=2
Aug 22 12:52:54 mail amavis[3555]: (03555-16) penpals: adj.bonus 0.950, age 0 12:23:51 (44631), SA score 0.817, <xxx@gmail.com> replying to <himself@jeanbruenn.info>, ref mail_id: weRr6tCck-FQ

#4 IP address reputation

Amavisd-new with version 2.9.0 has a new feature called IP address reputation.

When a Redis storage back-end is enabled, besides the existing pen pals functionality, it now also offers information updating and retrieval on IP address reputation. This function is enabled by default when @storage_redis_dsn is nonempty, but can be disabled by setting $enable_ip_repu to false (to 0 or undef), per policy bank if necessary.

For each mail message a list of public IP addresses (IPv4 or IPv6) is collected from its ‘Received’ trace header fields in a mail header section. A redis server maintains a database of each IP address encountered. For each IP address an entry carries a set of counters corresponding to the number of mail messages encountered in the past having this IP address in a trace header. These counters show: a number of spam messages, a number of ham messages, a number of banned or infected messages, and a total number of messages. Also a timestamp of the first and last encounter is kept. Each entry (a set of counters) is subject to automatic expiry, so that infrequently encountered IP addresses are eventually automatically purged from a database by a redis server itself.

May 9, 2014 amavisd-new-2.9.0 release notes

When using this feature whitelist your mailservers through:

@ip_repu_ignore_networks =
      qw( 192.0.2.44 192.0.2.45 198.51.100.0/24 2001:db8::1:25 );

If you followed my Post about how I configured amavis, I do disable IP reputation checks for my internal mailers / relays by adding 

enable_ip_repu => 0,

to the specific policy bank. Anyway. To see if this works, you can check the logs for “redis: IP”:

Aug 22 11:44:38 mail amavis[3554]: (03554-14) redis: IP 209.85.219.74 age: 0 0:13:56, last: 0 0:07:16, ttl: 7.5 h, s/h: 0/2, clean

#5 Bouncekiller

With amavis 2.6.0 another feature called Bouncekiller was implemented. In 2.6.2 this feature was improved. It uses the data from the pen pals feature which I explained previously here. This feature tries to help against Backscatter assuming that you got pen pals working you may enable it by just setting

$bounce_killer_score = 100;

Let me re-post the pre-requisites required for this feature to work:

A pre-requisite for proper operation of a bounce killer is a working SQL logging database (pen pals), or that outbound DSN messages have a Message-ID with a fully qualified domain name matching the @local_domains_maps list of lookup tables. Parts decoding must also not be disabled ($bypass_decode_parts=0), which is a default. Conditions are easily met when all mail from local users is submitted through a domain’s official mailer, which goes hand in hand with the requirement for DKIM signing and for other similar anti-spoofing techniques (SPF, whitelisting by IP address in Received trace, …).

December 15, 2008 amavisd-new-2.6.2 release notes

Also keep the following warning in mind:

The $bounce_killer_score should not be enabled when not all outgoing mail can be identified either by a local domain name in Message-ID or by being registered in pen pals SQL database, otherwise genuine bounces and returning MDN messages will be considered spam.

December 15, 2008 amavisd-new-2.6.2 release notes

I also disable this in my policy bank for the internal mailers.

Final thoughts

These 5 features aren’t all of amavisd-new available features. Just those I was interested in. Other features like OS-Fingerprinting (I was unable to get that working) and structured json logging to redis (I tried it, looks fine. I have no use for it, though) just to name a few.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.