I guess that most people use amavisd-new together with spamassassin and for example ClamAV. Probably a few more use features like DKIM verification and signing with amavis. However, there are some features which aren’t found in the usual howtos. Here are some of them.
1. Configuration Consistency: Reading Postfix Files
Did you know that Amavis can directly read and include Postfix hash files and CIDR files? If you manage Postfix using configuration files like I do, this feature is invaluable because it eliminates the need to maintain duplicate configuration sets.
The Problem of Configuration Duplication
Without this feature, configuration snippets look repetitive:
Postfix Configuration
mynetworks = 84.200.7.144/28 127.0.0.0/8
virtual_mailbox_domains = no-uce.de, jeanbruenn.infoAmavis Configuration
@mynetworks = qw( 127.0.0.0/8 84.200.7.144/28 );
@local_domains_maps = ( [ "no-uce.de", "jeanbruenn.info" ] );Every time a domain is added or an IP network changes, both files require updates. This is inefficient and prone to configuration errors.
The Solution: Read Postfix Files
Using read_cidr and read_hash allows the configuration to be centralized in Postfix, while Amavis simply consumes it. This ensures consistency and simplifies management.
Postfix
# Postfix main.cf
mynetworks = cidr:/etc/postfix/mynetworks
virtual_mailbox_domains = hash:/etc/postfix/virtual/domainsAmavis
# Amavis 50-user
@mynetworks = @{ read_cidr('/etc/postfix/mynetworks') };
@local_domains_maps = ( read_hash("/etc/postfix/virtual/domains") );Note: Special care must be taken with mynetworks. In Postfix, it defines a trusted sender for relaying; in Amavis, it defines an internal sender. While often identical, the function is different.
2. Enabling Advanced Features with Redis
Redis alone is just a fast key-value store, but it acts as the high-performance backend for several advanced Amavis features, including Pen Pals, IP Reputation, and the Bouncekiller.
Using Redis is straightforward. Make sure Redis is installed and listens only to the loopback interface (127.0.0.1).
Redis Storage Backend Configuration
@storage_redis_dsn = ( { server => '127.0.0.1:6377' } );30 days retention. Default is 16 days.
$storage_redis_ttl = 30*24*60*60;Using a specific database ID in a multi-instance setup
@storage_redis_dsn = ( {
server => '127.0.0.1:6377',
db_id => 1
} );Note: I prefer to work with multiple isolated Redis instances, which allows for better resource isolation and management (capacity planning).
3. Pen Pals: The Context-Aware Whitelist
The idea of this feature is that you may reduce the spam score if you receive an email from someone you wrote a mail to beforehand. This soft-whitelisting feature, first implemented in version 2.4.2, adds crucial context to your spam scoring. Redis support became viable starting in Amavis 2.8.1.
Prerequisites and Architecture
To ensure Pen Pals works correctly, you must satisfy several architectural prerequisites. These are sourced directly from the Amavisd-new release notes:
both the outgoing and the incoming mail must pass through amavisd (although outgoing mail may have checks disabled or made more permissive if desired);
June 27, 2006 amavisd-new-2.4.2 release notes
SQL logging must be enabled (@storage_sql_dsn) and records should be kept for at least several days (some statistics (2006-11 update): 90% of replied mail (or followups) is sent within 2 weeks since previous correspondence, 40% within 24 hours, 20% within 3 hours, 10% within 30 minutes, 5% within 12 minutes);
Note: You may ignore the SQL prerequisite because we use Redis as the high-performance storage backend.
@mynetworks and @local_domains_maps must reflect reality, allowing amavisd to distinguish between outgoing, incoming and internal-to-internal mail;
the information about client IP address must be available to amavisd, i.e. Postfix XFORWARD protocol extension must be enabled, or AM.PDP+milter;
June 27, 2006 amavisd-new-2.4.2 release notes
In my Postfix setup I do use a Milter for inbound mails (so I don’t need to change something here). I do use an after-queue filter for my clients, so I need to make sure the XFORWARD extension is enabled in my master.cf:
submission inet n - n - - smtpd
-o content_filter=amavisfeed:[127.0.0.1]:10023
-o syslog_name=postfix/submission
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
[..]
amavisfeed unix - - n - 2 lmtp
-o lmtp_data_done_timeout=1200
-o lmtp_send_xforward_command=yes
-o disable_dns_lookups=yes
-o max_use=20The lmtp_send_xforward_command is set to yes, which means everything is fine. If you’re using smtp instead of lmtp there is smtp_send_xforward_command.
configuration variable $penpals_bonus_score must be set to a positive value (such as 1.0, increase to perhaps 5 or 8 after seeing that it works), zero disables the feature and is a default;
June 27, 2006 amavisd-new-2.4.2 release notes
I use the following settings for testing purposes in my 50-user.conf:
#
# penpals
#
$penpals_bonus_score = 1;
$penpals_threshold_low = undef;
$penpals_threshold_high = undef;
$sql_clause{‘sel_penpals’} must contain a SELECT clause (which by default it does, unless overridden by an old assignment to %sql_clause in amavisd.conf);
June 27, 2006 amavisd-new-2.4.2 release notes
Not relevant due to our use of Redis.
sender/recipient address pair must exactly match recipient/sender pair of previous correspondence (except for allowed case-changes in domain part), which means that care must be taken when canonical and/or virtual mapping is performed by MTA (such as mapping between internal and external address forms) – if external address forms of local addresses are to be seen by a content filter then canonical mapping (int->ext) must be done *before* filtering and virtual mapping (ext->int) *after*; alternatively, if internal address forms are to be seen by a content filter, then canonical mapping should be done after filtering, and virtual mapping before; see README.postfix, section “TO DO ‘VIRTUAL ALIAS’ MAPPING AND OTHER POSTFIX CLEANUP PROCESSING BEFORE OR AFTER CONTENT FILTERING?” (P.S. later renamed to ‘Advanced Postfix and amavisd-new configuration’);
June 27, 2006 amavisd-new-2.4.2 release notes
The referenced documentation explains how to set up two cleanup services to achieve this behavior. Please read that document it’s really good. Exactly like in that guide I added a second cleanup service to Postfix master.cf. I do add -o syslog_name=postfix/pre-queue to it because without that I won’t notice if it works.
pre-cleanup unix n - n - 0 cleanup
-o syslog_name=postfix/pre-cleanup
-o virtual_alias_maps= then I modified the existing cleanup service:
cleanup unix n - n - 0 cleanup
-o mime_header_checks=
-o nested_header_checks=
-o body_checks=
-o header_checks= Finally I modified pickup, submission and smtp to contain -o cleanup_service_name=pre-cleanup. Mind that I add -o cleanup_service_name to my smtpd service due to my use of postscreen (instead of to smtp)
smtp inet n - n - 1 postscreen
smtpd pass - - n - - smtpd
-o smtpd_milters=${milter_amavis}
-o cleanup_service_name=pre-cleanup
[..]
submission inet n - n - - smtpd
-o content_filter=amavisfeed:[127.0.0.1]:10023
-o syslog_name=postfix/submission
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o cleanup_service_name=pre-cleanup
pickup unix n - n 60 1 pickup
-o cleanup_service_name=pre-cleanupCheck the logs (hence my usage of syslog_name) to make sure that this works and you can both receive as well as send emails 🙂
Settings
I will cite the RELEASE-NOTES again.
mail with (unadjusted) SA score below $penpals_threshold_low (1 by default) is exempted from pen pals check to save time and lighten the load on SQL; similarly for high score spam which would not have a chance of being ‘saved’ even by a maximal pen pals bonus score;
June 27, 2006 amavisd-new-2.4.2 release notes
Due to the use of Redis instead of a heavy sql database, I believe settings this lower would be fine. Depending on your load and available resources you might still want to limit this. I keep $penpals_threshold_low at undef; so that there is no bottom limit.
$penpals_threshold_high … when (SA_score – $penpals_bonus_score > $penpals_threshold_high) pen pals lookup will not be performed to save time, as it could not influence blocking of spam even at maximal penpals bonus (age=0); usual choice for value would be kill level or tag2 level, or other reasonably high value; undef lets the threshold be ignored and is a default (useful for testing and statistics gathering);
June 27, 2006 amavisd-new-2.4.2 release notes
I set this to $sa_kill_level_deflt:
$penpals_threshold_high = $sa_kill_level_deflt;configuration variable $penpals_bonus_score must be set to a positive value (such as 1.0, increase to perhaps 5 or 8 after seeing that it works), zero disables the feature and is a default;
[..]
$penpals_bonus_score … a maximal (positive) score value by which spam score is lowered when sender is known to have previously received mail from our local user from this mail system. Zero or undef disables pen pals lookups, and is a default.
$penpals_halflife … exponential decay time constant in seconds, defaults to 7 days; pen pal bonus is halved for each halflife period since the last mail sent by a local user to the current message’s sender;
June 27, 2006 amavisd-new-2.4.2 release notes
The value fo $penpals_halflife is fine in my opinion. The value for $penpals_bonus_score could be$sa_kill_level_deflt – 1:
$penpals_bonus_score = $sa_kill_level_deflt - 1;So all in all I do use:
#
# penpals
#
$penpals_bonus_score = $sa_kill_level_deflt - 1;
$penpals_threshold_low = undef;
$penpals_threshold_high = $sa_kill_level_deflt;If you set your log_level somewhat higher (I believe it was 2, I’m using 3, though). You can check the logs to see if it’s working:
Aug 22 12:52:54 mail amavis[3555]: (03555-16) penpals: (redis) found (3,2) age 0 12:23:51, refs: weRr6tCck-FQ (44631 s) rid=2
Aug 22 12:52:54 mail amavis[3555]: (03555-16) penpals: adj.bonus 0.950, age 0 12:23:51 (44631), SA score 0.817, <xxx@gmail.com> replying to <himself@jeanbruenn.info>, ref mail_id: weRr6tCck-FQ
4. IP Address Reputation (Local Threat Intelligence)
Amavisd-new version 2.9.0 introduced a powerful feature called IP address reputation. This function essentially turns your mail server into a collector of local threat intelligence.
When a Redis storage back-end is enabled, this feature automatically profiles IP addresses that send you mail, tracking metrics like spam, ham, banned, and infected messages. This process adds a critical, local layer to your anti-spam defense.
Here is the precise description from the release notes:
When a Redis storage back-end is enabled, besides the existing pen pals functionality, it now also offers information updating and retrieval on IP address reputation. This function is enabled by default when @storage_redis_dsn is nonempty, but can be disabled by setting $enable_ip_repu to false (to 0 or undef), per policy bank if necessary.
For each mail message a list of public IP addresses (IPv4 or IPv6) is collected from its ‘Received’ trace header fields in a mail header section. A redis server maintains a database of each IP address encountered. For each IP address an entry carries a set of counters corresponding to the number of mail messages encountered in the past having this IP address in a trace header. These counters show: a number of spam messages, a number of ham messages, a number of banned or infected messages, and a total number of messages. Also a timestamp of the first and last encounter is kept. Each entry (a set of counters) is subject to automatic expiry, so that infrequently encountered IP addresses are eventually automatically purged from a database by a redis server itself.
May 9, 2014 amavisd-new-2.9.0 release notes
Configuration and Observability
It is vital to whitelist your mail servers using @ip_repu_ignore_networks to prevent your own systems from being incorrectly profiled.
@ip_repu_ignore_networks =
qw( 192.0.2.44 192.0.2.45 198.51.100.0/24 2001:db8::1:25 );If you use Policy Banks, you can also disable the check selectively by adding enable_ip_repu => 0 to a specific policy bank (e.g., for trusted internal mailers).
enable_ip_repu => 0, To see this feature working (a core step in Observability), check the logs for “redis: IP”:
Aug 22 11:44:38 mail amavis[3554]: (03554-14) redis: IP 209.85.219.74 age: 0 0:13:56, last: 0 0:07:16, ttl: 7.5 h, s/h: 0/2, clean#5 Bouncekiller
With amavis 2.6.0 another feature called Bouncekiller was implemented. In 2.6.2 this feature was improved. It uses the data from the pen pals feature which I explained previously here. This feature tries to help against Backscatter assuming that you got pen pals working you may enable it by just setting
$bounce_killer_score = 100;Let me re-post the pre-requisites required for this feature to work:
A pre-requisite for proper operation of a bounce killer is a working SQL logging database (pen pals), or that outbound DSN messages have a Message-ID with a fully qualified domain name matching the @local_domains_maps list of lookup tables. Parts decoding must also not be disabled ($bypass_decode_parts=0), which is a default. Conditions are easily met when all mail from local users is submitted through a domain’s official mailer, which goes hand in hand with the requirement for DKIM signing and for other similar anti-spoofing techniques (SPF, whitelisting by IP address in Received trace, …).
December 15, 2008 amavisd-new-2.6.2 release notes
Also keep the following warning in mind:
The $bounce_killer_score should not be enabled when not all outgoing mail can be identified either by a local domain name in Message-ID or by being registered in pen pals SQL database, otherwise genuine bounces and returning MDN messages will be considered spam.
December 15, 2008 amavisd-new-2.6.2 release notes
I also disable this in my policy bank for the internal mailers.
Final Thoughts and Other Features
These five are just the features that caught my attention. Amavisd-new is packed with advanced functionality. I tried other features, like OS-Fingerprinting (was unable to get it working) and structured JSON logging to redis (looks fine, but I currently have no use for it). This entire exploration shows that there is always more to find in a complex system once you start digging.