A version amavisd-new-2.6.2 has been released, it is available at:
http://www.ijs.si/software/amavisd/amavisd-new-2.6.2.tar.gz
The 2.6.2 is mainly a maintenance release, but also adds some
new features. It is compatible with 2.6.1.
Release notes are also available on-line:
http://www.ijs.si/software/amavisd/release-notes.txt
amavisd-new-2.6.2 release notes
MAIN NEW FEATURES SUMMARY
- bounce killer: improved detection of nonstandard bounces;
- bounces to be killed no longer waste SpamAssassin time;
- tool to convert dkim-filter keysfile into amavisd configuration;
- compatibility with SpamAssassin 3.3 (CVS head) regained;
- rewritten and expanded documentation section on DKIM signing and
verification in amavisd-new-docs.html;
COMPATIBILITY WITH 2.6.1
- apart from small differences in logging and notifications, the
version 2.6.2 is compatible with 2.6.1, with its configuration file
and its environment;
- virus scanner entries were updated (as described below, most notably by
adding a regexp flag m), so be sure to update existing configuration file;
updated virus scanner entries can be used with 2.6.1 too;
- the %sql_clause default has changed in detail (see below), if its value
is overridden in a configuration file the setting may need updating;
BUG FIXES
- when feeding a message by SMTP back to MTA and MTA rejects a recipient as
invalid and a smtp connection cache is enabled, the SMTP protocol can get
out of step, rejecting the next message in the same connection with a
"503 5.5.1 Error: nested MAIL command"; this only affects (hopefully) rare
sites where recipient validation is performed after content filtering
instead of before content filtering; reported by Richard Smits;
- logging routines reporting warnings failed to include a diagnostics message
in a log, instead only a dry '_WARN:' or '_DIE:' with no explanation was
logged; a bug was introduced in 2.6.1; reported by Mike Cappella;
- amavisd-release: add a 'partition_tag' attribute to a release request if
a specified quarantine name ends up in a partition tag string in square
brackets; this feature was announced in 2.6.1 release notes, but never
made it into a distribution;
- amavisd-report failed on reading a message from SQL quarantine:
dispatch_from_quarantine failed: read: sql select failed,
DBD::Pg::st fetchrow_arrayref failed: no statement executing
reported by Achraf Tangui;
- while evaluating compiled regular expressions (qr), perl 5.10.0 ignores
flag m when present in the final expression but not in the qr itself,
causing messages containing multiple viruses not to report any virus
names (mail is still considered infected, but list of names is empty).
Changed regular expressions in virus entries by appending a /m flag
to regular expressions in the 6th element of each entry. According to
Perl maintainers this was a bug in 5.8.x and earlier, and the behaviour
of perl 5.10.0 is now according to specs; reported by Martin Huber;
- envelope sender address for administrator- and recipient notifications
($mailfrom_notify_admin, $mailfrom_notify_spamadmin, $mailfrom_notify_recip,
%mailfrom_notify_admin_by_ccat, %mailfrom_notify_recip_by_ccat) was
not expanded when their value is left unspecified in a configuration
file and defaults to parsing of $hdrfrom_notify_* settings. This leads
to MTA rejecting a notification from 'postmaster@${myhostname}' by a
'501 5.1.7 Bad sender address syntax'. Reported by Aleksey Chudov,
Jonas Jacobsson, Durk Strooisma, and Adam;
- remove unintentionally hard-coded SSL certificate and key file locations
stored in variables $smtpd_tls_key_file and $smtpd_tls_cert_file, they
are now configurable through a configuration file as intended;
- a macro 'rfc2822_sender' now returns a Sender address in a quoted form,
just like its cousin 'rfc2822_from';
- when stopping or restarting amavisd, check a PID file for being stale
_before_ testing whether a process exists, not the other way around;
previously an unlucky starting amavisd process could hit a:
Can't send SIG 0 to process [nnnn]: Operation not permitted
which prevented its startup when a stale PID was reused by an unrelated
process; reported by Zhang Huangbin;
- error reporting improvement: localize variables $@ and $! in all DESTROY
methods, thus preventing these variables from being clobbered behind
the scenes (e.g. by calling eval or system routines from DESTROY),
which could cause a surprising empty (or unrelated) error message
being reported by surrounding eval blocks;
- avoid problematic perl constructs open('|-') and open('-|') which fail
to catch certain fork errors, or waits indefinitely when resources
are tight; just explicitly create a pipe and call fork in subroutines
run_command, run_command_consumer and in run_as_subprocess. The change
possibly also solves some mystery cases where amavisd would appear
to hang when resources are tight (running out of swap space or near a
maxprocesses limit) instead of reporting a fork failure. Problem with
fork failing without giving a reason for failure reported by Uwe Kiewel;
- amavisd.conf-default: definition of %sql_clause default was out of date;
reported by Roland;
- releasing a non-existent message from a SQL quarantine produced an
inappropriate error message about a subsequent failure, instead of
reporting a missing record; reported by Rick (rn). Also let SQL treat
a NULL in mail_text.partition_tag as 0 by using coalesce() - changed
a $sql_clause{'sel_quar'} from:
SELECT mail_text FROM quarantine
WHERE partition_tag=? AND mail_id=?
ORDER BY chunk_ind
into:
SELECT mail_text FROM quarantine
WHERE coalesce(partition_tag,0)=coalesce(?,0) AND mail_id=?
ORDER BY chunk_ind
to facilitate transition from not having a partition_tag defined
(resulting in NULL partition_tag fields in SQL) into using it as a
numeric value (e.g. a week-of-the-year number);
- modified AV entry for a grisoft.com virus scanner by adding a regexp
flag /m to let ^ match at any line beginning of a possibly multi-line
response from a virus scanner; problem reported by John Beranek;
- recognize any 'ERROR:' result from a file(1) utility - not just an
'ERROR: Corrupted', and do not treat its exit status 1 as fatal,
but just log a warning;
- protect logging from being recursively re-entered when an error occurs
during writing of a log entry;
NEW FEATURES:
- bounce killer: improved parsing of nonstandard bounce messages (from
qmail, spamarrest.com and similar) yields more effective protection
against third-party bounces, including those without a Message-ID.
An analysis of 1000 previously passed bounces showed that 2/3 of those
are now recognized and blocked, bringing a bounce killer rate to 94 % of
all received bounces (with about 4 % of passed unverifiable bounces not
carrying an original mail header, and a tiny trickle of true bounces),
while still ensuring that bounces (in response to our genuine outbound
mail) and message disposition notifications (MDN, RFC 3798) are still
received reliably.
As a reminder: bounce killer is enabled by setting $bounce_killer_score
to a large value, e.g. 100. A pre-requisite for proper operation of
a bounce killer is a working SQL logging database (pen pals), or that
outbound DSN messages have a Message-ID with a fully qualified domain
name matching the @local_domains_maps list of lookup tables.
Parts decoding must also not be disabled ($bypass_decode_parts=0), which
is a default. Conditions are easily met when all mail from local users
is submitted through a domain's official mailer, which goes hand in hand
with the requirement for DKIM signing and for other similar anti-spoofing
techniques (SPF, whitelisting by IP address in Received trace, ...).
The $bounce_killer_score should not be enabled when not all outgoing
mail can be identified either by a local domain name in Message-ID or by
being registered in pen pals SQL database, otherwise genuine bounces and
returning MDN messages will be considered spam.
- to facilitate transition of DKIM signing from dkim-milter to amavisd-new,
a new command-line tool is available (the extra utility code is not
loaded during normal operation), taking a file name as its argument,
e.g.:
# amavisd convert_keysfile /var/db/dkim/keysfile.txt
and writing to stdout a set of lines that may be directly included into
amavisd.conf configurations file, matching semantics of a dkim-filter
keys file. It can be useful during transition, or for those who prefer
to specify signing keys and sender-to-key mappings as a file in a
syntax compatible with options -K -k of dkim-filter, and can live with
limitations of such syntax. See dkim-filter(8) man page for details
on the syntax.
The produced output consists of signing key declarations (calls to a
procedure dkim_key), where each call normally corresponds to exactly
one DNS resource record publishing a corresponding DKIM public key.
When necessary output also produces an assignment to a list of lookup
tables @dkim_signature_options_bysender_maps, which supplies non-default
mappings of sender domains to signing keys, e.g. when third-party
signatures are desired.
From the dkim-filter man page: The keyfile should contain a set of lines
of the form sender-pattern:signing-domain:keypath where sender-pattern
is a pattern to match against message senders (with a special character
"*" interpreted as "zero or more characters"), signing-domain is the
domain to announce as the signing domain when generating signatures
(or a '*', implying author's domain), and keypath is a path to the
PEM-formatted private key to be used for signing messages which match
the sender-pattern. The selector used in the signature will be the
filename portion of keypath. A line starting with "/" is interpreted as
a root directory for keys, meaning the keypath values after that line
in the file are taken relative to that path. If a file referenced by
keypath cannot be opened, the filter will try again by appending ".pem"
and then ".private". '#'-delimited comments and blank lines are ignored.
- DKIM verification now logs a note (at log level 2) when a signature
timestamp is in future;
- allow expiration time (tag x) to be requested with DKIM signing,
it is now supported since Mail::DKIM 0.29;
- when determining which DKIM-signing key should be applied or which
disclaimer options to apply, consider also addresses in all Resent-Sender
header fields. The search order is: From, followed by Resent-From and
Resent-Sender address pairs traversed top-down by resent blocks,
followed by Sender, and by envelope sender;
- amavisd-report no longer lets amavisd strip header fields found in a
quarantined message which were previously inserted by amavisd; most of
X-Spam-* and X-Amavisd-* header fields are now retained in a reported
message; suggested by Achraf Tangui;
- support IPv6 when connecting over an INET socket to virus scanners;
- support SMTP, LMTP and TCP_LOOKUP protocols also over Unix sockets;
- added a LDAP attribute amavisArchiveQuarantineTo to code and to
LDAP.schema; a patch was provided by Anand Palaniswamy (back in
October 2006); prompted by Quanah Gibson-Mount, both of zimbra.com.
- new file in the package: LDAP.ldif, same schema as in LDAP.schema, but
in ldif format; contributed by Quanah Gibson-Mount and independently
also by Michael Hall;
- @remove_existing_spam_headers_maps is now a per-recipient list of lookup
tables, so pre-existing X-Spam* header fields may be selectively removed
according to preferences of individual recipients or sub-domains, e.g.:
@remove_existing_spam_headers_maps = ({
'user@example.com' => 0,
'user@office.example.net' => 1,
'.office.example.net' => 0,
'.' => 1, # all the rest
});
- added a macro b64encode, which could be used in log template to
facilitate log parsing, perhaps by using the following in a $log_templ:
[? [:header_field|Subject]||, \
Subject: [:b64encode|[:header_field|Subject|100]]]#
Suggested by Rajkumar S;
- added a macro HEADER as a synonym for a macro header_field
for compatibility with SpamAssassin;
- added a configuration variable $logline_maxlen (default value is 980,
lower bound is 50), allowing user to customize syslog line wrapping
threshold; based on a patch by Charles A. Scheidecker;
- when loading a policy bank, most entries from an associative array (hash)
being loaded entirely replace entries of the same key in the currently
active policy bank, but entries which are references to a hash are
normally merged with existing hashes, replacing only specified key/values
but leaving remaining (non-existent in a new hash) key/values pairs
unchanged. In some rare cases it would be more desirable to entirely
replace existing hashes, which so far was not possible.
For this purpose a hack was introduced: if a hash in a policy bank being
loaded contains a key name 'REPLACE' (uppercase) and its value is true,
this hash replaces a current hash, instead of being merged with it.
For example, a policy bank 'DITCH' when loaded replaces a hash
%final_destiny_by_ccat entirely, leaving only a key CC_CATCHALL there:
$policy_bank{'DITCH'} = {
final_destiny_by_ccat => { REPLACE=>1, (CC_CATCHALL) => D_DISCARD },
};
Without specifying a REPLACE=>1 remaining key/value pairs in a hash
%final_destiny_by_ccat would stay unchanged and only the CC_CATCHALL
key/value pair would be replaced by a new setting (which may not be
desirable):
$policy_bank{'DITCH'} = {
final_destiny_by_ccat => { REPLACE=>0, (CC_CATCHALL) => D_DISCARD },
};
or equivalently:
$policy_bank{'DITCH'} = {
final_destiny_by_ccat => { (CC_CATCHALL) => D_DISCARD },
};
- rewritten and expanded documentation section on DKIM signing and
verification in amavisd-new-docs.html;
OTHER
- tested with perl 5.10.0;
- package Amavis::IO::FileHandle now supports a method READ (i.e. invoked
by a perl functions sysread through a tied hash), which is needed by
SpamAssassin revisions since 2008-09-25 (3.3), bringing a little speedup
to transferring a message from amavisd to SpamAssassin, and avoiding a
Perl I/O bug (perl bug 39060; SA: bug 5985) on some installations;
- updated @virus_name_to_spam_score_maps to recognize new malware name
formats used by some popular third-party ClamAV signatures (Sanesecurity,
MSRBL, MBL); thanks to Mike Cappella, Gary V, Wijatmoko U. Prayitno,
Steve Basford, Luca Gibelli, Bill Landry, Henrik K;
- keep only one (unique) copy of each malware/spam name when infection
is downgraded to spam through @virus_name_to_spam_score_maps or when
reported as a virus;
- macro F now only shows the first (if any) banned leaf part name,
preceded by comments from a banning rule regexp (if any), instead of
a list of multiple banned parts each with its full MIME/archive path.
Note that Perl syntax for a comment within a regexp is: (?# ... )
For example, given the following rule...
$banned_filename_re = new_RE(
qr'^\.(exe-ms|dll)$(?# rule #9)', # banned file(1) types
);
... a macro expansion of macros banning_rule_key, banning_rule_comment,
banning_rule_rhs, banned_parts and F will be:
banning_rule_key: (?-xism:^\\.(exe-ms|dll)$(?# rule #9))
banning_rule_comment: rule #9
banning_rule_rhs: 1
banned_parts:
multipart/mixed | application/octet-stream,.exe,.exe-ms,videos.exe
F: rule #9:application/octet-stream,.exe,.exe-ms,videos.exe
Likewise a SMTP response (with D_REJECT) would match a macro F as
before and would yield:
554 5.7.0 Reject, id=42721-01 - BANNED: rule #9:application/octet\
-stream,.exe,.exe-ms,videos.exe
A default administrator and recipient notification (still using a macro
F in their template), a main log entry, as well as a DSN and a rejection
message, will now be shorter and hopefully less confusing to an end user.
A full list of banned part paths (as previously produced by a macro F)
is now available under a new name as a macro 'banned_parts' and can be
used in custom templates to retain previous behaviour if desired.
Suggested by Andreas Schulze and Peer Heinlein;
- remove a 'LIMIT 1' from default $sql_clause{'sel_penpals_msgid'} and
from $sql_clause{'sel_penpals'} clauses, it is redundant and it happens
to make a MySQL 5.1 optimizer choose a slow plan;
investigated by Michael Scheidell;
- changed a default $sql_select_policy from:
SELECT *, users.id FROM users LEFT JOIN policy ON ...
into a:
SELECT users.*, policy.*, users.id FROM users LEFT JOIN policy ON ...
MySQL and PostgreSQL are happy with a 'SELECT *, users.id', but Oracle
wants 'SELECT users.*, policy.*, users.id', which is also acceptable
to MySQL and PostgreSQL and shouldn't make any difference;
problem reported and a solution provided by Chris Bryant;
- optimize storage of DKIM signing keys when multiple calls to dkim_key()
(from amavisd.conf) specify the same file to be associated with different
keys - now only one copy of a private key is kept in memory;
- sanitize (strip) bare CR characters in mail before DKIM-signing a message
and when forwarding it over SMTP (or LMTP) protocol which prohibits CR
characters outside of CRLF pairs. Previously a DKIM signature generated
by amavisd on messages with embedded bare CR characters broke when passed
back through Postfix (following a principle of garbage-in, garbage-out).
This was mainly an issue when a message was incorrectly sanitized or
a disclaimer added by an external program such as altermime, which (due
to a bug in versions 0.3.10 and older) could inappropriately introduce
CR characters into a message. Reported by Patrick Wong;
- retain original rfc822 quoting of envelope sender address when forwarding
mail, instead of using a sanitized version (de-quoted & re-quoted);
- insert autolearn=... information field into an X-Spam-Status header field,
similar to how SpamAssassin does it; suggested by Jonathan Skanes;
- an SQL field msgs.spam_level now receives a sum of SA score plus a minimum
of internally generated score boosts across all message recipients, to
facilitate coarse assessment by third party utilities without having
to look into msgrcpt records; previously it only reflected a SA score
(but field msgrcpt.bspam_level remain unchanged, storing a sum for each
individual recipient as before);
- insert "AM:BOOST=boost_scores_list" into a list of triggered spam tests
to make visible the internally generated per-recipient spam score boosts
(like from: pen pals, soft white/black-listing, bounce killer) in the log
and in the X-Spam-Status header field. The 'tests' list in X-Spam-Status
or in the log (macro %T) can now look like:
tests=[AM:BOOST=+1.3+0.51-1.1, BAYES_99=3.6, ...]
Multiple summands appear in multi-recipient messages where boost scores
differ between recipients (the list is squashed, only unique values are
shown). Apparent mismatch in score addition pointed out by John Beranek.
- when quarantining and generating notification for administrator and
recipients, and the per-recipient contents category differs from a
per-message summarized contents category, use the per-recipient contents
category for lookups into settings; a desirable side effect is that
recipients which are bypassing some tests (like tests for virus or spam)
no longer receive a recipient notification for cases they are not
interested in; undesired behaviour pointed out by Erin D. Hughes;
- drop log level of 'smtp resp to NOOP' and related messages from 2 to 3
to reduce log clutter; reported by kfx;
- log a warning during startup when DKIM verification is not enabled
(when $enable_dkim_verification is at its default value of undef).
To quench down the warning and keep DKIM verification disabled, set
the $enable_dkim_verification explicitly to 0;
- when mail with banned part is to be passed but defanged, provide a more
informative warning in the text part:
WARNING: banning rules detected suspect part(s),
do not open unless you know what you are doing
suggested by Gerald Macinenti;
- amavisd-agent: Content*Msgs* now take as a 100% reference the InMsgs$2
counter instead of Content$1Msgs;
- when exec in a forked process fails, call POSIX::_exit with exit status
6 (SIGABRT) instead of 8 which has different meanings on different OS;
- ensure a BDB cursor is unlocked in put_initial_snmp_data() even in case
of errors or signals during writing of the initial sys* set of SNMP
variables;
- provide a generous but firm 4 MB sanity limit on a header section size to
avoid excessive storage requirements while parsing and storing a runaway
header section; the limit also protects DKIM signature verifier on huge
headers; exceeded limit does not affect other mail checks and forwarding,
only access to individual header fields beyond the limit is crippled and
DKIM signatures would most likely be invalidated;
- do not pre-load module Mail::SPF::Query with versions of SpamAssassin
3.2.0 or later, it has been replaced by Mail::SPF;
- internal: modify mail_to_local_mailbox and do_quarantine to better deal
with suppressing multiple quarantining to the same mailbox, e.g. when
two quarantining methods are active but point to the same file;
- internal: brush up I/O modules for consistency: open method should
implicitly close a previously open file, print method should print all
its arguments, read methods now support reading to a buffer at an offset;
- internal: change most calls to lookup() into calls to a newer lookup2()
for added flexibility, adding option 'Label' to some calls to facilitate
debugging;
- internal: passing options to lookup2 (and to other subordinate lookup
methods) as a hash instead of a hashref;
- internal: renamed Amavis::In::Message::PerRecip methods:
banned_keys -> banning_rule_key, banned_rhs -> banning_rule_rhs,
added: banning_rule_comment and banning_reason_short;
- internal: replace subroutine unique with two: unique_list and unique_ref;
- internal: remove dependency on a module IO::Wrap;
- internal: many rather cosmetic changes for consistency, updated comments;
- README.sql-mysql: add 'ALTER table' suggestions to change CHAR to BINARY
and VARCHAR to VARBINARY data types; suggested by Peter Huetmannsberger:
Mark
------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
AMaViS-user mailing list
AMaViS-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/