AMAVIS POLICY DELEGATION PROTOCOL (AM.PDP) ========================================== Author: Mark Martinec Revisions: 2003-11-10 created 2005-03-18 2005-06-22 2006-08-18 (added attributes: version_server, insheader and quarantine), 2007-05-17 (allow attribute value: request=requeue) 2008-06-21 (new attribute partition_tag=xx on release requests) 2009-11-16 2010-01-22 (new attribute log_id on response) NOTE: at the end of this document there is a description (by Stephane Lentz) of the old protocol, now called AM.CL. Amavis policy delegation protocol (AM.PDP) is intended to replace the old amavis client protocol (AM.CL) as spoken between amavisd-new helper programs (amavis.c or amavis-milter.c) and the amavisd daemon. The server side is implemented by amavisd-new daemon. A sample AM.PDP client in Perl is helper-progs/amavis.pl, and a rewrite by Petr Rehor of the helper program amavis-milter.c to use the new AM.PDP protocol is available as a separate project, see: http://sourceforge.net/projects/amavisd-milter/ The new amavisd client/server protocol is based on the 'Postfix policy delegation protocol', described in the Postfix document file README_FILES/SMTPD_POLICY_README, which can be used to delegate policy decisions to an external server that runs outside Postfix. It is conceptually similar to sendmail/milter mechanism, but based on a documented text protocol which is easier to test and debug, instead of using an undocumented binary protocol such as sendmail/milter. There are a few departures from the original Postfix policy delegation protocol: - Postfix policy delegation protocol terminates lines with LF, AM.PDP protocol terminates lines with CR LF; ( This was changed to facilitate debugging via telnet, and to make it more similar to related protocols, e.g. SMTP, HTTP, FTP, ... ) - Postfix policy delegation protocol restricts the reply to one field only, AM.PDP allows arbitrary number of fields passed from server back to client; ( This allows additional functionality like modifying or removing recipient addresses, adding or editing mail header fields, etc. ) - Postfix policy delegation protocol expects the same attribute name to be used only once; AM.PDP allows repeated appearances of the attribute, which makes passing of lists easier (such as multiple recipient addresses); - attribute value may consist of more than one field. Fields are delimited by exactly one non-encoded space. Spaces within a field must be encoded like any other restricted character (see below). The protocol may be spoken over a Unix STREAM socket, or over an inet tcp socket. The client request is a sequence of name=value attributes, each terminated by CR LF. The sequence is terminated by an empty line (only CR LF). The server reply has the same structure. After client receives server response sequence terminated by an empty line, it may close the session, or issue another request on the same session. During normal operation the server should not close the socket until it has been closed by the client. Only under special circumstances is the server allowed to close the session, e.g. in response to a timeout or fatal error condition. The order of attributes does not matter, except for the 'request' attribute which must appear first in the client request. The policy client as well as the policy server should ignore any attributes that it does not care about. An attribute name must not contain non-encoded characters: "=", "%", space, null, CR or LF. An attribute value must not contain non-encoded characters: "%", space, null, CR, or LF. All restricted characters must be hex-encoded to a sequence of three characters: a percent sign, followed by two hex digits, e.g. %2A. Hexadecimal digits 0..9,a..f or 0..9,A..F are allowed. Any other character MAY be encoded as well. Although not mandated, it is prudent to encode all non-printable characters. The line length is not limited by this protocol. Both the server and the client must be prepared to handle arbitrary line lengths without breaking the protocol or rising security issues. Both are allowed to internally truncate unreasonably long lines to a sensible length, and issue a warning. Neither the client nor the server must make any assumptions that certain characters will not be used in the attribute name or values. E.g. a presence of encoded null or newline or other special character in the attribute name or value must be safely and appropriately handled. If such a character does not comply with the expected syntax, the case should be handled to the best of client or server understanding and capability, e.g. character ignored, attribute ignored, and/or a warning logged. The following example Perl expression may be used for encoding/decoding: to decode attribute name and attribute values: s/%([0-9a-fA-F]{2})/pack("C",hex($1))/eg to encode attribute name: s/([^0-9a-zA-Z_-])/sprintf("%%%02x",ord($1))/eg; to encode attribute values (each space-separated field individually): s/([^\041-\044\046-\176])/sprintf("%%%02x",ord($1))/eg; Attributes in the client request are: ------------------------------------- request=AM.PDP is a required first attribute, its value must be AM.PDP sender= specifies the envelope sender address (reverse-path). The attribute should appear exactly once. The attribute value syntax is specified in rfc5321 as 'Reverse-path' (i.e. smtp-quoted form, enclosed in <>); a null reverse path is specified as <>. recipient= specifies the envelope recipient address. The attribute appears once for each recipient address, the order of addresses must be preserved and might be significant for some setups or functions. The attribute value syntax is specified in rfc5321 as 'Forward-path'. tempdir=/var/amavis/amavis-milter-MWZmu9Di Specifies a temporary work directory to be used for mail unpacking, typically also containing the original mail file - see attribute 'mail_file' below. This attribute should be present exactly once. The server is allowed to use the specified directory to create additional temporary files if it chooses so. As a security precaution, currently amavisd restricts the temporary directory path, which must be a subdirectory under $TEMPBASE or $MYHOME. tempdir_removed_by=client Specifies the client will be responsible for removing the temporary directory. The server must not remove the file email.txt nor the directory, but it may remove temporary files and subdirectories it has created. tempdir_removed_by=server Specifies the server is responsible to remove the temporary directory if/when it deems appropriate. This is a default in the absence of this attribute (for compatibility with traditional amavis clients). mail_file=/var/amavis/amavis-milter-MWZmu9Di/email.txt Specifies a file name (full file path) of a file containing the original mail with header and body. This attribute should be present at most once. In its absence the file name defaults to /email.txt. delivery_care_of=client Specifies that server should NOT actively forward the mail to recipients, but should only report its opinion in its reply, and let the client act on it. This is a default in the absence of this attribute. delivery_care_of=server Specifies that server is responsible to actively forward the mail to recipients if it deems the message appropriate for forwarding. This attribute value indicates that the client has no capability or intention to forward mail by itself. queue_id=qid optional informational attribute: MTA queue id; protocol_name=ESMTP optional informational attribute: the name of the protocol used by the original SMTP client to deliver the mail to the client (or to its associated MTA). Common values are ESMTP, SMTP, LMTP. helo_name=b.example.com optional informational attribute: the value of the HELO or EHLO or LHLO command specified to our MTA by the original SMTP client; client_address=10.2.3.4 optional informational attribute: the IP address of the original SMTP client; client_name=mail.example.com optional informational attribute: the DNS name of the original SMTP client as obtained by DNS reverse mapping of the original SMTP client; policy_bank=TLS,ORIGINATING,MYNETS value is a comma-separated list of policy bank names. Names of nonexistent banks are silently ignored, so are leading and trailing spaces and TABs around each name. The order of policy bank loading generally follows the order in which information about a message were obtained: - interface or socket -based policy banks (when MTA connects to amavisd); - MYNETS (when client's IP address becomes known); - the list as specified in the policy_bank attribute of AM.PDP; - MYUSERS (when sender e-mail address becomes known); Attributes in the server response are: -------------------------------------- The current set of attributes maps almost exactly to the capabilities of libmilter, which should facilitate initial implementation. See sendmail libmilter documentation as well. version_server=2 Since amavisd-new-2.4.3 the 'version_server=2' is inserted, older versions did not provide this attribute (assumed version was 1). With version 2 the following changes to the protocol were made: - version_server=2 is provided by the server as the first attribute; - delheader and chgheader now stand in a response before insheader and addheader, assuming that milter MTA will execute these in the same order; - new attribute insheader, see below; - new attribute quarantine, see below. log_id=xxx Provided since amavisd-new-2.7.0, tells a log_id under which the amavisd daemon logged the request and related debugging events; delrcpt= The specified recipient should be removed from the list of recipients of this mail. The specified address must exactly match the recipient addresses as specified in the client request, i.e. a smtp-quoted form enclosed in <> should be specified; addrcpt= The specified recipient should be added to the list of recipients of this mail. Paired with 'delrcpt' the pair indicates an existing recipient address to be replaced by a modified address, e.g. to add an address extension. delheader=index hdr_head Specifies an existing mail header field should be removed. Index is a decimal integer indicating which of the header fields with the same head should be affected, count starts with 1. 'hdr_head' does not include a colon! chgheader=index hdr_head hdr_body Specifies an existing mail header field should have its body replaced by a new hdr_body. Index is a decimal integer indicating which of the header fields with the same head should be affected, count starts with 1. 'hdr_head' does not include a colon! insheader=index hdr_head hdr_body Similar to addheader, but specifies a mail header field to be inserted to the mail header at a given position, index 0 implies top of the header. Amavisd-new passes a value of $prepend_header_fields_hdridx as the index argument, which is configurable and defaults to 1 for compatibility with dkim-milter and dk-milter signing milters. Alternative useful value is 0, which may be used in absence of other milters inserting their header fields at index 1. Header fields to be prepended are listed in reverse order, the last one listed in AM.PDP is to appear at the top of a resulting mail header. Inserting a header field at an arbitrary position is a later addition to sendmail milter protocol (introduced with sendmail 8.13.0 2004-06-20) addheader=hdr_head hdr_body Specifies a mail header field to be appended to the mail header. Note the use of exactly two value fields, separated by exactly one space. As described above, spaces in each field must be hex-encoded. 'hdr_head' does not include the colon! replacebody=new_body Not implemented - for future consideration. quarantine=reason place message on hold or to a quarantine maintained by MTA, and supply a reason text (e.g. client may call smfi_quarantine milter routine; btw, smfi_quarantine was introduced with sendmail 8.13); For future use - it is currently (2.4.3 or earlier) never used. return_value=val where val can be any of: continue, accept, reject, discard, tempfail This attribute should be present exactly once, and indicates to the client what should be done with the mail and how the original SMTP session should be treated. setreply=rcode xcode message SMTP response code and text suggested by server to the client to be used in response to the original SMTP client; There are exactly three value fields, separated by a single space. exit_code=n Similar in semantics to return_value attribute, but uses sysexits.h -based exit values for the purpose. Useful with old amavis clients, but should be ignored by new clients, which should base its decision on return_value instead. An example ========== Indentation and text in parenthesis in the following example is not part of the actual session. $ telnet 127.0.0.1 9998 Trying 127.0.0.1... Connected to localhost.example.com. Escape character is '^]'. (client->server request) request=AM.PDP sender=me@example.com recipient=user1@example.net recipient=user2@example.net protocol_name=ESMTP client_address=10.2.3.4 tempdir=/var/amavis/amavis-milter-MWZmu9Di (server->client response) setreply=250 2.5.0 Ok,%20id=MWZmu9Di,%20continue%20delivery return_value=continue exit_code=0 (or:) setreply=250 2.7.1 Ok,%20discarded,%20UBE,%20id=mYOljdn2 return_value=discard exit_code=99 (or:) setreply=550 5.7.1 Message%20content%20rejected,%20UBE,%20id=S7uS4qvA return_value=reject exit_code=69 (or:) setreply=451 4.5.0 Error%20in%20processing,%20id=... return_value=tempfail exit_code=75 =============================================================================== Releasing a message from a quarantine: request=release (or request=requeue, available since 2.5.1) mail_id=xxxxxxxxxxxx secret_id=xxxxxxxxxxxx (authorizes a release) partition_tag=xx (optional, but recommended if info is available) quar_type=x F/Z/B/Q/M (file/zipfile/bsmtp/sql/mailbox) mail_file=... (optional: overrides automatics; $QUARANTINEDIR prepended) requested_by= (optional: lands in Resent-From:) sender= (optional: replaces envelope sender) recipient= (optional: replaces envelope recips) recipient= recipient=... In reply, for each recipient a SMTP status response is returned (similar to LMTP) quar_type defaults to Q if spam_quarantine_method is sql:, otherwise to F. =============================================================================== AMAVIS SIMPLE CLIENT/SERVER PROTOCOL (traditional) description by Stephane Lentz 2003-11-06 amavisd is the daemon part of AMAVIS in charge of scanning SMTP messages. It receives messages from other applications (clients) using either SMTP or a simple protocol which is detailed in this document. The protocol being used depends on the MTA and architecture chosen. The "simple protocol" is most often used with sendmail in a MILTER set-up (the client program in such a case is amavis-client). AMAVISD receives messages from clients through a UNIX socket : The UNIX socket used is by default /var/amavis/amavisd.sock . It is defined : - at the amavisd server level in amavisd.conf as $unix_socketname - at the client level when using MILTER (amavis-milter available in helper-progs) as a configure option : --with-sockname swith The protocol used between the client and server is simple (basic & limited). There is no possibility for the server to ask the client to remove/add/change headers. The server can only say if the message was detected as CLEAN, as UNSAFE and to be rejected/discarded) or not analysed successfully due to some errors. PROTOCOL IN DETAILS : The client connects to the AMAVISD server's socket. IF successful then for each incoming message : - the client computes and create a new temporary directory ($tempdir) to store the new incoming message. - the incoming message is stored as $tempdir/email.txt - the client sends the directory name to the SERVER - the server sends \1 to the client if the directory is ok - the client sends the envelope sender recipient address - the server sends \1 to the client if ok - the client sends the envelope recipient addresses one by one to the server: the client trims the address if it is longer than the maximum length possible the client sends this address to the server the server sends \1 to the client if ok - the client sends some request to analyze the message to the SERVER. The character used is EOT (end of transmission) :\3 - the server processes the mail stored in the directory ($tempdir/email.txt) - the server sends a STATUS number to the CLIENT. This number returned is either : EX_OK (2) : message CLEAN EX_UNAVAILABLE (69) : message UNSAFE to be rejected at the SMTP LEVEL (550 reject) 99 : message UNSAFE to be silently (250 code) discarded at the SMTP LEVEL EX_TEMPFAIL (75) : message not processed successfully (error in communication, or server error, ...)