Postfix aliases and duplicate e-mails, how to fix?

Question

I have aliases set up in postfix, such as the following:

all@mydomain.com:    foo@mydomain.com, bar@mydomain.com ...

When an email is sent to all@mydomain.com, and any of the recipients in that alias is cc:ed which is quite common (ie: "Reply all"), the e-mail is delivered in duplicates. For instance, if an e-mail is sent to all@mydomain.com and foo@mydomain.com is cc:ed, it'll get delivered twice. According to the Postfix FAQ, this is by design as Postfix sends e-mail in parallel without expanding the groups, which makes it faster than sendmail. Now that's all fine and dandy, but is it possible to configure Postfix to actually remove duplicate recipients before sending the e-mail?

I've found a lot of posts from people all over the net that has the same problem, but I have yet to find an answer. If this is not possible to do in Postfix, is it possible to do it somewhere on the way? I've tried educating my users, but it's rather futile I'm afraid...

I'm running postfix on Mac OS X Server 10.6, amavis is set as content_filter and dovecot is set as mailbox_command. I've tried setting up procmail as a content_filter for smtp delivery (as per the suggestion below), but I can't seem to get it right. For various reasons, I can't replace the standard OS X configuration, meaning postfix, amavis and dovecot stay put. I can however add to it if I wish.

Answer 1

There is a way how to do it, but not using Postfix itself.

If you are using a reasonably recent Dovecot with Pigeonhole implementation of Sieve, there is a special extension that can be used for on-delivery deduplication. In Dovecot older than 2.2.18, the name of this extension is "vnd.dovecot.duplicate" and it's disabled by default, in newer releases the extension is called "duplicate" and should be already enabled, but enabling it explicitly won't harm.

/etc/dovecot/conf.d/90-sieve.conf:

plugin {
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
sieve_before = /mnt/mail/users/global_sieve/deduplicate.sieve
sieve_before2 = /mnt/mail/users/global_sieve/antispam.sieve
sieve_extensions = +vnd.dovecot.duplicate  # for dovecot < 2.2.18
  #sieve_extensions = +duplicate             # for dovecot >= 2.2.18
}

Then create the sieve script to handle the deduplication itself (you can adjust the filename as you see fit).

/mnt/mail/users/global_sieve/deduplicate.sieve:

require "vnd.dovecot.duplicate"; # for dovecot < 2.2.18
# require "duplicate";           # for dovecot >= 2.2.18
if duplicate {
    discard;
    stop;
}

Compile the script using sievec and make sure the script is readable by dovecot user. More in Dovecot docs v2.3 / current.

If you are using Cyrus, there is a a duplicate message delivery suppression feature that can be enabled using suppress_duplicates = yes.

Answer 2

Postfix has no idea about duplicate emails due to the way it's structured. It is possible to do what you're suggesting by using procmail as your delivery agent.

Essentially, each message coming from a client should be delivered with a unique Message-Id. In the case that it's delivered to multiple people, the Message-Id should be the same, so we save any Message-Id headers we've seen and discard and future ones that match that list.

From http://novosial.org/procmail/

:0 Wh: msgid.lock
| formail -D 8192 ~/.procmail/msgid.cache

Answer 3

A solution that work for me is add -o receive_override_options=no_address_mappings on master.cf

Here the doc: http://www.postfix.org/postconf.5.html#receive_override_options

Answer 4

I was facing the same issue. I tried using the pigeonhole (sieve) approach but it resulted more complicated for such a minor feature (prevent duplicates when storing and forwarding at the same time). I also tried most of the suggestions here without success (is not that they are mistaken, they just didn't work for me).

One way to solve this is to clearly identify aliases vs mailboxes. If you are using emails as usernames, I guess you need to adjust this method. In my case, user accounts are like: user1000 for the email user@example.com. This is how to solve it:

1. In the configuration where you get the directory path for the user (e.g. mysql.users.cf), you have to be sure to use mailbox accounts only (not email addresses) to search:

SELECT home_dir FROM accounts WHERE CONCAT(username, '@', 'mail.example.com')='%s'

It should return the full path to the inbox if exists, or nothing if the username doesn't exists or is just an alias for another email (account without mailbox). You can test with this command:

postmap -q 'user1000@mail.example.com' mysql:/etc/postfix/mysql.users.cf

(please note that we use user1000@mail.example.com and not user@example.com)

NOTE: mail.example.com is the FQDN of your mail server and it is used to identify mailboxes from incoming emails and external accounts. You can not use your domain (e.g. example.com) here as postfix won't be able to tell the difference between incoming emails and forwarded ones (resulting in duplicate emails). You could use localhost instead, but postfix will complain as @localhost is used mainly for OS physical accounts.

2. In the alias configuration file (e.g. mysql.aliases.cf), return the addresses you want to store or forward to:

SELECT address FROM aliases WHERE email='%s'

You can return a list of addresses (one by line) or a single line separated by comma. You can test with this command:

postmap -q 'user@example.com' mysql:/etc/postfix/mysql.aliases.cf

Now, the trick here is that you must return only email addresses , do not return account names here, instead append your server FQDN as explained above. For example, the previous SQL should return something like (the order is not important):

SELECT address FROM aliases WHERE email='info@example.com'
sam_t100@yahoo.com   <-- external
peter123@mail.example.com  <-- internal/local
info4562@mail.example.com  <-- internal/local
root@mail.example.com      <-- internal/local
rose@gmail.com      <-- external

(note that we are returning local mailboxes as *@mail.example.com as explained above)

In this way, postfix can clearly identify what should be stored and what should be forwarded and duplicates emails are no more.

Answer 5

This is from some old postfix faq:

Postfix sends duplicate mail Some people will complain that Postfix sends duplicate messages. This happens whenever one message is mailed to multiple addresses that reach the same user. Examples of such scenarios are:

• One message is sent to the user, and to an alias that lists the user. The user receives one copy of the mail directly, and one copy via the alias.

  • One message is sent to multiple aliases that list the user. The user receives one copy of the mail via each alias.

Some people will even argue that this is the "right" behavior. It is probably more a matter of expectation and of what one is used to.

This can be "fixed" only by making Postfix slower. In the above examples, Postfix would first have to completely expand all distribution lists before starting any delivery. By design, Postfix delivers mail to different destinations in parallel, and local delivery is no exception. This is why Postfix can be faster than sendmail.

So by design you are seeing that behavior. Perhaps if you find a content filter that can strip duplicate message IDs, you can eliminate this after the delivery event.

Sendmail does not have this problem because it expands everything first and strips out duplicates.

Answer 6

The oficial solution is here.. http://osdir.com/ml/mail.postfix.devel/2007-05/msg00010.html

duplicate_filter_limit (10000) The maximal number of addresses remembered by the recipient duplicate filters for aliases(5) or virtual(5) alias expansion, or for showq(8) queue displays (with earlier Postfix releases the default limit was 1000).

duplicate_filter_style (strict) The duplicate recipient filter policy: strict or pragmatic.

Answer 7

enable_original_recipient=no

I put that in my main.cf and it worked fine but only for mail sent from my domain. I still have duplicates if I send email from outside my domain (yahoo to my domain for eg.)

source : http://article.gmane.org/gmane.mail.postfix.user/119783