List:       postfix-users
Subject:    Re: 1.000.000 email posted in a single delivery process
From:       wietse () porcupine ! org (Wietse Venema)
Date:       2001-09-26 21:14:15
[Download message RAW]

If you want to deliver 1000000 queue files in 3 hours, that is 10
milliseconds per file. That seems a bit optimistic to me, considering
the latencies of real disks (every file needs to be read from disk,
the cache is cold) and of real networks (having a local DNS server
should really make a difference).

Neither qmail nor Postfix currently has connection caching, which
can make a difference sending mail to sites that have slow servers.

0 - In the /etc/syslog.conf file, put a - in front of the maillog
    filename, so that the stupid syslogd does not hammer the disk
    after writing each logfile record.

1 - The default settings of two levels of directory hashing in the
    queue should suffice. If you disagree, change this with:

    postconf -e hash_queue_depth=3
    postfix reload

2 - Configure an adequate number of delivery agents in the master.cf
    file: 500+.  On your typical Linux box, this will run the kernel
    out of resources and Postfix will slam on the brakes until you
    fix it.

3 - Test the setup with the smtp-source and smtp-sink programs.
    For the test, specify a relayhost name that points to the
    smtp-sink program, and send messages with thousands of recipients
    into Postfix, and have Postfix perform one-recipient deliveries.

    On the Postfix host:

    postconf -e relayhost=smtp-sink.host:9999 \
        smtp_destination_recipient_limit=1

    On the mail sink host (can be same as postfix host, if you have
    enough memory and process slots and space for open files):

    ./smtp-sink -c :9999 1000

    On the smtp-source host (can be same as postfix host):

    ./smtp-source -c -m 1000 -s 10 -r 10000 -t foo@postfix.host postfix.host

    Postfix breaks up the deliveries into transactions of 1 recipient
    each, so one message with 10000 recipients will keep lots of
    delivery agents busy.

4 - Before submitting the real mail:

    postconf -e defer_transports=smtp relayhost= \
        smtp_destination_recipient_limit=\$default_destination_recipient_limit
    postfix reload

    (and don't let anyone execute a "sendmail -q" command).

5 - SUBMIT THE MAIL VIA SMTP, not via /usr/sbin/sendmail. Otherwise
    you are throwing away all the performance.

6 - On Doomsday,

    postconf -e defer_transports=
    postfix reload
    sendmail -q

7 - From here on, do not touch the box. Each time you stop or reload
    Postfix, or do a sendmail -q, it slows down dramatically.

    Wietse

Alexander:
> (I've posted this message in QMAIL and EXIM mailing list.
> Many people say me that POSTFIX can do a good work)
> 
> This is a case study and it's NOT for create a spamming system or other
> terrible mail bombing system.
> I HATE them !
> 
> Title: the best mode to deliver 1.000.000 email messages (recipient are ALL
> different and email are ALL different) in few hours during a single delivery
> session.
> 
> You suppose to have a SMP Linux server (i.e. dual P-III 800 Mhz, 512 MB Ram,
> SCSI Raid 5, kernel 2.4.9 and RaiserFS) and a good internet connection (4
> Mbit/sec full bandwidth) and a fast DNS server near me.
> 
> During the night a program running on the same server creates 1 million of
> email (getting user data and infos data from a SQL DB).
> This process need about 3 hours. Meanwhile email are created, they are
> "injected" in MTA queue (i.e. using qmail-inject or
> exim -odq -t) and they are NOT delivered: they stay in queue.
> The email are all different but short: about 800-1000 bytes each and the
> recipients are all different and are all remote.
> Yes, we can have a vary big number of different recipient domain but
> statistically we also have a large number of email directed on mass email
> system (Hotmail, Yahoo...)
> 
> So 1.000.000 x 1.000 bytes = 1 GBytes data. Yes, like a technical
> information email system with personalized info for each recipient.
> 
> Then every day at 8.00 AM the remote delivery needs to start (exim -q ,
> qmail-remote) and delivery is done once per day.
> 
> Target: send 1 million of email in a very short time, shortest as possibile,
> it must end as soon as possibile. Bandwidth occupation is not a problem: we
> can use all 4 Mbit/sec (about 1 GB / 500 Kbytes/sec = about 2000 seconds , 1
> hour and few minutes. 2 or 3 hours are ok anyway).
> 
> And now the questions:
> 
> 1) QMAIL vs EXIM vs POSTFIX: which is the best MTA to manage 1.000.000 email
> in queue ?
> (EXIM with option "split_spool_directory" and QMAIL patched to have a more
> "deep" in queue tree, POSTIFIX ??)
> 
> 2) EXIM can deliver all messages during a single SMTP session to the same
> email system (useful for mass system like HOTMAIL,YAHOO, ....), QMAIL no.
> Is it faster to have either for example 300 (more or less) qmail-remote
> process running and sending 1 messages each or  for example 20 (more or
> less) "exim -q" process running  sending more messages in a single SMTP
> connection ? Remember: the target is to use less time than possible.
> 
> Thanks for any suggestion about.
> POSTFIX ,I've never used it, can be an alternative ?
> 
> 
> 

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users