Sending large newsletter is slow

Use this forum for questions regarding adoption and functionality of OpenEMM

Moderator: moderator

Cybex
Posts: 4
Joined: Mon Mar 18, 2013 12:38 pm

Sending large newsletter is slow

Post by Cybex » Mon Mar 18, 2013 12:52 pm

Hi all,

This is the first time I am sending a large newsletter (over 520,000 recipients) with OpenEMM 2013. The performance isn't good, it takes too long to send all mails. The server has been sending mails for 22 hours now and only 133,000 mails are sent. It's a dedicated server with OpenEMM installed.

Server is an Intel Core i7 (2600) with 16GB memory installed.

Is there anything I can do to speed up the sending process?

Kind regards, John

Fatalin
Posts: 111
Joined: Wed Aug 05, 2009 10:51 am
Location: Troisdorf, Germany
Contact:

Re: Sending large newsletter is slow

Post by Fatalin » Tue Mar 19, 2013 9:41 am

Hi John,

Depending on your mailing-list quality it could be the case that your server get clogged by a large number of soft bounce related server messages (from the receiving mail-servers).

You may want to check your mailing-statistic. The bounce rate on the main statistic page shows only the hard bounces. If you click on the bounces-link you can also see the soft bounces. If there are several thousand soft bounces those server messages could block your server. It means that the send-out is slower than expected, but it still takes place.

You can also check your log files to see if there are many grey-listings and the like, that are additional causes for delay.

These things can always happen, if your mailing-list is not really clean. The problem will solve itself, through a number of future newsletter sendouts, because many of your current soft bounces will ultimately hard bounce and then be excluded from future send-outs. When this happens depends on your system settings. The default soft bounce -> hard bounce count is 7 (I believe). Scan the forum for further info on this bounce counter.

You can also check, if your server is on a black list. An easy way to check this is: http://zy0.de/q/YOUR_IP_ADDRESS

Example: http://zy0.de/q/46.163.77.12 for the Agnitas Newsletter.

I hope this helps you a bit.

Best Regards
Markus
Günstige Software gibt es bei http://www.Software-Butler.de

Cybex
Posts: 4
Joined: Mon Mar 18, 2013 12:38 pm

Re: Sending large newsletter is slow

Post by Cybex » Tue Mar 19, 2013 9:44 am

The bounces only work when the server is also receiving mail, right? I mean, the mx records in our DNS is not set to the OpenEMM server.

Fatalin
Posts: 111
Joined: Wed Aug 05, 2009 10:51 am
Location: Troisdorf, Germany
Contact:

Re: Sending large newsletter is slow

Post by Fatalin » Tue Mar 19, 2013 10:16 am

I believe the answer is yes and no.

The bounces, that the OpenEMM bounce-filter processes are EMAILS, return from a receiving mail server. Those may not be blocking your sending server.

However, there are a number of cases, were bounce situations do block your sending server. Some examplese are:
  1. Grey listing
    The receiving mail server "does not know your server" and requires a resend in 5/15/60 minutes (whatever the receiving server is set to). These SERVER MESSAGES are processed by your sending server, because they are related to the send-out. They also cause additional mail send outs, that are not show in the statistic
  1. Soft Bounce
    There are many different reasons why soft bounces occur. In some cases the server communication between sender and receiver can be clogged due to too many server messages.
Regards
Markus
Günstige Software gibt es bei http://www.Software-Butler.de

maschoff
Site Admin
Posts: 2197
Joined: Thu Aug 03, 2006 10:20 am
Location: Munich, Germany
Contact:

Re: Sending large newsletter is slow

Post by maschoff » Wed Mar 20, 2013 10:04 pm

You should check Sendmail's maillog from time to time if you are not happy with the performance to make sure there are no nasty surprises.
OpenEMM Maintainer

Cybex
Posts: 4
Joined: Mon Mar 18, 2013 12:38 pm

Re: Sending large newsletter is slow

Post by Cybex » Thu Mar 21, 2013 7:54 am

Thank guys, I'll check it to see how I can speed up this process the next time. It's still sending mail now :-)

Cybex
Posts: 4
Joined: Mon Mar 18, 2013 12:38 pm

Re: Sending large newsletter is slow

Post by Cybex » Thu Mar 21, 2013 10:42 am

Where can I find the sendmail log? When I monitor /var/log/maillog I don't see anything weird, except that lot's of mail being send.

Where can I see softbounces?

maschoff
Site Admin
Posts: 2197
Joined: Thu Aug 03, 2006 10:20 am
Location: Munich, Germany
Contact:

Re: Sending large newsletter is slow

Post by maschoff » Thu Mar 21, 2013 10:53 am

This is the maillog. Softbounces are recorded in the database (see appendix in extension guide) and the workflow is like this:

- Sendmail log realtime scanner slrtscn.py parses sendmail's log files in /var/log/maillog for instant bounces and writes identified bounces in file ~/var/spool/log/extbounce.log and state of processing (cursor) to ~/var/run/slrtscn.save.
- if you have configured OpenEMM for processing of delayed bounces, email replies are forwarded to Procmail (via configuration in mailertable).
- Procmail analyses via control file bav.rc (bounce address validation filter) email replies with ruleset in file bav.rule and forwards result to bavd (bounce address validation daemon).
- bavd writes copies of all incoming replies to ~/var/spool/filter and filtered bounces in file ~/var/spool/log/extbounce.log.
- update.py reads and processes extbounce.log and writes result to bounce_tbl.
- Softbounces in bounce_tbl are processed by softbounce.sh and written to softbounce_email_tbl. Softbounce rules are in script softbounce.py, beginning at about line 200.
OpenEMM Maintainer

Post Reply