Mailing not being sent - have to delete QUEUE

[atego]

A couple of times now mailings have been started but appear to stall, with no emails apparently being sent. For example, a recent mailing was set up and sent and the distribution status showed that 9,022 emails were generated, but after several hours zero had been sent. This is not usual behaviour but had happened once before. I checked /home/openemm/var/spool/QUEUE and saw over 11,000 items in the directory. I checked /home/openemm/var/spool/META and I could see .xml.gz, .stamp and .final files that appeared to be related to this mailing. I checked /var/log/maillog and could see nothing obviously wrong. I rebooted the server, I restarted OpenEMM and I checked /home/openemm/bin/scripts/pickdist.py process was running.

In the end I resorted to stopping openemm and deleting the contents of /home/openemm/var/spool/QUEUE. When I restarted openemm the QUEUE immediately refilled with 11,000+ files but the distribution status now showed that 9,022 emails were generated and 6,001 sent.

What I would like to know is:

1. What is the actual consequence of deleting the contents of the QUEUE?
2. Is it likely that the 6000 emails were actually sent or is this just because I removed the messages from the QUEUE?
3. Why does OpenEMM get into this state where deleting the QUEUE appears to be the only corrective action?

My assumption/hope is that by deleting the contents of the QUEUE, something that was causing OpenEMM to fail is removed and clears a blockage. I also hope that doing this simply causes openemm to requeue and attempt to resend the messages in the mailing.

My fear is that when I delete the QUEUE, OpenEMM interprets this as the messages having been successfully sent and so updates the statistics in the distribution status accordingly; thus presenting a false impression of the mailing status.

Any help or advice would be appreciated.

[maschoff]

If you empty the queue Sendmail can no longer send out these mails but - since the queue is empty - OpenEMM thinks the mails have been sent.

The real question is why the queue was full - probably because Sendmail coult not send out mails due to blocking MTAs. Please check the Sendmail logs for details.

BTW, if you have the notion that OpenEMM is hanging you can restart the application and the recovery process will take care that the sending of unfinished mailings will be resumed.

[atego]

Thanks Maschoff. That makes sense.

In the end I increased the queue size by editing /home/openemm/bin/scripts/pickdist.py. This seems to have prevented the mailings from stalling and so large mailings now seem to complete without my intervention.

However, I now have a different, but probably related issue. Despite OpenEMM reporting that the mailing has ended at a given time, the queue never appears to empty (or reduce to the kind of numbers of items I would expect). We just sent a mailing with 80,000 recipients and it is reported to have completed in OpenEMM. Yet the queue still contains about 20,000 items relating to that mailing - these are message parts (headers and body) from that mailing. Why would that be if the mailing is complete? I can also see what appear to be NDRs from the system reporting delivery failures but there are fewer than a 1000 of these so I can't understand why so many items are left over.

Can I assume that, given enough time, the queue will eventually empty?

[maschoff]

The OpenEMM GUI actually reports the number of mails distributed to the mailservers (in OpenEMM's case that is just one). If a mail has been delivered to the mailserver (Sendmail queue) but could not been send out due to a softbounce it is still reported as sent because the OpenEMM GUI gets no feedback from the mailserver(s).

[atego]

Thanks again. I now fully understand.

We were getting many soft bounces and this was causing congestion in the Sendmail queue. It seems that the queue was getting stuck because it was full of messages that were bouncing but Sendmail was continually retrying those same messages. Upon re-reading the documentation I found that by default Sendmail will retry for 5 days. I have therefore reduced the time out to 1 day before messages are purged from the queue as undeliverable. This appears to have allowed the queue to be processed more efficiently.

[maschoff]

Yes, depending on the ISPs you send to and the reputation of your mailserver IP the softbounce rate can be neglectable or really big.

[atego]

Maschoff,

It would appear that our sender reputation is pretty good - if you can believe the reputation databases/services on-line. I think it is quite likely that our contacts database needs some work and so hopefully the bounce management will start to have some effect there.

[manoj]

Hi,

In this post it is mentioned that by increasing the queue size by editing home/openemm/bin/scripts/pickdist.py restarting of openemm was prevented. But we are unable to see any parameters for this file. Kindly let us know the procedure for the same.

Also please suggest any thing that can prevent from restarting the openemm to push emails. Unless we restart every hour never the mails are being delivered. And we provisioned a powerful server of which not even 10% of RAM was ever found being used.

Appreciate your advice.

Regards,
Manoj