Maximum throughput

[chumpster]

I wonder if anyone can tell me the 'theoretical maximum' throughout for OpenEMM. I know many people are more interested in 'throttling' throughput, but I'd like to see if anyone knows the other end of the scale.

To qualify this, I'm not talking about deliveries to end users, or being out on the Internet with ISPs 'tar pitting' deliveries and IP addresses etc. Let say for theoretical sake, you're on a Gigbit LAN and sendmail is handing off delivery to a steroid pumped up smarthost that can take everything you give it in a snap and you're running a DB on a second server or disks. I hope you get the picture of an OpenEMM install running full throttle... it's not that I have such a thing, but people often read into a question something that's not there. I'm trying to gauge OpenEMM's theoretical performance limits, before considering external bottlenecks that might constrain it's real world performance.

By comparison, PHPList is a single threaded app. and they honestly say it has a maximum throughput of ~5,000/hour.

thanks....

[maschoff]

OpenEMM's hosted version eMM-Xpress (www.emm-xpress.com) guarantees 10,000 mails/hour in its TOS.

[emmulator]

I hadn't realized it might be so slow -- is there anything that can be done to make it go faster? What is the limiting factor keeping it to 10k/hour? We're using OpenEMM right now to support Mailings scheduled by users in our network, and we're starting to get backed up. When I look at the hourly throughput for the day (grepping the logs), I see the following counts getting passed along to our IronPort:

May 15 00: - 3230
May 15 02: - 5786
May 15 03: - 10889
May 15 04: - 4308
May 15 05: - 3569
May 15 06: - 2150
May 15 07: - 3532
May 15 08: - 9299
May 15 09: - 4155
May 15 10: - 4804
May 15 11: - 11057

Our users are scheduling up to around 20k/hr -- is there anything in particular we should be looking at to increase the throughput?

[emmulator]

Our solution was to edit the pickdist.py file at the following function:

def queueIsFree (self):
return len ([_f for _f in os.listdir (self.queue) if _f[:2] == 'qf']) < 5000

We replaced that 5000 with a 20000, to allow OpenEMM to place more emails into the Sendmail queue. We are now sending about 30k/hour, which is the limit imposed by the administrators of the IronPort we are using as a relay.

[maschoff]

Normally, soft bounces coming back from the receiving mail servers are what drags down sending performance. I would rather guess that your bigger hardware is responsible for the improved throughput. We know users who make 50,000 to 60,000 mails/hour on multi CPU machines.

In our opinion increasing the maximum size of the Sendmail queue beyond 5,000 mails does not help but could even drag down performance.

[emmulator]

It seems to me that what happened was the 5k baseline was filled up with bouncing emails, preventing new emails from getting put into the queue at a high enough frequency to achieve the needed throughput. Once we upped that limit for the queue, there was space for new emails to be put on the queue, even if there were lots of bounces sitting there for retry.

[snailsites.com]

hi openemm team,
would be cool if you read this:
first thanks for writing this
and second: our first mailing about 50k recipients takes about 24hours and still 10k mails to send
restarted openemm some times but this didnt help
than i read about to push up this 5k limit and pushed it to 50k
restarted and voila: in the next seconds the mailing was finished, the last 10k mails were sended

so now the things i wish:
parameterize this, put a variable in emm.properties or sth
repair the dkim stuff, this could give the key points to went not in spam folder
give me form where i can set blocksize and stepping as default for all mailings

THANKY YOU

[anondev]

Of course you can raise that cap on mails sent, but you will find yourself blacklisted in no time among all the major email providers (gmail, hotmail, etc...). You will be labeled as a spammer, because one shouldn't try to send that many emails within such a short timeframe (50 000 mails in a second, srsly?). You should read about deliverability.

[maschoff]

Based on our experience this is absolutely true!

[GrandG]

Hi,
Has anyone done any sort of benchmarking with something like smtp-sink or bhm (from postal)? I guess I'm also wanting to get an idea of system limits like the OP - if you get rid of all limits, plonk this behind a black hole for testing, what could it get up to?
Let's say we have a reasonably spec'ed DB server (say 2x hexacore XEON, 16 gig of RAM, with a decent RAID 5 or 10, or on a high-speed SAN or whatever, in short, fast!) and a reasonably spec'ed OpenEMM server (say 2x hexacore XEON, 16 gig of RAM with a decent RAID 5 or 10) - if sendmail were pushing to a super-highspeed relay (like /dev/null, for example), what could it get up too? I suppose it is very dependent on content, a small 3-5 KB email with 1-2 user-defined fields at one end up to a 50-60 KB email with 50 user-defined fields in it on the other. 100k/h? 500k/h? 1M/h?
Has anyone benchmarked anything like that? What would be issues to look out for?
I realise this is like asking about how fast a car can go - in real situations you shouldn't ever get that fast, but there are always situations where it might be interesting, like the Autobahn...
Cheers
GG.