Stress testing HCL Domino and your "Other" Mail Infrastructure

 How does 1.2M emails sound?

There are people out there who say HCL Domino can't handle the stress of modern times.

It is an old system(truth be told, Exchange isn't much younger) with limitations.

Well, yesterday, I did an unintended stress test of Domino and a client's internal infrastructure.

Like most large and well-known companies, they run many Domino applications that handle millions of dollars a day but also have an O365 infrastructure.

Mail doesn't come into Domino, but it does go out from there, and this is where it got interesting.

Domino sent over 275,000 emails over about 15 minutes +/-.

This was all internal SMTP, so we didn't get spam blocked or anything,

Normally, a company would have a choke hold option on mail sent per minute, but for some reason, that was not in place or was avoided due to routing configurations yet to be determined.

And my 275K grew to 1.2M inside the network architecture.

Fun day at the office, right?

No, I caused the problem and also knew how to fix it, but that small window of time was enough to unleash the tsunami.

Sometimes, the simplest things become the hardest things, and this was such a case. 

Even veterans of the email wars screw up.

I now have a great session for F*uck Up Nights if anyone is interested.

Definitely used up one of my IT 9 lives, this brings me down to about 5 left.

HCL Domino was amazing throughout. Not only did the servers not complain about anything, but they just kept going and going and going. Multithreading FTW!

O365/Exchange did not do so well, got clogged up and had issues because it still is not a multithreaded service.

I pulled the numbers from the Mail statistics and showed them on my Admin client Monitoring dashboard. The left side is tasks, which I covered in my previous blog post. On the right side, you can add statistics, something not many people do, but I like to see some things there and probably should do a separate session on the statistics, but that is for another time.

My peers, I am sure, can guess what caused the tsunami, so there is no reason to elaborate. But let's just say when you are a junior admin, this is one of the outcomes of your trial-by-fire Domino Administration education.

For those who think Ambassadors and long-time Yellowbleeders are these great Gods of tech, some really are titans, I admit my mistake and that, indeed, you are never too young or old to learn something new....or mess up royally.

A Tale of 2 Mail Servers Threading

This little server is an old Dell rack server, about 4 years old maybe and runs Windows Server 2003 and is up to date on all patches and fixes for the NOS and the Mail application.

The other is brand new (under a year) and runs Windows Server 2008, because it had to, and is supposed to be up to date as well.

The Old server is a happy go lucky type of server, runs mail, instant messaging and push email/mobile solutions in addition to a handful of applications for 6 domains.

The New server is also happy but not so lucky and ONLY runs mail for 2 domains.

The Old server is the primary work horse, all mail comes into it from the SPAM appliance, internal and external mail as well. Attachment sizes are reasonable(25MB) for sending and unlimited for incoming. We encourage good email habits this way.

The New server has no limits on sending or receiving. Here lies the problem.


If the New server needs to send an attachment that is 50MB, the server will refuse to accept any emails from the Old server while it waits for it's threading to clear from sending this monster file. This is bad. Very bad.
 
We do not manage the New server yet perpetually must explain how SMTP works, why security like TLS is important and why size matters for attachments.

I had no idea Exchange was single threaded, thought for sure by now it was mutli threaded, especially that beast of a 64bit server, but evidently not the way these guys built it.

Anyone have some insight on this I will be more than happy to point the New server team to how to resolve this.

In the interim Domino is perfectly happy to keep holding their mail in queue and DDM is sending us notifications about the issue, as it should be, so we are monitoring their servers in addition to ours. Should start charging them for it.