ClamAV and Domino

 I am not an expert on Anti-virus things and Domino, that would be Daniel Nashed.

But if HCL Domino has something new, I test it out and set it up on my server.

In this case, I had heard that ClamAV would now work with Domino as of 14.5.1.

Having used Clam for many years at the OS level, I wanted to play with this in Domino.

My server is Windows, not Linux, ClamAV Linux instructions are pretty common, Windows, not as much.

Boy was this the hardest "easy' thing to do ever.

To start, HCL provides no documentation.

There are only 3 references to ClamAV in the official docs:

https://help.hcl-software.com/domino/14.5.1/admin/wn_config_features_1451.html#wn_config_features_1451__section_fq1_wgc_23c

https://help.hcl-software.com/domino/14.5.1/admin/conf_configuringscanninginscscancfg.html?hl=clamav

https://help.hcl-software.com/domino/14.5.1/admin/conf_scanningattachmentsforviruses.html?hl=clamav

None of which tell you much about how to set up ClamAV or how to configure it for Domino integration.

I opened a ticket and eventually was directed to this post:

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0129703

Which is in Japanese. Use Google Translate or Chrome built in translstion.

I see now, at my urging, HCL has released the same document in English.

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0131220

Their perspective is that we should go to ClamAv to set it up and then magically know how to integrate it.

I asked why it wasn't in the actual documentation and got a runaround answer, given HCL announced the availability, not ClamAV.

The new technical document covers some of the set up required.

I am going to add what is missing, more for my benefit, and anyone else who will try this on their servers.

Step 1 is Download the ClamAV from the link in the document, https://www.clamav.net/downloads and install it. BUT this does not tell you how to set it up in Services.

For that, you perform this:

Open the Windows Command Prompt as an Administrator and navigate to your ClamAV folder.

Run the following command to install the ClamAV daemon service:
clamd.exe --install

Remember to set it to automatic.

Step 2 is to copy the config file as described and perform the edits.

Step 3, however, did not work for me no matter how I tried to get the certificate. Even with HCL Support help, they eventually had to send me the certificate required.

Step 4: You have to get the updated virus definitions. NOTE: This is a once off effort. And Manual.

If you want it to be automatic, and you do, you need to do these steps.

Open the Windows Command Prompt as an Administrator and navigate to your ClamAV folder.

Run the following command to install the ClamAV daemon service:
freshclam.exe --install

Once you Load mailscan at the Domino server console you will get the cscancfg.nsf and need to follow the document entries as provided.

NOTE: The ClamAV server name(DNS) must be 127.0.0.1. Do not put your server IP there, it will not work at all.

Make sure you open port 3310 with your local server firewall, and in my case, the outside ISP.

I changed the Subject Prefix Scanned field default text to something I would know came from my server, not a spammer.

Lastly, make sure you add Mailscan to your tasks line in your notes.ini also something not mentioned in the document that is kind of important, unless you like having your mail stuck in mail.box and spend 2 hours troubleshooting it one morning, like I just did.

I eventually figured out it was an AV issue when running SMTPDebugClient=1 and

Tell Router list run from the server console.

You see this:

Mbx NoteID          ID          State          Size Pri Count ScheduledDate           From

  1 000008FE 0028261D Wait AV        8443         1                         K Brooks/org

  2 00000902 00283C39 Wait AV       13306         1                         kbrooks@test.com

  1 00000902 002864E3 Wait AV        8749         1                         keith@gmail.com

  2 00000906 00287C9A Wait AV        8640         1                         keith@gmail.com

Oh, that makes sense after the server restarted late yesterday. Mailscan was not in tasks.

So, my time spent, is your time gained.

Stress testing HCL Domino and your "Other" Mail Infrastructure

 How does 1.2M emails sound?

There are people out there who say HCL Domino can't handle the stress of modern times.

It is an old system(truth be told, Exchange isn't much younger) with limitations.

Well, yesterday, I did an unintended stress test of Domino and a client's internal infrastructure.

Like most large and well-known companies, they run many Domino applications that handle millions of dollars a day but also have an O365 infrastructure.

Mail doesn't come into Domino, but it does go out from there, and this is where it got interesting.

Domino sent over 275,000 emails over about 15 minutes +/-.

This was all internal SMTP, so we didn't get spam blocked or anything,

Normally, a company would have a choke hold option on mail sent per minute, but for some reason, that was not in place or was avoided due to routing configurations yet to be determined.

And my 275K grew to 1.2M inside the network architecture.

Fun day at the office, right?

No, I caused the problem and also knew how to fix it, but that small window of time was enough to unleash the tsunami.

Sometimes, the simplest things become the hardest things, and this was such a case. 

Even veterans of the email wars screw up.

I now have a great session for F*uck Up Nights if anyone is interested.

Definitely used up one of my IT 9 lives, this brings me down to about 5 left.

HCL Domino was amazing throughout. Not only did the servers not complain about anything, but they just kept going and going and going. Multithreading FTW!

O365/Exchange did not do so well, got clogged up and had issues because it still is not a multithreaded service.

I pulled the numbers from the Mail statistics and showed them on my Admin client Monitoring dashboard. The left side is tasks, which I covered in my previous blog post. On the right side, you can add statistics, something not many people do, but I like to see some things there and probably should do a separate session on the statistics, but that is for another time.

My peers, I am sure, can guess what caused the tsunami, so there is no reason to elaborate. But let's just say when you are a junior admin, this is one of the outcomes of your trial-by-fire Domino Administration education.

For those who think Ambassadors and long-time Yellowbleeders are these great Gods of tech, some really are titans, I admit my mistake and that, indeed, you are never too young or old to learn something new....or mess up royally.

SMTP BlackListing, WhiteListing and Log and Reject/Tag

If you rely on your Domino server to handle all your mail, you probably have had numerous attacks on your server over time or even lately, as I did last week.

My personal Domino server is a mix of real code, websites, and active email, with various half-coded things and weird templates or customer testing.

However, I started getting harassed by sites looking for open SMTP accounts recently and figured something was amiss in my configuration document.

The official blacklist servers worked fine, but some of these rogues were missing.

Looking at my log file, I found a few domains/IP addresses and put them into the deny access group known as the Private Blacklist Filter found in the Servers Configuration document, as shown below.

But that wasn't enough to stop them. They kept coming. 

I wondered if 12.02.FP2 had some problems, so I opened a ticket with HCL.

Turns out the problem was on my end, but I still have some questions, but first, what was the problem?

I had a default configuration document, which was fine, but I  also had a separate one for my server explicitly named a relic from a test issue.

The explicit one took over the default one, and so while I thought I was maintaining one list, I was wasting my time.

I deleted the explicit one and just focused on the default document, it is my server after all.

And all was good, sort of.

I wanted to understand why I was still getting a few spam emails.

I had set the server to Log and tag instead of Log and reject. 

Here is where the problems got worse.

I decided to block all spam and set all fields to Log and reject messages. You probably can guess what happened next.

My inbox was very clean. Very few emails came through.

I thought I would whitelist what I needed, like bank mail, and HCL support mail (not so simple, someone at HCL should look into their SMTP issues that have them on a blacklist).

Still not getting lots of mail.

Next, I looked at what else was set in the doc and saw the verify domain lookup option was set, and rightly so as this does a great job.

However, I have learned that many organizations don't have good, clean SMTP/DKIM/SPF entries, and thus, they are getting blocked.

Sadly, I had to revert back to Log and tag to interact with customers and business partners.

Customers of mine with issues were notified, as was HCL, but if you have been playing with SMTP, something else always pops up. It needs babysitting.

While my mail is more stable now, I know I lost a few entities that got the denied server message and probably will not resend anything in the future. Which is a problem as some are bills and other items of usefulness.

If you are a new Domino administrator be careful with how you edit your Configuration document.

Email, and Email Services, are not a Commodity

Do you think Email is a commodity, but SMTP service is not? If you think it is, have you planned for what happens when your primary mail service, including SMTP goes down?

I doubt it, because most people do not even realize how email is broken down into parts that only a small bit are the responsibility of your own IT team.

When you have in house email, everything, except the ISP providing your bandwidth, is your own world. You break it, you fix it. You do, or do not, plan for business interruptions. Maybe you have a secondary ISP, even if it is some lousy DSL line, you have something, anything. You do, right?

What about when you get to cloud apps and you have a LDAP in one service or server, your app on one server, your mail transport on another, mobile site on another, your company website on yet another one.

When any of those parts fail, most of the rest fails and you have zero capability to resolve it. Zero. You are at the mercy of whatever Cloud provider, Host facility, kid with an iPhone controlling your VMs and whenever they get around to fixing your problem they will.

Utility? Commodity? Sure, but so are cars, yet many people will not drive a (fill in the blank for your country) because of the lousy service record or safety. Why is your Cloud provider any different.

You may ask about clustering, fail over, etc.. but that does mean you will get an answer that makes sense to you or to them sometimes.

Email is still the #1 business tool because so many rely on it. Do they have to rely on it? Not at all, but they have not made the leap yet to extended applications that notify via other methods. Maybe they did take the leap, but our customers have not yet. The customers expect to be able to interact with us through an old medium which is so easy even the term used, SMTP stands for Simple Mail Transfer Protocol, makes it sound easy. Can you plan for every contingency? No, sometimes stuff really happens you can not work around easily. But if you believe all your IT services are just commodities, then you will get what you pay for and should not expect better.

Running a mail server, and related services is not so simple and takes people that truly understand what they are looking at when they get RFC codes in reply and other "errors" as users report them.

I know, I have been doing it for 20 years, IBM, Microsoft, Google, Novell and so many long gone that it is hard to really believe anyone when they say they have a new email app. Email is an app, a huge one, and though you may argue it should just do email, we both know it does so much more and can do so much more. The problem is when the other services, which most people take for granted, stop working and your commodity or utility is unable to do anything.

Makes for a really different work day. or maybe you should spend the time thinking about how to get off the email drug and move to a better way to work.