ClamAV and Domino

 I am not an expert on Anti-virus things and Domino, that would be Daniel Nashed.

But if HCL Domino has something new, I test it out and set it up on my server.

In this case, I had heard that ClamAV would now work with Domino as of 14.5.1.

Having used Clam for many years at the OS level, I wanted to play with this in Domino.

My server is Windows, not Linux, ClamAV Linux instructions are pretty common, Windows, not as much.

Boy was this the hardest "easy' thing to do ever.

To start, HCL provides no documentation.

There are only 3 references to ClamAV in the official docs:

https://help.hcl-software.com/domino/14.5.1/admin/wn_config_features_1451.html#wn_config_features_1451__section_fq1_wgc_23c

https://help.hcl-software.com/domino/14.5.1/admin/conf_configuringscanninginscscancfg.html?hl=clamav

https://help.hcl-software.com/domino/14.5.1/admin/conf_scanningattachmentsforviruses.html?hl=clamav

None of which tell you much about how to set up ClamAV or how to configure it for Domino integration.

I opened a ticket and eventually was directed to this post:

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0129703

Which is in Japanese. Use Google Translate or Chrome built in translstion.

I see now, at my urging, HCL has released the same document in English.

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0131220

Their perspective is that we should go to ClamAv to set it up and then magically know how to integrate it.

I asked why it wasn't in the actual documentation and got a runaround answer, given HCL announced the availability, not ClamAV.

The new technical document covers some of the set up required.

I am going to add what is missing, more for my benefit, and anyone else who will try this on their servers.

Step 1 is Download the ClamAV from the link in the document, https://www.clamav.net/downloads and install it. BUT this does not tell you how to set it up in Services.

For that, you perform this:

Open the Windows Command Prompt as an Administrator and navigate to your ClamAV folder.

Run the following command to install the ClamAV daemon service:
clamd.exe --install

Remember to set it to automatic.

Step 2 is to copy the config file as described and perform the edits.

Step 3, however, did not work for me no matter how I tried to get the certificate. Even with HCL Support help, they eventually had to send me the certificate required.

Step 4: You have to get the updated virus definitions. NOTE: This is a once off effort. And Manual.

If you want it to be automatic, and you do, you need to do these steps.

Open the Windows Command Prompt as an Administrator and navigate to your ClamAV folder.

Run the following command to install the ClamAV daemon service:
freshclam.exe --install

Once you Load mailscan at the Domino server console you will get the cscancfg.nsf and need to follow the document entries as provided.

NOTE: The ClamAV server name(DNS) must be 127.0.0.1. Do not put your server IP there, it will not work at all.

Make sure you open port 3310 with your local server firewall, and in my case, the outside ISP.

I changed the Subject Prefix Scanned field default text to something I would know came from my server, not a spammer.

Lastly, make sure you add Mailscan to your tasks line in your notes.ini also something not mentioned in the document that is kind of important, unless you like having your mail stuck in mail.box and spend 2 hours troubleshooting it one morning, like I just did.

I eventually figured out it was an AV issue when running SMTPDebugClient=1 and

Tell Router list run from the server console.

You see this:

Mbx NoteID          ID          State          Size Pri Count ScheduledDate           From

  1 000008FE 0028261D Wait AV        8443         1                         K Brooks/org

  2 00000902 00283C39 Wait AV       13306         1                         kbrooks@test.com

  1 00000902 002864E3 Wait AV        8749         1                         keith@gmail.com

  2 00000906 00287C9A Wait AV        8640         1                         keith@gmail.com

Oh, that makes sense after the server restarted late yesterday. Mailscan was not in tasks.

So, my time spent, is your time gained.

12.0.2 FP8 didn't update version numbers

This is probably not a problem with FP8, to be upfront, but just in case I am using it for the title.

4 out of 8 servers at a customer did not chaaange tehir version number when I updated them to FP8.

They stayed FP7.

In the past, I have had this issue where updates don't happen, due to various reasons like a corrupted NAB, failed hot fix removals, etc..

But this was different because 4 servers, DEV domain, were fine.

QUAL domain is not updated.

How Did I Get Here

Servers were 12.0.2 FP7 HF1 Windoews 2019

I shut the Domino server down, including the Domino service.

Paused the Windows Management service(ONLY affects HF's for some reason), then ran the hot fix as administrator to remove it.

Started the server from the icon asd a regular application, so I can see it running, and a "sh ser" showed the correct FP7 version. Let it run for a few minutes till I saw HTTP start and some other tasks. Then shut it down.

Run FP8 as an administrator, runs fine, no errors.

Restart the box, actually, 2 servers had issues with the Domino service login, and I restarted the whole VM, the other 2 the service started fine.

Waited to see the server docs change, and nothing.

The upgrade.log showed everything looked ok.

I opened a ticket with HCL after the 1st 4 failed to update, then I checked the next 4 and those worked.

The Resolution and Reasoning

Sooraj from HCL Support provided me with the following resolution and an explanation after I provided some files and screenshots.

Take a backup of the files below and replace them from a "working" server [12.0.2FP8]

Domino\nstrings.dll
Domino\res\C\nstrings.res

Shutdown the incorrect Domino server
Take a backup of the two files mentioned above.
Copy similar files from a working 1202FP8 server and replace them.

(I left the existing ones, and added a _BAD at the end of the name)

Start the Domino server.

While I asked to close the ticket, I also wanted an explanation for future thoughts. 

Sooraj provided the following:

This problem was reported for the first time, and based on my analysis, it appears that those two files were not updated by the FP8 installer. It was still using the FP7 code. This ideally occurs due to an access issue with the account that started the installation, but unfortunately, I cannot confirm this because the upgrade.log doesn't show any such errors.

Possibly a server processing issue, a delay, or perhaps not fully completing a virus scan, or some other local issue was to blame.

But on 4 servers, I don't believe so. 

One, sure, things happen. 

In this situation, all 4, well, 8, are nearly identical setups.

Hope this helps others with similar issues at some time in their admin journeys.

 

SSL Cert Renewed in Certmgr but no one could see it

Certmgr is the greatest thing in Domino these days if you are an admin.

Autorenewing SSL saves so many problems, delays, and potential loss of revenue for customers that it is, in my opinion, one of the best things HCL has added to Domino.

Much of the credit for it goes to HCL Lifetime Ambassador Daniel Nashed. 

When you see him at Engage or DNUG, buy him a beer.

Daniel was on hand to help me with my problem tonight, and he was correct with his original assessment, Certmgr should just work. 

I agreed, and it was working, or so it showed in the view when verifying it using "tell certmgr show certs" at the server console, but we could not see the validated certificates for 2 domains.

Since I had manually renewed them today, we should have seen a date of expiration for March 18. Instead, we saw December 17th for the one that expired yesterday, and the other showed January 21.

The TLS cache should be auto-refreshing when it gets the new certificates, but appeared to not be doing the task.

We reviewed the basic configuration and tried some test requests, which should have triggered a cache refresh and resolved the issue. But that didn't help us see the correct certificates in our browsers.

While Daniel asked me about different parameters, I learned something about the updated certmgr, we don't need to put the .kyr name in the Security tab, TLS options field. 

Instead, we should be using the DNS name. I totally missed this. The .kyr name in the field is there for the legacy people who have yet to move to V12 or V14. See page 36 of the slide deck mentioned below.

You can read Daniel's slides from his OpenNTF session, which is full of deep technical information. https://blog.nashcom.de/presentations/openntf2021_domino_certmgr.pdf

The other part, which I did know, but had yet to remove from the customer server is the Internet Sites Basics tab, DSAPI Filters field no longer requires ncertmgrdsapi.

After doing these bits of cleanup, and restarting HTTP a few times, we were still left with the issue of incorrectly reported SSL certificate dates.

We turned on debugging for the cache using set config CERTSTORE_CACHELOG=1. 

Page 47 in the above slide deck.

And we got nothing.

Which surprised both of us.

And then we went to look at the notes.ini to see if anything was pointing to the wrong place.

And this is where we found the problem.

Now, there is a parameter that should not have been there at all, and there was only one Google reference for it that we found. Evidently, that reference should not have been public, but it was, and someone at the customer site had added it sometime in the last 60 days or so because Certmgr had been running fine for over a year already.

For the sake of some poor admin out there troubleshooting this, I will say that if you experience the same problem as I did, look in your Domino notes.ini for a line that starts with "SSL_DISABLE_TLS".

I will not put the rest of the command here because, as Daniel said, no one should be using it.

If you find something like this, just remove the line outright from your notes.ini.

You can use "set config ssl_disable_tls(rest of the name)=" to remove it from your active server.

There is no 0 or 1 to put to remove it.

Then, at your server console, type "restart task HTTP," which is the better way to restart HTTP.

And poof, like magic, it all worked again.

That command blocks the newer TLS Cache refresh implementation from running. Thus even though Certmgr could get the updated certificates, it could not run the refresh because this line was telling it not to run.

Customers are so cute when they tell you they didn't change anything.

SnTT - When you need OS Acces, but DON'T Have it, OPENNTF to the Rescue

 

I should tag this as #HCLAmbassadorTips, but that is so #2020.

Picked up a new client from my home state with a small but necessary business, which they run on their HCL Domino application.

The problem started when I realized I needed to scrape their old website from a hosting provider and then put it under HCL Domino.

Simple enough to get the site files and fodlers, but when I went to add the site to Domino realized I did not have RDP/VPN/FTP access to their server at the OS level.

What do I do?

As it turns out, there have been a few tools that existed that do this, but being an Admin and not a Developer, I never needed them and never looked into it.

I ended up finding a solution at Openntf.org with a project called Remote File Browser.

It was posted in 2010, still works, and I am using an R12 beta client to work with it on my end and the R11 server on the customer side.

It does just what I needed:

• Create a folder
• Delete files
• Move Files
• Upload file
• Download files

Did you note an extra letter between the last 2 items? It seems you can download multiple files, but you can only upload 1 file at a time.

Okay, a bit tedious. I had about 40 files, but if I needed to edit the notes.ini or to change the SSL keys(if you are not using LE4D) and did not have access at the time, this does the trick.

Follow the instructions for it, copy it to the server, sign it, assign an ACL and then start browsing the server files.

Created a folder under domino\data\domino\html and then moved all the files.

Edit the Internet Site Document, so it knows which directory to read from and which HTML file to use as the primary page and restart HTTP, and it works.

If you know of anything similar and is available, free or for a fee, include it in the comments or message me to include the other options for everyone.

The best part is I get to cancel a meeting for later today that was about access.

SnTT - Adding a 3rd party Task to the Domino Admin Client Monitoring Dashboard

 

This has bugged me for a while. I may have just been a bit lazy as well.

But that's what friends are for, and Ted Hardenburgh came to my rescue in our Openntf slack channel.

You can join the slack channel here: https://openntf.slack.com if you have FOMO or just want to ask us your crazy questions.

Anyway, back to Ted's help with my situation.

We have a client running a task that is not a native Domino task, and I wanted to monitor it from the Admin client's Server-Monitoring tab that acts as a Dashboard.

You did know you have a dashboard built into the Admin client, right?

I will presume you know how to add your servers to the dashboard.

But when you want to add a task, you select a server and right-click and select Monitor New Task from the list shown. The default only includes the native Domino tasks.

Naturally, I figured there must be a way to add the 3rd party task, I was hoping without too much under the cover messing around, and that is when Ted helped me out.

My thought was there is a subform someplace to edit, and Ted pointed me to it.

Open the domadmin.nsf file in the Designer client.

Look at the Forms, and you will see a long list of Task\names.

The easiest way to start on this journey is to copy one of the tasks and paste it back in and begin editing it.

In this case, I am creating a task for bccdbt, I already renamed the task.

The name has to match the task name you would run when at a server console.

The fields you will need to edit from the above screen are the following:

Task name, Task filename, and Task Monitor Name and Description.

Task name field = Name as you want it to be seen in the Add Monitor Task pop-up window.

Task filename = the task name as you would enter it in the server console to run it.

Task Monitor Name = Should be how the task is described when you run a sh ta command. Like Router, Schedule Manager, etc...

Description = Edit according to what you need to tell your fellow admins.

Save it and then test it.

The task should show up within a few minutes, the dashboard updates automatically.

That's it. Many thanks again to Ted.

The Resiliency of HCL Domino and how to get Multiple SSL Certs to work with it

First, thanks go to Daniel Nashed who told me it could be done and that in R12 it will be even easier! I asked him because I thought the HCL documentation was a bit vague about if it would work as I needed.

Second thanks go to Detlev Poettgen and Ulrich Krause of the midpoints LE4D (Let's Encrypt 4 Domino) team for support while I set it up and providing the community an awesome SSL certificate tool that keeps you rolling SSL certs for free. The link includes the request form to get it.

Lots have been posted about the LE4D and for whatever reason, I had not gotten around to it.

Part of the reason is the resiliency of Domino, will get to that in a minute. But there was a technical limitation since removed in Domino 11 related to how Domino handled the SNI(Server Name Indication). Prior to R11 you can only use one SSL certificate per IP address in Domino. Since I run about a dozen domains on my server, this was not helpful, but now, it works quite well.

What I found along the way was, if HTTP is turned on, it is pretty hard to screw up a website. IP, name, old name, odd folder, missing file, Domino will still publish something. You start working backward to figure out what is going on and along the way end up with a stronger configuration. SSL troubleshooting is a little harder since we do not get specific error messages out of the server.

And this is what happened to me, but perseverance won out.

I requested the LE4D tool (link above), which is really a Domino application and I added it to my Domino 11.0FP1 Windows server.

Did I mention it is Free! As in beer, well when we are at evening events at conferences.

Before you get started, you need the ENABLE_SNI=1 added to your notes.ini on your server as explained in the HCL doc at the top of this post. The document explains that your configuration may need to be tweaked, mine did, and that you need a default web site configured or at least one web site with an IP address configured to use as a starting point. More on this later.

NOTE: On IBM i, SNI is supported natively on IBM i enabled for SSL Plus for HTTP and not for System SSL API.

You can follow directions in their PDF so I won't waste time on those, but the guide provided expects you to know a  few things which I will itemize below in case the LE4D teams want to update their help doc. 

NOTE: Domino 901FP8 or newer is required due to a reliance on JVM 1.8. Also, if not running on R10 or R11 you will need the KYRTOOL file and the pdf helps you to get it.

Almost everything is built into the application, even a run and sign button so you can do everything from within the LE4D application.

While it is possible to create one settings document for all your domains, I found it better to create one for each domain. It makes troubleshooting easier, but more importantly, because I leverage separate folders for each domain, it allows me to customize the HTML HOME DIRECTORY field which became a problem.

The Let's Encrypt hash codes and certificates need a place to go under the domain so it can be updated automatically, but also to preserve each domain's specific certificates. Otherwise, as I saw, all my domains ended up using the same cert which would not work in the real world, although fine in testing/staging servers.

Also, if you have multiple domains, you need to name the KEYFILE NAME field something different for each, or else all your certs will get written over and that helps no one.

Once you have filled in the setting document, they provide an example in the PDF, save the document. Then enable it so it can run.

I did not have to do the IKEYMAN part on page 7 of the PDF which may be for prior to R10 servers or Linux, not sure.

Set up the automated process to run the program document to keep your server automatically SSL up to date.

You can manually run the agent from their database from the button in the top right corner.

There is an agent log at the bottom of each setting document which helps to troubleshoot it as well.

I found that when I clicked on Run the client would hang for about 2 minutes while it ran and then would come back and either had worked or failed.

Now, this is where Domino was stubborn or resilient, depends on how you view it.

I could not get SSL to work at all. The log showed the cert was downloaded so what was wrong?

This is when I asked Daniel what I was missing and he pointed out that my DEFAULT SITE needed an IP Address. BUT when I set a default site, there was no way to add the IP Address. Seemed odd to me, so I worked backward. 

In my case, the default site is Traveler as it is not going away, even if some other domains get retired.

A second auto-generated internet site document is created and that one gets the IP address. You also need to add the correct SSL to this 2nd document so your Traveler devices can connect.

The next thing I had to do was manually add the correct Key File Name SSLFILE.KYR file in the security tab under each domain's Internet Sites document. And then run HTTP Refresh at the server console. 

And then it worked. Prior to fixing it all, HTTP worked fine and SSL thought it worked but really just errored out or said I was using a different domains certificate. Domino is pretty resilient to keep going even though parts were wrong.

2 domains I had to try a few times, maybe it was network issues, but eventually, all got done and updated. I had some typos and used the wrong http folder name in one case, but if you have patience you can find all your mistakes and fix them like I did.