The Resiliency of HCL Domino and how to get Multiple SSL Certs to work with it

First, thanks go to Daniel Nashed who told me it could be done and that in R12 it will be even easier! I asked him because I thought the HCL documentation was a bit vague about if it would work as I needed.

Second thanks go to Detlev Poettgen and Ulrich Krause of the midpoints LE4D (Let's Encrypt 4 Domino) team for support while I set it up and providing the community an awesome SSL certificate tool that keeps you rolling SSL certs for free. The link includes the request form to get it.

Lots have been posted about the LE4D and for whatever reason, I had not gotten around to it.

Part of the reason is the resiliency of Domino, will get to that in a minute. But there was a technical limitation since removed in Domino 11 related to how Domino handled the SNI(Server Name Indication). Prior to R11 you can only use one SSL certificate per IP address in Domino. Since I run about a dozen domains on my server, this was not helpful, but now, it works quite well.

What I found along the way was, if HTTP is turned on, it is pretty hard to screw up a website. IP, name, old name, odd folder, missing file, Domino will still publish something. You start working backward to figure out what is going on and along the way end up with a stronger configuration. SSL troubleshooting is a little harder since we do not get specific error messages out of the server.

And this is what happened to me, but perseverance won out.

I requested the LE4D tool (link above), which is really a Domino application and I added it to my Domino 11.0FP1 Windows server.

Did I mention it is Free! As in beer, well when we are at evening events at conferences.

Before you get started, you need the ENABLE_SNI=1 added to your notes.ini on your server as explained in the HCL doc at the top of this post. The document explains that your configuration may need to be tweaked, mine did, and that you need a default web site configured or at least one web site with an IP address configured to use as a starting point. More on this later.

NOTE: On IBM i, SNI is supported natively on IBM i enabled for SSL Plus for HTTP and not for System SSL API.

You can follow directions in their PDF so I won't waste time on those, but the guide provided expects you to know a  few things which I will itemize below in case the LE4D teams want to update their help doc. 

NOTE: Domino 901FP8 or newer is required due to a reliance on JVM 1.8. Also, if not running on R10 or R11 you will need the KYRTOOL file and the pdf helps you to get it.

Almost everything is built into the application, even a run and sign button so you can do everything from within the LE4D application.

While it is possible to create one settings document for all your domains, I found it better to create one for each domain. It makes troubleshooting easier, but more importantly, because I leverage separate folders for each domain, it allows me to customize the HTML HOME DIRECTORY field which became a problem.

The Let's Encrypt hash codes and certificates need a place to go under the domain so it can be updated automatically, but also to preserve each domain's specific certificates. Otherwise, as I saw, all my domains ended up using the same cert which would not work in the real world, although fine in testing/staging servers.

Also, if you have multiple domains, you need to name the KEYFILE NAME field something different for each, or else all your certs will get written over and that helps no one.

Once you have filled in the setting document, they provide an example in the PDF, save the document. Then enable it so it can run.

I did not have to do the IKEYMAN part on page 7 of the PDF which may be for prior to R10 servers or Linux, not sure.

Set up the automated process to run the program document to keep your server automatically SSL up to date.

You can manually run the agent from their database from the button in the top right corner.

There is an agent log at the bottom of each setting document which helps to troubleshoot it as well.

I found that when I clicked on Run the client would hang for about 2 minutes while it ran and then would come back and either had worked or failed.

Now, this is where Domino was stubborn or resilient, depends on how you view it.

I could not get SSL to work at all. The log showed the cert was downloaded so what was wrong?

This is when I asked Daniel what I was missing and he pointed out that my DEFAULT SITE needed an IP Address. BUT when I set a default site, there was no way to add the IP Address. Seemed odd to me, so I worked backward. 

In my case, the default site is Traveler as it is not going away, even if some other domains get retired.

A second auto-generated internet site document is created and that one gets the IP address. You also need to add the correct SSL to this 2nd document so your Traveler devices can connect.

The next thing I had to do was manually add the correct Key File Name SSLFILE.KYR file in the security tab under each domain's Internet Sites document. And then run HTTP Refresh at the server console. 

And then it worked. Prior to fixing it all, HTTP worked fine and SSL thought it worked but really just errored out or said I was using a different domains certificate. Domino is pretty resilient to keep going even though parts were wrong.

2 domains I had to try a few times, maybe it was network issues, but eventually, all got done and updated. I had some typos and used the wrong http folder name in one case, but if you have patience you can find all your mistakes and fix them like I did.