Showing posts with label certificates. Show all posts
Showing posts with label certificates. Show all posts

Tuesday, December 19, 2023

SSL Cert Renewed in Certmgr but no one could see it

Certmgr is the greatest thing in Domino these days if you are an admin.

Autorenewing SSL saves so many problems, delays, and potential loss of revenue for customers that it is, in my opinion, one of the best things HCL has added to Domino.

Much of the credit for it goes to HCL Lifetime Ambassador Daniel Nashed. 

When you see him at Engage or DNUG, buy him a beer.

Daniel was on hand to help me with my problem tonight, and he was correct with his original assessment, Certmgr should just work. 

I agreed, and it was working, or so it showed in the view when verifying it using "tell certmgr show certs" at the server console, but we could not see the validated certificates for 2 domains.

Since I had manually renewed them today, we should have seen a date of expiration for March 18. Instead, we saw December 17th for the one that expired yesterday, and the other showed January 21.

The TLS cache should be auto-refreshing when it gets the new certificates, but appeared to not be doing the task.

We reviewed the basic configuration and tried some test requests, which should have triggered a cache refresh and resolved the issue. But that didn't help us see the correct certificates in our browsers.

While Daniel asked me about different parameters, I learned something about the updated certmgr, we don't need to put the .kyr name in the Security tab, TLS options field. 

Instead, we should be using the DNS name. I totally missed this. The .kyr name in the field is there for the legacy people who have yet to move to V12 or V14. See page 36 of the slide deck mentioned below.

You can read Daniel's slides from his OpenNTF session, which is full of deep technical information. https://blog.nashcom.de/presentations/openntf2021_domino_certmgr.pdf

The other part, which I did know, but had yet to remove from the customer server is the Internet Sites Basics tab, DSAPI Filters field no longer requires ncertmgrdsapi.


After doing these bits of cleanup, and restarting HTTP a few times, we were still left with the issue of incorrectly reported SSL certificate dates.

We turned on debugging for the cache using set config CERTSTORE_CACHELOG=1. 
Page 47 in the above slide deck.

And we got nothing.

Which surprised both of us.

And then we went to look at the notes.ini to see if anything was pointing to the wrong place.

And this is where we found the problem.

Now, there is a parameter that should not have been there at all, and there was only one Google reference for it that we found. Evidently, that reference should not have been public, but it was, and someone at the customer site had added it sometime in the last 60 days or so because Certmgr had been running fine for over a year already.

For the sake of some poor admin out there troubleshooting this, I will say that if you experience the same problem as I did, look in your Domino notes.ini for a line that starts with "SSL_DISABLE_TLS".

I will not put the rest of the command here because, as Daniel said, no one should be using it.

If you find something like this, just remove the line outright from your notes.ini.
You can use "set config ssl_disable_tls(rest of the name)=" to remove it from your active server.
There is no 0 or 1 to put to remove it.

Then, at your server console, type "restart task HTTP," which is the better way to restart HTTP.

And poof, like magic, it all worked again.

That command blocks the newer TLS Cache refresh implementation from running. Thus even though Certmgr could get the updated certificates, it could not run the refresh because this line was telling it not to run.

Customers are so cute when they tell you they didn't change anything.







Tuesday, November 17, 2020

The Resiliency of HCL Domino and how to get Multiple SSL Certs to work with it

First, thanks go to Daniel Nashed who told me it could be done and that in R12 it will be even easier! I asked him because I thought the HCL documentation was a bit vague about if it would work as I needed.

Second thanks go to Detlev Poettgen and Ulrich Krause of the midpoints LE4D (Let's Encrypt 4 Domino) team for support while I set it up and providing the community an awesome SSL certificate tool that keeps you rolling SSL certs for free. The link includes the request form to get it.

Lots have been posted about the LE4D and for whatever reason, I had not gotten around to it.

Part of the reason is the resiliency of Domino, will get to that in a minute. But there was a technical limitation since removed in Domino 11 related to how Domino handled the SNI(Server Name Indication). Prior to R11 you can only use one SSL certificate per IP address in Domino. Since I run about a dozen domains on my server, this was not helpful, but now, it works quite well.

What I found along the way was, if HTTP is turned on, it is pretty hard to screw up a website. IP, name, old name, odd folder, missing file, Domino will still publish something. You start working backward to figure out what is going on and along the way end up with a stronger configuration. SSL troubleshooting is a little harder since we do not get specific error messages out of the server.

And this is what happened to me, but perseverance won out.

I requested the LE4D tool (link above), which is really a Domino application and I added it to my Domino 11.0FP1 Windows server.

Did I mention it is Free! As in beer, well when we are at evening events at conferences.

Before you get started, you need the ENABLE_SNI=1 added to your notes.ini on your server as explained in the HCL doc at the top of this post. The document explains that your configuration may need to be tweaked, mine did, and that you need a default web site configured or at least one web site with an IP address configured to use as a starting point. More on this later.

NOTE: On IBM i, SNI is supported natively on IBM i enabled for SSL Plus for HTTP and not for System SSL API.

You can follow directions in their PDF so I won't waste time on those, but the guide provided expects you to know a  few things which I will itemize below in case the LE4D teams want to update their help doc. 

NOTE: Domino 901FP8 or newer is required due to a reliance on JVM 1.8. Also, if not running on R10 or R11 you will need the KYRTOOL file and the pdf helps you to get it.

Almost everything is built into the application, even a run and sign button so you can do everything from within the LE4D application.

While it is possible to create one settings document for all your domains, I found it better to create one for each domain. It makes troubleshooting easier, but more importantly, because I leverage separate folders for each domain, it allows me to customize the HTML HOME DIRECTORY field which became a problem.

The Let's Encrypt hash codes and certificates need a place to go under the domain so it can be updated automatically, but also to preserve each domain's specific certificates. Otherwise, as I saw, all my domains ended up using the same cert which would not work in the real world, although fine in testing/staging servers.

Also, if you have multiple domains, you need to name the KEYFILE NAME field something different for each, or else all your certs will get written over and that helps no one.

Once you have filled in the setting document, they provide an example in the PDF, save the document. Then enable it so it can run.

I did not have to do the IKEYMAN part on page 7 of the PDF which may be for prior to R10 servers or Linux, not sure.

Set up the automated process to run the program document to keep your server automatically SSL up to date.

You can manually run the agent from their database from the button in the top right corner.

There is an agent log at the bottom of each setting document which helps to troubleshoot it as well.

I found that when I clicked on Run the client would hang for about 2 minutes while it ran and then would come back and either had worked or failed.

Now, this is where Domino was stubborn or resilient, depends on how you view it.

I could not get SSL to work at all. The log showed the cert was downloaded so what was wrong?

This is when I asked Daniel what I was missing and he pointed out that my DEFAULT SITE needed an IP Address. BUT when I set a default site, there was no way to add the IP Address. Seemed odd to me, so I worked backward. 

In my case, the default site is Traveler as it is not going away, even if some other domains get retired.

A second auto-generated internet site document is created and that one gets the IP address. You also need to add the correct SSL to this 2nd document so your Traveler devices can connect.

The next thing I had to do was manually add the correct Key File Name SSLFILE.KYR file in the security tab under each domain's Internet Sites document. And then run HTTP Refresh at the server console. 

And then it worked. Prior to fixing it all, HTTP worked fine and SSL thought it worked but really just errored out or said I was using a different domains certificate. Domino is pretty resilient to keep going even though parts were wrong.

2 domains I had to try a few times, maybe it was network issues, but eventually, all got done and updated. I had some typos and used the wrong http folder name in one case, but if you have patience you can find all your mistakes and fix them like I did.

Thursday, May 14, 2009

Tuesday Afternoon (Applets List of updates) You WILL need this

An amazing record still 40 years later

Something....calls to me....Those gentle voices I hear, explain it all with a sigh.

42 years ago or so The Moody Blues predicted this day would come.

So since we keep getting asked and the list grows by the day, here is what we know and where to find it. I have listed the versions for each as well.

Lotus Domino 6.5.x, 7.0.x, 8.0.x, 8.5 Technote Which explains it all in detail but some of the files, when I went to get them, were not on Fix Central so go to this Technote with the FTP for these files

Lotus Quickr for Domino 8.0.0.2 and 8.1 Technote
Lotus Quickr services for WebSphere Portal is not affected by this certificate expiration.

Lotus Quickplace 7.0, QuickPlace(Team Workplace)6.5.1 Technote

Lotus Sametime 7.0, 7.5.1 CF1 and CF2, 8.0, 8.0.1, 8.0.2 Technote
If you are on Sametime earlier than 7.0 you are out of luck, you must upgrade or keep getting pop up warnings forever.

So how do you fix it or what can you do to mitigate your call center and help desk people from staging a walk out on Thursday?

1. Instruct users to select "Always Trust" content from IBM, as the applets are still valid. The warning message is only to notify users that the certificate used to sign the applet has expired. The expiration does not affect an applet's security or functionality. In most cases, the user can click to "Always Trust" content from IBM to stop the message from appearing in the future.

2. Upgrade Domino 7.x servers to Domino 7.0.4. Domino 7.0.4 servers that are accessed via browsers that are using JRE 1.5.0 or later will not experience the issue.

3. IBM recommends correcting this issue through the replacement of the .jar files affected. You can currently download .zip files containing re-signed applets either from Fix Central or the IBM Support FTP site, as described in Technote #4022981 - Download re-signed Java applets for Lotus Domino (May 18, 2009).The re-signed .jar files are applicable to any supported release of Notes/Domino for any 6.5x, 7.0.x, 8.0.x, or 8.5 release.

4. IBM recommends option #3, which allows customers to replace files manually without needing a specific hotfix. Additionally, an interim fix (hotfix) will be provided via Fix Central starting May 11th for the latest MR/Fix Pack levels. This interim fix can be applied on Domino servers that do not currently have a hotfix applied.

Hotfix postings to Fix Central will start May 11, 2009. These releases include the following platforms and releases:
Platforms: W32, AIX, IBM i (iSeries), & Linux
Releases:
- Domino 6.5.6 FP3
- Domino 7.0.3 FP1
- Domino 7.0.4
- Domino 8.0.2 FP1
- Domino 8.5.0


That is all....for now.