SSL Cert Renewed in Certmgr but no one could see it

Certmgr is the greatest thing in Domino these days if you are an admin.

Autorenewing SSL saves so many problems, delays, and potential loss of revenue for customers that it is, in my opinion, one of the best things HCL has added to Domino.

Much of the credit for it goes to HCL Lifetime Ambassador Daniel Nashed. 

When you see him at Engage or DNUG, buy him a beer.

Daniel was on hand to help me with my problem tonight, and he was correct with his original assessment, Certmgr should just work. 

I agreed, and it was working, or so it showed in the view when verifying it using "tell certmgr show certs" at the server console, but we could not see the validated certificates for 2 domains.

Since I had manually renewed them today, we should have seen a date of expiration for March 18. Instead, we saw December 17th for the one that expired yesterday, and the other showed January 21.

The TLS cache should be auto-refreshing when it gets the new certificates, but appeared to not be doing the task.

We reviewed the basic configuration and tried some test requests, which should have triggered a cache refresh and resolved the issue. But that didn't help us see the correct certificates in our browsers.

While Daniel asked me about different parameters, I learned something about the updated certmgr, we don't need to put the .kyr name in the Security tab, TLS options field. 

Instead, we should be using the DNS name. I totally missed this. The .kyr name in the field is there for the legacy people who have yet to move to V12 or V14. See page 36 of the slide deck mentioned below.

You can read Daniel's slides from his OpenNTF session, which is full of deep technical information. https://blog.nashcom.de/presentations/openntf2021_domino_certmgr.pdf

The other part, which I did know, but had yet to remove from the customer server is the Internet Sites Basics tab, DSAPI Filters field no longer requires ncertmgrdsapi.

After doing these bits of cleanup, and restarting HTTP a few times, we were still left with the issue of incorrectly reported SSL certificate dates.

We turned on debugging for the cache using set config CERTSTORE_CACHELOG=1. 

Page 47 in the above slide deck.

And we got nothing.

Which surprised both of us.

And then we went to look at the notes.ini to see if anything was pointing to the wrong place.

And this is where we found the problem.

Now, there is a parameter that should not have been there at all, and there was only one Google reference for it that we found. Evidently, that reference should not have been public, but it was, and someone at the customer site had added it sometime in the last 60 days or so because Certmgr had been running fine for over a year already.

For the sake of some poor admin out there troubleshooting this, I will say that if you experience the same problem as I did, look in your Domino notes.ini for a line that starts with "SSL_DISABLE_TLS".

I will not put the rest of the command here because, as Daniel said, no one should be using it.

If you find something like this, just remove the line outright from your notes.ini.

You can use "set config ssl_disable_tls(rest of the name)=" to remove it from your active server.

There is no 0 or 1 to put to remove it.

Then, at your server console, type "restart task HTTP," which is the better way to restart HTTP.

And poof, like magic, it all worked again.

That command blocks the newer TLS Cache refresh implementation from running. Thus even though Certmgr could get the updated certificates, it could not run the refresh because this line was telling it not to run.

Customers are so cute when they tell you they didn't change anything.

The Resiliency of HCL Domino and how to get Multiple SSL Certs to work with it

First, thanks go to Daniel Nashed who told me it could be done and that in R12 it will be even easier! I asked him because I thought the HCL documentation was a bit vague about if it would work as I needed.

Second thanks go to Detlev Poettgen and Ulrich Krause of the midpoints LE4D (Let's Encrypt 4 Domino) team for support while I set it up and providing the community an awesome SSL certificate tool that keeps you rolling SSL certs for free. The link includes the request form to get it.

Lots have been posted about the LE4D and for whatever reason, I had not gotten around to it.

Part of the reason is the resiliency of Domino, will get to that in a minute. But there was a technical limitation since removed in Domino 11 related to how Domino handled the SNI(Server Name Indication). Prior to R11 you can only use one SSL certificate per IP address in Domino. Since I run about a dozen domains on my server, this was not helpful, but now, it works quite well.

What I found along the way was, if HTTP is turned on, it is pretty hard to screw up a website. IP, name, old name, odd folder, missing file, Domino will still publish something. You start working backward to figure out what is going on and along the way end up with a stronger configuration. SSL troubleshooting is a little harder since we do not get specific error messages out of the server.

And this is what happened to me, but perseverance won out.

I requested the LE4D tool (link above), which is really a Domino application and I added it to my Domino 11.0FP1 Windows server.

Did I mention it is Free! As in beer, well when we are at evening events at conferences.

Before you get started, you need the ENABLE_SNI=1 added to your notes.ini on your server as explained in the HCL doc at the top of this post. The document explains that your configuration may need to be tweaked, mine did, and that you need a default web site configured or at least one web site with an IP address configured to use as a starting point. More on this later.

NOTE: On IBM i, SNI is supported natively on IBM i enabled for SSL Plus for HTTP and not for System SSL API.

You can follow directions in their PDF so I won't waste time on those, but the guide provided expects you to know a  few things which I will itemize below in case the LE4D teams want to update their help doc. 

NOTE: Domino 901FP8 or newer is required due to a reliance on JVM 1.8. Also, if not running on R10 or R11 you will need the KYRTOOL file and the pdf helps you to get it.

Almost everything is built into the application, even a run and sign button so you can do everything from within the LE4D application.

While it is possible to create one settings document for all your domains, I found it better to create one for each domain. It makes troubleshooting easier, but more importantly, because I leverage separate folders for each domain, it allows me to customize the HTML HOME DIRECTORY field which became a problem.

The Let's Encrypt hash codes and certificates need a place to go under the domain so it can be updated automatically, but also to preserve each domain's specific certificates. Otherwise, as I saw, all my domains ended up using the same cert which would not work in the real world, although fine in testing/staging servers.

Also, if you have multiple domains, you need to name the KEYFILE NAME field something different for each, or else all your certs will get written over and that helps no one.

Once you have filled in the setting document, they provide an example in the PDF, save the document. Then enable it so it can run.

I did not have to do the IKEYMAN part on page 7 of the PDF which may be for prior to R10 servers or Linux, not sure.

Set up the automated process to run the program document to keep your server automatically SSL up to date.

You can manually run the agent from their database from the button in the top right corner.

There is an agent log at the bottom of each setting document which helps to troubleshoot it as well.

I found that when I clicked on Run the client would hang for about 2 minutes while it ran and then would come back and either had worked or failed.

Now, this is where Domino was stubborn or resilient, depends on how you view it.

I could not get SSL to work at all. The log showed the cert was downloaded so what was wrong?

This is when I asked Daniel what I was missing and he pointed out that my DEFAULT SITE needed an IP Address. BUT when I set a default site, there was no way to add the IP Address. Seemed odd to me, so I worked backward. 

In my case, the default site is Traveler as it is not going away, even if some other domains get retired.

A second auto-generated internet site document is created and that one gets the IP address. You also need to add the correct SSL to this 2nd document so your Traveler devices can connect.

The next thing I had to do was manually add the correct Key File Name SSLFILE.KYR file in the security tab under each domain's Internet Sites document. And then run HTTP Refresh at the server console. 

And then it worked. Prior to fixing it all, HTTP worked fine and SSL thought it worked but really just errored out or said I was using a different domains certificate. Domino is pretty resilient to keep going even though parts were wrong.

2 domains I had to try a few times, maybe it was network issues, but eventually, all got done and updated. I had some typos and used the wrong http folder name in one case, but if you have patience you can find all your mistakes and fix them like I did.

SnTT - WildCard SSL Certs and Domino....needs 32bit Windows?

This post is for me, and you, to make life easy. 

Some of you out there are only now adding SSL certificates to your servers, I know because you are asking me to help you, and so I present today's Show n Tell Tuesday post.

There are some excellent SSL and Domino posts on this topic already from Gab Davis, Mats Ekman, and Sean Cull, IBM documentation is a bit lacking in this use case so I will not point you to it.
EDIT : Jared Roberts, also a fellow IBM Champion, wrote an excellent post on the wildcard and you should go use his, it is found here. (My blog post was written over a few months so I had missed Jared's post)

I used Gab's post a few times over the last few months and always in a situation where the customer had a stand-alone (individual server) SSL certificate and you can't mess it up, well you could typo the syntax, but her post is the easiest way to get this accomplished. Thank you, Gab!

However, a Wildcard SSL certificate is a little different and this is when I found Mat and Sean's post very helpful and I am taking from them to add my 2 cents.

Along the way of following their posts, I found that the IBM required tool only runs on a 32 bit Windows environment. Let's just say were it not for my TV PC, I would have to create a VM just to run this tool. 

You have been warned.

Everything else runs on 64 bit and you will need your Domino server and your Admin Client accessible. 
EDIT from Chuck's Comment below: You should be using at least a 9.0.1FP6 Notes client, not necessarily an Admin client but you may find it easier to do so.

This is how we include a wildcard 4096 bit cert issued from a registrar with a .PFX and .CSR file into Domino. (Always make sure to get the password used for the key, you WILL need it.)

What do you do with a .PFX file? You convert it into a .PEM file using OpenSSL.
EDITED FEB 9, 2022 after prior update of Nov 30, 2017 NOTE: If you have a .pfx file, an IBM HCL technote makes this much easier than the steps below. https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0029841 but I will leave it all for those who want to know more. Thanks Ted H. for the link.

What? Why? Never mind that now, but the steps to do the magic are below:

Steps:
1) Download the OpenSSL software (taken from Mat's post)
Homepage: https://www.openssl.org/source/
Easy precompiled: https://slproweb.com/products/Win32OpenSSL.html
The one Mat used: https://slproweb.com/download/Win64OpenSSL_Light-1_1_0g.exe
NOTE: These are direct links, so if it doesn't work, go to https://slproweb.com/products/Win32OpenSSL.html for 32 and 64 bit.

2) Download the KYRTool (again taken from Mat's Post)
Fixcentral short: http://ibm.co/1SAYX5E
Fixcentral long: http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=ibm%7ELotus&product=ibm/Lotus/Lotus+Domino&release=9.0.1.2&platform=All&function=fixId&fixids=KYRTool_9

3) Install the OpenSSL into its own directory (include the binaries)

4) Go to C:\OpenSSL-Win64\bin\openssl then run the openssl to get the command window you need so you can type in. (This took me a while to figure out, you are welcome):

5) this is all one line pkcs12 -in C:\location of mypfxfiles\wildcard_company_com.pfx -out  c:\somelocation\wildcard_cmpany_com.pem -nodes -chain

Remember, Domino requires both the .KYR file and the .STH files to be placed inside the \Domino\data directory.

As always have fun, ask for help when you need it and don't give up, as you can see, many of us have been in your shoes before.

HUGE Change Coming for non SSL Traveler Customers

This was to go into effect on January 1, 2017 but IBM has relented, edit is from yesterday, so you have 3 months, and not a nightmare on January 2nd to return to from vacation.

IBM published this Technote which lays it all out in black and white: 

IMPORTANT: You must ensure that your IBM Verse Mobile and Traveler connections are secure and compliant with these requirements by March 15, 2017. 

This is not about IBM Connections Cloud users but is about on-premises installations of IBM Notes Traveler.

As the Technote states:

In the coming months, IBM will be enhancing the IBM Verse for iOS, IBM Verse for Android, IBM Notes Traveler Companion and IBM Notes Traveler To Do mobile apps to require that a secure connection be used between the mobile app and the endpoint used for connecting to the IBM Traveler server. 

The gist of it is, without HTTPS and valid SSL certificates you will lose your connectivity.

Please read the Technote for all the details and plan accordingly if you have yet to setup your SSL certificates.

If you need help, just ask. 

Remember Young Admins...there are 2 files

Us old timers don't remember everything after all.

I set up a TLS environment using the post from Gregg Eldred.

Additionally we self certified it from Domino (checked the vendor and the spam system accepts it) and all seemed well and good.

Except I received numerous errors (and since we use DDM, um, 1,000's of emails about it)which said the same thing:

SSL error: Keyring File access error

So I did what anyone else would do, I phoned a friend. Called Gregg, left him a voice mail and then pondered what else I might have done wrong.

Within a few minutes, and a quick search of my own internal documentation, I realized the problem was I forgot to copy BOTH files from my client when I set up the keyring file. I only copied the .KYR but not the .STH.

DOH!

And thus ended my errors and all was good again.

If you have a similar error message and are 100% sure you copied both files, then I suggest you check out IBM technote #1218820, titled "
Frequently Asked Questions: Using Secure Socket Layer (SSL) with Notes/Domino".

That will probably lead you to this IBM technote #1109822 titled "Web server error: Keyring File access error or Page cannot be displayed".

So next time you have a problem leave Gregg a message describing the problem and you may have it solved within minutes. If you do, please do a session at MWLUG 2010 on this phenomena.

Lotus Domino SSL, Don't Force It (updated Post)

Can't save this for SNTT, that's on Yom Kippur.

Evidently one's intepretation of a setting can lead to double trouble.

A URL which should look like this(found on a Lotus Quickr server):
https://www.youarethe1.com/lotusquickr/...

Turned into this:
https://www.youarethe1.com/https//www.youarethe1.com/lotusquickr...

And of course prevents logging in.

The resolution for this odd appearance is the following:
In the Server document of the offending server go to:
Internet Protocols Section
Domino Web Engine Tab
HTTP Sessions Part
Force Login on SSL set it to NO

Setting it to YES caused this dilemna.
I am not going to delve into what this field is for exactly, there is no hover over or click on it help to tell me what it is for....so someone want to enlighten me I will be happy to add it on to this post.

COntinuing now with User Acceptance Testing.

Updated 4/23/2009 Per Patrick's post
Technote:
http://www-01.ibm.com/support/docview.wss?rs=899&uid=swg21283220

Q-SnTT - DA Configuration Reminder

When involved in the Quickr world, at times(ok, almost all the time) you may want to connect external people to your server.
How do you do it?
Create a Directory Assistance database (Ctrl+N, select the server and advanced templates to find it). Name it da.nsf.

In the DA document select LDAP if you need to include an external LDAP server or Notes for internal Directory Catalogs (DirCat) or Extended Directories.
Fill in the blanks. It is fairly obvious what to fill in where....EXCEPT for
the field called Domain, which is NOT asking for your Lotus Domain.
Name it anything else and then save it and you are good to go.

Why is this? I don't know, but it is.

On the tab marked rules(the middle tab usually) select trusted for credentials and say YES.
On the LDAP tab, if you selected it, if you are not using SSL make sure to select the proper options, otherwise you will not find your LDAP server.
Caveat, AD (Active Directory) requires a Base DN to be used, o=companyname most likely)
If you selected Notes instead of LDAP on this tab you would put any of the other Domino directories you will be using, no need to include the names.nsf itself, it searches that by default.
And don't forget to mark it Enabled.

Next save it all.
Then add the da.nsf to your server document on the basics tab.

Now go to a server console and type "Show X R" without the quotes.
You should see names.nsf first then your LDAP server.

Any problems, you know how to find me.

SSL and Quickr Using Network Solutions

Edit Update 7:30PM 9/15/2008:
IBM Support replied to my questions and I wanted to share it with you.

"You're correct in that the order of installation of the certificates is important and can be confusing if (a) the CA uses multiple trusted root certificates and (b) the CA does not document well which order the certificates need to be installed.

Regarding your blog entry on Domino's selection of which CA's trusted root certificates are included, I spoke with our L3 team about this issue. They stated that they regularly check which CA's are the most popular ones among both our customers and the public in general and use this information when deciding upon which CA's trusted root certificates will be automatically included with Domino. As you know, Network Solutions' trusted root certificates are not currently included in the creation of new SSL keyring files in Domino. Because of this, I have created SPR TJOR7JHSQL to request that its trusted root certificates be included in future releases of Domino as an enhancement."

Thanks team IBM and Lotus Support for helping on this and saving me one less headache, hopefully in R8.5 and beyond.
==========

What would normally be a simple SSL set up turned into a couple of hours of pain, for me, the client is happy of course.

I will not rewrite the help docs on how to setup SSL, but the problem stems from using a vendor not listed in the certsrv.nsf file.

Why, after all these years, hasn't Lotus included some more main stream vendors I can't tell you, but they have quite a few listed, evidently Network Solutions (NS) just doesn't come up much.

Requested the certs from NS and then the problem started. Never seen 4 certificates returned. 1 is the domain, easy enough. The other 3? Who knows what order or if they are all needed.

Bottom line is phone support from NS resolved it, online documentation was not very good which is why this post is out there. I could save it for SnTT but no, I want to help people sooner, rather than later with this one.

The proper order to install the certs is:
AddTrustExternalCARoot.crt
UTNAddTrustServer_CA.crt
NetworkSolutions_CA.crt
yourdomain.com.crt

Name the first 3 as you add them, individually, Network Solutions #1,#2 and #3.
The Domain gets added as the last one in Step 4 of the certsrv file.

Reload HTTP and it should be fine. If you have already set everything up according to the documentation.

And lest I forget, contrary to what you think, you must select anonymous under the SSL to enable the ability to see the Quickr logo files found in the /qphtml/html/login directory on the Login page.