SnTT - Does TOTP Work for users in a Secondary Directory via DA

TOTP, DA, and Domino

For the last 3 years, I have worked with TOTP inside HCL Domino and customers with unique requirements.

This has provided fodder for my blog, and today, we have a new entry into the TOTP Mystical Ways of the World.

Let me state my usual caveat upfront: TOTP is about the URL, not the server, the database, or the user.

You enable TOTP for each URL you want on your server.

PSA is completed. Let's discuss the circumstances that brought me here.

Like many of our customers, a customer has a large external user community relying on their applications.

The customer has licensed this with HCL, so I am not going to get involved in that discussion. However, be warned: It is not a comfortable one if you have been relying on some old licensing options and now fall under the new ones.

We have about 7,000 external customers. Some are undoubtedly old customers, but 7,000 is a lot of people.

Previously, I wrote about how to bulk add these people into your ID Vault, and that was all fine and good where we have only one names.nsf for everyone and everything. We may have had 2-3 servers in that org.

Now, the 7,000 are in a secondary external names.nsf via DA (Directory Assistance).

The Problem

1) How do you register and maintain the people in a secondary Directory?

2) Does the DA even work with TOTP? 

The Options I See

Officially, there is only one place, and one place only, where everyone gets registered: the names.nsf.

This is not very helpful, especially given the reliance on the ID Vault for many things these days. By changing licenses, there is no way to "convert non-ID people to Notes ID people."

What do you do?

1) Copy, not replicate, the name.nsf, to extnames.nsf, move the actual names.nsf out of the way, rename extnames to names and then register everyone to it. Once done, put back the original names.nsf and off you go.

As pointed out in our Openntf.org Discord channel, the problem with this is that the user and ID would not be properly found for encryption/certification. This is a very valid point I wasn't thinking about at first. Thank you, Ulrich and Detlev.

2) register all 7,000 into the names.nsf properly, like normal. Then, manually copy the 7,000 to the extnames.nsf. Then delete, just regular delete, not Adminp delete, the 7,000 from the names.nsf.

By doing this, we preserve the user's encryption/certification, and should a name need to be renamed/edited, we can copy the user back and fix it. One could also just create a new account etc.. and remove the bad one from extnames.nsf.

3) Create a new Domain and register everyone to it and cross certify it with the existing domain. This may or may not be the answer as well, depends on circumstances.

4) I have no other idea otherwise. However, there is an AHA idea asking HCL to think about a way to register people to some other .nsf. Take a look and vote over here. 

If anyone has any ideas, let me know in the comments.

I will get to the DA question shortly.

Details and Planning

I started with my blog post and the CSV file I needed with 4 test users.

Copied the 4 test users from the extnames.nsf to the names.nsf.

Went to register the users, verified information etc..

And got this error: "The user's flat name matches another user with a different hierarchical name."

You should know that I ran into a bug in the Notes Admin and Domino Server v12.0.2, which you can read about here. Upgrading the client/server is the basic answer to resolve this one. Since Domino was already at 14.0FP1, my Notes Admin client had yet to be updated. 

Once updated, everything went as it should.

Well, only some things. 

I need to do more testing, but I think the "just updating an existing user" registration option is not working properly because I now have 2 completely different entries for each test user. 

My theory is the existing users being "unregistered web users" with "other mail" were not seen as the same people, and so Domino created the new entries. I know the ways to work around this if it is the case, but more testing will validate what I need to do. After all i will have 7,000 to update, fixing one manually is fine, but all of them? For that, I have my Openntf Admin Snippets to help me. I will blog about this after testing is completed.

In any event, I copied the 4 people out of names.nsf, put them in the extnames.nsf, and reindexed both directories as it was testing time.

Testing TOTP and DA

Ready for testing, I turned on TOTP in the Security setting, edited the domcfg.nsf for the test URL, and checked that the extnames.nsf was enabled in the DA, then restarted Domino.

When Dominno comes up, everything looks okay. I open a browser, put in the URL, and see the login screen with the MFA details. So far, so good.

The first test is my own log-in. I am in the names.nsf, and my ID is in the ID Vault. I passed, and there are no issues.

Log in as one of the new test users for the second test. Invalid password.

It seems, and this may be a customer agent, that the users are not supposed to have a web password. I added there password back, it was in my CSV, and reindexed and tried again.

Different error. User not found type error.

I was logging in as FirstName LastName, which is how I logged in, but there is only one record of me, but 2 of the test users. I logged in with the Org domain name and got one step closer, this time it just crashed on me.

Ok, I cleared the browser cache, restarted Chrome, and tried again.

This time I received a invalid access error.

This is important because the HCL documentation does not say anywhere that the DA will work with TOTP on its own. It only discusses the DA via Cross-Domain Authentication, as you can see here.

I looked at how the DA was set up and changed the Group Authorization setting to Yes from No.

Made sure Trusted for Credentials in the Naming Contexts tab was set to YES.

Then I tried it again.

And it worked this time. I was prompted to set up the MFA and log in as the extnames test user.

Conclusion

I still have more procedures to test and document, but the ability to leverage TOTP in a secondary directory via DA is not a limitation for the rollout. 

Stress testing HCL Domino and your "Other" Mail Infrastructure

 How does 1.2M emails sound?

There are people out there who say HCL Domino can't handle the stress of modern times.

It is an old system(truth be told, Exchange isn't much younger) with limitations.

Well, yesterday, I did an unintended stress test of Domino and a client's internal infrastructure.

Like most large and well-known companies, they run many Domino applications that handle millions of dollars a day but also have an O365 infrastructure.

Mail doesn't come into Domino, but it does go out from there, and this is where it got interesting.

Domino sent over 275,000 emails over about 15 minutes +/-.

This was all internal SMTP, so we didn't get spam blocked or anything,

Normally, a company would have a choke hold option on mail sent per minute, but for some reason, that was not in place or was avoided due to routing configurations yet to be determined.

And my 275K grew to 1.2M inside the network architecture.

Fun day at the office, right?

No, I caused the problem and also knew how to fix it, but that small window of time was enough to unleash the tsunami.

Sometimes, the simplest things become the hardest things, and this was such a case. 

Even veterans of the email wars screw up.

I now have a great session for F*uck Up Nights if anyone is interested.

Definitely used up one of my IT 9 lives, this brings me down to about 5 left.

HCL Domino was amazing throughout. Not only did the servers not complain about anything, but they just kept going and going and going. Multithreading FTW!

O365/Exchange did not do so well, got clogged up and had issues because it still is not a multithreaded service.

I pulled the numbers from the Mail statistics and showed them on my Admin client Monitoring dashboard. The left side is tasks, which I covered in my previous blog post. On the right side, you can add statistics, something not many people do, but I like to see some things there and probably should do a separate session on the statistics, but that is for another time.

My peers, I am sure, can guess what caused the tsunami, so there is no reason to elaborate. But let's just say when you are a junior admin, this is one of the outcomes of your trial-by-fire Domino Administration education.

For those who think Ambassadors and long-time Yellowbleeders are these great Gods of tech, some really are titans, I admit my mistake and that, indeed, you are never too young or old to learn something new....or mess up royally.

Domino RESTAPI Bug and WorkAround

This is not my usual line of thought as an Admin, but sometimes, AdminOps is better than DevOps because troubleshooting is not an exact science.

While customers over the last year or so have been asking about the HCL Domino REST API, my reply is usually something like, I can install it, but you are on your own afterward, or I point them to a Developer friend.

To be fair, HCL will help them/me with getting started or "where is/How do I" questions. But this is about the bug my client and I discovered and how to work around it.

While updating the v1202FP3 servers to v14FP1, all went okay, even with the change in Java classes, until we got to the REST API server.

If you had downloaded and added my tasks update for the Admin client, you would see that RESTAPI was running on the machine, as I saw. I let the customer know I would upgrade it from 1.04 to 1.014.

Usually, it's not a big deal; you run a long string of commands or broken down into 4 lines like the example shown in the documentation which I prefer as my typing is not perfect.

java -jar restapiInstall-14r.jar ^ -d="C:\Program Files\HCL\Domino\Data" ^ -i="C:\Program Files\HCL\Domino\notes.ini" ^ -p="C:\Program Files\HCL\Domino" ^ -r="C:\Program Files\HCL\Domino\restapi" ^ -u

 You then type an A to accept the update, which will upgrade the REST API code.

It replaces the existing files with new ones and any updated files.

Great. I rebooted the server, and it looks like everything is up. The task view in the Admin client shows REST API is up.

Test a few things, but Swaager is not connecting to anything.

My initial thought was that the developers had some code that had been deprecated or maybe not valid with something in v14. I wasn't far off. I tried a few things to get it to work, and then I decided to look at the schemas. I found them deactivated, which seemed wrong to me, so I enabled one to test.

And it worked again.

Great! I did the same trick with the others, and they also worked.

One last test, shut down Domino and reboot the box clean to ensure it was ok.

No luck. Back to square one, but now the schemas all showed activated, but still just errors like this one:

This not being my first effort, I figured let's deactivate everything and reactivate it and test it.

Sure enough, it worked.

So, in my mind, the problem was somewhere between the REST API code and some type of flag on the schemas/databases that just wasn't being accepted.

Opened a ticket with HCL to discuss the bug I found.

After the usual back and forth, get us this, debug that, a copy of the NAB, a db, schema db, etc.. HCl said they could not recreate my problem, yet here it was.

We had an online meeting so the dev team could get a good luck at my testing and poke around deeper.

They agreed something was fishy and went off to look into it further.

In the interim, given this was a key production server, we wanted to revert back to the old REST API version, which worked fine prior to the upgrade. But would it work on v14, among other questions we had?

HCL and I discussed it with the customer, and this is how you revert to an older version.

We had the old code downloaded, which is key as sometimes HCL has a way to make older versions "disappear" from the public. The steps included:

1. Shut down Domino
2. Copy the Domino\restapi folder contents out from Domino to a backup space.
3. Once the RESTAPI folder is empty, you can then walk through the installation steps as I showed them above, but the last line instead of -u should be -a.
4. This will reinstall all you need(presuming all URLs, folders, etc are NOT being edited/changed)
5. Start Domino
6. Test

This worked well, and the customer was back up and running on V14FP1, albeit with the older REST API code. 

After a few days, we got back some information from HCL, which I quote directly below:

Problem:
REST API 1.014 Running the APIs in local swagger returns Error "Failed to Load API Definition"

Possible cause
The issue is that the scope's Server name is not correct.
This setting was not working correctly in older versions of DRAPI and was fixed in v1.0.6. It was pretty much ignored in v1.0.4.

Possible solution/Workaround:
With DRAPI 1.0.14 you have to add the CN name/hierarchical name of the Domino server instead of host name for example: CN=customerTest/O=HCLLabs in the Edit scope>>Server field>>CN=customerTest/O=HCLLabs or CustomerTest/HCLLabs
Earlier server name field in scope>>Server field was: Host name of the server i.e keithbrooks.com

Change it to either CN name or the hierarchical name of the Domino server or can be left blank stating any server that has this scope in the KeepConfig.nsf assumes the database / schema exists.

Moving further the product dev team is going to update the error message in the product code as 'You need to query a different server' so it makes more sense to the testing on affected DRAPI versions.

There you have it. My guess initially was partially correct. 

The configuration was correct for the old version, but because the customer did not upgrade the RESTAPI code along the way, they missed the changes in code at 1.06, which would have probably prompted an HCL ticket at that time as well, but it would have been easier for all of us to see as the issue at that time. as the change would have been fresh in their mind.

The story's moral is, of course, ABU or "Always Be Upgrading" because of security, code, and functionality adjustments over time. While we don't want to break production, sometimes you must do so for your benefit.

The second moral is to have a Development or Staging environment for these critical applications. In this case, the lower environment did not match the upper one entirely, so the problem was not seen when we updated the lower environment.

An Admin Present You Didn't Know You Needed

 Hi, welcome back to my burnt out blog. 1,500 posts and, well, I am kind of burnt out, but that doesn't stop me from giving to the community these little bits.

I'd like to write more but a lot of things have been internal client items that I can't write about, but I am active.

Preamble, excuses out of the way, so, who wants some goodies?

About 2 weeks ago, I gave an impromptu webinar for Openntf.org as a last-minute fill-in.

Openntf, for those that don't know, is the Notes/Domino+ community, where devs, admins, business people, HCL, and others share code and ideas, templates, and projects for the benefit of the greater worldwide community.

I wanted to inform people that monitoring Tasks in the Administrator client has some changes.

Why is this important? Because unless you are a 1 server company, you have a lot of information to remember, such as:

1. How do you know if DBMT ran? 
2. How do you know which server Certmgr runs on?
3. Which web server do you run the Domino REST API on?
4. Which server handles your Backups and Restores, presuming you leverage the v14 options?
   
5. Is NOMAD running?
   
6. Is your DirSync working?
   
7. Are you sure the awesome OnTime Group calendar is running?
8. Have you enabled Aautoupdate yet? One look and you know.

Intriguing questions, right?

Between v9 and v12, nothing changed in the tasks that could be monitored. Traveler seems to have been the last item added, and that was from 8.5, but it found its way into the Monitoring Dashboard in v9.

Now comes v14 and HCl has cleared out some older items, like x500 info and the Fax server....but did not add any of the newer tasks that have come along since v9.

To be fair to HCL, it is not as simple as a few fields and renaming a file.

But fear not my fellow Admins, for I have not only explained it all in my presentation, which you can watch over here on the Openntf YouTube page, I have made the tasks available for everyone to update their Admin Monitoring Dashboard.

If you just want the slides, go here.

And because I know you are probably as lazy as I am, I have made the forms available with instructions to help you get more from your Monitoring Dashboard.

Go get the tasks from my Openntf project over here.

If I missed a task that is not listed, let me know, and I will update the project database.