Domino RESTAPI Bug and WorkAround

This is not my usual line of thought as an Admin, but sometimes, AdminOps is better than DevOps because troubleshooting is not an exact science.

While customers over the last year or so have been asking about the HCL Domino REST API, my reply is usually something like, I can install it, but you are on your own afterward, or I point them to a Developer friend.

To be fair, HCL will help them/me with getting started or "where is/How do I" questions. But this is about the bug my client and I discovered and how to work around it.

While updating the v1202FP3 servers to v14FP1, all went okay, even with the change in Java classes, until we got to the REST API server.

If you had downloaded and added my tasks update for the Admin client, you would see that RESTAPI was running on the machine, as I saw. I let the customer know I would upgrade it from 1.04 to 1.014.

Usually, it's not a big deal; you run a long string of commands or broken down into 4 lines like the example shown in the documentation which I prefer as my typing is not perfect.

java -jar restapiInstall-14r.jar ^ -d="C:\Program Files\HCL\Domino\Data" ^ -i="C:\Program Files\HCL\Domino\notes.ini" ^ -p="C:\Program Files\HCL\Domino" ^ -r="C:\Program Files\HCL\Domino\restapi" ^ -u

 You then type an A to accept the update, which will upgrade the REST API code.

It replaces the existing files with new ones and any updated files.

Great. I rebooted the server, and it looks like everything is up. The task view in the Admin client shows REST API is up.

Test a few things, but Swaager is not connecting to anything.

My initial thought was that the developers had some code that had been deprecated or maybe not valid with something in v14. I wasn't far off. I tried a few things to get it to work, and then I decided to look at the schemas. I found them deactivated, which seemed wrong to me, so I enabled one to test.

And it worked again.

Great! I did the same trick with the others, and they also worked.

One last test, shut down Domino and reboot the box clean to ensure it was ok.

No luck. Back to square one, but now the schemas all showed activated, but still just errors like this one:

This not being my first effort, I figured let's deactivate everything and reactivate it and test it.

Sure enough, it worked.

So, in my mind, the problem was somewhere between the REST API code and some type of flag on the schemas/databases that just wasn't being accepted.

Opened a ticket with HCL to discuss the bug I found.

After the usual back and forth, get us this, debug that, a copy of the NAB, a db, schema db, etc.. HCl said they could not recreate my problem, yet here it was.

We had an online meeting so the dev team could get a good luck at my testing and poke around deeper.

They agreed something was fishy and went off to look into it further.

In the interim, given this was a key production server, we wanted to revert back to the old REST API version, which worked fine prior to the upgrade. But would it work on v14, among other questions we had?

HCL and I discussed it with the customer, and this is how you revert to an older version.

We had the old code downloaded, which is key as sometimes HCL has a way to make older versions "disappear" from the public. The steps included:

1. Shut down Domino
2. Copy the Domino\restapi folder contents out from Domino to a backup space.
3. Once the RESTAPI folder is empty, you can then walk through the installation steps as I showed them above, but the last line instead of -u should be -a.
4. This will reinstall all you need(presuming all URLs, folders, etc are NOT being edited/changed)
5. Start Domino
6. Test

This worked well, and the customer was back up and running on V14FP1, albeit with the older REST API code. 

After a few days, we got back some information from HCL, which I quote directly below:

Problem:
REST API 1.014 Running the APIs in local swagger returns Error "Failed to Load API Definition"

Possible cause
The issue is that the scope's Server name is not correct.
This setting was not working correctly in older versions of DRAPI and was fixed in v1.0.6. It was pretty much ignored in v1.0.4.

Possible solution/Workaround:
With DRAPI 1.0.14 you have to add the CN name/hierarchical name of the Domino server instead of host name for example: CN=customerTest/O=HCLLabs in the Edit scope>>Server field>>CN=customerTest/O=HCLLabs or CustomerTest/HCLLabs
Earlier server name field in scope>>Server field was: Host name of the server i.e keithbrooks.com

Change it to either CN name or the hierarchical name of the Domino server or can be left blank stating any server that has this scope in the KeepConfig.nsf assumes the database / schema exists.

Moving further the product dev team is going to update the error message in the product code as 'You need to query a different server' so it makes more sense to the testing on affected DRAPI versions.

There you have it. My guess initially was partially correct. 

The configuration was correct for the old version, but because the customer did not upgrade the RESTAPI code along the way, they missed the changes in code at 1.06, which would have probably prompted an HCL ticket at that time as well, but it would have been easier for all of us to see as the issue at that time. as the change would have been fresh in their mind.

The story's moral is, of course, ABU or "Always Be Upgrading" because of security, code, and functionality adjustments over time. While we don't want to break production, sometimes you must do so for your benefit.

The second moral is to have a Development or Staging environment for these critical applications. In this case, the lower environment did not match the upper one entirely, so the problem was not seen when we updated the lower environment.