Thursday, June 25, 2026

ClamAV and Domino

 I am not an expert on Anti-virus things and Domino, that would be Daniel Nashed.

But if HCL Domino has something new, I test it out and set it up on my server.

In this case, I had heard that ClamAV would now work with Domino as of 14.5.1.

Having used Clam for many years at the OS level, I wanted to play with this in Domino.

My server is Windows, not Linux, ClamAV Linux instructions are pretty common, Windows, not as much.

Boy was this the hardest "easy' thing to do ever.


To start, HCL provides no documentation.

There are only 3 references to ClamAV in the official docs:

https://help.hcl-software.com/domino/14.5.1/admin/wn_config_features_1451.html#wn_config_features_1451__section_fq1_wgc_23c

https://help.hcl-software.com/domino/14.5.1/admin/conf_configuringscanninginscscancfg.html?hl=clamav

https://help.hcl-software.com/domino/14.5.1/admin/conf_scanningattachmentsforviruses.html?hl=clamav


None of which tell you much about how to set up ClamAV or how to configure it for Domino integration.

I opened a ticket and eventually was directed to this post:

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0129703

Which is in Japanese. Use Google Translate or Chrome built in translstion.

I see now, at my urging, HCL has released the same document in English.

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0131220


Their perspective is that we should go to ClamAv to set it up and then magically know how to integrate it.

I asked why it wasn't in the actual documentation and got a runaround answer, given HCL announced the availability, not ClamAV.

The new technical document covers some of the set up required.


I am going to add what is missing, more for my benefit, and anyone else who will try this on their servers.

Step 1 is Download the ClamAV from the link in the document, https://www.clamav.net/downloads and install it. BUT this does not tell you how to set it up in Services.

For that, you perform this:

  • Open the Windows Command Prompt as an Administrator and navigate to your ClamAV folder.
  • Run the following command to install the ClamAV daemon service:
    clamd.exe --install

  • Remember to set it to automatic.

    Step 2 is to copy the config file as described and perform the edits.

    Step 3, however, did not work for me no matter how I tried to get the certificate. Even with HCL Support help, they eventually had to send me the certificate required.

    Step 4: You have to get the updated virus definitions. NOTE: This is a once off effort. And Manual.

    If you want it to be automatic, and you do, you need to do these steps.

  • Open the Windows Command Prompt as an Administrator and navigate to your ClamAV folder.
  • Run the following command to install the ClamAV daemon service:
    freshclam.exe --install

  • Once you Load mailscan at the Domino server console you will get the cscancfg.nsf and need to follow the document entries as provided.

    NOTE: The ClamAV server name(DNS) must be 127.0.0.1. Do not put your server IP there, it will not work at all.

    Make sure you open port 3310 with your local server firewall, and in my case, the outside ISP.

    I changed the Subject Prefix Scanned field default text to something I would know came from my server, not a spammer.

    Lastly, make sure you add Mailscan to your tasks line in your notes.ini also something not mentioned in the document that is kind of important, unless you like having your mail stuck in mail.box and spend 2 hours troubleshooting it one morning, like I just did.

    I eventually figured out it was an AV issue when running SMTPDebugClient=1 and

    Tell Router list run from the server console.

    You see this:

    Mbx NoteID          ID          State          Size Pri Count ScheduledDate           From

      1 000008FE 0028261D Wait AV        8443         1                         K Brooks/org

      2 00000902 00283C39 Wait AV       13306         1                         kbrooks@test.com

      1 00000902 002864E3 Wait AV        8749         1                         keith@gmail.com

      2 00000906 00287C9A Wait AV        8640         1                         keith@gmail.com


    Oh, that makes sense after the server restarted late yesterday. Mailscan was not in tasks.

    So, my time spent, is your time gained.


    Tuesday, May 12, 2026

    12.0.2 FP8 didn't update version numbers

    This is probably not a problem with FP8, to be upfront, but just in case I am using it for the title.

    4 out of 8 servers at a customer did not chaaange tehir version number when I updated them to FP8.

    They stayed FP7.

    In the past, I have had this issue where updates don't happen, due to various reasons like a corrupted NAB, failed hot fix removals, etc..

    But this was different because 4 servers, DEV domain, were fine.

    QUAL domain is not updated.

    How Did I Get Here

    Servers were 12.0.2 FP7 HF1 Windoews 2019

    I shut the Domino server down, including the Domino service.

    Paused the Windows Management service(ONLY affects HF's for some reason), then ran the hot fix as administrator to remove it.

    Started the server from the icon asd a regular application, so I can see it running, and a "sh ser" showed the correct FP7 version. Let it run for a few minutes till I saw HTTP start and some other tasks. Then shut it down.

    Run FP8 as an administrator, runs fine, no errors.

    Restart the box, actually, 2 servers had issues with the Domino service login, and I restarted the whole VM, the other 2 the service started fine.

    Waited to see the server docs change, and nothing.

    The upgrade.log showed everything looked ok.

    I opened a ticket with HCL after the 1st 4 failed to update, then I checked the next 4 and those worked.

    The Resolution and Reasoning

    Sooraj from HCL Support provided me with the following resolution and an explanation after I provided some files and screenshots.

    Take a backup of the files below and replace them from a "working" server [12.0.2FP8]

    Domino\nstrings.dll
    Domino\res\C\nstrings.res

    Shutdown the incorrect Domino server
    Take a backup of the two files mentioned above.
    Copy similar files from a working 1202FP8 server and replace them.

    (I left the existing ones, and added a _BAD at the end of the name)

    Start the Domino server.

    While I asked to close the ticket, I also wanted an explanation for future thoughts. 

    Sooraj provided the following:

    This problem was reported for the first time, and based on my analysis, it appears that those two files were not updated by the FP8 installer. It was still using the FP7 code. This ideally occurs due to an access issue with the account that started the installation, but unfortunately, I cannot confirm this because the upgrade.log doesn't show any such errors.

    Possibly a server processing issue, a delay, or perhaps not fully completing a virus scan, or some other local issue was to blame.

    But on 4 servers, I don't believe so. 

    One, sure, things happen. 

    In this situation, all 4, well, 8, are nearly identical setups.

    Hope this helps others with similar issues at some time in their admin journeys.



     

    Thursday, April 23, 2026

    Engage 2026, the Year of AI and my session

     There were 11 AI sessions out of about 70, not including roadmap or sponsor/commercial sessions.

    I figure in a year or 2 we won't be saying AI and just, look what I built.

    Thank you to everyone who messaged me asking where I was hiding. It wasn't my intention to miss the event, but I did not make it.

    Instead, I was live from my apartment, doing my session. What may have been a first for Engage, a solo session done remotely!

    Due to my flight travel plans being interrupted and changed a few times, it became clear I may not even get to Engage until after my session would have started, and have to stay a day later, but leave much earlier, and this caused quite a few issues, which I discussed with Kris and reluctantly had to give up trying to attend.

    Kris said he would hold my space in case things changed for the better, but alas, that was not the case.

    However, he said we could try to do it remotely. Webinar mode basically. 

    Thanks, Kris and Tom, for the chance and possibility!

    I agreed we could try it, not expecting much or even how we would do it, so I labored on with my slides.

    I asked the OpennNTF Board, which I am part of, if we could use our webinar meeting app and if someone could connect everything in Ghent.

    The Board agreed, and it was game on.

    Roberto was very helpful in getting it all set up, Thank you Roberto! I had presumed I would lose 5-10 minutes with setup and adjusted my session accordingly. It was only 7 minutes.

    My next thought was, why would anyone come to my session, knowing I wasn't there to give it?

    Turns out, about 15-20 people ventured into the room, and I think they were surprised by what they saw and heard.


    Full deck is here: https://speakerdeck.com/kbmsg/ai-making-admin-and-users-lives-better

    My session was supposed to be on AI tools/apps that anyone had created to help users and admins. To get people thinking more practically about what could be done.

    Along the way, I discussed why citizen developers who leverage AI, or anything, don't exist as they did 25 or even 10 years ago.

    With locked down browsers and IT blocking changes to applications, let alone updates, and the lack of a Domino Designer client installed (even though it is no longer an extra charge), it is just not easy for users, or even shadow it, to create anything, let alone use it. You can read more about this on page 8 of my slides.

    But we admins and developers are not as constrained, aside from browser lockdowns.

    I had hoped, had I been at Engage, to go to the Monday HCL workshop on vibe coding. There was also Bernd's session on Tuesday around the advances his company has made, which I looked forward to for his insights. Then there was Dominic's session on Wednesday, also aimed at Admins and AI, which, if I had read all the sessions in advance, I would have tried to compare notes earlier. To be fair, I often don't get around to reviewing the agenda until the week before the event.

    I work with some entities that I am not allowed to post about, which explains my dearth of blog posts. But that doesn't stop me from creating things I need to work with in their environments.

    My main point of the session was to show a script(LotusScript, server commands, Java, and formulas) building tool for Domino admins. I am not a developer, although I can edit existing code and make things do what I want. When it comes to doing it from scratch, I am useless.

    Enter AI, and I said, " I bet I could create something to make my life and other admins ' lives better".

    I give you my Domino Script AI Tool.


    You can go use it here: https://kbmsg.github.io/domino-scripts/

    Or pull it from my GitHub: https://github.com/kbmsg/domino-scripts

    The slide deck discusses some of the templates and what it does; the GitHub site provides more details and how to start using it.

    I built it to work with Claude.ai, and feedback already asked that I let Ollama be used too.

    I started the ollama config, but it is not quite working. If you see it say v66 or newer and followed by Ollama works, like it says just now, you are good to go!

    Domino Script AI v64



    This will be posted to https://OpenNTF.org as well.

    AI vibe coding is literally like building your parachute after you jump out of a plane. Sure, you get most of it right, but that last 10% is the problem, and it takes time.

    But, and this is a huge but, if I tried to do this from scratch, it would take me days and days. While I am on iteration 65 or so, all together it took only a few hours to get here, but stretched out over a few days, giving time, resources, and work.

    If you have ideas for me to include, feel free to comment here or via GitHub or the OpenNTF Discord channels. 

    PS - If you are in a licensing situation and need to sort the CCB vs CCX people, you may want to try the Dump ACL to CSV template. As discussed on page 16 in my slides.


    Sunday, June 29, 2025

    HCL Domino 14.5, Fix your iNotes Redirector for Verse

    By now, if you have updated your Domino server and relied on iNotes, you have found that it no longer works.

    HCL was clear about this, and there is no workaround, so if you want iNotes and not Verse, better keep a server on 14.0 and point everyone there.

    For the rest of us that move forward with the times, we will have seen that logging into the iNotes URL gets you to a pop-up which looks like this:

    HCL iNotes is no longer supported

    It states that iNotes is no longer available, and you need to open HCL Verse. If you click on the link, it directs you straight to Verse.

    The URL for verse is something like https://companyname.com/verse, and you change your bookmark, but that doesn't work exactly. When you hit that URL, you will get the framework of Verse but this message:


    So, you try the manual way, https;//companyname.com/mail/brooks.nsf, and that lets you in, via the 1st pop-up as above..

    The next morning, you open your browser, and the /verse URL is not working with the same error about recent contacts.

    What is going on here?

    iNotes required the iNotes redirector. Well, there is no new redirector, but it did get renamed to Verse redirector(same template name), but it does include some options you probably hadn't considered, and this is how you resolve the problem.

    If you haven't changed any settings in a while, now is a great time to update them. The redirector has been updated periodically along the way.

    These are the ones to change or review:

    Server Settings Button:

    Do you wish to force SSL for the entire session? Yes

    UI Setup Button:

    Your company logo should be there, so your users know it is a legitimate site.

    Enable Personal Options: Set to No, they were really about iNotes anyway.

    Enable HCL Verse: Yes, you need this now.

    Enable SAML options: If you use SAML, enable this option, which displays a secondary option for the Default Mail Application. Select Verse.

    When you have finished all your changes, click Save & Exit and then test to ensure everything works properly.

    And no more weird error messages about recent contacts.

    This tech doc can help those who want more details: https://help.hcl-software.com/verse_onprem/2.0.0/admin/vop_configuring_server.html

    Update June 30: I found that I now see this error the next morning.


    Step 19 is what you want.


    In my case, I already had a credstore created, and this new key would not get added to that credstore.

    I went back to the original technote, vop_configuring_server, and added a new entry in my domcfg for DWA, but that now redirects me to a different domain on my server, which has TOTP.

    Also verified that the 'inotes/verse' tab in the configuration document met the proper configuration as per the document.

    As it turns out, my secondary domain was set for SAML instead of single server in my Internet Site document, Domino Web Engine tab. Probably because I was testing something. Changing that back to single server now allows me to log in correctly with https://companyname.com/verse. With my erse redirectpr in place and everything now correct.



    Thursday, May 29, 2025

    What was my session about at Engage? Money and your Job

    It was a cathartic, contrarian approach to our current situation, in which we, the admins, developers, and business partners, are supporting our Domino worlds.

    I wish this were all in my head. But every single person in the room for my session at Engage was nodding their head at some point and others had very serious faces when confronted with my thoughts. The couple of people that talked with me afterwards were very happy they came to it. After all they gave up the Domino IQ roundtable for my session.

     

    How did we get here?

     

    The road that got us here is not the road that will get us out of here.

    But enough cliché’s let’s be realistic.

    We put ourselves in this situation.

    Some items to think about:

    (Almost) Every time we had a chance to ask for a budget, we failed.

    We got what we needed, but this is not what an Enterprise solution desires, deserves, or requires.

    We are selling our environments short if we only ask for licensing money, salaries, and some equipment. Way too short.

    If your annual budget request does not include enough money that a respected enterprise solution requires, your request will not go up to the budget committee. Each company sets these meetings at various levels. Enterprise ready products have a staggeringly large annual budget, because that is how you know what is important. Or so the executives believe.

    Maybe you think this is too much for “your team”.

    Here is a little secret: Security budget money usually does not come from your team. It comes from the security budget. Leverage that every year.

    Your environment needs to maintain perpetual security, which includes external and internal resources, people, apps, disaster recovery, development architectures, and more.

    Everything you do in your Domino environment is for the company's greater good, and extending easier, faster processes for everyone, in a secure and meaningful way, needs to be backed by a secure environment.

    I am not talking about 10k, 5k, or even 50k. You probably need to be asking for at least 100/250 or 500K to get to the budget executive committee. These days, maybe starting at a million. This is obviously relative to your situation and industry.

    Some of you reading now might think I am out of my mind, but I am just a contrarian who has learned from 30 years of mistakes.

    Maybe licensing is handled by a different group, so you didn’t need to ask for a budget. This also happens.

    Once you stop asking for a budget, you don’t get invited to ask for one in the future. You mysteriously get left off the group email, or worse, your boss stops asking you about it. It is very hard to get back on the budget list. VERY HARD. 

    For those who say, "No, we submit a budget," my question is, do they take it seriously, and do you get funding beyond the licensing basics?

    Developers, you fall under this spell as well.

    How many times have you said (as an employee), "Oh, I can put something together. Give me a few days or a week or two." Every time you did this but did not chargeback some business units' code or lay out the process and document why the app is needed and what it does for the company, you have also failed your Domino infrastructure.

    If you were an outside vendor, you would have a project plan and bill X amount of hours or a total cost.

    But you think that as an employee, you can’t do that, and you are wrong.

    Everything goes to a cost center somewhere.

    The group you are writing this for should be paying your team some of their budget.

    But I bet few of you have done this.

    While internal cost centers are a BS accounting move, they do work with your budget, so play ball by their rules, not the ones in your head.

    Fear of Unemployment

    I understand your predicament. You don’t want to lose your job, and I don’t want you to, either. However, no one has ever been fired for asking for budget money. A raise is a different matter.

    For some reason, employees started to think they would get punished if they asked for the budget. You don’t get everything, but have you budgeted for the 2-8 various updates (now multiply by how many servers you have) you may need to do all year for your Domino environment? That number gets large fast, especially if you assign a value to it.

    Why assign a value?

    Because you should be clawing back money from every cost center. Again, internal budget money matters. Think about your budget if you had been "paid back" by the other cost centers.

    When Lotusphere/Connect/Connected existed, you had to beg your company to go sometimes. How many of you went to the HCL Factory Tours? I hope you all knew they did these. If not, why not? You should all have been at them.

    Did you think you would get fired for asking to go?

    No, usually, the fear is to do anything that will get Domino seen inside your company. The CIO may demand someone's head for saying the D word, or your boss gets pushback for the old software. Whatever the reason, I understand, but keep in mind, you are not helping yourself in the long run because you aren't giving your company any reason to stick with Domino. 

    I also blame the HCL sales rep at this point because they should be talking to you regularly by phone, email, WhatsApp, SMS, X, TikTok, set up an Ai auto message, whatever, I don't care, but they need to be on your side and not just "we have a webinar" or licensing renewal times. 

    To be fair to the HCL rep, they do what makes them money first before anything else, but if they don't want to lose customers, they could be more proactive, so could most of us, that's all I will say here.

    Money is everything, and you don’t understand what money means.

    Money means one thing to you, but it is totally different to executives. 

    You think asking for 10k is too much when the executive deals with million-dollar deals.

    I am not saying deals don’t get lost because of price; that does happen. However, when properly laid out and defined, a solution for your company is not a 25k over-the-weekend app. If you want the executives to take it seriously, it is a 500k project or equivalent for your organization.

    They will take it even more seriously if you can show that the new app either helps make money or manages some part, if not all, of your company's manufacturing, selling, marketing, or finance. 

    Do this, and you can ask for even more money.

    I am not talking about fake ROI projections of “time saved,” but real money tied to real work processes.

    If you used your budget for licensing, what else would you ask for? I have listed what I expect to be in your budget in the spreadsheet I have uploaded for everyone. I presume you are the budget submitter; otherwise, you must bother your boss more.

    Spreadsheet for download.

    It includes guideline pricing, multiple items, education, conferences, T&E for these events, and other items that should be clawed back from cost centers.

    It is also laid out as though you built an environment properly, and you can understand why, as you start playing with it, how the numbers multiply easily the more servers you have, and some costs also multiply. I aimed to show that all your costs after the Year 1 hardware purchases should be budgeted for you and cover your expenses.

    Some of you may find this enlightening, while others may find it far from reality. I will wager that few of you even have something laid out like this, on which you are basing your budget. I hope I am wrong.

    UI/UX is NOT HCL's Fault

    All the BS talk about a poor UI/UX falls on your developers, now.

    Older apps did have limitations, but this is not 2001 and R5, why haven't you refreshed them? Even just using Restyle would help. NOMAD web can be your friend here to reduce the web overhead to convert an app from Notes only to web, but you probably do need a nicer UI.

    There are some gorgeous apps, Notes and web ones, so we know it is possible.

    Again, who is to blame for this? Your app's UI/UX is not HCL’s fault.

    Your executives see it as IBM/HCL's fault, but that only relates to the mail template. Your apps are your own to be creative.

    Speaking of executives.....

    New Executives have no idea what Domino does for your company

    Because NO ONE has told them.

    In this situation, I am talking about apps here. Mail is basically a cost of doing business. While on-premises and keeping your data inside are essential and more secure, email is a cost.

    Some of you have been trapped by this “cost” term. Just because email is a cost and you are told to reduce expenses(notice they never drop new applications or revise older ones to save money), you may have been mistaken in thinking you shouldn’t ask for more money. But this, again, is what got us into this mess in the first place.

    Servers do not update themselves; they can now under v14.5, but you get my point.

    But apps, apps run your world, make you money, reduce paperwork (hopefully), and should be so good that no one is looking to replace them. 

    Every time a new executive comes on board —and let’s face it, executives bounce between companies every 2-5 years—how have you briefed them on your Domino apps and what they do for the company, what they make for it, or what they provide that would cripple you if it suddenly stopped?

    Have you? Has your HCL rep? Have you even told your rep what your apps do? Your HCL rep can't help you be that internal champion if you don't help them with what exists.

    In my session, I used one of my clients as an example. They make millions of dollars a day using their Domino app worldwide inside their company. It handles everything from initial email requests for pricing and online sales to shipping, inventory, and manufacturing, tracking everything along the way. It is typical of Domino and an insanely elaborate application put together over two decades of teams of developers.

    And yet, EVERY MEETING that involves leadership, I have to go to bat for my client and their app. Their internal people are too afraid to be connected to the “old server” stuff. See the above section Fear of Unemployment".

    Imagine that meeting, discussing when we will move off Domino (I have been managing their environment for over 10 years or so) soon, but hearing this every year from me that it keeps them running.

    They back down and then ask what is replacing it, which I suggest nothing, since it is an active and constantly changing environment, why would they replace it. Keep in mind that they only know the app by its internal name, which is how I introduce the topic, because they have zero idea it is a Domino application. But it always comes up from the Microsoft guy.

    Of course, the business units say some cloud thing or an app made out of another database or workflow solution. Anything they move to would be a considerable cost, and ALL THEY WILL DO IS REWRITE WHAT THEY ALREADY HAVE, and from experience, end up with less functionality. I offer our help with whatever route they choose.

    Like I said, 10 years, same meetings, every year. Many partners have similar stories, because we believe in the product, management believes in their bonus and their next salary when they move on.

    This is exactly what one person in the meeting said about his situation. While he isn’t happy about the reduction in physical servers (personally I prefer less, but clustered, even when you have remote locations) he agreed that he will be there until he retires, nothing is moving that fast. 

    Management has made the decision in his case, but to be fair, their environment might be faster and perform better with less overhead from the dozens of servers being consolidated.

    Details matter, but for every excuse or peculiar circumstance, a completely normal one is ignored or under-budgeted until it becomes too late.

    I hope my session and this blog post help you never reach that point of no return.

    Conclusion

    TL:DR The problem is and has been us.

    I hit on other items that we all see or hear when advising clients. But almost everyone should know how to handle those by now.

    If not, reach out to me. The lack of integration FUD, the “old software” line, the “lack of knowledge out there” hiring managers, and the perpetual “everyone uses <insert some other company>” lazy man excuse from someone that doesn’t want to be accountable for anything.

    But we could turn this around by addressing the money issue and how we handle pricing/budgeting.

    If you need help with this discussion, contact me, my fellow HCL Ambassadors, your Business Partner (or dump them for me), your HCL sales Rep/ Customer Success Manager/Customer Success Executive.

    We are at a fulcrum. The cycle is coming back to in-house data from Cloud data, and now is the best time to Make Domino Great Again in your company and champion it as a proper Enterprise solution.

    Thursday, May 22, 2025

    My IQ has gone up thanks to Engage

    Engage just finished. The first one is under new management, and it is in safe hands.

    I don't think I have ever seen Theo so happy with a big grin on his face the whole time. Tom and Kris did really well for their first time out. And they told us next year Engage will be in Belgium.

    I had to miss last year's due to the timing and circumstances of the war at home, so it was great to talk to everyone. Thank you all for asking about the situation and the family. It means a lot.

    The star attraction was Domino IQ, which adds AI to the core product. As an Admin, all I can say about this is that you will need to spend time building this out properly if you want to move beyond the basic regurgitation of your knowledge base or other internal documents. Also, IQ needs some extra hardware and configuration, so be prepared and read the what's new and requirements docs. Your developers will be busy.

    It will be available starting with v14.5, which is coming out June 17! IQ is part of your license if you are under CCB Term(I believe). If you are a perpetual license org, this is your incentive to change licensing plans.

    I was happy to see friends from past Engage shows, and in some cases people I hadn't seen in many years. Yes, we all got older, but we still have fun together.

    There was a tease of Notes.Next but it is far from prime time ready so don't ask about it yet.

    Roadmap sessions were broad with partial timelines for new items.

    One thing that also stood out is the added accessibility to all the products which need to be up to certain laws soon and so that is taking over a lot of dev time.

    As an Admin, I want to let everyone know to go get Cormac's slides from his session. His session was filled with great ideas and details about cleaning up your administrative items, and as he said, even old admins can learn something new, which I did about SMTP failover routing.

    His blog is here, and hopefully, he will post his slides there or at the Engage website.

    Lastly, the Lotus brand might return for at least one product. Or not. While this was from Richard Jefts' opening keynote, later on, other HClers implied the name may not stay.
    I will leave the debate about the brand to others, as both sides of the argument have their validity.

    I was also in some partner/ambassador/HCL Roundtable sessions, which I can't say much about, but there are plans that will help the partners in the long run, which in turn will help you, the customers.

    I think my own session scared a lot of people, but I hope it also helped them rethink how they and their teams interact with upper management and budget committees. My slides will not be posted, and I don't publish my Competitive Intelligence info. I will probably make a second blog post about it over the weekend.

    Safe travels home everyone, talk to you soon or see you, at least virtually, some time soon.


    Tuesday, August 20, 2024

    SnTT - Does TOTP Work for users in a Secondary Directory via DA

    TOTP, DA, and Domino

    For the last 3 years, I have worked with TOTP inside HCL Domino and customers with unique requirements.

    This has provided fodder for my blog, and today, we have a new entry into the TOTP Mystical Ways of the World.

    Let me state my usual caveat upfront: TOTP is about the URL, not the server, the database, or the user.

    You enable TOTP for each URL you want on your server.

    PSA is completed. Let's discuss the circumstances that brought me here.

    Like many of our customers, a customer has a large external user community relying on their applications.

    The customer has licensed this with HCL, so I am not going to get involved in that discussion. However, be warned: It is not a comfortable one if you have been relying on some old licensing options and now fall under the new ones.

    We have about 7,000 external customers. Some are undoubtedly old customers, but 7,000 is a lot of people.

    Previously, I wrote about how to bulk add these people into your ID Vault, and that was all fine and good where we have only one names.nsf for everyone and everything. We may have had 2-3 servers in that org.

    Now, the 7,000 are in a secondary external names.nsf via DA (Directory Assistance).

    The Problem

    1) How do you register and maintain the people in a secondary Directory?

    2) Does the DA even work with TOTP? 

    The Options I See

    Officially, there is only one place, and one place only, where everyone gets registered: the names.nsf.

    This is not very helpful, especially given the reliance on the ID Vault for many things these days. By changing licenses, there is no way to "convert non-ID people to Notes ID people."

    What do you do?

    1) Copy, not replicate, the name.nsf, to extnames.nsf, move the actual names.nsf out of the way, rename extnames to names and then register everyone to it. Once done, put back the original names.nsf and off you go.

    As pointed out in our Openntf.org Discord channel, the problem with this is that the user and ID would not be properly found for encryption/certification. This is a very valid point I wasn't thinking about at first. Thank you, Ulrich and Detlev.

    2) register all 7,000 into the names.nsf properly, like normal. Then, manually copy the 7,000 to the extnames.nsf. Then delete, just regular delete, not Adminp delete, the 7,000 from the names.nsf.

    By doing this, we preserve the user's encryption/certification, and should a name need to be renamed/edited, we can copy the user back and fix it. One could also just create a new account etc.. and remove the bad one from extnames.nsf.

    3) Create a new Domain and register everyone to it and cross certify it with the existing domain. This may or may not be the answer as well, depends on circumstances.
    4) I have no other idea otherwise. However, there is an AHA idea asking HCL to think about a way to register people to some other .nsf. Take a look and vote over here

    If anyone has any ideas, let me know in the comments.

    I will get to the DA question shortly.

    Details and Planning

    I started with my blog post and the CSV file I needed with 4 test users.

    Copied the 4 test users from the extnames.nsf to the names.nsf.

    Went to register the users, verified information etc..

    And got this error: "The user's flat name matches another user with a different hierarchical name."

    You should know that I ran into a bug in the Notes Admin and Domino Server v12.0.2, which you can read about here. Upgrading the client/server is the basic answer to resolve this one. Since Domino was already at 14.0FP1, my Notes Admin client had yet to be updated. 

    Once updated, everything went as it should.

    Well, only some things. 

    I need to do more testing, but I think the "just updating an existing user" registration option is not working properly because I now have 2 completely different entries for each test user. 

    My theory is the existing users being "unregistered web users" with "other mail" were not seen as the same people, and so Domino created the new entries. I know the ways to work around this if it is the case, but more testing will validate what I need to do. After all i will have 7,000 to update, fixing one manually is fine, but all of them? For that, I have my Openntf Admin Snippets to help me. I will blog about this after testing is completed.

    In any event, I copied the 4 people out of names.nsf, put them in the extnames.nsf, and reindexed both directories as it was testing time.

    Testing TOTP and DA

    Ready for testing, I turned on TOTP in the Security setting, edited the domcfg.nsf for the test URL, and checked that the extnames.nsf was enabled in the DA, then restarted Domino.
    When Dominno comes up, everything looks okay. I open a browser, put in the URL, and see the login screen with the MFA details. So far, so good.
    The first test is my own log-in. I am in the names.nsf, and my ID is in the ID Vault. I passed, and there are no issues.
    Log in as one of the new test users for the second test. Invalid password.
    It seems, and this may be a customer agent, that the users are not supposed to have a web password. I added there password back, it was in my CSV, and reindexed and tried again.
    Different error. User not found type error.
    I was logging in as FirstName LastName, which is how I logged in, but there is only one record of me, but 2 of the test users. I logged in with the Org domain name and got one step closer, this time it just crashed on me.
    Ok, I cleared the browser cache, restarted Chrome, and tried again.
    This time I received a invalid access error.
    This is important because the HCL documentation does not say anywhere that the DA will work with TOTP on its own. It only discusses the DA via Cross-Domain Authentication, as you can see here.
    I looked at how the DA was set up and changed the Group Authorization setting to Yes from No.

    Made sure Trusted for Credentials in the Naming Contexts tab was set to YES.
    Then I tried it again.
    And it worked this time. I was prompted to set up the MFA and log in as the extnames test user.

    Conclusion

    I still have more procedures to test and document, but the ability to leverage TOTP in a secondary directory via DA is not a limitation for the rollout. 


    Wednesday, July 31, 2024

    Stress testing HCL Domino and your "Other" Mail Infrastructure

     How does 1.2M emails sound?

    There are people out there who say HCL Domino can't handle the stress of modern times.

    It is an old system(truth be told, Exchange isn't much younger) with limitations.

    Well, yesterday, I did an unintended stress test of Domino and a client's internal infrastructure.

    Like most large and well-known companies, they run many Domino applications that handle millions of dollars a day but also have an O365 infrastructure.

    Mail doesn't come into Domino, but it does go out from there, and this is where it got interesting.

    Domino sent over 275,000 emails over about 15 minutes +/-.

    This was all internal SMTP, so we didn't get spam blocked or anything,

    Normally, a company would have a choke hold option on mail sent per minute, but for some reason, that was not in place or was avoided due to routing configurations yet to be determined.

    And my 275K grew to 1.2M inside the network architecture.

    Fun day at the office, right?

    No, I caused the problem and also knew how to fix it, but that small window of time was enough to unleash the tsunami.

    Sometimes, the simplest things become the hardest things, and this was such a case. 

    Even veterans of the email wars screw up.

    I now have a great session for F*uck Up Nights if anyone is interested.

    Definitely used up one of my IT 9 lives, this brings me down to about 5 left.

    HCL Domino was amazing throughout. Not only did the servers not complain about anything, but they just kept going and going and going. Multithreading FTW!

    O365/Exchange did not do so well, got clogged up and had issues because it still is not a multithreaded service.

    I pulled the numbers from the Mail statistics and showed them on my Admin client Monitoring dashboard. The left side is tasks, which I covered in my previous blog post. On the right side, you can add statistics, something not many people do, but I like to see some things there and probably should do a separate session on the statistics, but that is for another time.

    My peers, I am sure, can guess what caused the tsunami, so there is no reason to elaborate. But let's just say when you are a junior admin, this is one of the outcomes of your trial-by-fire Domino Administration education.

    For those who think Ambassadors and long-time Yellowbleeders are these great Gods of tech, some really are titans, I admit my mistake and that, indeed, you are never too young or old to learn something new....or mess up royally.


    Tuesday, July 30, 2024

    Domino RESTAPI Bug and WorkAround

    This is not my usual line of thought as an Admin, but sometimes, AdminOps is better than DevOps because troubleshooting is not an exact science.

    While customers over the last year or so have been asking about the HCL Domino REST API, my reply is usually something like, I can install it, but you are on your own afterward, or I point them to a Developer friend.

    To be fair, HCL will help them/me with getting started or "where is/How do I" questions. But this is about the bug my client and I discovered and how to work around it.

    While updating the v1202FP3 servers to v14FP1, all went okay, even with the change in Java classes, until we got to the REST API server.

    If you had downloaded and added my tasks update for the Admin client, you would see that RESTAPI was running on the machine, as I saw. I let the customer know I would upgrade it from 1.04 to 1.014.

    Usually, it's not a big deal; you run a long string of commands or broken down into 4 lines like the example shown in the documentation which I prefer as my typing is not perfect.

    java -jar restapiInstall-14r.jar ^ -d="C:\Program Files\HCL\Domino\Data" ^ -i="C:\Program Files\HCL\Domino\notes.ini" ^ -p="C:\Program Files\HCL\Domino" ^ -r="C:\Program Files\HCL\Domino\restapi" ^ -u

     You then type an A to accept the update, which will upgrade the REST API code.

    It replaces the existing files with new ones and any updated files.

    Great. I rebooted the server, and it looks like everything is up. The task view in the Admin client shows REST API is up.

    Test a few things, but Swaager is not connecting to anything.

    My initial thought was that the developers had some code that had been deprecated or maybe not valid with something in v14. I wasn't far off. I tried a few things to get it to work, and then I decided to look at the schemas. I found them deactivated, which seemed wrong to me, so I enabled one to test.

    And it worked again.

    Great! I did the same trick with the others, and they also worked.

    One last test, shut down Domino and reboot the box clean to ensure it was ok.

    No luck. Back to square one, but now the schemas all showed activated, but still just errors like this one:

    Domino RESTAPI Fetch Error
    This not being my first effort, I figured let's deactivate everything and reactivate it and test it.
    Sure enough, it worked.

    So, in my mind, the problem was somewhere between the REST API code and some type of flag on the schemas/databases that just wasn't being accepted.

    Opened a ticket with HCL to discuss the bug I found.

    After the usual back and forth, get us this, debug that, a copy of the NAB, a db, schema db, etc.. HCl said they could not recreate my problem, yet here it was.

    We had an online meeting so the dev team could get a good luck at my testing and poke around deeper.

    They agreed something was fishy and went off to look into it further.

    In the interim, given this was a key production server, we wanted to revert back to the old REST API version, which worked fine prior to the upgrade. But would it work on v14, among other questions we had?

    HCL and I discussed it with the customer, and this is how you revert to an older version.

    We had the old code downloaded, which is key as sometimes HCL has a way to make older versions "disappear" from the public. The steps included:
    1. Shut down Domino
    2. Copy the Domino\restapi folder contents out from Domino to a backup space.
    3. Once the RESTAPI folder is empty, you can then walk through the installation steps as I showed them above, but the last line instead of -u should be -a.
    4. This will reinstall all you need(presuming all URLs, folders, etc are NOT being edited/changed)
    5. Start Domino
    6. Test
    This worked well, and the customer was back up and running on V14FP1, albeit with the older REST API code. 

    After a few days, we got back some information from HCL, which I quote directly below:

    Problem:
    REST API 1.014 Running the APIs in local swagger returns Error "Failed to Load API Definition"

    Possible cause
    The issue is that the scope's Server name is not correct.
    This setting was not working correctly in older versions of DRAPI and was fixed in v1.0.6. It was pretty much ignored in v1.0.4.

    Possible solution/Workaround:
    With DRAPI 1.0.14 you have to add the CN name/hierarchical name of the Domino server instead of host name for example: CN=customerTest/O=HCLLabs in the Edit scope>>Server field>>CN=customerTest/O=HCLLabs or CustomerTest/HCLLabs
    Earlier server name field in scope>>Server field was: Host name of the server i.e keithbrooks.com

    Change it to either CN name or the hierarchical name of the Domino server or can be left blank stating any server that has this scope in the KeepConfig.nsf assumes the database / schema exists.

    Moving further the product dev team is going to update the error message in the product code as 'You need to query a different server' so it makes more sense to the testing on affected DRAPI versions.

    There you have it. My guess initially was partially correct. 

    The configuration was correct for the old version, but because the customer did not upgrade the RESTAPI code along the way, they missed the changes in code at 1.06, which would have probably prompted an HCL ticket at that time as well, but it would have been easier for all of us to see as the issue at that time. as the change would have been fresh in their mind.

    The story's moral is, of course, ABU or "Always Be Upgrading" because of security, code, and functionality adjustments over time. While we don't want to break production, sometimes you must do so for your benefit.

    The second moral is to have a Development or Staging environment for these critical applications. In this case, the lower environment did not match the upper one entirely, so the problem was not seen when we updated the lower environment.



    Thursday, July 4, 2024

    An Admin Present You Didn't Know You Needed

     Hi, welcome back to my burnt out blog. 1,500 posts and, well, I am kind of burnt out, but that doesn't stop me from giving to the community these little bits.

    I'd like to write more but a lot of things have been internal client items that I can't write about, but I am active.

    Preamble, excuses out of the way, so, who wants some goodies?

    About 2 weeks ago, I gave an impromptu webinar for Openntf.org as a last-minute fill-in.

    Openntf, for those that don't know, is the Notes/Domino+ community, where devs, admins, business people, HCL, and others share code and ideas, templates, and projects for the benefit of the greater worldwide community.

    I wanted to inform people that monitoring Tasks in the Administrator client has some changes.

    Why is this important? Because unless you are a 1 server company, you have a lot of information to remember, such as:

    1. How do you know if DBMT ran? 
    2. How do you know which server Certmgr runs on?
    3. Which web server do you run the Domino REST API on?
    4. Which server handles your Backups and Restores, presuming you leverage the v14 options?
    5. Is NOMAD running?
    6. Is your DirSync working?
    7. Are you sure the awesome OnTime Group calendar is running?
    8. Have you enabled Aautoupdate yet? One look and you know.
    Intriguing questions, right?

    Between v9 and v12, nothing changed in the tasks that could be monitored. Traveler seems to have been the last item added, and that was from 8.5, but it found its way into the Monitoring Dashboard in v9.

    Now comes v14 and HCl has cleared out some older items, like x500 info and the Fax server....but did not add any of the newer tasks that have come along since v9.

    To be fair to HCL, it is not as simple as a few fields and renaming a file.

    But fear not my fellow Admins, for I have not only explained it all in my presentation, which you can watch over here on the Openntf YouTube page, I have made the tasks available for everyone to update their Admin Monitoring Dashboard.

    If you just want the slides, go here.

    And because I know you are probably as lazy as I am, I have made the forms available with instructions to help you get more from your Monitoring Dashboard.

    Go get the tasks from my Openntf project over here.

    If I missed a task that is not listed, let me know, and I will update the project database.


    Tuesday, December 19, 2023

    SSL Cert Renewed in Certmgr but no one could see it

    Certmgr is the greatest thing in Domino these days if you are an admin.

    Autorenewing SSL saves so many problems, delays, and potential loss of revenue for customers that it is, in my opinion, one of the best things HCL has added to Domino.

    Much of the credit for it goes to HCL Lifetime Ambassador Daniel Nashed. 

    When you see him at Engage or DNUG, buy him a beer.

    Daniel was on hand to help me with my problem tonight, and he was correct with his original assessment, Certmgr should just work. 

    I agreed, and it was working, or so it showed in the view when verifying it using "tell certmgr show certs" at the server console, but we could not see the validated certificates for 2 domains.

    Since I had manually renewed them today, we should have seen a date of expiration for March 18. Instead, we saw December 17th for the one that expired yesterday, and the other showed January 21.

    The TLS cache should be auto-refreshing when it gets the new certificates, but appeared to not be doing the task.

    We reviewed the basic configuration and tried some test requests, which should have triggered a cache refresh and resolved the issue. But that didn't help us see the correct certificates in our browsers.

    While Daniel asked me about different parameters, I learned something about the updated certmgr, we don't need to put the .kyr name in the Security tab, TLS options field. 

    Instead, we should be using the DNS name. I totally missed this. The .kyr name in the field is there for the legacy people who have yet to move to V12 or V14. See page 36 of the slide deck mentioned below.

    You can read Daniel's slides from his OpenNTF session, which is full of deep technical information. https://blog.nashcom.de/presentations/openntf2021_domino_certmgr.pdf

    The other part, which I did know, but had yet to remove from the customer server is the Internet Sites Basics tab, DSAPI Filters field no longer requires ncertmgrdsapi.


    After doing these bits of cleanup, and restarting HTTP a few times, we were still left with the issue of incorrectly reported SSL certificate dates.

    We turned on debugging for the cache using set config CERTSTORE_CACHELOG=1. 
    Page 47 in the above slide deck.

    And we got nothing.

    Which surprised both of us.

    And then we went to look at the notes.ini to see if anything was pointing to the wrong place.

    And this is where we found the problem.

    Now, there is a parameter that should not have been there at all, and there was only one Google reference for it that we found. Evidently, that reference should not have been public, but it was, and someone at the customer site had added it sometime in the last 60 days or so because Certmgr had been running fine for over a year already.

    For the sake of some poor admin out there troubleshooting this, I will say that if you experience the same problem as I did, look in your Domino notes.ini for a line that starts with "SSL_DISABLE_TLS".

    I will not put the rest of the command here because, as Daniel said, no one should be using it.

    If you find something like this, just remove the line outright from your notes.ini.
    You can use "set config ssl_disable_tls(rest of the name)=" to remove it from your active server.
    There is no 0 or 1 to put to remove it.

    Then, at your server console, type "restart task HTTP," which is the better way to restart HTTP.

    And poof, like magic, it all worked again.

    That command blocks the newer TLS Cache refresh implementation from running. Thus even though Certmgr could get the updated certificates, it could not run the refresh because this line was telling it not to run.

    Customers are so cute when they tell you they didn't change anything.