Tuesday, October 6, 2026

Setting Up Directory Assistance for LDAP in HCL Domino

Setting Up Directory Assistance for LDAP in HCL Domino

Integrating remote LDAP directories into your HCL Domino environment can streamline authentication and directory lookups. Here's a step-by-step guide to setting up directory assistance for an LDAP directory.

Prerequisites

Before diving in, ensure you have:

  • Network Connectivity: Verify that your Domino servers can reach the remote LDAP server. A simple ping test can confirm this.
  • Directory Assistance Database: Create a directory assistance database (da.nsf) from the da.ntf template and replicate it across servers that will utilize it. Each server must have a local replica to use directory assistance. (ibm.com)

Creating a Directory Assistance Document

  1. Access the Domino Administrator:

- Open Domino Administrator. - Navigate to the server that will use the directory assistance database.

  1. Open the Directory Assistance Database:

- Go to the Configuration tab. - Expand Directory > Directory Assistance. - If prompted with Server Error: File does not exist, ensure the server is set up to use the directory assistance database.

  1. Add a New Directory Assistance Document:

- Click Add Directory Assistance.

  1. Configure the Basics Tab:

- Domain Type: Select LDAP. - Domain Name: Assign a unique name different from other Directory Assistance documents. - Company Name: Enter the associated company's name. - Search Order: Define the order in which servers search this directory relative to others. - Make this domain available to: - Notes clients and Internet Authentication/Authorization: For mail addressing and client authentication. - LDAP Clients: To enable LDAP service referrals. - Group Authorization: Choose Yes if you want to search group memberships in this LDAP directory for database access control. - Enabled: Set to Yes.

  1. Configure the LDAP Tab:

- Host Name: Enter the LDAP server's hostname or IP address. - Port: Default is 389 for standard LDAP or 636 for LDAPS. - Optional Authentication Credential for Search: Provide credentials if the LDAP server requires authentication for searches. - Base DN for Search: Specify the base distinguished name (e.g., dc=example,dc=com). - Channel Encryption: Choose Yes if using SSL/TLS. - LDAP Vendor: Select the appropriate vendor (e.g., Active Directory).

  1. Save and Close the document.

Testing the Configuration

After setting up, it's crucial to test the configuration:

  • Verify Connectivity: Use the Verify button in the LDAP configuration section to test the connection. If you encounter errors like LDAP Server is NOT available, ensure the LDAP server is accessible and that network configurations are correct. (developer.ds.hcl-software.com)
  • Monitor the Domino Console: Restart the Domino server or the LDAP task and monitor the console for any error messages related to directory assistance.

Troubleshooting Tips

  • SSL/TLS Issues: If connecting over LDAPS (port 636), ensure that the Domino server trusts the LDAP server's SSL certificate. This might involve importing the certificate into Domino's key ring.
  • Authentication Failures: Double-check the credentials provided in the Optional Authentication Credential for Search field. Ensure they have the necessary permissions to search the LDAP directory.
  • Group Authorization: If group lookups aren't functioning, verify that the Group Authorization setting is enabled and that the LDAP directory contains the expected group structures.

The Outcome

By integrating a remote LDAP directory through directory assistance, your Domino environment can authenticate users and perform directory lookups seamlessly across multiple directories. This setup enhances flexibility and centralizes user management, especially in environments with diverse directory services.

Tuesday, September 22, 2026

Setting Up Domino's Native Backup: A Hands-On Guide

Why Native Backup Matters

If you've been juggling third-party backup solutions for your Domino servers, you know the drill: compatibility issues, complex configurations, and sometimes, less-than-ideal integration. With Domino 12, HCL introduced a native backup and restore feature that simplifies the process, offering a seamless way to protect your data without external tools. Let's dive into setting it up.

Prerequisites

Before we get our hands dirty, ensure:

  • Domino Version: You're running Domino 12 or later. This feature isn't available in earlier versions.
  • Operating System: Your server is on Windows or Linux. The native backup doesn't support other Operating Systems.

Step 1: Initialize the Backup Database

First things first, we need to create the dominobackup.nsf database, which will house our backup configurations and logs.

  • Run the Backup Task: Open your Domino console and execute:

`` load backup ``

This command initializes the backup database. If it's the first run, Domino will create dominobackup.nsf in the data directory.

  • Verify Creation: Navigate to your Domino data directory and confirm the presence of dominobackup.nsf.

Step 2: Configure Backup Settings

With the backup database in place, let's set up the backup configurations.

  1. Open dominobackup.nsf: Using the Domino Administrator client, access the backup database.
  2. Create a New Configuration:

- Platform Selection: Choose between 'Default' (applies to multiple servers) or 'Server' (specific to one server). For most setups, 'Default' suffices.

- Backup Paths: Specify directories for storing backup files. Ensure these paths have adequate storage and permissions.

- Excluded Databases: If there are databases you don't want to back up, list them here.

- Schedule: Define when backups should run. Regular scheduling ensures consistent data protection.

For a detailed breakdown of each configuration field, refer to the official documentation.

  1. Save and Close: Once configured, save your settings.

Step 3: Schedule the Backup Task

To automate backups:

  1. Create a Program Document:

- Program Name: backup

- Command Line: Leave this blank.

- Server to Run On: Specify your server.

- Schedule: Set the desired frequency and time for backups.

  1. Save the Document: This schedules the backup task as per your configuration.

Step 4: Test the Backup

Before relying on the setup:

  • Manually Run the Backup: In the Domino console, execute:

`` load backup ``

  • Monitor the Process: Check the console for any errors. Also, verify that backup files are created in the specified directories.

Step 5: Restore a Database

Testing the restore process ensures your backups are reliable.

  1. Open dominobackup.nsf: Navigate to the 'Restore' view.
  2. Select a Database: Choose a database to restore.
  3. Restore Options: Decide whether to restore to the original location or an alternate one. You can also choose to restore deleted documents.
  4. Initiate Restore: Click 'Restore' and monitor the process.

For more insights on the restore process, check out the Domino Backup Introduction.

To Review

Setting up Domino's native backup isn't just about ticking boxes; it's about ensuring your data's safety without unnecessary complexity. By following these steps, you integrate a robust backup solution directly into your Domino environment. Remember, regular testing of both backup and restore processes is crucial. After all, a backup is only as good as its restore.

For a visual walkthrough, consider watching this HCL Domino Backup/Restore Function video.

Thursday, September 17, 2026

Optimizing Domino Cluster Failover for Seamless High Availability

Ensuring high availability in an HCL Domino environment requires meticulous configuration of cluster failover mechanisms. Proper tuning not only guarantees seamless user access during server outages but also optimizes overall server performance. 

This guide delves into key settings and parameters essential for effective cluster failover management.

Understanding Domino Cluster Failover

In a Domino cluster, failover is the process by which the system redirects user requests from an unavailable server to a replica on another server within the cluster. This mechanism ensures continuous access to databases, even during server downtimes. The Cluster Manager on each server monitors the availability of other cluster members and manages the redirection process accordingly.
Learn more about how failover works here.

Configuring Silent Failover on Notes Clients

By default, when a Notes client encounters a server failure, it prompts the user with a dialog box before switching to a replica on another server. To enhance user experience by making this process transparent, you can enable silent failover:

    1. Set the FailoverSilent Parameter:

- Add the following line to the client's notes.ini file: `` FailoverSilent=1 `` - This setting suppresses failover prompts, allowing the client to switch servers without user intervention. Detailed information on this parameter.

    2. Deploy via Desktop Policy:

- Navigate to Policies - Settings - Desktop Settings in the Domino Directory. - Under the Mail tab, select Client Settings. - Enable the option "Enable silent failover when a server goes down". - This policy distributes the FailoverSilent=1 setting to all clients, ensuring consistency across the organization.

Enabling Mail Router Cluster Failover

To ensure that the Domino Mail Router can reroute emails to available servers during a mail server outage, configure the following:

    1. Set the MailClusterFailover Parameter:

- Add the following line to the server's notes.ini file: `` MailClusterFailover=1 `` - This setting allows the Router task to forward incoming messages to a cluster replica of the recipient's mail file on another available server. More details on this parameter.

    2. Configure via Server Configuration Document:

- Open the Domino Administrator. - Navigate to Configuration > Servers > Configurations. - Edit the relevant Server Configuration document. - Under the Router/SMTP tab, go to Advanced > Controls. - Enable the Cluster failover option.

Managing Server Availability and Load Balancing

To prevent servers from becoming overloaded and to manage failover effectively, adjust the following settings:

    1. Set the Server_Availability_Threshold Parameter:

- Add the following line to the server's notes.ini file: `` Server_Availability_Threshold=75 `` - This value (ranging from 0 to 100) defines the minimum acceptable level of available system resources. When a server's availability index falls below this threshold, it declines new database open requests, prompting clients to failover to less loaded servers. Further information on this parameter.

    2. Configure via Server Configuration Document:

- Open the Domino Administrator. - Navigate to Configuration > Servers > Configurations. - Edit the relevant Server Configuration document. - Under the Notes.ini Settings tab, add or modify the Server_Availability_Threshold setting.

Restricting Server Access During Maintenance

During maintenance windows, it's crucial to prevent new user connections while allowing existing sessions to conclude gracefully:

    1. Set the Server_Restricted Parameter:

- To restrict new database open requests: `` set config Server_Restricted=1 ` - This command can be issued directly on the server console. The value 1 restricts access until the next server restart, while 2` makes the restriction persistent across restarts. Detailed explanation of this parameter.

    2. Monitor and Manage Active Sessions:

- Use the Domino Administrator to monitor active user sessions. - Notify users of impending maintenance to minimize disruption.

Tuning the Cluster Repair Service

For environments utilizing symmetrical clusters, the Cluster Repair service ensures database replicas remain consistent across servers. To optimize its performance:

  1. Access the Cluster Configuration Document:

- Open the Domino Administrator. - Navigate to Configuration > Clusters > Configuration. - Edit the Cluster Configuration document.

  1. Adjust Repair Service Settings:

- Under the Tuning tab, configure the following: 

  • Number of repair threads: Set between 1 and 20 (default is 4). 
  • Check donor availability: Frequency (in minutes) to check donor server availability (default is 5 minutes).
  • Retry failed repairs after: Interval (in minutes) before retrying a failed repair (default is 5 minutes).
  • Maximum number of retries: Number of retry attempts before marking a repair as unrepairable (default is 3).
  • Repair performance: Level of CPU and disk I/O dedicated to the repair service, from 1 (slowest) to 5 (fastest) (default is 3).
  • Repair logging level: Level of logging detail, from 0 (none) to 4 (diagnostic) (default is 2).

- These settings help balance repair efficiency with system resource utilization. More on tuning the repair service.

Conclusion

By carefully configuring these parameters and settings, you can enhance the resilience and performance of your HCL Domino cluster. Regular monitoring and adjustments ensure failover processes remain seamless, providing users with uninterrupted access to critical services.

Wednesday, September 16, 2026

Booting up the old Blog

Once in a while, I get inspired, or in this case, someone beat me to doing something, and so I have forked their project for my own, and soon to be, your benefit.

When I started down the road with the AI Admin app, it was, like most people find, a small step forward, but one that helped me get a clearer picture of what works, doesn't work, and what makes you want to smash your laptop in pieces.

But that left me wondering, what else I could do to help my fellow admins and wayward developers who must do admin stuff?

5 days ago, although I only saw the post 3 days ago, Heiko posted about this site: https://bryanhsiao.github.io/domino-news/en/.

This is great for developers and people who speak Taiwanese.

I have been playing with a blog posting app, I have 2 others that are active and still tweaking, unrelated to my Domino work, for Admins, and realized Bryan had put this up on GitHub for people to adapt.

Well, I adapted it.

And it took 2 days of testing to get it to a point where it is ready and puts the posts into my blog feed.

I will be editing them slightly, but it gets 95% of what I want done on its own.

You can go look at the site and the GitHub links he has.

Right now, it is random topics while I work on prompting it for specific topics.

Do we need more tech info? Maybe, maybe not.

But sometimes you do want information in one place rather than hopping all over the HCL docs.

In any event, I hope to get 2-3 a week out if they meet the guidelines, so far only 1 out of 8 has met them.

That post will go out overnight, well, for my time zone.


Thursday, June 25, 2026

ClamAV and Domino

 I am not an expert on Anti-virus things and Domino, that would be Daniel Nashed.

But if HCL Domino has something new, I test it out and set it up on my server.

In this case, I had heard that ClamAV would now work with Domino as of 14.5.1.

Having used Clam for many years at the OS level, I wanted to play with this in Domino.

My server is Windows, not Linux, ClamAV Linux instructions are pretty common, Windows, not as much.

Boy was this the hardest "easy' thing to do ever.


To start, HCL provides no documentation.

There are only 3 references to ClamAV in the official docs:

https://help.hcl-software.com/domino/14.5.1/admin/wn_config_features_1451.html#wn_config_features_1451__section_fq1_wgc_23c

https://help.hcl-software.com/domino/14.5.1/admin/conf_configuringscanninginscscancfg.html?hl=clamav

https://help.hcl-software.com/domino/14.5.1/admin/conf_scanningattachmentsforviruses.html?hl=clamav


None of which tell you much about how to set up ClamAV or how to configure it for Domino integration.

I opened a ticket and eventually was directed to this post:

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0129703

Which is in Japanese. Use Google Translate or Chrome built in translstion.

I see now, at my urging, HCL has released the same document in English.

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0131220


Their perspective is that we should go to ClamAv to set it up and then magically know how to integrate it.

I asked why it wasn't in the actual documentation and got a runaround answer, given HCL announced the availability, not ClamAV.

The new technical document covers some of the set up required.


I am going to add what is missing, more for my benefit, and anyone else who will try this on their servers.

Step 1 is Download the ClamAV from the link in the document, https://www.clamav.net/downloads and install it. BUT this does not tell you how to set it up in Services.

For that, you perform this:

  • Open the Windows Command Prompt as an Administrator and navigate to your ClamAV folder.
  • Run the following command to install the ClamAV daemon service:
    clamd.exe --install

  • Remember to set it to automatic.

    Step 2 is to copy the config file as described and perform the edits.

    Step 3, however, did not work for me no matter how I tried to get the certificate. Even with HCL Support help, they eventually had to send me the certificate required.

    Step 4: You have to get the updated virus definitions. NOTE: This is a once off effort. And Manual.

    If you want it to be automatic, and you do, you need to do these steps.

  • Open the Windows Command Prompt as an Administrator and navigate to your ClamAV folder.
  • Run the following command to install the ClamAV daemon service:
    freshclam.exe --install

  • Once you Load mailscan at the Domino server console you will get the cscancfg.nsf and need to follow the document entries as provided.

    NOTE: The ClamAV server name(DNS) must be 127.0.0.1. Do not put your server IP there, it will not work at all.

    Make sure you open port 3310 with your local server firewall, and in my case, the outside ISP.

    I changed the Subject Prefix Scanned field default text to something I would know came from my server, not a spammer.

    Lastly, make sure you add Mailscan to your tasks line in your notes.ini also something not mentioned in the document that is kind of important, unless you like having your mail stuck in mail.box and spend 2 hours troubleshooting it one morning, like I just did.

    I eventually figured out it was an AV issue when running SMTPDebugClient=1 and

    Tell Router list run from the server console.

    You see this:

    Mbx NoteID          ID          State          Size Pri Count ScheduledDate           From

      1 000008FE 0028261D Wait AV        8443         1                         K Brooks/org

      2 00000902 00283C39 Wait AV       13306         1                         kbrooks@test.com

      1 00000902 002864E3 Wait AV        8749         1                         keith@gmail.com

      2 00000906 00287C9A Wait AV        8640         1                         keith@gmail.com


    Oh, that makes sense after the server restarted late yesterday. Mailscan was not in tasks.

    So, my time spent, is your time gained.


    Tuesday, May 12, 2026

    12.0.2 FP8 didn't update version numbers

    This is probably not a problem with FP8, to be upfront, but just in case I am using it for the title.

    4 out of 8 servers at a customer did not chaaange tehir version number when I updated them to FP8.

    They stayed FP7.

    In the past, I have had this issue where updates don't happen, due to various reasons like a corrupted NAB, failed hot fix removals, etc..

    But this was different because 4 servers, DEV domain, were fine.

    QUAL domain is not updated.

    How Did I Get Here

    Servers were 12.0.2 FP7 HF1 Windoews 2019

    I shut the Domino server down, including the Domino service.

    Paused the Windows Management service(ONLY affects HF's for some reason), then ran the hot fix as administrator to remove it.

    Started the server from the icon asd a regular application, so I can see it running, and a "sh ser" showed the correct FP7 version. Let it run for a few minutes till I saw HTTP start and some other tasks. Then shut it down.

    Run FP8 as an administrator, runs fine, no errors.

    Restart the box, actually, 2 servers had issues with the Domino service login, and I restarted the whole VM, the other 2 the service started fine.

    Waited to see the server docs change, and nothing.

    The upgrade.log showed everything looked ok.

    I opened a ticket with HCL after the 1st 4 failed to update, then I checked the next 4 and those worked.

    The Resolution and Reasoning

    Sooraj from HCL Support provided me with the following resolution and an explanation after I provided some files and screenshots.

    Take a backup of the files below and replace them from a "working" server [12.0.2FP8]

    Domino\nstrings.dll
    Domino\res\C\nstrings.res

    Shutdown the incorrect Domino server
    Take a backup of the two files mentioned above.
    Copy similar files from a working 1202FP8 server and replace them.

    (I left the existing ones, and added a _BAD at the end of the name)

    Start the Domino server.

    While I asked to close the ticket, I also wanted an explanation for future thoughts. 

    Sooraj provided the following:

    This problem was reported for the first time, and based on my analysis, it appears that those two files were not updated by the FP8 installer. It was still using the FP7 code. This ideally occurs due to an access issue with the account that started the installation, but unfortunately, I cannot confirm this because the upgrade.log doesn't show any such errors.

    Possibly a server processing issue, a delay, or perhaps not fully completing a virus scan, or some other local issue was to blame.

    But on 4 servers, I don't believe so. 

    One, sure, things happen. 

    In this situation, all 4, well, 8, are nearly identical setups.

    Hope this helps others with similar issues at some time in their admin journeys.



     

    Thursday, April 23, 2026

    Engage 2026, the Year of AI and my session

     There were 11 AI sessions out of about 70, not including roadmap or sponsor/commercial sessions.

    I figure in a year or 2 we won't be saying AI and just, look what I built.

    Thank you to everyone who messaged me asking where I was hiding. It wasn't my intention to miss the event, but I did not make it.

    Instead, I was live from my apartment, doing my session. What may have been a first for Engage, a solo session done remotely!

    Due to my flight travel plans being interrupted and changed a few times, it became clear I may not even get to Engage until after my session would have started, and have to stay a day later, but leave much earlier, and this caused quite a few issues, which I discussed with Kris and reluctantly had to give up trying to attend.

    Kris said he would hold my space in case things changed for the better, but alas, that was not the case.

    However, he said we could try to do it remotely. Webinar mode basically. 

    Thanks, Kris and Tom, for the chance and possibility!

    I agreed we could try it, not expecting much or even how we would do it, so I labored on with my slides.

    I asked the OpennNTF Board, which I am part of, if we could use our webinar meeting app and if someone could connect everything in Ghent.

    The Board agreed, and it was game on.

    Roberto was very helpful in getting it all set up, Thank you Roberto! I had presumed I would lose 5-10 minutes with setup and adjusted my session accordingly. It was only 7 minutes.

    My next thought was, why would anyone come to my session, knowing I wasn't there to give it?

    Turns out, about 15-20 people ventured into the room, and I think they were surprised by what they saw and heard.


    Full deck is here: https://speakerdeck.com/kbmsg/ai-making-admin-and-users-lives-better

    My session was supposed to be on AI tools/apps that anyone had created to help users and admins. To get people thinking more practically about what could be done.

    Along the way, I discussed why citizen developers who leverage AI, or anything, don't exist as they did 25 or even 10 years ago.

    With locked down browsers and IT blocking changes to applications, let alone updates, and the lack of a Domino Designer client installed (even though it is no longer an extra charge), it is just not easy for users, or even shadow it, to create anything, let alone use it. You can read more about this on page 8 of my slides.

    But we admins and developers are not as constrained, aside from browser lockdowns.

    I had hoped, had I been at Engage, to go to the Monday HCL workshop on vibe coding. There was also Bernd's session on Tuesday around the advances his company has made, which I looked forward to for his insights. Then there was Dominic's session on Wednesday, also aimed at Admins and AI, which, if I had read all the sessions in advance, I would have tried to compare notes earlier. To be fair, I often don't get around to reviewing the agenda until the week before the event.

    I work with some entities that I am not allowed to post about, which explains my dearth of blog posts. But that doesn't stop me from creating things I need to work with in their environments.

    My main point of the session was to show a script(LotusScript, server commands, Java, and formulas) building tool for Domino admins. I am not a developer, although I can edit existing code and make things do what I want. When it comes to doing it from scratch, I am useless.

    Enter AI, and I said, " I bet I could create something to make my life and other admins ' lives better".

    I give you my Domino Script AI Tool.


    You can go use it here: https://kbmsg.github.io/domino-scripts/

    Or pull it from my GitHub: https://github.com/kbmsg/domino-scripts

    The slide deck discusses some of the templates and what it does; the GitHub site provides more details and how to start using it.

    I built it to work with Claude.ai, and feedback already asked that I let Ollama be used too.

    I started the ollama config, but it is not quite working. If you see it say v66 or newer and followed by Ollama works, like it says just now, you are good to go!

    Domino Script AI v64



    This will be posted to https://OpenNTF.org as well.

    AI vibe coding is literally like building your parachute after you jump out of a plane. Sure, you get most of it right, but that last 10% is the problem, and it takes time.

    But, and this is a huge but, if I tried to do this from scratch, it would take me days and days. While I am on iteration 65 or so, all together it took only a few hours to get here, but stretched out over a few days, giving time, resources, and work.

    If you have ideas for me to include, feel free to comment here or via GitHub or the OpenNTF Discord channels. 

    PS - If you are in a licensing situation and need to sort the CCB vs CCX people, you may want to try the Dump ACL to CSV template. As discussed on page 16 in my slides.


    Sunday, June 29, 2025

    HCL Domino 14.5, Fix your iNotes Redirector for Verse

    By now, if you have updated your Domino server and relied on iNotes, you have found that it no longer works.

    HCL was clear about this, and there is no workaround, so if you want iNotes and not Verse, better keep a server on 14.0 and point everyone there.

    For the rest of us that move forward with the times, we will have seen that logging into the iNotes URL gets you to a pop-up which looks like this:

    HCL iNotes is no longer supported

    It states that iNotes is no longer available, and you need to open HCL Verse. If you click on the link, it directs you straight to Verse.

    The URL for verse is something like https://companyname.com/verse, and you change your bookmark, but that doesn't work exactly. When you hit that URL, you will get the framework of Verse but this message:


    So, you try the manual way, https;//companyname.com/mail/brooks.nsf, and that lets you in, via the 1st pop-up as above..

    The next morning, you open your browser, and the /verse URL is not working with the same error about recent contacts.

    What is going on here?

    iNotes required the iNotes redirector. Well, there is no new redirector, but it did get renamed to Verse redirector(same template name), but it does include some options you probably hadn't considered, and this is how you resolve the problem.

    If you haven't changed any settings in a while, now is a great time to update them. The redirector has been updated periodically along the way.

    These are the ones to change or review:

    Server Settings Button:

    Do you wish to force SSL for the entire session? Yes

    UI Setup Button:

    Your company logo should be there, so your users know it is a legitimate site.

    Enable Personal Options: Set to No, they were really about iNotes anyway.

    Enable HCL Verse: Yes, you need this now.

    Enable SAML options: If you use SAML, enable this option, which displays a secondary option for the Default Mail Application. Select Verse.

    When you have finished all your changes, click Save & Exit and then test to ensure everything works properly.

    And no more weird error messages about recent contacts.

    This tech doc can help those who want more details: https://help.hcl-software.com/verse_onprem/2.0.0/admin/vop_configuring_server.html

    Update June 30: I found that I now see this error the next morning.


    Step 19 is what you want.


    In my case, I already had a credstore created, and this new key would not get added to that credstore.

    I went back to the original technote, vop_configuring_server, and added a new entry in my domcfg for DWA, but that now redirects me to a different domain on my server, which has TOTP.

    Also verified that the 'inotes/verse' tab in the configuration document met the proper configuration as per the document.

    As it turns out, my secondary domain was set for SAML instead of single server in my Internet Site document, Domino Web Engine tab. Probably because I was testing something. Changing that back to single server now allows me to log in correctly with https://companyname.com/verse. With my erse redirectpr in place and everything now correct.



    Thursday, May 29, 2025

    What was my session about at Engage? Money and your Job

    It was a cathartic, contrarian approach to our current situation, in which we, the admins, developers, and business partners, are supporting our Domino worlds.

    I wish this were all in my head. But every single person in the room for my session at Engage was nodding their head at some point and others had very serious faces when confronted with my thoughts. The couple of people that talked with me afterwards were very happy they came to it. After all they gave up the Domino IQ roundtable for my session.

     

    How did we get here?

     

    The road that got us here is not the road that will get us out of here.

    But enough cliché’s let’s be realistic.

    We put ourselves in this situation.

    Some items to think about:

    (Almost) Every time we had a chance to ask for a budget, we failed.

    We got what we needed, but this is not what an Enterprise solution desires, deserves, or requires.

    We are selling our environments short if we only ask for licensing money, salaries, and some equipment. Way too short.

    If your annual budget request does not include enough money that a respected enterprise solution requires, your request will not go up to the budget committee. Each company sets these meetings at various levels. Enterprise ready products have a staggeringly large annual budget, because that is how you know what is important. Or so the executives believe.

    Maybe you think this is too much for “your team”.

    Here is a little secret: Security budget money usually does not come from your team. It comes from the security budget. Leverage that every year.

    Your environment needs to maintain perpetual security, which includes external and internal resources, people, apps, disaster recovery, development architectures, and more.

    Everything you do in your Domino environment is for the company's greater good, and extending easier, faster processes for everyone, in a secure and meaningful way, needs to be backed by a secure environment.

    I am not talking about 10k, 5k, or even 50k. You probably need to be asking for at least 100/250 or 500K to get to the budget executive committee. These days, maybe starting at a million. This is obviously relative to your situation and industry.

    Some of you reading now might think I am out of my mind, but I am just a contrarian who has learned from 30 years of mistakes.

    Maybe licensing is handled by a different group, so you didn’t need to ask for a budget. This also happens.

    Once you stop asking for a budget, you don’t get invited to ask for one in the future. You mysteriously get left off the group email, or worse, your boss stops asking you about it. It is very hard to get back on the budget list. VERY HARD. 

    For those who say, "No, we submit a budget," my question is, do they take it seriously, and do you get funding beyond the licensing basics?

    Developers, you fall under this spell as well.

    How many times have you said (as an employee), "Oh, I can put something together. Give me a few days or a week or two." Every time you did this but did not chargeback some business units' code or lay out the process and document why the app is needed and what it does for the company, you have also failed your Domino infrastructure.

    If you were an outside vendor, you would have a project plan and bill X amount of hours or a total cost.

    But you think that as an employee, you can’t do that, and you are wrong.

    Everything goes to a cost center somewhere.

    The group you are writing this for should be paying your team some of their budget.

    But I bet few of you have done this.

    While internal cost centers are a BS accounting move, they do work with your budget, so play ball by their rules, not the ones in your head.

    Fear of Unemployment

    I understand your predicament. You don’t want to lose your job, and I don’t want you to, either. However, no one has ever been fired for asking for budget money. A raise is a different matter.

    For some reason, employees started to think they would get punished if they asked for the budget. You don’t get everything, but have you budgeted for the 2-8 various updates (now multiply by how many servers you have) you may need to do all year for your Domino environment? That number gets large fast, especially if you assign a value to it.

    Why assign a value?

    Because you should be clawing back money from every cost center. Again, internal budget money matters. Think about your budget if you had been "paid back" by the other cost centers.

    When Lotusphere/Connect/Connected existed, you had to beg your company to go sometimes. How many of you went to the HCL Factory Tours? I hope you all knew they did these. If not, why not? You should all have been at them.

    Did you think you would get fired for asking to go?

    No, usually, the fear is to do anything that will get Domino seen inside your company. The CIO may demand someone's head for saying the D word, or your boss gets pushback for the old software. Whatever the reason, I understand, but keep in mind, you are not helping yourself in the long run because you aren't giving your company any reason to stick with Domino. 

    I also blame the HCL sales rep at this point because they should be talking to you regularly by phone, email, WhatsApp, SMS, X, TikTok, set up an Ai auto message, whatever, I don't care, but they need to be on your side and not just "we have a webinar" or licensing renewal times. 

    To be fair to the HCL rep, they do what makes them money first before anything else, but if they don't want to lose customers, they could be more proactive, so could most of us, that's all I will say here.

    Money is everything, and you don’t understand what money means.

    Money means one thing to you, but it is totally different to executives. 

    You think asking for 10k is too much when the executive deals with million-dollar deals.

    I am not saying deals don’t get lost because of price; that does happen. However, when properly laid out and defined, a solution for your company is not a 25k over-the-weekend app. If you want the executives to take it seriously, it is a 500k project or equivalent for your organization.

    They will take it even more seriously if you can show that the new app either helps make money or manages some part, if not all, of your company's manufacturing, selling, marketing, or finance. 

    Do this, and you can ask for even more money.

    I am not talking about fake ROI projections of “time saved,” but real money tied to real work processes.

    If you used your budget for licensing, what else would you ask for? I have listed what I expect to be in your budget in the spreadsheet I have uploaded for everyone. I presume you are the budget submitter; otherwise, you must bother your boss more.

    Spreadsheet for download.

    It includes guideline pricing, multiple items, education, conferences, T&E for these events, and other items that should be clawed back from cost centers.

    It is also laid out as though you built an environment properly, and you can understand why, as you start playing with it, how the numbers multiply easily the more servers you have, and some costs also multiply. I aimed to show that all your costs after the Year 1 hardware purchases should be budgeted for you and cover your expenses.

    Some of you may find this enlightening, while others may find it far from reality. I will wager that few of you even have something laid out like this, on which you are basing your budget. I hope I am wrong.

    UI/UX is NOT HCL's Fault

    All the BS talk about a poor UI/UX falls on your developers, now.

    Older apps did have limitations, but this is not 2001 and R5, why haven't you refreshed them? Even just using Restyle would help. NOMAD web can be your friend here to reduce the web overhead to convert an app from Notes only to web, but you probably do need a nicer UI.

    There are some gorgeous apps, Notes and web ones, so we know it is possible.

    Again, who is to blame for this? Your app's UI/UX is not HCL’s fault.

    Your executives see it as IBM/HCL's fault, but that only relates to the mail template. Your apps are your own to be creative.

    Speaking of executives.....

    New Executives have no idea what Domino does for your company

    Because NO ONE has told them.

    In this situation, I am talking about apps here. Mail is basically a cost of doing business. While on-premises and keeping your data inside are essential and more secure, email is a cost.

    Some of you have been trapped by this “cost” term. Just because email is a cost and you are told to reduce expenses(notice they never drop new applications or revise older ones to save money), you may have been mistaken in thinking you shouldn’t ask for more money. But this, again, is what got us into this mess in the first place.

    Servers do not update themselves; they can now under v14.5, but you get my point.

    But apps, apps run your world, make you money, reduce paperwork (hopefully), and should be so good that no one is looking to replace them. 

    Every time a new executive comes on board —and let’s face it, executives bounce between companies every 2-5 years—how have you briefed them on your Domino apps and what they do for the company, what they make for it, or what they provide that would cripple you if it suddenly stopped?

    Have you? Has your HCL rep? Have you even told your rep what your apps do? Your HCL rep can't help you be that internal champion if you don't help them with what exists.

    In my session, I used one of my clients as an example. They make millions of dollars a day using their Domino app worldwide inside their company. It handles everything from initial email requests for pricing and online sales to shipping, inventory, and manufacturing, tracking everything along the way. It is typical of Domino and an insanely elaborate application put together over two decades of teams of developers.

    And yet, EVERY MEETING that involves leadership, I have to go to bat for my client and their app. Their internal people are too afraid to be connected to the “old server” stuff. See the above section Fear of Unemployment".

    Imagine that meeting, discussing when we will move off Domino (I have been managing their environment for over 10 years or so) soon, but hearing this every year from me that it keeps them running.

    They back down and then ask what is replacing it, which I suggest nothing, since it is an active and constantly changing environment, why would they replace it. Keep in mind that they only know the app by its internal name, which is how I introduce the topic, because they have zero idea it is a Domino application. But it always comes up from the Microsoft guy.

    Of course, the business units say some cloud thing or an app made out of another database or workflow solution. Anything they move to would be a considerable cost, and ALL THEY WILL DO IS REWRITE WHAT THEY ALREADY HAVE, and from experience, end up with less functionality. I offer our help with whatever route they choose.

    Like I said, 10 years, same meetings, every year. Many partners have similar stories, because we believe in the product, management believes in their bonus and their next salary when they move on.

    This is exactly what one person in the meeting said about his situation. While he isn’t happy about the reduction in physical servers (personally I prefer less, but clustered, even when you have remote locations) he agreed that he will be there until he retires, nothing is moving that fast. 

    Management has made the decision in his case, but to be fair, their environment might be faster and perform better with less overhead from the dozens of servers being consolidated.

    Details matter, but for every excuse or peculiar circumstance, a completely normal one is ignored or under-budgeted until it becomes too late.

    I hope my session and this blog post help you never reach that point of no return.

    Conclusion

    TL:DR The problem is and has been us.

    I hit on other items that we all see or hear when advising clients. But almost everyone should know how to handle those by now.

    If not, reach out to me. The lack of integration FUD, the “old software” line, the “lack of knowledge out there” hiring managers, and the perpetual “everyone uses <insert some other company>” lazy man excuse from someone that doesn’t want to be accountable for anything.

    But we could turn this around by addressing the money issue and how we handle pricing/budgeting.

    If you need help with this discussion, contact me, my fellow HCL Ambassadors, your Business Partner (or dump them for me), your HCL sales Rep/ Customer Success Manager/Customer Success Executive.

    We are at a fulcrum. The cycle is coming back to in-house data from Cloud data, and now is the best time to Make Domino Great Again in your company and champion it as a proper Enterprise solution.

    Thursday, May 22, 2025

    My IQ has gone up thanks to Engage

    Engage just finished. The first one is under new management, and it is in safe hands.

    I don't think I have ever seen Theo so happy with a big grin on his face the whole time. Tom and Kris did really well for their first time out. And they told us next year Engage will be in Belgium.

    I had to miss last year's due to the timing and circumstances of the war at home, so it was great to talk to everyone. Thank you all for asking about the situation and the family. It means a lot.

    The star attraction was Domino IQ, which adds AI to the core product. As an Admin, all I can say about this is that you will need to spend time building this out properly if you want to move beyond the basic regurgitation of your knowledge base or other internal documents. Also, IQ needs some extra hardware and configuration, so be prepared and read the what's new and requirements docs. Your developers will be busy.

    It will be available starting with v14.5, which is coming out June 17! IQ is part of your license if you are under CCB Term(I believe). If you are a perpetual license org, this is your incentive to change licensing plans.

    I was happy to see friends from past Engage shows, and in some cases people I hadn't seen in many years. Yes, we all got older, but we still have fun together.

    There was a tease of Notes.Next but it is far from prime time ready so don't ask about it yet.

    Roadmap sessions were broad with partial timelines for new items.

    One thing that also stood out is the added accessibility to all the products which need to be up to certain laws soon and so that is taking over a lot of dev time.

    As an Admin, I want to let everyone know to go get Cormac's slides from his session. His session was filled with great ideas and details about cleaning up your administrative items, and as he said, even old admins can learn something new, which I did about SMTP failover routing.

    His blog is here, and hopefully, he will post his slides there or at the Engage website.

    Lastly, the Lotus brand might return for at least one product. Or not. While this was from Richard Jefts' opening keynote, later on, other HClers implied the name may not stay.
    I will leave the debate about the brand to others, as both sides of the argument have their validity.

    I was also in some partner/ambassador/HCL Roundtable sessions, which I can't say much about, but there are plans that will help the partners in the long run, which in turn will help you, the customers.

    I think my own session scared a lot of people, but I hope it also helped them rethink how they and their teams interact with upper management and budget committees. My slides will not be posted, and I don't publish my Competitive Intelligence info. I will probably make a second blog post about it over the weekend.

    Safe travels home everyone, talk to you soon or see you, at least virtually, some time soon.


    Tuesday, August 20, 2024

    SnTT - Does TOTP Work for users in a Secondary Directory via DA

    TOTP, DA, and Domino

    For the last 3 years, I have worked with TOTP inside HCL Domino and customers with unique requirements.

    This has provided fodder for my blog, and today, we have a new entry into the TOTP Mystical Ways of the World.

    Let me state my usual caveat upfront: TOTP is about the URL, not the server, the database, or the user.

    You enable TOTP for each URL you want on your server.

    PSA is completed. Let's discuss the circumstances that brought me here.

    Like many of our customers, a customer has a large external user community relying on their applications.

    The customer has licensed this with HCL, so I am not going to get involved in that discussion. However, be warned: It is not a comfortable one if you have been relying on some old licensing options and now fall under the new ones.

    We have about 7,000 external customers. Some are undoubtedly old customers, but 7,000 is a lot of people.

    Previously, I wrote about how to bulk add these people into your ID Vault, and that was all fine and good where we have only one names.nsf for everyone and everything. We may have had 2-3 servers in that org.

    Now, the 7,000 are in a secondary external names.nsf via DA (Directory Assistance).

    The Problem

    1) How do you register and maintain the people in a secondary Directory?

    2) Does the DA even work with TOTP? 

    The Options I See

    Officially, there is only one place, and one place only, where everyone gets registered: the names.nsf.

    This is not very helpful, especially given the reliance on the ID Vault for many things these days. By changing licenses, there is no way to "convert non-ID people to Notes ID people."

    What do you do?

    1) Copy, not replicate, the name.nsf, to extnames.nsf, move the actual names.nsf out of the way, rename extnames to names and then register everyone to it. Once done, put back the original names.nsf and off you go.

    As pointed out in our Openntf.org Discord channel, the problem with this is that the user and ID would not be properly found for encryption/certification. This is a very valid point I wasn't thinking about at first. Thank you, Ulrich and Detlev.

    2) register all 7,000 into the names.nsf properly, like normal. Then, manually copy the 7,000 to the extnames.nsf. Then delete, just regular delete, not Adminp delete, the 7,000 from the names.nsf.

    By doing this, we preserve the user's encryption/certification, and should a name need to be renamed/edited, we can copy the user back and fix it. One could also just create a new account etc.. and remove the bad one from extnames.nsf.

    3) Create a new Domain and register everyone to it and cross certify it with the existing domain. This may or may not be the answer as well, depends on circumstances.
    4) I have no other idea otherwise. However, there is an AHA idea asking HCL to think about a way to register people to some other .nsf. Take a look and vote over here. 

    If anyone has any ideas, let me know in the comments.

    I will get to the DA question shortly.

    Details and Planning

    I started with my blog post and the CSV file I needed with 4 test users.

    Copied the 4 test users from the extnames.nsf to the names.nsf.

    Went to register the users, verified information etc..

    And got this error: "The user's flat name matches another user with a different hierarchical name."

    You should know that I ran into a bug in the Notes Admin and Domino Server v12.0.2, which you can read about here. Upgrading the client/server is the basic answer to resolve this one. Since Domino was already at 14.0FP1, my Notes Admin client had yet to be updated. 

    Once updated, everything went as it should.

    Well, only some things. 

    I need to do more testing, but I think the "just updating an existing user" registration option is not working properly because I now have 2 completely different entries for each test user. 

    My theory is the existing users being "unregistered web users" with "other mail" were not seen as the same people, and so Domino created the new entries. I know the ways to work around this if it is the case, but more testing will validate what I need to do. After all i will have 7,000 to update, fixing one manually is fine, but all of them? For that, I have my Openntf Admin Snippets to help me. I will blog about this after testing is completed.

    In any event, I copied the 4 people out of names.nsf, put them in the extnames.nsf, and reindexed both directories as it was testing time.

    Testing TOTP and DA

    Ready for testing, I turned on TOTP in the Security setting, edited the domcfg.nsf for the test URL, and checked that the extnames.nsf was enabled in the DA, then restarted Domino.
    When Dominno comes up, everything looks okay. I open a browser, put in the URL, and see the login screen with the MFA details. So far, so good.
    The first test is my own log-in. I am in the names.nsf, and my ID is in the ID Vault. I passed, and there are no issues.
    Log in as one of the new test users for the second test. Invalid password.
    It seems, and this may be a customer agent, that the users are not supposed to have a web password. I added there password back, it was in my CSV, and reindexed and tried again.
    Different error. User not found type error.
    I was logging in as FirstName LastName, which is how I logged in, but there is only one record of me, but 2 of the test users. I logged in with the Org domain name and got one step closer, this time it just crashed on me.
    Ok, I cleared the browser cache, restarted Chrome, and tried again.
    This time I received a invalid access error.
    This is important because the HCL documentation does not say anywhere that the DA will work with TOTP on its own. It only discusses the DA via Cross-Domain Authentication, as you can see here.
    I looked at how the DA was set up and changed the Group Authorization setting to Yes from No.

    Made sure Trusted for Credentials in the Naming Contexts tab was set to YES.
    Then I tried it again.
    And it worked this time. I was prompted to set up the MFA and log in as the extnames test user.

    Conclusion

    I still have more procedures to test and document, but the ability to leverage TOTP in a secondary directory via DA is not a limitation for the rollout. 


    Wednesday, July 31, 2024

    Stress testing HCL Domino and your "Other" Mail Infrastructure

     How does 1.2M emails sound?

    There are people out there who say HCL Domino can't handle the stress of modern times.

    It is an old system(truth be told, Exchange isn't much younger) with limitations.

    Well, yesterday, I did an unintended stress test of Domino and a client's internal infrastructure.

    Like most large and well-known companies, they run many Domino applications that handle millions of dollars a day but also have an O365 infrastructure.

    Mail doesn't come into Domino, but it does go out from there, and this is where it got interesting.

    Domino sent over 275,000 emails over about 15 minutes +/-.

    This was all internal SMTP, so we didn't get spam blocked or anything,

    Normally, a company would have a choke hold option on mail sent per minute, but for some reason, that was not in place or was avoided due to routing configurations yet to be determined.

    And my 275K grew to 1.2M inside the network architecture.

    Fun day at the office, right?

    No, I caused the problem and also knew how to fix it, but that small window of time was enough to unleash the tsunami.

    Sometimes, the simplest things become the hardest things, and this was such a case. 

    Even veterans of the email wars screw up.

    I now have a great session for F*uck Up Nights if anyone is interested.

    Definitely used up one of my IT 9 lives, this brings me down to about 5 left.

    HCL Domino was amazing throughout. Not only did the servers not complain about anything, but they just kept going and going and going. Multithreading FTW!

    O365/Exchange did not do so well, got clogged up and had issues because it still is not a multithreaded service.

    I pulled the numbers from the Mail statistics and showed them on my Admin client Monitoring dashboard. The left side is tasks, which I covered in my previous blog post. On the right side, you can add statistics, something not many people do, but I like to see some things there and probably should do a separate session on the statistics, but that is for another time.

    My peers, I am sure, can guess what caused the tsunami, so there is no reason to elaborate. But let's just say when you are a junior admin, this is one of the outcomes of your trial-by-fire Domino Administration education.

    For those who think Ambassadors and long-time Yellowbleeders are these great Gods of tech, some really are titans, I admit my mistake and that, indeed, you are never too young or old to learn something new....or mess up royally.


    Tuesday, July 30, 2024

    Domino RESTAPI Bug and WorkAround

    This is not my usual line of thought as an Admin, but sometimes, AdminOps is better than DevOps because troubleshooting is not an exact science.

    While customers over the last year or so have been asking about the HCL Domino REST API, my reply is usually something like, I can install it, but you are on your own afterward, or I point them to a Developer friend.

    To be fair, HCL will help them/me with getting started or "where is/How do I" questions. But this is about the bug my client and I discovered and how to work around it.

    While updating the v1202FP3 servers to v14FP1, all went okay, even with the change in Java classes, until we got to the REST API server.

    If you had downloaded and added my tasks update for the Admin client, you would see that RESTAPI was running on the machine, as I saw. I let the customer know I would upgrade it from 1.04 to 1.014.

    Usually, it's not a big deal; you run a long string of commands or broken down into 4 lines like the example shown in the documentation which I prefer as my typing is not perfect.

    java -jar restapiInstall-14r.jar ^ -d="C:\Program Files\HCL\Domino\Data" ^ -i="C:\Program Files\HCL\Domino\notes.ini" ^ -p="C:\Program Files\HCL\Domino" ^ -r="C:\Program Files\HCL\Domino\restapi" ^ -u

     You then type an A to accept the update, which will upgrade the REST API code.

    It replaces the existing files with new ones and any updated files.

    Great. I rebooted the server, and it looks like everything is up. The task view in the Admin client shows REST API is up.

    Test a few things, but Swaager is not connecting to anything.

    My initial thought was that the developers had some code that had been deprecated or maybe not valid with something in v14. I wasn't far off. I tried a few things to get it to work, and then I decided to look at the schemas. I found them deactivated, which seemed wrong to me, so I enabled one to test.

    And it worked again.

    Great! I did the same trick with the others, and they also worked.

    One last test, shut down Domino and reboot the box clean to ensure it was ok.

    No luck. Back to square one, but now the schemas all showed activated, but still just errors like this one:

    Domino RESTAPI Fetch Error
    This not being my first effort, I figured let's deactivate everything and reactivate it and test it.
    Sure enough, it worked.

    So, in my mind, the problem was somewhere between the REST API code and some type of flag on the schemas/databases that just wasn't being accepted.

    Opened a ticket with HCL to discuss the bug I found.

    After the usual back and forth, get us this, debug that, a copy of the NAB, a db, schema db, etc.. HCl said they could not recreate my problem, yet here it was.

    We had an online meeting so the dev team could get a good luck at my testing and poke around deeper.

    They agreed something was fishy and went off to look into it further.

    In the interim, given this was a key production server, we wanted to revert back to the old REST API version, which worked fine prior to the upgrade. But would it work on v14, among other questions we had?

    HCL and I discussed it with the customer, and this is how you revert to an older version.

    We had the old code downloaded, which is key as sometimes HCL has a way to make older versions "disappear" from the public. The steps included:
    1. Shut down Domino
    2. Copy the Domino\restapi folder contents out from Domino to a backup space.
    3. Once the RESTAPI folder is empty, you can then walk through the installation steps as I showed them above, but the last line instead of -u should be -a.
    4. This will reinstall all you need(presuming all URLs, folders, etc are NOT being edited/changed)
    5. Start Domino
    6. Test
    This worked well, and the customer was back up and running on V14FP1, albeit with the older REST API code. 

    After a few days, we got back some information from HCL, which I quote directly below:

    Problem:
    REST API 1.014 Running the APIs in local swagger returns Error "Failed to Load API Definition"

    Possible cause
    The issue is that the scope's Server name is not correct.
    This setting was not working correctly in older versions of DRAPI and was fixed in v1.0.6. It was pretty much ignored in v1.0.4.

    Possible solution/Workaround:
    With DRAPI 1.0.14 you have to add the CN name/hierarchical name of the Domino server instead of host name for example: CN=customerTest/O=HCLLabs in the Edit scope>>Server field>>CN=customerTest/O=HCLLabs or CustomerTest/HCLLabs
    Earlier server name field in scope>>Server field was: Host name of the server i.e keithbrooks.com

    Change it to either CN name or the hierarchical name of the Domino server or can be left blank stating any server that has this scope in the KeepConfig.nsf assumes the database / schema exists.

    Moving further the product dev team is going to update the error message in the product code as 'You need to query a different server' so it makes more sense to the testing on affected DRAPI versions.

    There you have it. My guess initially was partially correct. 

    The configuration was correct for the old version, but because the customer did not upgrade the RESTAPI code along the way, they missed the changes in code at 1.06, which would have probably prompted an HCL ticket at that time as well, but it would have been easier for all of us to see as the issue at that time. as the change would have been fresh in their mind.

    The story's moral is, of course, ABU or "Always Be Upgrading" because of security, code, and functionality adjustments over time. While we don't want to break production, sometimes you must do so for your benefit.

    The second moral is to have a Development or Staging environment for these critical applications. In this case, the lower environment did not match the upper one entirely, so the problem was not seen when we updated the lower environment.