Optimizing Domino Cluster Failover for Seamless High Availability

Ensuring high availability in an HCL Domino environment requires meticulous configuration of cluster failover mechanisms. Proper tuning not only guarantees seamless user access during server outages but also optimizes overall server performance. 

This guide delves into key settings and parameters essential for effective cluster failover management.

Understanding Domino Cluster Failover

In a Domino cluster, failover is the process by which the system redirects user requests from an unavailable server to a replica on another server within the cluster. This mechanism ensures continuous access to databases, even during server downtimes. The Cluster Manager on each server monitors the availability of other cluster members and manages the redirection process accordingly.
Learn more about how failover works here.

Configuring Silent Failover on Notes Clients

By default, when a Notes client encounters a server failure, it prompts the user with a dialog box before switching to a replica on another server. To enhance user experience by making this process transparent, you can enable silent failover:

    1. Set the FailoverSilent Parameter:

- Add the following line to the client's notes.ini file: `` FailoverSilent=1 `` - This setting suppresses failover prompts, allowing the client to switch servers without user intervention. Detailed information on this parameter.

    2. Deploy via Desktop Policy:

- Navigate to Policies - Settings - Desktop Settings in the Domino Directory. - Under the Mail tab, select Client Settings. - Enable the option "Enable silent failover when a server goes down". - This policy distributes the FailoverSilent=1 setting to all clients, ensuring consistency across the organization.

Enabling Mail Router Cluster Failover

To ensure that the Domino Mail Router can reroute emails to available servers during a mail server outage, configure the following:

    1. Set the MailClusterFailover Parameter:

- Add the following line to the server's notes.ini file: `` MailClusterFailover=1 `` - This setting allows the Router task to forward incoming messages to a cluster replica of the recipient's mail file on another available server. More details on this parameter.

    2. Configure via Server Configuration Document:

- Open the Domino Administrator. - Navigate to Configuration > Servers > Configurations. - Edit the relevant Server Configuration document. - Under the Router/SMTP tab, go to Advanced > Controls. - Enable the Cluster failover option.

Managing Server Availability and Load Balancing

To prevent servers from becoming overloaded and to manage failover effectively, adjust the following settings:

    1. Set the Server_Availability_Threshold Parameter:

- Add the following line to the server's notes.ini file: `` Server_Availability_Threshold=75 `` - This value (ranging from 0 to 100) defines the minimum acceptable level of available system resources. When a server's availability index falls below this threshold, it declines new database open requests, prompting clients to failover to less loaded servers. Further information on this parameter.

    2. Configure via Server Configuration Document:

- Open the Domino Administrator. - Navigate to Configuration > Servers > Configurations. - Edit the relevant Server Configuration document. - Under the Notes.ini Settings tab, add or modify the Server_Availability_Threshold setting.

Restricting Server Access During Maintenance

During maintenance windows, it's crucial to prevent new user connections while allowing existing sessions to conclude gracefully:

    1. Set the Server_Restricted Parameter:

- To restrict new database open requests: `` set config Server_Restricted=1 ` - This command can be issued directly on the server console. The value 1 restricts access until the next server restart, while 2` makes the restriction persistent across restarts. Detailed explanation of this parameter.

    2. Monitor and Manage Active Sessions:

- Use the Domino Administrator to monitor active user sessions. - Notify users of impending maintenance to minimize disruption.

Tuning the Cluster Repair Service

For environments utilizing symmetrical clusters, the Cluster Repair service ensures database replicas remain consistent across servers. To optimize its performance:

1. Access the Cluster Configuration Document:

- Open the Domino Administrator. - Navigate to Configuration > Clusters > Configuration. - Edit the Cluster Configuration document.

1. Adjust Repair Service Settings:

- Under the Tuning tab, configure the following: 

• Number of repair threads: Set between 1 and 20 (default is 4). 
• Check donor availability: Frequency (in minutes) to check donor server availability (default is 5 minutes).
• Retry failed repairs after: Interval (in minutes) before retrying a failed repair (default is 5 minutes).
• Maximum number of retries: Number of retry attempts before marking a repair as unrepairable (default is 3).
• Repair performance: Level of CPU and disk I/O dedicated to the repair service, from 1 (slowest) to 5 (fastest) (default is 3).
• Repair logging level: Level of logging detail, from 0 (none) to 4 (diagnostic) (default is 2).

- These settings help balance repair efficiency with system resource utilization. More on tuning the repair service.

Conclusion

By carefully configuring these parameters and settings, you can enhance the resilience and performance of your HCL Domino cluster. Regular monitoring and adjustments ensure failover processes remain seamless, providing users with uninterrupted access to critical services.

SnTT - In Which Port Settings Don't Quite Work

Been a long time since I wrote a Show n Tell Thursday so let's get on with it.

Building an IBM Domino cluster is not as difficult as it was years ago and I was done with the basic parts and wanted to test the fail over.

I figured easiest way to do this was to enter at a server console "stop port tcpip" while connected via rdp to the server.

I tested mail clients could fall over to the 2nd server and mail routed. Great.

Back to the server and server console and tell it to "start port tcpip".

I didn't think much about it and since there were some server updates to add to the box, I decided to shut it down after the updates were done.

It came back up and was running and I was working on another server in the domain when the client let me know no one could connect to the server. Odd, I was there and it "looked" ok. On closer inspection, it was not ok.

Errors that said databases could not be found, no route found, no network could be found, trace connections and some other fine error messages that mean nothing to the average person.

After poking around a bit, I figured the notes.ini needs to be edited. I checked with Rob Kirkland, of this book fame on Domino System Administration, and followed it to solve the problem.

I found the TCPIP port listed under disabled ports. Deleted it form there.
Found TCPIP=TCP,0,15,0,,45088 was missing which is for compression and encryption (see below).
Also missing was Ports=TCPIP

After putting it all back together, rebooted and all is good again. Still not sure why the commands failed and did not help this 8.5.3FixPack6 server but at least someone else out there will not have to waste an hour or 2 tying to figure it out.

Selection NOTES.INI parameter

          Nothing TCPIP=TCP,0,15,0,,12288,

Compression only TCPIP=TCP,0,15,0,,12320,

Encryption only TCPIP=TCP,0,15,0,,45056,

Both TCPIP=TCP,0,15,0,,45088,

EDITED: October 20, 2020 The below is pulled from the most awesome list of ini settings ever.

https://entwicklercamp.de/customer/notesini.nsf/85255a87005060c585255a850068ca6f/8d0f57b937909eefc1256e9e0045eddb?OpenDocument

Configuration for a TCPIP port.

Examples: TCPIP1=TCP,0,15,0,,12288

TCPIP=TCP, 0, 15, 0 [,,x] <----------- This is the setup for your TCP Port
.......|...|..|... Application Data buffer size (0 is default which = 8000)
.......|...|..Placeholder not used by TCP
.......|...Placeholder not used by TCP
.......This is the port driver name

The last parameter x ([,,x] above) can be decoded as follows:
0 X 8000 Encryption is enabled
0 X 0020 Compression is requested
0 X 4000 driver is internal
0 X 2000 no-op
0 X 1000 always for V2 and V3
0 X 0002 set to log modem I/O
0 X 0004 set to enable RTS/CTS

8020 which would be Encryption plus Compression is in hexadecimal code 45088

Selection NOTES.INI parameter
Nothing TCPIP=TCP,0,15,0,,12288,
Compression only TCPIP=TCP,0,15,0,,12320,
Encryption only TCPIP=TCP,0,15,0,,45056,
Both TCPIP=TCP,0,15,0,,45088,

Quickr 8.5 for Websphere Portal Cluster Install doc

I have been on site all week at a client that was a little out of my depth. Sure Quickr is one thing, clustered Websphere Network Deployment (WAS ND for those in the know) with Network Deployment Manager on top of it is another story.

Most of our clients run Quickr Domino or J2EE usually in a stand alone situation.

How hard could it be to cluster it and get it all together? Right?

Let me just say building servers or clusters in a group is not easy, especially when I have no access to the outside world and am not the one doing the work. I was advising and translating IBM speak to the clients. But watched the team build a server, which we did pretty well considering we did not have this document that I found eventually.

Clustering IBM Lotus Quickr 8.5 for WebSphere Portal: An end-to-end Guide
By: Jeff Johnson, WebSphere Portal Customer Enablement Team
IBM Collaboration Solutions, IBM Software Group
Durham, NC US from February 2011

We had all types of issues, from server outages, forced updates during what I consider normal working hours(meaning before 8pm), oops I rebooted the server moments...in the middle of editing key parts and thus corrupting the install, servers being shut down by IT support without warning, IP/DNS issues, blocked ports on the servers themselves and well, not one of my better efforts including being unable to tether my netbook to my cell phone.

But this document really does a great job and following it will provide you to your ending.

Keep in mind when the installation says this will take a few minutes, it means 15-30 and the whole process can take hours to complete not including customizations and other configurations, just letting you know in case you ever need to do this for yourself or a client.

Fud Buster Friday #45 - Our Systems Never Go Down

Better late than never, as you can see I made it to the UK and have just now gotten my broadband hooked up at the house.

For anyone that still doesn't believe you need redundancy or clustering or whatever you prefer to call it, I leave you with this real lifer scenario.

A prominent cell phone telecom provider(T-) in the UK was down ALL day, basically from around 9am to after 5PM. While their service was sporadic for some people, I and my wife just had problems trying to get our phones and the new SIM chips to work.

So what should have been a 5 minute effort turned in to a 12 hour one. My wife's Blackberry only started getting email this morning. Mine didn't get proper configuration OTA until this afternoon when I had to manually change some email settings.

But the fact that T- answered their free phone call help service with "Sorry but our internal systems are down, can we help you with general questions" was not very helpful at all.

Nor was the 4 different people that I got disconnected from who claimed they would call me back. None ever called back.

Evidently their internal servers were either not set up for redundancy or for disaster planning. I have no idea if they had server issues or telecom issues, but either way, this is NOT how you ensure 100% uptime of your network.

So the next time someone says to you they don't need clustering or do not want to spend the money on a second server or enterprise license, do them a favor and figure it out for them and let them know they can pay you for it when they use it or lease them a backup box or co-location.

Living proof of this problem is everywhere, does you customer want to have these problems? Even Gmail has outages, but clustered Lotus Domino Servers don't have these problems. Well they do if you put all the clustered servers on the same VM machine as one of my Fortune 150 clients did and the VM fails.

Got Cluster? Why not? Don't be lazy

For those who never think about it or believe it is the scariest thing on Earth, because they heard that from some monolithic vendors who only run on one OS, go set up your Lotus Domino clustering today.

Right now!

Why is this important? Maintain 100% uptime to the user community is the main reason. But it enables you to manage upgrades better, no scheduled outages as far as users are concerned. Disaster recovery, business continuity too.

This isn't new to R8 although some UI changes exist as well as enhancements to policies/configurations since clustering came out in R5.

Really, 7 clicks and you are done with the basic setup. Can you handle it?

Caveat, you MUST be running Enterprise server or you are out of luck. AND I am not detailing all the steps to ensure your files are replicated in this post or about selective clustering or options to think about. This is really a quick benefit put in simple terms.

Ok, I lied, not exactly 5 clicks and you do need a 2nd Lotus Domino Enterprise server available first as well.

1) Open your Administrator client to the Configuration tab, open the Servers twistie and Select All Server Documents.

2) Now check in the left column which one(s) you want to cluster. Don't be shy, you can put more than 2 in there at a time.

3)Click Yes at the confirmation window.
4)Select create new cluster.
5)Enter a Name for the Cluster (unique to identify it).
6)Select yes or no to have Adminp do everything for you.
7)Go over to the Files tab and click on the Database tab in the Right column and select Cluster. Now you get to decide which files are in and which files are out of the cluster and if it is in/out of service and is replication enabled.

Done. Really.

Don't believe it, go test it.

The left column has Cluster listed, expand that twistie and select Clusters, your cluster should be listed.

Want more detail?
Go to the Server tab and the Analysis sub-tab and then in the right side click on Analyze then Cluster.

This will set up the Cluster Analysis db so select what information you need to know and then make sure to click on Results Database to specify where to place the file.

In your notes.ini you might want to include the following:
RTR_logging=1
this will show all the cluster replication messages on the server console for those that love watching fast moving objects from their chairs.

There are way more choices and options to monitor, configure and check on the cluster but this is just another great reason why Domino makes life easier for admins and users.

More information available from the IBM Infocenter for R8 here.

And for those with a single server, we can work with you to provide backup for you on our nice Domino farm. Hurricane season is back so be prepared.