Domino is down, not again I said

Funny how luck does find you and rarely the other way around.

I was driving out to a client site to literally plug in an external USB drive (don't ask, I tried to have the people in the office do it, server was not having it) so off I went.

5 minutes away I get a call that mail and sametime are down. Odd, I thought, staff has been there an hour and they now called? I said I was on my way already, see them in a few minutes. They must think I am a mind reader.

Naturally I get to the office and the server console is flying by with emails and such. BUT Traveler is showing errors in connecting to the very visible server being down.

Internet access was down, sounds like T-1 issue, or the Domain Controller died perhaps? Checking about saw the Firewall had the test light on, good old Sonicwall was letting me know something happened. When the test light is on, it means it went into SAFE MODE which is a euphemism for "nothing is going to work now, go get lunch and leave me alone".

OK, turning it on and off didn't change anything, power cut, still nothing. The only way to resolve this is to kill power, and kill power from the routers that cascaded off it. It seems the Sonicwall does not appreciate repeated attempts to flood itself with invalid packets. Who'd know :-)

Starting back up the Sonicwall then the 2 routers cleared the issue.

In the mean time had to reboot a bunch of servers that had invalid DNS entries.

So the problem we can now resolve, but what caused it to fail?

Best guess so far is there was a power surge or failure that affected either the Sonicwall unit or the Domain Controller and DNS server. No matter how many UPS units we use or what is plugged in where, some of the servers seem to be highly sensitive to fluctuations in power. Didn't want to go to heavy duty UPS but may be forced to do so. But for the few times a year this happens they can live.

Domino gets blamed for mail not working or Sametime or Traveler but as the client said, if they can't get to anything fromt he outside, they know its a firewall or T-1 down and that makes them MUCH happier than to think Domino is down.

In fact, aside from these outages, Domino is usually up for about 40-60 days before I pull it down for OS maintenance. Still trying to train the users it's not Domino, but at least the boss gets it.

PSA from Microsoft Office Live- We should learn from

Keep in mind, most internal IT groups don't send an email this informative, and I doubt anyone got something similar from Google the other day either.

While I may not always appreciate Microsoft, I do respect them enough to post what is good too, especially if you or my organization can learn from it.

Notice of scheduled Web site temporary downtime.


Dear Keith,
On September 12, 2009 and September 19, 2009 we will be upgrading the Microsoft Office Live Small Business servers that host your Web site(MY NOTE: a test site I used to play with note they didn't tell me which site). On only one of these days, your public Web site will be unavailable for up to three hours.

During this brief downtime, visitors to your site will see a message alerting them of the server upgrade, and asking them to check back shortly. We have scheduled these updates on Saturday, the lowest day for site traffic, in an effort to minimize the impact to your customers. You will be able to sign in to your Office Live Small Business account and access your e-mail during the upgrade, but you will not be able to use the site management tools.

While these upgrades will improve the performance, reliability, and stability of your Web site, we apologize for any inconvenience this scheduled maintenance may cause. For more information, please see our service upgrade blog posting.
Thank you for choosing Office Live Small Business.

Sincerely,
The Microsoft Office Live team