Fud Buster Friday #29 - One Outage isn't so bad.. is it?

Naturally it depends on who has the outage and how long it lasts and well, a million other things.

As we know Google had an outage this week. Ho hum. Welcome to IT guys. The world keeps spinning, we get blamed, we fix it, life moves on. Or does it?

Just spent over a week on a server crashing daily problem, because of my laziness and lack of attention to the clients plight. Well, to be fair, I was at 2 new clients this week, plus demoing for a 3rd and rewiring my office but the bottom line is I took a casual liberty attitude with a clients business and that's not good.

The client is extremely happy I have resolved the problem, will wait 72 hours before total acceptance.

But this taught me, and should teach you too, that one outage is too many for ANY client. And repeated ones spell disaster for any IT credit or expectations and it is VERY hard to recover from this and in some cases leads the customer to think about alternative email solutions.

The client had turned down clustering, even a backup replicated server(I am sure some PC could have been found but I digress). Their backup procedures are appalling and they have no idea why they were running LDAP, POP3,DECS, DIIOP or IMAP on the server. (Note to jr admins, just because you see a check box does NOT mean it must be checked.)

The server was throwing out the usual errors about calendar profiles, users not found in the NAB and a bunch of agent signing errors, which happen to run their apps but signed by a previous admin/developer, of course. And they had small quotas like 150MB. And of course you can guess which mail files had these errors. Funny how nonexistent people can fill up a mail file,waste disk space, routing, backup time and space and even have agents running! Also, if you have a donotreply@corp.com address, be prepared to have many replies to it. The fix is to just have an entry in the NAB and a mail file for it that purges every day or 7 or whatever you want. Otherwise, like the client, you could have 100's of dead mails in your 4 mail.box files.

Suggested to the client to take care of these, only to realize they could have been all along and never did. So as any good consultant would, we billed them for the mini clean up and are planning on a bigger clean up under a new project.

I saw the errors upfront, but we were working on a (thought we were)bigger problem and could fix these later. Remember, Eat the Frog, do the stuff you don't like or want to first and move forward. Taking the easy way out will never help you as an admin.

The other thing is delegate. It was bumped to me as others had tried to tackle it already. I was not happy none of them had resolved the errors or the small things.

Training is important, but walk the talk. As the Worst Practices session at LS08 showed, even the great ones don't always follow their own guidelines.

So I apologized to the client and oddly enough they were just happy anyone could resolve the problem. It has been a month or so off and on.

The downside is their clients just expect the server to be down and even if it stays up won't care. The worst part, for me, was after it all, they tell me they are going to move off Domino for mail. Working on that problem next and may blog it if it gets interesting.

Worst Practices Submission for LS10

I know Bill and Paul are not doing this session at Lotusphere this year but here's an oldie but a goody and it still happens today, er I mean recently(last week).

When moving your data center you should always:

1) Let everyone know with enough time in advance when you will be moving so things like the VOIP phones and Email/Webmail will be down and not surprise everyone.
2) At the cut over date, remind everyone in and NOT in the office you are about to down the servers.
3) Shut down ALL servers appropriately.
4) Tag and label each box, cable, jack, port, router, monitor, keyboard, hub...just....in....case!

Yes, there are a million and one other things to add to the list.

Now let's look at what really happened:

1) We were told the move would be on a Thursday/Friday. No time specified.
2) Thursday came everything was fine, figured they decided Friday was better.
Went to lunch, came back, nothing was up any more.
3) We hope they shut everything down properly, AIX box was the last man standing, luckily they hadn't removed it's console monitor or keyboard, although the whole network was already in pieces so we could shut it down gracefully.
4) Primary Web Server AWOL, although perfectly accessible internally, not accessible externally. Took a few days for them to ask me to look into it, I pinged it and yes it was down, but up on the inside. Pinged it inside, wrong IP address. Checked the box.....they plugged the LAN cable in the wrong Ethernet port on the server. Swapped and amazingly all was well again. Amazingly Sametime suddenly worked again too.

So the next time you move servers or data centers just remember the time you put in upfront saves hours or days in the end.