Wednesday, December 24, 2008

Worst Practices Submission for LS10

pilfered from worstpractices.com, someone teach them about Domino Security please
I know Bill and Paul are not doing this session at Lotusphere this year but here's an oldie but a goody and it still happens today, er I mean recently(last week).

When moving your data center you should always:

1) Let everyone know with enough time in advance when you will be moving so things like the VOIP phones and Email/Webmail will be down and not surprise everyone.
2) At the cut over date, remind everyone in and NOT in the office you are about to down the servers.
3) Shut down ALL servers appropriately.
4) Tag and label each box, cable, jack, port, router, monitor, keyboard, hub...just....in....case!

Yes, there are a million and one other things to add to the list.

Now let's look at what really happened:

1) We were told the move would be on a Thursday/Friday. No time specified.
2) Thursday came everything was fine, figured they decided Friday was better.
Went to lunch, came back, nothing was up any more.
3) We hope they shut everything down properly, AIX box was the last man standing, luckily they hadn't removed it's console monitor or keyboard, although the whole network was already in pieces so we could shut it down gracefully.
4) Primary Web Server AWOL, although perfectly accessible internally, not accessible externally. Took a few days for them to ask me to look into it, I pinged it and yes it was down, but up on the inside. Pinged it inside, wrong IP address. Checked the box.....they plugged the LAN cable in the wrong Ethernet port on the server. Swapped and amazingly all was well again. Amazingly Sametime suddenly worked again too.

So the next time you move servers or data centers just remember the time you put in upfront saves hours or days in the end.