Welcome to Troubleshooting Sametime Week

This week is Sametime Troubleshooting week for us.

Tuesday we are on site all day to resolve ST 8.5.1 issues where business cards are not working and there is a delay by a few seconds between sound in online meetings.

This holiday weekend I was moving a client's ST 801 server off of VM to a new nonVM server. Ran into a problem afterward where we lost HTTP access for ST. While I am not sure exactly what was different, the server is known by various names for historical reasons and possibly we have a DNS issue to narrow down.

This IBM technote, #1176728, resolved it very simply. Note it is useful for ALL versions from 7.0 up to 8.5.1.

Along with this is an upgrade to Traveler 8.5.2.2 which we had an issue yet to be cleared. While unpacking the file we got an error message about "Please select another location" and we await a new download to try again.

The Fun of it all. And then there was my accidental blocking of our internet password for accessing a client site. Easily cleared but amusing nonetheless.

RDP Gray Screen of Volcanic Ash

As you have read, we had a box get attacked last week. Still finding little things on it to fix. Like this gray box problem for RDP. Login and the screen would stay gray or the login box would just stay on top.

My friends online had some advice but the funny thing was no matter what we did, we could not manage to logon to the server at the console. Eventually I got RDP back up, after a LONG LONG delay.

How do you fix something that doesn't exist? HOTFIX! Maybe

Yes, you read that right, hotfix. Microsoft has released a specific hotfix:

Article ID: 942880 - Last Review: February 1, 2010 - Revision: 2.0

A Windows Server 2003-based computer may stop responding after you enter your user name and password in the Windows logon dialog box

Once the server is back up, run different virus programs than your normal one, Sophos runs a 30 day trial version, handles servers too. Run MalwareBytes too.

But the underlying issues manifest themselves in many ways. Check your Windows Server Services carefully. We found a rogue Com+ entry as well as some other references that we disabled.

One thing which we did that also caused a problem was to disable numerous items that we no longer used. Evidently uninstall doesn't really for everything, who knew, right?

Two things we disabled, ended up blocking our log in. One was a remote control viewer software that should no longer be there, yet changing it affected login. Cleaning that up in registries and uninstall codes.

But the worse for wear server is back now, cleaned from the sole Blacklist it landed and humming along. Along the way I found a different server which was up for over 120 days so far. Given some fix packs came out in that time not sure why it got skipped but it is scheduled for updates over the long weekend.

So keep your head above water, work on getting the server back up, virus free of course and accessible while you do your forensics.

"What is your Favorite Color?"

Blue, no Yellow...of course. Thank you Monty Python for being IBM lovers.

An interesting conversation from the other day about passion. In an interview they ask one about their passion.

While I thought this was an okay question, others believed it was intrinsic to defining someone and their space within the company. As a contrarian, naturally I disagreed.

It goes against my thinking that one would be hired solely for having a passion for IT, as an example. You can have all the passion in the world, but if you can't figure out why a server is down, what good are you in IT? Enthusiasm is good, inability to think is not. I prefer to interview people to understand how they think. When they will be working on a deadline or in the middle of the night I need to trust they can get the job done.

When hiring for sales, I want them to prove they can sell me something from my office. When hiring for marketing I want them to take a stand and back it up.

Maybe it is just me but I believe that a person is passionate about what they do or when they are applying for a job they seek what they prefer. Why would I apply for a developer job? It is not me even though I could probably meet the requirements in some shape or form.

And so if you saw a resume and decided to interview someone, I presume you would think they have a passion for the role.

I recognize this may not always be the case, but if I like IT and am a member of PETA, but need a job, and the only one open is with a leather clothing manufacturer, I may not have as much passion for my job. A bit extreme, but in this economy not unlikely either.

So how do you interview people? The manhole cover question is old.Yes technical knowledge is important but beyond the technical, how else do you do it? Psych questions are lame. Those of us in the social stream probably have enough footprints for managers to get to know us a bit, but many others have none, so you need to get to know them better.

LDAP and SPAM Attack Last Night on a Domino Server

Doing a post mortem this morning on a Domino server that was compromised last night. Wish this had happened a few weeks back when I was preparing my slides for The View conference for my session on breaking into Domino.

This was not a major scale customer, more SMB size and a little looser with their firewall and security appliances. But a good lesson to everyone.

While it looks like a purely spam attack effort, from the outside, the server log revealed a stronger LDAP attack.


Let's start with the basic details, in US Eastern time.

05/18/2011 03:24:21 AM  LDAP Server: Bind request for CN=LDAP USER,O=ORG failed: Invalid credentials specified: failed to authenticate

This was the first of 100's if not 1,000's of LDAP requests sent during the night. the account used to authenticate LDAP lookups is unique and has a password which should be okay. I say should be because who knows these days. Next we saw this:

05/18/2011 03:24:27 AM  SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 24.237.115.235
05/18/2011 03:24:33 AM  Router: No messages transferred to MAIL.RU (host mxs.MAIL.RU) via SMTP

What we found was the SHORTNAME was from a demo account. You can guess the password it had probably, evidently so could the SPAMMERS. This explains the SPAM that was sent out over the night. Every admins nightmare, demo accounts that don't get turned off or deleted. And of course once one is compromised, social media or collaboration takes off and every BOT tries to get in:

05/18/2011 03:25:24 AM  SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 71.80.203.51
05/18/2011 03:25:24 AM  SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 98.251.18.30
05/18/2011 03:25:25 AM  SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 24.2.139.106
05/18/2011 03:25:25 AM  SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 24.237.115.235
05/18/2011 03:25:25 AM  SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 76.1.160.187
05/18/2011 03:25:27 AM  SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 223.130.33.225
05/18/2011 03:25:30 AM  SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 203.169.117.225
05/18/2011 03:25:29 AM  SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 116.14.38.190

These lasted until 3:52AM so about 25 minutes or so before it stopped. Yuck indeed.


We found the Names.nsf had Author access. So conceivably, if the demo account had Admin rights, this could very well have been even worse.

What do you do?  Where to Start. Would you even know if any of this happened last night? While some of you go run to open your log files I will wait a minute.

Now that you have returned and hopefully not found any of these issues, let's review how I was alerted and what you may want to think about on your end as well.

 First, DDM, Domino Domain Monitoring is YOUR FRIEND, USE IT. Needless to say I received tons of emails alerting me to a problem. Granted at 3AM I am sleeping and so when I got up this morning, the fun began.

Why did DDM send me emails? Because we told it to. Yes, they can be annoying, yes at times it seems, like today, they will never stop. BUT which would you rather be? Ignorant of your security breaches? I didn't think so.

Open the DDM database on your server. Sort by date and you hopefully should see few entries if any, if you are on top of the environment. If you are not, well, get working, you have issues. Now we do not generally care about informational items like if a user was not found in the Directory type items. We care about fatal warnings, failures and Warning High items. Surprisingly enough this line:

nHTTP: SHOTNAME [166.137.8.176] authentication failure using internet password

is set to be a "Warning Low in Security" by IBM. So by default, in theory, one would not notice it if they only cared about the higher end items. BUT, you can go inside and edit the level of severity which is what we do for this and the LDAP, IMAP, and POP3 entries when they are applicable.

You make this change in severity type by opening the item from the list in DDM.
Next find the Severity and type line and at the end of the warning is a DOC LINK, click on it to go to the events4.nsf Database where you click on Edit Document.
You will now see Event Severity have a drop down. Set this to whatever level you prefer, we usually set it to Warning High or Failure.
Save and close.

This presumed you already had set up your notifications.

If you have not, shame on you if you are an admin, do this:
From the DDM database select Open Configuration.
This opens the events4.nsf database and you need to find Event Handlers, then By Server.
Click on New Event Handler.
Select the server(s) to monitor and a trigger event, but for this just use any event.
Click on Event and under Events must be this type, select Security.
Select if you want specific messages or all severity and leave Events can have any message checked.
Click on Action and Method, We use email but you have many choices for notifications.
Enter an email address and enable the notification.

Done

So what else do we do? We changed the account password and banished the account to the Deny Access Group because we want to see if they come back and if there are other accounts we are missing.

Make sure to check each item in the DDM and if the level of severity is not high enough for you, go raise it and make sure a new notifier is built around it.

Could this be handled in other ways. Yes, We have changed some SMTP settings to block the domains that the SPAMMERS were using.

In this case, because it was a real account that the password was guessed, an appliance could have seen masses of emails and shut it down. Beyond that it could be a gaping hole in some environments. Especially you that leave LDAP anonymous access.

Students want to Learn

Once again I will point you to my friend Luis Suarez. If only I could beat him once to the blog post.
Luis posted this great post and video about the state of education and which one are you or I?

Our ensuing conversation highlighted how both of us saw our education similarly thought we are across the Atlantic Ocean from each other.

Today, as posted earlier today, I took part in a forum to help junior and senior students in high school get some interviewing experience, advice and knowledge sharing.

The slides are on the previous post. I encouraged them top share with each other the knowledge they received not just from us but from each other.

Wish we could have spent more time together talking afterwards but the students needed to get back to their studies. The four of us who did the forum discussed the morning on our way out.

These students, about 25 of them, were very interested in listening to us. While we all take some basics, like dressing the part, as a given, to students it is odd. After all, if you would not wear a suit to work, why would you wear one to an interview?

Indeed, a hard one to explain without laughing at the state of our business world. But we explained that one, as well as many other items they asked about.

They were surprised by how much I and the others stressed customer service. If we did not have any customers, we would not have any business. The best way to have customers is to have great customer service. As I write this I owe someone a patch fix which was not working last I sat with it.

The students are taking a mix of science and technology courses and do their own school networks(cabling, drops, etc..) and Cisco router configurations. Just like most of us did back in high school. Web design was also a big highlight, Dreamweaver came up a bit and the graphic designers were so interested in designing new and cool things.

The need to resolve conflict and manage groups of different people was discussed and I am happy to say that most of them have some leadership backgrounds. If they can put that to good use in their future they will get very far. As I told them in my opening, the business world is still like high school, so if you can manage it here, you can manage it at work. They were very happy to hear this. Shocked, but happy.

So maybe Luis and I are not giving our current skills enough slack for teaching analysis or observing and learning from their experiences. This is active learning at its best! These students do get it and want it, now to help them find some internships and jobs for the summer.

This was harder than I expected due to the students not having the means for equipment, software or even internet access at home or easily accessible to them even if we wanted to give them a chance. Some can or will drive this summer which makes it easier, if they have a car to use, but for the majority they could do well with some small projects, if they had the opportunity.

A troubling conundrum that outlines the balance of home efforts with school efforts to provide a full environment to the students. Imagine if you were taking IT courses but could only do your homework at school.

Sending the Elevator Back Down Today

This morning I am spending my time with senior high school students from Miami Central High School. Some of you may recognize the name from when President Obama visited them in March.

This is a magnet school that has raised their levels from the lowest level to a C level. A huge gain, for the staff and the students in a short time. They are on their way to bigger and better things.

While I have been a part of the South Florida technology community almost since I returned to Florida 10 years ago, the community has changed just like the rest of the world. Today the umbrella organization, the South Florida Technology Alliance, is the primary IT community remaining. Various groups are a part of the SFTA, one such group, IT Women, accepted my offer to help them with the event today.

What has slowly been changing as well is the expectation we, the IT community, hold for the next generation of students who will also become employees soon enough.

With this in mind, the IT Women organization had asked for help with the students and their mock interviews to help them prepare for their upcoming internships or job interviews. I volunteered to help, not expecting to be the main presenter, but to assist in the mocks. To my delight and surprise, Sherry Giordano, Executive Director of the IT Women organization asked me to give the students some insight and advice.

The slides are a bit more textual, but it is an informative session for the students. Will follow up later today or tomorrow with my experiences. If you are interested in a similar event or found this looking for advice for your high school students, you can get the slides here.

Why comments are sparse on a blog

This doesn't usually bother me, but it seems to bother others.
While I am always amazed when there are comments, others expect them perhaps?

Some basics:
1) Never expect anything from your blog, unless you plan on making your blog your main money maker or the companies front lines.
2) Try not to require people to register to comment. this blog does allow anonymous or registered or via some other oAuth options. Due to spam, try using a Captcha program.

3) Don't ask for comments on topics. You can ask for help for a project or slidedeck, but that is all.
4) Not everything of importance to you, will be important to everyone else.

Musicians will tell you their hit songs, to them, usually were junk. But the junk made them millions! And the ones they loved or thought would be great, get no respect. That about sums up blogging and writing sometimes as well.

Now, psychologically speaking, let us look at this based on an introvert and extrovert theory.
If you are blogging in a technical way or for a technical community you may find fewer comments on your posts. Why? Because the introverts, which tend to be developers and more thinking people have a different view of the world. They may agree or not with your post but may feel no obligation to comment. They like to have discussions about why and how or based on a line of thinking they are investigating.

Compare this to the extroverts. Usually found in Business lines, Sales or Marketing but can be in IT as well. They like to be seen and heard and offer opinions because they like discussion. It's not always about being an ASW.

Keep in mind it is not your writing that is the problem. Quite the contrary, the more you write the better you get and the more people that can experience you and join in your conversations.

If you enjoy what you blog about, then keep going. The road to awareness of your blog takes time and when you can branch out from your local blog list and find new ones to be a part of that community will help you as well. Do not try to write as someone else, be authentic and real.

Reply to All now called Post to Twitter

My friend Luis Suarez who's blog and Twitter streams I read often, posted this yesterday on his blog.

Luis suggest that Activity Streams will save us from Information Overload over Craig Roth's post that this will just be another inbox..

This is an interesting point, one which Google, Facebook, Novell, IBM, Linkedin and many others are using to feed us information.

In one way, this is Reply to All gone horribly wrong. One can post or reply to people and everyone sees it, like in Twitter or Facebook or Novell's Vibe and was this way in Google's Wave. Luckily, and IT for the moment is happy about this part, none of it gets stored locally or on a server.....yet!

Surprisingly, one can learn some great bits of information from the stream, no question. But it also means one must be reading it 24x7. That doesn't make sense. Thus developers created Filters, and they were good. BUT how do I filter what I don't know? This is important. I can filter by person, or idea or product but what if someone posts about a rare concert by Warren Zevon? Do I set up filters for every band, then briefly read them for anything interesting?

Sounds like another form of inbox to me.

If you like one program, whatever it is does not matter, and it can feed all of your interests, feeds, pictures, etc.. shouldn't you use that as your only client/application?

The question will be, who's app wins.

The flip side is, in theory, better awareness of those around us, directly or peripherally. My friends around the world I can vaguely know where they are and what they are doing is interesting. Not very business helpful to me but I am sure others would like to know these details for their business.

This is also the issue with streams. Privacy. Making very clear which incoming and outgoing messages are private or not to be shared. The definition varies immensely.

Luis hit it perfectly when he described what the streams do for your organization in this line: They help flatten organisations and traditionally hierarchical structures. Now anyone can contact the CEO or VP of Marketing or whomever. In IBM one could always do it, though naturally one wouldn't on a daily basis. But many other companies, some people don't even know who the executives are, they might not even care.

But once you find your company personnel on a service, BAM! you can talk to anyone. Doesn't mean they will listen to you, but the fact that they, in theory, are listening, means a lot to employees and your vendors/suppliers or Business Partners. If something is important it will bubble up and be found or heard, but that doesn't mean that other topics of secondary or tertiary interest should be ignored.

The challenge is how to filter for those lower tier details which sometimes are more important to know about.

Tweetdeck is helpful for Twitter but unless one has a very wide screen or multiple screens, one can not see all the other streams because they each get a column. This is just one example of an inbox of the future. Not one giant stream of information but a break down in UI in a more granular way.

There is another aspect of all of this data and information flying past us and I will get to that in another post.

Microsoft to Buy RIM after Absorbing Nokia

Back in July 2008 I posted this.

What if the wrong messaging server buys RIM?

What if Microsoft buys RIM? Micorosft already has Nokia in their pocket so we could see the perfect device.
A UI designed by Nokia.
A form factor designed by RIM.
An OS designed by Google. No one wants a Wiindows Mobile OS. Maybe they tossed Symbian a bit too early?


And with RIM supporting multiple devices now, this would be a great opportunity...for Microsoft. That is, if they really want to own the mobile messaging and application space.

RIM has been buying some smaller niche companies that could be leveraged better by a company that owns the messaging infrastructure. IBM would be nice to see from my eyes, but at this point it looks more like Microsoft.

The announcement that Bing would be on the Playbook's just is another effort by RIM to find some cash in a down market for themselves.

Having used a 7 inch tablet for a while now I prefer the larger iPad size, but the applications RIM has for everyone may make up for the size factor. In this case, size does matter.

But if there was an opportunity for RIM to move forward, it will have to be with someone else, like Novell, sometimes the future can be better with new management and more importantly money.

So is Lotus, the IBM brand, dead? Or not?

Let me say at the outset, IBM has not terminated any Lotus products that are currently on the market. Older editions (R7) are sunset just like any other product line(IE6 is a good example).
IBM has not informed me of any such plans and if they had, well I couldn't really tell you any way due to NDA.
So just get that out of your head right now and send the Microsoft sales guy to me if he wants to argue about legacy products like Windows XP, Exchange 2003 or SQL 2000 that are still not sunset, see here for more listings: support.microsoft.com/lifecycle.

Customers that have been loyal to IBM and Lotus believe given the Lotus Knows campaign from part of 2010 and the continuous updates released across the product lines that all is good and just getting a face lift.

Customers that refused to touch Lotus lately are now being amenable to IBM branded items, even though they are still the same products as before, more or less. Perhaps the CXOs feel that an IBM solution is a safer and easier discussion to have compared to one that mentions Lotus. New Coke this isn't. This is just a branding exercise not a rewrite. And in the executive suites, names are MUCH more important than technical coolness.

Oddly enough if (no idea what the actual number is) 85 out of 100 Fortune 100 companies still use Domino applications this is weird to me that they refuse to believe it or mention it or anything else. We have a backyard but never mention it, just enjoy it.

IBM is taking it's time in the slow rebranding process. LotusLive is staying, right now, as Lotus. Notes and Domino look to be staying for the moment too(I expect January 2012 to be full of announcements). The newly announced IBM Champions skips Lotus for the IBM Collaboration Solutions moniker.

IBM still sells Smartsuite, mine still installs as Lotus, wonder if that will be rebranded too. After all it still makes money for IBM somewhere in the world.

Amanda Bauman, via the IBM Collaboration Solutions User Experience Blog posted about a number of shorter URLs
"To help make it easier for you to find the information you're looking for, we now have simplified URLs to some of our key Web sites."


What are they? All Lotus Links. http://learn.lotus.com , http://wiki.lotus.com, http://doc.lotus.com, http://m.lotus.com 
We have asked for this for a while, simpler, easier URLs. Thank You. But, it did not say the http://wiki.ibmcs.com exists. 


Lotusphere Comes To You is, for this year, Social Roadshows. No idea what next year's will be.


Lotusphere itself? No idea but maybe it should be rebranded with the Devcon name which is what it has become in the last few years. Admincon just doesn't have the same ring to it.


As a Business Partner IBM asks us to get certified in products, now product lines or areas, such as collaborative solutions. Sounds more like a sales effort than a technical one to me. We have a choice of different exams which no longer totally focus on a specific product which has both a up and a down side.


What we ask, as Business Partners, customers, admins and developers from IBM is simple. Just do it! If you want to rebrand, we get it, but we need your help as well in getting a clear, concise and definitive answer to the BPs and the world at large. 


Take the leap of faith, be Google, be Apple. 
Be definitive. 
Don't be wishy washy anymore!
If you believe this is the best way forward, back it up and show it.
Be a man, or woman, about it.


Inside IBMers wonder why the outside doesn't "get it" about the rebranding.


It is because you never really came out of the closet to tell anyone on the outside the plan.


Stuart said more in his post on this topic and I encourage you, if you made it this far, to read his thoughts as well.

No Phone and No Internet, but Bin Laden Got work done

sadly Somehow and that probably shocks so many Westerners who can't imagine being disconnected for even a few minutes.
Sadly it worked for him too ... for years.
But how did he get intelligence reports? Did he even care?
Was he a true CEO, so far removed from the front lines that the only thing that mattered was him and his thoughts and it mattered little whether they were winning or losing their battles?
Or was he the CEO thinker type? Preferring to be introverted and accomplish his work in the background?

Naturally he wasn't checking in on Foursquare or Gowalla or Facebook or Twitter but no doubt coded messages did get distributed through some intermediaries, probably through some of these and other countless systems. Intelligence gathering is endless, thus why IBM is betting so big on analytics. The sheer numbers of data crunching makes it impossible for anything but a computer to sort it all out. But interpreting the data correctly is still important.

Sure he was wanted and hunted by everyone, well, except the Pakistan government it seems. And he knew very well that any phone or internet would give him away. So without the distractions he could make his grandiose plans.

Not that this could have been seen as coming. The US security and military teams did an excellent job of keeping this all quiet. But if you think like a Poker player for a second, was the US being quieter about Osama since they found out where he was hiding? If so, would he have realized this as a bad sign and high tailed it to a new hiding place?

So maybe not being plugged in, even a little bit, was his downfall? Businesses that still refuse to pay attention to the online world in all its forms are missing out on information that could help them.

Congratulations to President Obama for not just "bombing the site to kingdom come" but finding a way to get the body and proof. I trust at some point the video/photos will be released, else the conspiracy theorists will come out.

Thanks to all the military personnel that were part of it and hope you all get to come home to your families sooner than later because of it.