Some days you just can't argue with simplicity. Duplicity is another story.
Junior admins remember, all is not as it appears.
While knocking off a number of issues at a client site upgrade it was reported a server was not accessible from the Domino Administration client. Funny thing is, the servers were replicating with it, no problem at all. But no clients could connect to it.
I asked the usual questions, have you verified DNS names, server document references, TCPIP references, etc.
All looked good they say. At a certain point it impedes my efforts so off I go to troubleshoot this one. Here is what I find:
1) Server document security page has no one allowed to access the server. Similarly the admin group and server group were not listed. Security settings were all default too. I know we set them, some box has an old copy that replicated, fine, changed, still nothing.
2) Can access the server via RDP and map a drive to it, which led me to #3
3) Windows 2008 Firewall. Damn, that was it. Although we set all the new servers to the exact same settings, this box did not have the clients cleared to be able to access the server.
REALLY! Having been on site at some government facilities that lock down every port, protocol and driver I should have checked it first, but you know you start with the larger focus and spiral down to smaller and smaller till you nail it.
Nailed it, now on to other things.
Speaker, Blogger, Author. Posts about HCL Digital Solutions, HCL Software, Lotus, Management, Collaboration, Competitive Intelligence and Strategy, Productivity, CRM and B2B Marketing.
Showing posts with label troubleshooting. Show all posts
Showing posts with label troubleshooting. Show all posts
Tuesday, May 8, 2012
Thursday, March 1, 2012
Humbled by an Expert
Yesterday I played hooky from work. Not the whole day, just the morning.
Why did I do it? Because I was learning. I was in need of seeking out someone who had more knowledge and experience than I to solve a problem.
The problem was important and although in the end I was on the right track, and indeed had validated the conclusion the expert came up with, I also could not solve it. As it turned out, neither could he, although we worked around it.
What I lacked in experience, I made up with in Troubleshooting ability. Yes we were surging power, yes we had tripped a fuse or 2. That was my laziness somewhat but the real problem was the loss of partial power.
Picture the light going on in your refrigerator but the motor not working to cool the food and you get my meaning.
So while we cleaned up some bits and ends in need, the problem still came back. It was only after looking at some very specific parts of other chips that we found what we were looking for. One chip had lost a leg and was causing a short. Soldering on a new one and replacing it put everything right again.
The kids should now be happy the pinball machine is fully functional again. Except for the bit which I was stuck on myself. I knew the wiring was good, the bulbs were good and power was good, but still no luck. We traced it back to the circuit board and well, for 2 lights not to work, I can live with that for now. The cost of a new board or repair can be up to $200.
But it was a great experience to learn more, see new ways to troubleshoot the machine and do stuff I would never do to a chip board. Surprised it didn't snap. But he knew what he was doing. And that is what you want from your experts. That ability to take something not seen or experienced before and work your way to the level of the issue. And then know how to work around it or fix it.
It is what I do for Lotus products and various other software programs and other technology, both new and old. It is why people pay us and come to us for help for our expertise.
While I was with him, someone else called me and needed me to come out to their house and resolve some wireless issues. The 2 of us talked about this as well and appreciated each others worlds.
When he left I said hope I don't see you again soon, although I do look forward to the next visit.
If you need Fred and are in South Florida, here is his website, http://www.homeusepinball.com and phone number (561) 683-5893. Thanks again for the lessons and my kids appreciate it even more.
Why did I do it? Because I was learning. I was in need of seeking out someone who had more knowledge and experience than I to solve a problem.
The problem was important and although in the end I was on the right track, and indeed had validated the conclusion the expert came up with, I also could not solve it. As it turned out, neither could he, although we worked around it.
What I lacked in experience, I made up with in Troubleshooting ability. Yes we were surging power, yes we had tripped a fuse or 2. That was my laziness somewhat but the real problem was the loss of partial power.
Picture the light going on in your refrigerator but the motor not working to cool the food and you get my meaning.
So while we cleaned up some bits and ends in need, the problem still came back. It was only after looking at some very specific parts of other chips that we found what we were looking for. One chip had lost a leg and was causing a short. Soldering on a new one and replacing it put everything right again.
The kids should now be happy the pinball machine is fully functional again. Except for the bit which I was stuck on myself. I knew the wiring was good, the bulbs were good and power was good, but still no luck. We traced it back to the circuit board and well, for 2 lights not to work, I can live with that for now. The cost of a new board or repair can be up to $200.
But it was a great experience to learn more, see new ways to troubleshoot the machine and do stuff I would never do to a chip board. Surprised it didn't snap. But he knew what he was doing. And that is what you want from your experts. That ability to take something not seen or experienced before and work your way to the level of the issue. And then know how to work around it or fix it.
It is what I do for Lotus products and various other software programs and other technology, both new and old. It is why people pay us and come to us for help for our expertise.
While I was with him, someone else called me and needed me to come out to their house and resolve some wireless issues. The 2 of us talked about this as well and appreciated each others worlds.
When he left I said hope I don't see you again soon, although I do look forward to the next visit.
If you need Fred and are in South Florida, here is his website, http://www.homeusepinball.com and phone number (561) 683-5893. Thanks again for the lessons and my kids appreciate it even more.
Wednesday, January 18, 2012
On Becoming and Training Troubleshooters
To many of us at Lotusphere, solving problems is what we do.
Having always enjoyed taking things apart and putting them back together again or fixing them is part of an innate curiosity that many of us share. The ability to learn from each other and share the information we find so the next person can benefit is part of the Lotus way.
What about the next generation of techies or Business Partners? Do they have this ability? Has Google made it too easy for people? What about your kids? Do your kids wait for you to come home to fix something or do they go out and get it done? What are we doing to not only help them but encourage them to do this?
Over the last few days here at Lotusphere a number of conversations have been about how we got started. Back in the old days, 1980's for me, there were no great books or Google on some topics and training was not really there so when I started out on networks they gave me some servers and said here are the diskettes and go build it. Also screenshot it and document the process for others to follow. Good luck.
Well, I did it, and like most of us blew up the motherboard on the first server I touched when I plugged in the wrong RJ45 line or patch cable. Along the way learned, first hand, about networks, topologies, routers, hubs, drivers, creating network protocols and memory management. These were the days when 640K was still a barrier and QEMM would eventually come along to help with this bit, but I digress.
File servers, access, modems, Digiboards, security, standards, diagrams and other pieces of the puzzle were always new and interesting.
Now much of that need to know information is no longer required. TCP/IP and Ethernet became the standards as did SMTP and so many other things which back then we had to learn and understand to get our work done. Now your average cell phone does so much, yet many understand so little about them and what they do. Maybe this is how everything works. Commerce started out as a barter before a standard method of payment existed and now credit cards are our money. Digital transfers no one thinks about, except for those of us building the infrastructure. Your average person on the street has no idea why their cell phone works or not or how it works and blames the carrier when it drops calls even if the reality may be you just lost coverage in between towers.
The students GBS brought to Lotusphere, kudos to GBS for doing it again this year, I hope, learned from all of us some pieces of the ideas mentioned above because it seemed like they do not grasp, as one example, the concept that security is a multi layer function or how or why it is important in business. No doubt many of them are or will be great developers, but I hope their professors and teachers also explain the ecosystems which are needed, not just to run the code, but how it interacts with other pieces.
I have been walking around with one of my Moo Cards on my badge that says "I Troubleshoot Anything". It has brought me questions about kids, spouses, cars and more. I am not an expert on every topic or product of IBM's but that innate ability to look at the problem from many angles and narrow down the issue is something I have been noticing in the workplace is missing from many IT personnel. On the flip side it is why ISSL and other Business Partners, Vendors and organizations seek me and you out for work. Because when something needs to get done and there are problems, they need someone who truly believes there is an answer to the problem and will just get to work.
"I don't know that product" or "it's not my scope" or "I only work on product X" or "I never got trained on it" are some of what I hear and see and this will lead to many issues in your organization if you are not careful. It's okay to not know something. Really. It..is..okay. Ask, learn, Google it or sit in font of a box and play with it till you get it. Instead of training on a new product's usage and some error messages, provide broken installations, down servers, screwed up Firewalls, reverse proxies and other real life scenarios to help the trainees enjoy training again and feel better about themselves.
No, I am not recommending you start performing neurosurgery or say you can do it. I am advocating that to advance yourself you need to look into the unknown and encourage yourself or your teams to do the same. Do not let them get away with being lazyMicrosoft paper engineers. Don't kill them either although you may want to do so at times.
To everyone at Lotusphere or around the world keep up the good work, keep learning and solving the little and the big problems. And don't forget to write a wiki, document, white paper, book, blog post or do a video so others can learn and benefit as well.
Having always enjoyed taking things apart and putting them back together again or fixing them is part of an innate curiosity that many of us share. The ability to learn from each other and share the information we find so the next person can benefit is part of the Lotus way.
What about the next generation of techies or Business Partners? Do they have this ability? Has Google made it too easy for people? What about your kids? Do your kids wait for you to come home to fix something or do they go out and get it done? What are we doing to not only help them but encourage them to do this?
Over the last few days here at Lotusphere a number of conversations have been about how we got started. Back in the old days, 1980's for me, there were no great books or Google on some topics and training was not really there so when I started out on networks they gave me some servers and said here are the diskettes and go build it. Also screenshot it and document the process for others to follow. Good luck.
Well, I did it, and like most of us blew up the motherboard on the first server I touched when I plugged in the wrong RJ45 line or patch cable. Along the way learned, first hand, about networks, topologies, routers, hubs, drivers, creating network protocols and memory management. These were the days when 640K was still a barrier and QEMM would eventually come along to help with this bit, but I digress.
File servers, access, modems, Digiboards, security, standards, diagrams and other pieces of the puzzle were always new and interesting.
Now much of that need to know information is no longer required. TCP/IP and Ethernet became the standards as did SMTP and so many other things which back then we had to learn and understand to get our work done. Now your average cell phone does so much, yet many understand so little about them and what they do. Maybe this is how everything works. Commerce started out as a barter before a standard method of payment existed and now credit cards are our money. Digital transfers no one thinks about, except for those of us building the infrastructure. Your average person on the street has no idea why their cell phone works or not or how it works and blames the carrier when it drops calls even if the reality may be you just lost coverage in between towers.
The students GBS brought to Lotusphere, kudos to GBS for doing it again this year, I hope, learned from all of us some pieces of the ideas mentioned above because it seemed like they do not grasp, as one example, the concept that security is a multi layer function or how or why it is important in business. No doubt many of them are or will be great developers, but I hope their professors and teachers also explain the ecosystems which are needed, not just to run the code, but how it interacts with other pieces.
I have been walking around with one of my Moo Cards on my badge that says "I Troubleshoot Anything". It has brought me questions about kids, spouses, cars and more. I am not an expert on every topic or product of IBM's but that innate ability to look at the problem from many angles and narrow down the issue is something I have been noticing in the workplace is missing from many IT personnel. On the flip side it is why ISSL and other Business Partners, Vendors and organizations seek me and you out for work. Because when something needs to get done and there are problems, they need someone who truly believes there is an answer to the problem and will just get to work.
"I don't know that product" or "it's not my scope" or "I only work on product X" or "I never got trained on it" are some of what I hear and see and this will lead to many issues in your organization if you are not careful. It's okay to not know something. Really. It..is..okay. Ask, learn, Google it or sit in font of a box and play with it till you get it. Instead of training on a new product's usage and some error messages, provide broken installations, down servers, screwed up Firewalls, reverse proxies and other real life scenarios to help the trainees enjoy training again and feel better about themselves.
No, I am not recommending you start performing neurosurgery or say you can do it. I am advocating that to advance yourself you need to look into the unknown and encourage yourself or your teams to do the same. Do not let them get away with being lazy
To everyone at Lotusphere or around the world keep up the good work, keep learning and solving the little and the big problems. And don't forget to write a wiki, document, white paper, book, blog post or do a video so others can learn and benefit as well.
Tuesday, January 10, 2012
TDI Troubleshooting Access Rights
Been working on a TDI project for the last few weeks and it looked to be almost done, then everything stopped working.
So, what to do? Asked around but did not find any great answers, just vague ideas or theories but nothing to point at and go AHA! So I did what I advocate and called in a PMR to IBM when nothing or no one else could help efficiently. Seems everyone is busy trying to finish work before Lotusphere.
Brian H. from support was helpful in pointing me in the right direction although there is a lack of info surrounding the problem experienced.
What we found was we suddenly had messages similar to this:
Checking under the logs directory under my TDISOL directory the ibmdi.log showed some errors but nothing that pointed to anything directly.
So off we went to find how to get more logging details. Knowing there is a log4J reference meant finding out how and where to set logging and what levels it has.
The directory also under TDISOL called etc is where you can find the log4j.properties file. In it you will find a line that says:
In our case it showed some jar files and other pieces could not be read, from the directory one ABOVE TDISOL which is called V70A on our box and in the jars directory, and it's sub directories(V70a/jars/connectors and V70A/jars/3rdparty) among them.. We asked for more access to cover the whole directory and all ran properly.
So it was an access issue and not a code problem, unlike my NULL post which is still testing before I post the conclusions.
So, what to do? Asked around but did not find any great answers, just vague ideas or theories but nothing to point at and go AHA! So I did what I advocate and called in a PMR to IBM when nothing or no one else could help efficiently. Seems everyone is busy trying to finish work before Lotusphere.
Brian H. from support was helpful in pointing me in the right direction although there is a lack of info surrounding the problem experienced.
What we found was we suddenly had messages similar to this:
Log4J: Error Either File or DatePattern options are not set for appender [null]Descriptive. I know.
Checking under the logs directory under my TDISOL directory the ibmdi.log showed some errors but nothing that pointed to anything directly.
So off we went to find how to get more logging details. Knowing there is a log4J reference meant finding out how and where to set logging and what levels it has.
The directory also under TDISOL called etc is where you can find the log4j.properties file. In it you will find a line that says:
log4j.logger.com.ibm.di.config=WARNEdit the WARN to ALL if you are having problems to get a very detailed log, save and run your scripts. More details on the options can be found in this Wiki http://www-10.lotus.com/ldd/lcwiki.nsf/dx/Troubleshooting_problems_with_Tivoli_Directory_Integrator_ic301
log4j.logger.com.ibm.di.loader=WARN
In our case it showed some jar files and other pieces could not be read, from the directory one ABOVE TDISOL which is called V70A on our box and in the jars directory, and it's sub directories(V70a/jars/connectors and V70A/jars/3rdparty) among them.. We asked for more access to cover the whole directory and all ran properly.
So it was an access issue and not a code problem, unlike my NULL post which is still testing before I post the conclusions.
Monday, March 15, 2010
Webmail Redirector #4 - Unable to Login Issue
I posted the other day that I was unable to login for iNotes for some unknown reason.
While that specific issue I did not test on THAT server yet, a different one has occurred and since it is all part of my session at The View conference, might as well help some poor admin pulling their hair out with what I learned, the hard way, and from ignoring my own notes in one case.
So no matter how I tried I could not login. I searched the web, I Googled till I could Google no more, IBM support toolbar probably hates me from asking so many variations of the same bad error message. Namely the infamous:
But I did not enter an invalid name or password.
Here in no specific order are what you need to do, take a deep breath, we will fix it! And if not, you know how to find me as many already have on this topic. If you have any other pointers please advise as well in the comments. Two paths for you to follow, specific users unable to login and ALL users unable to login.
If a Specific user:
1) Obvious, but check the mail file(if it is a specific user), that their administration server exists or is the proper one. In a demo environment like mine right now, it's easy to not pay attention to this. For a different but same conclusion, read this from David.
2) Replace the design on the mail file to make sure it is the correct version. Again in my demo environment I was using a slightly tainted template.
3) Check the ACL of the mail file that the user did not do something, um, user like, and change all the ACL settings.
4) Make sure the user's name is properly shown as the owner in the preferences section. (More-Preferences from the Inbox view)
5) Check the person document is correct for the server name, file location and name, etc. Mail jump, depending on how you configure it, can be, shall we say unsympathetic to your errors.
6) Change the users internet password...just in case.
When done, do a CTRL-SHIFT-F9 on the names.nsf and try again.
If no one can login:
1) Many places to check but let's start with the key databases. Did you set up the mail jump properly? Did you set up the Domino Web Services Configuration properly? Check the ACL's on both of those, set Anonymous to reader.
2) If you have many websites on the server or many services(like Traveler which is most common), check the Internet sites documents are set up for proper redirection or Override Session authentication. See this subsection from an Excellent IBM Technote, titled "Configuring IBM Lotus Notes Traveler 8.0.1.x and Domino Web Services on the same server". technote # 1298016
3) Check that the security sections in the Internet Sites Document shows this as well:
4) Make sure proper fully qualified name and/or hostnames or IP addresses are stored in the proper place of the Internet Site Document Basics tab under "Host names or addresses mapped to this site:"
5) This wiki page might help if you have an HTTP Authentication issue preventing people from completing their login.
6) Lastly if you did follow my previous posts and edited your login forms, try reverting it back to a basic login and see if that works.
Now I know some of you reading this are saying "Of course, who wouldn't think to do that"? The problem is not who wouldn't think of it, but who wouldn't stop to check they spelled something incorrectly or the IP/name was incorrect. Years of experience tell me all the time that what should work and doesn't is probably my fault, especially if it works everywhere else I set it up.
So the moral of the story is what you learn in a demo(or in my case what I am playing with for a demo) and what you configure in real life do not always equal perfectly. Different versions, even point releases can have major differences and while I and other bloggers, even the IBM Infocenters and wiki's, try to keep our posts up to date with changes in the software, don't rely only on what you read, your intuition will help you decide as well.
Take a vacation after this, you'll need it.
While that specific issue I did not test on THAT server yet, a different one has occurred and since it is all part of my session at The View conference, might as well help some poor admin pulling their hair out with what I learned, the hard way, and from ignoring my own notes in one case.
So no matter how I tried I could not login. I searched the web, I Googled till I could Google no more, IBM support toolbar probably hates me from asking so many variations of the same bad error message. Namely the infamous:
you provided an invalid user name and password
But I did not enter an invalid name or password.
Here in no specific order are what you need to do, take a deep breath, we will fix it! And if not, you know how to find me as many already have on this topic. If you have any other pointers please advise as well in the comments. Two paths for you to follow, specific users unable to login and ALL users unable to login.
If a Specific user:
1) Obvious, but check the mail file(if it is a specific user), that their administration server exists or is the proper one. In a demo environment like mine right now, it's easy to not pay attention to this. For a different but same conclusion, read this from David.
2) Replace the design on the mail file to make sure it is the correct version. Again in my demo environment I was using a slightly tainted template.
3) Check the ACL of the mail file that the user did not do something, um, user like, and change all the ACL settings.
4) Make sure the user's name is properly shown as the owner in the preferences section. (More-Preferences from the Inbox view)
5) Check the person document is correct for the server name, file location and name, etc. Mail jump, depending on how you configure it, can be, shall we say unsympathetic to your errors.
6) Change the users internet password...just in case.
When done, do a CTRL-SHIFT-F9 on the names.nsf and try again.
If no one can login:
1) Many places to check but let's start with the key databases. Did you set up the mail jump properly? Did you set up the Domino Web Services Configuration properly? Check the ACL's on both of those, set Anonymous to reader.
2) If you have many websites on the server or many services(like Traveler which is most common), check the Internet sites documents are set up for proper redirection or Override Session authentication. See this subsection from an Excellent IBM Technote, titled "Configuring IBM Lotus Notes Traveler 8.0.1.x and Domino Web Services on the same server". technote # 1298016
IBM Lotus Notes Traveler and Domino Web Access (DWA)
After Lotus Notes Traveler has been installed and configured, DWA is still functional, however with Internet Sites enabled and Single Server or Multiple Server specified for Session Authentication, the default DWA login mechanism is changed from a 401 challenge to an HTTP login form. Many devices and web clients are not equipped to handle the HTTP login form. Follow these steps to revert back to the 401 challenge.
* Once Lotus Notes Traveler is installed and configured open the Domino Administrator Client and connect to the Domino Server
* Navigate to the Configuration Tab --> Web --> Internet Sites
* Locate and open the Internet Site document entitled "IBM Lotus Notes Traveler Web" or your equivalent Internet Site document for Web (HTTP) protocol.
* Choose the Web Site button and select Create Rule.
* Enter these values for the rule:
* Description: DWA Rule (any value here is fine)
* Type of rule: Override Session Authentication
* Incoming URL pattern: /mail*
Note: Choose the URL pattern appropriate for your organization. The default URL for DWA is /mail/username.nsf so in general /mail* will work fine.
* Save and Close the rule.
* Restart the HTTP Server.
3) Check that the security sections in the Internet Sites Document shows this as well:
4) Make sure proper fully qualified name and/or hostnames or IP addresses are stored in the proper place of the Internet Site Document Basics tab under "Host names or addresses mapped to this site:"
5) This wiki page might help if you have an HTTP Authentication issue preventing people from completing their login.
6) Lastly if you did follow my previous posts and edited your login forms, try reverting it back to a basic login and see if that works.
Now I know some of you reading this are saying "Of course, who wouldn't think to do that"? The problem is not who wouldn't think of it, but who wouldn't stop to check they spelled something incorrectly or the IP/name was incorrect. Years of experience tell me all the time that what should work and doesn't is probably my fault, especially if it works everywhere else I set it up.
So the moral of the story is what you learn in a demo(or in my case what I am playing with for a demo) and what you configure in real life do not always equal perfectly. Different versions, even point releases can have major differences and while I and other bloggers, even the IBM Infocenters and wiki's, try to keep our posts up to date with changes in the software, don't rely only on what you read, your intuition will help you decide as well.
Take a vacation after this, you'll need it.
Monday, August 31, 2009
TroubleShooting Post #1
Discussed at MWLUG and although some numbers were stated, my own background/experience provided the simplest and most agreeable answer.
2
Hours that is. If you are stuck on a problem in your server, environment, car, whatever, don't spend more than 2 hours on it trying and retrying the same things.
Walk away. Forget about it. Get something to eat, go have a drink, just drop it. sleep on it.
It will come to you usually within an hour.
The subconscious is smarter than we think and is always working on what causes us problems.
Caution: Some people this will not help because they don't have the proper knowledge, but presuming you do, this should work for you.
2
Hours that is. If you are stuck on a problem in your server, environment, car, whatever, don't spend more than 2 hours on it trying and retrying the same things.
Walk away. Forget about it. Get something to eat, go have a drink, just drop it. sleep on it.
It will come to you usually within an hour.
The subconscious is smarter than we think and is always working on what causes us problems.
Caution: Some people this will not help because they don't have the proper knowledge, but presuming you do, this should work for you.
Thursday, December 4, 2008
Lotus/IBM Support Something you should know
Note to IBM, make the Support teams have some logo, nothing on the IBM site for it that i could find. Even an orange circle with an H inside would be nice.
Many years ago they were the best when you needed help. Then like many organizations do they had some trouble spots along the way, but I am happy to say they are back in full swing and really worth their money.
But you should remember they are just like you, and you may have been one of them once, or still are in some ways, so be kind, be nice, help them too if you can and listen to their ideas, especially if you are out of them.
1st thing to know is always have IBM Support on speed dial.
2nd thing is DO NOT HARASS them until you have done at least 5 things first on your own to trouble shoot the problem, otherwise they will probably make fun of you in their weekly meetings.
In general if you can't come up with 5 things to check for any problem, I teach classes in general troubleshooting and of course specifically about Domino too when asked. Yes, pinging someone is preferable over the lazy web way of twittering for an answer. We are all here to help, just ask us.
Seriously though Lotus Support is very good these days and better at raising a flag to the developers if necessary so go ahead try them again. I am sure they don't make fun of you, well, okay, maybe some of you, even me!. You know sometimes your brain goes fuzzy, too many things on your plate, the game is on in 45 minutes, significant other is ....you know we all have reasons that we need an answer NOW.
Here are basic guidelines:
NEVER, EVER call and say it's a Level 1 unless your server is really DOA, finished, game over. It's just not fair to scream fire all the time.
Use level 2 for anything else that is amiss or odd, except questions.
Questions of general or what if or how about are Level 3. For instance does R8 accept importing from Excel instead of a wk3 spreadsheet. No it doesn't, stop bothering them.
And with that this PSA is over and I can go back to building demo's for Lotusphere.
Subscribe to:
Posts (Atom)