Doing a post mortem this morning on a Domino server that was compromised last night. Wish this had happened a few weeks back when I was preparing my slides for The View conference for my session on breaking into Domino.
This was not a major scale customer, more SMB size and a little looser with their firewall and security appliances. But a good lesson to everyone.
While it looks like a purely spam attack effort, from the outside, the server log revealed a stronger LDAP attack.
Let's start with the basic details, in US Eastern time.
This was the first of 100's if not 1,000's of LDAP requests sent during the night. the account used to authenticate LDAP lookups is unique and has a password which should be okay. I say should be because who knows these days. Next we saw this:
What we found was the SHORTNAME was from a demo account. You can guess the password it had probably, evidently so could the SPAMMERS. This explains the SPAM that was sent out over the night. Every admins nightmare, demo accounts that don't get turned off or deleted. And of course once one is compromised, social media or collaboration takes off and every BOT tries to get in:
These lasted until 3:52AM so about 25 minutes or so before it stopped. Yuck indeed.
We found the Names.nsf had Author access. So conceivably, if the demo account had Admin rights, this could very well have been even worse.
What do you do? Where to Start. Would you even know if any of this happened last night? While some of you go run to open your log files I will wait a minute.
Now that you have returned and hopefully not found any of these issues, let's review how I was alerted and what you may want to think about on your end as well.
First, DDM, Domino Domain Monitoring is YOUR FRIEND, USE IT. Needless to say I received tons of emails alerting me to a problem. Granted at 3AM I am sleeping and so when I got up this morning, the fun began.
Why did DDM send me emails? Because we told it to. Yes, they can be annoying, yes at times it seems, like today, they will never stop. BUT which would you rather be? Ignorant of your security breaches? I didn't think so.
Open the DDM database on your server. Sort by date and you hopefully should see few entries if any, if you are on top of the environment. If you are not, well, get working, you have issues. Now we do not generally care about informational items like if a user was not found in the Directory type items. We care about fatal warnings, failures and Warning High items. Surprisingly enough this line:
You make this change in severity type by opening the item from the list in DDM.
Next find the Severity and type line and at the end of the warning is a DOC LINK, click on it to go to the events4.nsf Database where you click on Edit Document.
You will now see Event Severity have a drop down. Set this to whatever level you prefer, we usually set it to Warning High or Failure.
Save and close.
This presumed you already had set up your notifications.
If you have not, shame on you if you are an admin, do this:
From the DDM database select Open Configuration.
This opens the events4.nsf database and you need to find Event Handlers, then By Server.
Click on New Event Handler.
Select the server(s) to monitor and a trigger event, but for this just use any event.
Click on Event and under Events must be this type, select Security.
Select if you want specific messages or all severity and leave Events can have any message checked.
Click on Action and Method, We use email but you have many choices for notifications.
Enter an email address and enable the notification.
Done
So what else do we do? We changed the account password and banished the account to the Deny Access Group because we want to see if they come back and if there are other accounts we are missing.
Make sure to check each item in the DDM and if the level of severity is not high enough for you, go raise it and make sure a new notifier is built around it.
Could this be handled in other ways. Yes, We have changed some SMTP settings to block the domains that the SPAMMERS were using.
In this case, because it was a real account that the password was guessed, an appliance could have seen masses of emails and shut it down. Beyond that it could be a gaping hole in some environments. Especially you that leave LDAP anonymous access.
This was not a major scale customer, more SMB size and a little looser with their firewall and security appliances. But a good lesson to everyone.
While it looks like a purely spam attack effort, from the outside, the server log revealed a stronger LDAP attack.
Let's start with the basic details, in US Eastern time.
05/18/2011 03:24:21 AM LDAP Server: Bind request for CN=LDAP USER,O=ORG failed: Invalid credentials specified: failed to authenticate
This was the first of 100's if not 1,000's of LDAP requests sent during the night. the account used to authenticate LDAP lookups is unique and has a password which should be okay. I say should be because who knows these days. Next we saw this:
05/18/2011 03:24:27 AM SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 24.237.115.235
05/18/2011 03:24:33 AM Router: No messages transferred to MAIL.RU (host mxs.MAIL.RU) via SMTP
What we found was the SHORTNAME was from a demo account. You can guess the password it had probably, evidently so could the SPAMMERS. This explains the SPAM that was sent out over the night. Every admins nightmare, demo accounts that don't get turned off or deleted. And of course once one is compromised, social media or collaboration takes off and every BOT tries to get in:
05/18/2011 03:25:24 AM SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 71.80.203.51
05/18/2011 03:25:24 AM SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 98.251.18.30
05/18/2011 03:25:25 AM SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 24.2.139.106
05/18/2011 03:25:25 AM SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 24.237.115.235
05/18/2011 03:25:25 AM SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 76.1.160.187
05/18/2011 03:25:27 AM SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 223.130.33.225
05/18/2011 03:25:30 AM SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 203.169.117.225
05/18/2011 03:25:29 AM SMTP Server: Authentication succeeded for user SHORTNAME ; connecting host 116.14.38.190
These lasted until 3:52AM so about 25 minutes or so before it stopped. Yuck indeed.
We found the Names.nsf had Author access. So conceivably, if the demo account had Admin rights, this could very well have been even worse.
What do you do? Where to Start. Would you even know if any of this happened last night? While some of you go run to open your log files I will wait a minute.
Now that you have returned and hopefully not found any of these issues, let's review how I was alerted and what you may want to think about on your end as well.
First, DDM, Domino Domain Monitoring is YOUR FRIEND, USE IT. Needless to say I received tons of emails alerting me to a problem. Granted at 3AM I am sleeping and so when I got up this morning, the fun began.
Why did DDM send me emails? Because we told it to. Yes, they can be annoying, yes at times it seems, like today, they will never stop. BUT which would you rather be? Ignorant of your security breaches? I didn't think so.
Open the DDM database on your server. Sort by date and you hopefully should see few entries if any, if you are on top of the environment. If you are not, well, get working, you have issues. Now we do not generally care about informational items like if a user was not found in the Directory type items. We care about fatal warnings, failures and Warning High items. Surprisingly enough this line:
nHTTP: SHOTNAME [166.137.8.176] authentication failure using internet passwordis set to be a "Warning Low in Security" by IBM. So by default, in theory, one would not notice it if they only cared about the higher end items. BUT, you can go inside and edit the level of severity which is what we do for this and the LDAP, IMAP, and POP3 entries when they are applicable.
You make this change in severity type by opening the item from the list in DDM.
Next find the Severity and type line and at the end of the warning is a DOC LINK, click on it to go to the events4.nsf Database where you click on Edit Document.
You will now see Event Severity have a drop down. Set this to whatever level you prefer, we usually set it to Warning High or Failure.
Save and close.
This presumed you already had set up your notifications.
If you have not, shame on you if you are an admin, do this:
From the DDM database select Open Configuration.
This opens the events4.nsf database and you need to find Event Handlers, then By Server.
Click on New Event Handler.
Select the server(s) to monitor and a trigger event, but for this just use any event.
Click on Event and under Events must be this type, select Security.
Select if you want specific messages or all severity and leave Events can have any message checked.
Click on Action and Method, We use email but you have many choices for notifications.
Enter an email address and enable the notification.
Done
So what else do we do? We changed the account password and banished the account to the Deny Access Group because we want to see if they come back and if there are other accounts we are missing.
Make sure to check each item in the DDM and if the level of severity is not high enough for you, go raise it and make sure a new notifier is built around it.
Could this be handled in other ways. Yes, We have changed some SMTP settings to block the domains that the SPAMMERS were using.
In this case, because it was a real account that the password was guessed, an appliance could have seen masses of emails and shut it down. Beyond that it could be a gaping hole in some environments. Especially you that leave LDAP anonymous access.
No comments:
Post a Comment