Alarm Overload

While the title of the linked article is clearly link bait, the message is very important. We talk so much about logging everything that we forget that we actually need to pay attention to that stuff. It’s easy to check off the compliance box that says “centralized logging”, but it’s much harder to actually do something useful with that information.

This comes up almost every time I teach a class. If you’re never going to look at it, is it really worth spending all that time and money to build a logging system? You might as well just dump those logs to /dev/null and be done with it. At least that way you won’t look bad when the forensic results show that you missed 60,000 alarms.

Of course, the most prudent solution is a happy medium. Maybe that includes having your own SIEM and staffing it with trained people 24×7. It could be outsourcing the management and monitoring of the system. Regardless, there is a goldmine of valuable information in those logs and alerts, make sure someone is able and willing to take the appropriate action when needed.

“In the case of the BP Oil rig disaster in the Gulf of Mexico in 2010, it was discovered that an alarm to warn of explosive gas has been intentionally disabled. A crash on the Washington Metro system a year earlier, which killed nine people, happened partly because train dispatchers were overwhelmed by extraneous notifications. Similarly, much has been written about hospitals that are grappling with the massive quantities of alarms that are generated by a wide variety of sensors. Hospital alarms are crucial – they provide notifications of a patient’s condition but only a small percentage of such alarms are issued for new, clinically significant changes.”

via How the Target Breach and the Malaysian Flight MH370 Mystery are Related | The State of Security.

Have a Breach? Fire Your Sys Admin

Like all companies that have had a well publicized data breach, the NSA has decided to take very public action in the wake of the Snowden affair. The action it will be taking? Firing all but 10% of their sys admins. Presumably someone looked at this case and determined that since Snowden was a sys admin who had access to a lot of confidential material, the obvious fix was to get rid of those pesky sys admins. I can’t say I’m surprised. Having been a sys admin and a manager of sys admins I know that in many organizations sys admins are viewed as a necessary evil who is one step away from holding the entire organization hostage in order to get a raise.

The NSA is dropping the ball here to say the least. Again, I’m not surprised. It’s way easier to just get rid of a bunch of people than it is to look at processes and policies and figure out what went wrong in the organization. Now the NSA can fire 90% of the sys admins it employs and say they’ve solved the problem. No more breaches! Until the next time. The problem here is not the sys admins, the problem is the way the sys admins are granted access to information within the organization.

When I was a new sys admin, there was nothing better than knowing that I was the one with all the power, the keys to the kingdom. Without me the company would never be able to function. I pretty much had the company convinced of that too. What did that lead to? Being on call 24×7 even on holidays for a start. Not to mention setting myself up for the potential to have access to all sorts of things I had no need to access. As I began to pursue information security as an interest I started to realize what I was doing to myself and the company. No one person should have that kind of access and no one person needs it either.

This is where organizations like the NSA are blind to the real solution. The only reason Edward Snowden was able to access all that information was because someone gave him that access. It was probably because the simple thing to do was give the sys admin access to everything. It’s complicated having to continually grant access when someone needs it (well, not really but people seem to think so). The easy way is to give them everything and hope they don’t do anything bad with it. It’s a symptom of an organization that thinks information security is only about technology. Effectively what the NSA is doing is consolidating all of the control to 10% of its sys admins. That’s not necessarily a bad thing. The problem is that instead of limiting the other 90%, they’re going to fire them and give the remaining admins all the access and all the work. So now you’ll have severely overworked admins who have access to all the information in the organization. Hardly sounds like a recipe for success to me.