Change Controls: Ur Doin It Rong

More details are emerging in the case of Rajendrasinh Makwana, a former consultant at Fannie Mae, who allegedly planted malicious code on Fannie Mae's servers after he had been terminated. If the code had not been detected, it apparently would have destroyed data on a large number of Fannie Mae's servers on January 31st.

There's been a great deal of hand-wringing over the fact that Makwana continued to have sufficient access after he was terminated to allow him to plant the malicious code. Well, let's review the facts as presented by FBI Agent Jessica Nye's affidavit:

"On October 24, 2008 between 1:00 and 1:30pm, MAKWANA was terminated as an employee of [Fannie Mae]... At approximately 4:45pm, MAKWANA dropped of his badge and laptop to a [Fannie Mae] employee... Access to [Fannie Mae's] computers...[was not terminated] until late in the evening on October 24, 2008."

Certainly it's regrettable that Makwana had upwards of 12 hours of access to Fannie Mae's network after he was notified of his termination. But, while I'm not condoning this practice, it's a fact that managing to terminate Makwana's access the same day he left the building puts Fannie Mae in the upper 90th percentile of most IT organizations I've encountered.

The information in the FBI Agent's affidavit that really made me sit up and take notice was the following:

"On October 29, 2008, SK, [a Fannie Mae] senior Unix engineer, discovered malicious script embedded within a pre-existing, legitimate script... It was only by chance that SK scrolled down to the bottom of the legitimate script to discover the malicious script."

In other words, Fannie Mae got very, very lucky here. What I want to know is why Fannie Mae had to trust to luck to detect an attack that (again according to the FBI affidavit), "would have caused millions of dollars of damage and reduced if not shut down operations at [Fannie Mae] for at least a week."

Where the hell are the controls on Fannie Mae's change management process? How is it possible that Makwana was able to modify code that ran on Fannie Mae's production systems without that modification being detected? Having a front-end change control process is nearly useless if you don't have back-end controls to verify that the process is being followed correctly.

Fellow SANS Instructor Randy Marchany is fond of saying, "The time it takes your organization to deal with an incident is equal to the time it takes you to notice the problem plus the time it takes you to react." Well in this case, it was only through sheer luck apparently that Fannie Mae's "time to notice the problem" wasn't infinite. That's unacceptable in a high-performing IT organization. In fact, there's research that shows that effective change management and lack of tolerance for unplanned changes may be the key benchmark for the highest-performing IT organizations. And Fannie Mae clearly isn't there.

And there's such an easy fix for this problem: file integrity assessment tools like Tripwire, AIDE, and Samhain. Yes, there is overhead to using these tools in any IT organization, but it's easy to justify these tools if the alternative is potentially losing millions of dollars and having your operations offline or degraded for a week. That's a catastrophic event that would put many companies out of business.

Many of you reading this blog deal more with the aftermath of failures of IT processes than with establishing those processes themselves. But I think we'd all agree that the best incident response scenario is one where we don't have to respond to an incident at all because the incident wasn't allowed to happen. As the current economic climate leads to more layoffs and a greater potential for malice by disgruntled former employees, it's important that IT organizations look to shoring up the controls that will help them defend their infrastructures against possible threats.

Hal Pomeranz is an independent IT/Computer Security Consultant and a SANS Faculty Fellow. He's extremely happy to report that he has no financial relationships with Fannie Mae.