Thought Leaders

Gene Schultz, CTO of High Tower

Stephen Northcutt - April 4th, 2008

The Security Laboratory is pleased to interview Dr. Gene Schultz, one of the most experienced security practitioners in the field, and we certainly thank him for his time.

Gene, my experience with audit, and log collections begins with something you said in one of your security classes back in 1997. You said, at least turn on logging so that, if something ever happens, at least you have somewhere to start looking for answers. I still remember and live by that advice and it has helped me more than once. If you were giving advice to the home user / small office user related to logs, what would that advice be today?

In today's Windows systems, security logging is enabled by default, so the situation is now somewhat different in that users simply need to leave security logging on. Those who feel they need more logging can enable more Event Categories, because only a few Event Categories are by default enabled. Additionally, if users deploy personal firewalls, which have become essential in achieving defense in depth for PCs, they will also be able to obtain additional log data, as well as alerts that can help them learn about attacks that have occurred against their systems.

OK, and what is your basic advice for the corporate world, I promise we will drill down into the details later!

In the corporate arena, some minimal level of logging also needs to be enabled, and additional log data from personal firewalls also needs to be collected. In this arena, however, automated log data aggregation and correlation, the kind of functions that Security Information and Event Management (SIEM) tools perform, are a necessity. Trying to manually access and analyze log data from a plethora of PCs (as well as other types of hosts) is too unwieldy and costly a task for technical staff--automation is necessary.

Thank you for that. I remember your classes, you were a great instructor, are you still teaching today? What is your favorite venue?

I still teach quite a bit. I teach courses on intrusion detection and prevention, incident response and forensics, Windows security, and Unix and Linux security, as well as the Certified Information Security Manager (CISM) exam prep course.

Nice! And, I know you are a writing machine with over a hundred papers and multiple books, what is your main focus in writing at the present?

My blog site is my main current emphasis.[1] I write a minimum of two blog entries every week, covering a very wide range of issues. I also still periodically write papers for various journals, such as Computer Fraud and Security and the ISSA Journal.

I enjoy your blog, thanks for doing that, I particularly enjoyed http://www.high-tower.com/blogs/gschultz/wikileaks-pandoras-box-opened/ that was nicely balanced. I remember hearing that you are on the advisory board for Secure Defenses and you have a senior position at High Tower; will you share with our readers where else you are involved in governance?

Nowhere else. Just keeping up with the security issues at my company, High Tower, keeps me plenty busy with respect to governance issues. We eat our own home cooking, the things I talk about when I give lectures on governance are the things we do at High Tower.

Awesome, so you really do have a voice that should be listened to in our industry, so let's drill down shall we? Question number one, is the log analysis space really a separate space from the SIEM space? A lot of confusion still floats over this question, and I get asked about this a lot; what is your take?

The answer is, it depends. In a SIEM tool with full functionality, log analysis and the analysis performed by the tool are essentially the same. Both utilize event correlation algorithms to maximize correct detections and minimize false alarms. But some SIEM tools are really nothing more than log aggregators; they perform little, if any, log analysis.

What do you think of the current state of the log collection and management industry from a detection point of view? I keep asking customers to give me examples of where these systems are successfully detecting security incidents and the war stories are few and far between. On the other hand, finding operational problems seems to be a bit easier. What is your take on and what advice do you have for using these devices to collect actionable security information you might not otherwise receive?

It depends on the tool. I, of course, am biased toward the High Tower Cinxi SIEM appliance in which log analysis is based on sophisticated attack models. The MetaRules in this appliance fire wherever a combination of time sequence-dependent events occur, not simply when an intrusion detection system (IDS) reports that an attack signature has been matched (although an IDS alert based on a signature can be one of the chain of events).

Whoa there Gene, what do you mean by MetaRules, can you break that down for us?

A MetaRule is a special term that High Tower uses for its rules, which is pattern analysis based on chains of events that occur when real-world attacks occur. They are called “MetaRules” because they represent high-level abstract logic, as opposed to individual indicators such as simple intrusion detection signatures (although a signature can comprise a part of a logic chain). As an example, MetaRules don't look at specific protocols, but MetaRule 24 looks for any clear text protocols. If we see an incoming clear text protocol such as telnet or FTP, and it stimulates an encrypted reply, there is a very good chance an attacker has created a reverse shell.

OK, thanks for doing that, can you give me an example of a real world detection using Cinxi?

In one case, a High Tower Cinxi appliance was installed at a university and then the configuration of a number of hosts and devices was changed to send syslog and other data to this box. Within five minutes, the appliance reported that a number of Linux hosts in a particular subnet were giving indications that they had been compromised on the basis that they had been launching subtle probes against other internal systems and these hosts were listening on a suspicious port. By going to log data available before the Cinxi appliance was installed, a system administrator found that one host had originally been compromised to allow the attacker to gain unprivileged remote access, and then, shortly afterwards, the attacker gained root access. An ssh agent was running on the compromised system; the attack quickly used it to gain unauthorized root access to other hosts. Administrators had not previously noticed anything out of the ordinary. Once they understood the pattern of attacks, they found evidence that other hosts had been compromised by the same basic type of attack in parts of the network in which hosts did not send syslog output to the Cinxi appliance. Some other SIEM tools are capable of doing the same type of thing, but some of the tools are not really all so proficient in detecting attacks because their rules are based solely on intrusion detection signatures or, in some cases, rather esoteric attack taxonomies rather than models of how real-life attacks occur. In this case, these tools are much more likely to detect malfunctions such as misconfigured routers spraying packets across the network than bona fide attacks.

Thanks Gene, can we get another real world story?

Sure. The Cinxi appliance reported that a Windows host connected to an Internet website then, shortly afterwards, suddenly began sending large amounts of email to a variety of internal and external hosts. A subsequent investigation revealed that the host had visited a malicious website that injected an Active X executable into Internet Explorer via a cross site scripting vulnerability which caused the host to install a mail server and then send messages advertising the URL of the malicious site to addresses in the host’s address book. They noticed this behavior quickly because it is abnormal for a computer to visit a website and then start spewing email to a large number of systems. The MetaRule would have fired if it had not been mail, it would have fired if it was a scan as well. The pattern is visit a website and suddenly make lots of connections. This demonstrates the advantage of event correlation.

This is very helpful Gene, and it does show the importance of event correlation; what if that was a rootkit that got installed? That makes event correlation even more important, yes?

If that was a rootkit, Stephen, antivirus would not have found the malware; it has control of the kernel. Event correlation may be all that you have, though there are some new technologies starting to show up.

What about the BMC or Service Host, they are connected to the network, but they are not the main CPU. If we do not have event correlation, will we know what happens?

One thing that can be done, Stephen, is a dedicated CPU to scan systems: it connects to them and scans them. This is the new technology I was talking about; for instance, Copilot, is a PCI card with a CPU. You may have read the Usenix paper.[2] So you plug it in and it has its own CPU and its own memory and it can connect to the memory of the subverted machine. Right now they are pricey, but I think the cost will go down and the features will go up.

What is the impact of virtualization on SIEMs and security in general?

I am not sure, I think that will be the next level of challenge for these products. Virtualization is a double edge sword. Some people say we are doomed, the red pill - blue pill residing in the virtual environment, and all that. But, if you think about it, virtualization hasn't employed a lot of security. That needs to change and I think that will be a growth area for the industry. From a SIEM perspective, we simply need to learn the signs that a virtual environment has been compromised. Also, vendors can put markers in place, essentially tripwires, so that if code is running where it does not belong, that can be detected. Of course, the attackers will learn what the tripwires are, but it is a cat and mouse game. Right now, I would say the attackers that write rootkits for virtual environments have the advantage, but that will not always be so.

Very helpful! SIEMs are pricey beasts, I notice the Price to Earnings ratio of some of the SAN companies is fairly high so they must be loving the installation of SIEM and log analysis. What are your tips for acquiring and implementing a SIEM to get the best bang for your bucks?

Although some SIEM products are rather pricey, with starting prices well over $100K, and then installation costs that can easily match the purchase price, some are not. It is possible to get a good commercial SIEM tool for quite a bit less, so it is important to shop around. This having been said, the most direct value of SIEM technology is reduction of manpower costs--costs associated with combing through logs, line-by-line. So an extremely important criterion is the degree to which each SIEM tool under consideration streamlines and simplifies the log analysis process. Similarly, compliance considerations are becoming extremely important in information security. SIEM technology can help not only by archiving log data, but also by providing reporting functionality that shows that the organization that deploys this technology is being compliant with regulation provisions that require certain types of monitoring, or that external traffic is not getting through to internal hosts that process and store credit card data, or that 24 X 7 monitoring of critical business systems is occurring, or that vulnerability scanning has been occurring. So, another important criterion is how good each SIEM tool's compliance reporting function is; the better it is, the greater the cost effectiveness. Additionally, installation costs can be downright outrageous; more cost effective SIEM tools can be installed and deployed quickly. Other criteria such as performance (because SIEM tools that are slow experience memory full conditions that cause them to miss events or even crash), ease and reliability of archiving the data that are collected, user friendliness, and many other variables are also strongly related to cost effectiveness.

Where do you see this space in the next two years?

I'm not being very original here by saying that SIEM technology is already becoming mainstream in information security practices, and in two years it will be even more so. The Gartner Group, in fact, sees SIEM technology growing to a $10 billion industry in just two years. There are three main reasons: 1) substantial reduction of time and labor costs, 2) good SIEM technology makes compliance much easier and more efficient, and 3) it takes intrusion detection up to the next level by automating what outstanding intrusion detection analysts have done for years--performing pattern analysis on intrusion detection data based on knowledge of how real-life attacks work.

Gene, one of my concerns is that as a community we are losing the ability to detect attacks. My opinion is part of this is grew from when we all responded to Gartner's "Intrusion detection is dead" paper by implementing Intrusion Prevention Systems, we seemed to believe we could leave the detection to the system, not the human analyst. So it is great to see you mention intrusion detection several times; can you please expand, and be Cinxi specific, on how this helps us take intrusion detection to the next level. I know you touched on this with the first war story; can you take us into the technical weeds with an example of an attack that is hard to write a Snort rule for, and that you can help us detect?

I certainly disagreed with Gartner’s analysis and suggestion that IDS was dead. Intrusion detection is very much alive and well, and is still the foundation of intrusion prevention; prevention mechanisms cannot be unleashed unless an attack has been discovered, and discovery is the role of intrusion detection. Interestingly, many organizations that buy intrusion prevention tools leave them in intrusion detection rather than intrusion prevention mode. That said, the application of intrusion detection information may need to change, and leveraging the data for event correlation and alert fusion are two areas in which considerable progress has been made. The High Tower Cinxi appliance has event correlation logic that enables it to report chains of events that represent attacks that IDSs such as Snort will usually miss. An example is a telnet or other cleartext connection from an external to an internal IP address, a routine event in organizations such as universities and research organizations that usually allow certain "dangerous" protocols for the sake of free and open communication. If the internal host creates an encrypted connection back to the external host afterwards, IDSs will also overlook this event. But with MetaRule logic, the first event combined with the second, represents an attack because it has all the characteristics of an attacker initiating a reverse shell connection.

The work of Dr. Matt Bishop and his colleagues at the University of California at Davis has shown that ability to produce fused alerts, i.e., single alerts representing sets of highly interrelated actions by an attacker (something that is highly desirable from the perspective of an intrusion detection analyst), can be enhanced considerably by analyzing the capabilities of each attacker. Some attackers engage in only very elementary actions, whereas others are capable of very sophisticated ones. Knowing that actions are occurring and linking actions with capabilities of attackers allow the creation of mathematical models that can identify multi-stage attacks that mainstream IDSs would be likely to overlook.

Thanks, I appreciate that! You mention reduction of time and labor costs. Is that really correct? Where do the savings come from? I am a bit gun shy; I still remember implementing HP Openview and thinking I could operate it with two FTEs, then it ended up needing four!

Reduction of time and labor costs is not automatic with SIEM technology—it depends on the particular tool in question. For example, some SIEM tools require little time and effort to install and to use, whereas others are nightmares in this regard. But, long after the early phases of deployment are finished, the day-by-day cost savings are realized by avoiding the time and hassle of system, network and security administrators having to read audit log entries line-by-line. Heaven only knows how many hours per day of time (and thus money) is consumed by such activity, activity that is, for the most part, unnecessary given suitable SIEM technology functionality.

What are your thoughts about integrating SIEM with these passive sniffers, P0F, SourceFire RNA, Tenable Passive Sniffer? It seems like an economic way to keep the SIEM information up to date?

The integration with these tools is a great idea, and could be the next evolutionary step for SIEM tools. In fact, High Tower is already modeling advanced parallel computing systems that can deliver the processing speeds necessary to handle so much data. We’re also working on partnerships with other technology companies to explore new applications of the technology. From a security/risk management perspective, the more data available to the SIEM, the more likely it is to identify an attack or other malicious behavior. Attacks continue to grow in complexity and surreptitiousness - becoming ever more difficult to detect and defend, and networks aren’t getting any less complex either. A great SIEM system should be proficient at identifying Zero-day attacks using the type of behavior modeling I discussed earlier. The more information available for analysis, the more capable these systems will become. The trick will be developing the intelligence of the system - so it is able to make use of all the information it receives while becoming less dependent on signatures.

One of the things we like to do in the Security Laboratory Thought Leader series is give people a bully pulpit, a chance to express what is really burning on their heart. What would you like to share with our readers?

My number one passion right now is to evangelize the information security community and senior management concerning the need to embrace and apply the concept of information security governance. Governance is a number of activities that are exercised by high level management. These activities help ensure that strategic objectives of the organization are met. This is management oversight, planning, and evaluating, ensuring that whatever drives the business operations is always considered. You have to work out your own flavor of governance in an organization. What works in government may well not work in the insurance industry, may well not work in a private company. But I will tell you, Stephen, I am the CSO at High Tower and this works great for us. The CEO loves the idea that we have a strategic plan and priority. This filters down to the technical folks, the admins, and they are very thankful to have priority. If we do not give the technical people guidance, they have a very hard time. This concept is the most revolutionary one to emerge from within information security in decades. If understood and applied properly, it results in huge dividends, including (but not limited to) far better risk management and delivery of business-related value.

I realize this is a similar question to when I asked about advice for the corporate world, but if a close friend was taking a job as CSO of a fortune 500, and they already had a SIEM implementation, but it was partial and ongoing, what would be the most important advice you would give her?

Honestly, I'd look very hard at the SIEM tool that this person had purchased and on the basis of the features, functionality and reliability (or lack thereof) of this tool, make a recommendation to either accelerate and complete the SIEM implementation (because of the many benefits of this technology), or to scrap it and start over. I don't pull punches--there are some SIEM tools to which I would not even allocate rack space if they were given to me because they don't at all deliver what SIEM products should (although they can be quite amusing to watch because of all the lights that go on and off and other display gimmickry). Some others are really excellent; if the friend had picked one of these, I would strongly recommend going full speed ahead with the implementation.

Thanks, this has been great and I have really enjoyed it and learned a lot. Just one last question - can you tell us a bit about Dr. Gene Schultz the person? What do you do when you are not in front of a computer?

Well, not flying, that's for sure. I am on an airplane going to some part of the world just about all the time. (Would you like my bonus miles so that you, not I, can fly somewhere else?) On a serious note, I like bicycling, fishing, hiking in the mountains (where my wife and I have a small second home), and fiddling with the two model railroad sets that I have built. Model railroading is probably my biggest after hours passion, but being on travel so much, I do not get nearly as much time to work on my layouts as I would like. Interestingly, I recently read an article about pop singer Rod Stewart, who is also an avid model railroader. He brings kits with him while he is on tour, but has the advantage of owning his own plane, so he can take anything he wants with him. Perhaps I should take a clue from him. *smile*

1. http://www.high-tower.com/blogs/gschultz/
2. http://www.cs.umd.edu/~waa/pubs/USENIX-copilot.pdf