Last day to save $500 for SANS San Diego 2013

Leigh Purdie, InterSect Alliance, co-founder of Snare

Introduction


Table of Contents

Leigh Purdie, InterSect Alliance, co-founder of Snare

March 7th, 2008
By Stephen Northcutt



Perhaps, one of the hottest topics in 2008 is log file analysis (who would have guessed). And while the commercial tools are getting a lot of the press, an open source, and also commercial tool is ending up on a lot of systems. It is called Snare and Leigh Purdie is the thought leader behind the project. He has been willing to invest the time for a thought leadership interview with the Security Laboratory, and we certainly thank him for his time.


Leigh, how did this all begin? The need for industrial strength logging for Windows is extreme, how did you become the aspirin for this headache?

G'day Stephen, logging certainly has hit the security spotlight in the last few years; it's been a bit of a wild ride. The story of Snare is fairly entwined with that of InterSect Alliance - I don't think either would exist without the other, so I'll give you a bit of a background. Close to 10 years ago, I was a senior IT security specialist at Australia's information security authority, the Defence Signals Directorate. To provide some context for your North American readers, DSD is similar in mission to the NSA in the United States. DSD was a fascinating place to work, with excellent resources, a diverse IT ecosystem, and a culture of security appropriate to the confidentiality of the information it processed.


So, Leigh, I am guessing InterSect Alliance comes up pretty quickly?

Yes - it was at DSD where I met George Cora, the other founder of InterSect Alliance. What drew George and myself together was a shared a drive to make security less of a burden and more of a business asset. In most medium to large organisations at that stage, security was seen as the responsibility of a vague amorphous 'security cell' buried somewhere within the IT area.


I hear you! Marcus Sachs, one of the US cybersecurity czars, once commented that some of us heard the sound of a song (security) and started dancing to it very early on. A lot of people who looked at us funny then, are amazed at our prescience now. So what did you notice specifically?

Absolutely; both George and I got a few confused looks early on in the process when we told people we were interested in working on IT security - roughly the same sort of look you'd probably get if you announced that you were going to take up professional octopus wrestling. The tools just weren't well developed, and the organisational understanding of how to use IT security as not only a 'protection layer', but also as a business-enabling asset, really wasn't there. We both felt (and still do) that the tools to validate and enforce security controls, as much as possible, really need to be in the hands of the people who are responsible for the data. They know how important the data really is, and the people who should (or shouldn't) have access to it. People are familiar, and comfortable with this concept in the world of physical security, where we have layers of protection that narrow as the protected asset becomes more 'personal', or tied to responsibility of the individual. In the IT arena though, the complexity of the technology has forced us down the path of a centralised protection strategy, where the core IT security area has all the responsibility, but often no real feel for the individual security requirements of the underlying data.


Gotcha Leigh! I think of the dynamic tension between a data owner and data custodian very often. I have seen more than one data base administrator cheeky enough to try to hold an entire company hostage and determine who gets coding priority, database access priority - it is not a pretty sight. So what did you notice that was actionable?

True. Unless the general directional controls are in the hands of those that understand the data, you risk your IT cell driving corporate policy. That might work pretty well in some organisations that are very information-driven, or have very flat organisational structures, but in more traditional companies, it can give the CEO heart palpitations. George and I were particularly frustrated with the limitations of the crop of IT security products that were available at the time, as we were dealing day-in, day-out with the requirement to protect top secret, compartmented information for which we had no need (or wish!) to know. In addition, we were trying to cope with the massive quantities of audit log data produced by a high-security organisation, sourced from a raft of different operating systems, applications, and services. The commercial tools available to assist in the process of event log management at the time were generally single-platform, horribly expensive relative to their useful capabilities, and crashed when faced with the sorts of volumes we saw on a daily basis. Products also seemed to fall into the trap of 'fluffing up' their feature-set with overly complex stuff that often produced a flashy demo but was practically useless in an operational setting. It's a little bit like driving a Winnebago down to the shops just to pick up some milk - the extra features often just get in the way of what you are trying to accomplish. Our attempts to gain access to reasonable support also provided a few challenges - we never seemed to be able to get past the 'smiling salesman' layer to the people who could actually solve the problems (no offence intended to sales people). All of these experiences have had a major impact on IA and how the Snare applications have evolved over time.

So, like many IT organisations around the world, we started putting together a system of our own. It needed to cope with a huge quantity of log data, and a diverse range of source systems, and it also needed to produce reports that were optimised for data owners - not necessarily IT professionals; providing data owners with the capability to manage their own security (within clearly delineated boundaries).


OK, so you were in enough pain to start writing code, how did the early days go?

I'm a geek at heart, so it usually doesn't take a lot of pain for me to start coding, but in the context of our other priorities at the time, yes - it was going to be a big investment in time and resources that we didn't want to take on lightly. The system was a little crude behind the scenes, but had a friendly front end, and proved to be quite effective. At that stage, we were relying on automated bulk file transfers from client systems, and were using a loosely integrated collection of scripts to manage the user front end, but several of the key concepts that eventually led to the Snare framework were tried and tested in this sort of operational environment.

George and I eventually left DSD to start our own IT security company, InterSect Alliance. IA started out as a boutique security consulting company, but the problems and opportunities associated with log management continued to be a significant interest area for us. We were also watching the progress of the Linux operating system in the commercial and government market pretty closely. Government agencies, and defence contractors saw Linux as a potentially useful replacement for aging Unix systems, but could not easily get past the requirement for system-level auditing imposed by the US 'C2' security guidelines. During our time at DSD, George and I worked on the document 'ACSI 33', which is vaguely analogous to the C2 guidelines in the US. As a result we were keenly aware of the effect such a document can have on the infrastructure decisions of organisations who have high security requirements.


Oh my, my, my. That is a famous and interesting problem. You started a consulting company that became, to some extent, a software product company. I know that is the story of CORE Security as well, from my thought leadership interview with Ivan Arce.

Our drivers were reasonably similar to Ivan's - the lack of available tools, really pushed him into developing his own. Our customers really hauled us out of our niche when they saw the sorts of tools that we were putting together. The start of this process began with the frustration of our customers (and others) with the lack of an auditing framework for the Linux operating system. That need drove me to create an audit subsystem for Linux. We called it 'Snare' - System iNtrusion Analysis and Reporting Environment (with a slight tip of the hat to George's hobby as a drummer), and released it as open-source, under the GNU Public License.

The response to Snare was amazing. The existence of Snare for Linux, allowed many Linux installations to progress, that had previously been frustrated by the fact that C2 security requirements could not be met. Snare for Linux also included event filtering, in order to curtail the operating system's enthusiastic reporting of information, and could either send the results to a local file, or over the network to a custom UDP port, or via syslog.


So now this is a bit like the start of a ski run down double black diamonds, I bet you were on the edge and could not see what was before you! So, what happened?

As an Aussie, I don't tend to see a lot of snow - our water tends to stay nice and wet, rather than freezing. But yes, it felt a bit like dropping in on a 2 meter shore break wave, which sounds like a similar analogy. *smile*

Emboldened by the success of Snare for Linux, we went on to create 'Snare for Solaris', and 'Snare for Windows' - again, released for free, under the terms of the GPL open-source license. Service agents for Apache, IIS, Squid, ISA Server, and so on, soon followed, and we eventually had a suite of applications that could collect and report data from most common log producing applications and operating systems. We're now seeing somewhere around 10,000 downloads of the Snare Agents per month - each of which could represent a handful of agents installed, or maybe a few thousand agents installed in an organisation worldwide. It's absolutely fantastic to see so many people making use of the applications. We use a fair number of open source tools on a day to day basis ourselves, so it allows us to give a little bit back in return. The size and diversity of the Snare user community also really helps us ensure that the agents are extremely robust and flexible. No test lab we could put together would have a chance at replicating the variety of environments that our Snare community users deal with ever day. Community feedback also helps us keep our tools as simple, and practical from a feature standpoint, as possible.


Thousands of agents, hmmm, so where is the server in this story?

We started work on the server fairly early in the Snare agent development cycle. Ironic as it may seem, a number of the early leaders in the commercial logging space (mostly tools for forensic analysis and correlation) were already using our open source agents to push data to their commercial servers. We had no problems with that - in fact, we actively encouraged it. However, several of our local customers asked whether we had a 'server solution' that would allow them to collect the logs generated by the Snare Agents together, and perform reasonable levels of analysis on the data to address the system audit and reporting requirements proscribed by various regulations. Stephen, you probably remember that at the time the functionality of existing products were generally tailored to real-time signature-based threat detection or post-incident forensics - neither of which addressed the requirements that were directed our way. You see the majority of our clients at the time were intelligence agencies and years before SOX, PCI or HIPAA triggered a dramatic shift in the SIEM market toward compliance, these organisations needed to comply with very stringent system monitoring and reporting requirements. I think the commercial users that approached us also figured that the guys who wrote the agents, would be in the best place to provide the collection, analysis, reporting and archival framework as well. So, we started reviewing and reviving the ideas we had whilst we were developing our internal audit subsystem for DSD - concepts like distribution of security responsibility to data owners, the ability to cope with potentially massive quantities of incoming audit data, an intuitive user interface, the ability to easily configure for regulatory requirements such as ACSI 33 (and by extension, HIPAA, DCID/DIAM, NISPOM, PCI, and so on). The end result, was the Snare Server.

At first, we restricted the distribution of the Snare Server to our key Australian-based clients; we even had something up on our website to that effect - that Snare Server was only available in Australia. However, after six months, we had so many requests from organisations outside Australia for access to the Snare Server, that we had to do something, or risk spending all day writing apologetic email messages to frustrated potential customers.


I hear that! A little of me dies every time I have to tell someone no when it is a reasonable request. So here comes the global software company from down under?

Not dragged kicking and screaming quite as much as the previous paragraph would imply perhaps, but yes. These days, although our local Aussie customers are very important to us, our North American and European Union customers make up a fair majority of our Snare Server users. What is quite interesting is that, although we have a pretty wide range of source industries using Snare, we get a lot of customers from national governments, Fortune 500 companies, financial institutions, large defence contractors, and the health care industry - the sorts of people that we originally designed the Snare agents and server for. Our customers tend to have a real culture of security, either as a consequence of the need to meet regulatory requirements, or just because of a drive to protect customer or corporate information. Hopefully that means that we're on target from a development perspective, and the sorts of things that we needed in DSD, and have discovered from talking to our users on a day to day basis, are the sorts of things that other IT Security teams all over the world have requirements for as well.


Nice! So the server is working out well?

The Snare Server now completely funds our open source development efforts, allowing us to bring out new agents, and provide updates and a certain level of support for our existing agents. In response to user demand, we're also looking at adding agents for Microsoft SQL server, Oracle, MySQL, and a bunch of others. We're starting to see a real migration of important data away from operating systems, into the database and application layer, and it's important for us to collect from the applications that store, and control access to the data.

Of course, the ability to collect from more and more log sources, does come at a cost; the Snare infrastructure needs to be robust enough to deal with the rapid increase in log volume, and we also need to continually come up with effective ways to present the information to people who have to manage the security of the source data. As Snare finds it's way into systems that protect extremely compartmented information, our customers have also asked us for a few extra features in our agent and server infrastructure, such as strong encryption, and verifiable delivery.


Yes, I was trying to explain this to some of the decision makers at SANS. Log analysis today is the intrusion detection of ten years ago. As we get better at reading and analyzing and protecting we will finally have insight into the whole organization. But there is a lot of work to be done, just like the IDS world had a lot of work to do ten years ago. So what are people asking for?

Our larger customers are constantly pushing us to improve the confidentiality, integrity and availability of the end-to-end Snare system. Features like centralised agent configuration management, fail-over, and log caching are examples of some of the means to those ends. Version 4 of our Snare Server has taken a bit of a quantum leap down that path - agent and server now work much closer together, and we've implemented a new data store that has enabled us to wring a heck of a lot more performance out of the same hardware and thereby greatly improved the cost/benefit ratio for our customers. We've also made sure that our new storage architecture is implemented in such a way that it both assures the integrity of the log record and also allows our users to easily export the raw data into other systems. Having been in the customer's shoes in the past, we're very much in tune with our users desire to avoid vendor lock-in.

So as you can see, we're certainly kept on our toes. Luckily, we have a growing number of terrific partners in North America, and Europe, to help us keep up - all of which share our commitment to avoid the 'smiling salesman' interface layer for support. We also have the advantage of a small, focussed team that sincerely enjoys IT security, and log management (crazy though that may seem). We stay sane by occasionally taking the entire engineering team, laptops and all, down to the coast for days at a time - particularly when we're brainstorming for the next version of the Snare Server, or looking at ways to implement a new agent. At IA our 'board meetings' are something that everyone looks forward to - as long as the surf is up and the weather is fine. Some of our best ideas or strategies have come very early in the morning, while we're waiting for a wave. Snare continues to be an exciting ride for us, and provides us with an opportunity to help many thousands of dedicated IT security people just like ourselves.


So, if one of our readers wants to learn more about Snare, where should they go? What are the smart first steps?

I don't really want to go into 'sales mode' in an interview like this, so I'll keep this as short as I can. The web site is a fair starting point - pop 'snare' into your favourite search engine, and we'll be up the top there somewhere--as will some competitors who leverage our open source agents. We really like our potential customers to have access to our software, which seems to make us somewhat unique in IT security market. We recommend people install the agents, and actually use them on an operational basis. The agents are free, and can be pointed at a standard syslog server for collection. We also provide a free collection tool on our web site called 'Backlog', and even some sample server code for people who want to write their own backend rather than using the Snare Server. We even cover a bit of support via the 'sourceforge' web site, where you can ask questions, or download the full source code to the open source agents.

If your existing collection system isn't quite up to the task, or you'd like to take advantage of some additional analysis, reporting and archival functionality, then talk to us about Snare Server. Again - even with our commercial product our approach is to encourage people to get the software in their hands. We have an online demo server available, and if you like what you see, we can provide a 30 day trial pretty easily - no restricted capabilities, no limitations that might hide potential performance issues, and no sales engineer required to hold your hand (or hide warts) during the installation process. Around 80%+ of the people that try the Snare Server end up buying it - a stat which we're extremely proud of. Perhaps it's the fact that Snare was built by people who used to 'live the dream' so to speak; the interface, and the way Snare works, ends up being pretty intuitive for the people who implement it in most cases.


So, where do you see the Log analysis world in two years, what will we be able to accomplish with our infrastructure?

IT teams have more to do every day - spreading themselves thinner and thinner, as more technology comes on board. In contrast, things that used to require a huge amount of knowledge and training, such as setting up a stateful firewall in an appropriate way, are becoming so much easier, and are starting to migrate out to the people that control the data. These days, most internet users have a fairly comprehensive, stateful packet filtering firewall active and running as part of their ADSL/Cable modem; a few short years ago, you had to have a pretty good grasp of TCP/IP, packet flags, and even potentially IOS or IPChains, in order to configure your own firewall. Auditing won't quite make it as far as firewalls, or virus checkers in the next couple of years perhaps, but I'd love to see it take a few steps in the same direction - getting easier for the IT team to manage, and providing a bit of breathing room by pushing the capability out to the people that can use it most.

I suspect we'll see less focus on signature-based systems. Although they provide the potential for real-time reporting, when faced with the question "What exactly do you want to be woken up for, at 3am?", there are few people that could really pin the answer to a particular reporting signature such as "SQL buffer overflow attempted on host X". People seem to prefer something a little more holistic, such as "someone has grabbed a copy of my database", or "our primary web site has been defaced". Log analysis is hopefully going to start presenting information in terms that the average end-user can really grab on to - incorporating more meta-data and corporate intelligence, and less of the real geeky raw data, which can be hard to read, and is easily flooded by false positives.

The world of audit log data is one of contrasts. On one hand, we have to deal with a huge quantity of events on a day-to-day basis. On the other hand, every time someone downloads a 5 megabyte document from a file server, it may only result in a couple of 300-byte events. This could potentially open the door to the possibility of a 'managed' audit service, particularly for smaller organisations who have a requirement to meet something like HIPAA, PCI or even more local regulatory requirements such as California State Senate Bill 1386. Many small organisations find it difficult to spare the resources to what can be a somewhat challenging job like monitoring audit log information. Perhaps we'll start to see local or global service providers spring up to look after these sorts of customers. Maybe we'll even be one of them.


One of our traditions is to offer a bully pulpit, a chance to share what really burns in your heart related to security, what message do you have for our readers?

As both a geek, and as someone who runs a business, I have a bit of a split personality towards security where it relates to auditing.

My geek side wants to delve down into the technical weeds, and complain mightily about systems that report dates without time-zone hints (5/2/2008 - is that the 5th of February, or the 2nd of May? Argh!), or perhaps manufacturers that insist on attempting to report raw log data in a human readable format at the expense of structural consistency, when there's absolutely no chance that a human is going to go through the 150 megabytes of raw logs that have been generated from that device each day.

But, realistically - though those issues give me the grumbles on occasions, I think IT security is in a pretty good place at the moment. It is gradually becoming a core part of business strategy, and is starting to be driven by organisational risk assessments, rather than being just a reflexive reaction to a series of vaguely understood recent threats. This is an excellent development, and really gives the IT guys the ability to deliver targeted solutions to security problems, rather than just throwing a bunch of generic tools at the task. To support this, your corporate decision makers and data owners really need to 'own' security. This doesn't mean that they need to understand either the current threat profile in detail, or defensive mechanisms in depth, but they need to understand the differences in 'risk' to an asset or information, if particular security components such as access controls or auditing, are not put in place. The system security plan (or threat/risk assessment document) is a great vehicle for this - it's amazing to see a data owner's enthusiasm for security change when they have the reins, or alternatively, it's impressive to see a pen-hand start shaking when asked to sign off on a security risk assessed at 'very high' when no additional controls have been implemented yet.


Computing and security are fun, but there is more to life. Leigh, please tell us just a bit about yourself as a person. When you are not in front of a computer, what do you do?

Anton, from Loglogic, and I seem to share some common interests, based on your recent interview - volleyball and kayaking amongst them. I really enjoy fishing out of my kayak, and can often be seen walking the yak down to the water, with so many rods attached that you could be forgiven for mistaking it for a pincushion. My wife and two children are a source of constant amazement and delight, and offer a welcome distraction that serves to drag me up from the depths of contemplating some new log format or other, and inject the occasional large slice of reality into my day.
<< Thought Leader Home
top^
<< Thought Leader Home

SANS CDI 2013 - DC

Latest Whitepapers

Building and Maintaining a &quot;Certifiable&quot; Workforce
By Robert J. Mavretich

How Can You Build and Leverage SNORT IDS Metrics to Reduce Risk?
By Tim Proffitt

Daisy Chain Authentication
By Courtney Imbert

Latest Tweets

NEW #SANS Survey: Security Analytics & Intelligence 2nd [...]
October 1, 2013 - 6:30 PM

Tips on how to convince your boss to let you train online wi [...]
October 1, 2013 - 6:23 PM

#2 Tip on how 2 convince your boss 2 let u train online: Fle [...]
October 1, 2013 - 3:00 PM

Contact Us

(301) 654-SANS (7267)
Mon-Fri 9am - 8pm EST/EDT
info@sans.org

"It has really been an eye opener concerning the depth of security training & awareness that SANS has to offer."
- Michael Hall, Drivesavers

"This has been a great way to get working knowledge that would have taken years of experience to learn."
- Josh Carlson, Nelnet

"Because of the use of real-world examples it's easier to apply what you learn."
- Danny Hill, Friedkin Companies, Inc.