Leigh Purdie, InterSect Alliance, co-founder of Snare
March 7th, 2008 By Stephen Northcutt
Perhaps, one of the hottest topics in 2008 is log file analysis
(who would have guessed). And while the commercial tools are getting a
lot of the press, an open source, and also commercial tool is ending up
on a lot of systems. It is called Snare and Leigh Purdie is the
thought leader behind the project. He has been willing to invest the
time for a thought leadership interview with the Security Laboratory,
and we certainly thank him for his time.
Leigh, how did this all begin? The need for industrial strength
logging for Windows is extreme, how did you become the aspirin for this
G'day Stephen, logging certainly has hit the security spotlight in the
last few years; it's been a bit of a wild ride. The story of Snare
is fairly entwined with that of InterSect Alliance - I don't think
either would exist without the other, so I'll give you a bit of a
background. Close to 10 years ago, I was a senior IT security
specialist at Australia's information security authority, the Defence
Signals Directorate. To provide some context for your North American
readers, DSD is similar in mission to the NSA in the United States. DSD
was a fascinating place to work, with excellent resources, a diverse IT
ecosystem, and a culture of security appropriate to the confidentiality
of the information it processed.
So, Leigh, I am guessing InterSect Alliance comes up pretty quickly?
Yes - it was at DSD where I met George Cora, the other founder of
InterSect Alliance. What drew George and myself together was a shared a
drive to make security less of a burden and more of a business asset.
In most medium to large organisations at that stage, security was seen
as the responsibility of a vague amorphous 'security cell' buried
somewhere within the IT area.
I hear you! Marcus Sachs, one of the US cybersecurity czars, once
commented that some of us heard the sound of a song (security) and
started dancing to it very early on. A lot of people who looked at us
funny then, are amazed at our prescience now. So what did you notice
Absolutely; both George and I got a few confused looks early on in the
process when we told people we were interested in working on IT
security - roughly the same sort of look you'd probably get if you
announced that you were going to take up professional octopus
wrestling. The tools just weren't well developed, and the
organisational understanding of how to use IT security as not only a
'protection layer', but also as a business-enabling asset, really
wasn't there. We both felt (and still do) that the tools to validate
and enforce security controls, as much as possible, really need to be
in the hands of the people who are responsible for the data. They
know how important the data really is, and the people who should (or
shouldn't) have access to it. People are familiar, and comfortable with
this concept in the world of physical security, where we have layers of
protection that narrow as the protected asset becomes more 'personal',
or tied to responsibility of the individual. In the IT arena though,
the complexity of the technology has forced us down the path of a
centralised protection strategy, where the core IT security area has
all the responsibility, but often no real feel for the individual
security requirements of the underlying data.
Gotcha Leigh! I think of the dynamic tension between a data
owner and data custodian very often. I have seen more than one data
base administrator cheeky enough to try to hold an entire company
hostage and determine who gets coding priority, database access
priority - it is not a pretty sight. So what did you notice that was
True. Unless the general directional controls are in the hands of those
that understand the data, you risk your IT cell driving corporate
policy. That might work pretty well in some organisations that are very
information-driven, or have very flat organisational structures, but in
more traditional companies, it can give the CEO heart palpitations.
George and I were particularly frustrated with the limitations of the
crop of IT security products that were available at the time, as we
were dealing day-in, day-out with the requirement to protect top
secret, compartmented information for which we had no need (or wish!)
to know. In addition, we were trying to cope with the massive
quantities of audit log data produced by a high-security organisation,
sourced from a raft of different operating systems, applications, and
services. The commercial tools available to assist in the process of
event log management at the time were generally single-platform,
horribly expensive relative to their useful capabilities, and crashed
when faced with the sorts of volumes we saw on a daily basis. Products
also seemed to fall into the trap of 'fluffing up' their feature-set
with overly complex stuff that often produced a flashy demo but was
practically useless in an operational setting. It's a little bit
like driving a Winnebago down to the shops just to pick up some milk -
the extra features often just get in the way of what you are trying to
accomplish. Our attempts to gain access to reasonable support also
provided a few challenges - we never seemed to be able to get past the
'smiling salesman' layer to the people who could actually solve the
problems (no offence intended to sales people). All of these
experiences have had a major impact on IA and how the Snare
applications have evolved over time.
So, like many IT organisations around the world, we started putting
together a system of our own. It needed to cope with a huge quantity of
log data, and a diverse range of source systems, and it also needed to
produce reports that were optimised for data owners - not necessarily
IT professionals; providing data owners with the capability to manage
their own security (within clearly delineated boundaries).
OK, so you were in enough pain to start writing code, how did the early days go?
I'm a geek at heart, so it usually doesn't take a lot of pain for me to
start coding, but in the context of our other priorities at the time,
yes - it was going to be a big investment in time and resources that we
didn't want to take on lightly. The system was a little crude behind
the scenes, but had a friendly front end, and proved to be quite
effective. At that stage, we were relying on automated bulk file
transfers from client systems, and were using a loosely integrated
collection of scripts to manage the user front end, but several of the
key concepts that eventually led to the Snare framework were tried and
tested in this sort of operational environment.
George and I eventually left DSD to start our own IT security company,
InterSect Alliance. IA started out as a boutique security consulting
company, but the problems and opportunities associated with log
management continued to be a significant interest area for us. We were
also watching the progress of the Linux operating system in the
commercial and government market pretty closely. Government
agencies, and defence contractors saw Linux as a potentially useful
replacement for aging Unix systems, but could not easily get past the
requirement for system-level auditing imposed by the US 'C2' security
guidelines. During our time at DSD, George and I worked on the document
'ACSI 33', which is vaguely analogous to the C2 guidelines in the US.
As a result we were keenly aware of the effect such a document can have
on the infrastructure decisions of organisations who have high security
Oh my, my, my. That is a famous and interesting problem. You started a
consulting company that became, to some extent, a software product
company. I know that is the story of CORE Security as well, from my
thought leadership interview with Ivan Arce.
Our drivers were reasonably similar to Ivan's - the lack of available
tools, really pushed him into developing his own. Our customers really
hauled us out of our niche when they saw the sorts of tools that we
were putting together. The start of this process began with the
frustration of our customers (and others) with the lack of an auditing
framework for the Linux operating system. That need drove me to create
an audit subsystem for Linux. We called it 'Snare' - System iNtrusion
Analysis and Reporting Environment (with a slight tip of the hat to
George's hobby as a drummer), and released it as open-source, under the
GNU Public License.
The response to Snare was amazing. The existence of Snare for Linux,
allowed many Linux installations to progress, that had previously been
frustrated by the fact that C2 security requirements could not be met.
Snare for Linux also included event filtering, in order to curtail the
operating system's enthusiastic reporting of information, and could
either send the results to a local file, or over the network to a
custom UDP port, or via syslog.
So now this is a bit like the start of a ski run down double black
diamonds, I bet you were on the edge and could not see what was before
you! So, what happened?
As an Aussie, I don't tend to see a lot of snow - our water tends to
stay nice and wet, rather than freezing. But yes, it felt a bit like
dropping in on a 2 meter shore break wave, which sounds like a similar
Emboldened by the success of Snare for Linux, we went on to create
'Snare for Solaris', and 'Snare for Windows' - again, released for
free, under the terms of the GPL open-source license. Service agents
for Apache, IIS, Squid, ISA Server, and so on, soon followed, and we
eventually had a suite of applications that could collect and report
data from most common log producing applications and operating systems.
We're now seeing somewhere around 10,000 downloads of the Snare Agents
per month - each of which could represent a handful of agents
installed, or maybe a few thousand agents installed in an organisation
worldwide. It's absolutely fantastic to see so many people making use
of the applications. We use a fair number of open source tools on a day
to day basis ourselves, so it allows us to give a little bit back in
return. The size and diversity of the Snare user community also really
helps us ensure that the agents are extremely robust and
flexible. No test lab we could put together would have a chance at
replicating the variety of environments that our Snare community users
deal with ever day. Community feedback also helps us keep our
tools as simple, and practical from a feature standpoint, as possible.
Thousands of agents, hmmm, so where is the server in this story?
We started work on the server fairly early in the Snare agent
development cycle. Ironic as it may seem, a number of the early leaders
in the commercial logging space (mostly tools for forensic analysis and
correlation) were already using our open source agents to push data to
their commercial servers. We had no problems with that - in fact, we
actively encouraged it. However, several of our local customers
asked whether we had a 'server solution' that would allow them to
collect the logs generated by the Snare Agents together, and perform
reasonable levels of analysis on the data to address the system audit
and reporting requirements proscribed by various regulations.
Stephen, you probably remember that at the time the functionality of
existing products were generally tailored to real-time signature-based
threat detection or post-incident forensics - neither of which
addressed the requirements that were directed our way. You see
the majority of our clients at the time were intelligence agencies and
years before SOX, PCI or HIPAA triggered a dramatic shift in the SIEM
market toward compliance, these organisations needed to comply with
very stringent system monitoring and reporting requirements. I
think the commercial users that approached us also figured that the
guys who wrote the agents, would be in the best place to provide the
collection, analysis, reporting and archival framework as well. So, we
started reviewing and reviving the ideas we had whilst we were
developing our internal audit subsystem for DSD - concepts like
distribution of security responsibility to data owners, the ability to
cope with potentially massive quantities of incoming audit data, an
intuitive user interface, the ability to easily configure for
regulatory requirements such as ACSI 33 (and by extension, HIPAA,
DCID/DIAM, NISPOM, PCI, and so on). The end result, was the Snare
At first, we restricted the distribution of the Snare Server to our key
Australian-based clients; we even had something up on our website to
that effect - that Snare Server was only available in Australia.
However, after six months, we had so many requests from organisations
outside Australia for access to the Snare Server, that we had to do
something, or risk spending all day writing apologetic email messages
to frustrated potential customers.
I hear that! A little of me dies every time I have to tell
someone no when it is a reasonable request. So here comes the global
software company from down under?
Not dragged kicking and screaming quite as much as the previous
paragraph would imply perhaps, but yes. These days, although our
local Aussie customers are very important to us, our North American and
European Union customers make up a fair majority of our Snare Server
users. What is quite interesting is that, although we have a pretty
wide range of source industries using Snare, we get a lot of customers
from national governments, Fortune 500 companies, financial
institutions, large defence contractors, and the health care industry -
the sorts of people that we originally designed the Snare agents and
server for. Our customers tend to have a real culture of security,
either as a consequence of the need to meet regulatory requirements, or
just because of a drive to protect customer or corporate information.
Hopefully that means that we're on target from a development
perspective, and the sorts of things that we needed in DSD, and have
discovered from talking to our users on a day to day basis, are the
sorts of things that other IT Security teams all over the world have
requirements for as well.
Nice! So the server is working out well?
The Snare Server now completely funds our open source development
efforts, allowing us to bring out new agents, and provide updates and a
certain level of support for our existing agents. In response to user
demand, we're also looking at adding agents for Microsoft SQL server,
Oracle, MySQL, and a bunch of others. We're starting to see a real
migration of important data away from operating systems, into the
database and application layer, and it's important for us to collect
from the applications that store, and control access to the data.
Of course, the ability to collect from more and more log sources, does
come at a cost; the Snare infrastructure needs to be robust enough to
deal with the rapid increase in log volume, and we also need to
continually come up with effective ways to present the information to
people who have to manage the security of the source data. As Snare
finds it's way into systems that protect extremely compartmented
information, our customers have also asked us for a few extra features
in our agent and server infrastructure, such as strong encryption, and
Yes, I was trying to explain this to some of the decision
makers at SANS. Log analysis today is the intrusion detection of ten
years ago. As we get better at reading and analyzing and protecting we
will finally have insight into the whole organization. But there is a
lot of work to be done, just like the IDS world had a lot of work to do
ten years ago. So what are people asking for?
Our larger customers are constantly pushing us to improve the
confidentiality, integrity and availability of the end-to-end Snare
system. Features like centralised agent configuration management,
fail-over, and log caching are examples of some of the means to those
ends. Version 4 of our Snare Server has taken a bit of a quantum leap
down that path - agent and server now work much closer together, and
we've implemented a new data store that has enabled us to wring a heck
of a lot more performance out of the same hardware and thereby greatly
improved the cost/benefit ratio for our customers. We've also made sure
that our new storage architecture is implemented in such a way that it
both assures the integrity of the log record and also allows our users
to easily export the raw data into other systems. Having been in the
customer's shoes in the past, we're very much in tune with our users
desire to avoid vendor lock-in.
So as you can see, we're certainly kept on our toes. Luckily, we have a
growing number of terrific partners in North America, and Europe, to
help us keep up - all of which share our commitment to avoid the
'smiling salesman' interface layer for support. We also have the
advantage of a small, focussed team that sincerely enjoys IT security,
and log management (crazy though that may seem). We stay sane by
occasionally taking the entire engineering team, laptops and all, down
to the coast for days at a time - particularly when we're brainstorming
for the next version of the Snare Server, or looking at ways to
implement a new agent. At IA our 'board meetings' are something that
everyone looks forward to - as long as the surf is up and the weather
is fine. Some of our best ideas or strategies have come very
early in the morning, while we're waiting for a wave. Snare continues
to be an exciting ride for us, and provides us with an opportunity to
help many thousands of dedicated IT security people just like ourselves.
So, if one of our readers wants to learn more about Snare, where should they go? What are the smart first steps?
I don't really want to go into 'sales mode' in an interview like this,
so I'll keep this as short as I can. The web site is a fair starting
point - pop 'snare' into your favourite search engine, and we'll be up
the top there somewhere--as will some competitors who leverage our open
source agents. We really like our potential customers to have access to
our software, which seems to make us somewhat unique in IT security
market. We recommend people install the agents, and actually use
them on an operational basis. The agents are free, and can be pointed
at a standard syslog server for collection. We also provide a free
collection tool on our web site called 'Backlog', and even some sample
server code for people who want to write their own backend rather than
using the Snare Server. We even cover a bit of support via the
'sourceforge' web site, where you can ask questions, or download the
full source code to the open source agents.
If your existing collection system isn't quite up to the task, or you'd
like to take advantage of some additional analysis, reporting and
archival functionality, then talk to us about Snare Server. Again -
even with our commercial product our approach is to encourage people to
get the software in their hands. We have an online demo server
available, and if you like what you see, we can provide a 30 day trial
pretty easily - no restricted capabilities, no limitations that might
hide potential performance issues, and no sales engineer required to
hold your hand (or hide warts) during the installation process. Around
80%+ of the people that try the Snare Server end up buying it - a stat
which we're extremely proud of. Perhaps it's the fact that Snare was
built by people who used to 'live the dream' so to speak; the
interface, and the way Snare works, ends up being pretty intuitive for
the people who implement it in most cases.
So, where do you see the Log analysis world in two years, what will we be able to accomplish with our infrastructure?
IT teams have more to do every day - spreading themselves thinner and
thinner, as more technology comes on board. In contrast, things that
used to require a huge amount of knowledge and training, such as
setting up a stateful firewall in an appropriate way, are becoming so
much easier, and are starting to migrate out to the people that control
the data. These days, most internet users have a fairly comprehensive,
stateful packet filtering firewall active and running as part of their
ADSL/Cable modem; a few short years ago, you had to have a pretty good
grasp of TCP/IP, packet flags, and even potentially IOS or IPChains, in
order to configure your own firewall. Auditing won't quite make
it as far as firewalls, or virus checkers in the next couple of years
perhaps, but I'd love to see it take a few steps in the same direction
- getting easier for the IT team to manage, and providing a bit of
breathing room by pushing the capability out to the people that can use
I suspect we'll see less focus on signature-based systems.
Although they provide the potential for real-time reporting, when faced
with the question "What exactly do you want to be woken up for, at
3am?", there are few people that could really pin the answer to a
particular reporting signature such as "SQL buffer overflow attempted
on host X". People seem to prefer something a little more
holistic, such as "someone has grabbed a copy of my database", or "our
primary web site has been defaced". Log analysis is hopefully
going to start presenting information in terms that the average
end-user can really grab on to - incorporating more meta-data and
corporate intelligence, and less of the real geeky raw data, which can
be hard to read, and is easily flooded by false positives.
The world of audit log data is one of contrasts. On one hand, we have
to deal with a huge quantity of events on a day-to-day basis. On the
other hand, every time someone downloads a 5 megabyte document from a
file server, it may only result in a couple of 300-byte events.
This could potentially open the door to the possibility of a 'managed'
audit service, particularly for smaller organisations who have a
requirement to meet something like HIPAA, PCI or even more local
regulatory requirements such as California State Senate Bill 1386. Many
small organisations find it difficult to spare the resources to what
can be a somewhat challenging job like monitoring audit log
information. Perhaps we'll start to see local or global service
providers spring up to look after these sorts of customers. Maybe we'll
even be one of them.
One of our traditions is to offer a bully pulpit, a chance to
share what really burns in your heart related to security, what message
do you have for our readers?
As both a geek, and as someone who runs a business, I have a bit of a
split personality towards security where it relates to auditing.
My geek side wants to delve down into the technical weeds, and complain
mightily about systems that report dates without time-zone hints
(5/2/2008 - is that the 5th of February, or the 2nd of May? Argh!), or
perhaps manufacturers that insist on attempting to report raw log data
in a human readable format at the expense of structural consistency,
when there's absolutely no chance that a human is going to go through
the 150 megabytes of raw logs that have been generated from that device
But, realistically - though those issues give me the grumbles on
occasions, I think IT security is in a pretty good place at the moment.
It is gradually becoming a core part of business strategy, and is
starting to be driven by organisational risk assessments, rather than
being just a reflexive reaction to a series of vaguely understood
recent threats. This is an excellent development, and really gives
the IT guys the ability to deliver targeted solutions to security
problems, rather than just throwing a bunch of generic tools at the
task. To support this, your corporate decision makers and data owners
really need to 'own' security. This doesn't mean that they need to
understand either the current threat profile in detail, or defensive
mechanisms in depth, but they need to understand the differences in
'risk' to an asset or information, if particular security components
such as access controls or auditing, are not put in place. The
system security plan (or threat/risk assessment document) is a great
vehicle for this - it's amazing to see a data owner's enthusiasm for
security change when they have the reins, or alternatively, it's
impressive to see a pen-hand start shaking when asked to sign off on a
security risk assessed at 'very high' when no additional controls have
been implemented yet.
Computing and security are fun, but there is more to life. Leigh,
please tell us just a bit about yourself as a person. When you are not
in front of a computer, what do you do?
Anton, from Loglogic, and I seem to share some common interests, based
on your recent interview - volleyball and kayaking amongst them. I
really enjoy fishing out of my kayak, and can often be seen walking the
yak down to the water, with so many rods attached that you could be
forgiven for mistaking it for a pincushion. My wife and two children
are a source of constant amazement and delight, and offer a welcome
distraction that serves to drag me up from the depths of contemplating
some new log format or other, and inject the occasional large slice of
reality into my day.
<< Thought Leader Home