SANS Intrusion Detection FAQ: What is Meta-Intrusion Detection Systems?

Patrick Ethier

Definition

George Ho describes Meta IDS as a technology that allows a single security console to accept from and communicate with all deployed devices that are from different vendors [1]. Pete Loshin, from Information Security Magazine adds that it is a system that can accept security alerts from all deployed security devices, massage the raw data, extract useful information and present that information in a manageable format [3].

Taken in this context, Meta-IDS uses information provided by various security devices and analyzes, performs trend analysis, sorts, correlates and presents security information to the network administrator. Since the intrusion detection doesn't originate from network data it is called Meta IDS from the use of meta data used in its analysis. But Meta IDS is much more than a super console for intrusion detection.

Meta IDS is the solution for enterprise-wide security deployments. Many vendors such as ISS and NFR offer collection servers that are able to centralize the management and monitoring of their various sensors. Certain technologies, such as IceCAP from ISS/NetworkICE are able to monitor and manage both HIDS and NIDS solutions. The convergence of IDS solutions and the growing acceptance of Intrusion Detection Message Exchange Format [2] [4](IDMEF) will undermine the current definition of Meta IDS given above. Meta IDS will need to rely upon the other advantages that it offers to the enterprise-wide deployment in order to prove its worth.

Identifying the need

The need for meta IDS arises from the inability of current intrusion detection systems to gain awareness of the "network" due to lack of scope. This is because individual IDS systems access data flowing through a particular channel at a particular point on a network. If an attack is aimed at a portion of the network which the sensor doesn't cover or the attack is launched on a host beyond the control point offered by the IDS sensor then that sensor cannot relate to events detected by other IDS systems deployed on the network.

HIDS and NIDS rely on raw data affecting a host or flowing through a network to identify anomalies or look for attack signatures. The result is a constant flow of trigger messages based on some sort of pattern matching. The problem is that regular, harmless network activities can contain these patterns and the result is the production a lot of alerts of which a good percentage indicate normal traffic. Many network administrators will disable the detection of certain attacks in order to reduce the amount of information that needs to be processed. Automating the preliminary operations that a network security technician would do in analyzing data as well as providing the ability to take this interpreted data and remove known or explained occurrences is another identified need for enterprise-wide deployment of IDS.

A huge problem often encountered in the deployment of security monitoring in an enterprise is the topology of the network. Network security specialists do not always deal with ideal situations. Internal connections, business-to-business links, flat networks, segmented networks; heterogeneous networks already pose enough of a headache with IDS in terms of coverage, performance, and efficiency. Added to this is the need to deal with vulnerabilities, storage of sensitive information and the need to keep systems available for business operations. Providing the ability to process IDS events in terms of Vulnerability Assessments, Risk Assessments and Threat Assessment identifies one of the most interesting advantages that MIDS offers to large organizations.

Deployment

When deploying MIDS, it is important to take a few factors into consideration. First, MIDS must access IDS sensors that are deployed throughout a network. Secondly, MIDS must be accessible to multiple security operators from various locations in order for them to take advantage of the system's features. Thirdly, MIDS must be deployed in a secured environment. These conditions are usually found in the Network Operations Center (NOC) of a company. Properly deployed operations centers usually offer redundant, out of band connections with controlled access.

Considerations in deploying MIDS on an enterprise network should include scalability and flexibility. Scalability means that the capacity for the MIDS to process a set amount of ids events should be distributable across multiple engines without losing any of the advantages offered by the MIDS correlation engine. Flexibility means that the MIDS engine should be open to handling new types of events as well as accept information from new types of IDS engines after the initial deployment without removing any efficiency from the correlation engine.

Features

Features that should be found on these meta IDS systems must address the handling of events in a network context, correlation based on access policy, vulnerability assessment, threat assessment and risk assessment, a distributed architecture to spread the load of processing imposing amounts of data and to make the system redundant as well as a modular/programmable interface to allow for the expansion of the system to adopt future technologies.

Handling of events in a network context means that events generated throughout the network are analyzed and matched across technologies to enforce, or promote the importance of a series of events. In this case correlation can be presented in various forms.

Gauging the importance of an intrusion event is derived from a relationship between a few things. Meta IDS can automate this process. In reference [6], there is discussion of the classification of attacks in terms of danger and transferability. The proposition here is that more factors be used in assigning criticality with respect to an organization. MIDS must be able to compute these factors in near real-time.

Intrusion attempt must be compared to a vulnerability assessment. The MIDS contains a listing of hosts on the network and a listing of the last vulnerability assessment. If the host has been identified as being "patched" for this particular intrusion attempt then the alerted is demoted. If a doubt exists about the resistance of the host to that particular vulnerability then the intrusion attempt is promoted. Secondly, the intrusion attempt must be compared to a threat assessment. If the host is situated in a position where it is prone to being attacked and that measures have been taken to reduce the threat then the intrusion attempt is demoted. If the host is situated in an area that has not been deemed prone to attack and that the measures taken to protect are not as tight then the intrusion attempt is promoted. Thirdly, the intrusion attempt must be compared to a risk assessment. If either target or source host contains highly sensitive information or if the risk imposed by the possibility that the host has been compromised is high then the intrusion attempt is promoted. If the information on the host is insignificant and the access controls on the network make it so that this host being compromised is of little importance then the intrusion attempt is demote.

Another facet of correlation is scenario matching. Scenario matching consists of taking a series of events together and turning them into a single event. Hence, vulnerability scans on a mail server matched with a zone transfer from your DNS server and ARP floods of a port on your switch might indicate that somebody is trying to take over your mail system. Being able to explain the relationship between events might take an event that is otherwise perceived as being harmless and put it into context of a global attack. This type of correlation can be done by an expert system, which learns by taking scenarios first inputted by security engineers and learned over time or it can be done using data mining techniques adapted to intrusion detection.

Wenke Lee and Salvatore Stolfo [7] [5] have discussed at great lengths the implications of using data mining techniques for intrusion detection. Although their paper discusses the use of data mining approach on tcpdump data and on sendmail logs, it is possible to abstract their methods to be used on IDMEF messages.

Correlation faces many challenges. The most important is the lack of standardization for the relationship between a known vulnerability and a type of attack. Although the arachnids database, the CVE database and other commercial databases of the sort aim to label all known vulnerabilities, no two vendors use the same convention to report similar findings. Hence, a port scan, which is a very common and general occurrence, can be detected in a multitude of ways. Port scans detected by SNORT and port scans detected by BlackICE do not always mean the same thing. IDMEF [2] was conceived to exchange data between intrusion detection systems but does not provide a mechanism to say, "This is a SYN scan" in a universal language. In order to apply data mining on data sets produced from varied technologies MIDS must overcome this challenge.

The possibility of tracking events is another feature offered by MIDS. This leaks over to the realm of ticketing and CRM software but is also an important part of intrusion detection. Offering the capability of an operator to see if a similar event has happened in the past and how the situation was resolved is of immeasurable value to an organization in dealing with security related issues. This feature is also important to allow for coordination between geographically separated security experts working on a common case. Using the MIDS as a dispatch center, to fill forms and store data about events becomes a crucial aspect in the race to secure a network by synchronizing actions undertaken by personnel. Lastly, the ability to gauge the efficiency of the technologies deployed and to offer statistics indicating how many events were detected and how many were explained/resolved means that a clear picture can be painted for the need to "beef up" security in certain areas and the justify budget for the maintenance of security levels in others.

MIDS offers the ability to act upon events. Taking into account that correlation and incident tracking can provide a certain window of warning, the MIDS engine can provide the option to gracefully shut down a server and minimize the loss of information. Understandably, care must be taken to avoid new types of denial of service attacks using these automated mechanisms. Certain IDS environments, such as SNORT using the flexresp module, already offer this feature. These IDS can deal with certain low-level decision making in order to automate responses but their lack of scope means that they are not well suited in making decisions involving many factors. MIDS has this scope and can therefore push the envelope of streamlined decision making to a higher level.

MIDS is able to manage multiple security devices from a common platform. This facilitates the application of policies throughout the network. Certain technologies, such as OPSEC and SNMP already offer the possibility to remotely reconfigure devices. Hence, MIDS should be conceived to convert between different vendor formats and allow operators to tune individual sensors from one platform. There is some debate on whether this approach should be taken using a deployed software agent [] on each host or a modular network agent that can convert between a universal language and the agents already deployed/integrated with each host.

The main feature of MIDS should be the ability to provide a global state of security. This means that reports should be generated to include common targets, common intruders, comparisons between events being detected by one type of technology but not another, etc. This state of security will allow security experts to make sound decisions as they have solid metrics on which to base their decisions on. MIDS, linked with industry best practices, has the ability to make the networking world a better and safer place.

In conclusion, MIDS is a relatively new technology which, like the network management consoles in the early 90's will allow for the ability to overcome the problems of deploying a large amount of IDS on a network and reducing the amount of effort and resources that need to be dedicated to them. This will let organizations to pursue their business and not worry about integration of security technologies onto their networks.

References:

1 - http://rr.sans.org/intrusion/tomorrow.php
2 - http://www.ietf.org/ids.by.wg/idwg.html
3 - http://www.infowar.com/iwftp/icn/05Jul2001_standardized_IDS_reporting_format.shtml
4 - http://www.infosecuritymag.com/articles/august01/cover.shtml
5 - http://www.securityfocus.com/data/library/ieee_sp99_lee.ps
6 - http://www.cs.nps.navy.mil/people/faculty/rowe/barruspap.html
7 - http://www.cs.columbia.edu/~sal/hpapers/USENIX/usenix.html