Intrusion Detection FAQ: How to place IDS sensor in redundant networks?

By: Chris Calabrese

Background and Introduction

The IDS world has finally gotten its arms around switched LAN's, but what about networks with redundant components? How do you know you're seeing all the attacks?

This paper examines common forms of network redundancy, strategies for placing network IDS probes in redundant networks, and the effectiveness of those strategies.

The goal here is to allow a single IDS probe to see all the traffic associated with a particular attack from one host to another regardless of how many packets the attack takes, how many network-level "sessions" the attack takes, or what network paths the packets traverse.

Redundant Network Elements

The following table gives a taxonomy of redundant network elements we'll consider in this paper:

How to place IDS sensor in redundant networks?
Chris Calabrese Background and Introduction The IDS world has finally gotten its arms around switched LAN's, but what about networks with redundant components? How do you know you're seeing all the attacks?

This paper examines common forms of network redundancy, strategies for placing network IDS probes in redundant networks, and the effectiveness of those strategies.

The goal here is to allow a single IDS probe to see all the traffic associated with a particular attack from one host to another regardless of how many packets the attack takes, how many network-level "sessions" the attack takes, or what network paths the packets traverse.. Redundant Network Elements The following table gives a taxonomy of redundant network elements we'll consider in this paper:

Redundant Network Elements
	Failover of network access	Multiple points of network access	Multiple access, single address
System/Node Availability	System with failover between between network interface cards (NIC's)	NIC multi-pathing through route advertisements	NIC multi-pathing through switch link-aggregation advertisements
Application Availability	Cluster with failover of IP address	Cluster with independent IP addresses (e.g., DNS secondaries)	Cluster with shared IP address (e.g, using a web server load balancer)
Network Availability	Multiple network switches	Parallel network routes	Cluster of transparent, stateful routing elements (i.e., load balanced firewalls)

Note that these elements are orthogonal and may be arbitrarily combined. However, it would be unusual to combine two elements from the same row, such as NIC failover and route-based NIC multi-pathing, since they're aimed at the same purpose.

The rest of this section serves as a brief introduction to how these elements operate, how they're useful, and, and their challenges for IDS sensor placement.

NIC failover

With NIC failover, an individual network node (typically a server or router) has multiple links to its LAN (or LAN's) through redundant NIC's that allow failover of the system's network address(es) from a primary NIC to a secondary NIC. This can ensure that the system continues operating in the face of a NIC failure.

In this scenario, failover software on the network node itself detects that the primary NIC is inoperative, de-configures it from use at the OS level, and configures the secondary NIC to take its place. In more sophisticated implementations, both the IP address and MAC address will fail over, allowing the secondary NIC to start talking on the network immediately. In less sophisticated implementations only the IP address will fail over, causing delays as ARP caches time out.

If we assume that it's highly unlikely for a NIC to fail in the middle of an attack, then the IDS placement challenge is to ensure that the traffic to/from both NIC's is covered, but not necessarily by the same IDS probe.

NIC multi-pathing through route advertisements

With NIC multi-pathing in general, an individual network node (typically a server or router) has multiple links to its LAN (or LAN's) through redundant NIC's operating in parallel. This allows higher network throughput compared to NIC failover.

With multi-pathing through route advertisements, the system has one IP address per NIC and advertises routes to its various IP addresses through each of its other IP addresses. Solaris 8, for example, does this by responding appropriately to Router Discovery Protocol requests[ 1 ].

The IDS challenge with multi-pathing through route advertisements is for one IDS probe to see all the network traffic regardless of which NIC the traffic is going to. Otherwise the IDS would have to do complex multi-probe correlations to construct a single attack.

Multi-pathing through switch link-aggregation advertisements

This is very similar to the multi-pathing through route advertisements, but here the system advertises paths to itself at the switching level instead of the routing level. The HP-UX implementation, for example, uses Cisco's Fast EtherChannel protocol to tell the switch how to send packets for the system's MAC address(es)[ 2].

The IDS challenge is the same as for multi-pathing through route advertisements.

Failover clustering

With failover clustering, network nodes (servers, routers, firewalls, etc.) can take over each other's network addresses when the primary address holder fails. This is similar to the failover NIC scenario discussed above, but with the failover happening at the entire-machine level rather than at the NIC level. As with NIC failover, just the IP address or the IP and MAC addresses can fail over depending on the sophistication of the software. This can ensure that a key service to continues operating in the face of a server failure.

If we assume that it's highly unlikely for a system to fail in the middle of an attack unless the result of the attack, then the IDS placement challenge is to ensure that the traffic to/from both systems is covered, but not that traffic to all cluster members must be covered by a single IDS probe.

Independent IP clustering

With independent IP clustering, each system in the cluster has its own unique IP address, and the client applications must decide how to distribute the load among the different servers. This is the basis of DNS clustering (the client choses among multiple DNS servers that are authoritative for the domain) and SMTP clustering (the client choses among multiple SMTP servers that have MX records in the domain)[ 3 ].

The IDS placement challenge with independent IP clustering is to ensure that a single probe sees all the traffic for a particular source/destination address pair. This may require multiple probes if the systems are geographically disparate.

Shared IP clustering

With shared IP clustering, an external load balancer is used to distribute requests sent to a shared/virtual network address to multiple servers. Each server in the cluster has their own unique/real IP address, and traffic between the load balancer and the real servers reflects these addresses rather than the shared address. This is how most popular web server load balancing systems work.

The IDS placement challenge with shared IP clustering is to ensure that a single probe sees all the traffic for a particular source/destination address pair (i.e., all traffic for a single attack). However, if we assume that the load balancer keeps all traffic from a particular source going to a particular destination as long as that destination is alive, and if we further assume that it's highly unlikely for a server to go down during an attack except as a result of the attack, than this collapses to all the traffic for a particular source regardless of which server the load balancer decided to send the traffic to.

Multiple switches

Merely having multiple switches handling the same network segment does not itself make the switches redundant or contribute to higher network availability. But multiple switches can be combined with redundant NIC's, failover routers, etc. so that the network will stay up in the event of a single switch failure.

The IDS placement challenge is to ensure that all data from all switches is seen by the IDS probes.

Parallel routes

In this scenario, a network may have redundant links internally (i.e., it is not a tree/hierarchy) or have multiple links to the outside world (i.e., multiple ISP links). This is typically used to ensure that network connectivity can survive a single link being down and also to increase network throughput.

The IDS placement challenge is to ensure that a single probe sees all the traffic that may flow between a particular source/destination pair so that multi-probe correlation is not necessary to construct a single attack.

Load-balanced firewalls

This is similar to the shared IP clustering scenario, but here the load balancers are placed both in front of and behind the firewall farm. This is used both to ensure that network connectivity will remain in the face of a firewall failure and to increase overall throughput through the firewall farm.

The IDS placement challenge is to ensure that a single probe sees all the traffic associated with a particular attack, and therefore a particular IP source/destination pair. However, similarly to the shared IP clustering scenario, load balancers keep all traffic for a particular source/destination pair going through the same firewall as long as the firewall is alive, and if we further assume that it's highly unlikely for a firewall to go down during an attack except as a result of the attack, than a single attack will be tied to a single firewall.

IDS Placement Strategies

In this section we'll consider the following placement strategies:

Single-segment IDS probe placed outside area of redundancy
Multiple single-segment probes
Multi-segment probe
Multiple multi-segment probes

Single-segment IDS probe placed outside area of redundancy

A single IDS probe is used, but carefully placed to avoid the problems introduced by redundant network elements. The probe could be a general-purpose system running a software IDS, a purpose-built IDS appliance, or a special purpose IDS card for a network switch such as Cisco's Catalyst 6000 IDS Module[ 4 ] or Intrusion, Inc.'s SecureCom 6000 series[ 5 ].

The challenge is finding a location where the IDS probe can see all the traffic. The table below shows where a probe might be located when used with different redundant network elements. Remember when reading the table, however, that the elements are orthogonal, so a location can only be used if it works for all the redundant elements in the network. For example, a switch SPAN port only works if you have a single switch or if the switches you have support multi-switch spanning.

Redundant Network Element	IDS Probe Location
Failover NIC's	Switch SPAN port
NIC multi-pathing	Switch SPAN port
Failover cluster	Switch SPAN port
Independent IP cluster	Switch SPAN port, if not geographically disparate
Shared IP cluster	In front of load balancer
Multiple switches	Multi-switch SPAN port
Parallel routes	Whatever part of the network is not parallel, if any
Load-balanced firewalls	In front of or behind entire cluster

Aswitch SPAN port is a network switch port setup so that that the switch copies all packets traveling through the swich (or a particular VLAN or set of VLAN's) to that port. This is sometimes referred to as port mirroring, though technically that refers to copying packets from one particular port to another, rather than copying all the packets from all the ports.

A multi-switch SPAN port refers to a SPAN port that sees data from multiple switches. In this way a single IDS probe can see span traffic from multiple switches. The poor man's version of this is to connect the SPAN port of one switch into another switch, but this can be cumbersome and cause performance problems and delays. Some high end switches such as the Cisco Catalyst 6000 family of switches have native support for inter-switch spanning over the normal inter-switch backbone connections[ 6 ].

Multiple single-segment probes

Next up the food chain is using multiple single-segment probes to cover multiple network segments. This gives us more flexibility to cover situations where failover network elements are used in conjunction with other redundant elements such as multiple switches. However, this technique still can not address issues where the traffic from a single attack may be sent along multiple paths that can't be spanned, such as multi-path NIC's on multiple switches that don't support multi-switch spanning. Having multiple probes also increases acquisition and management costs.

The following table summaries where multiple single-segment probes can offer an improvement over a single single-segment probe.

Redundant Network Element	Advantages/Disadvantages of Multiple Probes vs. Single Probe
Failover NIC's	Improves flexibility to work with other redundant elements
NIC multi-pathing	No difference
Failover cluster	Improves flexibility to work with other redundant elements
Independent IP cluster	Improves performance, ability to handle geographic diversity (locate by individual servers rather than in front of cluster)
Shared IP cluster	Improves performance (locate by individual servers rather than in front of cluster)
Multiple switches	No difference
Parallel routes	No difference
Load-balanced firewalls	Improves performance (locate by individual firewalls rather than in front of / behind cluster)

Multi-segment probe

An IDS probe that can listen on multiple network segments can cover arbitrarily complex redundant network scenarios as long as the whole network is in the same building. And less probes means lower management costs. But should we always jump to this design if a single single-segment IDS probe won't cut it, or are there situations where multiple single-segment probes would be more effective? Or even a combination of the single- and multi-segment probes. The issues are going to be:

What is the acquisition cost vs. multiple single-segment probes?
Can a single complex probe handle the traffic levels?

The answers to these questions depend greatly on exactly how the probe is constructed. We have two options:

Probe with multiple NIC's
Probe connected to a specialized switch

Let's look at these two options in more detail.

Probe with multiple NIC's

Not all NIDS software supports multiple NIC's. For example, SNORT can not look at more than one data stream at a time[ 7 ]. Only higher-end versions other systems may support this feature, especially on appliance-based probes, and the added cost may not be justified if other high-end features (such as probe performance) aren't required. Regarding performance, many IDS technologies are not able to make use of SMP technology and therefore are limited to roughly 100Mb/s total throughput[ 7 ]. Even systems that support SMP may have very high costs relative to performance due to non-linear scaling with the number of processors.

Probe connected to specialized switch

It is possible to connect IDS systems that do not support multiple NIC's to a specialized switch such as the Top Layer AppSwitch that feeds the IDS probe from multiple ports on the switch[ 8 ]. The cost of a Top Layer AppSwitch starts at around $10,000 at the time of this writing[ 9 ], so again, this may not be a cost effective solution. However, the AppSwitch is also capable of feeding multiple IDS probes and making sure that each network flow goes to one and only one probe, so this may be an excellent choice if very high performance is necessary.

To put this in our familiar format:

Redundant Network Element	Advantages/Disadvantages of Multi-Segment Probe vs. Multiple Single-Segment Probes
Failover NIC's	No difference
NIC multi-pathing	Increases effectiveness/flexibility
Failover cluster	No difference
Independent IP cluster	Decreases performance, ability to handle geographic diversity
Shared IP cluster	Decreases performance
Multiple switches	Increases effectiveness/flexibility
Parallel routes	Increases effectiveness/flexibility
Load-balanced firewalls	Decreases performance

Multiple multi-segment probes

Same advantages of performance as multiple single probes, but with the flexibility of multi-segment probes.

Redundant Network Element	Advantages/Disadvantages of Multiple Multi-Segment Probes vs. One Multi-Segment Probe
Failover NIC's	No difference
NIC multi-pathing	No difference
Failover cluster	No difference
Independent IP cluster	Increases performance, ability to handle geographic diversity
Shared IP cluster	Increases performance
Multiple switches	No difference
Parallel routes	No difference
Load-balanced firewalls	Increases performance

IDS Probe "Sweet Spots"

Application Redundancy (Clustering)	System Redundancy	Network Redundancy
Application Redundancy (Clustering)	System Redundancy	Multiple Switches (no multi-switch port SPANning)	Multiple Switches and Multiple Links / Segments	Multiple Switches, Multiple Links / Segments, and Firewall Load Balancer (high data rate)
No Clustering or Failover Clustering	Failover NIC's	Simple probe	Multiple simple probes	Multiple simple probes
No Clustering or Failover Clustering	NIC multi-pathing	Multi-segment probe	Multi-segment probe	Multiple multi-segment probes
Geographically Disparate Independent IP Cluster	Failover NIC's	Multiple simple probes	Multiple simple probes	Multiple simple probes
Geographically Disparate Independent IP Cluster	NIC multi-pathing	Multiple multi-segment probes	Multiple multi-segment probes	Multiple multi-segment probes
Shared IP Cluster (high data rate)	Failover NIC's	Simple probe	Multiple simple probes	Multiple simple probes
Shared IP Cluster (high data rate)	NIC multi-pathing	Multiple multi-segment probes	Multiple multi-segment probes	Multiple multi-segment probes

Conclusions

No matter what kind of bizarre network architecture you have, it's always possible to find some way of monitoring it with a network IDS. However more complex networks may require multiple single-segment probes, a multi-segment probe, or even multiple multi-segment probes.

References

Mark Garner, "IP Network Multipathing," Sun BulePrints OnLine, February 2001. Available from http://www.sun.com/blueprints/0201/Multipathing.pdf .
Hewlett Packard, " Cisco Systems and hp fast etherchannel and auto-port aggregation software ." Available from http://www.hp.com/products1/unixserverconnectivity/infolib/cisco.html .
Paul Albitz and Cricket Liu, DNS and BIND, 4th Edition , O'Reilly and Associates, 2001.
Cisco Systems, "Catalyst 6000 Intrusion Detection System Module Data Sheet." Available from http://www.cisco.com/warp/public/cc/pd/si/casi/ca6000/prodlit/6kids_ds.htm .
Intrusion.Com, "SecureCom Chassis Overview." Available from http://www.intrusion.com/products/productcategory.asp?lngCatId=21 .
Cisco Systems, "Configuring the Catalyst Switched Port Analyzer (SPAN) Feature." Available from http://www.cisco.com/warp/public/473/41.html .
Martin Roesch, comments made during SNORT BoF at SANS Network Security 2001 Conference, October 2001.
Gary Kessler, "IDS-in-Depth: Top Layer's AppSwitch filters a copy of traffic flows to downstream IDSeS," Information Security Magazine, August 2001.
Top Layer Networks, private correspondence, November 2001.

< Previous Question | Back to Intrusion Detection FAQ Home | Next Question >

Check Them Out!

SANS is the fastest way to go from an Information Security beginner to an Information Security guru.
-Dave Howard, Emerson