When "Redundant" Yields Different Results

One question that often comes up with I'm talking about Digital Forensics in SANS SEC 506 is, "There are so many ways to get at the same data on a Linux/Unix system, which method should we choose?" My response is, "All of them." And then I show them this little example to explain why.

Let's take the case of active network connections on the system. There are all sorts of ways to get at this data, including "lsof" and "netstat":

# lsof -i :22
# netstat -anp | grep :22
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 172.1.1.136:22          172.1.1.1:39967         ESTABLISHED -

This is definitely a Whiskey-Tango-Foxtrot kind of situation. First we're asking "lsof" to show us all current network connections associated with port 22, and it says there's nothing going on. However when we run "netstat" to dump the same information and "grep" for the port 22 traffic, we see both a local server process listening on that port as well as an active network connection with another system. If you look closer at the "netstat" output you'll see that where the "-p" option should be providing process ID information about these processes in the last column of output, we're only getting a "-", indicating that "netstat" is unable to associate a process with these network connections.

What's happening on this system is that I've used a kernel-level rootkit to hide the SSH daemon process in the kernel process table. Since "lsof" comes at the problem by scanning through the process table, it misses our hidden processes and their open sockets. On the other hand, "netstat" starts from the list of open sockets and (if requested with "-p") tries to find related processes in the process table. Even if it can't find an associated process, however, it still can output the fact that the network sockets are in use.

This is why it's so critical to capture data using multiple tools when you're trying to verify an incident or capture volatile data about the current state of a compromised system. If we had only relied on "lsof", we would have missed important information. In this case, the discrepancy in the output of these two tools tells us a lot about what's happened on the system. So make sure you use all the tools in your toolkit.

Hal Pomeranz is an independent Computer Security/IT consultant and SANS Institute Faculty Fellow. He once developed the perfect kernel-level rootkit, but it was so good at hiding itself that it disappeared.