Directory Link Counts and Hidden Directories

One of the things I love about teaching at SANS is that the students are smart people and come up with great ideas. Sometimes these ideas even lead to useful tools, as was the case a few years ago when we were talking about hidden directories in the Digital Forensics section of SEC 506.

First, a little background information. Unix file systems keep track of a "link count" to all objects in the file system. This "link count" value is the number of different directory entries that all point to the inode associated with the object. In the case of a regular file, the link count is the number of hard links to that file.

However, Unix file systems don't let you create hard links to directories, yet the link count on a directory is always at least two, and even increases by one for each sub-directory in that directory. Why is this so?

• Any object in the file system must have a directory entry that connects it into the file system. For example, if you have a directory like "/tmp", there's a pointer in the root directory ("/") that points to the "tmp" directory entry. So that gives you one link.
• Every directory contains the "." link that points back to itself. So that gives us the minimum value of 2 links per directory.
• Every subdirectory has a ".." link that points back to its parent, incrementing the link count on the parent directory by one for each subdirectory created.

For one thing, the above behavior is why it's important to monitor the link count on critical directories in your file system using a file integrity assessment tool like Tripwire, Samhain, or AIDE. You can detect people adding or deleting directories when the link counts change.

But consider the following output from a compromised system:

# ls -a /foo
.  ..
# ls -dl /foo
drwxr-xr-x    3 root     root         4096 Jun 12 18:46 /foo

Our "/foo" directory is empty except for the normal "." and ".." links, meaning we'd expect the link count to be 2. Yet we see from the "ls -l" output that the link count on this directory is listed as 3 (look for the link count in the second column after the permissions flags and before the file ownership). What's going on here?

What's happening is that I've used a kernel-level rootkit to hide a subdirectory of "/foo". However the rootkit was not able to decrement the link count of the parent directory without causing a file system discrepancy that would show up the next time you fsck-ed the file system. In fact, I've never encountered a kernel-level rootkit that has attempted to mask the parent directory link count in any fashion when it hides a directory.

As my students pointed out, this suggests an obvious heuristic for detecting the presence of hidden directories on a system. Simply write a tool that traverses the entire file system searching for directories where the number of subdirectories in a given directory is not equal to the "link count minus two". While this technique will only tell you that a hidden directory exists and not necessarily give you the name of the hidden directory, it will pinpoint exactly where to start looking once you get a chance to analyze the file system image on a system that doesn't have the kernel-level rootkit loaded.

In any event, the tool was extremely simple to write. It's called "chkdirs" and it's now part of the chkrootkit distribution. And it's all thanks to some smart and interested SANS students.

Hal Pomeranz is an independent IT/Computer Security Consultant and a SANS Faculty Fellow. He believes that when the student is ready the master will appear... even if that master is one of your students!