Intrusion Detection FAQ: What are telltale signs that a Unix system has been compromised?

Earlier this year I wrote a real-world story for Linux Gazette describing a way to search a computer for evidence of a cracker. I received a great deal of positive feedback which has prompted me to write this article detailing a more abstract version of the process I use, including the reason for each step, rather than a simple cookbook walk-through. When searching for a cracker, I look at four distinct, yet related areas. These are the filesystem and timestamps, logfiles, the contents of unrecognized files and their functions.

Filesystem analysis centers around the use of 'find / -name "*" -o -name ".*" -ls' or 'ls -RalF /'. These two commands generate roughly equivalent output; a recursive listing of every file with attributes and file type on all mounted filesystems. I trust find more than I trust ls since ls seems to be a part of every rootkit out there and as such is liable to "miss" things. Find also has a few more options to allow you to search for specific creation, modification and access times. I've also heard that ls changes the access times of a file, while find doesn't. I've yet to prove that true or false, but I suspect it depends on the version of ls used. Once you have a listing of all the files, you can seach for certain things that may be out of place. A directory called '...' is out of place, so is /root/{ls,ps,rm,mv,cp,*sh}, as are things like linsniff, crack, or anything else you wouldn't normally expect in a certain location. Intrusion detection is about finding anomalies, and this is what we are doing here. Root should not have '.' or '$HOME' in the path, or if those elements are in the path, they should be last. This prevents /root/ls from being run rather than /bin/ls and creating setuid root shells, hiding files or destroying them. Speaking of setuid and setgid files, those should be regarded with a healthy measure of paranoia as well.

Timestamps and logfiles should be viewed together. In my Gazette article I note that our cracker read manpages for various nfs subsystems, and the console error log showed an attempt to mount "/" remotely. This sort of correlation is what you are looking for. Checking the lastlog for odd login names is, admittedly a bit of a stretch, but you might find something. Perhaps the login times will indicate a particular user whose account has been compromised. Though each log is usually created by a different service, often some event will spill a few waves into other logs. The security log might complain about an unauthorized mount attempt, authlog shows some password guessing, mountd, nfsd, and the TCP wrappers all whimpering into syslog within a reasonable time window should raise some eyebrows. Also, process accounting may be useful. Inactive accounts should not be spawning processes and using up CPU time.

Returning to the issue of not leaving '.' or $HOME in root's path, we can consider the contents of a suspicious file. Often, rootkit binaries will include marker strings, passwords, readable text, or debugging information. Occasionally an executable will be linked against an older version of a dynamic library. I cannot think of any operating system that ships unstripped system binaries using outdated support code. Certain files such as the kernel are notable exceptions, but usually a file that violates one of the previous three conditions is not factory-issue. The file utility can be used to determine whether a file has been stripped or not; ldd, if it exists can reveal what libraries are expected, and strings will extract the ASCII text from a binary. Debugging information can easily quadruple the size of a file; oversized utilities can be a good indicator that something is wrong. If a file isn't stripped, a debugger such as GDB can be used to find out more about the file as well as what it does. I've seen one version of ls that, when run by root, would first create a setuid shell in /tmp/.trash and then run as normal. It seems that most rootkits for Linux seem to be linked with libc5, not libc6 like the rest of the system. Rootkits may have changed, but library mismatches are fairly accurate indicators of something amiss.

Finally, when examining the function of an unknown executable, I suggest that it is unwise to merely run it and see what smokes, so to speak. Once again, a debugger comes in handy, and even a general examination of the program's flow will be useful. Tracing a program that claimed to be a tcp wrapper, I found that indeed it did a socket(), fork(), connect(), but it also did an exec() in a non-daemonly fashion. To safely test the unknown program, I had to make a miniature LAN out of two 'disposable workstations' and run on that. A disposable workstation is one where the contents of the disk is trivially reproducible, usually from a CD, and is designed so that one can be used as a sacrificial lamb to the Gods of Network Security. By portscanning the machine before and after running this program, I found that it opened a back door on a high-numbered port, wired onto a root shell. The program also rewrote it's own argv[] array in order to hide from a process listing. For other rootkit binaries, a simple chroot hole will do. Build a bare minimum system, chroot to that virtual system and play with the suspect utilities there. While not perfectly secure, this will tell you what to look for, and what sort of traps you may be up against.

After carefully cataloguing one suspect machine, you now have a checklist of things to search others for. In a manner of speaking, you know the cracker's modus operandi. A portscanner in a shell script can quickly flag machines with backdoor daemons running. NFS exports should be reviewed if there is evidence that illegitimate exports were attempted. And similiar suggestions apply to other suspicious things. One good way to prove that something just has a funny name is to see if the file exists on similar machines. Only after seeing the rpc.snoopd program on fresh install of IRIX would I believe that that really was a system binary.

References

C. Kuethe, "Through the Looking Glass...", Linux Gazette
Seattle, 1999. http://www.linuxgazette.com/issue36/kuethe.html


--------------------------------------------------------------------------------

Disclaimer
: While every effort has been made to ensure the accuracy of the information presented here, this document is not a substitute for competent professional advice. This information is presented as a guide on an "as-is" basis; all warranties of fitness for a particular purpose, either implied or otherwise are hereby disclaimed.