2 Days Left to Save $400 on SANS Albuquerque 2014

Intrusion Detection FAQ: Verifying Files with Red Hat's RPM

By Chris Brenton

It happens to all of us at least once. Something on your system does not seem quite right and you start wondering if someone has penetrated your defenses. The only way to find out for sure is to verify that none of your files have been changed. At this point you really wish you had found the time to install TripWire of some other file auditing tool.

Luckily the folks at Red Hat have developed a tool which can be quite handy in these situations. The program is called Red Hat Package Manager or RPM for short. There are package managers for other flavors of Linux as well as other Unix variations. For the purpose of this paper however we will focus on the Red Hat implementation.

What Can RPM Do For Me?

RPM is a great utility used to install, upgrade and verify software packages on your Red Hat system. It’s the verify feature that we are most interested in as this can be used to check our files and make sure they have not been modified or replaced. Besides file size and date/time stamp (which can be altered by a savvy attacker) RPM will also check the Message Digest or MD5 signature of the files.

MD5 is described in detail in RFC 1321. In short, MD5 takes the contents of a file and runs it through a mathematical formula. The result is a 128 bit signature that is unique to that file. Change the file in any way and the resulting signature changes. While it has been argued that it is theoretically possible to alter a file in such a way that it will still generate the same MD5 signature, to date no one has accomplished this task. This means that if you do a before and after check of a file’s MD5 signature you can be 99.9999% certain that the file has not been altered.

How Do I Use RPM to Check My Files?

There is a couple of RPM switches you will want to focus in on. The first is the "-V" switch which allows you to check the integrity of all files associated with a specific package. The syntax is:
rpm -V package_name_to_verify 


So if we had Sendmail running on the system and wanted to verify the files associated with this application we would type:

rpm -V sendmail 


The output we receive may look something like this:

[root@fubar /root]# rpm -V sendmail 
S.5....T c /etc/aliases
missing /etc/mail/ip_allow
S.5....T c /etc/mail/relay_allow
S.5....T c /etc/sendmail.cf
S.5....T c /etc/sendmail.cw
S.5....T /usr/sbin/sendmail
S.5....T /var/log/sendmail.st
[root@fubar /root]#

The only files that get listed are the files that fail the verification. Any files that are not listed are assumed to be OK. On the left of this output we have the reason why this particular file failed the verify check. A legend of the results would be as followed:

S = size change 
M = permissions change
5 = MD5 changed
L = Symlink changed
D = Device change
U = User change
G = Group change
T = Date/Time change
missing = file is gone


So if we check the output from our Sendmail verification, we see that the files aliases, relay_allow, sendmail.cf and sendmail.cw has changed. Note that size, MD5 and date/time has changed. This is a clear indication that these files have been modified. However since each is a configuration file, this is probably not a cause for concern. Its quite possible (in fact probable) that the default configuration was not exactly what we needed so these files where changed to suit our environment.

What is a cause for concern is that /usr/bin/sendmail has been changed. This is the executable which listens on port 25 to accept inbound mail connections. Unless the binary has been updated, it should not fail this verification check. Clearly someone has modified or replaced the original sendmail file. This is bad as the new version may contain a Trojan providing some unscrupulous attacker backdoor access to our machine.

Our output also indicates that ip_allow has been deleted or renamed. This should be checked into as well as this file is used to control the relaying of SPAM. This missing file may somehow be related to our sendmail binary being modified or replaced.

When reviewing RPM’s output, keep an eye out for very strange entries like a change in the MD5 value but the date/time stamp and/or file size is reported as being the same. This could be caused by an attacker who has modified or replaced files but is trying to cover their tracks.

It can be quite time consuming to check each package one at a time. To check all packages at once, use the "-a" switch as shown:

rpm -Va > /root/rpm_chk.txt & 


This command tells RPM to verify all installed packages. Instead of printing the output to the screen, the entries are stored in the file rpm_chk.txt for later review. The trailing "&" is optional and tells the system to run this command as a background process. This immediately frees up the command prompt so you can continue to check your system.

One last trick before moving on. Let’s say you run across a file and you are not sure which package uses it. In this case you can use the "-qf" option to see which package installed the file:

[root@fubar /root]# rpm -qf /usr/sbin/sendmail 
sendmail-8.8.7-20
[root@fubar /root]#


This tells us that the sendmail binary was installed as part of the sendmail-8.8.7-20 RPM package. If a file has no association, you would see output similar to the following:

[root@fubar /root]# rpm -qf /sbin/.vile_stuff 
file /sbin/.vile_stuff is not owned by any package
[root@fubar /root]#


Be very careful of any file running on your system that you can not verify!

Where to Begin

First, you must have root access to run RPM. While RPM will run from a regular user account, the information reported will be incorrect for any files to which you do not have at least read access. This means that unless you are root, you will not be able to check all system files.
v The RPM binary is stored in /bin while the RPM databases are saved in /var/lib/rpm.

The safest approach to copy these databases as well as the RPM binaries off to a CD before bringing the server up on line. This way you know that all of your tools are safe to use. For the purpose of this paper we’ll assume that no precautions have been taken and you are stuck working with a suspect RPM system.

The first thing to do is check the contents of /var/lib/rpm. The databases should have a date/time stamp of when you originally installed the system. If you see a different date, be suspect of the integrity of the databases.

Next, we can actually use RPM to verify its own integrity. You should see output similar to the following:

[root@fubar /root]# rpm -V rpm 
[root@fubar /root]#


The lack of output tells us that RPM appears to be OK. Remember that you are using a suspect binary to check itself. This is not the safest practice and is not recommended under normal circumstances. When possible, use tool which have been secured to CD. If you do not have any secured tools to work with however, having rpm check itself is better than nothing.

Now that we know RPM appears to be OK, we are ready to run a check on our system:

rpm -Va > /root/rpm_chk.txt & 


Once the check is complete we can review the rpm_chk.txt file for any unusual entries.

A slick trick is to generate a verification file prior to hooking your system up to the wire. You can then run later verifications and run a file compare between the new and old verification files. This keep you from having to review the entire file and allows you to focus in on just the changes made since your last audit.

Conclusion

While RPM was not specifically designed to be a file auditing tool, it can fill in when your in a pinch. The nice thing about RPM is that it is available on all recent versions of Red Hat. This means that if you are working with a Red Hat installation, the tool is already there and ready to use. The MD5 verification provides a high degree of accuracy when checking for file changes. The only caveat is that you have to be sure that RPM and all its databases have not been altered by an attacker in an effort to cover their tracks.