In the Linux/Unix realm we have tools like shred for securely overwriting files before deleting them in order to prevent recovery of the deleted file. If your adversary is sufficiently advanced (or just not lazy), they can obviously use these tools to frustrate your forensic investigation. Previously, I had thought that shred removed all traces of the file from disk. But in the course of some other file system research I was doing, I've realized that there may be a few lingering artifacts.
Let me begin by creating a small file containing a pattern of all 1's and using the istat tool from the Sleuthkit to get a list of the blocks in the file:
$ for ((i=0; $i<20480; i++)); do
echo -ne \xFF\xFF\xFF\xFF;
done >testfile
$ df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/Root-Home 599G 158G 411G 28% /home
$ ls -li testfile
9863292 -rw-r--r-- 1 hal hal 81920 Dec 5 07:59 testfile
$ sudo istat /dev/mapper/Root-Home 9863292
inode: 9863292
[...]
Direct Blocks:
9881605 9881606 9881607 9881608 9881609 9881610 9881611 9881612
9881613 9881614 9881616 9881617 9881619 9881620 9881621 9881622
9881623 9881624 9881625 9881626
Indirect Blocks:
9881618
Now I'm going to use shred to overwrite this file with nulls and then delete it. Then I'll dump the blocks where the file used to reside and see what's there.
$ shred -u -z testfile $ sudo blkcat -h /dev/mapper/Root-Home 9881605 20 0 00000000 00000000 00000000 00000000 .... .... .... .... 16 00000000 00000000 00000000 00000000 .... .... .... .... 32 00000000 00000000 00000000 00000000 .... .... .... .... [...] 53232 00000000 00000000 00000000 00000000 .... .... .... .... 53248 13c89600 14c89600 15c89600 16c89600 .... .... .... .... 53264 17c89600 18c89600 19c89600 1ac89600 .... .... .... .... 53280 00000000 00000000 00000000 00000000 .... .... .... .... [...] 81872 00000000 00000000 00000000 00000000 .... .... .... .... 81888 00000000 00000000 00000000 00000000 .... .... .... .... 81904 00000000 00000000 00000000 00000000 .... .... .... ....
The vast majority of the original file content is showing up as all nulls- this is what we would expect. But it looks like there's some residual data in the middle of the file. What's up with that?
This file system has a block size of 4096, so byte offset 53248 corresponds to the beginning of the 13th consecutive block of the original file. That would be block 9881618 in our example, which your can see from the istat output above is listed as an indirect block. I've written about indirect blocks on this blog previously, and you can read my earlier article for more detail, but basically these indirect blocks are file metadata that ends up being placed in the midst of the block runs that make up the file. shred is apparently unable to clobber this information.
In hindsight, I feel like I should have realized that this behavior would happen. If you look at the shred source code, it simply opens the file like any normal user process, jumps to the beginning of the file, and proceeds to overwrite the file from beginning to end. As a normal user process, it has access to the data blocks that make up the file content, but not the indirect block metadata. The file system drivers in the OS "hide" the indirect blocks from the shred program. A program that wanted to clobber the indirect block data would have to have superuser privileges so that it could open the raw disk device and attack the necessary blocks directly. Normal user space programs like shred aren't going to have this level of access.
Well this is all very interesting in an abstract knowledge kind of way, but why am I writing about it in the SANS Forensics Blog? Consider that this finding means that even when shred is used, there is residue from the original file left on disk. As I hinted in my previous article about indirect blocks, it is possible to write a signature for finding indirect blocks in unallocated space ("look for a block whose first four bytes are the address of the next block" works 99% of the time). So if you find part of a file system that's largely "empty" but for a bunch of what appear to be indirect blocks, that may be an indicator that a disk wiping tool like shred was used. Depending on the circumstance, deleting evidence can be construed as obstruction, which may allow you to get a conviction (even in the absence of other evidence of crime).
The other implication, of course, is that if you're "wiping" data off a drive using shred, you should just be aware that there actually will be residue left on the drive. If you really want to wipe a drive, you're better off using dd as root to write zeros over the entire drive via the raw disk device.
Hal Pomeranz is an Independent IT/Security Consultant, a SANS Institute Faculty Fellow, and a GCFA. Yes, he really does do this sort of thing for fun.
