Paul A. Henry Forensics and Recovery.com Follow me on Twitter
As a follow up to my recent SANS Forensic Blog post "How To — Digital Forensics Copying A VMware VMDK" that provided insight in to making a "GUI tool" based copy of a VMware VMDK, I have put together a How To that addresses creating a forensically sound image of a VMware VMDK on the ESXi console, that is able to provide the "chain of custody" needed in a digital forensics investigation.
Important note: In the simplest of terms a VMDK is an abstraction of a physical disk for a VM contained within a file (VMDK-flat). We are making a bit by bit copy of this file without the use of a traditional write blocker - however we are not altering any of the data contained "within" the file. It is expected that worst case in our efforts we may alter the time stamps of the underlying VMDK file itself but we will not alter any of the time stamps or data contained "within" the actual abstraction of the disk for the respective VM.
As noted in my previous post, the RedHat Linux console in ESX is not available in ESXi having been replaced with a "limited" BusyBox Linux console that is officially "un-supported"(Figure 1) by VMware. In order to enable the un-supported console you have to jump through a few hoops; enable the root password (Figure 2), enable tech support (Figure 3) and lastly if you plan to use SSH to access the un-supported console, you need to enable SSH (remember this opens a potential security hole) (Figure 4). There is a nice capability in 4.1 to limit the amount of time that the Tech Support console is available (Managing Tech Support Timeout) , however as we have no firm idea on how long the imaging process is going to take we will have to leave the timeout disabled.
While it is possible to image a running VM it is not practical from a forensics perspective as the VMDK for a running VM will be under going constant change and it will be impossible to validate the image integrity (before/after MD5 hashes) in our effort to establish evidence integrity. It is suggested to "suspend" the VM (Figure 5) within vSphere or directly via the vSphere client before creating the dd image. While it eliminates the integrity issue associated with validating the respective MD5 hashes it also provides a bonus - VMware creates a VMSS file that represents a snapshot of the VM's RAM at the moment the VM was suspended. This VMSS file can potentially contain valuable forensic evidence of what was occurring in the running VM RAM and should be imaged along with the VMDK file for possible extended forensic analysis.
With the console enabled we can log in to the ESXi console (Figure 6) and begin the work of creating the forensically sound images of the VMDK and VMSS files. The first step is connecting a NAS device that will be used as the destination for the images. I use a large RAID 5 NAS (4TB) for my images and it is simply a matter of setting the IP address in my NAS so it is available on the same network as the respective VM and literally plugging it in. I like to make sure the address of the NAS is reachable with a simple ping prior to trying to connect to it with ESXi (Figure 7). Once I determine it is reachable, I use the esxcfg-nas command (Figure 7) in the ESXi console to connect ESXi to the NAS. With the connection made we can verify it by doing a quick check at the command line, making sure that the NAS is actually displayed correctly within ESXi under /vmfs/volumes/ (Figure 8).
Before beginning our imaging it is a good idea to check / create the md5 hash values of the respective files using md5sum. I store each value in a respective .md5 file in the directory for the VM (Figure 9). With MD5 hash values established from before our forensic bit by bit copy using dd it is time to create the actual dd images and send them to the NAS. I will start with the VMDK file (Figure 10) and then do the VMSS file (Figure 11). After the images are complete we copy the respective .md5 files from the VM directory over to our NAS (Figure 12) as well.
With our imaging and hashing completed we can take a quick look in our NAS directory to see that all files have been created (Figure 13). Next I use an add-on Windows 7 utility HashTab 3.0 that integrates with the Windows Explorer and lets me right-click on a file in any directory and select properties to generate the MD5 and SHA hash for the respective file. In the directory on the NAS using Windows Notepad I open the individual .md5 files that were created from the original files on ESXi and compare the md5 values of the created images to the MD5 hash of the original evidence to validate integrity of the dd images on the NAS (Figure 14). This is a critically important step as it ensures our digital evidence copy matches the original evidence.
IMPORTANT: Once evidence collection is complete, do not forget to disable Trouble Shooting Mode (console access) and SSH (if you enabled it) in ESXi and resume the suspended VM.
With valid dd images of our VMDK and VMSS you can now create copies (always validate the hashes) to work with your favorite analysis tools (never work with your original dd images). For the sake of illustration I created a case in FTK3.1 and imported the VMDK dd image in as evidence. Note the file directory tree of the VM displayed in FTK as well as the verification of Image Integrity, which validates that the MD5 of the evidence in FTK matches the original evidence MD5 hash (Figure 15).
