Analyzing Shellcode Extracted from Malicious RTF Documents

During the analysis of malicious documents designed to exploit vulnerabilities in the programs which load them (thereby allowing the running of arbitrary code), it is often desirable to review any identified shellcode in a debugger. This allows an increased level of control and flexibility during the discovery of it's capabilities and how it implements the payload of the attack.

MalHost-Setup, part of the OfficeMalScanner suite allows the analyst to generate an executable which runs the shellcode embedded in malicious documents. To use this tool, we first need to determine the offset within the infected document, or extracted OLE file at which the shellcode begins, we then specify this offset as a parameter to MalHost-Setup when generating the executable. This executable can then be loaded into a debugger, allowing the analyst to step through the assembly instructions of the shellcode to understand it's functionality.

Shellcode techniques for locating secondary embedded payloads

The shellcode may be designed to search for a second stage payload or other embedded artifacts elsewhere in the originating file. This may be the case if the buffer being exploited was limited in size, the malware author may have placed the secondary stage shellcode, or perhaps even an embedded obfuscated executable, elsewhere in the document within a buffer that has significantly greater capacity.

In order to locate the specific offset within the document that the secondary stage code resides, the shellcode may try to locate itself either in memory or on the hard drive and then use the known offset to the next piece of code to make reference to, extract and execute it. One way this can be achieved (that I've recently seen) is by identifying and making use of the handle which refers to the document from which the shellcode originated, which would typically have been created by the program which loaded the file. A popular way to find the handle is to iterate through all possible handle values and making use of the Microsoft Windows GetFileSize API call which is designed to return the file size related to the specified handle. As the author knows the expected size of their malicious document, they are able to hard code this in, enabling this process to take place. Therefore, it doesn't matter where on the hard drive or in memory the malicious document resides.

This can be demonstrated using the assembly pseudo-code below:

At this point, you would encounter issues if the shellcode was being run from from a new file. In the case of a malicious RTF, this could be an OLE object extracted using RTFscan rather than the original file, which would inevitably have a different size to the original document. Therefore the handle to the original document would not be found in the context of the process, the referencing of embedded artifacts would fail, and this would hinder our analysis.

So here is what I did when I encountered this problem (+ releasing an open source tool!)

A potential solution would be to create a handle to the original file within the newly formed process, as this would allow the shellcode to make reference to the original document and extract the data it requires. Without the source code to MalHost-Setup, this is slightly more difficult, but we can achieve this using a capability built into Windows which allows handles from a parent process to be inherited to any child processes launched, the steps to achieve this are listed below.

1. Create a handle to a file using the 'CreateFile' API call
2. Launch a new process using the 'CreateProcess' API call, specifying the security parameters to enable the child to inherit the parent's handles.

I have written a small tool named 'HandleInheriter.exe' which takes an executable to launch and file to which we want a handle to be created as parameters and launches using the method detailed above. This is available along with it's C++ source code from my github repository by here [source code] [binary]

I've also been experimenting with re-writing an open source version of Malhost-Setup from scratch (called jmp2it) with the above functionality implemented as an option, if you're interested the current version can be downloaded here [source code] [binary] comments would be greatly appreciated.

Happy reversing!

— Adam

(I've just joined Twitter, follow me! @CyberKramer)