Gh0st in the Dshell: Decoding Undocumented Protocols

While many types of malware use well-documented protocols, such as HTTP, HTTPS, or IRC for command and control, any network traffic analyst will eventually encounter malware that uses an undocumented, custom protocol. This traffic is sometimes encrypted but often relies on simple obfuscation techniques or security through obscurity to avoid detection. These protocols must be decoded to understand what an attacker is doing on a victim system and develop signatures to detect it. The art of reverse-engineering undocumented network protocols can be a difficult and time-consuming process, but can be greatly simplified by using Dshell, a network traffic analysis framework developed by the US Army Research Lab and recently released open-source to the information security community. Dshell comes with a number of powerful built-in decoders, but also allows analysts to write custom decoder and parser modules for new network protocols. This powerful and extensible framework will prove a valuable tool for decoding many protocols not readable by other tools. This case study will demonstrate the process of reverse engineering a command and control protocol and writing a Dshell decoder for it, using the Gh0st remote access Trojan (RAT)'s proprietary network communication protocol as an example.