SANS - Information Security Resources

By: Joseph (Randy) Taylor

Overview
Robert Grahamâs release of Sidestep in early February 2001 identified basic Intrusion Detection System (IDS) evasion techniques in the BackOrifice application, DNS (Domain Name Service), FTP (File Transfer Protocol), HTTP (Hyper-Text Transfer Protocol), RPC (Remote Procedure Call), and SNMP (Simple Network Management Protocol). Sidestep demonstrates only one method of evading signature-based IDSes using the RPC protocol. This paper goes deeper into RPC to show how it operates, its inherent weaknesses in terms of network security, a full disclosure of all known evasion techniques, and solution sets to counter these weaknesses for both network-level security and IDS technology.

We will present this information "in plain English" first - the technical details will be included in the Appendices. "" The key points of this discussion are:

There are an infinite number of possible ways to evade IDSes using the weaknesses inherent in the RPC protocol.
One evasion method constitutes a Denial-of-Service (DoS) attack against RPC servers and scales easily and rapidly into a full Distributed Denial-of-Service (DDoS) attack. This evasion method also contains DoS and DDoS implications for all forms of IDSes.
Solution sets from both network-security and IDS technology perspectives are, fortunately, relatively simple to implement and effective.
The relative threat severity of these attacks is, at most, medium.

The RPC Protocol "In Plain English"

What the RPC Protocol Does
According to the Merriam-Webster Dictionary, a protocol is "a set of conventions governing the treatment and especially the formatting of data in an electronic communications system". Put simply, a protocol defines the rules a client must follow to communicate with a server (and vice-versa) for a particular type of network service offering.

The chain of events for the RPC protocol begins in OSI Layer 5 (Session Layer) with the core code of the RPC communication conventions. At OSI Layer 6 (Presentation), XDR (eXternal Data Representation) encodes the input to and decodes output from the portmap/rpcbind server at OSI Layer 7 (Application). For the purposes of this paper, portmap and rpcbind are equivalent services with different names because they run on different versions of UNIX. They both run on TCP and UDP port 111 and/or 32771 through 32779. You can think of portmap/rpcbind as a phone book that RPC-based programs must use to talk with each other. The portmap (Linux, xBSD) / rpcbind (Solaris) serverâs purpose is to act as a central registration facility for all other RPC-based services, such as NFS (Network File System) and NIS (Network Information System). When these services start up, they register the ports they will be listening on with portmap/rpcbind. When an RPC call comes in from another application or service, it must ask portmap/rpcbind for the port of the service it wants to talk with. When portmap/rpcbind returns this information to the caller, the caller will use it to begin direct communication with the service it was seeking. The evasion and attack techniques we will discuss target portmap/rpcbind.

A Normal RPC Request-for-Information "Conversation"
The rpcinfo utility is provided in UNIX for querying portmap/rpcbind for information it has about the RPC-based services registered with it. The -p option to rpcinfo asks portmap/rpcbind to return all of the information available about every service registered with it. rpcinfo -p is often used by hackers or system crackers to gather information about what RPC services are offered by a given host so that they can decide which, if any, RPC-based attack methods they have at the ready will apply to the targeted host.

A normal rpcinfo -p command equates to the following English conversation:

rpcinfo: "This is the last fragment of data I have to send you. The data fragment is 40 bytes long. I want to use this tracking number to keep our conversation synchronized. I am placing a call message. I am using RPC Version number 2. I need to speak to portmap/rpcbind and I expect its program version number to also be 2. I am asking portmap/rpcbind to give all the information it has about every registered service on the local host. I am not using any authentication or verification mechanism."

portmap/rpcbind: "Because the data you are sending me represents the last fragment of data you have to send I can begin processing it. This is the last fragment of data I will be sending to you and it is 988 bytes in length. I acknowledge your conversation tracking number by sending it back to you. I am sending a reply message to your request. Because your input was formatted properly I am accepting your call message. Because you presented me no authentication or verification data, I wonât use any, either. Your call message executed successfully. I have the information you requested. Each piece of information will be of different lengths. When I have completed sending you all of the information I have, I will tell you that there is no information left to send and our conversation will be concluded."

A Sidestep RPC Request-for-Information "Conversation"
The Sidestep program also performs an rpcinfo -p request, but by a different method that attempts to evade IDSes by changing the pattern of the conversation.

Sidestep: "This is not the last fragment of data I have to send you. This data fragment is one byte long. Here is that data byte. I will repeat this message 39 more times"

portmap/rpcbind: "Because the data you are sending me does not represent the last fragment of data you have to send, I will not be able to begin processing your request until you tell me that you have sent the last data fragment"

Sidestep: ..this is the last data fragment. It is one byte long. Here is that byte. My message is complete."

portmap/rpcbind: ..it took a little longer than usual, but I now have all of the data you sent me, so I can begin processing it. This is the last fragment of data I will be sending to you and it is 988 bytes in length. I acknowledge your conversation tracking number by sending it back to you. I am sending a reply message to your request. Because your input was formatted properly I am accepting your call message. Because you presented me no authentication or verification data, I wonât use any, either. Your call message executed successfully. I have the information you requested. Each piece of information will be of different lengths. When I have completed sending you all of the information I have, I will tell you that there is no information left to send and our conversation will be concluded."

Beyond Sidestep
The conversation pattern used by Sidestep is static from one invocation to the next. That fact allows a signature-based IDS to identify its technique and generate an alert. Unfortunately, the static technique used by Sidestep is not the only one possible. Here is a variation on the evasion theme.

Hacked-up RPC Client: "This is not the last fragment of data I have to send you. This data fragment is seven bytes long. Here are those bytes. This is not the last fragment of data I have to send you. This data fragment is three bytes long. Here are those bytes. This is not the last fragment of data I have to send you. This data fragment is five bytes long. Here are those bytes. This is not the last fragment of data I have to send you. This data fragment is nine bytes long. Here are those bytes. This is not the last fragment of data I have to send you. This data fragment is two bytes long. Here are those bytes. This is not the last fragment of data I have to send you. This data fragment is six bytes long. Here are those bytes. This is not the last fragment of data I have to send you. This data fragment is seven bytes long. Here are those bytes. This is the last data fragment. It is one byte long. Here is that byte. My message is complete."

portmap/rpcbind: "Because the data you are sending me does not represent the last fragment of data you have to send, I will not be able to begin processing your request until you tell me that you have sent the last data fragment"

portmap/rpcbind: ..it took a little longer than usual, but I now have all of the data you sent me, so I can begin processing it. This is the last fragment of data I will be sending to you and it is 988 bytes in length. I acknowledge your conversation tracking number by sending it back to you. I am sending a reply message to your request. Because your input was formatted properly I am accepting your call message. Because you presented me no authentication or verification data, I wonât use any, either. Your call message executed successfully. I have the information you requested. Each piece of information will be of different lengths. When I have completed sending you all of the information I have, I will tell you that there is no information left to send and our conversation will be concluded."

From the Sidestep conversation and a variant, it becomes clear that any conversation (evasion) pattern can be created, provided the following rules are followed:

All fragment message byte values used must add up to 40.
All data bytes must be encoded properly in the message.
All data fragment messages except the last must indicate that more data fragments follow and must properly indicate the number of data bytes that follow.
A single last fragment message must be sent with valid length and associated data bytes.

Item (1) above implies that the total numbers of possible conversation variations are 40! or 8.159 X 10⁴⁷. A signature-based IDS cannot possibly cope with such a large number of signatures even if one could write them all. That forces signature-based IDSes to incorporate a RPC protocol decoding algorithm to extract the "normal conversation" and run that result through signature-matching analysis. We have observed that the minimum number of bytes for a request-for-information conversation (a normal rpcinfo -p command) is 44 bytes of data. Sidestep uses the maximum number of bytes for its static variant, which comes in at 200 bytes. All conversation variants will use no less than 44 bytes and no more than 200 bytes. From a decoding standpoint, this is not a lot of data to manipulate, so the IDS RPC protocol decoder should not incur a significant additional workload. Unfortunately, there is one conversation technique that is toxic to both portmap/rpcbind and IDSes that employ RPC protocol decoding.

The Toxic RPC Request-for-Information "Conversation" a.k.a. "Null-Byte Encoding"
In the preceding discussions, we have always assumed a set of logical conditions for RPC fragment messages - that we were or were not sending the last fragment of data and that the data bytes had some length greater than zero. What happens when you set data length to zero? Hereâs what that conversation looks like - think Jack Nicholson in The Shining ;-)

Hacked-up RPC Client: "This is not the last fragment of data I have to send you. This data fragment is zero bytes long. This is not the last fragment of data I have to send you. This data fragment is zero bytes long. This is not the last fragment of data I have to send you. This data fragment is zero bytes long ...

(Millions of such messages later)

This is not the last fragment of data I have to send you. This data fragment is zero bytes long. This is not the last fragment of data I have to send you. This data fragment is zero bytes long. This is not the last fragment of data I have to send you. This data fragment is zero bytes long. This is not the last fragment of data I have to send you. This data fragment is zero bytes long....
(While this is happening, portmap/rpcbind is thinking ...)

portmap/rpcbind: "I cannot do anything but wait until you notify me that the last data fragment has been sent. I cannot talk to anyone else until your conversation has been completed."

The conversation constitutes a denial-of-service attack against both the remote portmap/rpcbind server and the IDS that is attempting to decode the conversation. In our experiments with "null-byte" conversations, we have found that portmap/rpcbind will deny connections to any other RPC-based application that needs its services while the toxic conversation is taking place. We have not found that portmap/rpcbind will crash, but we have not pushed the limits very hard. The largest single null-byte encoded stream we have tried came in at 400 million bytes. This attack denied service to or from portmap/rpcbind for about 10 minutes. If you think of this conversation as a module that can plug in to DDoS programs like TFN, Trin00, or Stacheldraht, youâll see that the null-byte technique quickly and easily scales out to a network-spanning DoS attack that could consume significant bandwidth. In single-host DoS and network DDoS cases, the IDSes attempting protocol decode on this technique are likely to be overwhelmed.

Further, we have found that legitimate queries and evasions can be embedded in null-byte message streams. If the query or evasion is formed correctly, portmap/rpcbind will respond eventually. The embedding can be inserted as either a single "chunk", or it can be interspersed throughout the null encoded stream. That capability increases the number of possible evasions to infinity, for all practical purposes.

Observations, Solutions, etc.

Observation: The core problem with RPC hinges on its message fragmentation feature. This is called the âLast Fragment/Fragment Lengthâ message (or LF/FL for short). portmap/rpcbind will not begin processing a request until it know it has its last data fragment. Null-Byte RPC encoding (Last Fragment set to No and Fragment Length of Zero) appears to be an oversight on the part of the developers of the RPC protocol. This oversight allows denial-of-service attacks against individual hosts offering RPC services and scales out to network DDoS bandwidth-consumption attacks.

Solution(s): There are, fortunately, a lot of good ways for networks to mitigate against null-byte RPC flooding:
- Block TCP port 111 and 32771 through 32779 (portmap/rpcbind) at the network head-end. We would also suggest a careful analysis of internal-to-internal network requirements - where TCP ports 111 and 32771 through 32779 are not required for intranetwork communications, block them on internal routers as well.
- Deploy Wietse Venemaâs replacement portmap/rpcbind on all UNIX hosts offering RPC services. When Wietseâs replacement portmapper is used, inbound null-byte streams are killed almost immediately if the calling host is not authorized to make the connection.
- The RPCSEC_GSS (RFC 2203) credentialing and authentication mechanism appears to hold some promise, but we have not seen an implementation of its features in an active network. It is not clear at this point whether or not an RPCSEC_GSS-compliant null-byte or evasion message could be created. It is also not immediately clear what advantages RPCSEC_GSS holds over non-AUTH_NULL RPC authentication mechanisms in terms of this type of attack.
These are obvious workarounds, but they donât really go to the core of the problem. A better solution would be to compensate for null-byte messages in the RPC protocol itself through a bugfix or algorithm redesign. This could be accomplished in at least two ways. One would be to detect and kill the null-byte message immediately upon receipt and shut down the offending session. This introduces another hidden denial-of-service possibility if a small null-byte message is sent expressly for the purpose of making portmap/rpcbind close down communications with an otherwise legitimate client. The second method would also detect null-byte, but instead of shutting down the connection, it would instead write a report of the event to syslog. From there, a host-based IDS could pick up the syslogged notification and report to its server or management console. This solution adds case-specific IDS functionality to the RPC protocol.

We have not found a legitimate need for null-byte messages in the normal course of RPC traffic. We have also not found a case in which the "Last Fragment" portion of an inbound RPC message should be set to "No", either, even if it has a valid "Fragment Length" and associated data bytes properly encoded. Both of these conditions could be detected in the RPC protocol stack and reported to syslog when they occur as "Protocol Evasion" attempts. The "Protocol as NIDS" concept may be able scale out to encompass protocols other than RPC. This has the advantage of turning some portion of network core communication mechanisms into malicious activity detectors.

Observation: IDS Evasion and DoS/DDoS are possible only in TCP. UDP is not at risk.

portmap/rpcbind runs on TCP and UDP at ports 111 and 32771 through 32779. The reason UDP is not affected is because LF/FL messages are not used in UDP-based conversations. Data cannot be fragmented, so null-byte encoding is not possible in UDP. Encoding variations targeted at IDS evasion are also not possible in UDP-based messages.

Observation: There are far too many valid RPC encodings for a signature-based IDS to handle.

Solution: Detect portmap/rpcbind replies instead of requests. While there are practically an infinite number of possible rpcinfo -p encodings, there is only one reply - the portmap/rpcbind Dump Reply message discussed earlier. That reply will be consistently detectable by an IDS, unless the portmap/rpcbind host server has been compromised - in that case, there are much larger security problems to deal with than just protocol evasions. The IDS may not know what evasion method was used, but it will know a reply was sent. If the reply does not correlate with a normal rpcinfo -p request, then an evasion must have been used.

Observation: IDSes that employ strict protocol decoding procedures open themselves up to the potential of resource-starvation and denial-of-service when processing null-byte RPC messages.

Solution: An RPC-specific protocol decoding algorithm should be used that takes malicious encodings into account. The basic algorithm is as follows:

Assumptions: Inbound data is directed at the portmap/rpcbind program (TCP ports 111 and 32771-32779)
IP and TCP Headers are processed/stripped elsewhere

ALLOCATE BUFFER of variable length
LOOP
   READ LAST FRAGMENT/FRAGMENT LENGTH
     CASE

       LAST FRAGMENT === NO and FRAGMENT LENGTH ==
       Zero
         FLAG Level Two RPC Protocol Evasion
         STOP Processing this packet
         BREAK OUT OF CASE and LOOP
       LAST FRAGMENT != YES AND FRAGMENT LENGTH !=
       Zero
         FLAG Level One RPC Protocol Evasion
         READ number of bytes designated by FRAGMENT LENGTH into BUFFER
       LAST FRAGMENT == YES
         IF FRAGMENT LENGTH == Zero THEN
           FLAG Level One RPC Protocol Evasion
           SEND BUFFER to signature matching engine
           BREAK OUT OF CASE and LOOP
         ELSE READ number of bytes designated by FRAGMENT LENGTH into BUFFER
         SEND BUFFER to signature matching engine
         BREAK OUT OF CASE and LOOP
     END CASE
END LOOP
FREE BUFFER

Here is the same algorithm with comments:

ALLOCATE BUFFER of variable length
LOOP

# RPC calls are encoded by XDR which operates on data
# in chunks of length "n mod 4" (4, 8, 12, 16...bytes). The
# command strings are all four bytes, though.
   READ LAST FRAGMENT/FRAGMENT LENGTH      CASE

     #
     # If the four bytes are all zero, it means a potential
     # resource-starvation/denial-of-service attack. Just flag the protocol
     # violation and let the outbound signatures handle it.
     # Put simply, this is a NO-OP attack of potentially infinite
     # length and we don't want to go there. ;)
     #

       LAST FRAGMENT == NO and FRAGMENT LENGTH ==
       Zero
         FLAG Level Two RPC Protocol Evasion
         STOP Processing this packet
         BREAK OUT OF CASE and LOOP

       #
       # This is the Sidestep evasion or a variation of it.
       # This evasion can be embedded in a NO-OP attack or it
       # can stand on its own.
       # If we detect Sidestep followed by NO-OP, we break
       # out of it in the NO-OP section above to prevent DoS.
       #

     LAST FRAGMENT != YES AND FRAGMENT LENGTH !=
     Zero
         FLAG Level One RPC Protocol Evasion
         READ number of bytes designated by FRAGMENT LENGTH into BUFFER

       #
       # "80 00" in the first two bytes means we are at the
       # last fragment of data so we should prepare to wrap
       # things up...
       #

       LAST FRAGMENT == YES

         #
         # A Sidestep evasion can be constructed that would
         # encapsulate all of the valid data before the "last
         # fragment" flag is set. In that case, the fragment
         # length must be zero - we need to be aware of that.
         #

         IF FRAGMENT LENGTH == Zero THEN
           FLAG Level One RPC Protocol Evasion

           #
           # "80 00 00 0x" means the BUFFER now has the
           # last fragment of data (and that data is of
           # length "x" bytes). If "x" is zero, then there's
           # no need to write anything to the output
           # BUFFER and we're done.
           #

           SEND BUFFER to signature matching engine
           BREAK OUT OF CASE and LOOP

         #
         # Otherwise, go ahead and fill out the rest of the
         # output BUFFER and send it to the signature engine.
         #

         ELSE READ number of bytes designated by FRAGMENT LENGTH into BUFFER          SEND BUFFER to signature matching engine
         BREAK OUT OF CASE and LOOP
     END CASE
END LOOP
FREE BUFFER

The Bottom Line
At most, Iâd assess the risk of the vulnerabilities described in this paper as medium. There are too many good existing workarounds to warrant anything more alarmist. Block TCP and UDP ports 111 and 32771 through 32779 at the head-end and intranet-to-intranet, deploy Wietseâs portmap/rpcbind replacement and youâll be in pretty good shape at the local level. That doesnât help the long-haul networks much, but thereâs (arguably) a lot of junk on the wire already masquerading as valid content. ;-) This attack is just another brick in the DoS/DDoS wall from the perspective of practical network security.

For IDSes that employ strict protocol decoding, dealing with RPC protocol evasion techniques will be problematic. Null-byte streams could lead to resource starvation within the IDS itself. If a strict decode approach is pursued, we would expect the IDS to wait for a corresponding reply to the inbound query - and as we have demonstrated, that wait could be a long one. If numerous null-byte streams are sent simultaneously, the affected IDS will have numerous threads tied up in the decode process, hence the potential for resource starvation. Our results indicate that a modified protocol decode approach should be taken to avoid resource consumption. In essence, an IDS RPC decoder needs only to identify evasive fragmentation encoding techniques, flag that evasion attempt, and let its signature engine detect outbound replies to encoded queries. This modified decoding approach solves the resource-starvation problem, identifies evasion attempts, and detects query replies.

"as through a glass, and darkly"
Just some unanswered questions and other related thoughts:

There is a lot of work yet to be done analyzing the RPC protocol, and I doubt Iâll have a chance to get to it any time soon. For instance, while we know that initial RPC call setups must run though portmap/rpcbind, we donât know if the subsequent conversations that ensue after the handoff to RPC services such as mountd and the like can be evaded, DoSâed, etc. Chances are good that thereâs probably more than a few bogeys waiting to be found down there.
Are there really valid states in which Last Fragment can be set to No and carry data payloads? We havenât found any yet, but that doesnât mean they arenât supposed to be there. In the case of RPC calls through portmap/rpcbind, it would seem to me that fragmentation should be handled by IP - I donât see an immediate reason for allowing fragmentation inside RPC itself. Null-byte encoding is a Real Bad Thing, obviously. If UDP RPC doesnât need fragmentation support, why is it in TCP RPC?
Can any protocol be evaded? Thatâs where Iâm going next. If there are unintentional consequences in RPC, you can bet thereâll be some fun surprises elsewhere.
Can the concept of "Protocol as NIDS" be extended beyond RPC? Iâd like to think so because it puts intrusion detection capability into the core elements of the network itself... and that can only be a Real Good Thing.

Appendix A: Citation of Sources / List of References

Robert Graham
SideStep: IDS evasion tool
February 2001
http://www.robertgraham.com/tmp/sidestep.html
RFC 1057
RPC: Remote Procedure Call Protocol Specification Version 2
June 1988 Sun Microsystems, Inc.
http://www.rfc-editor.org/rfc/rfc1057.txt
RFC 1831
RPC: Remote Procedure Call Protocol Specification Version 2
R. Srinivasan August 1995 Sun Microsystems, Inc.
http://www.rfc-editor.org/rfc/rfc1831.txt
RFC 1832
XDR: External Data Representation Standard
R. Srinivasan August 1995 Sun Microsystems, Inc.
http://www.rfc-editor.org/rfc/rfc1832.txt
RFC 1833
Binding Protocols for ONC RPC Version 2
R. Srinivasan August 1995 Sun Microsystems, Inc.
http://www.rfc-editor.org/rfc/rfc1833.txt
RFC 2203
RPCSEC_GSS Protocol Specification
M. Eisler, A. Chiu, L. Ling September 1997
http://www.rfc-editor.org/rfc/rfc2203.txt
Power Programming with RPC
John Bloomer
OâReilly & Associates, 1st Edition February 1992
ISBN: 0-937175-77-3
OpenBSD Source Code
Various Distribution Revisions
http://www.openbsd.org
FreeBSD Source Code
Various Distribution Revisions
The FreeBSD Project
http://www.freebsd.org

What is polymorphic shell code and what can it do?

How does an attacker evade Intrusion Detection Systems with Session Splicing?

IDFAQ: Intrusion Detection System Evasion and Denial of Service Using RPC Design Flaws

Latest Whitepapers

Latest Tweets @SANSInstitute

Contact Us