Fun with FIFOs (Part II): Output Splitting

Several months ago now, I wrote up a little article on using FIFOs to trick the script command into writing output over the network. But there are other neat hacks you can do with FIFOs, and I want to show you one right now that can save you lots of time.

Suppose you had a disk image and you wanted to pull out both the ASCII and Unicode strings from a specific partition. The classic approach is to read the partition twice- once to gather the ASCII strings and once to pull out the Unicode. But on a large partition, reading the image even once can take a huge amount of time. The good news is we can use some Unix FIFO magic along with the frequently overlooked tee command and only have to read the partition once.

Let me show you the magic incantation first, and then explain what's happening:

$ mkfifo part-output
$ strings -a -t d -e l part-output >strings.uni
[1] 23281
$ dd if=drive-image.dd bs=512 skip=273168 count=160587 | 
    tee part-output | strings -a -t d >strings.ascii
160587+0 records in
160587+0 records out
82220544 bytes (82 MB) copied, 2.82945 s, 29.1 MB/s

My first command is simply creating a FIFO object called "part-output". You can call this FIFO anything you want and put it in any directory you want- the naming convention is completely irrelevant.

Next I run the strings command which is going to dump out our Unicode strings (see the "-e l" on the command-line? that's how you tell strings to look for Unicode). I tell it to read it's input from the part-output FIFO and write the strings to a file called strings.uni (">strings.uni"). The trailing ampersand ("&") tells the command shell to run this process in the background so I get my command prompt back. But nothing is going to start happening with this process until I start putting data into the FIFO.

Putting data into the FIFO is the magic on the next line. First I'm using dd to carve a particular partition out of my disk image and send the data to the standard output (see my recent article on manipulating disk images for more information on where this dd command-line comes from). I send that output to the tee command, which is a simple utility that reads data on the standard input and writes that data both to the standard output and to a file simultaneously. Think of it like a "T-joint" (hence the name "tee") in your plumbing or your ductwork: one input, two outputs. The problem is that one of those outputs has to be a file, which is where our FIFO comes in handy.

In this case, I tell tee to write the data into our FIFO instead of a regular file. All of this data is then read by the strings command we set up with the previous command to read the FIFO and output the Unicode strings. As for the other output channel from tee- the standard output- I just pipe that into another strings command which dumps out the ASCII strings. The tee command ensures both strings commands get the same input, and I get to extract both types of strings simultaneously. This means I only have to read through the partition once to get all the data I need. And that's a huge time-saver.

Now I only needed to split the output into two streams for this example, but there's nothing that says you couldn't create multiple FIFOs and use multiple tee commands to accomplish further splitting. As a silly example, let me simultaneously create the two strings files and compute an MD5 checksum over the partition data using three output streams:

$ mkfifo str-uni
$ strings -a -t d -e l str-uni >strings.uni &
[1] 23724
$ mkfifo str-asc
$ strings -a -t d str-asc >strings.asc &
[2] 23727
$ dd if=drive-image.dd bs=512 skip=273168 count=160587 | 
    tee str-uni | tee str-asc | md5sum
160587+0 records in
160587+0 records out
82220544 bytes (82 MB) copied, 2.71264 s, 30.3 MB/s
efafa128fa2f8ab0544b633c33f28559  -

First I create FIFOs named "str-uni" and "str-asc" and run a strings command against each FIFO to extract the different types of strings. We then run the dd command as before and pipe the output into tee which writes to one of the FIFOs we created above. But the standard output of the first tee goes into the input of a second tee command which writes to the other FIFO. The output of the second tee command gets fed into md5sum so we can get our checksum value (you can see it on the last line of output after the output from dd).

Keep stacking up FIFOs and tee commands and you can split your output into an arbitrary number of streams. Ultimately, of course, the CPU load of all the processing you're doing will set a limit on this. But if you've got a multi-core CPU, you've probably got more than enough processing power for typical tasks.

Anyway, the short summary is that FIFOs can save time and let you do lots of cool things you never thought you could do. It's a shame more people don't know about them. But the good news is that this lack of understanding in the general population makes you look like a big-time guru when you pull tricks like this out of your bag.

Hal Pomeranz, GCFA, is an Independent IT/Security Consultant and a SANS Institute Faculty Fellow. He will also never be a rich man, since he has apparently failed to grasp the fact that saving time is not a virtue when one is charging clients by the hour.