Digital Forensics: PS3 Linux file system analysis and network forensics

Let me start by noting how much fun I had while investigating and analyzing everything for this forensics challenge, I was able to apply many different techniques, from analyzing logs to file carving and network forensics. 

The first thing I did was grab the images and pcap files from the DFRWS site and mounted those images on my workstation for analysis:

asm forensics # mount -o loop,noexec,ro /forensics/nssal-linux-side-fs.dd /mnt/ps3asm forensics # mount -o loop,noexec,ro /forensics/jhuisi-linux-side-fs.dd /mnt/ps3-jhs

At this point I tried generating a timeline but I ended up with two huge files, 240 MB and 75 MB, it would not be easy to go through that by hand so I tried to narrow down the date range by analyzing the images I had and trying to figure out when the suspicious activity started.

I started with "nssal-linux-side-fs.dd" and looked at the /etc/shadow file to see what users were on the system and I found these 3 user names:

root:$6$0GL.Dibt$.2dVOk9BcGVR6NJKdl.ZacR63CHB0hhJ3DBDRg/dT/XeULxdblHdOJWrjfP0QyInFcWlxXNreuIHre9VyMxxb1:14258:0:99999:7:::nssal:$1$Hf2Vk/7k$HikXkhyPtzxzJSXvSqTYf0:14217:0:99999:7:::jhuisi:$6$HqshGlKp$PwsgCEwcQ2u57aMkHvwLB25gWGrQSC.jvnPE5aOTAFbNdoLCs0GWVvXStAXWmWnMUfePK5BKY0u0q5RPtQVqU.:14314:0:99999:7:::

It would be possible to decrypt the password for the user nssal since it uses MD5 but it wouldn't be very useful to us. What I did was go through that user's home directory and I found some images that looked like Mardi Gras pictures (~4MB total). Nothing too suspicious there, so I went through the other user's home directory. There aren't many files there except .bash_history:

asm jhuisi # pwd
/mnt/ps3/home/jhuisi
asm jhuisi # cat .bash_history
ls
cd Examples
ls
cd
cd ..
ls
cd nssal/
ls
cd Images/
ls
cd
ls
./backdoor
ls -l
~/backdoor
ls -l
sudo nssal ./backd00r
su
netstat -an
netstat -an | more
telnet localhost 45541
ps auxww | grep klog
ps auxww | grep back
netstat -an | more
exit

Ok, this looks interesting. The user seems to have run a backdoor with root privileges, possibly listening on port TCP 45541 and with a process name of klog. These would be some good keywords for a string search. Using the stat command you can see that the history file has a modification date of 2009-03-11. This would be a good time to look at the system logs and find login times and ssh session start times, the files i looked at where /var/log/wtmp, /var/log/auth.log, /var/log/messages and /var/log/ConsoleKit/history.

asm log # last -f wtmp<p>nssal pts/3 128.220.249.83 Fri Aug 25 14:17 - 07:02 (195+16:44)
jhuisi pts/2 128.220.249.83 Wed Oct 8 01:01 - 19:50 (34373+18:48
nssal pts/1 137.30.251.152 Wed Jul 16 11:01 - 03:01 (-18057+-7:0</p>asm log # grep 128.220.249.83 auth.log<p>Mar 11 11:49:43 nssal-ps3 sshd[3208]: reverse mapping checking getaddrinfo for ps3.isi.jhu.edu.249.220.128.in-addr.arpa [128.220.249.83] failed - POSSIBLE BREAK-IN ATTEMPT!
Mar 11 11:49:45 nssal-ps3 sshd[3208]: Accepted password for jhuisi from 128.220.249.83 port 51874 ssh2
Mar 11 11:52:00 nssal-ps3 sshd[3266]: reverse mapping checking getaddrinfo for ps3.isi.jhu.edu.249.220.128.in-addr.arpa [128.220.249.83] failed - POSSIBLE BREAK-IN ATTEMPT!
Mar 11 11:52:02 nssal-ps3 sshd[3266]: Accepted password for jhuisi from 128.220.249.83 port 51942 ssh2
Mar 11 11:56:09 nssal-ps3 sshd[3326]: reverse mapping checking getaddrinfo for ps3.isi.jhu.edu.249.220.128.in-addr.arpa [128.220.249.83] failed - POSSIBLE BREAK-IN ATTEMPT!
Mar 11 11:56:11 nssal-ps3 sshd[3326]: Accepted password for jhuisi from 128.220.249.83 port 51075 ssh2
Mar 11 12:02:26 nssal-ps3 sshd[3369]: reverse mapping checking getaddrinfo for ps3.isi.jhu.edu.249.220.128.in-addr.arpa [128.220.249.83] failed - POSSIBLE BREAK-IN ATTEMPT!
Mar 11 12:02:30 nssal-ps3 sshd[3369]: Accepted password for nssal from 128.220.249.83 port 51109 ssh2</p>asm ConsoleKit # tail history<p>1236791463.488 type=SEAT_SESSION_REMOVED : seat-id='Seat5′ session-id='Session5′ session-type=" session-x11-display=" session-x11-display-device=" session-display-device='/dev/ssh' session-remote-host-name='128.220.249.83′ session-is-local=FALSE session-unix-user=1000 session-creation-time='2009-03-11T17:02:30.355257Z'</p>

I found a few lastlog entries for both nssal and jhuisi and two IP addresses, I also found successful logins from both users on 2009-03-11.I decided it was time to have a look at the other PS3's image (jhuisi-linux-side-fs.dd). Again, I began by looking at the user's home directories, in this case they were jhuisi and goatboy. Once again I found suspicious files and activity.

asm goatboy # cat /mnt/ps3-jhs/home/goatboy/.bash_history<p>ls
mkdir Recipes
ls
cd Recipes/
ls
tar xvf recipes.tar
ls customers/
mv Tír\ na\ nÓg customers/
ls
rm recipes.tar
exit
ls
ls -l Recipes/
exit
ls
cd Examples
ls
cd ..
clear
ls
cd Recipes
ls
cat bateman\'s
talk
ytalk
ntalk
write jhuisi
ls
cat custmers
write jhuisi
cat customers
cd customers
ls
clear
ls -l
cat Tír\ na\ nÓg
ls
write jhuisi
I feel younger already!
write jhuisi
write
write jhuisi
Nice doing business with you!
clear
ls
cd
clear
ls</p>asm jhuisi # pwd<p>/mnt/ps3-jhs/home/jhuisi</p>asm jhuisi # ls<p>Desktop Documents Examples Music Pictures Public Templates Videos a.out backd00r.c</p>asm jhuisi # strings a.out...<p>root
klogd -x
jhuisi
Backdoor by darkXside
Enter the second password.
Password accepted!
:Welcome!psyBNC@lam3rz.de NOTICE * :psyBNC2.3.2-4
[backdoor]#
/dev/.tty01
chdir
exit
See ya later...</p>asm log # grep goatboy auth.log...<p>Mar 11 12:49:25 ps3 sshd[5382]: Accepted password for goatboy from 137.30.123.40 port 34612 ssh2
Mar 11 12:49:25 ps3 sshd[5382]: pam_unix(sshd:session): session opened for user goatboy by (uid=0)
...</p>

Ok, now it's getting more interesting, we found what apparently was the source code for the backdoor that was used and its binary version, and also some interesting bash_history and ssh sessions for user goatboy. I have to admit, at first I thought it could not be that easy to find the source and binary for the backdoor, maybe it had been left there as a decoy, so I thought I had to go back and create my timeline with a start date of 2009-03-11, take a good look at it and then do some string searching on the file systems.I used The SleuthKit's fls to create what's called a body file from the images and then created a timeline with mactime.

asm forensics # fls -r -m / nssal-linux-side-fs.dd nssal-linux-side-fs.dd > nssal-linux-side-fs.bodyfileasm forensics # fls -r -m / jhuisi-linux-side-fs.dd > jhuisi-linux-side-fs.bodyfileasm forensics # mactime -d -b nssal-linux-side-fs.bodyfile -z CST6CDT 2009-03-11 > nssal-linux-side-fs.timelineasm forensics # mactime -d -b jhuisi-linux-side-fs.bodyfile -z EST5EDT 2009-03-11 > jhuisi-linux-side-fs.timeline

Why did I use timezones with mactime? Because the timeline needs to be on the same timezone than the system it was created for. How did I find out the timezone for each system?

asm etc # strings /mnt/ps3/etc/localtime<p>...
CST6CDT,M3.2.0,M11.1.0</p>asm etc # strings /mnt/ps3-jhs/etc/localtime
...
EST5EDT,M3.2.0,M11.1.0

Mactime generated two files (the -d option tells it to index files in comma separated format) which I opened with OpenOffice's Calc.Looking at the timelines from both systems I found evidence of a few interesting facts. On March 11, 2009 at approximately 12:49 EST (11:49 CST) sftp-server was executed on the John's Hopkins PS3 and the sftp client was executed on NSSAL's PS3 which would indicate a file transfer took place. There's evidence on the PCAP files and system log files to back this up.

asm etc # tail -100 /mnt/ps3/var/log/auth.log<p>Mar 11 11:49:43 nssal-ps3 sshd[3208]: reverse mapping checking getaddrinfo for ps3.isi.jhu.edu.249.220.128.in-addr.arpa [128.220.249.83] failed - POSSIBLE BREAK-IN ATTEMPT!
Mar 11 11:49:45 nssal-ps3 sshd[3208]: Accepted password for jhuisi from 128.220.249.83 port 51874 ssh2
Mar 11 11:49:45 nssal-ps3 sshd[3208]: pam_unix(sshd:session): session opened for user jhuisi by (uid=0)</p>asm etc # tail -100 /mnt/ps3-jhs/var/log/auth.log<p>Mar 11 12:49:25 ps3 sshd[5382]: Accepted password for goatboy from 137.30.123.40 port 34612 ssh2
Mar 11 12:49:25 ps3 sshd[5382]: pam_unix(sshd:session): session opened for user goatboy by (uid=0)
Mar 11 12:49:25 ps3 sshd[5411]: subsystem request for sftp</p>Extracted from jhuisi-capture-1.pcap165720 2009-03-11 13:49:28.800357 137.30.123.40 128.220.249.83 TCP 34612 > 22 [SYN] Seq=0 Win=5840 Len=0 MSS=1380 TSV=213441 TSER=0 WS=2165721
2009-03-11 13:49:28.800455 128.220.249.83 137.30.123.40 TCP 22 > 34612 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=219936 TSER=213441 WS=2
165722 2009-03-11 13:49:28.860085 137.30.123.40 128.220.249.83 TCP 34612 > 22 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=213456 TSER=219936
165723 2009-03-11 13:49:28.883103 128.220.249.83 137.30.123.40 SSHv2 Server Protocol: SSH-2.0-OpenSSH_5.1p1 Debian-3ubuntu1\r
...
166834 2009-03-11 13:52:24.069081 128.220.249.83 137.30.123.40 TCP 51942 > 22 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=263753 TSER=0 WS=2
166837 2009-03-11 13:52:24.118586 137.30.123.40 128.220.249.83 TCP 22 > 51942 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1380 TSV=257270 TSER=263753 WS=2
166838 2009-03-11 13:52:24.118676 128.220.249.83 137.30.123.40 TCP 51942 > 22 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=263765 TSER=257270
Session size = ~4MB (possible SFTP transfer)<p></p>Extracted from nssal-capture-2.pcap<p>34671 2009-03-11 13:49:02.101469 137.30.123.40 128.220.249.83 TCP 34612 > 22 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=213441 TSER=0 WS=2
34672 2009-03-11 13:49:02.161136 128.220.249.83 137.30.123.40 TCP 22 > 34612 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1380 TSV=219936 TSER=213441 WS=2
34673 2009-03-11 13:49:02.161235 137.30.123.40 128.220.249.83 TCP 34612 > 22 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=213456 TSER=219936
34674 2009-03-11 13:49:02.236815 128.220.249.83 137.30.123.40 SSHv2 Server Protocol: SSH-2.0-OpenSSH_5.1p1 Debian-3ubuntu1\r
...
35804 2009-03-11 13:51:57.409715 128.220.249.83 137.30.123.40 TCP 51942 > 22 [SYN] Seq=0 Win=5840 Len=0 MSS=1380 TSV=263753 TSER=0 WS=2
35805 2009-03-11 13:51:57.409862 137.30.123.40 128.220.249.83 TCP 22 > 51942 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=257270 TSER=263753 WS=2
35807 2009-03-11 13:51:57.459273 128.220.249.83 137.30.123.40 TCP 51942 > 22 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=263765 TSER=257270</p>
Session size = ~4MB (possible SFTP transfer)

I could tell from this network traffic that there was SSH/SFTP activity at that time on both systems, however it wasn't possible to see what was being transferred because everything was encrypted. I suspect that it was an sftp file transfer of the Mardi Gras pictures considering the size in bytes of the session which is almost the same as the total size of the Images directory on nssal's home directory. Wireshark's 'Follow TCP Stream' feature was very useful to determine when that particular session started and ended, I didn't include the FIN packets from when the session was closed here but you can easily find them using that particular Wireshark feature or doing some filtering.

asm etc # ls -alh /mnt/ps3/home/nssal/Images/<p>total 3.9M</p>

It isn't over yet.. there was more to see. I mentioned earlier that I had possibly found the source code for a backdoor that looked like it was used. I did some string searching and file carving and was able to recover different versions of the same backdoor, but all the evidence points to that same file that was sitting there in plain sight. My dirty words list was:

asm forensics # cat keywords.txt<p>Mardi grass
nssal
jhuisi
goatboy
backd00r
#define password
45541
137.30.251.152
128.220.249.83</p>

In order to recover that deleted backdoor source code i had to extract the unallocated space from the images i had, TSK's blks is what I used.

asm forensics # blkls nssal-linux-side-fs.dd > nssal-linux-side-fs.unalloc
asm forensics # blkls jhuisi-linux-side-fs.dd > jhuisi-linux-side-fs.unalloc

Then I extracted strings from unallocated space using srch_strings with the "-t d" option to get a byte offset for each string

asm forensics # srch_strings -a -t d nssal-linux-side-fs.unalloc > nssal-linux-side-fs.unalloc.asc
asm forensics # srch_strings -a -t d jhuisi-linux-side-fs.unalloc > jhuisi-linux-side-fs.unalloc.asc

Now that I had strings from unallocated space I was able to go through it and look for suspicious strings. There's several ways to do this, you could manually go through the file using the less command or you could be smart and use your dirty words list and grep

asm forensics # grep -i -f keywords.txt jhuisi-linux-side-fs.unalloc.asc
...
3762245958 #define password "q1w2e3r4"
... 

I found many interesting strings but after a lot of trial and error I was able to extract a source code for the backdoor. This line

3762245958 #define password "q1w2e3r4" 

looked like part of a C code so i tried to recover the file that contained that string from unallocated space. I used fsstat to get the file system's block size from the image, its 4096, this is needed to calculate the block number (in unallocated space) where this string was found, then I used blkcalc to calculate the block number on the original image and blkcat to extract the block's content.

asm forensics # expr 3762245958 / 4096<p>918517
asm forensics # blkcalc jhuisi-linux-side-fs.dd -u 918517
1273872
asm forensics # blkcat jhuisi-linux-side-fs.dd 1273872 > backd00r.c
asm forensics # file backd00r.c
backd00r.c: ASCII C program text, with CRLF line terminators</p>
There you have it. The source code for the backdoor that was used, if you look at the source you will find the lines

<p>#define PORT 45541</p>
<p>#define HIDE "klogd -x"</p>

<p>...</p><p>servaddr.sin_port = htons(PORT);</p><p>...</p>memset(argv[0], ' ', strlen(argv[0]));
strcpy(argv[0], HIDE);<p>...</p>

You'll see that the backdoor listens on port TCP 45541 and attempts to "hide" itself by running with the process name "klogd -x" which for the untrained eye would seem like a normal process on a Linux box. By realizing the port it uses and what I mentioned earlier in jhuisi's bash history file I thought it would be a good idea to look at the pcap file again and look for more evidence in terms of network traffic involving the two PS3's over TCP port 45541. 

Extracted from jhuisi-capture-1.pcap: 

173999 2009-03-11 14:10:26.035241 128.220.249.83 137.30.123.40 TCP 56515 > 45541 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=534240 TSER=0 WS=2
174000 2009-03-11 14:10:26.084796 137.30.123.40 128.220.249.83 TCP 45541 > 56515 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1380 TSV=527756 TSER=534240 WS=2
174002 2009-03-11 14:10:26.084964 128.220.249.83 137.30.123.40 TCP 56515 > 45541 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=534252 TSER=527756

We can clearly see here a TCP connection being established between the two systems over TCP port 45541, again I used wireshark's Follow TCP Stream here to be able to see what was going on, and this time everything was in clear text, nothing was encrypted.

jhuisi
Backdoor by darkXside
Enter the second password.
mac
Password accepted!
[backdoor]# ls
backd00r
Examples
[backdoor]# ls
backd00r
Examples
[backdoor]# rm backd00r
[backdoor]# ls
Examples
[backdoor]# cd ..
[backdoor]# cd nssal
[backdoor]# ls
channels
crosstool
d
Desktop
Documents
downloads
dump
Examples
file
find.py
first
first100
Images
kmem
memdump
memdump.new.kernel
memdump-powerpc.tar
memdump.tct
mem.find.pics
mem.hithere
mod
nulls
off.c
ppu-binutils-2.18.50-21.ppc.rpm
ppu-gcc43-4.3.2-2.ppc.rpm
ps3git
Public
ram.dd
Recipes
screenshots
strings.hithere
Templates
test
Videos
[backdoor]# who
nssal tty7 2009-03-11 11:31 (:0)
nssal pts/0 2009-03-11 11:32 (:0.0)
nssal pts/1 2009-03-11 11:47 (:0.0)
[backdoor]# ls
channels
crosstool
d
Desktop
Documents
downloads
dump
Examples
file
find.py
first
first100
Images
kmem
memdump
memdump.new.kernel
memdump-powerpc.tar
memdump.tct
mem.find.pics
mem.hithere
mod
nulls
off.c
ppu-binutils-2.18.50-21.ppc.rpm
ppu-gcc43-4.3.2-2.ppc.rpm
ps3git
Public
ram.dd
Recipes
screenshots
strings.hithere
Templates
test
Videos
[backdoor]# cd Recipes
[backdoor]# ls
andromachi
bateman's
stanley's
stoughton's
[backdoor]# cd ..
[backdoor]# ls
channels
crosstool
d
Desktop
Documents
downloads
dump
Examples
file
find.py
first
first100
Images
kmem
memdump.new.kernel
memdump-powerpc.tar
memdump.tct
mem.find.pics
mem.hithere
mod
nulls
off.c
ppu-binutils-2.18.50-21.ppc.rpm
ppu-gcc43-4.3.2-2.ppc.rpm
ps3git
Public
ram.dd
Recipes
screenshots
strings.hithere
Templates
test
Videos
[backdoor]# ls Videos
[backdoor]# ls screenshots
[backdoor]# ls Images
3316820191_4737c3edf4.jpg
3317048368_639213e24b_b.jpg
3317820492_cefb7ca452_b.jpg
3318492402_731ae5cdc3_b.jpg
3318589824_35fe706451_b.jpg
3320345810_6acc8185b2_b.jpg
3321856153_0a2c9577bd_b.jpg
3322040743_08a3b99acd_b.jpg
3322064459_d61e14dea5_b.jpg
3323556994_59a3982d61_b.jpg
3323673964_94e64ebddd_b.jpg
[backdoor]# fdisk -l
[backdoor]# ls -l /media
total 4
lrwxrwxrwx 1 root root 6 2008-12-03 18:14 cdrom -> cdrom0
drwxr-xr-x 2 root root 4096 2008-12-03 18:14 cdrom0
[backdoor]# ls -l /mnt
total 4
drwxr-xr-x 2 root root 4096 2009-03-06 15:01 usb
[backdoor]# ls -l /mnt/usb
total 0
[backdoor]#

We finally found more evidence of illicit access to that system. I didn't have time to look at the USB drive's image, it would be interesting to see what's in there. There's also probably more data to be extracted from the packet captures, such as images or HTTP traffic, it could easily be done using a tool like NetworkMiner.—Juan is a Network Security Engineer for an IT company in Uruguay. He holds a GCIA certification, Security+ and CCNA. He is working towards getting his GCFA.