10.8Mb/sec

Something isn’t right…

This afternoon, not long before leaving work, I just happened to check out bandwidthd running on nuxx.net. I was a bit surprised that it looked like this.

Yes, something was hacked. So, what happened?

Well, the first thing I did was just run top as root to see if anything stood out. The first thing I saw was the user katester running perl and taking up loads of CPU, having consumed something like 26 hours of CPU time. In short: ack!

Next I took a quick stab at what was doing it…

root@rowla:~# ps auxw | grep katester | grep perl
katester 23802 94.4 0.1 3184 2520 ?? R Wed01PM 1582:07.71 perl udp.pl irc.world-crew.net 6667 1200 (perl5.8.8)

Well, there’s the culprit… So, I killed it off, and it didn’t come back. Traffic immediately went back to normal, as can be seen here, in a mirror taken a bit after fixing things.

root@rowla:~# kill 23802

Next I shut down all of the sites owned by katester by commenting out their includes in the lighttpd config file and then restarting lighttpd.

Not finding any copies of udp.pl anywhere on the box, with nothing weird beneath /tmp or ~katester I threw udp.pl into Google. There were a few results, most of them showing this file, which appears to be a tool for DOSing boxes rather inefficiently by throwing random data at a port.

One of the next things I saw were two zombie processes and a copy of bash running as katester, along with the PHP processes normally allocated to that account:

root@rowla:~katester# ps -U katester
PID TT STAT TIME COMMAND
17603 ?? Z 0:00.01 <defunct>
17605 ?? S 0:03.76 ./bash
23800 ?? Z 0:00.00 <defunct>
51838 ?? I 7:27.67 /usr/local/bin/php-cgi
51842 ?? I 7:17.46 /usr/local/bin/php-cgi
51843 ?? I 7:29.32 /usr/local/bin/php-cgi
51851 ?? I 7:19.06 /usr/local/bin/php-cgi
59959 ?? Is 0:00.93 /usr/local/bin/php-cgi
67261 ?? I 6:31.64 /usr/local/bin/php-cgi

I noticed (but didn’t capture) that the copy of bash there was started at the same time as udp.pl. Because I was trying to isolate this user as much as possible I went ahead began killing off the PHP processes. As soon as I killed 59959 the next one, 67261 died as well. I then noticed that bash and both of the zombies were gone.

Now I figured it was time to start looking for where the infection came from so I could get the four downed sites back up as soon as possible. I just simply used grep to look through the httpd’s log files for things occurring during 13:00:00 EDT on 05-Sep-2007. (I also checked the 12:00:00 and 14:00:00 blocks and didn’t see anything suspectious.) After a few moments I noticed this:

142.166.3.123 www.katester.net - [05/Sep/2007:13:37:02 -0400] "GET /blog/index.php/feed/ HTTP/1.1" 200 17637 "-" "radianrss-1.0"

Throwing radianrss-1.0 into Google gave me no results, which is more than a little suspicious for what sounds like an RSS reader’s useragent. I looked through the other logs for the site and I could see other visits from 142.166.3.123 and found something interesting.

I next went ahead and called the site’s owner, discussed it with her, archived .../blog, and deleted that directory from the server. I was then able to bring the katester PHP processes and sites back online.

While there were other visits from 142.166.3.123 accessing /blog/index.php/feed/, every other one before that time had a useragent of Jakarta Commons-HttpClient/3.1-beta1. Here’s an example:

142.166.3.123 www.katester.net - [05/Sep/2007:03:19:47 -0400] "GET /blog/index.php/feed/ HTTP/1.1" 200 17637 "-" "Jakarta Commons-HttpClient/3.1-beta1"
142.166.3.123 www.katester.net - [05/Sep/2007:08:27:11 -0400] "GET /blog/index.php/feed/ HTTP/1.1" 200 17637 "-" "Jakarta Commons-HttpClient/3.1-beta1"

Then, next was the first entry with the odd useragent. All right around the time things began, and nothing else from that IP but those specific requests. I’ve now got tcpdump running, listening for anything from that IP, so I can (hopefully) see exactly what the malicious request was. There have been other requests since yesterday afternoon with the same radianrss-1.0 useragent, so I expect (and hope) the exploit will be attempted again.

Here’s what I suspect happened:

· The copy of WordPress, known for it’s numerous security holes, which happened to be running on katester.net was somehow exploited.
· The exploit was able to write a small script to the machine and execute it.
· This script then grabbed, spawned, and deleted udp.pl.
· I found things and cleaned them up.

I suppose in the future that I could have left udp.pl running for a little longer so that I could better investigate it, but I didn’t want to allow the DOSing that was happening to continue.

As some of you probably remember, back on 08-Jul-2006 the then-server hosting nuxx.net and many other sites was compromised. That happened because I had all the vhosts, PHP, and the httpd running as the same user with one of those vhosts running an old, vulnerable copy of phpBB. When things finally settled down and I moved to the new box I was able to run the httpd as its own user, with separate banks of PHP processes running for each user. Each user’s files were also held under their own home directories with accessible, but not modifiable (by the user) log files.

While the exploiting script last time was able to access (and delete) the contents of all the root directories of each user’s website, the more-proper privilege separation in new configuration mitigated potential risk. Fortunately it appears that this attack did little more than DOS an IRC server, but had it tried to do more it’s nice to know that it would have likely been contained to just ~katester.