How to check if your server has been hacked

Root compromises

This means someone has full access to the system, here are the tell tale signs in order of most likely to give you a quick feel for what’s going on.

1. Have a look for system files that have changed recently. This is the first thing I would do.

find /etc /var -mtime -2

The “-2” means 2 days, i.e. show me all files modified in the last 2 days.

Now if you haven’t installed any new software on your server for a while then this command will run and produce very little output. For a server I investigated there were references to postfix. clearly someone had installed a mail server probably for sending spam.

2. Run who

who
user1 pts/2 2012-03-28 13:38 (128.114.44.209)

This should give you a list of users on the system, what you’re looking for is users other than yourself especially root.

3. History

history

Login as root and run history, this will give you a list of recently run commands. If you’re the only person who logs in as root you should have an idea if anything looks suspicious or not.

4. Netstat
It’s a good idea at this point to see what processes are running on which ports.

netstat --listen -A inet

tcp 0 0 *:64010 *:* LISTEN
tcp 0 0 *:http-alt *:* LISTEN 
tcp 0 0 *:ssh *:* LISTEN 
tcp 0 0 *:https *:* LISTEN

Application Breaches
An example of an application breach might be a shopping cart that allows file uploads these uploads can be executed as say PHP scripts (This happened with oscommerce). The attacker doesn’t have root access but they can certainly use up resources.

5. High load and Memory.
Sudden spikes in memory and load usage are an indication of possible root breaches but sometimes the only indication an application has been breached.

If you’re a ServerMonitoringHQ.com user then you’ll receive notifications of CPU and Memory spikes. (Along with bandwidth and more).

You can use the following command to track load (CPU usage). Generally values over 1 indicate the CPU is getting used quite hard.

cat /proc/loadavg

0.04 0.35 0.26 1/83 27412

The first column 0.04 is the number to look at

6. Top
The top command is a quickly way to see what processes are consuming resources.

7. Bandwidth Spikes
As with process and memory if you don’t know what you normally consume it’s difficult to tell if you get a spike or not.

netstat -i 
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR 
eth0 1500 0 2937582 0 0 0 3127979 0 0 0 BMRU 
lo 16436 0 531312 0 0 0 531312 0 0 0 LRU

The larger numbers for TX and RX are bytes transferred (roughly), run this twice, a minute apart will give you an indication of bandwidth transfer per minute.

And finally, good luck.
We’ve all been there, best of luck getting your server up and running again.