Detecting Alteration to 'ps' and Friends



Attackers backdoor and modify compromised systems to secure future access to the machine, and to leverage newly-gained access to an individual system into access to many related machines ( Cheswick and Bellovin creepily refer to this as metastasis) . The former objective is typically met by altering the authentication mechanisms of the compromised system (usually, this means backdooring "login(1)" to allow root access with a secret password). The latter objective is almost always met by installing a sniffer.

Unfortunately for attackers, admins tend to notice processes on their systems that last for weeks and consume significant amounts of CPU time. In order to mitigate the risk of having their sniffer programs detected and pulled, and their breakins investigated, most attackers try to conceal the presence of their illicit activities on a compromised system. The most popular technique used by attackers (after logfile modification) is to backdoor the "ps" program to prevent it from reporting selected programs.

Modifying 'ps' is a flawed approach for a number of reasons, not the least of which is the fact that a savvy admin can keep a backup copy of 'ps' stashed away somewhere which will report accurately after the systemwide copy is altered. Because of these types of problems, the computer underground has come up with progressively more complex ways to attempt to conceal processes on a compromised system.

One of the first of these attempts exploited the fact that many Unix system status utilities rely on a single library to facilitate access to kernel data structures. It is possible to modify this library (usually called "libkvm") to tamper with the data 'ps' receives from the kernel. Besides beating naieve attempts to literally copy 'ps' to a safe location for later use, this technique also has the added benefit of consistantly hiding processes from other status utilities, like 'top'.

Of course, this trick can be easily defeated by statically linking a backup copy of 'ps'. A truely dedicated admin could also sanity-check the output of his 'ps' program by directly pulling the list of processes out of "/dev/kmem". Clearly, the only reliable way to hide the existance of a process on a Unix system is to modify the kernel, so that information about evil processes isn't available to userland programs at all.

This is particularly easy to accomplish on systems that rely on the "procfs" virtual filesystem for access to process status and control data. An attacker need only modify the procfs code to skip over "directories" in the process filesystem that represent illicit processes. This can be done in a few lines of code.

There are many other ways to hide processes from userland by altering the kernel. Essentially, as soon as your kernel is compromised, you are fighting a losing battle against the attacker, who can manipulate virtually anything you can see from userland.

All is not lost, however. There's a fairly serious problem facing an attacker trying to conceal the existance of processes from the operator of a system: many different interfaces into the kernel from userland leak process information. An obvious example of this is the "kill()" system call, which will return the error code ESRCH when it is asked to kill a nonexistant process.

Want to see if you're been hacked by a 12 year old? Try compiling and running this:

#include 
#include 
#include 
#include 
#include 

#define PID_MAX	30000	/* 4.4BSD, YMMV */

int main(int argc, char **argv) {
	int i = 0;
	
	for(i = 1; i < PID_MAX + 1; i++) 
		if(kill(i, 0) != -1 || errno != ESRCH)
			printf("%d\n", i);

	exit(0);
}

This trivial example uses kill() (with an argument of '0' for the signal to send, which causes kill() to perform checking without sending an actual signal) to print a list of all processes on the system. If kill() doesn't return "-1", the process exists and you have enough creds to send it a signal. If it returns "-1" and "errno" isn't ESRCH, the process exists, but you can't send signals to it. This program takes negligable time to run.

Compare the output of this program to the output of "ps -auxww". Do any process ID's show up in this output that don't in 'ps's? It's possible that someone has backdoored your 'ps' command. Investigate.

This is not the most devious way to apply this technique. It is fairly trivial for an attacker to modify kill() to report ESRCH for hidden processes. However, an attacker can't modify the general-purpose process-finding function in the kernel ("pfind()", on 4.4BSD) without potentially breaking their own programs. It is Difficult (note caps) to individually hack every point in the kernel that looks up processes and returns distinctive error codes for real ones. Think about using ioctl()'s and fcntl() to map out valid PIDs. Think about grepping for "pfind()" in the 4.4BSD kernel and finding every place that can report ESRCH, and cross check each of these interfaces for consistancy.