Model-Based Intrusion Detection

9.10.5 Model-Based Intrusion Detection

Yet another approach to defending a machine is to install an IDS (Intrusion Detection System ). There are two basic kinds of IDSes, one focused on inspect- ing incoming network packets and one focused on looking for anomalies on the CPU. We briefly mentioned the network IDS in the context of firewalls earlier; now we will say a few words about a host-based IDS. Space limitations prevent us from surveying the many kinds of host-based IDSes. Instead, we will briefly sketch one type to give an idea of how they work. This one is called static model-based intrusion detection (Hua et al., 2009). It can be implemented using the jailing technique discussed above, among other ways.

SECURITY CHAP. 9 In Fig. 9-37(a) we see a small program that opens a file called data and reads it

one character at a time until it hits a zero byte, at which time it prints the number of nonzero bytes at the start of the file and exits. In Fig. 9-37(b) we see a graph of the system calls made by this program (where print calls wr ite ).

int main(int argc char argv[]) * {

open

int fd, n = 0; char buf[1];

write fd = open("data", 0);

read

if (fd < 0) { printf("Bad data file\n"); exit(1);

exit } else {

close

while (1) { read(fd, buf, 1); if (buf[0] == 0) {

write

close(fd); printf("n = %d\n", n); exit(0);

Figure 9-37. (a) A program. (b) System-call graph for (a).

What does this graph tell us? For one thing, the first system call the program makes, under all conditions, is always open . The next one is either read or wr ite , depending on which branch of the if statement is taken. If the second call is wr ite , it means the file could not be opened and the next call must be exit . If the second call is read , there may be an arbitrarily large number of additional calls to read and ev entually calls to close , wr ite , and exit . In the absence of an intruder, no other se- quences are possible. If the program is jailed, the jailer will see all the system calls and can easily verify that the sequence is valid.

Now suppose someone finds a bug in this program and manages to trigger a buffer overflow and inserts and executes hostile code. When the hostile code runs, it will most likely execute a different sequence of system calls. For example, it might try to open some file it wants to copy or it might open a network connection to phone home. On the very first system call that does not fit the pattern, the jailer knows definitively that there has been an attack and can take action, such as killing the process and alerting the system administrator. In this manner, intrusion detec- tion systems can detect attacks while they are going on. Static analysis of system calls is just one of the many ways an IDS can work.

SEC. 9.10

DEFENSES

When this kind of static model-based intrusion detection is used, the jailer has to know the model (i.e., the system-call graph). The most straightforward way for it to learn it is to have the compiler generate it and have the author of the program sign it and attach its certificate. In this way, any attempt to modify the executable program in advance will be detected when it is run because the actual behavior will not agree with the signed expected behavior.

Unfortunately, it is possible for a clever attacker to launch what is called a mimicry attack , in which the inserted code makes the same system calls as the program is supposed to, so more sophisticated models are needed than just tracking system calls. Still, as part of defense in depth, an IDS can play a role.

A model-based IDS is not the only kind, by any means. Many IDSes make use of a concept called a honeypot, a trap set to attract and catch crackers and mal- ware. Usually it is an isolated machine with few defenses and a seemingly inter- esting and valuable content, ripe for the picking. The people who set the honeypot carefully monitor any attacks on it to try to learn more about the nature of the at- tack. Some IDSes put their honeypots in virtual machines to prevent damage to the underlying actual system. So naturally, the malware tries to determine if it is run- ning in a virtual machine, as discussed above.