226
solutions to consider. You should not hastily implement a solution until you have thought out the consequences. With lnx1, solutions ranged from rebooting the system to reinstalling
software. I chose the simplest first and rebooted the system. 9. Implement and evaluate your solution. Once you have decided on a solution and have
implemented it, you should confirm the proper operation of your system. Depending on the scope of the changes needed, this may mean extensive testing of the system and all related
systems.
With our running problem, this was not necessary. Connectivity was fully restored when the system was rebooted. What caused the problem? That was never fully resolved, but since the
problem never recurred, it really isnt an issue.
If restarting the system hadnt solved the problem, what would have been the next step? In this case, the likely problem was corrupted system software. If you are running an integrity
checker like tripwire, you might try locating anything that has changed and do a selective reinstallation. Otherwise, you may be faced with reinstalling the operating system.
One last word of warning. It is often tempting to seize on an overly complex explanation and ignore simpler explanations. Frequently, problems really are complex, but not always. It is worth asking
yourself if there is a simpler solution. Often, this will save a tremendous amount of time.
12.2 Task-Specific Troubleshooting
The guidelines just given are a general or generic overview of troubleshooting. Of course, each problem will be different, and you will need to vary your approach as appropriate. The remainder of
this chapter consists of guidelines for a number of the more common troubleshooting tasks you might face. It is hoped that these will give you further insight into the process.
12.2.1 Installation Testing
Ironically, one of the best ways to save time and avoid troubleshooting is to take the time to do a thorough job of testing when you install software or hardware. You will be testing the system when
you are most familiar with the installation process, and you will avoid disruptions to service that can happen when a problem isnt discovered until the software or hardware is in use.
This is a somewhat broad interpretation of troubleshooting, but in my experience, there is very little difference between the testing you will do when you install software and the testing you will do when
you encounter a problem. Overwhelmingly the only difference for most people is the scope of the testing done. Most people will test until they believe that a system is working correctly and then stop.
Failures, particularly multiple failures, may leave you skeptical, while some people tend to be overly optimistic when installing new software.
12.2.1.1 Firewall testing
Because of the complexities, firewall testing is an excellent example of the problems that installation testing may present. Troubleshooting a firewall is a demanding task for several reasons. First, to avoid
disruptions in service, initial firewall testing should be done in an isolated environment before moving on to a production environment.
227
Second, you need to be very careful to develop an appropriate set of tests so that you dont leave gaping holes in your security. Youll need to go through a firewall rule by rule. You wont be able to
check every possibility, but you should be able to test each general type of traffic. For example, consider a rule that passes HTTP traffic to your web server. You will want to pass traffic to port 80 on
that server. If you are taking the approach of denying all traffic that is not explicitly permitted, potentially, you will want to block traffic to that host at all other ports. You will also want to block
traffic to port 80 on other hosts.
[2]
Thus, you should develop a set of three tests for this one action. Although there will be some duplicated tests, youll want to take the same approach for each rule.
Developing an explicit set of tests is the key step in this type of testing.
[2]
If you doubt the need for this last test, read RFC 3093, a slightly tongue-in-cheek description of how to use port 80 to bypass a firewall.
The first step in testing a firewall is to test the environment in which the firewall will function without the firewall. It can be extraordinarily frustrating to try to debug anomalous firewall behavior only to
discover that you had a routing problem before you began. Thus, the first thing you will want to do is turn off any filtering and test your routing. You could use tools like ripquery to retrieve routing tables
and examine entries, but it is probably much simpler to use ping to check connectivity, assuming ICMP ECHO_REQUEST packets arent being blocked. If this is the case, you might try tools like
nmap or hping.
Youll also want to verify that all concomitant software is working. This will include all intrusion detection software, accounting and logging software, and testing software. For example, youll
probably use packet capture software like tcpdump or ethereal to verify the operation of your firewall and will want to make sure the firewall is working properly. I hate to admit it, but Ive started packet
capture software on a host that I forgot was attached to a switch and banged my head wondering why I wasnt seeing anything. Clearly, if I had used this setup to make sure packets were blocked without
first testing it, I could have been severely misled.
Test the firewall in isolation. If you are adding filtering to a production router, admittedly this is going to be a problem. The easiest way to test in isolation is to connect each interface to an isolated host that
can both generate and capture packets. You might use hping, nemesis, or any of the other custom packet generation software discussed in
Chapter 9 . Work through each of your tests for each rule with
the rule disabled and enabled. Be sure you explicitly document all your tests, particularly the syntax. Once you are convinced that the firewall is working, it is time to move it online. If you can schedule
offline testing, that is the best approach. Work through your tests again with and without the filters enabled. If offline testing isnt possible, you can still go through your tests with the filters enabled.
Finally, dont forget to come back and go through these tests periodically. In particular, youll want to reevaluate the firewall every time you change rules.
12.2.2 Performance Analysis and Monitoring
