Incident Response by Kenneth R. Van Wyk, Richard Forno

From the Publisher

Seventy percent of businesses reported security breaches in 2000, and the rate is on the rise. Is your organization ready to respond to such an incident head-on? Will you be able to tell whether an incident is an attack or a glitch in the system? Do you know how to assess the possible damage from an incident? "Incident Response shows you how to answer questions like these and create a plan for exactly what to do before, during, and after an incident.

The authors of "Incident Response draw on years of experience developing and taking part in incident response teams at the highest levels of government and business. They guide you through both the technical and administrative details of effective incident response planning as they describe: What incident response is, and the problems of distinguishing real risk from perceived riskThe different types of incident response teams, and advantages and disadvantages of eachPlanning and establishing an incident response teamState of the Hack® information about different types of attacksRecommendations and details about available tools for incident response teamsResources available to incident response teams

Whatever your organization's size or purpose, "Incident Response shows how to put in place an incident-response process that's as planned, efficient, and businesslike as any other IT operation in a mature organization. Incidents happen, and being able to respond to them effectively makes good business sense.

Product Details

• Table of Contents
• Read an Excerpt

Table of Contents

Read an Excerpt

Chapter 7: Tools of the Trade

Anyone who reads an information security magazine or spends time on the Internet quickly realizes that there are a myriad of security tools available. These tools range from small, simple freeware security utilities to comprehensive professional security suites of tools designed to solve all your security problems. Because there are a vast number of tools, it can be difficult for the incident response professional to select the right tools for the job. But, contrary to vendor claims, no panacea or silver bullet exists that can solve all your problems. More importantly, very few, if any, security tools are designed from the ground up to be incident response tools. There are tools for encryption, authentication, intrusion detection, and so on, but there are (arguably) no tools designed specifically for incident response. Many of the available tools are useful for incident response activities, but were not designed for that purpose. Thus, it is up to the incident response practitioner to study the available tools and their applicability to incident response operations, and then to adapt to the tools they deem appropriate for the task at hand.

This chapter describes a large percentage--though certainly not all--of the currently available tools and discusses each tool's strengths and weaknesses specifically with regards to its utility and applicability as an incident response tool. The discussions about the tools are based, in almost every case, on our experiences using them in actual incidents. The operational aspects of the tool, such as the vendor's ability to support it, are stressed in the discussions here. We do not seek to endorse or condemn any tools, only to show the novice incident responder the capabilities, pros, and cons of various tools you will encounter. Where one tool comes up short, you can bet others will fill in the gaps nicely.

What's Out There?

As we've already indicated, the spectrum of available tools is broad. Even under ideal situations, not all of them are applicable to incident response operations. Although some can be considered system administration tools as well, we're going to focus on the tools that can be used as part of incident response operations. The list of tools we've assembled has been put together through actual incident response operations. As such, it has been growing, changing, shrinking, and updating for some time now.

One of the many encouraging and rewarding aspects of the incident response community (especially FIRST) has been its willingness and eagerness to share technical ideas among teams and individuals. FIRST holds periodic Technical Colloquia; the primary purpose of these TCs is to exchange technical data not related to actual sites that have been under attack. One of the more successful topics of discussion at numerous TCs has been technical presentations about practical experiences with different security tools. Most of the member teams are more than willing to candidly discuss their successes and failures in using security tools. This sort of open exchange of nonclient technical information is a tremendous benefit to FIRST and to all of the member teams that participate in the TCs.

In order for a tool to be truly useful in an incident response operation, it has to be able to support at least some of the basic operations that incident response practitioners encounter in their work. Some of the requirements that we've found over the years are as follows:

Flexibility

Above all else, the tools used the most in incident response operations are the ones that give the operator the maximum amount of flexibility. Each incident has the potential to be different. You could quickly find yourself needing different configurations of the tool, saving information to different magnetic and optic media in a variety of data formats, and so forth. New, previously unforeseen situations constantly arise in incident response operations. As responders, our tools of the trade need to support them!

Functionality

In general, there's a fairly small number of actions that the tools need to support. These include data collection, 24/365 notification of critical events, data analysis and documentation, litigation support, and attack visualization. While these criteria will no doubt evolve over time, it's a fairly accurate list of the top-level functions and features needed as of this writing.

Stealth

Almost without exception, incident response operations need to be carried out discretely so that the subject or target does not know she is being observed. The processes that we use need to be undetectable--within reason, of course--to the environment they are being deployed in. Sometimes this means connecting a computer to a network in such a way that the computer cannot be electronically observed; sometimes it means analyzing a computer system's contents in a way that doesn't disturb the contents at all; but it always means doing things quietly, efficiently, and promptly.

It is often the case that a suitable commercial or freeware tool cannot be found for a particular task. All incident response teams should be capable of putting together one-off tools for such occasions. These tools are often just used once and then stored in a repository for a future operation in which parts of the tool can be "plagiarized" for some other purpose. Examples of such one-off tools are almost always written as shell scripts, Perl scripts, batch files, or some other simple (but effective) programming language that can be very quickly tossed together into a simple archive. Indeed, some of the most useful tools we've encountered in operations were Perl scripts coded at 3:00 in the morning that served their one-time purpose and have quietly sat in a repository someplace to be resurrected someday when called upon.

One final side note before we get into the details about the specific tools regards the need to support law enforcement-grade evidentiary information. The data gathered by the tools that we're going to discuss may need to be presented in a courtroom situation as evidence. The collection and storage requirements for information that may be used in court are very different from normal network and IT diagnostic information. In a courtroom, the opposing counsel is likely to impugn your information by casting doubt on its integrity and how it was handled or prepared from the time of capture to the day of trial. For example, could the information have been fabricated or tampered with during the collection, analysis, or storage phase of the operation? If the answer to that question unveils any doubt, then it is likely that the information that you collected will at best be ineffective at convincing the judge or jury; at worst it will be dismissed as evidence and thrown out of the courtroom entirely. While electronic evidence-handling is a separate book unto itself, here are a few guiding points anytime you are handling computer evidence:

Two-person rule

Throughout the entire information gathering process, the data collected should be handled by two people, ensuring that no single person could have tampered with the information without the knowlege of another person.

Process documentation

The process, tools, etc., used in the collection of the information should be documented and signed by the people doing the collection. You can never document too much!

Tools sealed and locked away

Any tools used for the gathering, analysis, or documentation of information--in particular if they are of the one-off script variety such as we described previously--should be copied onto magnetic or optical media and locked away in a signed and sealed container. Ideally, the container should be stored in a trusted third-party's storage facility--we've found that storing sensitive information in our clients' General Counsels' offices is effective. Any access to that locked storage facility should be documented, dated, and signed.

Information sealed and locked away

Likewise, any actual data collected that might be needed as evidence should be stored on magnetic or optical media, sealed, and locked away--quite possibly for a very long time. Of course, the information that you've collected will need to be analyzed. So, prior to sealing it and storing it away--until needed in a courtroom, preferably--a duplicate copy should be made. That duplicate will be used for analysis and documentation. The duplicate itself should be backed up prior to doing the analysis. So, there should be three identical copies of the information: the original (locked away), the working duplicate (on which you do your forensics), and the back up of the duplicate (locked away in case you need to make another working duplicate).

At some point in nearly every incident response operation, the decision of whether to handle information as potential evidence should be made. As you can no doubt tell from the previous description of how to handle evidence, there's a rather large administrative burden when doing so. That burden can cause undue difficulties on the team when they are trying to get a system back in business as quickly as possible. If the information is not going to be used as evidence, then it is often best to make that determination early on and get on with the operation in the absence of the evidentiary burden. If the determination is made to treat the information as evidence, be sure to record who accesses, handles, processes, and prepares it, and which methods or tools are used. The chain of custody is a crucial part of proving in a courtroom that electronic evidence has not been tampered with or inadvertently modified.

We've often found that when business operations are placed at risk from an incident, the best thing to do is make your backup images and then allow the system admins to restore their systems while you do forensic analysis and investigation someplace else. Remember, business always comes first!

With that being said, let's dive into our list of tools and their applicability to incident response operations. Many of these tools are ones that you might want to include in a fly-away kit, a set of tools and equipment that is always ready to use when responding to an incident. As you read about the following tools, you might consider which of these you would want to include in your own kit. Fly-away kits are described in more detail later in this chapter in "The Incident Kit" section.

Network-Based Tools

Many general-purpose network diagnostic tools can be useful during incident response operations. These include network protocol analyzers, sniffers, and network-based Intrusion Detection Systems (IDSs). In fact, network-based tools are probably the most useful diagnosis and analysis available today to the incident response practitioner. Their applicability is enormous. The most common incident response uses of network-based tools include the following:

Attack detection

Although not part of incident response per se, a good Intrusion Detection System (IDS) architecture can be a superb means of electronically watching over an information infrastructure for indicators of attack. In fact, an IDS and an IRT function are really two complementary parts of a robust information security program. Further, many IDS tools have features that are highly applicable to incident response operations. Note that we are referring to Intrusion Detection here in a very broad sense, not just dedicated special-purpose IDS tools. Included in this category of Attack Detection is any network or system component that provides event logging data that detects whether an incident has taken place. This could include, for example, standard operating system event logs, router event logs, and so forth. When these logs are appropriately monitored for anomalies, they can be tremendously useful at alerting the operations staff to possible security incidents.

Attack diagnosis

Once an incident has been detected--correctly or incorrectly--it is usually necessary to diagnose the actual nature of the attack. For example, was an outside intruder successful at gaining access to an internal system? If so, what information did he get to? Was any of that information tampered with? What vulnerability did he exploit in order to gain the access? Network tools can be very helpful in going through the network-level information to analyze the attack and answer these types of questions.

Attack analysis and visualization

Once an attack has been observed, captured, and diagnosed on a network-based collection tool such as a protocol analyzer or IDS, it needs to be fully understood. Unlike the process of diagnosing the attack, this is an intense process of going through every aspect of the attack and trying to figure out what the attacker was doing. The most important aspect is trying to figure out what the attacker's intent was. One of the most useful ways of accomplishing this is by using a tool that supports attack visualization, or playback. This is a process in which the analyst can replay the attack and essentially watch what the attacker would have seen on his own screen during the attack, just like a VCR replaying whatever it has recorded on television. We should warn you that very few commercial tools do this well at all, and this particular feature is something of a holy grail.

Event notification

Another fairly common process requirement for the incident response team is to be notified of a particular network event, such as an intruder reentering a system or an employee suspected of wrongdoing accessing a file or area of a system that he is not authorized to. A full-featured network-based tool can support this kind of requirement by providing out-of-band notification (e.g., dial-out to a pager) so that the IRT can be alerted around the clock.

Out-of-Band Communication Out-of-band communications are simply alternative methods of communicating. For example, in a networked environment, out-of-band communication tools are telephone, cellphone, wireless networking (such as Ricochet), and pagers.

Media adaptation

Although a somewhat more mundane requirement, it is very often necessary to connect network-based tools to strange networks or devices not commonly used. These can include the kinds of very high speed media that are often found in data centers, such as FDDI or other fiber optic cabling, all the way through external Wide Area Network (WAN) connections such as T1, T3, OC3, OC12, and so on. Most out of the box network-based tools described here are going to be equipped with a standard selection of supported network media, and the IRT is likely to find the need to connect to other media from time to time. For these cases, having a diverse collection of network media adaptors, connectors, terminators, network interface cards, and related connection devices is invaluable. Nothing is worse than a fire truck showing up to a house fire and then realizing they've got the wrong type of hose to connect to the hydrant. Guess what happens to the house during the delay.

Network Monitors and Protocol Analyzers

Although primarily a mainstay as a diagnostic tool for network administrators, network monitors and protocol analyzers can be tremendously useful to the incident response team. Such tools should be standard issue for every incident response team. Their primary uses for incident response is their ability to dig deep into the network datagrams for low-level attack analysis and their ability to store network data to disk for further analysis. In short, these tools are like microscopes that reveal the inner contents of the data you're dealing with on an incident.

The one common and vital requirement for the network monitors, protocol analyzers, and the network-based intrusion detection systems discussed later is that they must perform their tasks silently and invisibly. We frequently refer to this as blackening the network monitor. A properly-configured network monitor should be completely invisible on a target network. In other words, an intruder should never be able to discover that he is being monitored. There are numerous ways of accomplishing the blackening of the tools, some mechanical and some logical, but any network monitoring device used for incident response should almost certainly be blackened, and the blackening should be thoroughly tested prior to actually using the tool. While there may be exceptions to this rule, they are few and far between....