Microsoft Windows Security Resource Kit by Ben Smith, Microsoft Corporation Staff, Brian Komar, Eric Schultze, Staff of Microsoft Security Team

From Barnes & Noble

The Barnes & Noble Review
Securing Windows 2000 and Windows XP isn’t an “occasional” task: It requires a systematic approach, a strong understanding of the available tools, and ongoing attention. The most complete and useful Windows security blueprint we’ve found is the Microsoft Windows Security Resource Kit.

With the help of Microsoft’s own security team, this book’s authors cover every aspect of security and privacy: principles, securing the core operating system, securing common services, performing assessments and incident response, and managing security updates.

Ben Smith and Brian Komar start by outlining 20 immutable laws of security and security administration (for example, “Absolute anonymity isn’t practical,” “Security only works if the secure way also happens to be the easy way,” “There really is someone out there trying to guess your passwords,” and most important: “Technology is not a panacea”).

Next, following Sun Tzu’s classic directive “Know your enemy and know yourself,” they help you assess where you stand. What are your own skills? What kind of support will you receive for your security initiatives? How well is your network documented?

Then, it’s down to cases with Windows. The authors begin with securing Active Directory -- starting with user accounts and passwords. You know you’re not supposed to use administrative accounts for routine activities, but do you always pay attention? (Try RunAs, a.k.a. Secondary Logon, as an easy alternative.) Think you know all about passwords? So are you using Passfilt.dll to require harder-to-guess passwords?

There’s a full chapter on securing AD objects and attributes: configuring DACLs (including command line shortcuts); applying least privilege, and so forth. The authors cover group policies in depth, including object filtering and loopback mode processing. They also offer complete guidance on designing more secure AD forests and domains -- and as with every chapter in the book, they offer specific best practices. (“Use multiple forests if you require discrete isolation.” “Control membership in security groups with high security requirements.”)

One step at a time, you’ll walk through securing permissions and services; implementing TCP/IP security; IE6 (you have upgraded, right?), and Microsoft Office XP -- including Outlook 2002. There’s a chapter on applying security templates to individual computers and via group policies; another on auditing Windows security events; and one on securing mobile computers and wireless networks.

There’s detailed coverage of securing Windows’ key services: DNS, Terminal Services, DHCP, WINS, routing and remote access, certificate services, and IIS 5 (but not IIS 6 -- but then you probably haven’t finished qualifying Windows Server 2003 yet, have you?)

You’ll find a six-step plan for managing patches from notification through validation; as well as practical coverage of Microsoft’s Baseline Security Analyzer and Software Update Services. There’s an excellent section on privacy -- on both your corporate web site and within the enterprise. Last but not least, there’s a CD-ROM full of Microsoft security tools and resources you could probably track down on your own -- if you had days to do it. Bill Camarda

Bill Camarda is a consultant, writer, and web/multimedia content developer. His 15 books include Special Edition Using Word 2000 and Upgrading & Fixing Networks for Dummies, Second Edition.

From the Publisher

The definitive security reference for Microsoft operating systems, servers, clients, networks, and Internet services-straight from the Microsoft product group. This official Microsoft resource kit delivers IT professionals with comprehensive operations and deployment information plus essential tools that information security professionals can put to work right away. The authors-members of Microsoft's security teams-describe how to plan and implement a comprehensive security strategy, assess security threats and vulnerabilities, configure system security, safeguard data, monitor and detect security events, respond to internal and external attacks, and effectively apply security technologies and best practices. The RESOURCE KIT also provides must-have security tools, checklists, templates, and other on-the-job resources on CD-ROM and the Web.

Product Details

• Table of Contents
• Read an Excerpt

Table of Contents

Read an Excerpt

Implementing TCP/IP Security

• Securing TCP/IP
• Understanding Internet Layer Protocols
• Understanding Transport Layer Protocols
• Common Threats to TCP/IP
• Configuring TCP/IP Security in Windows 2000 and Windows XP
• Using IPSec
• Securing Data Transmission with IPSec Protocols
• Choosing Between IPSec Modes
• Selecting an IPSec Authentication Method
• Creating IPSec Policies
• How IPSec Works
• Monitoring IPSec
• Best Practices
• Additional Information

TCP/IP is an industry-standard suite of protocols designed to facilitate communication between computers on large networks. TCP/IP was developed in 1969 by the U.S. Department of Defense Advanced Research Projects Agency (DARPA), as the result of a resource-sharing experiment called ARPANET (Advanced Research Projects Agency Network). Since 1969, ARPANET has grown into a worldwide community of networks known as the Internet, and TCP/IP has become the primary protocol used on all networks. Unfortunately, TCP/IP was not designed with security in mind and thus has very few security components by default. Consequently, it is often a source of network vulnerabilities. On your Microsoft Windows 2000 and Windows XP computers, you can secure the TCP/IP protocol in several ways, which include securing the TCP/IP stack itself and using IP Security (IPSec). We will examine both techniques in this chapter.

Securing TCP/IP

You cannot successfully secure computer networks without knowing how TCP/IP works. Nearly all computers today use TCP/IP as their primary network communication protocol. Thus, without physical access to a computer, an attacker must use TCP/IP to attack it. Consequently, TCP/IP security is often your first line of defense against attackers attempting to compromise your organization’s network and therefore should be part of any defense-in-depth strategy for securing networks. You can secure the TCP/IP protocol in Windows 2000 and Windows XP to protect a computer against common attacks, such as denial-of-service attacks, and to help prevent attacks on applications that use the TCP/IP protocol.

Understanding Internet Layer Protocols

TCP/IP primarily operates at two levels in the OSI model: the Internet layer and the transport layer. The Internet layer is responsible for addressing, packaging, and routing functions. The core protocols of the Internet layer include the Internet Protocol (IP), Address Resolution Protocol (ARP), and Internet Control Message Protocol (ICMP):

• IP  A routable protocol responsible for logical addressing, routing, and the fragmentation and reassembly of packets
• ARP  Resolves IP addresses to Media Access Control (MAC) addresses and vice versa
• ICMP  Provides diagnostic functions and reporting errors for unsuccessful delivery of IP packets

The TCP/IP protocol suite includes a series of interconnected protocols called the core protocols. All other applications and protocols in the TCP/IP protocol suite rely on the basic services provided by several protocols, including IP, ARP, and ICMP.

IP

IP is a connectionless, unreliable datagram protocol primarily responsible for addressing and routing packets between hosts. Connectionless means that a session is not established to manage the exchange data. Unreliable means that delivery is not guaranteed. IP always makes a best-effort attempt to deliver a packet. An IP packet might be lost, delivered out of sequence, duplicated, or delayed. IP does not attempt to recover from these types of errors. The acknowledgment of packets delivered and the recovery of lost packets is the responsibility of a higher-layer protocol, such as TCP. IP is defined in RFC 791.

An IP packet consists of an IP header and an IP payload. The IP header contains information about the IP packet itself, and the IP payload is the data being encapsulated by the IP protocol to be transmitted to the receiving host. The following list describes the key fields in the IP header:

• Source IP Address  The IP address of the source of the IP datagram.
• Destination IP Address  The IP address of the destination of the IP datagram.
• Identification  Used to identify a specific IP datagram and all fragments of a specific IP datagram if fragmentation occurs.
• Protocol  Informs IP at the destination host whether to pass the packet up to TCP, UDP, ICMP, or other protocols.
• Checksum  A simple mathematical computation used to verify the integrity of the IP header. If the IP header does not match the checksum, the receiving host will disregard the packet. This checksum does not include any information outside the IP header.
• Time To Live (TTL)  Designates the number of networks on which the datagram is allowed to travel before being discarded by a router. The TTL is set by the sending host and used to prevent packets from endlessly circulating on an IP network. When forwarding an IP packet, routers decrease the TTL by at least one.
• Fragmentation And Reassembly  If a router receives an IP packet that is too large for the network to which the packet is being forwarded, IP fragments the original packet into smaller packets that fit on the downstream network. When the packets arrive at their final destination, IP on the destination host reassembles the fragments into the original payload. This process is referred to as fragmentation and reassembly. Fragmentation can occur in environments that have a mix of networking technologies, such as Ethernet and Token Ring. The fragmentation and reassembly works as follows:

1. When an IP packet is sent, the sending host places a unique value in the Identification field.
2. The IP packet is received at the router. If the router determines that the Maximum Transmission Unit (MTU) of the network onto which the packet is to be forwarded is smaller than the size of the IP packet, the router fragments the original IP payload into multiple packets, each of which is smaller than the receiving network’s MTU size. Each fragment is sent with its own IP header that contains the following:

The original Identification field, which identifies all fragments that belong together.

The More Fragments flag, which indicates that other fragments follow. The More Fragments flag is not set on the last fragment because no other fragments follow it.

The Fragment Offset field, which indicates the position of the fragment relative to the original IP payload.

3. When the fragments are received by the destination host, they are identified by the Identification field as belonging together. The Fragment Offset field is then used to reassemble the fragments into the original IP payload.

ARP

Address Resolution Protocol performs IP address–to–MAC address resolution for outgoing packets. As each outgoing addressed IP datagram is encapsulated in a frame, source and destination MAC addresses must be added. Determining the destination MAC address for each frame is the responsibility of ARP. ARP is defined in RFC 826.

ICMP

Internet Control Message Protocol provides troubleshooting facilities and error reporting for packets that are undeliverable. For example, if IP is unable to deliver a packet to the destination host, ICMP sends a Destination Unreachable message to the source host. Table 9-1 shows the most common ICMP messages.

Table 9-1  Common ICMP Messages

Message	Description
Echo Request	Troubleshooting message used to check IP connectivity to a desired host. The Ping utility sends ICMP Echo Request messages.
Echo Reply	Response to an ICMP Echo Request.
Redirect	Sent by a router to inform a sending host of a better route to a destination IP address.
Source Quench	Sent by a router to inform a sending host that its IP datagrams are being dropped because of congestion at the router. The sending host then lowers its transmission rate.
Destination Unreachable	Sent by a router or the destination host to inform the sending host that the datagram cannot be delivered.

When the result of an ICMP request is a Destination Unreachable message, a specific message is returned to the requestor detailing why the Destination Unreachable ICMP message was sent. Table 9-2 describes the most common of these messages.

Table 9-2  Common ICMP Destination Unreachable Messages

Unreachable Message	Description
Host Unreachable	Sent by an IP router when a route to the destination IP address cannot be found
Protocol Unreachable	Sent by the destination IP node when the Protocol field in the IP header cannot be matched with an IP client protocol currently loaded
Port Unreachable	Sent by the destination IP node when the destination port in the UDP header cannot be matched with a process using that port
Fragmentation Needed and DF Set	Sent by an IP router when fragmentation must occur but is not allowed because of the source node setting the Don’t Fragment (DF) flag in the IP header

ICMP does not make IP a reliable protocol. ICMP attempts to report errors and provide feedback on specific conditions. ICMP messages are carried as unacknowledged IP datagrams and are themselves unreliable. ICMP is defined in RFC 792.

Understanding Transport Layer Protocols

The transport layer is responsible for providing session and datagram communication services over the IP protocol. The two core protocols of the transport layer are the Transmission Control Protocol (TCP) and User Datagram Protocol (UDP):

• TCP  Provides a one-to-one, connection-oriented, reliable communications service. TCP is responsible for the establishment of a TCP connection, the sequencing and acknowledgment of packets sent, and the recovery of packets lost during transmission.
• UDP  Provides a one-to-one or one-to-many, connectionless, unreliable communications service. UDP is used when the amount of data to be transferred is small (such as data that fits into a single packet), when the overhead of establishing a TCP connection is not desired, or when the applications or upper layer protocols provide reliable delivery.

How TCP Communication Works

When two computers communicate using TCP, the computer that initiates the communication is known as the client, regardless of whether it is running a client or server OS, and the responding computer is known as the host. If the client and host are on the same network segment, the client computer first uses ARP to resolve the host’s MAC address by sending a broadcast for the IP address of the host. Once the client has the MAC address of the host, it can commence communication to the port on the host by using the transport layer protocol specified by the application. There are 65,535 TCP and UDP ports, beginning with 0. Ports 1023 and below are regarded as well-known ports for legacy reasons, and ports above 1023 are known as high ports. Functionally, no difference exists between the well-known ports and the high ports. On the host, an application is bound to a certain port it specifies and is initialized in a listening state, where it waits for requests from a client. When the client initiates a connection to a TCP port, a defined series of packets, known as a three-way handshake and illustrated in Figure 9-1, constructs a session for reliable packet transmission. The steps for establishing connections follow:

1. The client sends the host a synchronization (SYN) message that contains the host’s port and the client’s Initial Sequence Number (ISN). TCP sequence numbers are 32 bits in length and used to ensure session reliability by facilitating out-of-order packet reconstruction.
2. The host receives the message and sends back its own SYN message and an acknowledgement (ACK) message, which includes the host’s ISN and the client’s ISN incremented by 1.
3. The client receives the host’s response and sends an ACK, which includes the ISN from the host incremented by 1. After the host receives the packet, the TCP session is established.

Figure 9-1  Three-way TCP handshake (Image unavailable)

When the communication between the client and host is complete, the session is closed once the following steps occur:

1. The client sends a finalization (FIN) message to the host. The session is now half closed. The client no longer sends data but can still receive data from the host. Upon receiving this FIN message, the host enters a passive closed state.
2. The host sends an ACK message, which includes the client’s sequence number augmented by 1.
3. The server sends its own FIN message. The client receives the FIN message and returns an ACK message that includes the host’s sequence number augmented by 1.
4. Upon receiving this ACK message, the host closes the connection and releases the memory the connection was using.

To see port activity on your Windows 2000 or Windows XP computers, you can use the Netstat.exe command. Netstat.exe will also show the status of TCP ports. The syntax for using Netstat.exe follows, and Table 9-3 describes the options available when using this command.

NETSTAT [-a] [-e] [-n] [-o] [-s] [-p proto] [-r] [interval]

Table 9-3  Netstat.exe Options

Option	Description
-a	Displays all connections and listening ports.
-e	Displays Ethernet statistics. This can be combined with the -s option.
-n	Displays addresses and port numbers in numerical form.
-o	Displays the owning process ID (PID) associated with each connection. This option exists in Windows XP only.
-p protocol	Shows connections for the protocol specified by protocol, which can be TCP, UDP, TCPv6, or UDPv6. If used with the -s option to display per-protocol statistics, the value for protocol can be IP, ICMP, TCP, or UDP.
-r	Displays the routing table.
-s	Displays per-protocol statistics. By default, statistics are shown for IP, ICMP, TCP, and UDP.
interval	Determines the refresh interval for the data displayed by Netstat.

As mentioned in Table 9-3, the -o option of Netstat.exe is not available in Windows 2000; however, you can download utilities from the Internet that have similar functionality and will run on Windows 2000.

Common Threats to TCP/IP

Several types of threats to TCP/IP can either compromise network security or lead to information disclosure. Although these attacks are more prevalent on the Internet, you should be concerned about them on internal computers as well. These common threats include:

• Port scanning
• Spoofing
• Denial of service

Port Scanning

In order to communicate with TCP/IP, applications running on host computers must listen for incoming TCP or UDP connections, and host operating systems must listen for broadcast and other network maintenance traffic. By scanning a computer to see what ports a host is listening for and what protocols it uses, an attacker might be able to locate weaknesses in the host that he can later use to attack the computer. Attackers often perform port scans to reveal this information. Several types of port scans exist:

• Ping sweeps  An attacker might use an automated tool to send ICMP Echo Request packets to entire networks or subnets. By default, all active hosts will respond. This lets the attacker know that the host exists and is active. An attacker can also analyze the structure of the ICMP packet to determine the OS running on the host.
• Port enumeration  At attacker might want to enumerate all the services running on a host computer. Because hosts must respond to client computers to carry out legitimate operations, attackers can exploit this behavior to obtain critical information.

---

TIP:
You can download a command-line port-scanning tool from Microsoft called Portqry.exe. This tool tests the security of hosts and performs network diagnostics. In addition, many free utilities that can perform port scans are available on the Internet.

---

• Banner grabbing  Many common services respond with banners when sessions are initiated or requested. These banners contain basic information on the service or server. For example, by using Telnet to connect to port 25 of a Windows 2000 server running the default Simple Mail Transfer Protocol (SMTP) service, you can retrieve this banner:

220 SFOFS001.finance.woodgrovebank.com Microsoft ESMTP MAIL Service, 
Version: 5.0.2195.5329 ready at Sat, 12 Oct 2002 16:18:44 -0800

From interpreting this banner, you can determine that the target server is named SFOFS001. SFOFS001 is probably a file server running Windows 2000 with Service Pack 3 installed and is physically located in the Pacific Time zone—most likely in San Francisco. The server is running a built-in instance of the SMTP service, which is installed as part of Microsoft Internet Information Services (IIS) 5.0. Knowing that IIS is installed by default in Windows 2000 and that this server does not appear to be a Web server, it is likely that the server has a default installation of Windows 2000.

---

IMPORTANT:
Changing service banners can also break applications that rely on them for information about the server they are communicating with. Furthermore, changing banners can break an application running on the computer that uses the information from service banners from other services running on the computer.

---

• Half scan  This type of port scanning does not follow the precise TCP three-way handshake protocol and leaves TCP connections half open. Because most host system logs do not log packets until the host receives the final ACK, half scans can enable an attack to gain information about a host without being detected.

Spoofing

Attackers might want to spoof, or mimic, a legitimate TCP/IP packets to attack a computer or network. Usually spoofing a packet requires that the attacker handcraft a TCP/IP packet and send it to either the host he wants to attack or a third party host that he has previously compromised in order to attack the targeted host or network. Many types of spoofing attacks exist. These following three are among the most well-known:

• Land attack  Takes advantage of security flaws in the many implementations of TCP/IP. To carry out a land attack, an attacker opens a legitimate TCP session by sending a SYN packet but spoofs the packet so that the source address and port and the destination address and port match the host IP address and the port the packet is being sent to.

For example, to carry out a land attack on an e-mail server with the IP address 192.168.183.200, an attacker can create a packet with the source address of 192.168.183.200 and the source port of 25, rather than using the source address and port of his own computer. Now the source and destination addresses will be the same, as will the source and destination ports. If not patched to protect against the land attack, the e-mail will continually attempt to make a connection with itself on its own port 25, resulting in a denial-of-service situation.

• Smurf attack  Uses a third-party network to carry out a denial-of-service attack on a host system by spoofing an ICMP Echo Request packet. The attacker obtains the host IP address and creates a forged ICMP Echo Request packet that looks like it came from the host IP address. The attacker sends thousands of copies of the spoofed packet to the broadcast address on an improperly secured third-party network. This results in every computer in the third-party network responding to each spoofed packet by sending an ICMP Echo Reply packet to the host system. The amount of ICMP traffic that is generated by this attack will deny legitimate traffic from reaching the target host.
• Session hijacking  Takes advantage of flaws in many implementations of the TCP/IP protocol by anticipating TCP sequence numbers to hijack a session with a host. To hijack a TCP/IP session, the attacker creates a legitimate TCP session to the targeted host, records the TCP sequence numbers used by the host, and then computes the round-trip time (RTT). This step often takes many exchanges in sequence. Using the stored sequence numbers and the RTT, the attacker can potentially predict future TCP sequence numbers. The attacker can then send a spoofed packet to another host, using the targeted host IP address as the source address and the next sequence number. If successful, the second host system will believe the packet originated from the targeted system and accept packets from the attacker. This type of attack is particularly effective when the second host trusts the targeted host.

---

MORE INFO:
IP spoofing by predicting TCP/IP sequence numbers was the basis for the famous Christmas 1994 attack on Tsutomu Shimomura by Kevin Mitnick. The attack is chronicled in the book Takedown: The Pursuit and Capture of Kevin Mitnick, America’s Most Wanted Computer Outlaw—By The Man Who Did It (Hyperion, 1996).

---

Denial of Service

Denial-of-service attackers attempt to exploit the way the TCP/IP protocol works to prevent legitimate traffic from reaching the host system. One of the most common types of denial-of-service attacks is a SYN flood. A SYN flood attempts to create a situation in which the host system’s maximum TCP connection pool is locked in a half-open state, thus denying legitimate traffic to and from the host. To carry out a SYN flood, the attacker creates a spoofed IP packet with an unreachable IP address for a source address, or she clips the receive wire on the Ethernet cable she is using. When the host receives the packet, it responds by sending a SYN/ACK response and waits for the final ACK in the TCP three-way handshake, which never comes. The session will remain in the half-open state until the predefined time-out is reached. This process is repeated until no more TCP sessions are allowed by the host system, which then cannot create any new sessions.

Configuring TCP/IP Security in Windows 2000 and Windows XP

The remainder of this section presents several ways you can secure your Windows 2000 and Windows XP computers against attacks on TCP/IP, including basic TCP/IP binding configurations, custom registry settings, and TCP/IP filtering.

Implementing Basic TCP/IP Security

Three basic settings, outlined in the following list, will increase the security of TCP/IP for each network adapter in Windows 2000 and Windows XP. You will need to ensure that each of these settings is compatible with your network and the applications that either run on the computer or must be accessible from the computer.

• File And Printer Sharing For Microsoft Networks  By default, File and Printer Sharing for Microsoft Networks is bound on all network interfaces. The File and Printer Sharing for Microsoft Networks component enables other computers on a network to access resources on your computer. By removing the binding to File and Printer Sharing for Microsoft Networks from a network interface, you can prevent other computers from enumerating or connecting to files and printers that have been shared through that network interface. After removing this binding from a network interface, the computer will no longer listen for direct Server Message Block (SMB) connections on TCP ports 139 or 445 of that interface. Removing this setting will not interfere with the computer’s ability to connect to other shared files or printers. You can unbind File and Printer Sharing for Microsoft Networks in the Network And Dial-Up Connections Control Panel applet or on the properties of the network interface.
• NetBIOS Over TCP/IP  Windows 2000 and Windows XP support file and printer sharing traffic by using the SMB protocol directly hosted on TCP. This differs from earlier operating systems, in which SMB traffic requires the NetBIOS over TCP/IP (NetBT) protocol to work on a TCP/IP transport. If both the direct hosted and NetBT interfaces are enabled, both methods are tried at the same time and the first to respond is used. This allows Windows to function properly with operating systems that do not support direct hosting of SMB traffic. NetBIOS over TCP/IP traditionally uses the following ports:

NetBIOS name	137/UDP
NetBIOS name	137/TCP
NetBIOS datagram	138/UDP
NetBIOS session	139/TCP

---

NOTE:
Direct hosted "NetBIOS-less" SMB traffic uses port 445 (TCP and UDP). If you disable NetBIOS Over TCP/IP (NetBT) and unbind File And Printer Sharing For Microsoft Networks, the computer will no longer respond to any NetBIOS requests. Applications and services that depend on NetBT will no longer function once NetBT is disabled. Therefore, verify that your clients and applications no longer need NetBT support before you disable it.

---

• DNS Registration  By default, Windows 2000 and Windows XP computers attempt to automatically register their host names and IP address mappings in the Domain Name System (DNS) for each adapter. If your computer is using a public DNS server or cannot reach the DNS server, as is often seen when the computer resides in a screened subnet, you should remove this behavior on each adapter.

Configuring Registry Settings

Denial-of-service attacks are network attacks aimed at making a computer or a particular service on a computer unavailable to network users. Denial-of-service attacks can be difficult to defend against. To help prevent denial-of-service attacks, you can harden the TCP/IP protocol stack on Windows 2000 and Windows XP computers. You should harden the TCP/IP stack against denial-of-service attacks, even on internal networks, to prevent denial-of-service attacks that originate from inside the network as well as on computers attached to public networks. You can harden the TCP/IP stack on a Windows 2000 or Windows XP computer by customizing these registry values, which are stored in the registry key HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\:

• EnableICMPRedirect  When ICMP redirects are disabled (by setting the value to 0), attackers cannot carry out attacks that require a host to redirect the ICMP-based attack to a third party.
• SynAttackProtect  Enables SYN flood protection in Windows 2000 and Windows XP. You can set this value to 0, 1, or 2. The default setting, 0, provides no protection. Setting the value to 1 will activate SYN/ACK protection contained in the TCPMaxPortsExhausted, TCPMaxHalfOpen, and TCPMaxHalfOpenRetried values. Setting the value to 2 will protect against SYN/ACK attacks by more aggressively timing out open and half-open connections.
• TCPMaxConnectResponseRetransmissions  Determines how many times TCP retransmits an unanswered SYN/ACK message. TCP retransmits acknowledgments until the number of retransmissions specified by this value is reached.
• TCPMaxHalfOpen  Determines how many connections the server can maintain in the half-open state before TCP/IP initiates SYN flooding attack protection. This entry is used only when SYN flooding attack protection is enabled on this server—that is, when the value of the SynAttackProtect entry is 1 or 2 and the value of the TCPMaxConnectResponseRetransmissions entry is at least 2.
• TCPMaxHalfOpenRetired  Determines how many connections the server can maintain in the half-open state even after a connection request has been retransmitted. If the number of connections exceeds the value of this entry, TCP/IP initiates SYN flooding attack protection. This entry is used only when SYN flooding attack protection is enabled on this server—that is, when the value of the SynAttackProtect entry is 1 and the value of the TCPMaxConnectResponseRetransmissions entry is at least 2.
• TCPMaxPortsExhausted  Determines how many connection requests the system can refuse before TCP/IP initiates SYN flooding attack protection. The system must refuse all connection requests when its reserve of open connection ports runs out. This entry is used only when SYN flooding attack protection is enabled on this server—that is, when the value of the SynAttackProtect entry is 1, and the value of the TCPMaxConnectResponseRetransmissions entry is at least 2.
• TCPMaxDataRetransmissions  Determines how many times TCP retransmits an unacknowledged data segment on an existing connection. TCP retransmits data segments until they are acknowledged or until the number of retransmissions specified by this value is reached.
• EnableDeadGWDetect  Determines whether the computer will attempt to detect dead gateways. When dead gateway detection is enabled (by setting this value to 1), TCP might ask IP to change to a backup gateway if a number of connections are experiencing difficulty. Backup gateways are defined in the TCP/IP configuration dialog box in Network Control Panel for each adapter. When you leave this setting enabled, it is possible for an attacker to redirect the server to a gateway of his choosing.
• EnablePMTUDiscovery  Determines whether path MTU discovery is enabled (1), in which TCP attempts to discover the largest packet size over the path to a remote host. When path MTU discovery is disabled (0), the path MTU for all TCP connections will be fixed at 576 bytes.
• DisableIPSourceRouting  Determines whether a computer allows clients to predetermine the route that packets take to their destination. When this value is set to 2, the computer will disable source routing for IP packets.
• NoNameReleaseOnDemand  Determines whether the computer will release its NetBIOS name if requested by another computer or a malicious packet attempting to hijack the computer’s NetBIOS name.
• PerformRouterDiscovery  Determines whether the computer performs router discovery on this interface. Router discovery solicits router information from the network and adds the information retrieved to the route table. Setting this value to 0 will prevent the interface from performing router discovery.

Table 9-4 lists the registry entries that you can make to harden the TCP/IP stack on your Windows 2000 and Windows XP computers.

Table 9-4  Registry Settings to Harden TCP/IP

Value	Data (DWORD)
EnableICMPRedirect	0
SynAttackProtect	2
TCPMaxConnectResponseRetransmissions	2
TCPMaxHalfOpen	500
TCPMaxHalfOpenRetired	400
TCPMaxPortsExhausted	5
TCPMaxDataRetransmissions	3
EnableDeadGWDetect	0
EnablePMTUDiscovery	0
DisableIPSourceRouting	2
NoNameReleaseOnDemand	1
PerformRouterDiscovery	0

Additionally, you can secure the TCP/IP stack for Windows Sockets (Winsock) applications such as FTP servers and Web servers. The driver Afd.sys is responsible for connection attempts to Winsock applications. Afd.sys has been modified in Windows 2000 and Windows XP to support large numbers of connections in the half-open state without denying access to legitimate clients. Afd.sys can use dynamic backlog, which is configurable, rather than a static backlog. You can configure four parameters for the dynamic backlog:

• EnableDynamicBacklog  Switches between using a static backlog and a dynamic backlog. By default, this parameter is set to 0, which enables the static backlog. You should enable the dynamic backlog for better security on Winsock.
• MinimumDynamicBacklog  Controls the minimum number of free connections allowed on a listening Winsock endpoint. If the number of free connections drops below this value, a thread is queued to create additional free connections. Making this value too large (setting it to a number greater than 100) will degrade the performance of the computer.
• MaximumDynamicBacklog  Controls the maximum number of half-open and free connections to Winsock endpoints. If this value is reached, no additional free connections will be made.
• DynamicBacklogGrowthDelta  Controls the number of Winsock endpoints in each allocation pool requested by the computer. Setting this value too high can cause system resources to be unnecessarily occupied.

Each of these values must be added to the registry key HKLM\System\CurrentControlSet\Services\AFD\Parameters. Table 9-5 lists the parameters and the recommended levels of protection.

Table 9-5  Registry Settings to Harden Winsock

Value	Data (DWORD)
DynamicBacklogGrowthDelta	10
EnableDynamicBacklog	1
MinimumDynamicBacklog	20
MaximumDynamicBacklog	20,000

Using TCP/IP Filtering

Windows 2000 and Windows XP include support for TCP/IP filtering, a feature known as TCP/IP Security in Windows NT 4.0. TCP/IP filtering allows you to specify which types of inbound local host IP traffic are processed for all interfaces. This feature prevents traffic from being processed by the computer in the absence of other TCP/IP filtering, such as that provided by Routing and Remote Access (RRAS), Internet Connection Firewall (on Windows XP), and other TCP/IP applications or services. TCP/IP filtering is disabled by default.

When configuring TCP/IP filtering, you can permit either all or only specific ports or protocols listed for TCP ports, UDP ports, or IP protocols. Packets destined for the host are accepted for processing if they meet one of the following criteria:

• The destination TCP port matches the list of TCP ports.
• The destination UDP port matches the list of UDP ports.
• The IP protocol matches the list of IP protocols.
• The packet is an ICMP packet.

---

NOTE:
TCP/IP port filtering applies to all interfaces on the computer and cannot be applied on a per-adapter basis. However, you can configure allowed ports and protocols on a per-adapter basis.

---

In addition to being able to configure TCP/IP filtering on the Options tab of the TCP/IP advanced properties in the user interface, you can apply the settings directly to the registry. Table 9-6 lists the registry values to configure TCP/IP filtering. TCP/IP filtering is set in the key HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters, while the specific settings for each interface are configured in the key HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\Interface_GUID.

Table 9-6  Registry Values for TCP/IP Filtering

Setting	Type	Description
EnableSecurityFilters	DWORD	1 enables TCP/IP filtering; 0 disables TCP/IP filtering.
UdpAllowedPorts	MULTI_SZ	0 allows all UDP ports; an empty (null) value blocks all UDP ports; otherwise, the specific allowed UDP ports are listed.
TCPAllowedPorts	MULTI_SZ	0 allows all TCP ports; an empty (null) value blocks all TCP ports; otherwise, the specific allowed TCP ports are listed.
RawIpAllowedProtocols	MULTI_SZ	0 allows all IP protocols; an empty (null) value blocks all IP protocols; otherwise, the specific allowed IP protocols are listed.

Using Internet Connection Firewall in Windows XP

Windows XP includes a personal firewall called Internet Connection Firewall (ICF). ICF is a stateful firewall—it monitors all aspects of the communications between the Windows XP computer and other hosts, and it inspects the source and destination address of each message that it handles. To prevent unsolicited traffic from the public side of the connection from entering the private side, ICF keeps a table of all communications that have originated from the ICF computer. When used in conjunction with Internet Connection Sharing (ICS), ICF creates a table for tracking all traffic originated from the ICF/ICS computer and all traffic originated from private network computers. Inbound Internet traffic is allowed to reach the computers in your network only when a matching entry in the table shows that the communication exchange originated within your computer or private network. You can enable ICF on a per-interface basis on the Advanced tab of the interface.

You can configure services to allow unsolicited traffic from the Internet to be forwarded by the ICF computer to the private network. For example, if you are hosting an HTTP Web server service and have enabled the HTTP service on your ICF computer, unsolicited HTTP traffic will be forwarded by the ICF computer to the HTTP Web server. A set of operational information, known as a service definition, is required by ICF to allow the unsolicited Internet traffic to be forwarded to the Web server on your private network. The Services tab of ICF is shown in Figure 9-2.

Figure 9-2  Services tab of ICF (Image unavailable)

In addition, you can add custom services to the Services tab of ICF. ICF can also perform port translation for incoming connections. When you create a custom service, you will need to specify the following:

• Description of service  Determines how the service is displayed on the Services tab
• Name or IP address  Determines the host name or IP address of the computer offering the service if the service is not hosted on the local computer
• External port  Defines the TCP or UDP port on the ICF computer that will listen to inbound traffic to the service
• Internal port  Defines the TCP or UDP port to which the ICF computer will forward the inbound traffic to the computer defined in the Name Or IP Address field

Communications that originate from a source outside the ICF computer, such as the Internet, are dropped by the firewall unless an entry in the Services tab is made to allow passage. ICF silently discards unsolicited communications, preventing common attacks, such as port scanning and NetBIOS enumeration. ICF can create a security log so you can view the activity that is tracked by the firewall. You can choose whether to log dropped, successful, or dropped and successful packets. By default, packets are logged to c:\windows\pfirewall.log. The log file has a default maximum size of 4098 KB. Table 9-7 describes the fields in the packet log file.

Table 9-7 Description of Information Logged by ICF

Field	Description
Date	Specifies date that the recorded transactions occurred in the format YY-MM-DD.
Time	Specifies time that the recorded transaction occurred in the format HH:MM:SS.
Action	Specifies which operation was observed by the firewall. The options available to the firewall are OPEN, CLOSE, DROP, and INFO-EVENTS-LOST. An INFO-EVENTS-LOST action indicates the number of events that happened but were not placed in the log.
Protocol	Specifies which IP protocol was used for the communication.
Src-ip	Specifies the source IP address of the computer attempting to establish communications.
Dst-ip	Specifies the destination IP address of the communication attempt.
Src-port	Specifies the source port number of the sending computer. Only TCP and UDP will return a valid src-port entry.
Dst-port	Specifies the port of the destination computer. Only TCP and UDP will return a valid dst-port entry.
Size	Specifies the packet size in bytes.
Tcpflags	Specifies the TCP control flags found in the TCP header of an IP packet: ACK Acknowledgment field significant FIN No more data from sender PSH Push function RST Reset the connection SYN Synchronize sequence numbers URG Urgent Pointer field
Tcpsyn	Specifies the TCP synchronization number in the packet.
Tcpack	Specifies the TCP acknowledgment number in the packet.
Tcpwin	Specifies the TCP window size in bytes in the packet.
Icmptype	Specifies a number that represents the Type field of the ICMP message.
Icmpcode	Specifies a number that represents the Code field of the ICMP message.
Info	Specifies an information entry that depends on the type of action that occurred. For example, an INFO-EVENTS-LOST action will create an entry of the number of events that happened but were not placed in the log since the last occurrence of this event type.

Using IPSec

By its design, TCP/IP is an open protocol created to connect heterogeneous computing environments with the least amount of overhead possible. As is often the case, interoperability and performance design goals do not generally result in security—and TCP/IP is no exception to this. TCP/IP provides no native mechanism for the confidentiality or integrity of packets. To secure TCP/IP, you can implement IP Security. IPSec implements encryption and authenticity at a lower level in the TCP/IP stack than application-layer protocols such as Secure Sockets Layer (SSL) and Transport Layer Security (TLS). Because the protection process takes place lower in the TCP/IP stack, IPSec protection is transparent to applications. IPSec is a well-defined, standards-driven technology.

The IPSec process encrypts the payload after it leaves the application at the client and then decrypts the payload before it reaches the application at the server. An application does not have to be IPSec aware because the data transferred between the client and the server is normally transmitted in plaintext.

IPSec is comprised of two protocols that operate in two modes with three different authentication methods. IPSec is policy driven and can be deployed centrally by using Group Policy. To deploy IPSec, you must determine the

• Protocol
• Mode
• Authentication methods
• Policies

Securing Data Transmission with IPSec Protocols

As mentioned, IPSec is comprised of two protocols: IPSec Authentication Header (AH) and IPSec Encapsulating Security Payload (ESP). Each protocol provides different services; AH primarily provides packet integrity services, while ESP provides packet confidentiality services. IPSec provides mutual authentication services between clients and hosts, regardless of whether AH or ESP is being used.

Using AH

IPSec AH provides authentication, integrity, and anti-replay protection for the entire packet, including the IP header and the payload. AH does not provide confidentiality. When packets are secured with AH, the IPSec driver computes an Integrity Check Value (ICV) after the packet has been constructed but before it is sent to the computer. With Windows 2000 and Windows XP, you can use either the HMAC SHA1 or HMAC MD5 algorithm to compute the ICV. Figure 9-3 shows how AH modifies an IP packet.

Figure 9-3  AH modifications to an IP packet (Image unavailable)

The fields in an AH packet include these:

• Next Header  Indicates the protocol ID for the header that follows the AH header. For example, if the encrypted data is transmitted using TCP, the next header value would be 6, which is the protocol ID for TCP.
• Length  Contains the total length of the AH.
• Security Parameters Index (SPI)  Identifies the security association (the IPSec agreement between two computers) that was negotiated in the Internet Key Exchange (IKE) protocol exchange between the source computer and the destination computer.
• Sequence Number  Protects the AH-protected packet from replay attacks in which an attacker attempts to resend a packet that he has previously intercepted, such as an authentication packet, to another computer. For each packet issued for a specific security association (SA), the sequence number is incremented by 1 to ensure that each packet is assigned a unique sequence number. The recipient computer verifies each packet to ensure that a sequence number has not been reused. The sequence number prevents an attacker from capturing packets, modifying them, and then retransmitting them later.
• Authentication Data  Contains the ICV created against the signed portion of the AH packet by using either HMAC SHA1 or HMAC MD5. The recipient performs the same integrity algorithm and compares the result of the hash algorithm with the result stored within the Authentication Data field to ensure that the signed portion of the AH packet has not been altered in transit. Because the TTL, Type of Service (TOS), Flags, Fragment Offset, and Header Checksum fields are not used in the ICV, packets secured with IPSec AH can cross routers, which can change these fields.

Using ESP

ESP packets are used to provide encryption services to transmitted data. In addition, ESP provides authentication, integrity, and antireplay services. When packets are sent using ESP, the payload of the packet is encrypted and authenticated. In Windows 2000 and Windows XP, the encryption is done with either Data Encryption Standard (DES) or 3DES, and the ICV calculation is done with either HMAC SHA1 or HMAC MD5.

ESP encrypts the TCP or UDP header and the application data included within an IP packet. It does not include the original IP header unless IPSec tunnel mode is used. Figure 9-4 shows how ESP modifies an IP packet.

Figure 9-4  ESP modifications to an IP packet (Image unavailable)

The ESP header has two fields that are inserted between the original IP header and the TCP or UDP header from the original packet:

• Security Parameters Index (SPI)  Identifies the SA that was negotiated between the source computer and the destination computer for IPSec communication. The combination of the SPI, the IPSec protocol (AH or ESP), and the source and destination IP addresses identifies the SA used for the IPSec transmission within the ESP packet.
• Sequence Number  Protects the SA from replay attacks. This field is incremented by 1 to ensure that packets are never received more than once. If a packet is received with a previous sequence number, that packet is dropped.

The ESP trailer is inserted after the application data from the original packet and includes the following fields:

• Padding  A variable length from 0–255 bytes that brings the length of the application data and ESP trailer to a length divisible by 32 bits so that they match the required size for the cipher algorithm.
• Padding Length  Indicates the length of the Padding field. After the packet is decrypted, this field is used to determine the length of the Padding field.
• Next Header  Identifies the protocol used for the transmission of the data, such as TCP or UDP.

Following the ESP trailer, the ESP protocol adds an ESP authentication trailer to the end of the packet. The ESP authentication trailer contains a single field:

• Authentication Data  Contains the ICV, which verifies the originating host that sent the message and ensures that the packet was not modified in transit. The ICV uses the defined integrity algorithm to calculate the ICV. The integrity algorithm is applied to the ESP header, the TCP/UDP header, the application data, and the ESP trailer.

ESP provides integrity protection for the ESP header, the TCP/UDP header, the application data, and the ESP trailer. ESP also provides inspection protection by encrypting the TCP/UDP header, the application data, and the ESP trailer.

Choosing Between IPSec Modes

IPSec operates in two modes: transport mode and tunnel mode. IPSec transport mode is used for host-to-host connections, and IPSec tunnel mode is used for network-to-network or host-to-network connections.

Using IPSec Transport Mode

IPSec transport mode is fully routable, as long as the connection does not cross a network address translation (NAT) interface, which would invalidate the ICV. Used this way, IPSec must be supported on both hosts, and each host must support the same authentication protocols and have compatible IPSec filters configured and assigned. IPSec transport mode is used to secure traffic from clients to hosts for connections where sensitive data is passed.

Using IPSec Tunnel Mode

IPSec tunnel mode is used for network-to-network connections (IPSec tunnels between routers) or host-to-network connections (IPSec tunnels between a host and a router). Used this way, IPSec must be supported on both endpoints, and each endpoint must support the same authentication protocols and have compatible IPSec filters configured and assigned. IPSec tunnel mode is commonly used for site-to-site connections that cross public networks, such as the Internet.

Selecting an IPSec Authentication Method

During the initial construction of the IPSec session—also known as the Internet Key Exchange, or IKE—each host or endpoint authenticates the other host or endpoint. When configuring IPSec, you must ensure that each host or endpoint supports the same authentication methods. IPSec supports three authentication methods:

• Kerberos
• X.509 certificates
• Preshared key

Authenticating with Kerberos

In Windows 2000 and Windows XP, Kerberos is used for the IPSec mutual authentication by default. For Kerberos to be used as the authentication protocol, both hosts or endpoints must receive Kerberos tickets from the same Active Directory directory service forest. Thus, you should choose Kerberos for IPSec authentication only when both hosts or endpoints are within you own organization. Kerberos is an excellent authentication method for IPSec because it requires no additional configuration or network infrastructure.

To remove the exemption for Kerberos and RSVP, set the value NoDefaultExempt to 1 in the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPSEC, or use the Nodefaultexempt.vbs script located in the Tools\Scripts folder on the CD included with this book.

Authenticating with X.509 Certificates

You can use X.509 certificates for IPSec mutual authentication of hosts or endpoints. Certificates allow you to create IPSec secured sessions with hosts or endpoints outside your Active Directory forests, such as business partners in extranet scenarios. You also must use certificates when using IPSec to secure VPN connections made by using Layer Two Tunneling Protocol (L2TP). To use certificates, the hosts must be able to validate that the other’s certificate is valid.

Authenticating with Preshared Key

You can use a preshared key, which is a simple, case-sensitive text string, to authenticate hosts or endpoints. Preshared key authentication should be used only when testing or troubleshooting IPSec connectivity because the preshared key is not stored in a secure fashion by hosts or endpoints.

Creating IPSec Policies

IPSec is a policy-driven technology. In Windows 2000 and Windows XP, you can have only one IPSec policy assigned at a time. IPSec policies are dynamic, meaning that you do not have to stop and start the IPSec service or restart the computer when assigning or unassigning IPSec policies. You can also use Group Policy to deploy IPSec policies to Windows 2000 and Windows XP clients. Windows 2000 and Windows XP include three precreated IPSec policies:

• Client (Respond Only)  A computer configured with the Client policy will use IPSec if the host it is communicating with requests using IPSec and supports Kerberos authentication.
• Server (Request Security)  A computer configured with the Server policy will always attempt to negotiate IPSec but will permit unsecured communication with hosts that do not support IPSec. The Server policy permits unsecured ICMP traffic.
• Secure Server (Require Security)  A computer configured with the Secure Server policy will request that IPSec be used for all inbound and outbound connections. The computer will accept unencrypted packets but will always respond by using IPSec secured packets. The Secure Server policy permits unsecured ICMP traffic.

In addition to the built-in policies, you can create custom IPSec policies. When creating your own IPSec policies, you must configure rules that include the following settings:

• IP Filter List
• Tunnel Settings
• Filter Actions
• Authentication Methods
• Connection Types

IPSec rules determine what types of network traffic will initiate IPSec between the computer and the host or endpoint it is communicating with. A computer can have any number of IPSec filters. You should ensure that only one rule is created for each type of traffic. If multiple filters apply to a given type of traffic, the most specific filter will be processed first.

IP Filter List

The IP filter list defines the types of network traffic that the IPSec rule applies to. You must define the following details for each entry in the filter list:

• Source address  Can be a specific IP address, a specific IP subnet address, or any address.
• Destination address  Can be a specific IP address, a specific IP subnet address, or any address.
• Protocol  The protocol ID or transport protocol used by the protocol. For example, Point-to-Point Tunneling Protocol (PPTP) uses Generic Routing Encapsulation (GRE) packets. GRE packets are identified by their protocol ID, which is protocol ID 47. Telnet, on the other hand, uses TCP as its transport protocol, so an IPSec filter for Telnet would only define the protocol type as TCP.
• Source port  If the protocol were to use TCP or UDP, the source port could be defined for the protected connection. The source port is set to a specific port or to a random port, depending on the protocol being defined. Most protocols use a random port for the source port.
• Destination port  If the protocol uses TCP or UDP, the protocol uses a specific port at the server to accept transmissions. For example, Telnet configures the server to listen for connections on TCP port 23.

When configuring IP filter lists for transport mode connections, you should always choose to have the IPSec rule mirrored to secure the return communication defined in the rule. For tunnel mode connections, you must manually specify both the inbound and outbound filter list.

Tunnel Settings

The tunnel setting determines whether IPSec operates in transport or tunnel mode. If you want IPSec to operate in transport mode, select This Rule Does Not Specify A Tunnel when creating an IPSec rule using the Security Rule Wizard. If you want the filter to operate in tunnel mode, you must specify the IP address of the endpoint of the tunnel.

Filter Actions

For each filter rule, you must choose a filter action. The filter action defines how the traffic defined in the IP filter will be handled by the filter rule. The three filter actions are listed here and shown in Figure 9-5.

• Permit  Allows packets to be transmitted without IPSec protection. For example, Simple Network Management Protocol (SNMP) includes support for devices that might not be IPSec aware. Enabling IPSec for SNMP would cause a loss of network management capabilities for these devices. In a highly secure network, you could create an IPSec filter for SNMP and set the IPSec action to Permit to allow SNMP packets to be transmitted without IPSec protection.
• Block  Discards packets. If the associated IPSec filter is matched, all packets with the block action defined are discarded.
• Negotiate Security  Allows an administrator to define the desired encryption and integrity algorithms to secure data transmissions if an IPSec filter is matched.

Figure 9-5  IPSec filter actions

In addition to these three basic actions, you can define settings that indicate how the Windows 2000–based computer will react if non-IPSec protected data is received and how frequently new session keys are defined to protect the IPSec data. Options include the following:

• Accept Unsecured Communication, But Always Respond Using IPSec  You use this option when the IPSec protection is enforced only at the servers, not at the clients. In a typical IPSec deployment, clients are configured to use IPSec if requested by the server but to never initiate an IPSec SA. This setting allows the initial packet to be received by the server, which then starts the IKE process to negotiate an SA between the client and the server. Although it is riskier to have the initial packet of a data transmission accepted by using plaintext, the response packet sent from the server will not be transmitted until an SA is established.
• Allow Unsecured Communication—With Non-IPSec-Aware Computers  In a mixed network, this option allows non-IPSec aware clients to connect to the server. Windows 2000 clients, if configured to do so, will connect to the server and negotiate IPSec protection. Non-IPSec-aware clients will still be allowed to connect by using unprotected data streams.
• Session Key Perfect Forward Secrecy  Using Perfect Forward Secrecy will ensure that an existing key is never used as the foundation of a new key. When you use Perfect Forward Secrecy, all keys will be generated without using existing keys. This reduces the risk of continual data exposure should a key be compromised because previous keys cannot be used to determine future keys.

Authentication Methods

For each filter rule, you must choose an authentication method. You can enable multiple authentication methods for each rule and determine their order of precedence by editing the filter rule after it has been created.

Connection Types

You must specify what type of interfaces each filter rule applies to. In Windows 2000 and Windows XP, you can choose to have the rule apply to the following:

• All network connections
• Local area network (LAN) connections
• Remote access connections

How IPSec Works

IPSec can be initiated by either the sending host or the receiving host. The two hosts or endpoints enter into a negotiation that will determine how the communication will be protected. The negotiation is completed in the IKE, and the resulting agreement is a set of security associations, or SAs.

IKE has two modes of operation, main mode and quick mode. We will examine each mode momentarily. IKE also serves two functions:

• Centralizes SA management, reducing connection time
• Generates and manages the authenticated keys used to secure the information

The SA is used until the two hosts or endpoints cease communication, even though the keys used might change. A computer can have many SAs. The SA for each packet is tracked using the SPI.

Main Mode

During the main mode negotiation, the two computers establish a secure, authenticated channel—the main mode SA. IKE automatically provides the necessary identity protection during this exchange. This ensures no identity information is sent without encryption between the communicating computers, thus enabling total privacy. Following are the steps in a main mode negotiation:

1. Policy negotiation These four mandatory parameters are negotiated as part of the main mode SA:
• The encryption algorithm (DES or 3DES)
• The hash algorithm (MD5 or SHA1)
• The authentication method (certificate, preshared key, or Kerberos v5 authentication)
• The Diffie-Hellman (DH) group to be used for the base keying material

If certificates or preshared keys are used for authentication, the computer identity is protected. However, if Kerberos v5 authentication is used, the computer identity is unencrypted until encryption of the entire identity payload takes place during authentication.

2. DH exchange (of public values) At no time are actual keys exchanged; only the base information needed by DH to generate the shared, secret key is exchanged. After this exchange, the IKE service on each computer generates the master key used to protect the final step: authentication.
3. Authentication The computers attempt to authenticate the DH exchange. Without successful authentication, communication cannot proceed. The master key is used, in conjunction with the negotiation algorithms and methods, to authenticate identities. The entire identity payload—including the identity type, port, and protocol—is hashed and encrypted by using the keys generated from the DH exchange in the second step. The identity payload, regardless of which authentication method is used, is protected from both modification and interpretation.

After the hosts have mutually authenticated each other, the host that initiated the negotiation presents an offer for a potential SA to the receiving host. The responder cannot modify the offer. Should the offer be modified, the initiator rejects the responder’s message. The responder sends either a reply accepting the offer or a reply with alternatives. After the hosts agree on an SA, quick mode negotiation begins.

Quick Mode

In this mode, SAs are negotiated on behalf of the IPSec service. The following are the steps in quick mode negotiation:

1. Policy negotiation The IPSec computers exchange their requirements for securing the data transfer:
• The hash algorithm for integrity and authentication (MD5 or SHA1)
• The algorithm for encryption, if requested (3DES or DES)
• A description of the traffic to protect

2. Session key material refresh or exchange IKE refreshes the keying material, and new, shared, or secret keys are generated for authentication and encryption (if negotiated) of the packets. If a rekey is required, a second DH exchange takes place or a refresh of the original DH exchange occurs.
3. SA exchange The SAs and keys are passed to the IPSec driver, along with the SPI.

A common agreement is reached, and two SAs are established: one for inbound communication, and one for outbound communication.

During the quick mode negotiation of shared policy and keying material, the information is protected by the SA negotiated during main mode. As mentioned in step 3, quick mode results in a pair of SAs: one for inbound communication and one for outbound communication, each having its own SPI and key. Figure 9-6 shows a summary of what is negotiated during main mode and quick mode.

Figure 9-6  Main mode and quick mode negotiation (Image unavailable)

• IP protocol ID 50  Create inbound and outbound filters to allow ESP traffic to be forwarded.
• IP protocol ID 51  Create inbound and outbound filters to allow AH traffic to be forwarded.
• UDP port 500  Create inbound and outbound filters to allow IKE traffic to be forwarded.

Because of the nature of the NAT and port address translation (PAT) technologies, which require that packets be altered to change IP address and port information, IPSec is not compatible with NAT. IPSec does not allow manipulation of packets during transfer. The IPSec endpoint will discard packets that have been altered by NAT because the ICVs will not match. At the time of the printing of this book, research into encapsulating IPSec packets in UDP packets so that they can pass through NAT devices is under way.

Monitoring IPSec

You can monitor IPSec in Windows 2000 with IPSecmon.exe and in Windows XP the IP Security Monitor Microsoft Management Console (MMC) snap-in. In addition, you can create log files in both Windows 2000 and Windows XP to view IPSec negotiations.

Using IPSecmon in Windows 2000

In Windows 2000, you can view the status of IPSec SAs and basic information on IPSec sessions by running IPSecmon from the Run prompt. IPSecmon displays information about each SA and the overall statistics of IPSec and IKE sessions. Figure 9-7 shows IPSecmon in Windows 2000. The built-in Server IPSec policy is applied to the Windows 2000 computer named SFOFS001. The SFOFS001 computer has attempted to negotiate IPSec with three other computers: SEADC001, SFODC001, and SFOXP001. However, SFOFS001 has successfully negotiated an SA with SFOXP001 only. The IPSec session with SFPXP001 uses the IPSec protocol ESP with 3DES as the encrypting algorithm and HMAC SHA1 as the authentication algorithm.

Figure 9-7  Using IPSecmon in Windows 2000 (Image unavailable)

Using the IP Security Monitor MMC Snap-In

In Windows XP, IPSecmon has been replaced with an MMC snap-in that provides all the information that IPSecmon did in Windows 2000, only in much greater detail. You can use the IP Security Monitor MMC snap-in to view details of each SA, whereas in Windows 2000, you could view only the basic details of an SA. Figure 9-8 shows the IP Security Monitor MMC snap-in in Windows XP, which enables you to view the exact SA details negotiated during both main mode and quick mode.

Figure 9-8  Using IP Security Monitor MMC snap-in in Windows XP (Image unavailable)

Using IPSec Logs in Windows 2000 and Windows XP

In both Windows 2000 and Windows XP, you can have IPSec log the IKE exchanges to a log file on the hard drive for troubleshooting or monitoring needs. To have your computer log IKE exchanges, you must create a registry value named EnableLogging in the registry key HKLM\System\CurrentControlSet\Services\PolicyAgent\Oakley. To enable logging, set the value to 1 and restart the IPSec services. The log file will be written to the file %systemroot%\debug\oakley.log. Ipseclog.vbs automatically configures the registry in Windows 2000 and Windows XP to enable IPSec logging. This file is located in the Tools\Scripts folder on the CD included with this book.

Best Practices

• Create a TCP/IP hardening policy.  Ensure that the TCP/IP stack on your Windows 2000 and Windows XP computers is appropriately secure, based on the threats to it. This is especially true of any computer directly connected to the Internet or in screened subnets.
• Use ICF for mobile and home computers running Windows XP.  ICF provides an excellent degree of protection for mobile clients and home computers. Be certain to provide training to users on how to enable and disable ICF.
• Use IPSec hardware accelerators when possible.  By using IPSec hardware accelerators on computers that will have many IPSec sessions at a time, such as servers, you can prevent the computer’s CPU performance from being overly taxed.

Additional Information

• Internet Assigned Numbers Authority’s (IANA) TCP and UDP port number assignment list
• IANA’s IP protocol ID number list
• "5-Minute Security Advisor—Essential Security Tools for Home Office Users"
• Internet Connection Firewall Overview—Windows XP Help File
• "IPSec Architecture" white paper
• "IPSec Implementation" white paper
• "Security Considerations for Network Attacks" white paper
• "Best Practices for Preventing DoS/Denial of Service Attacks" white paper
• 309798: "How to Configure TCP/IP Filtering in Windows 2000"