IPSec Technical Reference

IPSec Technical Reference
Internet Protocol security (IPSec) in the Microsoft Windows Server 2003 operating system helps protect networks
from active and passive attacks by securing IP packets through the use of packet filtering, cryptographic security
services, and the enforcement of trusted communications. IPSec is integrated within the security framework of
Windows Server 2003, which allows you to use Active Directory domains to provide identity and manage trust
relationships between users and computers in different departments within an organization.
This subject provides a high-level description of IPSec that includes the scenarios for which it is intended and not
In This Subject
What Is IPSec?
How IPSec Works
IPSec Tools and Settings
What Is IPSec?
What Is IPSec?
In This Subject
IPSec Scenarios
IPSec Dependencies
IPSec and ICF
Related Information
Internet Protocol security (IPSec) is a framework of open standards for helping to ensure private, secure
communications over Internet Protocol (IP) networks through the use of cryptographic security services. IPSec
supports network-level data integrity, data confidentiality, data origin authentication, and replay protection.
Because IPSec is integrated at the Internet layer (layer 3), it provides security for almost all protocols in the
TCP/IP suite, and because IPSec is applied transparently to applications, there is no need to configure separate
security for each application that uses TCP/IP.
IPSec helps provide defense-in-depth against:
Network-based attacks from untrusted computers, attacks that can result in the denial-of-service of
applications, services, or the network
Data corruption
Data theft
User-credential theft
Administrative control of servers, other computers, and the network.
You can use IPSec to defend against network-based attacks through a combination of host-based IPSec packet
filtering and the enforcement of trusted communications.
IPSec is integrated with the Windows Server 2003 operating system and it can use the Active Directory directory
service as a trust model. You can use Group Policy to configure Active Directory domains, sites, and
organizational units (OUs), and then assign IPSec policies as required to Group Policy objects (GPOs). In this
way, IPSec policies can be implemented to meet the security requirements of many different types of
This section describes the solution that IPSec is intended to provide by providing information about core IPSec
scenarios, IPSec dependencies, and related technologies.
The following figure shows an Active Directory-based IPSec policy being distributed to two IPSec peers and
IPSec-protected communications being established between those two peers.
Two IPSec Peers Using Active Directory-based IPSec Policy
The MicrosoftWindows implementation of IPSec is based on standards developed by the Internet Engineering
Task Force (IETF) IPSec working group. For a list of relevant IPSec RFCs, see the “Related Information” section
later in this subject.
IPSec Scenarios
IPSec is a general-purpose security technology that can be used to help secure network traffic in many scenarios.
However, you must balance the need for security with the complexity of configuring IPSec policies. Additionally,
due to a lack of suitable standards, IPSec is not appropriate for some types of connectivity. This section describes
IPSec scenarios that are recommended, IPSec scenarios that are not recommended, and IPSec scenarios that
require special consideration.
Recommended Scenarios for IPSec
IPSec is recommended for the following scenarios:
Packet filtering
End-to-end security between specific hosts
End-to-end traffic through a Microsoft Internet Security and Acceleration (ISA) Server-secured network
address translator
Secure server
Layer Two Tunneling Protocol (L2TP) over IPSec (L2TP/IPSec) for remote access and site-to-site virtual
private network (VPN) connections
Site-to-site IPSec tunneling with non-Microsoft IPSec gateways
Packet Filtering
IPSec can perform host-based packet filtering to provide limited firewall capabilities for end systems. You can
configure IPSec to permit or block specific types of unicast IP traffic based on source and destination address
combinations and specific protocols and specific ports. For example, nearly all the systems illustrated in the
following figure can benefit from packet filtering to restrict communication to only specific addresses and ports.
You can strengthen security by using IPSec packet filtering to control exactly the type of communication that is
allowed between systems.
Filtering Packets by Using IPSec
As illustrated in this figure:
The internal network domain administrator can assign an Active Directory-based IPSec policy (a
collection of security settings that determines IPSec behavior) to block all traffic from the perimeter
network (also known as a demilitarized zone [DMZ], demilitarized zone, or screened subnet).
The perimeter network domain administrator can assign an Active Directory-based IPSec policy to block
all traffic to the internal network.
The administrator of the computer running Microsoft SQL Server on the internal network can create an
exception in the Active Directory-based IPSec policy to permit structured query language (SQL) protocol
traffic to the Web application server on the perimeter network.
The administrator of the Web application server on the perimeter network can create an exception in
the Active Directory-based policy to permit SQL traffic to the computer running SQL Server on the
internal network.
The administrator of the Web application server on the perimeter network can also block all traffic from
the Internet, except requests to TCP port 80 for the HyperText Transfer Protocol (HTTP) and TCP port
443 for HTTPS (HTTP over Secure Sockets Layer/Transport Layer Protocol [SSL/TLS]), which are used
by Web services. This provides additional security for traffic allowed from the Internet in case the
firewall was misconfigured or compromised by an attacker.
The domain administrator can block all traffic to the management computer, but allow traffic to the
perimeter network.
You can also use IPSec with the IP packet-filtering capability or NAT/Basic Firewall component of the Routing and
Remote Access service to permit or block inbound or outbound traffic, or you can use IPSec with the Internet
Connection Firewall (ICF) component of Network Connections, which provides stateful packet filtering. However,
to ensure proper Internet Key Exchange (IKE) management of IPSec security associations (SAs), you must
configure ICF to permit UDP port 500 and port 4500 traffic needed for IKE messages.
End-to-End Security Between Specific Hosts
IPSec establishes trust and security from a unicast source IP address to a unicast destination IP address (end-toend). For example, IPSec can help secure traffic between Web servers and database servers or domain
controllers in different sites. As shown in the following figure, only the sending and receiving computers need to
be aware of IPSec. Each computer handles security at its respective end and assumes that the medium over
which the communication takes place is not secure. The two computers can be located near each other, as on a
single network segment, or across the Internet. Computers or network elements that route data from source to
destination are not required to support IPSec.
Securing Communications Between a Client and a Server by Using IPSec
The following figure shows domain controllers in two forests that are deployed on opposite sides of a firewall. In
addition to using IPSec to help secure all traffic between domain controllers in separate forests, as shown in the
figure, you can use IPSec to help secure all traffic between two domain controllers in the same domain and
between domain controllers in parent and child domains.
Securing Communications Between Two Domain Controllers in Different Forests by Using IPSec
End-to-End Traffic Through an ISA-Secured Network Address Translator
Windows Server 2003 supports IPSec NAT Traversal (NAT-T). IPSec NAT-T allows traffic to be secured by IPSec
and also to be translated by a network address translator. For example, you can use IPSec transport mode to
help secure host-to-host traffic through a computer that is running ISA Server and that is functioning as a
network address translator if ISA (or any other NAT device) does not need to inspect the traffic between the two
hosts. IPSec transport mode is used to protect traffic between hosts and it can provide security between
computers that are on the same local area network (LAN) or connected by private wide area network (WAN)
links. In the following figure, a computer running Windows Server 2003 and Microsoft Internet Security and
Acceleration (ISA) Server is functioning as a network address translator. The IPSec policy on Server A is
configured to secure traffic to the IP address of Server B, while the IPSec policy on Server B is configured to
secure traffic to the external IP address of the computer running ISA Server.
Securing Communications Through an ISA-Secured NAT by Using IPSec NAT-T
Secure Server
You can require IPSec protection for all client computers that access a server. In addition, you can set restrictions
on which computers are allowed to connect to a server running Windows Server 2003. The following figure shows
IPSec in transport mode securing a line of business (LOB) application server.
Securing an Application Server by Using IPSec
In this scenario, an application server in an internal corporate network must communicate with clients running
Windows 2000 or Windows XP Professional; a Windows Internet Name Service (WINS) server, Domain Name
System (DNS) server, and Dynamic Host Configuration Protocol (DHCP) server; Active Directory domain
controllers; and a non-Microsoft data backup server. The users on the client computers are company employees
who access the application server to view their personal payroll information and performance review scores.
Because the traffic between the clients and the application server involves highly sensitive data, and because the
server should only communicate with other domain members, the network administrator uses an IPSec policy
that requires ESP encryption and communication only with trusted computers in the Active Directory domain.
Other traffic is permitted as follows:
Traffic between the WINS server, DNS server, DHCP server, and the application server is permitted
because WINS servers, DNS servers, and DHCP servers must typically communicate with computers
that run on a wide range of operating systems, some of which might not support IPSec.
Traffic between Active Directory domain controllers and the application server is permitted, because
using IPSec to secure communication between domain members and their domain controllers is not a
recommended usage.
Traffic between the non-Microsoft data backup server and the application server is permitted because
the non-Microsoft backup server does not support IPSec.
L2TP/IPSec for Remote Access and Site-to-Site VPN Connections
You can use L2TP/IPSec for all VPN scenarios. This does not require the configuration and deployment of IPSec
policies. Two common scenarios for L2TP/IPSec are securing communications between remote access clients and
the corporate network across the Internet and securing communications between branch offices.
Windows IPSec supports both IPSec transport mode and tunnel mode. Although VPN connections are
commonly referred to as “tunnels,” IPSec transport mode is used for L2TP/IPSec VPN connections.
IPSec tunnel mode is most commonly used to help protect site-to-site traffic between networks, such as
site-to-site networking through the Internet.
L2TP/IPSec for remote access connections
A common requirement for organizations is to secure communications between remote access clients and the
corporate network across the Internet. Such a client might be a sales consultant who spends most of the time
traveling, or an employee working from a home office. In the following figure, the remote gateway is a server
that provides edge security for the corporate intranet. The remote client represents a roaming user who requires
regular access to network resources and information. An ISP is used as an example to demonstrate the path of
communication when the client uses an ISP to access the Internet. L2TP/IPSec provides a simple, efficient way to
build a VPN tunnel and help protect the data across the Internet.
Securing Remote Access Clients by Using L2TP/IPSec
L2TP/IPSec for site-to-site VPN connections
A large corporation often has multiple sites that require communication — for example, a corporate office in New
York and a sales office in Washington. In this case, L2TP/IPSec provides the VPN connection and helps protect
the data between the sites. In the following figure, the router running Windows Server 2003 provides edge
security. The routers might have a leased line, dial-up, or other type of Internet connection. The L2TP/IPSec VPN
tunnel runs between the routers only and provides protected communication across the Internet.
Establishing an L2TP/IPSec VPN Tunnel Between Sites
Site-to-Site IPSec Tunneling with Non-Microsoft Gateways
For interoperability with gateways or end systems that do not support L2TP/IPSec or Point-to-Point Tunneling
Protocol (PPTP) VPN site-to-site connections, you can use IPSec in tunnel mode. When IPSec tunnel mode is
used, the sending gateway encapsulates the entire IP datagram by creating a new IP packet that is then
protected by one of the IPSec protocols. The following figure illustrates site-to-site IPSec tunneling.
Establishing an IPSec Gateway-to-Gateway Tunnel Between Sites
In this figure, traffic is being sent between a client computer in a vendor site (Site A) and a File Transfer Protocol
(FTP) server at the corporate headquarters site (Site B). Although an FTP server is used for this scenario, the
traffic can be any unicast IP traffic. The vendor uses a non-Microsoft IPSec-enabled gateway, while corporate
headquarters uses a gateway running Windows Server 2003. An IPSec tunnel is used to secure traffic between
the non-Microsoft gateway and the gateway running Windows Server 2003.
Scenarios for Which IPSec Is Not Recommended
IPSec policies can be quite complex to configure and manage. Additionally, IPSec can incur performance
overhead to establish and maintain secure connections, and it can incur network latency. In some deployment
scenarios, the lack of standard methods for user authentication and address assignment make IPSec an
unsuitable choice. Because IPSec depends on IP addresses for establishing secure connections, you cannot
specify dynamic IP addresses. It is often necessary for a server to have a static IP address in IPSec policy filters.
In large network deployments and in some mobile user cases, using dynamic IP addresses at both ends of the
connection can increase the complexity of IPSec policy design. For these reasons, IPSec is not recommended for
the following scenarios:
Securing communication between domain members and their domain controllers
Securing all traffic in a network
Securing traffic for remote access VPN connections using IPSec tunnel mode
Securing Communication Between Domain Members and their Domain Controllers
Using IPSec to help secure traffic between domain members (either clients or servers) and their domain
controllers is not recommended because:
If domain members were to use IPSec-secured communication with domain controllers, increased
latency might occur, causing authentication and the process of locating a domain controller to fail.
Complex IPSec policy configuration and management is required.
Increased load is placed on the domain controller CPU to maintain SAs with all domain members.
Depending on the number of domain members in the domain controller’s domain, such a load might
overburden the domain controller.
Securing All Traffic in a Network
In addition to reduced network performance, using IPSec to help secure all traffic in a network is not
recommended because:
IPSec cannot secure multicast and broadcast traffic.
Traffic from real-time communications, applications that require Internet Control Message Protocol
(ICMP), and peer-to-peer applications might be incompatible with IPSec.
Network management functions that must inspect the TCP, UDP, and other protocol headers are less
effective, or cannot function at all, due to IPSec encapsulation or encryption of IP payloads.
Securing Traffic for Remote Access VPN Connections by Using IPSec Tunnel Mode
IPSec tunnel mode is not a recommended technology for remote access VPN connections, because there are no
standard methods for user authentication, IP address assignment, and name server address assignment. Using
IPSec tunnel mode for gateway-to-gateway VPN connections is possible using computers running Windows Server
2003. But because the IPSec tunnel is not represented as a logical interface over which packets can be forwarded
and received, routes cannot be assigned to use the IPSec tunnel and routing protocols do not operate over IPSec
tunnels. Therefore, the use of IPSec tunnel mode is only recommended as a VPN solution for site-to-site VPN
connections in which one end of the tunnel is a non-Microsoft VPN server or security gateway that does not
support L2TP/IPSec. Instead, use L2TP/IPSec or PPTP for remote access VPN connections.
IPSec Uses That Require Special Considerations
The following scenarios merit special consideration, because they introduce an additional level of complexity for
IPSec policy configuration and management:
Securing traffic over IEEE 802.11wireless networks
Securing traffic in home networking scenarios
Securing traffic in environments that use dynamic IP addresses
Securing Traffic Sent over 802.11 Networks
You can use IPSec transport mode to protect traffic sent over 802.11 wireless networks. However, IPSec is not
the recommended solution for providing security for corporate 802.11 wireless LAN networks. Instead, it is
recommended that you use either 802.11 Wired Equivalent Privacy (WEP) encryption or Wi-Fi Protected Access
(WPA) and IEEE 802.1X authentication.
To use IPSec to help secure traffic sent over 802.11 networks, you must ensure that client computers and
servers support IPSec. Configuration management and trust are also required on client computers and servers
when IPSec is used. Because many computers on a network do not support IPSec or are not managed, it is not
appropriate to use IPSec alone to protect all 802.11 corporate wireless LAN traffic.
Securing Traffic in Home Networking Scenarios
Although IPSec is not optimized for use in general home networking scenarios, when network security
administrators deploy IPSec with appropriate scripts and support tools, it can be used effectively on home
computers for specific scenarios.
IPSec can be used to connect home computers to a corporate intranet for remote access. Network security
administrators can use scripts and support tools to deploy IPSec on the home computers of employees who
require secure connectivity to the corporate network. For example, an administrator can use a Connection
Manager profile to deploy an L2TP/IPSec-based VPN connection on home computers. Employees can then
establish IPSec-secured connections across the Internet to the corporate network by using the VPN client built-in
to Network Connections.
In some cases, non-Microsoft VPN or firewall clients might disable the IPSec service, which is required
for IPSec to function. If you encounter this problem, it is recommended that you contact the VPN or
firewall vendor.
IPSec is not recommended for end users in general home networking scenarios for the following reasons:
The IPSec policy configuration user interface (IP Security Policy Management) is intended for
professional network security administrators, rather than for end users. Improper policy configuration
can result in blocked communications, and if problems occur, built-in support tools are not yet available
to aid end users in troubleshooting.
Some home networking applications use broadcast and multicast traffic, for which IPSec cannot
negotiate security.
Many home networking scenarios use a wide range of dynamic IP addresses.
Many home networking scenarios involve the use of a network address translator. To use IPSec across a
NAT, both IPSec peers must support IPSec NAT-T.
Securing Traffic in Environments That Use Dynamic IP Addresses
IPSec depends on IP addresses for establishing secure connections, and it is often necessary for a server to have
a static IP address in IPSec policy filters. In large network deployments and in some mobile user cases, using
dynamic IP addresses at both ends of the connection can increase the complexity of IPSec policy design.
IPSec Dependencies
There is no single optimal environment for IPSec. However, there are dependencies that are critical to the
successful deployment of IPSec. This section describes how the following two IPSec dependencies affect the
deployment of IPSec:
Active Directory (if your deployment requires the use of Active Directory-based IPSec policies, rather
than local IPSec policies)
Successful mutual authentication
Active Directory
For organizations with large numbers of computers that must be managed in a consistent way, it is best to
distribute IPSec policies by using Group Policy to configure Active Directory domains, sites, and organizational
units (OUs), and then assigning IPSec policies as required to Group Policy objects (GPOs). Although you can
assign local IPSec policies to computers that are not members of a trusted domain, distributing IPSec policies and
managing IPSec policy configuration and trust relationships is much more time-consuming for computers that are
not members of a trusted domain.
If you do use Active Directory-based IPSec policies, IPSec policy design and management must take into account
the delays that result from the replication of Group Policy data from domain controllers to domain members.
Often, the first step in troubleshooting a problem with IPSec connectivity is to determine whether the computer in
question has the most current Group Policy assignment. To do this, you must be a member of the local
Administrators group on the computer for which troubleshooting is being performed.
Successful Mutual Authentication
For IPSec-secured communications to be established, there must be successful mutual authentication between
IPSec peers. IPSec requires the use of one of the following authentication methods: Kerberos version 5, an
X.509 version 3 computer certificate issued by a public key infrastructure (PKI), or a preshared key. The two
IPSec peers must use at least one common authentication method or communication will fail. Make sure that you
choose an authentication method that is appropriate for your environment.
When you deploy IPSec to negotiate security for upper-layer protocols such as TCP connections, failures to
communicate are often caused by the failure of IPSec to mutually authenticate the two communication endpoints.
Authentication might succeed for some computers and fail for others due to issues within the authentication
system itself (typically, Kerberos or public key certificates, rather than preshared keys, because preshared keys
are not recommended). For these reasons, it is important to evaluate how the dependency of IPSec connectivity
on authentication affects your environment and support practices. Additional training is recommended so that
administrators can quickly determine whether a connectivity problem is caused by an IPSec authentication failure
and understand how to investigate the authentication system.
IPSec and ICF
IPSec is similar to the ICF feature of Network Connections. However, there are important differences between
these two technologies as well. It is important to understand the similarities and differences, so that you can
deploy IPSec where it is truly needed and obtain the maximum benefits from the security that IPSec provides.
This section describes similarities and differences between IPSec and ICF.
Microsoft ICF is a locally managed, stateful host firewall that, by default, discards all incoming packets except
those sent in response to packets sent by the host. One primary difference between IPSec and ICF is that IPSec
provides complex static filtering based on IP addresses, while ICF provides stateful filtering for all addresses on a
network interface. For example, when you use IPSec, you can configure a filter to block all inbound traffic that is
sent to a specific IP address over a specific protocol and port (for example, “Block all inbound traffic from the
Internet to TCP port 135”). You can also configure exemptions to permit specific types of traffic from specific
source IP addresses. When you use ICF, you can configure exemptions to permit traffic based solely on the port,
regardless of the source IP address.
It is recommended that you use ICF when you want a firewall for a network interface that can be accessed
through the Internet. It is recommended that you use IPSec when you want to secure traffic over upper-layer
protocols or when you need to allow access only to a group of trusted computers. Note that it is easier to
configure ICF to permit traffic over a certain port than it is to configure an IPSec policy.
IPSec is not a full-featured host firewall. However, it does provide the ability to centrally manage policies that can
permit, block, or secure unicast IP traffic based on specific addresses, protocols, and ports. Some of the functions
found in standard firewalls that IPSec does not provide include stateful inspection, application protocol
awareness, intrusion detection, and packet logging. Although IPSec lacks some features of firewalls, the packet
blocking and filtering it provides can be effective in helping to limit the spread of viruses and thwart specific
attacks known to use specific ports. You can also use IPSec to prevent specific applications and services from
being used on the network.
Related Information
The following resources contain additional information that is relevant to this section.
Active Directory Collection
How IPSec Policy Extension Works
How TCP/IP Works
How VPN Works
How 802.11 Wireless Works
The following RFCs and Internet Drafts are relevant to IPSec. To find the RFCs, type the appropriate RFC number in the IETF RFC
Database. To find the Internet Drafts, type the appropriate keyword in the IETF Internet Drafts database.
RFC 2401: Security Architecture for the Internet Protocol
RFC 2402: IP Authentication Header
RFC 2406: IP Encapsulating Security Payload (ESP)
RFC 2408: Internet Security Association and Key Management Protocol (ISAKMP)
RFC 2409: The Internet Key Exchange (IKE)
UDP Encapsulation of IPSec Packets (draft-ietf-ipsec-udp-encaps-02.txt)
Negotiation of NAT-Traversal in the IKE (draft-ietf-ipsec-nat-t-ike-02.txt)
How IPSec Works
How IPSec Works
In this section
IPSec Architecture
IPSec Protocols
IPSec Processes and Interactions
Network Ports and Protocols Used by IPSec
Related Information
In the Microsoft Windows Server 2003 operating system, Internet Protocol security (IPSec) helps provide
defense-in-depth against network-based attacks from untrusted computers. IPSec provides protection from
attack in host-to-host, virtual private network (VPN), site-to-site (also known as gateway-to-gateway or routerto-router), and secure server environments. You can configure IPSec policies to meet the security requirements
of a computer, an organizational unit, a domain, a site, or a global organization.
IPSec uses packet filtering and cryptography. Cryptography provides user authentication, ensures data
confidentiality and integrity, and enforces trusted communication. The strong cryptographic-based authentication
and encryption support that IPSec provides is especially effective for securing traffic that must traverse untrusted
network paths, such as those on a large corporate intranet or the Internet. IPSec also is especially effective for
securing traffic that uses protocols and applications that do not provide sufficient security for communications.
To successfully deploy IPSec for Windows Server 2003, you must ensure the following:
If your scenario requires Active Directory-based IPSec policy (a collection of IPSec rules that determine
IPSec behavior), the Active Directory directory service and Group Policy must be configured correctly on
the corporate network, appropriate trusts must be defined, and appropriate permissions must be
applied. Although Group Policy applies to both users and computers, IPSec policy is a computer
configuration Group Policy setting that applies only to computers.
Each computer that will establish IPSec-secured communications must have an IPSec policy assigned.
This policy must be compatible with the IPSec policy that is assigned to other computers with which that
computer must communicate.
Authentication must be configured correctly and an appropriate authentication method must be
specified in the IPSec policy so that mutual authentication can occur between IPSec peers.
Routers, firewalls, or other filtering devices must be configured correctly to permit IPSec protocol traffic
on all parts of the corporate network, if IPSec negotiation messages and IPSec-secured traffic must
pass through these devices.
Computers must run operating systems that automatically support IPSec or must have appropriate
client software installed.
If computers are running different versions of the Microsoft Windows operating system (for example,
Windows Server 2003, the Microsoft Windows XP operating system, and the Microsoft Windows 2000
operating system), you must address the compatibility of the IPSec policies.
If clients must establish IPSec-secured connections with servers, those servers must be adequately
sized to support those connections. If necessary, you can use IPSec hardware offload network
The number of IPSec policies are kept to a minimum, and the IPSec policies are made as simple as
Systems administrators who will configure and support IPSec must be properly trained and must be
members of the appropriate administrative groups.
IPSec Architecture
Several Requests for Comments (RFCs) define the architecture and components of IPSec. These components and
their interrelationship comprise the logical architecture of IPSec. This section briefly describes the fundamental
components of the IPSec logical architecture and then explains how these components are implemented in
Windows Server 2003.
For comprehensive descriptions of IPSec architecture and components, see the RFCs that are listed in “Related
Information” later in this section.
Logical Architecture
The IPSec architecture can be categorized into four main areas:
Security Associations
SA and key management support
IPSec protocols
Algorithms and methods
Security Associations
Security Associations (SAs) are a combination of a mutually agreeable policy and keys that defines the security
services, mechanisms, and keys used to protect communications between IPSec peers. Each SA is a one-way or
simplex connection that provides security services to the traffic that it carries.
Because SAs are defined only for one-way communication, each IPSec session requires two SAs. For example, if
both IPSec protocols, Authentication Header (AH) and Encapsulating (ESP), are used for an IPSec session
between two peers, then four SAs would be required.
SAs for IPSec-secured communications require two databases: a security policy database (SPD) and security
association database (SAD). The SPD stores the security requirements or policy requisites for an SA to be
established. It is used during both inbound and outbound packet processing. IPSec checks inbound packets to
ensure that they have been secured according to policy. Outbound packets are secured according to policy.
The SAD contains the parameters of each active SA. The Internet Key Exchange (IKE) protocol automatically
populates the SAD. After an SA is established, the information for each SA is stored in the SAD. The following
figure shows the relationship between SAs, the SPD, and the SAD.
SA, SPD, and SAD Architecture
SA and Key Management
IPSec requires SA and key management support. The Internet Security Association and Key Management
Protocol (ISAKMP) defines the framework for authentication and key exchange by providing procedures for
negotiating, establishing, changing, and deleting SAs. It does not define the actual key exchange: it merely
provides the framework.
IPSec requires support for both manual and automatic management of SAs and keys. IKE is the default
automated key management protocol for IPSec. IKE is a hybrid protocol that incorporates parts of the Oakley key
exchange protocol and the SKEME keying techniques protocol. The following figure shows the relationship
between the ISAKMP, IKE, Oakley, and SKEME protocols.
ISAKMP, IKE, Oakley, and SKEME Protocol Architecture
The Oakley protocol uses the Diffie-Hellman key exchange or key agreement algorithm to create a unique,
shared, secret key, which is then used to generate keying material for authentication or encryption. For example,
such a shared secret key could be used by the DES encryption algorithm for the required keying material. A
Diffie-Hellman exchange can use one of a number of groups that define the length of the base prime numbers
(key size) which are created for use during the key exchange process. The longer the number, the greater the
key strength. Well-known groups include Groups 1, 2, and 14.
The following figure shows the relationship between the Oakley protocol, the Diffie-Hellman algorithm, and wellknown Diffie-Hellman key exchange groups.
Oakley Protocol, Diffie-Hellman Key Exchange Algorithm, and Well-Known Diffie-Hellman Groups
The Oakley protocol defines several modes for the key exchange process. These modes correspond to the two
negotiation phases defined in the ISAKMP protocol. For phase 1, the Oakley protocol defines two principle modes:
main and aggressive. IPSec for Windows does not implement aggressive mode. For phase 2, the Oakley protocol
defines a single mode, quick mode.
IPSec Protocols
To provide security for the IP layer, IPSec defines two protocols: Authentication Header (AH) and Encapsulating
Security Payload (ESP). These protocols provide security services for the SA. Each SA is identified by the Security
Parameters Index (SPI), IP destination address, and security protocol (AH or ESP) identifier.
The SPI is a unique, identifying value in an SA that is used to distinguish among multiple SAs on the receiving
computer. For example, IPSec communication between two computers requires two SAs on each computer. One
SA services inbound traffic and the other services outbound traffic. Because the addresses of the IPSec peers for
the two SAs are the same, the SPI is used to distinguish between the inbound and outbound SA. Because the
encryption keys differ for each SA, each SA must be uniquely identified.
The following figure shows the relationship between the SA, SPI, IP destination address, and security protocol.
IPSec Protocols and SA Architecture
Algorithms and Methods
The IPSec protocols use authentication, encryption, and key exchange algorithms. Two authentication or keyed
hash algorithms, HMAC-MD5 (Hash Message Authentication Code - MD5) and HMAC-SHA-1, are used with both
the AH and ESP protocols, The DES and 3DES encryption algorithms are used with ESP. The following figure
shows the relationship between the authentication and encryption algorithms and the AH and ESP protocols.
IPSec Protocols and Algorithms for Authentication and Encryption
The authentication methods for IPSec, as defined by the IKE protocol, are grouped into three categories: digital
signature, public-key, and pre-shared key. The following figure shows the relationship between the IKE protocol
and the authentication methods.
IKE Protocol and Authentication Methods Architecture
Windows Server 2003 IPSec Architecture and Components
The components and architecture of Windows Server 2003 IPSec are based on the IPSec RFCs. The basic IPSec
architecture for Windows Server 2003 has the following components: Active Directory, a Policy Agent, the IKE
protocol, an IPSec driver, and a TCP/IP driver.
The following figure illustrates how these components interact.
Windows Server 2003 IPSec Architecture
The following table describes each of these components.
IPSec Components
Windows Server 2003 Active Directory stores domain-wide IPSec policies for
computers that are members of the domain. Active Directory-based IPSec policies
are polled and retrieved by the Policy Agent.
Policy Agent
The Policy Agent retrieves IPSec policy from an Active Directory domain, a
configured set of local policies, or a local cache. The Policy Agent then distributes
authentication and security settings to the IKE component and the IP filters to the
IPSec driver.
IKE receives authentication and security settings from the Policy Agent and waits
for requests to negotiate IPSec SAs. When requested by the IPSec driver, IKE
negotiates both kinds of SAs (main mode and quick mode) with the appropriate
endpoint requested by the IPSec driver based on the policy settings obtained
from the Policy Agent. After negotiating an IPSec SA, IKE sends the SA settings to
the IPSec driver.
IPSec driver
The IPSec driver monitors and secures outbound unicast IP traffic and monitors,
decrypts, and validates inbound unicast IP traffic. After the IPSec driver receives
the filters from the Policy Agent, it determines which packets are permitted,
which are blocked, or which are secured. For secure traffic, the IPSec driver
either uses active SA settings to secure the traffic or requests that new SAs be
created. The IPSec driver is bound to the TCP/IP driver to provide IPSec
processing for IP packets that pass through the TCP/IP driver.
The TCP/IP driver is the Windows Server 2003 implementation of the TCP/IP
protocol. It is a kernel-mode component that is loaded from the tcpip.sys file
during startup.
The architecture of the Policy Agent, IKE protocol, and IPSec driver are described in more detail in the following
Policy Agent Architecture
The Policy Agent retrieves IPSec policy information, handles the internal interpretation and processing of the
policy, and sends it to the other IPSec components that require the information to perform security services. The
Policy Agent has the following components: policy store, Policy Agent service, local registry, local cache, and
Interface Manager.
The following figure shows the architecture of the Policy Agent.
Policy Agent Architecture
The following table briefly describes the Policy Agent components.
Policy Agent Components
Policy store
The IPSec policy store maintains both IPSec policy descriptions and interfaces to
applications and other tools that provide policy data management. The policy
store accesses IPSec policy data that is stored in either the local registry or in
Active Directory.
Policy Agent
The IPSec Policy Agent controls the retrieval and distribution of IPSec policy and
maintains the data about the configured policy for the IPSec driver and IKE.
Local registry
The local registry stores the locally configured IPSec policies, the local cache,
and other IPSec settings.
Local cache
The local cache stores IPSec policies after they are downloaded from an Active
Directory domain controller by the Policy Agent.
Interface Manager manages a list that contains items that correspond to each
physical and logical network adapter on the system.
The following sections provide additional detail about each of these components.
Policy store
The policy store organizes IPSec policy data and stores it in a format that the Policy Agent can use. In Windows
Server 2003, policy data can be stored in the following:
Active Directory
Local and remote registry
A file (for exporting and importing only)
In addition to providing an interface that UI services can use to store policy in each of these media, the policy
store does the following:
Provides policy data for default IPSec policies
Checks policy information for consistency
Retrieves policy version information
The policy store reads and writes policy information both to and from persistent storage and is aware of shared
policy-setting dependencies. This ensures that all policies using shared settings are marked as changed when
they are modified and that Windows Server 2003 IPSec components download the modified policies.
Policy Agent
The Policy Agent retrieves IPSec policy information and delivers it to other IPSec components that require this
information to perform security services, as shown in the following illustration.
Policy Agent Service Retrieving and Delivering IPSec Policy Information
The Policy Agent performs the following tasks:
Retrieves the appropriate IPSec policy (if one has been assigned) from Active Directory if the computer
is a domain member or from the local registry if the computer is not a member of a domain
Determines filter list order
Delivers the assigned IPSec policy information (IP filters) to the IPSec driver
Delivers both main mode and quick mode settings to IKE
Polls for changes in policy configuration. If the computer is a member of a domain, policy retrieval
occurs when the computer starts, at the interval specified in the IPSec policy, and at the default
Winlogon polling interval. You can also manually poll Active Directory for policy by using the gpupdate
/target:computer command.
If there are no IPSec policies in Active Directory or the registry, or if the IPSec Policy Agent cannot connect to
Active Directory, the IPSec Policy Agent waits for policy to be assigned or activated.
The Policy Agent appears in the list of computer services in the Services snap-in under the name IPSEC Services
and starts automatically as part of the initialization of the Local Security Authority (LSA) service.
Local registry
Each computer running Windows XP or a Windows Server 2003 has only one local Group Policy object, often
called the local computer policy. Using this local Group Policy object allows Group Policy settings to be stored on
individual computers regardless of whether they are members of an Active Directory domain.
The local registry maintains the IPSec policy configuration in the following registry key and its
If you assign local IPSec policies and you do not assign Active Directory-based IPSec policies, the local policies
are stored in the following registry
If you assign Active Directory-based IPSec policies, the policies are read from Active Directory and stored in the
local cache.
Local cache
The local cache is part of the registry. If you assign an IPSec policy in Active Directory, the policy is stored in and
read from Active Directory. A copy of the current policy in Active Directory is maintained in a cache in the local
registry at: HKEY_LOCAL_MACHINE\Software\Policies \Microsoft\Windows\IPSec\Policy\Cache.
If a computer to which an IPSec policy in Active Directory is assigned cannot connect to the domain, the cached
copy of the policy in Active Directory is applied. When the computer reconnects to the domain, new policy
information for that computer replaces old, cached information. You cannot configure or manage the cached copy
of an IPSec policy in Active Directory.
Interface Manager
Interface Manager maintains a list of physical and logical network adapters on the computer and notifies the
Policy Agent when interface and address changes occur. Interface Manager also maintains a complete list of
generic filters. Generic filters are filters that are configured to use My IP Address either as a source address or as
a destination address. Generic filters are saved in the appropriate IPSec policy storage location with either a
source address or a destination address of and a corresponding subnet mask of
IKE Module Architecture
The IKE module receives authentication and security settings from the Policy Agent and waits for requests to
negotiate SAs. When the IKE module receives a request to negotiate an SA from the IPSec driver, the IKE module
negotiates both kinds of SAs (the main mode SA and the quick mode SA) with the appropriate endpoint based on
the request of the IPSec driver that the policy settings obtained from the Policy Agent. After it has negotiated an
SA, the IKE module sends the SA settings to the IPSec driver.
The IKE module has the following components:
Diffie-Hellman Cryptographic Service Provider (CSP)
Certificate store
Security Support Provider Interface (SSPI)
Kerberos Security Support Provider (SSP)
The following figure shows the architecture of the IKE module.
IKE Module Architecture
A cryptographic service provider (CSP) is an independent software module that provides
implementations of cryptographic standards and algorithms. The CSP carries out the cryptographic
functions of CrytoAPI, creating keys, destroying them, and using them to perform a variety of
cryptographic operations.
The following table briefly describes the IKE module components.
IKE Module Components
CryptoAPI provides a set of functions that allows applications based on Windows to encrypt or
digitally sign data in a flexible manner while providing protection for the user's sensitive
private key data. Actual cryptographic operations are performed by independent modules
known as CSPs.
The IKE negotiation must be encrypted. This encryption is limited by what can be configured in
IPSec policy. The standard CryptoAPI functions for keyed hashing (using HMAC-MD5 and
HMAC-SHA1) and data encryption (using DES and 3DES) are used
The Diffie-Hellman CSP contains the implementation of the Diffie-Hellman key exchange and
determination algorithm. IKE uses only the Microsoft Base or Enhanced CSP for Diffie-Hellman.
However, the Diffie-Hellman calculation can be accelerated using the CryptoAPI exponentiation
offload interface (OffloadModExpo), as documented in the CryptoAPI SDK.
The RSA CSP contains the implementation of the Rivest-Shamir-Adleman (RSA) cryptographic
algorithms. When certificate authentication is selected, IKE checks the CryptoAPI default
provider to see if it is capable of performing RSA 512-bit digital signatures. If so, then IKE
uses this default CSP. If not, IKE enumerates the RSA providers, selects hardware-based
providers first and ensures that they can provide 512-bit signatures. IKE performs these
actions to open the certificate store.
The CSP that is used for signature operations during IKE negotiation is specified by the
certificate selected during the IKE negotiation; the certificate’s associated CSP for the private
key signature is used. As long as the RSA CSP supports the NOHASHID flag for
the CryptSignHash( ) API call, IKE can use that CSP for private key signing of IKE payloads.
Because the enumeration process does not permit IKE to know if the CSP supports the
NOHASHID option, it is possible to choose a certificate that appears valid, but whose CSP does
not allow IKE to construct the proper signature.
The certificate store is a permanent storage location where certificates, certificate revocation
lists, and certificate trust lists are stored. The certificate store is a physical store on the
Windows Server 2003 computer, but it is viewed logically as belonging to the user account of
the currently logged-on user, a service account, or the computer account. IKE can use only the
computer account, usually referred to as the computer store. You can view the computer store
by using the Certificates snap-in.
The SSPI enables network applications to access one of several security providers to establish
authenticated connections and exchange data securely over those connections.
The Kerberos SSP contains an implementation of the Kerberos security protocol. The Kerberos
SSP is an SSPI provider
IPSec Driver Architecture
The IPSec driver is a kernel-mode component that monitors and secures IP packets. In addition to the Policy
Agent and IKE, the IPSec driver uses the following components: the Security Association Database (SAD), the
Security Policy Database (SPD), the TCP/IP driver, TCP/IP applications, and the network interface.
The IPSec driver matches IP packet information with the IP filters that are configured in the active SPD. If traffic
must be secured, the IPSec driver either uses the appropriate SA to determine how to provide packet security or
requests that the IKE module negotiate SAs to be used to provide packet security. After the IPSec driver
determines which SA to use, it creates and validates encrypting, decrypting, and hashing to create or interpret
the AH and ESP headers on an IPSec-protected packet.
The following figure shows the IPSec driver architecture and how the driver interacts with other components in
Windows Server 2003.
IPSec Driver Architecture
The following table briefly describes the IPSec driver components.
IPSec Driver Components
The SAD is a database in the IPSec driver that contains the parameters
associated with each active SA. This database is populated automatically from
the IKE module.
The SPD is a database in the IPSec driver that specifies the filter lists and
associated settings that determine the status of all inbound or outbound IP
traffic. Inbound packets are checked to ensure that they have been secured
according to policy. Outbound packets are permitted, blocked, or secured,
according to policy. For secured traffic, the security policy that is used is the
negotiated SA, which is stored in the SAD.
TCP/IP driver
The TCP/IP driver is the Windows Server 2003 implementation of the TCP/IP
protocol. It is a kernel-mode component that is loaded from the Tcpip.sys file
during startup.
TCP/IP applications use TCP/IP and access TCP/IP network services through an
appropriate network API, such as Windows Sockets, NetBIOS, or Remote
Procedure Call (RPC).
The network interface is the logical or physical interface over which IP packets
are sent and received. The details of the Network Driver Interface Specification
(NDIS) interface, the network adapter driver, and the physical media over which
the IP packets are sent and received are beyond the scope of this subject.
Policy Data Structure
The data in a policy indicates the desired protection for the traffic between computers on a network. The data is
made up of various computer-related attributes (for example, IP address and port number), the communication
methods allowed (for example, algorithms and key lengths), and the IKE key negotiation and management. The
policy store updates and stores the policy data. The Policy Agent retrieves the stored policy data and makes it
available to all IPSec components.
As shown in the following figure, an IPSec policy contains several subsets of information.
IPSec Policy Structure
The following table describes the components of an IPSec policy.
IPSec Policy Components
Policy-wide parameters specify the polling interval used to detect changes in policy.
The policy-wide parameters are configured on the General tab in the properties of an IPSec
ISAKMP policy
The ISAKMP policy contains IKE parameters, such as encryption key lifetimes, and other
settings. The ISAKMP policy settings are configured in the Key Exchange Settings dialog
box, which is available from the Advanced button on the General tab in the properties of
an IPSec policy.
The ISAKMP policy also contains a list of security methods for protecting the identity of
IPSec peers during authentication. These methods are, listed in order of preference, and are
configured in the Key Exchange Security Methods dialog box, which is available from
the Methods button on the Key Exchange Settings dialog box.
IPSec rules
IPSec rules contain a statement that associates a filter list with a filter action, an
authentication method, an IPSec mode, and other settings. Typically, an IPSec rule is
configured for a specific purpose (for example, "Block all inbound traffic from the Internet to
TCP port 135"). You can define many IPSec rules in a single IPSec policy.
IPSec rules are configured on the Rules tab in the properties of an IPSec policy.
IPSec rules associate IKE negotiation parameters with one or more IP filters. The following table describes the
components of an IPSec rule.
Download PDF