Virtual Private Network (VPN) is technology used to establish secure connection between two hosts or two networks, it extends the private network across a public network.
One of the most implemented VPN protocols is IPSec, which is standard protocol defined by IETF in many RFCs. IPSec operates at the Network layer of the OSI model, it uses the following protocols to provide security services:
Authentication Header (AH) – provides data origin authentication (Achieved by computing MAC for the entire datagram), anti-replay protection using sequence number in the AH header (already covered in the authenticated data), and integrity protection (tampering with data will impact the recalculated MAC).
Encapsulating Security Payload (ESP) – provides all AH services in addition to confidentiality using the encryption service.
These two protocols can be implemented individually or combined in one of two modes; tunnel mode or transport mode. Which mode to use depends on scope of the service; do we want to protect the entire packet or just the payload, and at what level we want to apply the encryption; at the outer level (Gateway-to-gateway) or at the inner level (host-to-host).
Policies are in place to determine the different IPSec options to choose; which protocol to use (AH, ESP), and what mode of operation (Tunnel vs. Transport). The security services we are looking for, plus additional security requirements (for example, the hashing algorithm to be used) guide construction of the policy.
Next we emphasis more on the two modes of operation.
As mentioned above, transport mode is constructed between end points, which mean that end points should be IPSec-aware. In other words, they should be able to understand the policy parameters, negotiate for the connection, establish the connection, and manage the connection through its life cycle. One advantage of this mode is that it extends scope of the services end-to-end, including internal network. However, this requires end points to perform additional tasks such as encryption/decryption which may impact the performance.
AH in transport mode places the AH header between IP header and the TCP header. The authentication service covers the entire datagram.
ESP in transport mode places ESP header between the original IP header and the TCP header. Also, it adds ESP trailer and ESP authentication trailer at end of datagram, this of course increase size of the datagram and cause additional processing at both VPN gateways.
Tunnel mode is provided between intermediate gateways, which are responsible for managing the VPN connection. Compared with transport mode, this will offload the end points from all operations related to the IPSec connection. However, it doesn’t provide end-to-end security services which mean the connection between the intermediate gateway and the end point will not be protected.
Additionally, tunnel mode provides protection for the entire datagram, including the original IP address. If the traffic is intercepted, the third party will not be able to figure out the original IP addresses if the ESP is used to provide confidentiality services.
AH in tunnel mode adds additional IP header to the datagram by the gateway device. It places the AH header between the original and new IP headers.
ESP protocol in tunnel mode also adds new header to the datagram by the gateway device, the encryption service covers the entire original datagram in addition to the ESP trailer, while the authentication service covers the complete datagram except the new IP header and the ESP auth header where the authentication data is located.
I is possible to use both of AH and ESP together. Since ESP protects the payload, AH can be used in conjunction with ESP to protect the entire datagram including the ESP headers. However, this adds processing overhead.