Virtual Private Networks, Part 3: Protocols

Last time we looked at how you can set up Windows NT and Windows 98 to participate in a virtual private network (VPN). The Windows NT approach to VPNs uses a protocol called PPTP (Point to Point Tunneling Protocol). Yet there are other protocols that can be used for VPNs, as you will see. It’s worth taking a few paragraphs to explain the protocols used by VPNs. While we won’t bother with the details you should know the acronyms and what each protocol has to offer when you consider a VPN.

You’re going to hear the term tunnel a lot when talking about VPNs. A tunnel is simply a connection between two machines over a network, such as the Internet, that can be maintained. Both ends of a VPN establish the tunnels to each other, and that tunnel exists until broken by one end or the other. A tunnel is a virtual network connection in a larger internetwork (similar to a telephone conversation over a much larger telephone network: the two ends are talking to each other without concern for all the other traffic going on over the telephone system).

There are four protocols used for VPNs: PPTP, IPSec, L2TP (Layer 2 Tunneling Protocol), and L2F (Layer 2 Forwarding). PPTP was developed by Ascend and Microsoft and is the most widely used when using Windows NT servers for VPNs. L2F was developed by Cisco. Both PPTP and L2F were supposed to be simple tunneling protocols and PPTP has caught on widely because of Microsoft’s support. Ascend, for its part, has pushed PPTP with its products and many ISPs using Ascend products have PPTP built into their systems. L2F has been kept proprietary by Cisco and is found only in their products, although some of the L2F features have been incorporated into L2TP.

Despite all the fuss about PPTP, IPSec is probably the best choice for a VPN protocol. IPSec is a specification that allows individual vendors to tailor the protocol the way they need, usually to support some feature specific to that vendor, although this could lead to interoperability problems. IPSec has very strong security, which makes it attractive for commercial, governmental and military markets. Finally, IPSec can be used over LANs, WANs, and many architectures, covering many different types of machines and operating systems.

The IPSec protocol is similar to TCP/IP and uses IP for routing. However, IPSec’s strength is its cryptographic systems used to ensure the data is unreadable by snoopers, that the two ends of a connection are authenticated, and that the packets of data are complete and data integrity is excellent. IPSec does this by combining a number of well-known techniques. (For those who care, public key cryptography is used for encryption with DES, Diffie-Hellman key exchanges are used for authentication, keyed hash algorithms are used to validate packets, and digital certificates are used to manage public keys.) IPSec is implemented by a large number of VPN vendors. Companies that offer IPSec-compatible products include 3COM, Cisco, Ascend, Nortel Networks, IBM, Lucent, Cabletron, and many more. (Microsoft is absent from the list: you need to buy add-on software to support IPSec on Windows platforms.)

Tunneling protocols work rather simply, once you understand the basics of packaging information for transmission across networks. Without going into the nitty-gritty let’s look at how PPTP works. PPP (Point to Point Protocol) has been around for many years and is used for connections to ISPs. Because security is a major issue with VPNs, PPP had to be modified to allow for better security both for transmitted data and for authentication of the machines involved in a VPN. PPP has two authentication systems called PAP (Password Authentication Protocol) and CHAP (Challenge Handshake Authentication Protocol). Both PAP and CHAP verify that the machines sending and receiving data are who they claim to be, and not some imposter intercepting the messages or masquerading as one of the valid machines. For PPTP, Microsoft developed a modification of CHAP called MS-CHAP.

Encryption of data is an important part of PPTP, so Microsoft developed an encryption method called MPPE (Microsoft Point to Point Encryption). When one end of a VPN wants to talk to the other end, PPP is used to construct the block of data to be sent. The block is then encrypted with MPPE. Next, the authentication algorithm MS-CHAP is used to make sure the two ends of the connection are valid. Then the data is sent over standard transmission and routing protocols, such as TCP/IP. The combination of PPP, encryption through MPPE, and authentication through MS-CHAP is all called PPTP.

Finally, L2TP is really a successor to PPTP as it is based on the PPTP specifications. L2TP combines the best of PPTP with the security of IPSec to form a hybrid product that is starting to gain wide support. L2TP is harder to install and configure, as well as manage, than either PPTP or IPSec, but it is starting to gain popularity because of its security and reliability. To date, only a few L2TP products are available, but the number is sure to grow as VPNs become more popular.