TCP/IP is at the heart of the Internet, and the Transport layer is at the heart of TCP/IP. The Transport layer is responsible for end-to-end connections between the sender and receiver over a TCP/IP network. The two most common Transport layer protocols are Transmission Control Protocol (TCP) and User Datagram Protocol (UDP). Nearly all Internet traffic uses either TCP or UDP.

TCP is connection-oriented, which means that a connection between the client and server is established before data can be sent. The TCP protocol provides reliable ordering and error-checked delivery. UDP is a connectionless protocol, which means it provides only minimal information and has no handshaking procedure. UDP does not offer a guarantee of ordering or delivery. Of course, the brevity of UDP makes it much faster than the steady and careful TCP, so applications that don't require a high level of reliability tend to use UDP.

TCP and UDP were created in more innocent days of an Internet, when networks did not face the security challenges we deal with today. The Transport Layer Security (TLS) protocol (and its predecessor SSL) were developed to provide encryption for communication security at the Transport layer. TLS offers privacy and data integrity between two communicating network nodes; however, it requires a reliable transport protocol, which means it won't work with the simple and unreliable UDP. TLS assumes that the packets arrive in the correct order, which TCP ensures but UDP does not.

Does this absence of ordering control mean that you can't use UDP for encrypted communication? Not exactly. UDP is intended as a foundation, upon which developers can add new protocols and services. The Datagram Transport Layer Security (DTLS) protocol [1] was developed to bring TLS-like privacy and encryption to UDP. The creators of DTLS wanted it to be as much like TLS as possible, adding only the necessary services needed to make TLS-style encryption work correctly.

This article introduces you to DTLS and offers some thoughts on how you can integrate DTLS into you own programs using OpenSSL or GnuTLS.

How It Works

DTLS reuses most of the protocol elements from TLS, with minor but crucial modifications for it to work properly with datagram protocols like UDP. To cope with the unreliability of connectionless protocols, DTLS had to implement a solution for packet loss and packet reordering.

DTLS messages are grouped into a series of message flights. Although each flight may consist of a number of messages, the flight should be viewed as monolithic for the purpose of time out and retransmission. Figure 1 shows the DTLS 1.2 handshake procedure.

Figure 1: The DTLS 1.2 handshake.

Because the DTLS handshake takes place over unreliable datagram transport, it is vulnerable to two types of Denial of Service (DoS) attacks. The first scenario is a standard resource-consumption attack. An adversary could start multiple handshake requests and cause the server to allocate system resources to burdensome cryptographic operations. This attack could slow or disrupt other network connections. The second attack is an amplification attack, in which an adversary sends a ClientHello message to the server and receives a large CertificateMessage response. To cause the damage, an adversary could employ IP spoofing to cause the DTLS server to send large CertificateMessage messages to the victim's IP address.

To protect against DoS attacks, DTLS adds a stateless cookie exchange to the handshake. The term cookie is universally used in computer science to describe a small packet of information that is sent and received without changes. The term "stateless" means that a cookie should be generated in such a way that it does not require keeping state on the server, which would require memory consumption.

To prevent DoS attacks, the DTLS server responds to the client's ClientHello message with a HelloVerifyRequest message, which contains a cookie. The client then has to repeat the ClientHello message with the cookie attached. The server verifies the cookie and proceeds with the handshake only if the cookie is valid.

The "Denial-of-Service Countermeasures" section of RFC 6347 [2] states the following:

"DTLS servers SHOULD perform a cookie exchange whenever a new handshake is being performed. If the server is being operated in an environment where amplification is not a problem, the server MAY be configured not to perform a cookie exchange. The default SHOULD be that the exchange is performed, however."

Note the keywords "SHOULD" and "MAY." The keyword "SHOULD" indicates items that can be omitted given valid reasons. The keyword "MAY" indicates features that can be arbitrarily omitted.

According to RFC 6347, the cookie exchange must be enabled by default on the server side, and a user has no duty to activate it. Because the cookie exchange is not a mandatory attribute of the handshake procedure, it can theoretically be turned off if the user really wants to remove it and understands all consequences of this decision. Network servers that are more sensitive to overall handshake latency can skip the HelloVerifyRequest message and instead respond with a ServerHello message, in which case the protocol behavior is the same as in the TLS protocol. Servers that choose to make this optimization can still be used as DoS amplifiers and should therefore not skip HelloVerifyRequest in environments where an amplification attack is a possibility.

RFC 6347 gives permission to disable the cookie exchange mechanism during the handshake procedure. The question is how this capability is realized in popular software libraries. The following sections look at implementing DTLS in OpenSSL (1.1.1i) and GnuTLS (3.6.15).

OpenSSL

OpenSSL [3] is a very popular library and the de facto standard open source TLS implementation. The architecture of OpenSSL consists of three parts: the context (CTX), the session (SSL), and the basic input/output subsystem (BIO). The context is responsible for the protocol (SSL/TLS/DTLS), the session cache, and other global parameters. Each new session is represented by the SSL object created from the existing context. The SSL object holds the session state and a BIO object. The BIO object is used to communicate with a network socket. The scheme of the DTLS server is defined as shown in Listing 1.

01 /* Functions to proccess cookies */
02 int gen_cookie(...){...};
03 int verify_cookie(...){...};
04 int main(void){
05 /* Library initialization */
06 SSL_load_error_strings();
07 SSL_library_init();
08 /* Create DTLS Context */
09 mtd = DTLSv1_server_method();
10 ctx = SSL_CTX_new(mtd);
11 /* Load SSL certificates */
12 /* Bind callbacks */
13 SSL_CTX_set_cookie_generate_cb(...);
14 SSL_CTX_set_cookie_verify_cb(...);
15 /* Create UDP socket */
16 fd = socket(...);
17 /* Process UDP packets */
18 for(;;){
19    /* Prepare new session */
20    BIO *bio = BIO_new_dgram(...);
21    SSL *ssl = SSL_new(ctx);
22    SSL_set_bio(ssl, bio, bio);
23    /* Waiting for ClientHello msg */
24    while(DTLSv1_listen(...) <= 0);
25    /* Encrypted data FROM client */
26    /* Process data */
27    /* Encrypted data TO client */
28    do_session(ssl, &client);
29    }
30 }

The function DTLSv1_listen() waits for incoming ClientHello messages, responds with a HelloVerifyRequest message, and returns  , which means that no client has been verified yet and it must be called again to continue listening. When the client sends a ClientHello message with a valid cookie attached, the function will return 1 and the sockaddr structure of the verified client. There are two callback functions to provide cookie exchange operations on the server side: gen_cookie() and verify_cookie().

When a cookie has to be generated for a HelloVerifyRequest message, the gen_cookie() function is called. After receiving an answer from the client with a cookie attached to a ClientHello message, the verify_cookie() function is used. Note that the programmer must implement these functions. I should emphasize that the implementation of gen_cookie() and verify_cookie() is a mandatory requirement within the existing architecture of the OpenSSL library. If the callbacks are not binded to the CTX by using SSL_CTX_set_cookie_generate_cb() and SSL_CTX_set_cookie_verify_cb(), or a NULL value is used instead of the function's address as a parameter of the binding function, the DTLS server will not respond to the client after receiving a ClientHello message.

By default the cookie exchange is enabled, and you do not have a way to change this part of the handshake procedure. There is an option called SSL_OP_COOKIE_EXCHANGE in the public API of the OpenSSL library that could be used to regulate the cookie exchange mechanism. But this option is used by the OpenSSL library itself in the body of the function DTLSv1_listen(). Therefore, this option is useless for the programmer. Within the current implementation of the library, the cookie exchange is always on.

GnuTLS

GnuTLS [4] is a C-based library that implements protocols ranging from SSL 3.0 to TLS 1.3, accompanied with the required means for authentication and public key infrastructure. The strong side of GnuTLS is a detailed documentation that is available online [5]. The documentation contains a brief introduction to the secure communication protocols and many examples of source code. GnuTLS provides essential means to write DTLS applications. One problem is that the library does not provide a function like DTLSv1_listen() from OpenSSL. You should construct the whole DTLS handshake procedure in your own program using functions provided by the library, which means that the text of the source code and the program structure of the application will be different depending on the programmer's decisions. If you want an RFC-compliant application, you must design and write the code as RFC-compliant.

Listing 2 shows the skeleton for a DTLS server in GnuTLS.

01 /* Data structures for cookies */
02 gnutls_dtls_prestate_st prestate;
03 gnutls_datum_t        cookie_key;
04 /* Init library */
05 gnutls_global_init();
06 /* Prepare key for cookie */
07 gnutls_key_generate(...);
08 /* Create socket */
09 fd = socket(...);
10 /* Wait for incoming udp packets */
11 for(;;){
12    /* Get udp payload */
13    ret = recvfrom(...);
14    if(ret > 0){
15    /* try to verify cookie */
16    ret = gnutls_dtls_cookie_verify();
17    if(ret < 0){
18       /* Send HelloVerifyRequest */
19       /* with attached cookie    */
20       gnutls_dtls_cookie_send(...);
21       continue;
22    }
23    }
24    else continue;
25    /* Prepare session */
26    /* ... */
27    gnutls_dtls_prestate_set(...);
28    /* Do DTLS handshake */
29    gnutls_handshake(...);
30    /* Do DTLS data exchange */
31    for(;;){
32       /* Encryptead data exchange */
33       do_session(...);
34    }
35 }

The GnuTLS approach is very similar to the OpenSSL scheme. The main difference is that the cookie exchange procedure is not a built-in part of gnutls_handshake() on the server side. From the documentation:

"Because datagram TLS can operate over connections where the client cannot be reliably verified, functionality in the form of cookies is available to prevent denial of service attacks to servers… If successful, the session should be initialized and associated with the cookie using gnutls_dtls_prestate_set before proceeding to the handshake. Note that the above apply to server side only and are not mandatory. Not using them, however, allows denial of service attacks. The client-side cookie handling is part of gnutls_handshake."

Unlike the OpenSSL library, GnuTLS allows you to forego the cookie exchange without any limitations (see the box entitled "GnuTLS Without Cookies,") but this option is best reserved for secure networks. If you plan to use DTLS on the Internet, the cookie exchange is an important protection against denial of service attacks.

GnuTLS Without Cookies

If you do not want to use the cookie exchange mechanism, you should remove all parts of the source code where cookies are used. In the files provided with this article [6], you will find a file named cookie.diff where you can see all steps to remove the cookie exchange from the source code of the DTLS server. You can obtain a version of the DTLS server without cookie exchange by executing the following command "[UCC:x20-kl-listing-bold]patch -b server.c cookie.diff[/UCC] " in your terminal program. When you get this version, you should compile it and then collect the network traffic using tcpdump. After loading the dump in Wireshark, you will see a screen view similar to one in Figure 2.

As you can see, the traffic dump does not contain a HelloVerifyRequest message with attached cookie. The DTLS connection was successfully established without the cookie exchange during the handshake procedure.

Figure 2: DTLS connection without the cookie exchange.