The Applications to Secure

93

Chapter 5. SSLTLS Programming

The main feature of the OpenSSL library is its implementations of the Secure Sockets Layer SSL and Transport Layer Security TLS protocols. Originally developed by Netscape for secure web transactions, the protocol has grown into a general solution for secure stream-based communications. Netscapes first public version of SSL is what we now call SSL Version 2. From that point, security experts began working to improve upon some of the flaws in SSLv2, and that gave birth to SSL Version 3. Development of a standard for transport layer security based on SSL was being done concurrently, which resulted in TLS Version 1. Because of the security flaws with SSLv2, modern applications should not support it. In this chapter, well discuss only programming with the SSLv3 and TLSv1 protocols in OpenSSL. Unless otherwise noted, when we refer to SSL, we refer to both SSLv3 and TLSv1. From a design perspective, we need to know more than just that we want to use SSL in our application. The correct implementation of an SSL-enabled program can be difficult due to complexities in protocol setup, the large size of the API, and developer inexperience with the library. OpenSSLs SSL support was originally designed to mimic the Unix socket interface; however, the likenesses quickly fade as we get into the subtleties of the API. In order to make the process of becoming acquainted with the massive library easier, we take a small example client and server through a step-by-step process of making it SSL-enabled and secure. To aid understanding, we start with some simplifying assumptions that may not be practical for real- world applications. From that point, well bridge the gap to the more advanced OpenSSL features. The goal of this chapter is to compartmentalize the features of the library into smaller groups to establish a sense of process. We hope that this process will serve as a template for developers when it comes to implementing SSL in their own applications. In adding SSL to an application, the applications unique requirements must be considered, and the best decision must be made for both security and functionality.

5.1 Programming with SSL

OpenSSLs API for SSL is large and can be daunting to inexperienced programmers. Additionally, as discussed in Chapter 1 , SSL can be ineffective at accomplishing its security goals if implemented incorrectly. These factors compound to leave the developer with a difficult task. In hopes of disentangling the mystery of implementing secure programs, we attack the problem in three steps. At each step, the developer must provide some application-specific knowledge to make sure SSL does its job. For example, the choices made by a developer of a highly compatible web browser will be different from those made by a developer of a highly secure server application. The steps below provide a template for developers to follow when implementing an SSL client or server. We will start with a small example and build upon it. This example will not be secure to our satisfaction until all of the steps have been thought through. In each step, we will introduce a small dose of the API; after all the steps, the developer should be able to think through the design of an SSL-enabled application much more clearly. Completing these steps is not the end of the road, however. In order to address the requirements of many applications, we need to go further and look into the advanced features of the API.

5.1.1 The Applications to Secure

94 We will be using two very simple applications: a client and server in which the server simply echoes data from the client to the console. Our goal is to augment our two applications so that they can perform their tasks in a hostile environment. In other words, well implement each program to strictly authenticate connecting peers. As we walk the path to creating SSL-enabled versions of our programs, we discuss the choices that a developer must make at each stage. Before moving forward, lets look at our sample applications. There a total of four files: common.h, common.c , client.c, and server.c. The code for each is presented in Example 5-1 through Example 5-4 . Also, we use the code presented in Example 4-2 so we can use multiple threads. For Unix systems, well continue to use POSIX threads. Starting with common.h in Example 5-1 , Lines 1-5 include relevant headers from OpenSSL. Right now, we dont make use of anything from a few of these headers, but we soon will, so they are included. Lines 22-24 define the strings for the client and server machines as well as the servers listening port. In addition, the header contains some definitions for convenient error handling and for threading in a platform-independent manner similar to the definitions set forth in Chapter 4 s threading examples. Example 5-1. common.h 1 include opensslbio.h 2 include opensslerr.h 3 include opensslrand.h 4 include opensslssl.h 5 include opensslx509v3.h 6 7 ifndef WIN32 8 include pthread.h 9 define THREAD_CC 10 define THREAD_TYPE pthread_t 11 define THREAD_CREATEtid, entry, arg pthread_createtid, NULL, \ 12 entry, arg 13 else 14 include windows.h 15 define THREAD_CC _ _cdecl 16 define THREAD_TYPE DWORD 17 define THREAD_CREATEtid, entry, arg do { _beginthreadentry, 0, arg;\ 18 tid = GetCurrentThreadId; \ 19 } while 0 20 endif 21 22 define PORT 6001 23 define SERVER splat.zork.org 24 define CLIENT shell.zork.org 25 26 define int_errormsg handle_error__FILE__, __LINE_ _, msg 27 void handle_errorconst char file, int lineno, const char msg; 28 29 void init_OpenSSLvoid; Example 5-2 , the file common.c, defines our error reporting function handle_error . The error handling in our example applications is a bit draconian, and youll most likely want to handle errors in your own applications in a much more user-friendly manner. In general, its not appropriate to handle all possible errors by bringing the application to such an abrupt halt. The file common.c also defines a function that will perform common initialization such as setting up OpenSSL for multithreading, initializing the library, and loading error strings. The call to SSL_load_error_strings loads the associated data for error codes so that when errors occur and we print the error stack, we get human-readable information about what went wrong. Since 95 loading these diagnostic strings does take memory, there are circumstances in which we would not want to make this call, such as when were developing applications for embedded systems or other machines with limited memory. Generally, its a good idea to load the strings since it makes the job of decoding error messages much easier. As we build in SSL support, the file common.c will hold the implementations of functions used by both our client and server, and common.h will receive the prototypes. Example 5-2. common.c 1 include common.h 2 3 void handle_errorconst char file, int lineno, const char msg 4 { 5 fprintfstderr, s:i s\n, file, lineno, msg; 6 ERR_print_errors_fpstderr; 7 exit-1; 8 } 9 10 void init_OpenSSLvoid 11 { 12 if THREAD_setup || SSL_library_init 13 { 14 fprintfstderr, OpenSSL initialization failed\n; 15 exit-1; 16 } 17 SSL_load_error_strings; 18 } The bulk of our client application is in Example 5-3 , client.c. At a high level, it creates a connection to the server on port 6001, as specified in common.h. Once a connection is established, it reads data from stdin until EOF is reached. As data is read and an internal buffer is filled, the data is sent over the connection to the server. Note that at this point, although were using OpenSSL for socket communications, we have not yet enabled the use of the SSL protocol. Lines 27-29 create a new BIO object with a BIO_METHOD returned from BIO_s_connect ; the call to BIO_new_connect is a simplified function to accomplish this task. As long as no error occurs, lines 31-32 do the work of actually making the TCP connection and checking for errors. When a connection is successfully established, do_client_loop is called, which continually reads data from stdin and writes that data out to the socket. If an error while writing occurs or an EOF is received while reading from the console, the function exits and the program terminates. Example 5-3. client.c 1 include common.h 2 3 void do_client_loopBIO conn 4 { 5 int err, nwritten; 6 char buf[80]; 7 8 for ;; 9 { 10 if fgetsbuf, sizeofbuf, stdin 11 break; 12 for nwritten = 0; nwritten sizeofbuf; nwritten += err 13 { 96 14 err = BIO_writeconn, buf + nwritten, strlenbuf - nwritten; 15 if err = 0 16 return; 17 } 18 } 19 } 20 21 int mainint argc, char argv[] 22 { 23 BIO conn; 24 25 init_OpenSSL; 26 27 conn = BIO_new_connectSERVER : PORT; 28 if conn 29 int_errorError creating connection BIO; 30 31 if BIO_do_connectconn = 0 32 int_errorError connecting to remote machine; 33 34 fprintfstderr, Connection opened\n; 35 do_client_loopconn; 36 fprintfstderr, Connection closed\n; 37 38 BIO_freeconn; 39 return 0; 40 } The server application in Example 5-4 , server.c, differs from the client program in a few ways. After making the call to our common initialization function line 44, it creates a different kind of BIO , one based on the BIO_METHOD returned from BIO_s_accept . This type of BIO creates a server socket that can accept remote connections. In lines 50-51, a call to BIO_do_accept binds the socket to port 6001; subsequent calls to BIO_do_accept will block and wait for a remote connection. The loop in lines 53-60 blocks until a connection is made. When a connection is made, a new thread to handle the new connection is spawned, which then calls do_server_loop with the connected sockets BIO . The function do_server_loop simply reads data from the socket and writes it back out to stdout . If any errors occur here, the function returns and the thread is terminated. As a note, we call ERR_remove_state on line 33 to make sure any memory used by the error queue for the thread is freed. Example 5-4. The server application 1 include common.h 2 3 void do_server_loopBIO conn 4 { 5 int err, nread; 6 char buf[80]; 7 8 do 9 { 10 for nread = 0; nread sizeofbuf; nread += err 11 { 12 err = BIO_readconn, buf + nread, sizeofbuf - nread; 13 if err = 0 14 break; 15 } 97 16 fwritebuf, 1, nread, stdout; 17 } 18 while err 0; 19 } 20 21 void THREAD_CC server_threadvoid arg 22 { 23 BIO client = BIO arg; 24 25 ifndef WIN32 26 pthread_detachpthread_self; 27 endif 28 fprintfstderr, Connection opened.\n; 29 do_server_loopclient; 30 fprintfstderr, Connection closed.\n; 31 32 BIO_freeclient; 33 ERR_remove_state0; 34 ifdef WIN32 35 _endthread; 36 endif 37 } 38 39 int mainint argc, char argv[] 40 { 41 BIO acc, client; 42 THREAD_TYPE tid; 43 44 init_OpenSSL; 45 46 acc = BIO_new_acceptPORT; 47 if acc 48 int_errorError creating server socket; 49 50 if BIO_do_acceptacc = 0 51 int_errorError binding server socket; 52 53 for ;; 54 { 55 if BIO_do_acceptacc = 0 56 int_errorError accepting connection; 57 58 client = BIO_popacc; 59 THREAD_CREATEtid, server_thread, client; 60 } 61 62 BIO_freeacc; 63 return 0; 64 } Now that we understand the sample application, were ready to take the steps necessary to secure the communications with SSL.

5.1.2 Step 1: SSL Version Selection and Certificate Preparation