Rebuilding the Server Part 2 - Poll!

Where We Left Off

Over the last several months I have been busy with full-time courses, part-time work, and very little hands on programming. Fortunately, I found some time to get back to diving into code and revising my web server.

As discussed in my last post, the original version of the web server was crude. Even after the improvements described in that post, the server spawned a child process to handle each request, wait for the child to complete, reap the child, and proceed to the next request. The server was still slow. When I last worked on the server, I was able to get up to around 2500 requests per second, however when I revisited the project I was only getting around 750 requests per second without any changes to the code!

I could not identify why there had been a nearly 75% decrease in performance without a code change. Originally I assumed there may have been an update to my development laptop that was causing trouble, but running the web server in virtual machines using different versions of Linux and FreeBSD resulted in no noticeable difference in performance. I can only assume that the earlier 2500 req/sec was a fluke.

Either way, I was not satisfied with the web server. There needed to be a better way.

A Quick Fix

The first way to improve performance was to revise the signal handler. At this stage in the design the server had a dedicated signal handler that would interrupt its normal execution and reap a child if it received a SIGCHLD interrupt. However, when handling multiple concurrent connections, such as during the hey tests, these signals would pile up and stall the execution of the server.

Addressing this was simple, just ignore the SIGCHLD signals from the child processes. I updated the signal handler to the following:

1struct sigaction sa;
2sa.sa_handler = SIG_IGN;
3sa.sa_flags = SA_NOCLDWAIT;
4sigaction(SIGCHLD, &sa, NULL);

This update told the server to handle a SIGCHLD signal by ignoring it. Assigning the sigaction sa_flags members to SA_NOCLDWAIT instructed the kernel to automatically reap the child process immediately upon exit and don’t bother sending its parent a SIGCHLD call at all. The results were impressive. Running the standard test we had previously used (hey -z 30s -c 8 -disable-keepalive http://127.0.0.1:8080/index.html), the throughput jumped from roughly 750 req/sec to ~20 000 req/sec!

No doubt that this was a great improvement, but nginx was still handling roughly five times as many requests per second. To improve performance further a redesign of the whole server was needed.

A Change to Event-Driven Architecture

A new design was needed for the server to handle multiple concurrent connections. The fork-per-request model was inefficient and needed to be scrapped in favour of an event-driven architecture. I came across a solution while reviewing blocking I/O in Beej’s Guide to Network Programming.

In my original design, each request is handled sequentially by spawning a new process.

The server would listen on the desiginated ports for incoming connections. When a client connected, accept() was called and the server forks a child process. That child blocked while reading the entire request from the client, parsed the request, generated a response, and then blocked again while writing the response back to the client if the socket’s send buffer was full. If the request was for a CGI program, the server forked yet another child process, introducing additional blocking I/O and inter-process communication between the web server and the CGI process.

While simple, the design did not scale well. Every request required at least one new process, and each process spent much of its lifetime waiting for I/O rather than performing useful work. As the number of concurrent clients increased, the process creation and context switching between them became a significant bottleneck.

The solution was to move to an event-driven architecture using poll(). Rather than dedicating an entire process to each client, a single process could monitor many sockets simultaneously and perform work only when a socket became ready for reading or writing. Not only would this eliminate the massive overhead of spawning multiple processes, but it would also eliminate most of the unnecessary waiting on blocking I/O, and would allow the server to handle many more concurrent connections with greater efficiency.

What is poll()?

The great Beej has a wonderful introduction to poll, but in short, the poll() system call is a POSIX function used for multiplexing I/O.

A file descriptor is registered by creating a struct pollfd entry. Each pollfd contains the file descriptor we want to monitor, along with the events we are interested in. For example, POLLIN indicates that we want to be notified when data is available to read, while POLLOUT indicates that we want to know when the descriptor is ready for writing.

An array of pollfd structures is passed to the poll() syscall, which blocks until one or more descriptors have an event read. When poll() returns, it provides the number of descriptors that have events ready. The revents field of each pollfd structure is updated with the actual event that occurred.

The server can then iterate through the array of pollfd structures and perform the appropriate action on each socket.

Beej provides a great example of using poll() with his simple poll chat server. In this example, clients can register with a server, send messages, and have those messages relayed to the other connected clients. This chat program was a great introduction to the poll() syscall, but it had an important limitation: it only tells the server that a socket has data

The event POLLIN guarantees that at least one byte is sitting in the socket’s receive buffer, not that the full HTTP request has been received. Similarly, POLLOUT lets us know that the socket send buffer has space available, not that the whole HTTP response will fit inside of the buffer. If the socket is still in its default blocking mode, a recv() or send() call can sit and wait until more bytes arrive to get the full request, or when writing out wait until there is more space in the buffer to write the entire response. In a single-process server, like my Simple Server, the wait doesn’t just stall the one client. It stalls all other client connections too while it waits!

For Beej’s simple chat server with short lines of text this isn’t a problem, but for an HTTP server it can cause problems. A slow client trickling in a request one byte at a time would cause a stall for all other clients. So too would an unresponsive client that stops reading its response mid-stream. The syscall poll() doesn’t protect against any of this. It only tells you that you can try a read/write operation, not that it will run to completion.

Using poll() With Non-Blocking Sockets

The fix to this problem is not abandoning poll, but changing the behaviour of the sockets that poll() monitors.

Every listening and accepted client socket is set to non-blocking mode via the fcntl() call and the flag O_NONBLOCK. On a non-blocking socket the calls recv() and send() never wait. If the operation can complete right now it does, even if it only gets a partial message and returns to the calling function. If the operation cannot complete it returns immediately, but this time with errno set to EAGAIN or EWOULDBLOCK instead of sitting around and waiting for more data or more buffer space respectively ¹.

poll() and O_NONBLOCK are two separate mechanisms that work best when used together. poll() will tell you when it’s worth attempting a read or write, meanwhile O_NONBLOCK guarantees that the attempt can never turn into an indefinite wait. Without O_NONBLOCK poll()’s “ready” is a false sense of safety as you can still block after reading or writing a single byte. Without poll(), non-blocking sockets would force the program to retry when receiving an EAGAIN response in a busy loop, wasting CPU cycles instead of sleeping efficiently.

To make use of syncronicity between poll() and the non blocking socket, the revised Simple Server main function runs a dedicated loop for each HTTP request. With each loop poll() reports that a socket is readable and then passes socket to the function do_read(). The job of this function is to effectively drain each socket, getting all of the currently available data received from the socket and then deciding what to do next. do_read() does this by calling recv() in a loop and appending whatever it receives to a buffer until one of two things happens. Either recv() receives a EAGAIN response, meaning the kernel has nothing more queued up in the socket’s buffer, and do_read() returns to main to begin reading the request from the next client, or do_read() continues reading until it has determined that the full HTTP request has been received and begins parsing the request.

The main loop has a similar process for writing a completed request to a client. Once it has been determined that the full HTTP request has been received and the response has been generated for the client, the server updated the pollfd for the client to notify when the socket is ready to write data with the event POLLOUT. If the socket is ready to be written to, the do_write() function is called and will begin to write from the response buffer to the socket. It essentially does what do_read() does in reverse, it runs a loop calling send() until it either sent the entire response, or until it receives an EAGAIN response, in which case it immediately returns to the main loop to continue once the socket becomes ready once more.

Implementing an Event-Driven Web Server

Unlike the first iteration of my server, which largely relied on if-statements, confusingly named and disorganised local variables scattered across the stack, and a fair amount of spaghetti code, this version was designed with maintainability in mind [2]. The source is organized around reusable data structures, expressive enumerations that model the state of each client connection, and a central event loop that orchestrates the lifecycle of every connection from acceptance to shutdown.

The first step in the server revision was to create a custom struct, Client, to handle the requests from the connecting clients.

 1typedef struct Client
 2{
 3	char *in_buf; 
 4	char *out_buf;
 5	
 6	FILE *file_ptr;
 7	off_t file_size;
 8	
 9	size_t input_length; 
10	size_t input_capacity;
11	size_t output_length; 
12	size_t output_capacity;
13	size_t output_sent;
14	
15	size_t header_len; 
16	size_t body_len;
17	size_t body_sent;
18
19	enum client_response
20	    resp_val; 
21	enum client_state state; 
22					
23	char timestamp[32]; 
24	char client_addr[INET6_ADDRSTRLEN]; 
25} Client;

The Client structure is the central data structure of the revised web server. A new instance is allocated for each incoming client connection and persists for the lifetime of that connection.

The struction contains two dynamically allocated buffers; an input buffer used to receive and store HTTP requests, and an output buffer used to store te HTTP response before it is transmitted to the client ². Associated length and capacity fields are track how much data is currently stored in each buffer and how much space has been allocated, while output_sent records how many bytes of the response have been written to the socket.

The members FILE *file_ptr* and off_t file_size reference the requested resource and record its size. Rather than repeatedly reopeing or rescanning the file while sending the response, the server retains this information within the Client structure for the duration of the request.

Several members exist primarily to support debugging and logging. header_len, body_len, and body_sent record the size and transimission progress of the HTTP response, while timestamp and client_addr store the time the request was processed and the client’s IP address for access logging.

Finally, the structure contains two enumerations values that drive the server’s behaviour. The resp_val member indicates which HTTP response should be generated, such as a successful 200 OK response or an error like 404 Not Found. More importantly, state records the client’s current position within the server’s event-driven state machine. Rather than record handling an entire request in a single blocking operation, the server advances each client through a sequence of states as it processes the request before closing the client connection. This state machine forms the foundation of the revised poll()-based architecture.

It is worth looking at the different states for the Client structure.

1enum client_state
2{
3	READING, /**< Accumulating inbound HTTP request bytes */
4	PROCESSING, /**< Full headers received; building the response */
5	SENDING_HEADER, /**< Writing HTTP response headers to the socket */
6	SENDING_BODY, /**< Writing response body (file data or CGI output) */
7	CLOSING, /**< Connection is done; pending removal from poll set */
8	NUM_CLIENT_STATE,
9};

The Client structure progresses through a series of well-defined states during the lifetime of a connection. Unlike the original fork-per-connection implementation, when a child process handled a entirely from start to finish, the revised server advances each client incrementally as the associated socket becomes ready for I/O. This finite-state machine allows a single process to manage many simultaneous client connections efficiently.

Each new client begins in the READING state. Here, the server accumulates bytes from the socket until it determines that the complete HTTP request has been received. Because this iteration of the server only implements HTTP/1.0 GET requests, this occurs once the terminating blank line (\r\n\r\n) marking the end of the request header has been read ³.

Once the request is complete, the client transitions to the PROCESSING state. During this stage, the server parses te HTT request, resolves the requested resource, determines the appropriate response code, constructs the HTTP response header, and prepares the response body.

The next two states indicate the transmission of the response. SENDING_HEADER writes the response header to the client, while SENDING_BODY streams the response body, whether that is a static file, a generated directory listing, or a CGI program output. Separating these operations into distinct states makes it straightforward to handle partial writes, allowing the server to resume sending where it left off if a socket temporarily cannot accept additional data.

Finally, the client enters the CLOSING state once the response has been fully transmitted. At this state the connection is removed from the poll() set, and any allocated resources are freed, and the socket is closed.

The current implementation supports only HTTP/1.0 and closes the connection after every response. A future revision to the server that support for HTTP/1.1 would extend the state machine by returning the client to the READING state after completing a response instead of immediately closing the connection.

Although five client states are defined, the main loop actively dispatches on only three of them: READING, SENDING_HEADER, and CLOSING. The other two, PROCESSING and SENDING_BODY, exist in the enum but are never observed by poll().

PROCESSING is intentionally transient. Once a completed request has been read, do_read() transitions the client into PROCESSING, parses the request, constructs response, and advances the client to SENDING_HEADER before returning to the event loop. As a result, no client is caught sitting in the PROCESSING state between poll() iterations because the transition happens and completes within a single function call.

PROCESSING is intentionally transient. Once a completed request has been read, do_read() transitions the client into PROCESSING, parses the request, constructs the response, and advances the client to SENDING_HEADER before returning to the event loop. As a result, no client is caught sitting in the PROCESSING state between poll() iterations, because the transition happens and completes within a single function call.

SENDING_BODY is unreachable for a different reason: the main loop’s dispatch check only tests for SENDING_HEADER, so SENDING_BODY is never checked, and no client’s state field is ever set to it. do_write() currently sends the entire response, both the header and body together, in one pass under SENDING_HEADER alone. SENDING_BODY is included in the state machine now because it reflects the future architecture rather than the present implementation. In a later phase of this project, when static files are transmitted using sendfile(2), the HTTP headers and response body will follow genuinely different code paths. The header still written from a buffer via send(), but the body handed directly from the file descriptor to the socket by the kernel. When implemented, SENDING_BODY will become a real dispatched state rather than a placeholder.

A Walk Through main()

Let’s discuss the main() function of the revised Simple Server as it’s where the magic happens. For the sake of brevity I will only focus on the aspects that related to handling HTTP requests from a client connection. Functions that are called to establish the running state of the server, such as the setFlags() function which parses the command line arguments, will not be the focus.

We start by allocating two dynamic arrays, one for Client structs and the second for struct pollfd structures. These two heap allocated arrays are kept in parallel.

 1
 2	int fd_size = INITIAL_SIZE;
 3	int fd_count = 0;
 4
 5	struct pollfd *pfds = malloc(sizeof(*pfds) * fd_size);
 6	Client *clients = malloc(sizeof(Client) * fd_size);
 7
 8	if (!pfds || !clients) {
 9		perror("malloc");
10		exit(EXIT_FAILURE);
11	}
12	

For every active client connection, pfds[i] contains the socket information monitored by poll() while clients[i] stores the per-connection state. The exception to this the first num_listener entries, which correspond to the listening sockets. Since these entries do not represent client connections, the corresponding Client structures are left unused.

 Index        0          1          2          3          4
           ┌────────┬────────┬────────┬────────┬────────┬─────
pfds       │ IPv4   │ IPv6   │ fd=11  │ fd=14  │ fd=17  │ ...
           ├────────┼────────┼────────┼────────┼────────┼─────
clients    │ unused │ unused │Client  │Client  │Client  │ ...
           └────────┴────────┴────────┴────────┴────────┴─────

The initial capacity of these arrays, tracked by fd_size, is set to 32 by a macro. If the number of active Client structures exceeds the current capacity, both arrays are reallocated so they continue to grow together preserving the one-to-one correspondence between pdfs[i] and clients[i].

Next, we create two dedicated file descriptors for listening on the desired port. One for the IPv4 and another for the IPv6 sockets on which the server will run. If both are successfully created, a struct pollfd will register each file descriptor and be configured to return when they are ready for a POLLIN event. Once the listeners have been registered, fd_count is initialized to the number of valid entries for the array. As additional clients connect, new entries are appended and fd_count grows accordingly..

 1	int num_listeners = 0;
 2	int listener_v4 = -1;
 3	int listener_v6 = -1;
 4
 5	int use_v4 = (app_flags & B4_FLAG) || !(app_flags & B6_FLAG);
 6	int use_v6 = (app_flags & B6_FLAG) || !(app_flags & B4_FLAG);
 7
 8	if (use_v4)
 9		listener_v4 = get_listener_v4();
10
11	if (use_v6)
12		listener_v6 = get_listener_v6();
13
14	if (-1 != listener_v4) {
15		pfds[num_listeners].fd = listener_v4;
16		pfds[num_listeners].events = POLLIN;
17		pfds[num_listeners].revents = 0;
18		num_listeners++;
19	}
20
21	if (-1 != listener_v6) {
22		pfds[num_listeners].fd = listener_v6;
23		pfds[num_listeners].events = POLLIN;
24		pfds[num_listeners].revents = 0;
25		num_listeners++;
26	}
27
28	if (num_listeners == 0) {
29		fprintf(stderr, "no listeners could be created; exiting\n");
30		exit(EXIT_FAILURE);
31	}
32
33	fd_count = num_listeners;

It is worth noting that when we create the sockets, they are set with the flag SO_REUSEADDR. Without this flag, restarting the server shortly after starting would often fail with EADDRINUSE as the previous server process’s socket lingers in a TIME_WAIT state for a few minutes after closing. This is a TCP mechanism to catch stray packets from the old connection before the port can be reused. SO_REUSEADDR instructs the kernel to permit a new socket bind to the port anyway, helpful when I have to frequently stop, rebuild and test the server during development ⁴.

 1/*
 2 * get_listener_v4.c
 3 */
 4
 5// ... code omitted from post
 6
 7struct sockaddr_in server_v4;
 8
 9if ((listener_v4 = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
10		perror("opening IPv4 sock stream");
11		exit(EXIT_FAILURE);
12	}
13
14	(void)setsockopt(listener_v4,
15	    SOL_SOCKET,
16	    SO_REUSEADDR,
17	    &yes,
18	    sizeof(yes));
19
20	int listener_flags = fcntl(listener_v4, F_GETFL, 0);
21	if (listener_flags < 0 ||
22	    fcntl(listener_v4, F_SETFL, listener_flags | O_NONBLOCK) < 0) {
23		perror("fcntl O_NONBLOCK listener_v4");
24		exit(EXIT_FAILURE);
25
26// ... code omitted from post

From there, we start the main loop.

 1	// Let's get this baby spinnin!
 2	while (running) {
 3		DBG("Entering first for loop\n");
 4		int poll_count = poll(pfds, fd_count, -1);
 5		if (-1 == poll_count) {
 6			if (errno == EINTR) {
 7				continue;
 8			}
 9			perror("poll");
10			exit(EXIT_FAILURE);
11		}
12
13		for (int i = 0; i < fd_count; i++) {
14		// Inner loop omitted from post
15		}
16
17		if (log_ptr) {
18			fflush(log_ptr);
19		}
20	}

The main loop repeatedly calls poll() on every file descriptor registered in the pfds array. This includes both the listening sockets and every active client connctoin. The call blocks until one or more file descriptors becomes ready for an event, such as a new incoming connection (ie. a POLLIN event on a listening socket), data arriving from a client (again a POLLIN), or a socket becoming ready to transmit more data to a client (POLLOUT).

When poll() returns, it populates the revents field of thestruct pollfd entry to indicate an which events have occurred. The inner for loop then walks the array, examinging each entry in turn and dispatching the appropriate action baded upon the reported events, and for for client connections the current state in the Client structure.

It is worth noting that every iteration of the event loop performs a linear scan of the pfds array, regardless of how many file descriptors are actually ready. Consequentally, the work performed by the event loop grows linearly with the number of registered file descriptors. This O(n) scanning is charactirstic of poll(), and is one of the reasons why interfaces such as epoll() on Linux and kqueue() on the BSD’s scale more effectively to a very large number of concurrent connections.

The inner loop iterates over every active entry in the pfds array. The first task is to distinguish between listening sockets and client connections

 1for (int i = 0; i < fd_count; i++) {
 2			if (i < num_listeners) {
 3				if (pfds[i].revents & POLLIN) {
 4					DBG("New connection found\n");
 5
 6					accept_new_conn(pfds[i].fd,
 7					    &pfds,
 8					    &clients,
 9					    &fd_count,
10					    &fd_size);
11				}
12				continue; // We don't want to treat a listener
13					  // as a client!
14			}
15// Client specific code omitted

The first num_listener entries in the array are permanently reserver for the listening sockets. This arrangement means the event loop never needs to ask whether a particular file descriptor is a listening socket or a Client struct. The role is entirely determined by its position inthe array. The first num_listener entries are always listeners, while every subsequent entry represents an active client connection.

Whenever one of the listening sockets reports a POLLIN event it indicates that a new TCP connection is waiting to be accepted. The server responds by calling accept_new_conn(), which accpets the new connection, allocated a new Client struct, and appends both the new Client struct and its corresponding struct pollfd entry to their respective arrays. If either has reached their current capacity, both are reallocated before the new connection is added. Once the listener has been processed the loop immediately executes a continue to prevent the listening sockets from being handled by the client-specific dispatch logic that follows.

 1for (int i = 0; i < fd_count; i++) {
 2    // listener socket specific code omitted 
 3
 4			short revents = pfds[i].revents;
 5			if (revents == 0) {
 6				continue;
 7			}
 8
 9			// Check if error and close if so
10			if (revents & (POLLERR | POLLHUP | POLLNVAL)) {
11				close_conn(i, &fd_count, pfds, clients);
12				i--; // close_con will replace current value of
13				     // i, decrement to get the new value
14				continue;
15			}
16
17			if ((revents & POLLIN) &&
18			    (READING == clients[i].state)) {
19				DBG("Incoming from client detected\n");
20				// handle read for new request
21				do_read(i, clients, pfds, magic);
22			}
23
24			if (revents & POLLOUT &&
25			    ((SENDING_HEADER == clients[i].state) ||
26				(SENDING_BODY == clients[i].state))) {
27				// handle writing
28
29				do_write(i, clients);
30			}
31
32			if (CLOSING == clients[i].state) {
33				DBG("Closing connection\n");
34				close_conn(i, &fd_count, pfds, clients);
35				i--;
36				continue;
37			}
38		}

The remaining entries in the loop are all active client connections. For these entiries, the servermakes it dispatch decision using two pieces of information: the events reported by the kernel via poll() in the revents field of the corresponding struct pollfd, an the current state of the stored in the associated Client structure.

If a client is in the READING state and poll() reports a POLLIN event, the socket has data available to be read without blocking. The server therefore calls do_read(), which accumulates the HTTP request, parses it once complete, and prepares the corresponding response.

Conversely, if a client is in either the SENDING_HEADER or SENDING_BODY state and the socket reports a POLLOUT event, teh socket is ready to accept the additional output. In this case, the server calls do_write() to continue transmitting the HTTP response to the client. Once the write is complete, the Client’s state is set to CLOSING.

Neither do_read() nor do_write() perform blocking I/O. If a read or write operation cannot proceed immediately, the underlying system call returns EAGAIN indicatnig that the operation would block. The function then returns control to the main event loop, allowing poll() to wait until the socket becomes ready again. This allows the server to make progress on other client connections rather than waiting on a single slow client.

Notice that neither POLLIN nor POLLOUT alone determine what action the server should take. A socket may be readable, but if the associated Client is waiting to send a response, the server ignores the read event. Likewise, a writable socket is only useful once a response has been prepared. The combined of kernel-reported events and application-maintained states is what allows a single event loop to coordinate hundreds or or thousands of client connections without blocking on any one of them.

Finally, if the Client has entered the CLOSING state, the server performs a connection clean up. The client’s allocated buffers are freed, the file descriptors closed, and the client’s entries are removed from both the clients and pfds arrays. The remaining entries are compacted so that the arrays continue to contain only active connections.

Performance Improvements

In the previous iteration of the server I was able to get around 2500 requests per second during the initial tests, and around 750 requests per second during subsequent tests when testing with hey -z 30s -c 8 -disable-keepalive http://127.0.0.1:8080/index.html.

The previous implementation of the server needed the -disable-keepalive flag due to the fact that the response headers did not provide the Content-Length of the response, nor specify that the connection was to close after receiving the response. Without knowing the response size, clients could not reiabily determine where the response ended, forcing each request to use a new TCP connection.

The previous implementation of the server never declared Content-Length in its response headers. Without it clients like hey, which attempt persistent connections by default, have no reliable way to detect where the response body ended or if should end. To side step this issue the -disable-keepalive flag was added when running benchmarks with hey. Other tools for measuring requests, such as wrk did not work at all as the server did not provide the proper information in the HTTP response header to properly test throughput. Fortunately, the revised version of the server provides both Content-Length and Connection: close in its headers, allowing the tools to correctly determine the response boundary. This means that the tool wrk can be used to test the number of responses per second, and hey can be run without the -disable-keepalive flag.

 1jholloway  …/server_revision  phase_1-poll via C v19.1.7-clang
 2 ❯ hey -z 30s -c 8  http://127.0.0.1:8080/index.html
 3
 4Summary:
 5  Total:        30.0002 secs
 6  Slowest:      0.0009 secs
 7  Fastest:      0.0001 secs
 8  Average:      0.0002 secs
 9  Requests/sec: 50613.9724
10
11  Total data:   170063936 bytes
12  Size/request: 170 bytes
13
14Response time histogram:
15  0.000 [1]     |
16  0.000 [406445]        |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
17  0.000 [558469]        |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
18  0.000 [30671] |■■
19  0.000 [3051]  |
20  0.000 [1019]  |
21  0.001 [276]   |
22  0.001 [45]    |
23  0.001 [14]    |
24  0.001 [2]     |
25  0.001 [7]     |
26
27
28Latency distribution:
29  10%% in 0.0001 secs
30  25%% in 0.0001 secs
31  50%% in 0.0002 secs
32  75%% in 0.0002 secs
33  90%% in 0.0002 secs
34  95%% in 0.0002 secs
35  99%% in 0.0003 secs
36
37Details (average, fastest, slowest):
38  DNS+dialup:   0.0001 secs, 0.0000 secs, 0.0008 secs
39  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0000 secs
40  req write:    0.0000 secs, 0.0000 secs, 0.0007 secs
41  resp wait:    0.0001 secs, 0.0000 secs, 0.0006 secs
42  resp read:    0.0000 secs, 0.0000 secs, 0.0005 secs
43
44Status code distribution:
45  [200] 1000000 responses

The response is a significant increase in throuput with over 50613 requests per second!

One of the primary motivations for replacing the fork-per-connection architecture with a poll()-based event loop was to improive scalablility under concurrent load. With the previous implementation, benchmarks were limited to only eight concurrent connections. The revised server comfortably handles several hundred simultaneous connections. For example, running hey on my laptop with -c 500 produces over 56 700 requests per second.

Increasing the concurrency further to 1000 simultaneous connections produces a noticeable increase in client-side errors. At this point it becomes difficult to determine whether the bottleneck lies in the benchmarking tool, the operating system, or the current server implementation. Regardless, hey becomes less suitable for stressing the server at this scale, so subsequent testing will use wrk, which is designed for generating much higher levels of concurrent load.

 1 ❯ wrk -t16 -c1000 -d30s  http://127.0.0.1:8080/index.html
 2Running 30s test @ http://127.0.0.1:8080/index.html
 3  16 threads and 1000 connections
 4  Thread Stats   Avg      Stdev     Max   +/- Stdev
 5    Latency    11.93ms   47.78ms   1.17s    94.48%
 6    Req/Sec     6.49k     1.66k   22.07k    75.53%
 7  3099515 requests in 30.10s, 848.35MB read
 8  Socket errors: connect 0, read 98, write 0, timeout 253
 9Requests/sec: 102976.87
10Transfer/sec:     28.19MB
11
12 ❯ wrk -t16 -c10000 -d30s  http://127.0.0.1:8080/index.html
13Running 30s test @ http://127.0.0.1:8080/index.html
14  16 threads and 10000 connections
15  Thread Stats   Avg      Stdev     Max   +/- Stdev
16    Latency    14.64ms   61.02ms   1.27s    94.49%
17    Req/Sec     6.01k     2.19k   44.81k    89.59%
18  2869406 requests in 30.10s, 785.37MB read
19  Socket errors: connect 0, read 705, write 0, timeout 2141
20Requests/sec:  95327.81
21Transfer/sec:     26.09MB
22
23 ❯ wrk -t16 -c20000 -d30s  http://127.0.0.1:8080/index.html
24Running 30s test @ http://127.0.0.1:8080/index.html
25  16 threads and 20000 connections
26  Thread Stats   Avg      Stdev     Max   +/- Stdev
27    Latency    19.62ms   96.37ms   2.00s    95.01%
28    Req/Sec     3.98k     3.38k   31.15k    53.95%
29  1878926 requests in 30.10s, 514.27MB read
30  Socket errors: connect 4642, read 174, write 18, timeout 3372
31Requests/sec:  62431.49
32Transfer/sec:     17.09MB

Using wrk, the server sustained approximately 100 000 requests per second while serving between 1000 and 10 000 concurrent connections. Increasing to 20 000 connections exposed te limits of the current implementation and test environment, with throughput falling to roughly 62 000 requests per second, with wrk reporting an increasing number of connection failures and timeouts.

Nginx Comparison

For fun, lets compare this iteration of Simple Server to one of the best production web servers out there, nginx. Is it fair to compare my home made application to a hardened production quality server with decades of optimization? Nope! But it’s fun!

First let’s see what response we get from the nginx server:

 1 ❯ curl -v http://127.0.0.1:80/index.html
 2*   Trying 127.0.0.1:80...
 3* Established connection to 127.0.0.1 (127.0.0.1 port 80) from 127.0.0.1 port 48243
 4* using HTTP/1.x
 5> GET /index.html HTTP/1.1
 6> Host: 127.0.0.1
 7> User-Agent: curl/8.21.0
 8> Accept: */*
 9>
10* Request completely sent off
11< HTTP/1.1 200 OK
12< Server: nginx/1.30.3
13< Date: Thu, 23 Jul 2026 02:18:26 GMT
14< Content-Type: text/html
15< Content-Length: 112
16< Last-Modified: Wed, 22 Jul 2026 05:11:09 GMT
17< Connection: keep-alive
18< ETag: "6a6050ed-70"
19< Accept-Ranges: bytes
20<
21<html>
22<body>
23<h1>Hello World!</h1>
24<p>This page has been served to you by the nginx server</p>
25</body>
26</html>
27* Connection #0 to host 127.0.0.1:80 left intact

I configured the index.html file served by nginx to match the size of the page returned by Simple Server. Even so, it is worth noting that the HTTP response headers differ considerably. In addition to providing substantially more metadata, nginx advertises two features that materially affect benchmarking: it speaks HTTP/1.1, whereas Simple Server currently implements a simplified subset of HTTP/1.0, and it includes the header Connection: keep-alive.

The latter is particularly important. Under HTTP/1.1, persistent connections are enabled by default, allowing multiple HTTP requests and responses to be exchanged over the same TCP connection. This avoids the overhead of repeatedly creating and tearing down connections for every request. By contrast, the current iteration of Simple Server closes the client connection after each response, requiring a new TCP connection for every subsequent request.

We can see the improvement in performance when using hey, as nginx will handle well over 150% the requests per second as Simple Server with the same number of concurrent connections:

 1 ❯ hey -z 30s -c 1000  http://127.0.0.1:80/index.html
 2Summary:
 3  Total:        30.0118 secs
 4  Slowest:      3.1116 secs
 5  Fastest:      0.0037 secs
 6  Average:      0.0299 secs
 7  Requests/sec: 82573.9709
 8
 9  Total data:   277537904 bytes
10  Size/request: 277 bytes
11
12Response time histogram:
13  0.004 [1]     |
14  0.315 [999945]        |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
15  0.625 [0]     |
16  0.936 [1]     |
17  1.247 [0]     |
18  1.558 [48]    |
19  1.868 [0]     |
20  2.179 [0]     |
21  2.490 [0]     |
22  2.801 [0]     |
23  3.112 [5]     |
24
25
26Latency distribution:
27  10%% in 0.0093 secs
28  25%% in 0.0097 secs
29  50%% in 0.0101 secs
30  75%% in 0.0106 secs
31  90%% in 0.0160 secs
32  95%% in 0.0249 secs
33  99%% in 0.0317 secs
34
35Details (average, fastest, slowest):
36  DNS+dialup:   0.0000 secs, 0.0000 secs, 0.0057 secs
37  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0000 secs
38  req write:    0.0000 secs, 0.0000 secs, 0.0041 secs
39  resp wait:    0.0272 secs, 0.0019 secs, 0.2521 secs
40  resp read:    0.0000 secs, 0.0000 secs, 0.0043 secs
41
42Status code distribution:
43  [200] 1000000 responses

Similarily we can test nginx using wrk:

 1 ❯ wrk -t16 -c1000 -d30s  http://127.0.0.1:80/index.html
 2Running 30s test @ http://127.0.0.1:80/index.html
 3  16 threads and 1000 connections
 4  Thread Stats   Avg      Stdev     Max   +/- Stdev
 5    Latency     9.88ms   39.32ms   1.36s    99.59%
 6    Req/Sec     6.97k     1.16k   28.99k    92.21%
 7  3332877 requests in 30.10s, 1.08GB read
 8  Socket errors: connect 0, read 271970, write 0, timeout 0
 9Requests/sec: 110729.71
10Transfer/sec:     36.85MB
11
12 ❯ wrk -t16 -c10000 -d30s  http://127.0.0.1:80/index.html
13Running 30s test @ http://127.0.0.1:80/index.html
14  16 threads and 10000 connections
15  Thread Stats   Avg      Stdev     Max   +/- Stdev
16    Latency    12.30ms   44.33ms   1.49s    98.57%
17    Req/Sec     6.98k     2.32k   39.89k    77.33%
18  3333062 requests in 30.01s, 1.08GB read
19  Socket errors: connect 0, read 517937, write 19, timeout 232
20Requests/sec: 111067.76
21Transfer/sec:     36.97MB
22
23 ❯ wrk -t16 -c20000 -d30s  http://127.0.0.1:80/index.html
24Running 30s test @ http://127.0.0.1:80/index.html
25  16 threads and 20000 connections
26  Thread Stats   Avg      Stdev     Max   +/- Stdev
27    Latency    11.79ms   36.47ms   1.52s    99.11%
28    Req/Sec     6.48k     1.37k   31.75k    82.89%
29  3096246 requests in 30.10s, 1.01GB read
30  Socket errors: connect 0, read 504877, write 5, timeout 81
31Requests/sec: 102876.76
32Transfer/sec:     34.24MB
33
34 ❯ wrk -t16 -c30000 -d30s  http://127.0.0.1:80/index.html
35Running 30s test @ http://127.0.0.1:80/index.html
36  16 threads and 30000 connections
37  Thread Stats   Avg      Stdev     Max   +/- Stdev
38    Latency    37.69ms  158.05ms   2.00s    94.60%
39    Req/Sec     3.74k     3.27k   34.87k    57.69%
40  1706053 requests in 30.10s, 567.83MB read
41  Socket errors: connect 5743, read 256456, write 1, timeout 10282
42Requests/sec:  56682.72
43Transfer/sec:     18.87MB

Nginx can handle a higher number of concurrent connections before encountering errors. It easily surpasses the 20 000 concurrent connections that simple server struggled to handle, and did not see a noticeable drop until testing with 30 000 connections.

However, this comparison is not quite fair. As mentioned when looking at the curl output for nginx, it is using HTTP/1.1 and keeping the connections alive. Simple Server is hard coded to close the connections, meaning each request will need to be done on a new TCP connection. We can bring back the “-disable-keepaliveflag forhey` to force the nginx connection to use a new TCP connection for each request.

 1 ❯ hey -z 30s -c 1000 -disable-keepalive http://127.0.0.1:80/index.html
 2Summary:
 3  Total:	35.7988 secs
 4  Slowest:	1.0168 secs
 5  Fastest:	0.0012 secs
 6  Average:	0.0102 secs
 7  Requests/sec:	43076.7563
 8  
 9  Total data:	172218144 bytes
10  Size/request:	172 bytes
11
12Response time histogram:
13  0.001 [1]	|
14  0.103 [998119]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
15  0.204 [0]	|
16  0.306 [952]	|
17  0.407 [0]	|
18  0.509 [0]	|
19  0.611 [0]	|
20  0.712 [0]	|
21  0.814 [0]	|
22  0.915 [0]	|
23  1.017 [928]	|
24
25
26Latency distribution:
27  10%% in 0.0039 secs
28  25%% in 0.0048 secs
29  50%% in 0.0055 secs
30  75%% in 0.0063 secs
31  90%% in 0.0070 secs
32  95%% in 0.0075 secs
33  99%% in 0.0089 secs
34
35Details (average, fastest, slowest):
36  DNS+dialup:	0.0014 secs, 0.0000 secs, 1.0051 secs
37  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0000 secs
38  req write:	0.0000 secs, 0.0000 secs, 0.0034 secs
39  resp wait:	0.0087 secs, 0.0001 secs, 0.2390 secs
40  resp read:	0.0000 secs, 0.0000 secs, 0.0046 secs
41
42Status code distribution:
43  [200]	1000000 responses

With the flag -disable-keepalive, hey reports nginx as responding with 43076.7563 requests per second. This is nearly half the number of requests compared to keeping connections open.

Unlike hey, wrk does not provide a command-line flag specifically named to disable keep-alive connections. Instead, request behaviour can be modified through its Lua scripting interface. To do so we create close.lua, which comprises the single line: wrk.headers["Connection"] = "close"

This script, when passed through to wrk via the command line, will add the Connection: close header to each HTTP request. The client is informing the server that the TCP connection should be closed after the response is completed, forcing subsequent requests to establish new connections.

 1 ❯ wrk -t16 -c1000 -d30s -s close.lua http://127.0.0.1:80/index.html
 2Running 30s test @ http://127.0.0.1:80/index.html
 3  16 threads and 1000 connections
 4  Thread Stats   Avg      Stdev     Max   +/- Stdev
 5    Latency     3.93ms   11.63ms   1.13s    99.60%
 6    Req/Sec     5.43k     1.55k   24.53k    74.19%
 7  2597654 requests in 30.10s, 852.20MB read
 8  Socket errors: connect 0, read 4116, write 84, timeout 59
 9Requests/sec:  86303.62
10Transfer/sec:     28.31MB
11
12 ❯ wrk -t16 -c10000 -d30s -s close.lua http://127.0.0.1:80/index.html
13Running 30s test @ http://127.0.0.1:80/index.html
14  16 threads and 10000 connections
15  Thread Stats   Avg      Stdev     Max   +/- Stdev
16    Latency    10.04ms   72.38ms   1.50s    98.88%
17    Req/Sec     5.56k     2.60k   62.84k    84.38%
18  2642243 requests in 30.02s, 866.82MB read
19  Socket errors: connect 0, read 4110, write 67, timeout 866
20Requests/sec:  88020.21
21Transfer/sec:     28.88MB
22
23 ❯ wrk -t16 -c20000 -d30s -s close.lua http://127.0.0.1:80/index.html
24Running 30s test @ http://127.0.0.1:80/index.html
25  16 threads and 20000 connections
26  Thread Stats   Avg      Stdev     Max   +/- Stdev
27    Latency     7.52ms   54.08ms   1.48s    99.17%
28    Req/Sec     5.67k     2.14k   32.25k    77.90%
29  2699747 requests in 30.02s, 0.86GB read
30  Socket errors: connect 0, read 4141, write 51, timeout 617
31Requests/sec:  89924.53
32Transfer/sec:     29.50MB
33
34 ❯ wrk -t16 -c30000 -d30s -s close.lua http://127.0.0.1:80/index.html
35Running 30s test @ http://127.0.0.1:80/index.html
36  16 threads and 30000 connections
37  Thread Stats   Avg      Stdev     Max   +/- Stdev
38    Latency    68.24ms  200.34ms   2.00s    93.16%
39    Req/Sec     5.47k     1.99k   57.93k    85.45%
40  2611290 requests in 30.10s, 856.67MB read
41  Socket errors: connect 0, read 4809, write 36, timeout 12853
42Requests/sec:  86763.66
43Transfer/sec:     28.46MB

As we can see, when nginx is forced to close each client connection after every response, throughput drops significantly compared to persistent connections. Instead of maintaining existing TCP connections, the server and client must repeatedly perform connection establishment and teardown, adding additional kernel work and network protocol overhead. With a keep-alive connection, nginx has a roughly 20-25% improvement under this workload.

Comparing Simple Server under these conditions provides a more meaningful baseline. Since the current iteration of Simple Server already closes connections after each response, the Connection: close nginx results more closely represent the workload Simple Server is designed to handle. It also gives a direction for future improvements in Simple Server.

Limitations of This Iteration

The new server still has some limitations. One of the original requirements of CS631 project was to handle CGI-BIN requests. While the server can handle them, it is extremely inefficient. The second is the limited request methods the server can handle.

Common Gateway Interface Limitations

Currently the server has two problems when handling CGI request.

The first is that it does not properly follow the protocol for the Commong Gateway Interface setout in RFC 3875. When I wrote my initial naive version of the server for my own education, I was unfamiliar CGI applications and the requirements for them. As I was working full time and building the server for my own desires, I instead wrote a very simplified implementation for CGI requests that would simply set whatever the query parameters were as Unix environment variables and fork a child process. Not only did this not follow the requirements setout in the respective RFC, but it was extremely unsafe. A malicious client could inject arbitrary values into the CGI process environment by manipulating HTTP query parameters, allowing request data to overwrite variables that should have been controlled by the server. This has been addressed in the most recent version of Simple Server, with the query string being set as a dedicated, specific environment variable. However, the server still lacks the proper implementation of the CGI protocol.

The second issue with CGI requests is that they are blocking. A client can make a request to a CGI file. The server will then fork a child process, read the CGI-binary’s output via a pipe, wait for the child to exit(), and then return the response. The problem is not that a child process is created; process creation is a reasonable isolation mechanism for CGI. The problem is that the parent process synchronously waits for the child process to complete. While waiting, the event loop cannot service unrelated client connections. This method is extremely inefficient and counters many of the improvements made by updating to the poll() based event-loop.

For example, I created helloWorld.cgi that does nothing but simply print “Hello World!” to stdout. Using hey to benchmark this resulted in a disappointing number of requests per second.

 1 ❯ ./simple_server -c ./cgi-bin
 2
 3 ❯ hey -z 30s -c 1000  http://127.0.0.1:8080/cgi-bin/helloWorld.cgi
 4
 5Summary:
 6  Total:	36.1100 secs
 7  Slowest:	3.1841 secs
 8  Fastest:	0.0162 secs
 9  Average:	1.3431 secs
10  Requests/sec:	512.8772
11  
12  Total data:	169000 bytes
13  Size/request:	13 bytes
14
15Response time histogram:
16  0.016 [1]	|
17  0.333 [74]	|■
18  0.650 [837]	|■■■■■■■■■
19  0.967 [1374]	|■■■■■■■■■■■■■■
20  1.283 [3808]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
21  1.600 [3129]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
22  1.917 [2223]	|■■■■■■■■■■■■■■■■■■■■■■■
23  2.234 [1456]	|■■■■■■■■■■■■■■■
24  2.551 [78]	|■
25  2.867 [11]	|
26  3.184 [9]	|
27
28
29Latency distribution:
30  10%% in 0.8139 secs
31  25%% in 1.0320 secs
32  50%% in 1.3195 secs
33  75%% in 1.6536 secs
34  90%% in 1.9348 secs
35  95%% in 2.0553 secs
36  99%% in 2.1986 secs
37
38Details (average, fastest, slowest):
39  DNS+dialup:	0.0104 secs, 0.0000 secs, 1.0046 secs
40  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0000 secs
41  req write:	0.0001 secs, 0.0000 secs, 0.0069 secs
42  resp wait:	1.3326 secs, 0.0155 secs, 2.2640 secs
43  resp read:	0.0000 secs, 0.0000 secs, 0.0020 secs
44
45Status code distribution:
46  [200]	13000 responses

The results with wrk were not much better…

 1 ❯ wrk -t16 -c1000 -d30s http://127.0.0.1:8080/cgi-bin/helloWorld.cgi
 2Running 30s test @ http://127.0.0.1:8080/cgi-bin/helloWorld.cgi
 3  16 threads and 1000 connections
 4  Thread Stats   Avg      Stdev     Max   +/- Stdev
 5    Latency     1.52s   299.99ms   2.00s    73.78%
 6    Req/Sec    25.75     28.97   200.00     87.89%
 7  8336 requests in 30.10s, 1.19MB read
 8  Socket errors: connect 0, read 2192, write 1322, timeout 1414
 9Requests/sec:    276.92
10Transfer/sec:     40.56KB

Compared with static file serving, CGI execution reduces throughput by approximately two orders of magnitude. This is expected because every request requires process creation, pipe communication, and synchronous waiting.

Fortunately, this limitation is not a fundamental problem with the event-driven architecture. The same approach used for client sockets can be extended to CGI pipes. The file descriptors connected to the child process can be placed into the poll() set and tracked through additional Client states, allowing the server to continue servicing network connections while waiting for CGI output.

A future implementation would likely introduce states such as CGI_RUNNING and READING_CGI_OUTPUT, where the event loop advances the CGI request only when the child process pipe becomes readable. This would preserve the non-blocking design goals of the server while still providing CGI support.

Limited Handling of HTTP Request Methods

At the moment, Simple Server can only handle GET requests and HEAD. It cannot currently handle request methods such as POST, PUT, or DELETE, which require additional request parsing and, in many cases, processing of a request body. This will be important as I work to update the server to properly implement CGI requests.

Currently the server detects when the complete HTTP request header has been received, but only parses the request line containing the method, URI, and HTTP version. The remaining header fields are discarded rather than interpreted. Furthermore, as the server only looks for the header terminator (\r\n\r\n), it does not even check if has received a request body.

Updating the Simple Server to properly process POST, PUT, and DELETE requests will require updating the parsing mechanisms of the HTTP requests. It will need to understand additional information sent by the client, such as the length of the requests and ensure it has received the entire request body.

Supporting request bodies will also require changes to the Client structure and state machin. A client connection may transition from receiving a request header to receiving an arbitrary amount of body data, potentially over multiple poll() events. The server cannot assume that an entire request arrives in a single read() call, so the body must be accumulated until the number of bytes specified by Content-Length has been received.

Conclusion

Hopefully, upgrading the server to HTTP/1.1 and implementing persistent connections will provide a similar improvement in throughput. The development of the poll()-based event loop and the Client state machine provides the server with the foundation required for this change. While additional work will be required to support multiple request/response cycles on a single connection, the current architecture is already structured around maintaining independent client states, which is the core requirement for an HTTP/1.1 server.

Similarly, expanding support for additional HTTP request methods and implementing a proper Common Gateway Interface implementation can be built upon this same architecture. Features such as request bodies, CGI processes, and persistent connections all require the server to maintain more detailed state about each client connection, which is exactly the problem the event-driven design was created to solve.

However, each phase of this project will need to be developed in a controlled and methodical manner. Before adding additional complexity, the server needs a stronger foundation of automated testing and portability. Future improvements will include expanding the Criterion test suite and integrating continuous integration testing across multiple Unix-like platforms, including FreeBSD, Linux, macOS, and illumos. Since the project is built around POSIX APIs and Unix networking primitives, maintaining compatibility across different implementations of Unix will be an important part of ensuring the server remains portable and reliable.

The goal of Simple Server has never been to create a replacement for mature production web servers. Instead, this project is an exploration of the systems programming concepts that make those servers possible: network protocols, event-driven architectures, process management, asynchronous I/O, and operating system interfaces. By developing each feature incrementally, the server can continue evolving from a simple educational project into a deeper exploration of how high-performance network software is designed..

The full source code for this phase of the project can be found on the GitHub page for Simple Server in the fork phase_1-poll