Quantdream

From Protocols to Production Systems

Abstract

In this third and final part, we move beyond protocol mechanics and into real-world implementation. Having examined how data travels across the network stack and how TCP and UDP behave under load, loss, and congestion, we now focus on how these principles translate into production-grade trading infrastructure.

This section connects transport-layer theory to actual system design decisions. We will analyze how a WebSocket client is implemented using Boost.Beast in C++, and how concepts such as non-blocking I/O, event loops, kernel buffers, and congestion control manifest in real code. We will examine architectural choices including threading models, lock-free queues, message dispatching, and backpressure management, and explain how these directly influence latency determinism and system stability.

Rather than treating networking as a black box hidden behind library abstractions, this part exposes the interaction between user-space code and the kernel networking stack. We will discuss how socket configuration, buffer sizing, TCP options, and scheduling decisions impact performance in low-latency trading environments. Special attention will be given to failure scenarios: reconnect logic, heartbeat handling, burst traffic, and recovery behavior under packet loss.

The goal of this final section is to bridge theory and engineering practice. By the end, the reader should not only understand how transport protocols work, but also how to design, implement, and reason about networking components in a high-performance trading system

Part 1: Networking basics
Part 2: TCP and UDP in trading systems
Part 3: WebSocket implementation

High-Level Architecture

At a high level, the design consists of:

  • A single persistent TLS-secured WebSocket connection
  • A dedicated receiver thread. It performs one task only, reading from the socket as fast as possible and pushing messages into an inbound queue. It does not parse, validate, or process business logic. Its responsibility is purely I/O.
  • A dedicated sender thread. It performs the inverse operation. It pulls preconstructed JSON messages from an outbound queue, applies rate limiting logic, and writes them to the WebSocket.
  • Single-producer single-consumer lock-free queues
  • A rate limiter controlling outbound flow

Business logic runs independently from both networking threads. It consumes inbound messages and produces outbound messages via queues. This decoupling prevents the most common failure mode in trading systems: allowing computational work to delay socket I/O.

If the receiver thread were responsible for both reading from the socket and processing messages, a temporary spike in CPU usage could cause the TCP receive buffer to fill. Once the kernel receive buffer fills, packet loss or connection instability can occur. By isolating I/O from processing, the system minimizes this risk.

WebSocketBeast

Why a Dedicated Transport Wrapper?

In my architecture, WebSocketBeast represents the lowest user-space boundary between my trading system and the operating system’s networking stack. It is the only component that understands DNS resolution, TCP sockets, TLS negotiation, and WebSocket framing. Everything above it interacts through a minimal interface: connect, send, read, and close. The internal type makes the protocol layering explicit:

websocket::stream<ssl::stream<net::ip::tcp::socket>> ws_;
C++

By encapsulating this layering inside one class, I prevent transport complexity from leaking into business logic. The sender and receiver threads do not need to know how TLS is configured or how the WebSocket handshake works. They only operate on strings.

Constructor

The constructor configures TLS for the websocket:

WebSocketBeast()
    : ctx_(ssl::context::tlsv12_client),    // SSL Context
      resolver_(net::make_strand(ioc_)),    // Resolver
      ws_(net::make_strand(ioc_), ctx_)     // WebSocket
{
    // Configuration
    ctx_.set_default_verify_paths();        // Load Certificates
    ctx_.set_verify_mode(ssl::verify_none); // OK for testnet, not suitable for production!
}
C++

The first element in the initializer list constructs the SSL context. This configures the object as a TLS client using TLS 1.2. The SSL context defines global properties of the encrypted session, including protocol version, certificate validation rules, and cipher negotiation. In other words, this object governs how encryption and authentication will behave during the TLS handshake.

The resolver is responsible for translating the hostname into one or more IP addresses before establishing the TCP connection. I bind it to a strand associated with the io_context. Even though this implementation is synchronous, using a strand ensures that handlers associated with this execution context will not run concurrently. It reflects a design choice that keeps the transport layer thread-safe and future-proof if later converted to asynchronous operation.

The WebSocket stream internally owns a TLS stream, which in turn owns a TCP socket. By passing the SSL context into the constructor, I bind the TLS configuration to this stream. Every read and write operation performed through ws_ will therefore travel through encryption and decryption layers automatically.

Inside the constructor body, I configure certificate handling:

ctx_.set_default_verify_paths();
ctx_.set_verify_mode(ssl::verify_none);
C++

set_default_verify_paths() loads the system’s trusted certificate authorities. In a production environment, this would allow the client to verify the exchange’s certificate during the TLS handshake. For testnet development, I disable certificate verification with verify_none. This simplifies setup and avoids issues related to missing or misconfigured trust stores. In production, this setting would be changed to enforce strict certificate validation.

Connection Lifecycle in Code

The connect() method is where the abstract transport wrapper turns into a live network session. This function performs four distinct steps:

  1. DNS resolution
  2. TCP connection establishment
  3. TLS handshake
  4. WebSocket upgrade handshake

Each step corresponds to a different layer of the stack.

void connect() {
    LOG_INFO("Starting Deribit WebSocket connection...");
    
    // 1. DNS Resolution
    auto const results = resolver_.resolve(DERIBIT_HOST, DERIBIT_PORT);
    
    // 2. TCP Connection
    auto ep = net::connect(ws_.next_layer().next_layer(), results);
    // Host port definition for SNI
    std::string host_port =
        std::string(DERIBIT_HOST) + ":" + std::to_string(ep.port());
    // SNI
    if (!SSL_set_tlsext_host_name(ws_.next_layer().native_handle(),
                                  DERIBIT_HOST)) {
        beast::error_code ec(
            static_cast<int>(::ERR_get_error()),
            net::error::get_ssl_category());
        throw beast::system_error{ec, "Failed to set SNI"};
    }
    
    // 3. TLS Handshake
    ws_.next_layer().handshake(ssl::stream_base::client);
    // Boost.Beast timeout settings for the client
    ws_.set_option(
        websocket::stream_base::timeout::suggested(
            beast::role_type::client));
    // 3-way handshake
    ws_.handshake(host_port, DERIBIT_PATH);
}
C++

At DNS resolution, the hostname test.deribit.com is translated into one or more IP addresses. This is a blocking operation. The resolver may consult the local DNS cache, or it may perform network queries to external DNS servers. From a performance perspective, this is an important step because DNS latency is not deterministic. In high-performance systems, this resolution is typically performed once and cached, or replaced entirely with a static IP to remove variability. In this implementation, the resolution cost is paid during connection establishment, not during message transmission.

// 2. TCP Connection
auto ep = net::connect(ws_.next_layer().next_layer(), results);
C++

Once the IP addresses are available, the code initiates a TCP connection. In order to reach the underlying TCP socket we have to deal with the nested nature of our websocket, as we defined it above. The expression ws_.next_layer().next_layer() accesses the raw net::ip::tcp::socket inside the TLS stream, which itself is wrapped by the WebSocket stream.
Calling net::connect() triggers the TCP three-way handshake at the kernel level. The operating system sends a SYN packet to the server, waits for a SYN-ACK response, and replies with an ACK. Only after this exchange completes is the connection considered established. During this time, the kernel allocates send and receive buffers, initializes sequence numbers, and sets up congestion control state. If packets are lost during this handshake, retransmissions are handled entirely by the TCP stack inside the kernel. From user space, the call simply blocks until success or failure. At this stage, the connection exists at the TCP level but is still unencrypted.

// Host port definition for SNI
std::string host_port =
    std::string(DERIBIT_HOST) + ":" + std::to_string(ep.port());
// SNI
if (!SSL_set_tlsext_host_name(ws_.next_layer().native_handle(),
                              DERIBIT_HOST)) {
    beast::error_code ec(
        static_cast<int>(::ERR_get_error()),
        net::error::get_ssl_category());
    throw beast::system_error{ec, "Failed to set SNI"};
}
C++

Before initiating the TLS handshake, I explicitly configure the Server Name Indication (SNI).
SNI is a TLS extension that informs the server which hostname the client intends to connect to. This is essential when multiple domains are hosted on the same IP address. Without SNI, the server may present the wrong certificate. By setting the hostname explicitly, I ensure that the server selects the correct certificate during the handshake.

// 3. TLS Handshake
ws_.next_layer().handshake(ssl::stream_base::client);
// Boost.Beast timeout settings for the client
ws_.set_option(
    websocket::stream_base::timeout::suggested(
        beast::role_type::client));
// 3-way handshake
ws_.handshake(host_port, DERIBIT_PATH);
C++

The TLS handshake is then performed. This operation upgrades the plain TCP connection into an encrypted session. During this handshake, several things occur: the client and server negotiate the cipher suite, exchange cryptographic parameters, perform key exchange, and derive shared symmetric session keys. The server presents its certificate, and depending on the verification settings, the client may validate it against trusted certificate authorities. This step introduces additional latency compared to raw TCP because it involves asymmetric cryptography and potentially multiple round trips. However, it is performed once per connection and amortized across all subsequent messages.

Although we are no longer in plain HTTP mode, WebSocket starts with an HTTP-based upgrade request. The client sends an HTTP GET request containing special headers indicating that it wishes to upgrade the connection to the WebSocket protocol. The server responds with a 101 Switching Protocols response if the upgrade is accepted. Once this exchange completes, the connection transitions from HTTP semantics to WebSocket framing semantics. From this point onward, the connection behaves as a persistent WebSocket session. Message boundaries are preserved at the WebSocket layer, while TCP continues to provide reliable, ordered byte delivery underneath. TLS continues to encrypt all data.

Blocking Send and Read Semantics

void send(const std::string& msg) {
    try {
        ws_.write(net::buffer(msg));
        LOG_DEBUG("WS Send: {}", msg);
    } catch (const std::exception& e) {
        LOG_ERROR("WS Send error: {}", e.what());
    }
}
C++

This is a synchronous, blocking write, in fact, when ws_.write() is called, several layers are involved:

  1. The WebSocket layer frames the message.
  2. The TLS layer encrypts the framed payload into one or more TLS records.
  3. The encrypted bytes are passed to the TCP layer.
  4. The kernel attempts to copy those bytes into the TCP send buffer.

If the kernel send buffer has space available, the call returns quickly. The data is now queued for transmission and will be handled by TCP’s congestion control and the NIC driver. If the send buffer is full, the call blocks. This can happen if:

  • The network is congested.
  • The congestion window has shrunk.
  • The receiver is slow to acknowledge packets.
  • The outbound rate temporarily exceeds the link capacity.

Because this implementation is blocking, the sender thread will pause until the kernel can accept more bytes. This design is intentional. The blocking behavior creates natural backpressure instead of allowing unbounded buffering in user space. Importantly, TCP does not preserve WebSocket message boundaries. From TCP’s perspective, it is just a stream of bytes. The WebSocket framing layer handles message structure, but once encrypted and handed to TCP, it becomes part of a continuous byte stream.

Blocking Read Semantics

std::string read() {
    if (shutting_down_.load(std::memory_order_acquire)) {
        return "";
    }

    beast::flat_buffer buffer;
    ws_.read(buffer);
    return beast::buffers_to_string(buffer.cdata());
}
C++

ws_.read() is also blocking. This call blocks until:

  • A complete WebSocket frame is received.
  • Or an error occurs.
  • Or the connection is closed.

Under the hood, the following occurs:

  1. TCP receives encrypted bytes into the kernel receive buffer.
  2. TLS decrypts records as they arrive.
  3. WebSocket deframes the decrypted payload.
  4. Once a full WebSocket message is assembled, control returns to user space.

If packet loss occurs and a TCP segment is missing, the read will block due to head-of-line blocking. Even if later segments arrive, TCP will not deliver them until the missing segment is retransmitted and received.

Shutdown and State Control

void close() {
    shutting_down_.store(true, std::memory_order_release);

    try {
        ws_.close(websocket::close_code::normal);
        LOG_INFO("WebSocket closed.");
    }
    catch (const std::exception& e) {
        LOG_DEBUG("WebSocket close during shutdown: {}", e.what());
    }
}
C++

Transport shutdown is one of the most delicate aspects of a concurrent networking system. It is easy to focus on connection establishment and message flow while overlooking how connections terminate. However, in trading systems, clean teardown is just as important as fast transmission. Improper shutdown can result in stalled threads, inconsistent session state, or abrupt TCP resets.

In my implementation, shutdown behavior is coordinated through an atomic flag:

std::atomic<bool> shutting_down_{false};
C++

This flag is shared across threads and serves as a signal that the connection is transitioning toward termination. Because the sender and receiver operate in separate threads, it is essential that state transitions are visible across cores in a predictable way. The atomic variable ensures that when one thread updates the shutdown state, the other threads observe it consistently. Before attempting a blocking read, the code checks this flag:

if (shutting_down_.load(std::memory_order_acquire)) {
    LOG_WARN("WebSocket is shutting down, aborting read.");
    return "";
}
C++

This check prevents a subtle race condition. Without it, the receiver thread could remain blocked inside ws_.read() while another thread attempts to close the connection. In such a situation, the underlying socket might be destroyed while still being accessed, leading to undefined behavior or exceptions at unpredictable points. By checking the shutdown state before entering the blocking read, the receiver thread can exit cleanly when termination is initiated. The close() method initiates the shutdown:

if (shutting_down_.load(std::memory_order_acquire)) {
    LOG_WARN("WebSocket is shutting down, aborting read.");
    return "";
}
C++

Setting the shutdown flag with release semantics ensures that any prior operations are visible before other threads observe the shutdown state. This ordering matters in concurrent systems. It guarantees that no stale state remains when the receiver thread detects the shutdown condition.

Calling ws_.close() performs a proper WebSocket close handshake. This is important because WebSocket is not just a raw TCP stream. It defines a protocol-level close frame that signals normal termination. When the close handshake is executed correctly, the remote peer understands that the session ended intentionally rather than due to a transport error. If instead the underlying TCP socket were destroyed abruptly, the kernel might emit a TCP reset. A reset does not indicate correct termination. It indicates an error condition. Exchanges may treat resets differently from clean closes, potentially impacting session recovery logic or rate-limiting behavior.

Another subtle point is exception handling during shutdown. It is common for blocking reads to throw exceptions once the socket is closed. In this implementation, such exceptions are caught and treated as expected if shutdown is already in progress. This prevents noisy error propagation during normal termination.

Request Sender

The outbound pipeline transforms logical trading requests into controlled, rate-limited, kernel-bound byte streams. It is intentionally simple in responsibility but critical in behavior. Its design reflects an understanding that network determinism is achieved not by complexity, but by strict separation of concerns and careful control of blocking boundaries.

std::thread worker;
std::atomic<bool> running{false};
C++

Once the transport layer is established, the next critical component is the outbound pipeline. In my architecture, sending messages to the exchange is not performed directly by business logic. Instead, outbound traffic is routed through a dedicated background worker: RequestSender. In a low-latency trading system, I/O must be isolated from computation. The sender thread exists solely to take preconstructed messages and push them to the network as predictably as possible.

When start() is called, a background loop begins executing:

worker = std::thread([this] {
    while (running.load()) {
        while (!rl.allow_request()) {
            std::this_thread::sleep_for(std::chrono::milliseconds(1));
        }

        auto req = queue.pop();
        if (!req) continue;

        std::string msg = std::move(*req);
        ws.send(msg);
    }
});
C++

This thread take strings from a queue and write them to the WebSocket. This separation is crucial. If outbound I/O were performed inside the strategy thread, a temporary CPU spike, memory allocation pause, or exception could delay socket writes. Because TCP is sensitive to burst behavior and congestion window dynamics, such delays can introduce latency jitter!By dedicating a thread to outbound traffic, I isolate network timing from computational variability.

Lock-Free Queue Boundary

The sender reads from a single-producer single-consumer queue:

SPSCQueue<std::string, 1024>& queue;
C++

This queue forms the boundary between business logic and the transport layer.
The SPSC design has important properties:

  • No mutex locking.
  • No kernel-level synchronization.
  • Predictable memory access patterns.
  • Minimal cache-line contention.

Because only one producer pushes into the queue and only the sender thread pops from it, the memory model remains simple and efficient. In high-frequency systems, eliminating locks in hot paths reduces tail latency and avoids priority inversion issues. If the queue is empty, the sender simply continues. If the queue were to fill, upstream components would naturally experience backpressure. This prevents unbounded memory growth.

Rate Limiting as Flow Control

while (!rl.allow_request()) {
    std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
C++

Exchanges impose strict rate limits on API requests. Violating these limits can result in throttling or temporary bans. Instead of allowing business logic to worry about compliance, rate limiting is enforced centrally in the outbound pipeline. The rate limiter effectively shapes traffic at the application layer. Even though TCP provides congestion control at the transport layer, that mechanism protects the network, not the exchange’s API policy. The token-bucket logic in RateLimiter ensures that request bursts remain within acceptable bounds. The sleep_for call introduces waiting. This is a deliberate tradeoff. It prioritizes simplicity and predictability over microsecond-level optimization. In a production HFT system, this could be replaced with a more precise timing mechanism or busy-waiting under CPU affinity control. However, in this case I wanted to use a simpler solution.

Authentication Injection

Before sending private RPC calls, the sender injects an access token:

if (msg.find("\"private/") != std::string::npos) {
    const std::string& token = auth->get_access_token();

    if (!token.empty()) {
        size_t pos = msg.rfind('}');
        if (pos != std::string::npos) {
            msg.insert(pos,
                R"(,"access_token":")" + token + "\"");
        }
    }
}
C++

This keeps authentication logic at the edge of the system rather than in strategy code. The business layer produces logical messages. The transport layer ensures they are compliant with exchange protocol requirements.

Interaction with TCP and Kernel Buffers

When ws.send(msg) is called, the message travels through several layers:

  1. WebSocket frames the text message.
  2. TLS encrypts it into records.
  3. TCP accepts encrypted bytes into its send buffer.
  4. The kernel schedules transmission based on congestion control.

If the kernel send buffer is full because the network is congested or acknowledgements are delayed, ws_.write() will block. In this design, only the sender thread blocks. Strategy logic remains unaffected. This blocking behavior acts as a natural backpressure mechanism. It prevents the system from building unbounded user-space buffers and aligns application throughput with actual network capacity. From a systems perspective, this is an important property. It ensures that the transport layer remains flow-controlled by real network conditions rather than by arbitrary application buffering.

Receiver

If the outbound pipeline controls how messages leave the system, the inbound pipeline determines how safely and efficiently data enters it. In a trading environment, inbound traffic can include order acknowledgements, fills, cancellations, heartbeats, and market data updates. The system must consume this stream continuously and without delay.
For this reason, I isolate inbound I/O inside a dedicated component: Receiver.

class Receiver {
    std::thread th;
    std::atomic<bool> running{false};
    WebSocketBeast& ws;
    SPSCQueue<std::string, 4096>& queue;
};
C++

Dedicated Receive Thread

When start() is called, a thread begins executing the run() loop:

void start() {
    running.store(true, std::memory_order_release);
    th = std::thread([this]() { run(); });
}

void run() const {
    while (running.load(std::memory_order_acquire)) {
        std::string msg = ws.read();
        if (msg.empty()) break;

        queue.push(std::move(msg));
    }
}
C++

The receiver does not parse JSON, It reads and forwards. Blocking reads interact directly with the kernel receive buffer. If the application delays reading from the socket, the kernel buffer fills. When the receive buffer fills completely, TCP advertises a zero window to the peer. The remote side must stop sending until buffer space is available again. In a trading system, this can create cascading latency spikes. Therefore, the receiver thread is designed to read as fast as possible and immediately offload processing to another component via the queue.

From Wire to User Space

When ws.read() is called, several layers are involved:

  1. TCP receives encrypted segments into the kernel receive buffer.
  2. The TLS layer decrypts complete TLS records.
  3. The WebSocket layer reassembles frames.
  4. Once a full WebSocket message is available, control returns to user space.

If packet loss occurs, TCP enforces in-order delivery. That means if a segment is missing, later segments are buffered but not delivered upward. This head-of-line blocking occurs below the WebSocket layer and is invisible to the application except as a delay. The receiver thread blocks inside ws.read() until a full message is available, as we explained in the previous articles. This blocking behavior is acceptable because it is isolated. No other part of the system depends on this thread for computation.

Lock-Free Inbound Queue

After reading a message, the receiver pushes it into a single-producer single-consumer queue:

queue.push(std::move(msg));
C++

The queue decouples transport from processing. The receiver thread produces messages. The strategy or dispatcher thread consumes them. Using an SPSC queue eliminates locks and reduces contention. The producer and consumer operate on separate ends of the buffer, minimizing cache-line bouncing and synchronization overhead.

If the inbound queue becomes full, the current implementation drops the message. Blocking inside the receiver would risk filling the kernel receive buffer and stalling the TCP flow. Dropping messages locally may be preferable to destabilizing the connection entirely, depending on application requirements. In a production-grade system, one might implement monitoring, backpressure signaling, or recovery logic. However, the key principle remains: transport must not be blocked by business logic.

Clean Termination

Shutdown is coordinated using the running flag:

void stop() {
    running.store(false, std::memory_order_release);
    ws.close();

    if (th.joinable())
        th.join();
}
C++

Calling ws.close() causes any blocking read() to terminate. The thread then exits its loop and joins cleanly.

Backpressure and Failure Modes

A networking system is not defined by how it behaves when everything is working. It is defined by how it behaves under stress. In trading infrastructure, stress can take many forms: burst traffic, exchange throttling, network congestion, packet loss, or remote disconnects. The design choices in the sender and receiver become most visible under these conditions.

Outbound Backpressure

The outbound path consists of three buffering layers:

Strategy → SPSC outbound queue → TCP send buffer

If the strategy produces messages faster than the sender can transmit them, the outbound queue begins to fill. Because the queue has a fixed capacity, it cannot grow indefinitely. This prevents unbounded memory usage and forces pressure to propagate upstream.

The rate limiter introduces an additional layer of control. Even if the network can transmit faster, the application intentionally limits its outbound rate to comply with exchange policies. This is a form of application-layer backpressure independent of transport-layer congestion control.

Inbound Backpressure

The inbound side has a similar but inverted structure:

Kernel receive buffer → Receiver thread → SPSC inbound queue → Strategy

If the strategy is slow to consume messages, the inbound queue begins to fill. If the queue reaches capacity and the receiver continues pushing, messages may be dropped. Blocking inside the receiver would be dangerous. If the receiver stops reading from the socket because it is waiting for queue space, the kernel receive buffer will fill. When that buffer is full, TCP advertises a zero window. The exchange is then forced to pause transmission. Under certain conditions, this can cause retransmission delays or connection instability.

By prioritizing socket reads over application processing, the system protects transport stability at the cost of potentially losing application-level messages. In practice, recovery mechanisms at higher layers can mitigate this risk.

Connection Loss and Transport Failures

Several failure modes are possible at the transport layer. If the TCP connection drops due to network failure, ws.read() or ws.write() will throw an exception. The receiver thread will exit its loop, and the sender thread may encounter write errors. The system must then trigger reconnection logic at a higher level. If the TLS handshake fails during connection establishment, connect() throws. This failure is deterministic and immediate. It does not affect other components because the connection lifecycle is encapsulated. If the exchange closes the WebSocket session gracefully, ws.read() returns an error corresponding to close frame reception. The receiver exits cleanly, and shutdown proceeds in a controlled manner.

A more subtle case is network stall. If packet loss occurs and retransmissions are delayed, TCP enforces head-of-line blocking. The receiver thread remains blocked inside read(). This is expected behavior. It reflects TCP’s ordered delivery guarantee. The application layer must tolerate temporary pauses.

Threading Model and Determinism

The threading model of this system is intentionally simple: one thread for outbound I/O, one thread for inbound I/O, and separate threads for business logic. This is not accidental. It reflects a deliberate tradeoff between architectural complexity and latency determinism.

In many modern networking systems built with Boost.Asio, it is common to use a fully asynchronous model driven by an event loop. Callbacks are registered for read and write readiness, and a single thread multiplexes many operations. While this model scales well for thousands of concurrent connections, it introduces additional complexity in reasoning about execution order and shared state. In this design, I chose blocking I/O inside dedicated threads.

This simplicity has several important consequences.

  1. Reasoning about timing becomes easier. The sender thread is either waiting for rate limit permission, waiting for queue data, or blocked on a kernel write. The receiver thread is either blocked on read() or pushing data to a queue. There are no interleaved callbacks, no hidden reentrancy, and no complex state machines in user space.
  2. Contention is minimized. The sender and receiver operate on separate queues with single-producer single-consumer semantics. There are no shared locks across the transport boundary. Memory ordering is explicit through atomic flags.
  3. Latency jitter is easier to diagnose. If outbound writes stall, the cause is either rate limiting or kernel backpressure. If inbound reads stall, the cause is either no incoming data or TCP-level delay.

The tradeoff is scalability. This model is ideal for a small number of persistent connections, such as one WebSocket session per exchange. It would not scale efficiently to thousands of simultaneous sockets. In high-frequency trading systems, however, the number of connections is typically small and long-lived. Determinism and clarity are more important than multiplexing efficiency. Another benefit of this model is compatibility with CPU affinity and isolation techniques. Because the sender and receiver are dedicated threads, they can be pinned to specific cores if needed. This reduces context switching and cache pollution, improving latency consistency. The cost of this approach is that each connection consumes dedicated threads. In large-scale distributed systems, this may be inefficient. In a trading engine interacting with a limited number of exchanges, it is acceptable and often preferable.

Ultimately, the threading model reflects a core principle: isolate I/O from computation, minimize shared mutable state, and keep execution paths predictable. Determinism in trading systems is achieved not by eliminating blocking, but by controlling where blocking occurs.

Conclusions

This project began as an exploration of networking fundamentals, but it evolved into something more practical and more important: understanding how protocol theory translates into real system design decisions.

In the first part, I examined how data travels across networks, from frames and packets to IP addressing and sockets. In the second part, I moved deeper into the transport layer, analyzing TCP and UDP, congestion control, retransmission, head-of-line blocking, and the behavior of algorithms such as CUBIC. That theoretical foundation was essential. Without it, transport behavior remains a black box. In this final part, I implemented those concepts in a real WebSocket client using Boost.Beast. The code is intentionally simple in structure but deliberate in design. The WebSocketBeast class isolates the entire transport stack. The RequestSender and Receiver enforce strict separation between I/O and business logic. Lock-free queues create clean boundaries. Blocking operations are confined to dedicated threads. Backpressure is allowed to propagate naturally rather than being hidden behind unbounded buffers.

What matters is not that the system can send and receive messages. What matters is that its behavior under load, under packet loss, under congestion, and during shutdown is predictable. Determinism comes from controlled blocking boundaries, bounded queues, and well-defined ownership of state.

This architecture reflects a core principle in trading infrastructure. Networking is a primary performance determinant. The latency of an order submission is not defined by how fast a string is constructed. It is defined by congestion window dynamics, kernel buffer availability, TLS record boundaries, and thread scheduling decisions.

By building the transport layer explicitly and reasoning about each step, I gain visibility into those mechanisms. I can trace a message from strategy construction to kernel transmission. I can understand where latency can accumulate. I can reason about failure modes and recovery. Ultimately, the goal of this system is not to be clever. It is to be controlled. In high-performance trading systems, control over timing, state transitions, and resource ownership is more valuable than abstraction alone.

This walkthrough of networking was an excellent opportunity to revisit foundational concepts and deepen my understanding, particularly in the second part, where I explored the transport layer from a more theoretical and systems-oriented perspective. As always, there is room for refinement and improvement. I welcome any feedback or suggestions that could strengthen the analysis or the implementation.

Leave a Reply

Your email address will not be published. Required fields are marked *