erXos: An Operating System in ErlangChapter 17The Network Stack
Contents
  1. 1Operating Systems from the Ground Up
  2. 2Microkernels and Capabilities
  3. 3Erlang and OTP
  4. 4History and Goals
  5. 5Design Overview of erXos
  6. 6Correct by Construction
  7. 7One Source for Every Fact
  8. 8The Kernel Interface and Fiasco
  9. 9Processes, Runtimes and Two-Level Scheduling
  10. 10Domains: A Wall Around Some Processes
  11. 11Authority and Security
  12. 12Memory and Budgets
  13. 13Native Programs, Pipes and POSIX-lite
  14. 14Files, Namespaces and Service Trees
  15. 15Devices and Drivers
  16. 16Storage and Updates
  17. 17The Network Stack
    1. 17.1Why FreeBSD’s stack
    2. 17.2The construction
    3. 17.3The packet path
    4. 17.4The TCP core
    5. 17.5The fast path
    6. 17.6Sockets act for one domain
    7. 17.7Time from the network
    8. 17.8What fails, and what it reaches
    9. 17.9Design choices
    10. Summary
    11. Where to read next
  18. 18WiFi
  19. 19Bluetooth
  20. 20The Command Shell and the UNIX Surface
  21. 21The Live Machine
  22. 22Displays, Windows and the Web
  23. 23Apps and Packages
  24. 24Startup, Configuration and Shutdown
  25. 25Checking the System
  26. AGlossary
  27. BReading the Design Documents
  28. CBuilding and Booting erXos
  29. DDependencies and Licenses
  30. ELineage

Chapter 17The Network Stack

Background needed. Chapter 1 (drivers, DMA), chapter 3 (processes, supervisors, binaries, NIFs), chapter 6 (the functional core), chapter 10 (lanes, the funnel, exports, the envelope), chapter 11 (facets) and chapter 15 (drivers and the device NIF).

A TCP connection meets every kind of trouble a network can send. Segments arrive out of order, twice, or not at all. A stranger sprays forged resets in the hope that one lands inside the window. A flood of SYNs asks the machine to remember a million half-open connections. Each case has an answer that mature stacks took years to get right, and the answer is in their code, not in the RFCs.

erXos’s stack, erx_net, is modelled on FreeBSD’s for its protocol behaviour; its structure is its own. The stack is Erlang, in a domain of its own, so a bug in it is not a kernel bug. Every piece of state has one owner process, every decision is a pure function, every queue between processes is bounded by a credit, and every connection is a process that the inspector shows. The chapter follows the rules of construction, a packet’s path, the TCP core and the fast path, then who may use the network, network time and failure.

17.1Why FreeBSD’s stack

A TCP stack written from the RFCs alone fails in the field, so the stack takes its protocol behaviour from one implementation with thirty years of interoperation in its code, FreeBSD’s default TCP stack: BSD-licensed, and part of the UNIX erXos follows (chapter 4).

17.1.1What the stack matches

Three things fix the stack’s behaviour (design-network-stack.md, What the stack matches):

  • The wire. What the stack sends and how it answers follow its model; a deliberate departure is stated in the design document with its reason.
  • The Erlang API. Existing gen_tcp and gen_udp code runs unchanged.
  • erXos’s architecture. Every connection is a process of the network domain, per-packet work is Erlang and per-byte work is C, and the facet a client domain holds bounds what it can make the network domain spend (section 17.6.2).

17.1.2Why the default stack

The alternative TCP stacks, RACK and BBR, each replace the whole input and output path and together are about twice the default’s size, so they stay behind seams (design-network-stack.md, Why FreeBSD’s default stack). Most of a kernel stack’s volume is locking, buffer handling, tunables and logging, which an Erlang stack does not need; only the decisions carry over.

17.1.3What a naive Erlang stack gets wrong

An Erlang stack that starts from the RFC’s state diagram shows what goes wrong (design-network-stack.md, What a naive Erlang stack gets wrong). One fault needs background. A SYN flood sends opening SYNs from forged addresses, each asking the receiver to remember a half-open connection. The answer is a syncache, a small bounded table of half-open connections, and, when it is full, SYN cookies: the connection’s state is encoded in the SYN-ACK’s sequence number, so the receiver remembers nothing until the peer’s ACK returns it.

FaulterXos’s answer
one malformed frame crashes the unsupervised interface processdecoders that never raise; a failing step drops the frame and keeps the state
initial sequence number 0RFC 6528’s keyed hash
any RST inside the window resets the connectionRFC 5961’s checks
a process per SYN, never expireda bounded syncache in the interface process, with SYN cookies
window arithmetic that breaks at the 2^32 wrapsequence numbers as unbounded integers (section 17.2.4)
no congestion control, dropped out-of-order dataloss detection, CUBIC and a reassembly queue
a payload copied four times on transmit, checksum in Erlangone copy each way, which is designed to compute the checksum too (section 17.3.4)

Behind the table lie three habits: a timer per connection, mailboxes that grow without bound, and bytes parsed again at every layer. Section 17.2 removes each.

17.1.4Decisions, not structure

erXos ports by redesign: matching behaviour is the test, and the structure is new. Where the model offers a knob, the stack does what its default does.

The stack is also an ordinary OTP application. Four seams (the driver’s transport, the clock and entropy, the per-byte C primitives, and the runtime’s calls for facets) each have an erXos implementation (erlang/apps/erx_net/erxos/) and a portable one (erlang/apps/erx_net/portable/). So the whole stack runs on stock OTP on a development host, against real peers through a tap device, where most of its tests run (design-network-stack.md, Two places, four seams). This is Built.

17.2The construction

A kernel TCP stack spends much of its length on locks, timers that must be stopped safely, buffers that can grow, and sequence-number comparisons that survive wrapping. Four rules make each disappear (design-network-stack.md, The construction).

17.2.1One owner, one step

Every piece of protocol state belongs to exactly one process:

OwnerProcessState it owns
an interfaceerx_net_ifaddresses, neighbour cache, routes, IP reassembly, its share of the syncache, rate limiters
a TCP connectionerx_tcp_connthe control block, send and receive buffers, reassembly queue
a listening socketerx_tcp_listenthe connections queued for accept
a UDP socketerx_udp_sockits receive queue and options
a client domain’s net facet (section 17.6.2)erx_net_grantthe domain’s limits and account

Each owner is a loop around one pure function of its kind, the shape of chapter 6, section 6.3. In sketch, with names that vary by owner:

step(Event, State, Context) -> {State, [Action]}

Context carries what the decision needs from outside: the monotonic time, the credit the transmit path offers, the path’s MTU. The step reads no clock, no shared table and no other process, and returns the next state and actions from a closed set: transmit these frames, deliver these bytes, reply to this caller, end. The loop performs the actions and arms its one timer. Because the step never sends, an exception inside it leaves the state as it was before the event.

Because no state has two writers, the stack has no locks, no reference counts and no epochs. A connection ends by its process exiting. Other processes read only what an owner publishes in an ETS table, an in-memory table that one Erlang process owns and others may read, and every row has one writer.

Random values follow the same rule: each owner keeps a generator, a pure value of erx_net_rand built on SipHash, in its state, so a test that fixes the key gets the same stream, while an observer cannot predict it.

17.2.2Deadlines, not timers

A TCP connection has five timers: resending unacknowledged data (retransmission), probing a peer that advertised a zero window (persist), testing an idle connection (keepalive), delaying an ACK so it can ride with data, and the wait after close (2MSL). A naive Erlang stack starts an Erlang timer for each and cancels it when it is no longer wanted. Every cancel then races with a timer message already in the mailbox, and each handler must recognise a stale one.

Instead a connection’s state holds its deadlines, erx_tcp:next_deadline/1 returns the earliest, and after every step the process makes sure one hint is armed no later than that deadline. When a hint arrives, the step runs every deadline that is due. A timer message carries no meaning of its own, so there is nothing to cancel: restarting the retransmission timer on an ACK is a field update, and a hint that arrives early finds nothing due.

The rule that the retransmission and persist timers are never both armed is a type, so it cannot be broken. The field in erlang/apps/erx_net/include/erx_tcp.hrl, whose integers are deadlines:

send_timer = none :: none | {rexmt, integer()} | {persist, integer()},

17.2.3Credited queues

ERTS bounds no mailbox, and a stack under attack is a set of mailboxes that someone else fills. So no stack process hands packets to another except against a credit: a counter the receiver creates and publishes, which every sender must reserve from before it sends. erx_net_credit:reserve/2, in erlang/apps/erx_net/src/erx_net_credit.erl, is the whole sender’s side:

reserve({Ref, Limit}, Units) when is_integer(Units), Units > 0 ->
    case atomics:add_get(Ref, 1, Units) of
        InFlight when InFlight =< Limit ->
            ok;
        _ ->
            atomics:sub(Ref, 1, Units),
            full
    end.

The counter is an OTP atomics array, which any process of the node can add to without a message. A sender adds the units it is about to send; past the limit it takes them back and drops the packet, counting the drop. The receiver subtracts the units of each message it takes. Every queue between the driver, the interface process and the sockets is therefore bounded. A connection that falls behind fills its credit, the interface drops further segments for it, and the peer’s congestion control sees loss and slows down. Across a lane (chapter 10) the socket process gives the connection a credit in bytes by message, so what a program does not read waits in the network domain’s receive buffer, never in a mailbox.

17.2.4Packets become checked records once; sequence numbers are integers

A frame is decoded once, in the interface process, by a chain of total decoders: functions that return {ok, Record, Rest} or {error, Reason} for every input and never raise. Each checks its lengths against the layer that contains it, and its checksum first, so no step sees an unchecked length or a segment whose checksum was not checked. The only atoms a decoder returns are its own error reasons, a closed set, so no packet can grow the atom table.

Inside a connection, sequence and acknowledgement numbers, SACK edges and timestamps are unwrapped: unbounded integers that count from the initial sequence number and never wrap. A connection would have to move 2^59 bytes before they outgrew ERTS’s small integers, the ones that fit in one machine word. The conversion lives in one module, erlang/apps/erx_net/src/erx_tcp_seq.erl, in two functions (shown without their comments):

lift(Wire, Reference) when is_integer(Wire), Wire >= 0, Wire < ?MODULUS,
                           is_integer(Reference) ->
    Offset = (Wire - Reference + ?HALF) band (?MODULUS - 1),
    Reference - ?HALF + Offset.

wire(Integer) when is_integer(Integer) ->
    Integer band (?MODULUS - 1).

lift returns the integer congruent to the 32-bit wire value that lies nearest a reference (rcv_nxt for a received sequence number, snd_una for an acknowledgement), and wire returns the low 32 bits for encoding. After lifting, the protocol code uses ordinary < and +, and no wrap-safe comparison macros are needed.

The lifted segment is a different record from the decoded one, so a step cannot confuse the two. Lifting differs from modular comparison only for a number exactly 2^31 away, which modular comparison calls both greater and smaller; the stack drops such a segment.

Together the four rules remove per-connection locks and their order, reference counts and epochs, the races of stopping a timer, and buffer-chain surgery, which binaries and sub-binaries replace (design-network-stack.md, What disappears).

17.3The packet path

A received TCP segment crosses two local messages and one lane on its way from the device to the program that reads it. Its bytes are copied once inside the network domain, four times end to end. This section follows it.

17.3.1The process tree

Figure 17.1 shows the network domain’s processes (erlang/apps/erx_net/src/erx_net_sup.erl).

erx_net_sup: restarts nothingerx_net_tables: shared tableserx_net_grantsone erx_net_grant per client's net facetconnections, listeners, UDP socketserx_net_interfaceslink supervisor vtnet0driverinterface processDHCP client, when configuredmints of net and network_ctl
Figure 17.1.The network domain’s process tree. The mints make a facet process each time the system domain introduces a client. Sockets are linked to their facet, not supervised: a restarted TCP connection would be a different connection.

Every child of the top supervisor, erx_net_sup, is permanent, and the supervisor restarts none of them: the end of any ends the stack, and with it the network domain, which the domain service restarts with its device. Restarting the whole domain is simpler than a restart at the top: every client gets a fresh facet, and no code has to work out which facets a partial restart broke.

Ordinary failures stay lower down. Each interface, named by its driver (vtnet0 on QEMU, genet0 on the Pi 4), has a link supervisor; the driver’s end ends the link, which the interfaces supervisor starts again, losing only soft state such as neighbour entries (chapter 15, section 15.1.1). The facet process traps exits, so a socket’s crash ends that socket alone.

17.3.2A received segment, end to end

Figure 17.2 follows one TCP segment for an established connection whose program runs in client domain D (design-network-stack.md, Receiving).

frame in a DMA buffer, interruptbatch of frames (copy 1, out of the buffer)merge, classify, reserve credit, verifyitems for this connection (no copy)erx_tcp:step folds over the segmentspush, over the lane (copies 2 and 3, in and out){tcp, Socket, Data} (pieces joined, copy 4)devicedriverinterface processconnectionsocket process, domain Dprogram, domain D
Figure 17.2.A received segment from driver to socket. Inside the network domain the payload is copied once; the lane copies it in and out, and the socket process joins its pieces into the one binary gen_tcp hands over.

On the interrupt message, the device NIF copies completed frames out of their DMA buffers into fresh binaries, and the driver strips the device’s header and sends the batch to the interface process in one message, against the interface’s credit. The interface merges consecutive in-order segments of one flow (section 17.5), then classifies, reserves credit for and verifies each frame (section 17.3.3). It groups the batch by destination, so each connection gets one message holding its segments in order, as sub-binaries of the frames. The connection takes every message waiting in its mailbox, folds erx_tcp:step/3 over the segments, sends the acknowledgements to the driver in one message, and pushes the in-order data its socket process has credit for. The push, {erx_net, Conn, {data, Pieces}}, is a frame on the lane to D, which D’s funnel delivers in the origin envelope (chapter 10, section 10.5). The socket process checks it and presents it as the socket’s options say: in active mode, as one local {tcp, Socket, Data} message.

A sub-binary is a reference into part of a larger binary, made without copying, and it keeps the whole frame alive. So a held payload smaller than a quarter of its frame is copied out, and a peer cannot pin many frames with one-byte segments.

17.3.3Classify, reserve, then verify

The interface process works for every client of the domain, so each frame it drops must cost it little. classify/2, a pure function of erx_net_input, decodes the Ethernet and IP headers and finds the destination: a socket, the interface itself, or nowhere. The frame then takes one unit of its destination’s credit (for ARP or a ping, a token of the interface’s ingress bucket). Only a frame that holds its unit reaches verify/1, which runs the TCP, UDP or ICMP decoder, checksum first; a frame that fails gives its unit back and is dropped. So a frame dropped for want of room costs a header decode and one lookup. Checking the checksum after the reservation changes nothing on the wire, since a bad frame is dropped either way (design-network-stack.md, Receiving).

17.3.4Transmitting, and checksums in the copy

The send path mirrors the receive path. gen_tcp:send/2 calls the socket process, which sends the data over the lane in pieces of at most 64 KiB, one at a time, each after the connection’s reply to the one before; the caller is answered after the last reply (erlang/apps/erx_net_client/src/erx_gen_tcp_core.erl). The connection appends each piece to its send buffer and runs output. All frames of one step go to the driver in one message, each as a header binary and payload sub-binaries, and the device NIF lays each in a DMA buffer: the one per-byte copy of the transmit path.

The Internet checksum is a sum of 16-bit words, and a copy reads every word, so the copy can sum what it copies. On receive each layer derives its checksum from the frame’s sum by subtracting the headers it reads anyway. On transmit the TCP code leaves only the pseudo-header’s sum in the checksum field, as a stack does for a network card that offloads checksums, and the copy finishes it. The copy hands over a sum, never a verdict, because it parses nothing (design-network-stack.md, Checksums in the copy). The host’s tap transport does both, Built (erlang/apps/erx_net/portable/erx_tap.erl). In erXos’s device NIF the summing copy is designed and not built, and the decoders compute each sum with the runtime’s C checksum primitive.

17.4The TCP core

The whole of TCP, from the handshake to TIME-WAIT, is one pure function over one record, with the connection process a thin loop around it, so the core can be explored state by state on a host.

17.4.1The control block

The control block is one Erlang record, #tcb{}. Where a word of flags would allow combinations that are not legal, the record keeps a variant that can hold only the legal ones, the approach of chapter 6, section 6.3.4:

FieldValues
finnone, queued or {sent, Seq}
sackoff, requested or on
send_timernone, {rexmt, Deadline} or {persist, Deadline}, beside a record of the other deadlines
snd_una, snd_nxt, rcv_nxt and the restunwrapped integers (section 17.2.4)

A connection cannot be “FIN sent” without the FIN’s sequence number, because the record has no way to say so.

17.4.2Events, the step and actions

erx_tcp:step/3, in erlang/apps/erx_net/src/erx_tcp.erl, is the core’s one entry for events. Its input type lists everything that can happen to a connection:

-type event() :: {segments, [segment()]}
               | {user, request()}
               | {expired, Hint :: integer()}
               | {icmp, erx_tcp_icmp:report()}
               | {buffer, snd | rcv, Bytes :: non_neg_integer() | refused}.

-spec step(event(), #tcb{}, #tcp_ctx{}) -> {#tcb{}, [action()]}.

Nothing else enters. The step hands the event to the module for its kind (erx_tcp_input, erx_tcp_user, erx_tcp_timer) and then runs output. Its actions are a closed set: transmit, deliver, notify, reply, reserve a buffer, reset, and a few more. Only the connection process around the step touches the world; the step is a function a test can call.

The step also limits what an attacker can learn. A RST that answers a segment is sent only within the interface’s RST rate limit, and a challenge ACK to a SYN or an ACK is charged against that limit. A challenge ACK to a RST charges nothing: a charge there would let a blind attacker learn that its forged RST landed inside a window, the class of side channel behind CVE-2016-5696, a shared challenge-ACK limit (design-network-stack.md, Events and the step).

17.4.3Output, the send log and reassembly

The sender’s record of what it has sent is one structure, erx_tcp_sendlog: an ordered tree, keyed by the integer sequence numbers, from each sent range to its segment size, last transmission time, transmission count, and whether it is SACKed, lost or retransmitted. The loss modules read and mark the log, and output takes the next range to send from it. A SACK block marks only whole segments the sender cut, so a peer cannot split the log finer than what was sent or grow it.

On the receiving side, erx_tcp_reass queues out-of-order segments, trimming overlaps, up to a cap of 100 segments, and erx_tcp_sack builds the SACK blocks from that queue.

17.4.4Loss detection and congestion control

Two Erlang behaviours, OTP’s form of an interface that a module implements by exporting named callbacks, make loss detection and congestion control replaceable. erx_tcp_loss‘s callbacks handle an acknowledgement, a rewind when the retransmission timer expires, and the module’s own deadline. erx_tcp_cc’s modules are erx_tcp_cc_cubic, the default, and erx_tcp_cc_newreno, and a socket names one by a string looked up in a closed table, so the name never becomes an atom. A loss module never calls the congestion module: it returns events, and the connection makes the calls in a fixed order, so either side can be replaced alone. All of this is Built.

17.4.5How the core is checked

A pure core can be explored rather than sampled. test/erlang/erx_tcp_explore.erl drives a connection with small buffers and windows, and a model of its peer, over an abstract alphabet: segments with every combination of SYN, ACK, FIN and RST, with or without data, at each class of sequence position (below the window, at rcv_nxt, inside, at the edge, beyond), user requests and timer expiries. It applies every event in every state, breadth first, merging states that agree on an abstraction of the control block, until no new state appears or a depth limit, 8 in the full run. A second pass starts just below 2^32, so every class is also crossed at the wrap. On every transition it checks properties such as these (design-network-stack.md, The TCP core: exhaustive exploration):

  • snd_una =< snd_nxt =< snd_max, and rcv_nxt never decreases;
  • no byte is delivered twice, out of order, or beyond what the peer sent;
  • no segment is sent beyond the peer’s window, except a one-byte window probe;
  • every state change is one RFC 9293’s state diagram allows;
  • where header prediction, the shortcut for the common cases, accepts a segment, the general path gives the same result.

The explorer is Built, and runs in the host tests. The packet decoders have fuzz targets.

17.5The fast path

A stack that decides every segment in a pure step pays for that step on every segment. The design measured how much, found the cost spread too thin to optimise away, and changed the unit of work instead (design-network-stack.md, The fast path).

17.5.1What was measured

The stack ran on a development host, natively, on stock OTP with the JIT and its C primitives as a NIF, over a tap device to a peer running FreeBSD’s stack (an AMD Ryzen Threadripper PRO 5995WX under FreeBSD 16.0-CURRENT, 2026-11-09). Each transfer moved 512 MiB in 64 KiB sends, median of three. The baseline was lwIP 2.2.1, a C stack, one thread, -O2, over the same tap.

erx_netlwIP
send890 Mbit/s6,537 Mbit/s
receive1,143 Mbit/s6,809 Mbit/s
cost of one 1,448-byte segment12.3 us sending, 10.1 us receiving, of the connection’s wall time1.5 to 1.6 us of CPU for the whole path

The connection process is the serial stage and its cost is per segment: a busy wait of 3 us added to each segment cut sending to 711 Mbit/s (the mean of two runs), where one serial stage predicts 715. The cost has no hot spot, about 330 calls per segment and no function above 4 percent, and nothing was batched.

17.5.2Why the cost is architectural

Removing every measured share from the per-segment step would bring it down perhaps twofold, not the sixfold the gap needs. C stacks pay those costs once per batch or never: they hand a capable network card one large segment to cut and take merged segments from large receive offload (LRO). So a step takes and emits many segments at once, and the work left for one segment is made lean. Its bulk target is two thirds of lwIP’s throughput each way.

17.5.3Runs and merges

A run is one TCP header, a contiguous range of payload, and the segment size it is cut at: consecutive segments of one connection that differ only in sequence number, lengths and checksums. Output emits runs, and the copy the medium forces anyway cuts a run into frames, byte for byte what per-segment output would have built. Everything above the copy pays once per run.

On receive the interface merges each batch before it classifies: consecutive in-order segments of one flow become one merged segment, by the rule of LRO, which takes one classification and one step but still one credit unit per segment. Every TCP decision stays in Erlang: the C of the cut and the merge (erlang/apps/erx_net/c_src/) copies, sums, writes the header values Erlang chose and groups frames by a fixed rule.

17.5.4What is built

The merge is Built on every transport: the QEMU suite checks that vtnet0 merged bursts at least once (tools/lib/erx/net_device.rb). Output’s run rule and the C that cuts a run (c_src/erx_tcp_cut.c) are Built, and the host’s tap transport cuts runs in its copy; cutting runs in erXos’s device NIF is designed and not built, so on erXos output sends one frame per segment. Designed and not built too are the lean step, which splits the control block into a small record that changes per segment and a cold one, and a faster host test transport over netmap, a framework for moving packets without a system call per packet. The development manifests run the network domain on the interpreter rather than the JIT, so the figures above say nothing about QEMU or the Pi 4.

17.6Sockets act for one domain

A domain without the network service’s facet cannot open a socket, and every socket acts for one domain alone (design-network-stack.md, Authority and admission).

17.6.1Who holds the network device

Only the network domain holds a network device, by the system manifest and its build check (chapter 7), and no client domain reaches a driver, an interface process or a shared table. In the design, a WiFi domain drives the Raspberry Pi 4’s WiFi chip and lends that medium to the network domain as an interface; that is designed and not built (chapter 18).

The system domain, the trusted base, holds no net facet, so no socket’s traffic ever enters it and the trusted base parses no packet (chapter 11, section 11.2.2). The QEMU suite checks it: the system domain’s command shell holds no net facet, and its ifconfig, ping and host are refused.

17.6.2The net facet

A domain’s use of the network is one facet of the network service (chapter 11, section 11.6), which the client’s runtime installs under the name net. Unlike most facets, its process lives in the network domain, and the stack’s code calls it erx_net_grant. It starts the domain’s sockets and charges their work to its account, within the manifest’s facet row and the bounds of abi/services/net.txt: at most 4,096 sockets, 128 KiB to 64 MiB of buffers across them, and ports below 1024 only when the row lists them. The facet process monitors the network domain’s connection to its client before it exports itself, so the connection’s end ends the facet and every socket it started.

Every message from another domain passes one admission module first: erx_net_admit in the network domain and erx_sock_admit in the client’s socket process alone take messages out of their origin envelope, which a build check enforces (tools/lib/erx/net_checks.rb). Both are Built by hand; generating them from abi/services/net.txt is designed and not built.

17.6.3A socket is two processes

Root, on a machine connected to the Internet, types this in its Erlang shell:

{ok, S} = gen_tcp:connect("example.org", 80, [binary, {active, true}]).

gen_tcp dispatches every call on a socket of the form {'$inet', Module, Handle} to Module, and on erXos the default backend is erx_gen_tcp (erlang/apps/erx_net_client/src/erx_gen_tcp.erl). It starts a socket process in root’s session domain, a gen_statem that presents the stream as gen_tcp does. The socket process calls the session’s net facet, which starts a connection process in the network domain, linked to it. The socket process alone talks to the connection and sends the program the socket’s messages, so stock receive {tcp, S, Data} -> ... code works.

The connection’s pid is exported only to the client’s domain, with send alone (chapter 10, section 10.4), and every request must come from the socket’s own socket process. Any process of that domain may use the socket through it: the domain is the unit of authority. The QEMU suite runs this path both ways, connecting out to an echo server on the host and accepting a connection from the harness.

17.6.4Ports and the demultiplexing table

The demultiplexing table maps a key to the one socket that receives for it: a connection’s four-tuple, a listener’s local address and port, a connected or bound UDP socket. Lookup goes from the most specific key to the wildcard. Every insertion goes through erx_net_bind with ETS’s insert_new, which inserts only when the key is absent, so a key is held by one socket and ERTS decides a race between two binders.

17.6.5Name resolution

Each domain resolves names itself, with inet_res, OTP’s DNS resolver, over its own gen_udp. The facet’s resolver operation returns the name servers and search list the network domain learnt, and erx_inet checks them and loads them into the domain’s inet_db, in place of a resolv.conf. Query IDs come from the kernel’s random bytes rather than inet_db‘s counter, so an off-path attacker must guess both the ID and the source port. The QEMU suite checks that host resolves a name, answers NXDOMAIN (DNS’s “no such name”) for a missing one and maps an address back.

17.6.6Native programs

A native program’s socket descriptor names an open-file description, a process of the program’s session (chapter 13), which is the socket’s client at the network service. A blocking read becomes a watch for readability followed by a nonblocking read, so no request waits inside the network domain (design-network-stack.md, The interface towards POSIX programs). Sockets for native programs are designed and not built.

17.7Time from the network

A Raspberry Pi 4 has no battery-backed clock, so every boot starts near 1970 and every TLS certificate looks not yet valid. Measuring the time needs the network; writing the clock is the system domain’s, which parses no packet, so the design splits the job (design-time.md).

17.7.1Who measures and who sets

The SNTP client, ntpd, is an app with a session of its own. That session holds a net facet and a clock facet of the system domain’s clock service with the operation sample and not set. ntpd measures the server’s clock against the kernel’s monotonic clock and reduces each exchange to one integer: the server’s time minus monotonic time, in nanoseconds. That is the number the kernel’s time page stores, so a sample does not depend on what the wall clock reads, and the system domain decides on an integer and parses nothing from the network. Root’s own right to set the clock with date, and what checks it, is chapter 11, section 11.8.

17.7.2Before the network: the build date and the real-time clock

The build’s date is the commit time of the image’s source. A clock service that starts on an earlier clock sets it to that date, shown in /proc as approximate until a sample or date sets it. So HTTPS works from boot, at the cost of accepting a certificate that expired between the build and the real date. A machine with a real-time clock reads it at boot instead: on QEMU virt the driver erx_rtc_pl031 reads the PL031 through a hardware description that names only the data register, so it cannot change the clock.

17.7.3The step rules

erx_clock_core:decide/4 takes the first row of this table whose condition holds, with D the sample’s distance from the clock (design-time.md, Steps):

ConditionAction
the sample is before the build’s daterefuse
the clock is as the boot or the approximate start left itstep to the sample
abs(D) =< 128 mshold
128 ms < abs(D) < 1000 s, and the last sample erred the same waystep
128 ms < abs(D) < 1000 s, otherwisehold, and remember the sign
abs(D) >= 1000 srefuse

Whether the boot has changed the clock is read from the time page, which nothing resets during a boot, so no restart of ntpd or the clock service can renew the one unrestricted step. There is no slew: the clock moves only by steps.

The rules filter one bad measurement and bound nothing over time: without authentication, the server, or anyone on the path to it, decides the wall clock (design-time.md, Security). ntpd runs where the machine’s manifest starts it: the Raspberry Pi 4 image starts it against pool.ntp.org, named in the session’s environment (config/manifest/raspi4b-development.txt). Enabling it from /etc/start.conf is designed and not built. The QEMU suite runs ntpd against the harness and checks the step, ntpctl step and date. The rest of section 17.7 is Built.

17.8What fails, and what it reaches

Each failure ends the smallest thing that holds the failed state (design-network-stack.md, What fails and what survives).

What failsWhat endsWhat the reader sees
a malformed framenothing: the decoder refuses it, the frame is dropped and countednothing
an interface process’s step raisesnothing: the step left the state as it was, the frame is droppednothing
a connection’s step raisesthat connection, with a RST{tcp_error, S, econnaborted} on that socket alone
the network driver crashes or is restartedthe link: driver, interface and DHCP client restart in orderconnections carry on; a QEMU suite shows an open connection surviving (chapter 15)
a client domain endsits net facet, and every socket the facet started, each connection with a RSTthe peers see resets
any top-level process of the stackthe network domain, which the domain service restartsevery connection ends; clients get new facets

The stack has known limits. Nothing sums the facets’ buffer limits against the network domain’s memory, so many facets at their maximum could exhaust it. And the interface process decodes every frame the link delivers, so a flood at link rate costs it its full time (design-network-stack.md, Residuals).

17.9Design choices

A stack in C, such as lwIP, in the network domain. lwIP moved six to seven times the bulk throughput in section 17.5. But it is a second TCP with its own decisions, its memory errors would be the network domain’s, and the inspector could not see its connections. erXos keeps every TCP decision in Erlang and moves into C only work on bytes: the copy, the checksum, the cut and the merge.

A process per layer. An Erlang design might give Ethernet, IP and TCP each a process, as the layers suggest. Every packet would then cross three mailboxes with no bound between them, and each layer would decode the bytes again. The stack has one process per owner of state, decodes once, and hands checked records on.

Summary

erXos’s TCP/IP stack is Erlang, in the network domain, modelled on FreeBSD’s for its protocol behaviour and with a structure of its own. Each piece of state has one owner process that decides by a pure step, every queue is bounded by a credit, and the TCP core, one pure function over one record, is explored exhaustively on a host. Merging makes one step serve many segments. A domain uses the network only through its net facet, the trusted base parses no packet, and network time reaches the clock as one integer.

  • design-network-stack.md: the stack’s whole design, the departures from its model and their reasons, the fast path and the verification plan.
  • design-time.md: network time, the clock service, the approximate start and the step rules.
  • Chapter 18 and chapter 19: the wireless media.
  • Marshall Kirk McKusick, George V. Neville-Neil and Robert N. M. Watson, The Design and Implementation of the FreeBSD Operating System, 2nd ed., 2014, its networking chapters: the stack erXos’s is modelled on.
  • W. Richard Stevens, TCP/IP Illustrated, Volume 1: the protocols, one packet trace at a time.
  • RFC 9293, “Transmission Control Protocol”, 2022; RFC 6675, SACK-based loss recovery; RFC 5961, defences against blind in-window attacks; RFC 6528, initial sequence numbers: the RFCs the design cites most.