erXos: An Operating System in ErlangChapter 19Bluetooth
Contents
  1. 1Operating Systems from the Ground Up
  2. 2Microkernels and Capabilities
  3. 3Erlang and OTP
  4. 4History and Goals
  5. 5Design Overview of erXos
  6. 6Correct by Construction
  7. 7One Source for Every Fact
  8. 8The Kernel Interface and Fiasco
  9. 9Processes, Runtimes and Two-Level Scheduling
  10. 10Domains: A Wall Around Some Processes
  11. 11Authority and Security
  12. 12Memory and Budgets
  13. 13Native Programs, Pipes and POSIX-lite
  14. 14Files, Namespaces and Service Trees
  15. 15Devices and Drivers
  16. 16Storage and Updates
  17. 17The Network Stack
  18. 18WiFi
  19. 19Bluetooth
    1. 19.1Bluetooth Low Energy, peripheral first
    2. 19.2Where the parts run
    3. 19.3The HCI codec
    4. 19.4The controller process
    5. 19.5Supervision and failure
    6. 19.6The design above the controller
    7. 19.7Testing a radio without a lab
    8. 19.8Design choices
    9. Summary
    10. Where to read next
  20. 20The Command Shell and the UNIX Surface
  21. 21The Live Machine
  22. 22Displays, Windows and the Web
  23. 23Apps and Packages
  24. 24Startup, Configuration and Shutdown
  25. 25Checking the System
  26. AGlossary
  27. BReading the Design Documents
  28. CBuilding and Booting erXos
  29. DDependencies and Licenses
  30. ELineage

Chapter 19Bluetooth

Background needed. Chapter 3, sections 3.3 and 3.4: links, supervisors and restart strategies. Chapter 6, section 6.3: a pure core with its effects at the edge. Chapter 14, sections 14.2 and 14.5: file trees, stream files and keepers, ctl files. Chapter 15, sections 15.1 and 15.5: drivers and the unit rule.

Anyone within radio range of a Raspberry Pi 4 can send bytes to its Bluetooth chip. The chip’s firmware is closed: it parses what comes over the air, then hands the host what it decided to pass on. A host stack that trusts those bytes trusts both the firmware and every stranger in the room.

erXos parses every byte from the controller in a domain of its own, the Bluetooth domain, written in Erlang, which holds no device, so a parser defect is never a defect in kernel mode. The system domain holds the chip’s UART and serves it as a file tree of two files, data and ctl, parsing none of the bytes it moves. Above the line, one controller process, built around a pure core, drives the chip, and a link process per connection carries each phone’s traffic.

19.1Bluetooth Low Energy, peripheral first

What does a phone see when it meets an erXos machine? In the design, a device that advertises, accepts a connection and answers reads and writes: the peripheral role, the side that advertises and waits.

The design offers a phone four things (design-bluetooth.md):

  1. Its name and the UUIDs of its advertised services, in a phone’s scanner.
  2. The characteristics that apps on the machine declare, to read, write and subscribe to.
  3. A byte stream between a phone and an app, shaped like Nordic’s UART service so that existing phone terminal apps open it.
  4. Pairing and bonding by LE Secure Connections, so that a characteristic can require an encrypted or an authenticated link. Pairing makes a shared key for one connection; bonding keeps that key, so the phone and the machine recognize each other next time.

Of these, only the advertising data of the first is built.

The central role, which scans and connects, belongs to the design of HID over GATT, for keyboards and mice. Classic Bluetooth, the older BR/EDR radio of headsets, is neither built nor designed, and neither is the legacy pairing that LE Secure Connections replaced.

19.2Where the parts run

Which domain holds the chip, and which reads what it says? The system domain holds the chip’s UART; the Bluetooth domain reads its bytes.

In the rest of this chapter a plain “handle” means a connection handle, not a kernel handle of chapter 8.

On the Raspberry Pi 4 the controller is a Cypress CYW43455, wired to UART0 (design-bluetooth.md, The hardware). The board’s five UARTs are copies of the PL011, Arm’s standard UART design, and share one register page and one interrupt line. Under the unit rule of chapter 15 they are one unit, held whole by the system domain, which needs UART3 for the host link. The Bluetooth domain cannot hold UART0 without holding the host link too.

So the system domain shares the controller’s line the erXos way, as a file tree with a ctl file (chapter 14). The line tree, declared in abi/services/line.txt, has two files:

FileReadWrite
datathe bytes the controller sent, waiting while there are nonebytes to send, in order
ctlrefusedreset, and nothing else

“Line” here means a serial line, not chapter 15’s interrupt line. The tree, erx_line_fs, and its keeper, erx_line (chapter 14), move bytes between the files and the UART driver, erx_pl011, parsing only ctl’s single word, which has a fuzz target. A waiting read is held by the keeper and holds up no write. If the controller overruns the 64 KiB receive buffer, the line is broken: the buffer is dropped and every read answers eio until the next reset. The tree and its keeper are Built, with host tests. Serving them as a file grant of the line interface (chapter 11) is Designed; the manifest check already allows at most one such grant, since one line service serves one line.

A reset puts the line in one known state, whatever the last run left (design-bluetooth.md, A reset): it stops the UART driver, turns the controller off, restarts the driver with the UART programmed from nothing, turns the controller on, and discards what the controller sent while it powered up. The steps without power are Built in erx_line.erl and erx_pl011.erl, with host tests over a model driver. The power steps are Designed: the enable line is BT_ON, a pin of the firmware’s GPIO expander (research-wifi-pi4.md), whose pin and polarity a board probe settles.

In the design, the Bluetooth domain mounts its grant of the line at /line and opens /line/data and /line/ctl as ordinary files. Its manifest row holds that grant, a store for its bonds (19.4.5) and, when the machine offers a login console over the serial service, the system domain’s login facet. The manifest check is designed to refuse anything more, because a stranger in radio range reaches whatever the domain holds through a parser defect. Figure 19.1 shows the path.

system domainBluetooth domainradioUART0file grantfile grantble facetphonecontroller chiperx_pl011 driverappline tree: data, ctlreadercontroller processwriterlink processesstore
Figure 19.1.The Bluetooth controller process and its line to the chip. The system domain moves bytes and parses none; every byte from the chip is decoded in the Bluetooth domain, which holds no device.

The build makes a release of the stack, erx_bt, for a domain named bt, but no manifest erXos ships names that domain or grants a line, so no boot runs it. The ble facet is Designed.

19.3The HCI codec

How does a stream of bytes from a hostile chip become something an Erlang process can act on? The H4 framing cuts the stream into packets, the HCI codec decodes each packet into a checked term, and nothing after the codec sees raw bytes. Both are pure functions.

The framing, erx_h4:split/1 (design-bluetooth.md, H4 framing), takes the bytes read so far and returns every whole packet and the remainder, an incomplete packet that waits for the next read. Erlang’s binary matching does the work:

split(<<?ERX_H4_INDICATOR_EVENT, Code, Length, Parameters:Length/binary,
        Rest/binary>>, Packets) ->
    split(Rest, [{event, Code, Parameters} | Packets]);
split(<<?ERX_H4_INDICATOR_ACL, _:16, Length:16/little, _/binary>>, _Packets)
  when Length > ?ERX_BT_ACL_RX_MAX ->
    {error, {acl_too_long, Length}};
split(<<?ERX_H4_INDICATOR_ACL, Word:16/little, Length:16/little,
        Data:Length/binary, Rest/binary>>, Packets) ->
    split(Rest, [acl_packet(Word, Data) | Packets]);
split(<<Indicator, _/binary>> = Rest, Packets)
  when Indicator =:= ?ERX_H4_INDICATOR_EVENT;
       Indicator =:= ?ERX_H4_INDICATOR_ACL ->
    {lists:reverse(Packets), Rest};
split(<<Indicator, _/binary>>, _Packets) ->
    {error, {bad_indicator, Indicator}};
split(<<>>, Packets) ->
    {lists:reverse(Packets), <<>>}.

The first clause matches only when the event’s whole length is present, so nothing reads past the input. An ACL packet longer than 251 bytes, the host’s own bound, is an error before its data is read, and so is any indicator but an event’s or an ACL packet’s. The specification calls either case a loss of synchronization, after which a reset is needed (Core Specification 6.3, Vol 4 Part A). In erXos the controller process stops, and its restart resets the line and the controller. The bound is a constant, not a value the controller reported, so a controller cannot raise it.

The codec, erx_hci, decodes each event against the layout the specification gives. A length that does not match gives malformed, never a partial read. An event the host does not use gives {unknown, Code}, which the controller process counts and drops. No atom is made from input, so a chip cannot fill the atom table (chapter 3).

Every code and bound the codec uses comes from abi/formats/bluetooth.txt, which cites the specification’s section for each and from which the build generates the macros, by the rule of chapter 7.

The framing and the codec are Built, with a fuzz target that mutates inputs written from the specification’s layouts and must catch each of its seeded faults, such as the ACL bound removed (tools/fuzz/erx_bt_hci.rb).

19.4The controller process

One process, erx_bt_hci, owns everything about the controller: the bring-up, the command in flight, the buffers, the map from handles to connections and the advertising. Its pure core, erx_bt_hci_core, takes every decision, in the shape of chapter 6: step(Input, Now, Core) returns the next state and a list of actions. The process turns each message (a decoded event, the writer’s answer, a timer, a link’s request) into an input and performs the actions: reset the line, write bytes, start, end or tell a link, or stop with a reason. All of this is Built.

19.4.1The bring-up, a line of steps

A freshly powered controller knows nothing of the host. The bring-up is a fixed list of steps that the core builds when it is created (design-bluetooth.md, The bring-up):

  1. Reset the line: write reset to ctl.
  2. HCI_Reset.
  3. Identify the controller: its version, its supported commands and features, its address.
  4. Load the patch, where the manifest names one, then reset and identify again.
  5. Ask the controller’s buffer sizes, and tell it the longest packet the host accepts.
  6. Configure: the events the host wants, the machine’s address, and the advertising parameters and data.
  7. Ready: enable advertising.

Each command step waits for its Command Complete with status success. Any other answer, or none before the deadline, stops the process with the step and the status as its reason, so a bring-up reaches ready or ends with a stated cause. The steps after the patch are distinct from the first ones, so the list has no loop.

The core keeps at most one deadline, in one field that also says what it guards: a line reset, a wait, a command due or a command sent. A second timer, or cancelling one never armed, cannot be written, and an early or repeated wake-up changes nothing.

19.4.2The patch

Hosts that drive this chip first load a patch into its RAM: a vendor file of HCI commands, BCM4345C0.hcd for the Pi 4, replayed one record at a time. erx_bt_patch reads the file whole first and accepts it only if it consists entirely of complete Write RAM and Launch RAM records, so a damaged file sends nothing. A controller that reports another manufacturer than the manifest names stops the bring-up, so no vendor command reaches a chip it was not written for. The reader and these checks are Built, with host tests; whether the Pi 4’s chip needs the patch is a question for a board probe.

19.4.3Commands and controller credits

The process keeps one command in flight, whatever the controller accepts, since the host’s commands are few and never urgent; an answer that names any other command stops the process.

Each free buffer in the controller is a controller credit: room to send it one data packet. This is not the channel credit of chapter 13, which limits a native program’s requests. Each live handle counts its packets in flight, and the free credits are the controller’s total less the sum. A report of completed packets for a handle that is not live, or for more packets than that handle has outstanding, stops the process. A forged report therefore cannot grow the free count, and the count cannot fall below zero. Credits are derived, never stored, so a disconnection returns a handle’s credits by removing the handle.

19.4.4A connection, end to end

Figure 19.2 follows one phone from the machine’s advertising to one attribute read.

LE Set Advertising EnableCommand Completeadvertising: name, UUIDsconnectLE Connection Complete (handle 1)start link {1, Serial}, ConnATT Read Request (ACL, handle 1)ACL packetto_linksend ATT Read ResponseACL packet (one controller credit)Number Of Completed Packetscompleted, 1phonecontroller chipcontroller processlink process
Figure 19.2.Advertising, connecting and one attribute read. The controller process owns the handle and the controller credits; the link process owns the conversation. Service UUIDs in the advertising and the attribute read itself are Designed: the built link processes drop every ACL packet and count it.

When the controller reports a connection, it has already stopped advertising. The core starts a link process named {Handle, Serial}, the serial counting the core’s connections, and checks that name on every request the link makes. If handle 1 passes to the next phone, a request the old link sent late changes nothing. The link also gets a fresh Erlang reference, its Conn, the name apps use in the design. It is the rule of chapter 15: a name dies at once, and the hardware’s resource, here the handle’s credits, returns only when the controller reports Disconnection Complete. Advertising stays on exactly while connections are below the manifest’s limit, one by default.

19.4.5Advertising and the machine’s address

The advertising data and the scan response, with the name and the appearance (a number that tells a phone what kind of device to show), are a pure function, erx_bt_adv:data/1. Each is at most 31 bytes; a name that does not fit is shortened at a character boundary. The function can carry service UUIDs, but the controller process passes none, since no service is declared yet.

The machine’s address on the air is a static random address, made from the system’s random generator, which the root service keeps. Like the IPv6 addresses of chapter 17, it does not embed a hardware address, so it names neither the board nor its chip (design-bluetooth.md, The identity address). The store, erx_bt_store, makes the address at its start; the controller process asks for it at each of its own starts, so a restart of the controller keeps the machine’s identity. Both are Built.

19.5Supervision and failure

What happens when the controller sends garbage, or a link misbehaves? Almost always the same thing: the part that saw it stops, and its restart resets the line and the controller, so the next bring-up starts from one state.

The supervision tree in erlang/apps/erx_bt is Built:

erx_bt_sup, the top, is rest_for_one with the store first and the controller group second, so a restart of the group leaves the store and its address alone. The group, erx_bt_host_sup, is one_for_all, at most five restarts in 60 seconds, over the link supervisor, the line’s reader and writer, and the controller process, so all four always belong to one run of the core. Links are temporary children, never restarted: a link that ends is a disconnection.

What failsWhat endsWhat the next state is
a malformed or unanswered exchange with the controller, or a failed read, write or reset of the linethe controller groupthe line reset, the controller reset, a fresh bring-up
a link process crashesthat linkthe core disconnects its handle with reason Remote User Terminated Connection; the phone may reconnect
a late request from a link that has gonenothingthe core ignores it
more than five group restarts in 60 secondsthe group, so erx_bt_sup restarts itOTP’s default intensity applies: a second such failure within five seconds stops erx_bt_sup and the erx_bt application

A restart costs one bring-up without advertising, and every connection. The alternative is a case for each half-finished state a controller can be left in; the reset removes all of them.

19.6The design above the controller

What does an app write to offer a phone a characteristic, and who decides whether the phone may read it? The design answers with configuration and one function. Everything in this section is Designed.

An app declares its services as data, one checked-in file per service under abi/gatt/ (design-bluetooth.md, Declarations). The serial service’s file reads:

service nus 6e400001-b5a3-f393-e0a9-e50e24dcca9e advertise
characteristic rx 6e400002-b5a3-f393-e0a9-e50e24dcca9e 244 write open write_without_response open
characteristic tx 6e400003-b5a3-f393-e0a9-e50e24dcca9e 244 notify open

Each characteristic names its accesses and the level each requires: open, encrypted or authenticated. The manifest selects which services a machine serves. The Bluetooth domain builds its attribute table once at its start, and nothing adds or moves an attribute while the domain lives, so no attribute handle moves under a connected phone.

Every access then passes one pure function, the gate (design-bluetooth.md, One table, one gate), which compares the level the declaration requires with the link’s security and answers allowed or the specification’s ATT error. A phone cannot send the link’s security: it is set only when the controller reports that encryption started, and a link is encrypted with at most one key in its lifetime, so the level cannot change under it. No app checks security, so none can forget to.

Pairing happens only in a window that someone opened by opening the control tree’s pair file, and the window ends with its first pairing attempt, whether it succeeds or fails. After a failure the next attempt waits, from 2 up to 64 seconds, and the wait outlives every window. A pairing never replaces a bond; only an explicit forget ends one (design-bluetooth.md, The pairing window).

19.7Testing a radio without a lab

QEMU emulates no Bluetooth controller, and the Pi 4 is tested by hand. How, then, is the stack tested? Mostly on the host, where the cores need no radio.

The controller’s core is explored exhaustively (test/erlang/erx_bt_hci_tests.erl): the test walks, breadth first, every state reachable from start by the inputs it models. It explores two configurations, one with a patch and one without, each allowing two connections at once; a path makes up to three connections in turn, so a handle is reused while the other is live. A comment beside the test’s budget puts the walk at about 160,000 states for each. Every state must keep the free controller credits at zero or above and connections within the limit, ignore requests from gone links, and stop only for a listed reason. Seeded faults (chapter 6), such as a handle kept live after its Disconnection Complete, must make it fail.

Over a model of the line (test/erlang/erx_bt_group_tests.erl), the controller group reaches ready again after its controller process is killed, keeps its address, and turns a killed link into one disconnect command. The patch reader, the line tree and the PL011 programming have host tests too. All of these tests are Built.

For the protocol above HCI, the design uses Bumble, a Bluetooth stack written in Python, whose virtual controllers speak H4 over a TCP socket (design-bluetooth.md, Hosted mode against Bumble). erXos runs in hosted mode on one virtual controller, while a Bumble peer on another scans, connects, reads and pairs. Each test has a seeded fault, such as advertising data built wrong, which must make the UUIDs vanish from the peer’s scan. This harness is Designed.

What only a board answers, the design lists as board probes (design-bluetooth.md, Board probes): the UART’s pins and the controller’s power as the firmware leaves them, the enable pin, and whether the patch is needed. A manual checklist with a phone follows. No probe is recorded as run, and the stack has met no real controller.

19.8Design choices

A domain of its own, not the domain that holds the board’s other low-speed devices: one runtime more, so that a parser defect reachable from radio range reaches no driver.

Declarations rather than services registered at run time. The stacks the design surveyed have an application register its attribute table in code, which brings attribute handles allocated while phones are connected, announcements that the table changed, and conflicts between apps. Configuration removes them; a newly installed app’s table takes effect at the domain’s next start.

A power cycle and a fresh UART at every reset. Without them, each bring-up would need a case for every state the last one could leave: a controller at another rate, patched or half-patched, a UART holding the last controller’s bytes.

The patch named by the manifest, not chosen by the controller. The driver could choose the patch file by the name the controller reports, but that name is text the controller writes, so in erXos it chooses nothing.

115200 baud rather than 3 Mbaud. A faster rate needs vendor commands that differ between hosts and a switch timed to a response. BLE’s traffic fits 115200; the faster rate is an optimization to measure.

Summary

Every byte a Bluetooth controller sends is hostile, so erXos decodes them in a Bluetooth domain that holds no device; the system domain moves them through a two-file line tree and parses none. A pure core drives a fixed bring-up, derives its controller credits from the live handles and names each connection by handle and serial. Almost every failure ends the controller group, whose restart resets the line and the controller. This is built and tested on the host, on no board; the GATT table, its gate and pairing are designed.

  • design-bluetooth.md: the whole design.
  • design-shared-devices.md, The unit rule: why the system domain holds the controller’s UART.
  • research-wifi-pi4.md: the Pi 4’s wireless chip in the board’s device tree.
  • Bluetooth SIG, Bluetooth Core Specification, version 6.3, 2026: Vol 4 Part A for the H4 transport, Vol 4 Part E for HCI, Vol 3 Parts F and G for ATT and GATT, Vol 3 Part H for pairing.
  • Google, Bumble (github.com/google/bumble): the Python stack the design tests against.