erXos: An Operating System in ErlangAppendix BReading the Design Documents
Contents
  1. 1Operating Systems from the Ground Up
  2. 2Microkernels and Capabilities
  3. 3Erlang and OTP
  4. 4History and Goals
  5. 5Design Overview of erXos
  6. 6Correct by Construction
  7. 7One Source for Every Fact
  8. 8The Kernel Interface and Fiasco
  9. 9Processes, Runtimes and Two-Level Scheduling
  10. 10Domains: A Wall Around Some Processes
  11. 11Authority and Security
  12. 12Memory and Budgets
  13. 13Native Programs, Pipes and POSIX-lite
  14. 14Files, Namespaces and Service Trees
  15. 15Devices and Drivers
  16. 16Storage and Updates
  17. 17The Network Stack
  18. 18WiFi
  19. 19Bluetooth
  20. 20The Command Shell and the UNIX Surface
  21. 21The Live Machine
  22. 22Displays, Windows and the Web
  23. 23Apps and Packages
  24. 24Startup, Configuration and Shutdown
  25. 25Checking the System
  26. AGlossary
  27. BReading the Design Documents
    1. B.1What the documents are, and what they are not
    2. B.2A reading order
    3. B.3What each document is for
    4. B.4The common layout
    5. B.5Where decisions are recorded
    6. B.6From the book to the documents
    7. Summary
  28. CBuilding and Booting erXos
  29. DDependencies and Licenses
  30. ELineage

Appendix BReading the Design Documents

The book explains erXos; the design documents decide it. Behind the chapters stand forty-five documents named design-*.md, together more than a hundred thousand lines, each the contract of one subsystem. Each is written for the people who build or review its subsystem and assumes they know the rest. A reader who opens one cold meets terms defined three documents away. This appendix is the way in: what the documents are, an order to read them in, what each is for, the layout most share, where decisions are written down, and which chapter prepares a reader for which documents.

B.1What the documents are, and what they are not

A design document states a contract. It says what a subsystem promises, what it refuses, which invariants hold, who owns what, how each part fails and how the claims are tested. When the book and a design document disagree, the design document is right and the book is fixed (README.md, Conventions).

Three properties of the documents surprise a newcomer.

  • They describe the target, in the present tense. A design document says how the subsystem works when it is complete. It is not trimmed to match the code and says nothing of what is finished. To learn what is built, read the root README.md, Current status, the tree and its tests, and TODO.md, which lists open defects and known gaps. The book’s Built and Designed marks come from those, not from the design documents’ tense.
  • Three describe the kernel interface in a stronger form than the system’s kernel gives it. design-kernel.md defines the objects, capabilities and calls of the kernel interface of chapter 8 as a kernel that would hold every isolation property itself; design-scheduler.md describes CPU time in the same form, and design-arch-aarch64.md binds that form to the processor. The system’s kernel is Fiasco. design-fiasco.md says which of those properties Fiasco gives, how liberx carries the interface over it, and what the system does without the rest. Where they differ, design-fiasco.md describes the running system, and the book follows it.
  • They carry the project’s planning notes. The documents say in places how the work is staged; the book leaves those notes out, and plan.md, the roadmap, holds them.

Around the design documents sit others with different jobs. overview.md is a short introduction for a reader new to the idea. design.md is the architecture: the shape of the system, every decision that constrains more than one subsystem, and the Document map from which section B.3 is drawn. style.md is the code style guide. The research records, research-*.md, are dated reports of what was measured or read, never edited after. The user guide, docs/guide/, says how to build and use the system.

B.2A reading order

A reader who wants the documents without the book does best in this order, in which each step defines terms the next one uses.

  1. overview.md. The idea in a few pages: why an operating system around Erlang, and what a user sees.
  2. design.md, five sections. Context and Requirements say what problem the system solves. The model defines the vocabulary every other document uses: domain, facet, lane, funnel, budget and the rest. Overview shows what runs where, and Invariants lists the properties the system holds by construction. Leave the rest of design.md for later; it is a reference.
  3. design-fiasco.md, the Contract first. What the kernel gives the system, how liberx carries the kernel interface over Fiasco, and what Fiasco does not give.
  4. design-domains.md. Domains, the pid space, connections, exports, the funnel and facets: how two walled groups of Erlang processes talk as one Erlang system.
  5. design-system.md. The system domain: the root task, the system manifest, the domain library, devices and drivers, and the system services. It is long; read its opening and the sections on the manifest and the domain library, and return for the rest.
  6. design-runtime.md. How ERTS runs in user mode on liberx, the runtime images, the device NIF and the C library seam.
  7. design-files.md. Namespaces and the file protocol, which most services use to present themselves.
  8. The subsystem the reader came for, from the table in section B.3.
  9. design-generation.md, design-abi.md and design-verification.md. How the documents’ tables become code and how their claims are checked.

design.md’s Decisions is worth a skim after step 2 and a careful read after step 7, when its rows name things the reader has met.

B.3What each document is for

The documents fall into groups that match the book’s parts. Each row gives the question the document settles.

DocumentWhat it settles
design.mdThe architecture: the model, the invariants and every decision that spans subsystems
design-fiasco.mdHow the system runs on Fiasco: the liberx backend, the call table, the scheduler and budgets on Fiasco, and what Fiasco does not give
design-kernel.mdThe kernel model behind the kernel interface: objects, capabilities, budgets, memory, threads, IPC, interrupts, boot and L4 lineage
design-scheduler.mdThe two levels of scheduling: groups and their CPU quotas in the kernel model, urgency, park and wake, service work, and the runtime contract
design-arch-aarch64.mdThe AArch64 binding of the kernel’s port contract
design-platforms.mdThe boards: QEMU virt, the Raspberry Pi 4 and 5
design-runtime.mdRuntimes in user mode: ERTS recombined, runtime images, the device NIF, the C library seam, liberx
design-shared-code.mdOne image of loaded code mapped into every domain
design-domains.mdDomains, the pid space, connections, exports, the funnel, frames and facets
design-system.mdThe root task, the system manifest, the domain library, devices and drivers, the system services and the vault
design-programs.mdNative programs and the POSIX-lite personality
design-pipes.mdPipes in user space
design-files.mdNamespaces, the file protocol (9P2000.L), file servers and shared mappings
design-tmp.mdThe RAM file system at /tmp
design-storage.mdPartitions, the boot image, system partitions, EROFS, FAT and updates through two slots
design-f2fs.mderXos’s F2FS, the main read-write file system, in Erlang
design-stopgap.mdlittlefs in the storage domain and its contract
design-shared-devices.mdDevices that share a register page or an interrupt line
design-device-dma.mdDMA by devices that are not cache-coherent, and the descriptor forms the device NIF drives
design-device-tree.mdThe firmware’s device tree in the root task
design-pcie.mdPCIe: the enumerator, BAR placement, bus mastering and interrupts
design-usb.mdUSB: xHCI, hubs, HID and mass storage
design-header-io.mdThe Raspberry Pi 4’s 40-pin header: GPIO, I2C, SPI and PWM
design-board-sensors.mdThe Raspberry Pi 4’s firmware readings and /proc/sensors
design-network-stack.mdThe TCP/IP stack, FreeBSD’s behavior in Erlang, and its fast path
design-time.mdNetwork time and the clock service
design-wifi.mdWiFi on the Raspberry Pi 4’s CYW43455
design-bluetooth.mdBluetooth Low Energy, peripheral first
design-crypto.mdCryptography: OpenSSL in the runtime images, OTP’s crypto applications, HTTPS clients
design-ssh.mdSSH: the SSH domain, logins, keys and channels
design-shell.mdThe command shell: rc’s language, the command namespace, text pipelines, job control
design-vi.mdevi, the screen editor run as vi
design-live.mdThe live machine: console, ttys, attributed text, the inspector and /proc, tracing, the code store, the host link
design-log.mdThe system log
design-display.mdComposition, input routing, window buffers, the draw protocol, the terminal, widget trees, kiosks, graphics hardware
design-desktop-programs.mdThe desktop’s own programs and the shared visual theme
design-find-act.mdFind and act: the session’s palette and routes from values to rc lines
design-web.mdThe web client and its two front ends
design-apps.mdApps as principals, the start table and service
design-packaging.mdPackages: one file per app, admission, install and update at run time, the SDK
design-app-trees.mdAn app’s own tree at /svc/NAME
design-generation.mdGenerated bindings, inspection, catalogs, test scaffolding, and the hardware and terminal sources
design-abi.mdThe declaration grammar, layouts, bounded readers and numeric assignments
design-build.mdThe build and host tooling: deadlines, source records, evidence boundaries
design-host-oracle.mdStock OTP as the test oracle, in an isolated Linux container
design-verification.mdPure protocol cores, TLA+ models and the evidence that ties a model to its code

Table B.1. The design documents, grouped by subject, drawn from design.md, Document map.

Sizes vary by a factor of sixty, from under 150 lines for littlefs to over nine thousand for design-kernel.md; in a long one, read the table of contents first.

B.4The common layout

Most design documents share one skeleton. The sections below appear roughly in this order, as in design-pipes.md; the counts say how many of the forty-five carry each one under that name.

  • Context (35). The problem, the consumers that need the subsystem and what breaks without it.
  • Requirements (33). What the subsystem must do. Each requirement should name the consumer that needs it; one that names none is a defect in the document, since the project questions every requirement before building for it.
  • Overview (22) or The construction (10). The parts, where each runs and who owns which state. A large subsystem gets one section per part instead, as in design-wifi.md.
  • Data model (21) and Interfaces (24). The records, formats and calls, often as tables that a generator reads (chapter 7).
  • Algorithms and behavior (19). How the parts act over time, usually as pure functions and state machines.
  • Invariants (27). The properties that always hold, each stated so that a test or a build check can fail on it.
  • Concurrency and Failure handling, diagnostics and debuggability, under these or shorter names: a Concurrency section in 17 documents, a Failure section in 30. What runs at once, what each failure reaches, which supervisor restarts what, and what an operator sees.
  • Refusals and error results (9) and Untrusted input (2). What the subsystem says no to, with which status, and which bytes from outside it parses.
  • Security (31) and Performance (30). The threat each part faces and the costs that matter, with the measurement that settles each unmeasured one.
  • Verification or Verification and testing (42 together). The tests, fuzz targets, models and build checks, each tied to the claim it checks.
  • Alternatives considered (36). The designs that lost, and why.
  • Risks and open questions (16) or Open decisions (8). What is not settled.
  • References (36) or Sources (3). The specifications, papers and source code read, with versions.

Seven documents add a section On Fiasco, which says how the subsystem’s design is carried on the system’s kernel; design-fiasco.md holds the rest of that story. Two add a lineage section, on which Appendix E draws. design-kernel.md records where the kernel model keeps L4’s and Fiasco’s choices and where it departs; design-wifi.md, which idea came from which WiFi implementation.

Two habits make the documents faster to read. First, read Context, then Invariants, then Failure, then Verification: together they say what the subsystem promises and how anyone would know, and the middle sections become detail to look up. Second, follow the first link to a term rather than guessing. The documents define each term once, in design.md, The model or in the document that owns it, and use it exactly.

B.5Where decisions are recorded

Every decision has one home. The rule is the documents’ own application of “one source for every fact” (chapter 7): a decision written in two places comes to say two things.

  • A decision that constrains more than one subsystem is a row of design.md, Decisions: the decision, its rationale, and the document that specifies it. The two-level scheduling contract, a pid that names its owner domain and the use of FreeBSD’s network behavior are rows there.
  • Any other decision lives in the document it shapes, in the section the decision concerns. Eight documents also collect theirs in a Decisions section.
  • The designs that lost are in each document’s Alternatives considered, with the reason. design.md’s Alternatives considered holds the system-wide ones, such as running each domain as a stock node on a Linux-compatible layer. It answers most questions of the form “why not the familiar way?”
  • What is not yet decided is in Open decisions or Risks and open questions, each with the options and, where there is one, a recommendation. design.md’s Tensions names the trade-offs the architecture accepts knowingly.
  • Simpler designs proposed and not adopted wait in todo-simplification.md with their evidence. When one is adopted, it moves into its design document and leaves that file.
  • Defects and known gaps are items of TODO.md, each with a severity and a component.
  • Evidence is in the dated research records that decisions cite.

When a decision changes, its text is edited in place. git holds the history, and commit messages say what changed and why, so a design document never says “now”, “previously” or “as of”: if it does, that is a defect.

B.6From the book to the documents

Table B.2 gathers the design documents that each chapter’s “Where to read next” names.

ChapterLeads into
1, 2overview.md; design-fiasco.md; research-kernel-bases.md for the kernels compared
3design-runtime.md; research-erts-internals.md
4design.md (Context, Requirements, Decisions); design-fiasco.md; research-hierarchical-scheduling.md; research-prior-art.md
5overview.md; design.md; design-fiasco.md; design-domains.md; design-system.md
6design.md (Correct by construction in code); design-fiasco.md (Contract); design-device-dma.md; style.md
7design-generation.md; design-abi.md; design-system.md (the manifest); design-device-dma.md; design-verification.md
8design-fiasco.md; design-runtime.md
9design-scheduler.md; design-fiasco.md (the scheduler on Fiasco); design-runtime.md; research-hierarchical-scheduling.md
10design-domains.md; design-runtime.md; design-system.md
11design.md (Security); design-fiasco.md (security on Fiasco, rights); design-domains.md; design-files.md; design-time.md (who may set it); design-system.md (the vault); design-live.md (escape rules)
12design-fiasco.md (memory and budgets); design-system.md
13design-programs.md; design-pipes.md; design-shell.md; design-fiasco.md (vCPU upcalls on Fiasco)
14design-files.md; design-tmp.md; design-app-trees.md; design-live.md (/proc)
15design-system.md (devices and drivers); design-runtime.md (the device NIF); design-device-dma.md; design-device-tree.md; design-shared-devices.md; design-pcie.md; design-usb.md; design-header-io.md; design-board-sensors.md
16design-storage.md; design-stopgap.md; design-f2fs.md; design-platforms.md; design-fiasco.md (the end path)
17design-network-stack.md; design-time.md
18, 19design-wifi.md; design-bluetooth.md; design-shared-devices.md
20design-shell.md; design-vi.md; design-ssh.md; design-live.md (ttys); design-apps.md
21design-live.md; design-log.md; design-system.md (the crash record); design-fiasco.md (reporting kernel objects, the trace ring)
22design-display.md; design-desktop-programs.md; design-find-act.md; design-web.md
23design-apps.md; design-packaging.md; design-app-trees.md; design-storage.md
24design-system.md; design-fiasco.md; design-kernel.md (boot); design-abi.md; design-platforms.md
25design-verification.md; design-host-oracle.md; design-build.md; design-network-stack.md (its tests)

Table B.2. Which chapter prepares a reader for which documents, from each chapter’s “Where to read next” and the outline of chapters 24 and 25.

design-arch-aarch64.md has no chapter: it binds design-kernel.md’s form of the interface to AArch64’s registers and instructions, and is for a reader of that document. Building and booting the system is the subject of Appendix C, and design.md’s dependency table, with every component’s license, of Appendix D.

Summary

The design documents are contracts, one per subsystem. They describe the complete design in the present tense, so whether a thing is built is found in README’s Current status, the tree and TODO.md, not in their tense. Where a document describes kernel properties, design-fiasco.md says what the system’s kernel, Fiasco, gives. A newcomer reads overview.md, part of design.md and five core documents before the subsystem of interest. Most documents share one layout, and Context, Invariants, Failure and Verification together give a document’s promise. Every decision has one home: design.md’s Decisions for those spanning subsystems, the shaping document for the rest, Alternatives considered for what lost, Open decisions for what is not settled.