erXos: An Operating System in Erlang
This file is the plan of a book: its title, its readers, the order of its chapters and, for each chapter, what its sections explain, the design documents they draw on and the figures they want. The chapters themselves are written one at a time from this outline, each checked against the design documents and the tree.
What the book is
The design documents in docs/ are contracts. Each states one subsystem’s invariants, interfaces, failure paths and tests, precisely enough that an implementer can build it and a reviewer can check it, and each assumes the reader already knows the rest of the system. They are the wrong place to start. The book is the place to start: it tells the whole system from the beginning, in the order a reader needs it, with the reason for each choice next to the choice, and with every term explained where it first appears.
The model is McKusick, Neville-Neil and Watson, The Design and Implementation of the FreeBSD Operating System (2nd edition, Addison-Wesley, 2014). The book borrows its arc and its manner, not its text: an opening part of background, then a part on history, goals and the overall design; then processes, memory, the I/O system, communication and the operation of the running system; each chapter opening with the concepts and the terms, then the structures that carry them, then how they behave and fail.
Readers
- Programmers who know UNIX, as a user and a little from the inside, and want to know what an operating system built around Erlang looks like.
- Erlang programmers who want to know what happens below the runtime when every Erlang process is the operating system’s process.
- Students and researchers of operating systems, for the design choices: a capability microkernel under a language runtime, two-level scheduling, drivers as supervised processes, and a system whose properties are designed to hold by construction.
- People who will work on erXos, as the reading that comes before the design documents.
The book assumes a reader who can program, in any language, and may know little about operating systems, microkernels or Erlang. It does not assume C, Erlang, L4, Plan 9 or OTP’s internals. Every term is explained where it first appears: in Part I for the three subjects everything rests on, and in a Background box at the head of the chapter that first needs anything more specialised (DMA, TCP, USB, WiFi and the like). The glossary, Appendix A, lists every such term and points to where it is explained.
Reading paths.
- No background: read in order, from chapter 1.
- Knows operating systems but not microkernels or Erlang: chapters 2 and 3, then from chapter 4.
- Knows Erlang and OTP but not operating systems: chapters 1 and 2, the rest of chapter 3 only for ERTS (section 5), then from chapter 4.
- Knows operating systems and Erlang, new to L4 and capabilities: chapter 2, then from chapter 4.
- Knows all three: from chapter 4. A term met on the way is linked to its chapter or Background box.
- Wants one thread: each chapter’s “Background needed” line names what it builds on, so the network (chapters 17 to 19), the display (22) or startup (24) can be read after the chapters named there.
Built and designed
The book describes erXos as it is today, running on the Fiasco microkernel behind erXos’s kernel interface, and, where the design is not yet built, as it is designed to be on that system. The design documents describe the whole design. The book describes both what runs and what is designed, and never presents the second as the first. Every chapter carries a status box, and every passage about something designed and not built says so in its own sentence. The box uses two marks:
| Mark | Meaning |
|---|---|
| Built | works in the tree today, in a QEMU suite, on the board or in host tests, as the top-level README, Current status and the code show it |
| Designed | specified in the design documents and not implemented |
The outline’s marks are what README.md’s “Current status” says; the writer of each chapter checks them again against the tree before writing. When “Current status” changes, the status boxes of the chapters it touches are checked again in the same change.
Preface: why this order
The book runs from overview through processes, the I/O system and communication to system operation. Five choices of order follow from erXos’s structure or from its readers.
- Background comes first. The book does not assume that its reader knows what a kernel, a page table and a system call are: its readers include programmers who have never opened an operating system and Erlang programmers who have never met a microkernel. Part I (chapters 1 to 3) therefore teaches, from the ground up and with no erXos in it, the three subjects every later chapter leans on: operating systems, microkernels and capabilities, and Erlang with OTP. A reader who knows one of them skips its chapter, and the reading paths above say which. What only one chapter needs (DMA and cache coherence for devices, TCP for the network stack, the handshake of WiFi) is not in Part I: it opens that chapter, in a set apart Background box, so it is read where it is used. Each chapter begins with a “Background needed” line naming the chapters and boxes it builds on. The alternatives were worse: an appendix of background is not read before the chapters that need it, and boxes alone would repeat three large subjects in every chapter.
- Method comes early. Two chapters of Part II (6 and 7) explain how erXos is designed before any subsystem is: decisions as pure functions and effects at the edge, properties that hold by construction instead of by a list of cases, rules of the form “only X may do Y” as build checks, and one declared source for every shared fact, from which C, Erlang and Ruby are generated. Every later chapter uses these ideas (the device DMA plan of chapter 15 is a plan validated whole; the kernel calls of chapter 8 are generated), so the reader meets them once, with small examples, and then sees them recur.
- The domain comes before memory. In erXos the Erlang process is the unit of execution and failure, and the domain, a group of processes behind an address space, is the unit of protection. So Part III goes process and scheduler, then domains, then authority and security, then memory and budgets, then native programs, which are ports of a session and make sense only once domains and authority do.
- The user’s machine is a part of its own. The shell, the live machine, the display and apps are not utilities on top of the system: each is a service built from the pieces of Parts III to V (domains, facets, file trees, pure cores), and each shows one of erXos’s distinctive ideas (UNIX tools over live Erlang state, a machine you can inspect while it runs, one interface rendered natively by every display). They come after the parts they are built from and before system operation.
- Checking comes last. How erXos knows it is right (OTP’s own suites, musl’s tests, fuzz targets, model checking, seeded faults, reproducible and deterministic boots) draws on every chapter, so it closes Part VII.
Kernel services are chapter 8, the kernel interface, because in erXos the kernel’s services are exactly its interface: everything else runs in user mode.
Conventions
- Every chapter follows the prose guide, style.md: its voice, the shape of a chapter, background for newcomers, honesty about what is built, and the checklist each writer runs before committing.
- Each chapter is a file in this directory, in GitHub Flavored
Markdown, its figures Mermaid where GitHub renders it and text diagrams
otherwise, as Decisions settles. The file name is the
two-digit chapter number and a short slug,
NN-slug.md; the appendices area-slug.mdtoe-slug.md. The table below is the one list of names. - A link from one chapter to another is a relative link to the file, with
a section anchor where the chapter has one (
10-domains.md#103-...). A chapter that is not yet written is linked to its outline section in this file (README.md#chapter-10-domains-a-wall-around-some-processes), and the link changes to the file when the chapter lands. - Each chapter cites the design documents by relative link and section, and the tree by path. It explains; it decides nothing. When a chapter and a design document disagree, the design document is right and the chapter is fixed.
- Plain language: the concrete file, call or check, and a term explained where it first appears and listed in the glossary, Appendix A, with a pointer to where it is explained. A term of operating systems, microkernels or Erlang is explained in Part I or in the Background box of the chapter that first needs it; a chapter never uses such a term unexplained, and never assumes that the reader met it elsewhere. UNIX in capitals; “license”.
- Each chapter opens with a “Background needed” line, naming the earlier chapters and boxes it depends on, so a reader can follow one thread (the network, say) without reading all that comes before. Its “Background boxes” are short titled passages, set apart, that explain a concept before the chapter first uses it; the outline lists them.
- Present tense throughout. Whether a thing is built is said in the chapter’s status box and in the sentence that describes it.
| Chapter | Title | File |
|---|---|---|
| 1 | Operating Systems from the Ground Up | 01-operating-systems.md |
| 2 | Microkernels and Capabilities | 02-microkernels.md |
| 3 | Erlang and OTP | 03-erlang.md |
| 4 | History and Goals | 04-history.md |
| 5 | Design Overview of erXos | 05-overview.md |
| 6 | Correct by Construction | 06-construction.md |
| 7 | One Source for Every Fact | 07-one-source.md |
| 8 | The Kernel Interface and Fiasco | 08-kernel-interface.md |
| 9 | Processes, Runtimes and Two-Level Scheduling | 09-processes.md |
| 10 | Domains: A Wall Around Some Processes | 10-domains.md |
| 11 | Authority and Security | 11-security.md |
| 12 | Memory and Budgets | 12-memory.md |
| 13 | Native Programs, Pipes and POSIX-lite | 13-native-programs.md |
| 14 | Files, Namespaces and Service Trees | 14-files.md |
| 15 | Devices and Drivers | 15-devices.md |
| 16 | Storage and Updates | 16-storage.md |
| 17 | The Network Stack | 17-network.md |
| 18 | WiFi | 18-wifi.md |
| 19 | Bluetooth | 19-bluetooth.md |
| 20 | The Command Shell and the UNIX Surface | 20-shell.md |
| 21 | The Live Machine | 21-live-machine.md |
| 22 | Displays, Windows and the Web | 22-display.md |
| 23 | Apps and Packages | 23-apps.md |
| 24 | Startup, Configuration and Shutdown | 24-startup.md |
| 25 | Checking the System | 25-checking.md |
| Appendix A | Glossary | a-glossary.md |
| Appendix B | Reading the Design Documents | b-design-documents.md |
| Appendix C | Building and Booting erXos | c-building.md |
| Appendix D | Dependencies and Licenses | d-dependencies.md |
| Appendix E | Lineage | e-lineage.md |
Contents
Part I. Background
Part II. Overview
- History and Goals
- Design Overview of erXos
- Correct by Construction
- One Source for Every Fact
- The Kernel Interface and Fiasco
Part III. Processes
- Processes, Runtimes and Two-Level Scheduling
- Domains: A Wall Around Some Processes
- Authority and Security
- Memory and Budgets
- Native Programs, Pipes and POSIX-lite
Part IV. The I/O System
Part V. Communication
Part VI. The User’s Machine
- The Command Shell and the UNIX Surface
- The Live Machine
- Displays, Windows and the Web
- Apps and Packages
Part VII. System Operation
Appendices. A. Glossary, B. Reading the Design Documents, C. Building and Booting erXos, D. Dependencies and Licenses, E. Lineage
About 710 pages in all, at about 400 words a page.
Part I. Background
The three chapters of this part teach what the rest of the book takes for granted, to a reader who can program and may know little about operating systems, microkernels or Erlang. They describe systems in general; erXos appears only where a chapter says that it does. Each is written to be skipped by a reader who already knows its subject (see the reading paths in Readers).
Chapter 1. Operating Systems from the Ground Up
About 27 pages. What an operating system is for and how one works: the kernel, the two modes of the processor, processes, virtual memory, files, devices and the boundary between a program and the machine.
Background needed. None. This is where the book starts; it assumes only that the reader can program.
Status. Background: no mark. Chapters 2 and 3 do the same for microkernels and for Erlang; the chapters after them say what erXos does.
- What an operating system is for. Sharing one machine among many programs, hiding the hardware behind a few simple ideas (a file, a process, an address space), and keeping programs from harming each other. The kernel, the programs, the libraries and the services, and which of them run with what power. Draws on: overview.md, design.md, The model.
- The processor’s two modes, and the way across. User mode and kernel mode, why a program cannot touch the hardware directly, and the three ways into the kernel: a system call, an exception such as a page fault, and an interrupt. The timer interrupt and preemption. The names an architecture gives its modes (AArch64’s exception levels) in one paragraph. Subsections: 1.2.1 three ways into the kernel; 1.2.2 a system call from start to finish; 1.2.3 the timer; 1.2.4 protection without modes (language-based safety as the alternative); 1.2.5 the modes on AArch64. Draws on: design-arch-aarch64.md.
- Processes, threads and scheduling. What a process and a thread are, the context switch, run queues, priorities and fairness, and cooperative against preemptive scheduling. Subsections 1.3.1 to 1.3.5 follow that order; 1.3.6 How a process ends covers exit, reaping and what the kernel reclaims. Draws on: design-scheduler.md.
- Virtual memory. Address spaces, pages, page tables and the TLB, a page fault walked step by step, demand paging, copy-on-write, memory-mapped files, and protection bits including writable-or-executable. Subsection 1.4.7 Who pays for memory: overcommit, running out of memory, and charging every page to an account. Draws on: design-kernel.md.
- Devices, interrupts, DMA and caches. Memory-mapped registers, device interrupts and the interrupt controller, drivers, direct memory access, why a cache can hide a device’s write from the processor and what an IOMMU does about a device that writes where it should not. Subsection 1.5.6 Finding the devices: device trees and bus enumeration. Draws on: design-device-dma.md.
- Files and block devices. The file as the central abstraction, file systems on block devices, directories, mounting, and what a path names. Subsection 1.6.5 Permissions, and the power of a user. Draws on: design-files.md.
- Talking and sharing. Locks, atomic operations and memory ordering in one page; pipes, sockets and messages; shared memory. Draws on: design-pipes.md.
- From power-on to the first program. Firmware, a boot loader, the kernel and the first user program. Draws on: design-platforms.md.
- When a part fails. What a crashing program, a kernel panic and a device that writes where it should not each cost in a conventional system: where the line between kernel mode and user mode runs decides how far a failure reaches, which the later chapters answer. A summary (1.10) and where to read next (1.11) close the chapter.
Background boxes. None: the chapter is the background. Further reading (the chapter’s writer checks each citation): Ritchie and Thompson, “The UNIX Time-Sharing System” (1974); McKusick, Neville-Neil and Watson, The Design and Implementation of the FreeBSD Operating System.
Figures. 1.1: the layers of a computer, from hardware to a program. 1.2: a system call from a program into the kernel and back. 1.3: a virtual address translated through a page table. 1.4: a device’s DMA write with and without a cache that is coherent with it.
Chapter 2. Microkernels and Capabilities
About 18 pages. The design family erXos belongs to: a kernel that does as little as it can, services in user mode, and authority held as unforgeable references.
Background needed. Chapter 1 (kernel and user mode, address spaces, system calls).
Status. Background: no mark. The sentences that say what erXos takes from L4 and Fiasco are in chapter 8.
- Monolithic kernels and microkernels. What a monolithic kernel holds (file systems, network stack, drivers), what a microkernel leaves out, and the trade: isolation of a failing driver against the cost of crossing between servers. Mach, QNX, MINIX and the L4 family as examples. Draws on: research-kernel-bases.md.
- Inter-process communication. Synchronous message passing between address spaces, the rendezvous, why L4 made it fast, shared memory beside it, and a file or network server as a user-mode process that clients call. Draws on: design-kernel.md.
- Capabilities. An unforgeable reference to an object together with the rights to use it; ambient authority (a user id, a path anyone can open) against held authority; the confused deputy; delegation, rights that only shrink, and revocation. KeyKOS, EROS, seL4, Zircon and Capsicum as examples. Draws on: research-prior-art.md.
- Security by construction. What a trusted computing base is, why a small one is checkable, and how a design where every access goes through a capability makes whole classes of mistake impossible to write, which is the idea the rest of the book returns to. Draws on: design.md, The model.
- The L4 family and Fiasco. L4 and its descendants, Fiasco and L4Re, the root task that creates every kernel object, and why erXos runs on Fiasco. Draws on: design-fiasco.md.
- What fails, and what a failure reaches. A microkernel does not stop servers from failing; it changes what a failure reaches, from the whole machine to one address space, and what stays unprotected.
- Design choices. The alternatives to a microkernel with capabilities (a hardened monolithic kernel, protection by language, access lists) and why erXos does not take them. A summary and where to read next close the chapter.
Background boxes. None: the chapter is the background. Further reading (the chapter’s writer checks each citation): Liedtke, “On micro-kernel construction” (1995); Dennis and Van Horn, “Programming Semantics for Multiprogrammed Computations” (1966); Saltzer and Schroeder, “The Protection of Information in Computer Systems” (1975); Elphinstone and Heiser, “From L3 to seL4: What Have We Learnt in 20 Years of L4 Microkernels?” (2013); Hardy, “The Confused Deputy” (1988); Shapiro, Smith and Farber, “EROS: A Fast Capability System” (SOSP 1999); the FreeBSD manual page capsicum(4).
Figures. 2.1: a monolithic kernel and a microkernel with servers, side by side, showing what a driver’s bug reaches in each. 2.2: a client calling a server by IPC. 2.3: a capability as a slot in a table, with its rights. 2.4: the confused deputy with an access list and with capabilities.
Chapter 3. Erlang and OTP
About 23 pages. Enough Erlang to read the rest of the book: the language, processes and messages, failure handled by supervisors, OTP’s behaviours, and the virtual machine that runs it.
Background needed. Chapter 1 (processes, scheduling); no Erlang.
Status. Background: no mark. A reader fluent in Erlang and OTP skips to section 5, the BEAM and ERTS, which is what erXos takes apart.
- The language in a sitting. Terms (numbers, atoms, tuples, lists, maps, binaries), pattern matching, immutable data, functions and modules, recursion in place of loops, and the book’s notation for specs and records. Draws on: style.md.
- Processes and messages. Lightweight processes, mailboxes,
spawn,!andreceive, selective receive, process identifiers and registered names, and why nothing is shared. Draws on: design-runtime.md. - Let it crash. Links and exit signals, monitors, trapping exits, supervisors, restart strategies and supervision trees: why a process that meets the unexpected stops and its supervisor starts it again. Draws on: design-runtime.md.
- OTP behaviours.
gen_server,gen_statem,supervisor,applicationand releases: a callback module around a generic process, and what each behaviour saves the programmer from writing. Draws on: design-runtime.md. - The BEAM and ERTS. Bytecode and the JIT (BeamAsm), one scheduler per processor, reductions, a heap and garbage collection per process, ports, NIFs and built-in functions, and the atom table. Draws on: research-erts-internals.md.
- Code loading and distribution. Two versions of a module at once, hot code loading, nodes, node names and cookies, and how a message crosses between nodes, which chapter 10 reuses across a domain’s wall. Draws on: design-runtime.md.
- Looking at a running system. Tracing, the shell, and the tools an Erlang programmer uses, as the baseline that chapter 21 extends to the whole machine. Draws on: design-live.md.
- Where the model stops. What Erlang’s isolation does not give on a stock system: native code, one shared memory, the atom table, shared stores, distribution with a cookie, and CPU time.
- How erXos uses these ideas. Every Erlang process an operating-system process; distribution across a domain’s wall; supervisors all the way down; a functional core with a behaviour at the edge; ERTS as a parts bin; hot code loading with a hash; and the chapter’s ideas at a glance. A summary and where to read next close the chapter.
Background boxes. None: the chapter is the background. Further reading (the chapter’s writer checks each citation): Armstrong, “Making Reliable Distributed Systems in the Presence of Software Errors” (2003); the OTP documentation shipped with the tree.
Figures. 3.1: a process and its mailbox. 3.2: a supervision tree and one restart. 3.3: the BEAM inside one operating-system process: schedulers on threads, processes on schedulers. 3.4: one message between domains.
Part II. Overview
Chapter 4. History and Goals
About 13 pages. Why build an operating system around Erlang, what erXos takes from earlier systems, and the goals and principles that shape it.
Background needed. Chapters 1 to 3, for the terms; a reader who knows operating systems, microkernels and Erlang starts here.
Background boxes. None; each term of Part I is linked to its chapter at first use.
Status. History, goals and principles: no mark. What runs today: Built, as README.md’s “Current status” records it.
- Erlang systems already behave like operating systems. Millions of isolated processes, preemptive scheduling, failure handled by links and supervisors, code replaced while running: all of it inside one process of another system, which sees none of it. The section states the problem erXos starts from: make the Erlang process the system’s process, and draw hardware protection around groups of processes only where trust requires it. Draws on: design.md, Context, overview.md.
- Thirty years of two-level scheduling. Scheduler activations, KSE, Solaris M:N threads and Windows user-mode scheduling all tried to let a user-level scheduler run on kernel threads, and all were abandoned. The section explains their common failure (a kernel re-entering user code at arbitrary points, for C code that may block anywhere) and the two facts that let erXos keep two levels: its runtimes are few and its own, and Erlang processes never block a thread. Draws on: research-hierarchical-scheduling.md, design.md, Context.
- The lineage. Four subsections give the sources that shaped erXos most: UNIX and FreeBSD for the surface; L4 and Fiasco for the kernel; Plan 9 for names and files; EROS, Safe Erlang and Zircon for authority and payment (EROS’s space banks, pids as capabilities, no ambient authority). A table (4.1) lists the rest with the idea each gave: QNX, Genera, Inferno, Singularity, SPIN, Midori, Genode and Barrelfish. Section 4.1.4 adds the earlier attempts to run Erlang below the application (Nerves, GRiSP, LING, HydrOS) and the gap they leave. Draws on: research-prior-art.md, README.md, Architecture.
- Goals and non-goals. The nine functional requirements (run OTP applications unmodified, protection, one system, generic runtimes, a live machine, Erlang drivers, users, storage and networking, UNIX programs where they pay) and the non-functional ones (performance within 1.2 times ERTS on a conventional system inside a domain, security on equal footing with correctness, reproducible images, portability). The non-goals matter as much: POSIX is a personality, never the base; no Linux binaries; no confidentiality between processes of one domain. Draws on: design.md, Requirements.
- Five principles. Erlang wherever practical; erXos owns OTP and changes it freely; simplicity; declarative where it simplifies; UNIX look and feel, honestly, with no skeuomorphism (nothing wears the name or shape of something it is not). The section shows each principle at work in one concrete choice that later chapters return to. Draws on: design.md, Decisions, overview.md.
- What runs today. A short account of the system as it runs: on
QEMU
virtand the Raspberry Pi 4, on Fiasco, with the kiosk, the command shell over SSH and native programs, and what in the rest of the book is designed and not built. No release names. Draws on: README.md, Current status. - What the approach costs, and how it fails. A runtime per domain costs memory, the system domain is critical, device isolation needs hardware, and the system’s isolation rests on Fiasco and on the root service. A summary and where to read next close the chapter.
Figures. 4.1: the lineage as a timeline: UNIX, L4, EROS, Plan 9, Erlang/OTP and the other sources of section 3, each with the idea erXos took from it.
Chapter 5. Design Overview of erXos
About 18 pages. The whole system in one chapter: seven concepts, what runs where, and a day in the life of the machine.
Background needed. Chapters 1 to 4.
Background boxes. Reading a sequence diagram, as the day in the life uses one. What a boot image, a manifest and a package are, one paragraph each, before the chapter names them.
Status. Built: a machine boots on Fiasco into the system domain,
which starts the storage, network and display domains and logs root in
at the console; shell commands and a pipeline of a native program and
Erlang commands; the system’s state under /proc; inspecting any
process; creating a domain and seeing a link report its end; memory and
threads held as records, and ending a domain ends everything charged to
it. Designed: memory limits that refuse a domain more, CPU limits per
domain, rights on handles checked on every use, a login that gives each person a domain of their own, and
toybox’s UNIX programs in the session’s /bin.
- Seven concepts, each with one owner. Process, domain, runtime, group, budget, capability and export, principal: what each is, who owns it, and why each has exactly one owner. The section opens with one process and the seven questions it answers. It defines the terms the rest of the book uses (connection, executor, own and outer budget, handle and generation, export table, session template, session, package) in plain words, as a reader’s map. Draws on: design.md, The model.
- What runs where. Fiasco, the only code in kernel mode, and everything else in user mode: the system domain (the root task, its root service, domain library and system services), Erlang domains, driver domains, and native programs in sub-spaces of a session. For each: what it runs, who trusts it, and what a failure reaches, from “the domain dies and links fire” to “the machine resets”. Draws on: design.md, Overview, design.md, Trust placement.
- A day in the life. The machine boots, the system domain starts the
domains the system manifest lists, a session starts, a process sends
to a service through a facet, a domain crashes and a link reports it,
and
find /tmp -type f | wc -lruns a native program and an Erlang command in one pipeline. Each step names the chapter that explains it and whether it is built or designed. Draws on: design.md, Overview. - Signature experiences. What a person can do on erXos that a conventional system does not offer: the shell is the operating system; the machine is an object you edit; a driver is a process, killed and restarted; a wall around some processes; UNIX tools over live Erlang state; a user with a domain of their own (Designed). Each is shown as a short transcript or sketch, marked built or designed. Draws on: design.md, Signature experiences, DEMOS.md.
- Four decisions every chapter assumes.
liberxis the only issuer of kernel calls; every domain is a node; system code is Erlang, drivers included; formats are erXos’s own. Each with its one-line reason. Draws on: design.md, Decisions. - Tensions and alternatives. Where the design pulls against itself (performance against isolation, Erlang against C, a trusted base above the kernel, a familiar surface over a different model, shared services and shared CPU) and the large alternatives rejected: Linux underneath, POSIX as the base, one runtime for the whole machine, a verifier instead of hardware protection. Draws on: design.md, Tensions, design.md, Alternatives considered, design.md, Why no verifier.
Figures. 5.1: the layer diagram of design.md, Overview, redrawn: domains, sub-spaces, connections, the system domain, the kernel. 5.2: the seven concepts and their owners as a table-diagram. 5.3: the day in the life as a sequence diagram, each step tagged with its chapter and status.
Chapter 6. Correct by Construction
About 18 pages. How erXos is designed: properties that hold because the wrong state cannot be written, not because every case was remembered.
Background needed. Chapters 1 to 3 (chapter 1 for DMA and caches, chapter 2 for capabilities, chapter 3 for Erlang).
Background boxes. Pure functions and side effects, with an Erlang example. Model checking, fuzzing and static checks in a paragraph each. Why a device’s write can bypass a cache, enough for the worked example; chapter 15 explains it in full.
Status. The method is Built and runs through the tree: pure Erlang
cores that return {State, Actions}, with edges that perform the
actions; the device NIF’s pure cores, the independent check of every
plan and the one effect loop; the funnel; the chokepoint build checks,
each with a seeded violation; and the stack proof of the C17 code that
Fiasco links. Of the properties in section 6.2, three are Designed: the
memory wall of a domain’s budget, Fiasco’s refusal of a mapping that is
writable and executable at once, and rights on handles checked on
every use. The rest are Built, with the limits the
chapter states.
- Open-ended enumeration is the smell. A correctness argument that depends on remembering every caller, path or exception breaks with the next caller. The section works through the two questions that replace such a list (“what is this list protecting?” and “what would make the list empty?”) and the three ways out: make the illegal state unrepresentable, make the invariant a derived fact nothing can destroy, make one funnel the only way to the operation. Draws on: design.md, Invariants, design.md, Correct by construction in code.
- The properties erXos holds by construction. A tour of the list
as it holds on Fiasco: only the root service creates kernel objects;
a destroyed object’s handles, and a handle used after its own
close, answer
ERX_ESTALEand never name a successor; a domain’s memory is charged to its own budget, and the wall that refuses a charge past it is Designed; a pid is a name, not an authority; no process acts on a message from another domain that its protocol’s owner did not admit; every layer agrees on the ABI because all are generated from one file; the root service refuses a mapping that is writable and executable at once, and Fiasco’s own refusal of one is Designed. For each, the one mechanism that makes it hold, and the chapter that shows it. Draws on: design.md, Invariants, design-fiasco.md, Contract, design-domains.md, Invariants. - Functional core, effects at the edge. Every decision that can be a pure function is one: a state machine that returns the next state and a list of actions, a plan validated whole before anything is applied. The section shows the shape in C and in Erlang, why the opposite shape, a caller stringing helpers together in the right order, keeps producing sequencing bugs, and why variants beat flags. Draws on: design.md, Functional core, effects at the edge, style.md, Functional core.
- A worked example: the device DMA plan. A preview of chapter 15. A driver’s C part never pokes a device freely: its pure core, a form for each kind of device, produces a plan, and one independent check reads the whole plan before anything touches the hardware. Every device address stays inside the queue’s DMA memory, and no region ever has two writers. Cache maintenance and barriers are derived from the plan by one effect loop, so no driver can forget one, and a build check keeps them there. Draws on: design-device-dma.md, Calls, observations and plans.
- What construction cannot say. A structure proves where an operation can be done from, not that the code allowed to do it does it only when it should. Model checking, litmus tests, CBMC and fuzz targets cover the rest, and seeded faults prove each check catches what it guards. Draws on: design-verification.md.
- Only X may do Y: chokepoints as build checks. A rule such as “only
the device shim touches a device’s registers” or “only
liberxissues kernel calls” is a check that fails the build on a new site, and each check has a test that plants a violation and expects the failure. The section explains why a type or visibility boundary comes first (opaque types,staticfunctions, unexported Erlang functions) and a chokepoint check covers only what such a boundary cannot say. Draws on: design.md, Chokepoint checks,tools/lib/erx/chokepoint_rules/. - Bounded stacks. The C17 code erXos links into Fiasco recurses only through tail calls the compiler must turn into jumps, and each build computes its worst-case stack depth from the call graph. Parsers outside the kernel bound their depth too. Draws on: design.md, Bounded stacks.
- One interface below every runtime. Every runtime and program
reaches the kernel through
liberx, every shared format is erXos’s own, and none of Fiasco’s data model reaches Erlang code or a user, so a component below the interface can change without anything above it moving. Draws on: design.md, Decisions. - When a construction fails, and the choices. What each failure looks like (a chokepoint violation, a stack proof failure, a refused DMA plan, a core that acts after stopping, a malformed frame), the failures construction does not prevent, and the alternatives: careful review, checks at run time everywhere, a verified kernel, a language that enforces more.
Figures. 6.1: a pure core and its effect loop: request in, either “observe these reads” or “decide: this plan and the next state”, the effect loop performing reads or the checked plan. 6.2: the “by construction” list as a table of property, mechanism and chapter. 6.3: a chokepoint check with its seeded violation.
Chapter 7. One Source for Every Fact
About 13 pages. Declarative description in erXos: facts stated once, and everything that follows from them generated or checked.
Background needed. Chapter 6; chapter 3 for Erlang terms.
Background boxes. What an application binary interface is and why two languages must agree on one. Code generation from a specification, against writing the same table twice. Datalog in a page: facts, rules, queries. How a chip vendor describes registers (CMSIS-SVD).
Status. Built: the ABI file and its generators; register accessors generated from the hardware sources; the theme and its contrast rules; the errno and signal tables; the system manifest and its build-time check; the declared services and inspection schema. Designed: the ABI file declaring every kernel call (it declares one); budgets that refuse a charge at run time; per-domain CPU limits; the ADMA2 descriptor form for the SD controller.
- Parallel copies, and the rule that replaces them. Two places that must change together become a list of places to update, and the list is an enumeration that someone forgets. The section states the rule (every decision, table and format has one implementation; C, Erlang and Ruby share a table only by generation) and its companion: a fact the build can compute is never written by hand and then checked. Draws on: design.md, One source for the ABI and the hardware, design-generation.md, Sources and outputs.
- The ABI file. Status codes, limits, rights and every fixed binary
layout (rings, reader pages, frames, boot information, descriptors)
are declared in one text file, and Ruby generators produce the C
headers, the Erlang modules and the Ruby tables. The section walks
one line from text to each output, and one layout to its offsets and
the compiler’s word on them.
Draws on: design-abi.md, Source grammar,
design-abi.md, A complete declaration example,
abi/erx_abi.txt. - The rules every spec follows. State facts and derive the rest; close every set, strictly; no expressions; fail whole; sorted and deterministic; generated files are not checked in. Tables that must match someone else’s, such as FreeBSD’s errno numbers.
- The hardware has one source too. Device registers come from CMSIS-SVD files and the processor’s own registers from Arm’s machine-readable data; accessors are generated and no register bit is packed by hand. The section follows one register from XML to a driver. Draws on: design-generation.md, Hardware and terminal sources, design-generation.md, Register inputs.
- One theme, three languages. The desktop’s type roles, spacing, radii, shadows and palette are one declaration with contrast rules the build checks, and the Erlang, C and TypeScript sides are generated from it. Draws on: design-desktop-programs.md, The theme: one declaration.
- The descriptor formats behind the DMA check. Each kind of device’s descriptors are declared once, and the device NIF’s forms and the check read the same declaration.
- The system manifest: the machine’s configuration, checked. One file lists every configured domain with its principal, image, budget, devices by unit, facets and introductions. A build check refuses a manifest that breaks a property the security argument rests on, and each property has a seeded test that plants one violation and expects its diagnostic. Draws on: design-system.md, The manifest, design-abi.md, System-manifest representation.
- Declared services and inspection paths. Every cross-domain facet is
declared in
abi/serviceswith the design section that justifies it, and every value the system observes is an inspection path declared inabi/inspection.txt; a build check refuses what the tables do not list. Draws on: design-generation.md, Service interfaces, design-generation.md, Inspection and diagnostics. - When a source is wrong. A malformed spec, a well-formed but wrong value, and a wrong configuration, and what catches each.
- Where logic languages fit, and where they do not. Barrelfish’s system knowledge base and the BOOM project described systems in a logic language; erXos’s principles allow Datalog where it turns a property into a checked constraint, never where it adds complexity. The section says where that holds and where plain generators win. Draws on: design-verification.md, Focused tool evaluations, research-os-ideas.md, Declarative OS services from BOOM and Dedalus, research-os-ideas.md, Barrelfish and declarative hardware management.
- Design choices. Headers written by hand and tested, the C header as the source, a general interface language, checking the configuration at boot: why each was not chosen.
Figures. 7.1: the generation graph: the ABI file, the hardware sources, the theme and the inspection schema on the left, the generators in the middle, every generated header, module and table on the right. 7.2: one declaration and its C, Erlang and Ruby outputs side by side. 7.3: the manifest check as a pipeline: reader, normalized facts, properties, diagnostics, with one seeded violation traced through.
Chapter 8. The Kernel Interface and Fiasco
About 23 pages. The kernel’s services are its interface: objects, handles, threads, memory, IPC and interrupts. Fiasco, the L4 microkernel, is the kernel behind it, and the system domain’s root service builds every domain over it.
Background needed. Chapters 1 to 3; chapter 2 for capabilities, IPC gates, the UTCB and the root task. Chapters 5 to 7 help but are not required.
Background boxes. A capability table recalled from chapter 2. Fiasco, L4Re and the root task in a page.
Status. Built: liberx with its Fiasco backend and its hosted
backend, held to one test set; the root service, its request decoder
and its tables; handles as entries of the root’s per-task tables, each
carrying its object’s generation; memory objects of seven kinds,
mappings and the root’s page pool; doorbells and device lines; thread
creation, umtx waits and parking; fault gates and exit records; the
machine’s end on a fault of the root task; budgets as records that end
what is charged to them. Designed: priority bands and per-domain CPU
sets; signals through vCPU upcalls; DMA spaces behind an IOMMU; rights
on handles checked on every use; and the system-call
fuzz campaign.
- What the kernel knows, and what it does not. Spaces, threads, memory objects, IRQ objects, gates and budgets; nothing of Erlang. The section explains why the line falls there. Draws on: design-fiasco.md, Contract, design.md, Trust placement.
liberx: the one issuer of kernel calls. Every kernel call in every runtime and program goes throughliberx, whose numbers come from the ABI file; behind it sit the Fiasco backend, whose calls are local or service calls, and the hosted backend, which pass one test set. A call the backend cannot carry out fails withERX_ENOTSUP, and a value it cannot measure is reported as unsupported, never invented. A build check holds the seam. Draws on: design-runtime.md, liberx, design-runtime.md, The liberx interface.- The root service: who holds what. One thread of the system domain is the only creator of kernel objects and the only writer of page tables, so its tables need no lock. What a domain holds, the request decoder as the trust boundary, and one request followed end to end: a domain’s crash area. Draws on: design-fiasco.md, The construction, design-fiasco.md, Who holds what, design-fiasco.md, The call table.
- Handles and destroying an object. A handle is an entry in the
caller’s table in the root service; the entry records its object’s
generation, and the handle’s value carries the entry’s own, so a
destroyed object’s handles and a handle used after its own close
both answer
ERX_ESTALEand never name a successor. Free entries are reused first in, first out. Entries record rights, and only the creator may destroy; rights narrowed per creation kind and checked on every use, and interrupt objects rung straight through Fiasco, are Designed. Handles flow only down, from a space’s creator into it. Draws on: design-fiasco.md, Destroying an object, design-fiasco.md, Passing a handle. - Threads, waits, gates and faults. Threads and the reaper, one semaphore per thread for umtx waits and parking, gates that carry fault and exit records, how a fault reaches the right handler, what a fault of the root task does to the machine, and signals through vCPU upcalls (Designed). Draws on: design-fiasco.md, Threads, waits and exit, design-fiasco.md, vCPU upcalls on Fiasco.
- Memory objects and mappings. One page, one address in the root’s window; reserve, commit and map through one range table; the kinds of memory object; who pays.
- Interrupts, doorbells and devices. One dispatcher per task for device lines and doorbells alike, and what a driver domain holds for a device. Draws on: design-fiasco.md, Interrupts and readiness.
- What Fiasco gives, and what it does not. Isolation between domains rests on Fiasco and on the root service’s decoder and placement code; where no IOMMU confines a DMA-capable device, also on the domain that drives it. The section states plainly what a hostile domain could do and what fails when, as a property of the system. Draws on: design-fiasco.md, Security on Fiasco, design-fiasco.md, What a hostile app can do.
- Design choices. Runtimes talking to Fiasco directly, a root of several threads, Fiasco’s own interface, a refusal as a best effort: why each was not chosen.
Figures. 8.1: the kernel interface: runtimes and programs above,
liberx and its generated calls, the Fiasco and hosted backends below
it. 8.2: the root service between the domains and Fiasco: requests in,
kernel objects placed in each domain’s tables. 8.3: a handle’s
life: created, passed, stale or closed, reused.
Part III. Processes
Chapter 9. Processes, Runtimes and Two-Level Scheduling
About 15 pages. Every Erlang process is a process of the operating system; Fiasco shares the processors among threads, and each runtime schedules its own processes on its own threads.
Background needed. Chapter 8 for liberx, the root service,
threads and waits; section 3.5.2 for schedulers and reductions.
Background boxes. A C library, and musl.
Status. Built: ERTS in user mode in every domain, with its Erlang
processes as the system’s one kind of process; Fiasco running every
thread but the watchdog at one priority; executors and workers and how
they park and wake; the event table that turns a doorbell into work;
the system domain on the interpreter, with BeamAsm available to other
domains; musl under ERTS, with FreeBSD’s errno and signal numbers;
hosted mode; ps over Erlang processes; and erx:trace. Designed:
priority bands, CPU sets and thread limits; refusing an executor every
other blocking wait; refusing code that is not installed; one shared
copy of common code; native programs in ps; and services that admit
each client within a share of their time.
- The Erlang process is the system’s process. What a user sees
(
pslists Erlang processes) and what a programmer sees (OTP applications run unmodified), and the two levels in one picture: Fiasco schedules threads, each runtime its processes. Draws on: design.md, The model. - How Fiasco shares the processors. Every thread but the watchdog runs at one priority, round robin. Bands, CPU sets and thread limits from the manifest are designed and not built; a band orders domains and measures no share of CPU time. Draws on: design-fiasco.md, Scheduling, design-fiasco.md, Bands, design-fiasco.md, Threads, bands and stops.
- The runtime contract. Executors run Erlang processes and never block except to park; workers may block. The lost wakeup, and the epoch that closes it; park and wake on Fiasco through one semaphore per thread, the dispatcher and the group’s ready list; what a runtime does not get from the kernel. Draws on: design-scheduler.md, The runtime contract, design-scheduler.md, Park and wake, design-fiasco.md, The park protocol.
- ERTS as a parts bin. One mechanism per job; the inside of ERTS on
erXos, with the adapter as the one part that includes
liberx; the scheduler’s sleep in code; the event table, which gives readiness without file descriptors. Draws on: design-runtime.md, One mechanism per job, design-runtime.md, The adapter, design-runtime.md, The event table. - Code: the interpreter, BeamAsm and shared code. Three runtime images, chosen by facts fixed at boot; the system domain’s code service loads modules by hash; refusing code that is not installed, and one shared image of common code, are designed and not built. Draws on: design-runtime.md, BeamAsm, design-runtime.md, Code in the system domain, design-shared-code.md, The central construction.
- The C library seam and hosted mode. musl over
liberxthrough one seam, and the same runtime as a macOS process under the sanitizers and a debugger. Draws on: design-runtime.md, The C library seam, design-runtime.md, Hosted mode. - Two schedulers you can watch as one. One timeline from OTP’s trace and the root service’s trace ring, and a worked example of three domains sharing a service. Draws on: design-live.md, Tracing, design-fiasco.md, The trace ring, design-scheduler.md, A worked example.
- What happens when a part fails. A looping process, a looping NIF or a blocked executor, a crashed runtime, a stuck machine: each stays where it started. Draws on: design-runtime.md, Crash handling, design-fiasco.md, The watchdog.
- Design choices. Every Erlang process a kernel thread, scheduler activations, shares instead of priorities, lending time to a service, stock ERTS on a POSIX layer: why each was not chosen.
Figures. 9.1: the two levels: Fiasco schedules threads, each runtime
switches among its own processes on its executors. 9.2: park and wake,
drawn for one epoch. 9.3: ERTS on erXos: the adapter and musl above
liberx, and liberx above Fiasco and the root service.
Chapter 10. Domains: A Wall Around Some Processes
About 19 pages. The unit of isolation: a domain is an address space with its own runtime and an Erlang node of its own, and processes in different domains link, monitor and message each other as Erlang processes always have.
Background needed. Chapter 1 for address spaces and shared memory;
chapter 2 for capabilities and the root task; chapter 3 for processes,
links, nodes and the external term format; chapter 8 for liberx, the
root service, handles and doorbells. Chapter 9 helps but is not
required.
Background boxes. Address spaces, recalled. Rings and doorbells. Frames, recalled.
Status. Built: domains, each with its own ERTS in its own address
space and its own Erlang node; connections of two lanes with doorbells;
the funnel; export tables; the {erx_origin, D, Msg} envelope; links,
monitors, exits and spawns across domains; {domain_down, D, Info} at a
domain’s end; the domain service, the domain library and the root
service creating and ending domains; a quota of node names per
principal. Designed: budgets that refuse connection memory past their
limit; handles that carry rights; a spawn right for a domain’s creator
at boot; gen_server and gen_statem servers that take calls from
other domains directly; a decoder in the system domain that creates no
atom from another domain’s frame.
- A domain is a node. What a domain holds; domain ids, creations and pids, so a pid carries its owner domain; where the picture of Erlang distribution inside one machine stops. Draws on: design-domains.md, Domain ids, pids and references, design-domains.md, Atoms, code, ETS and names.
- Connections, lanes and doorbells. A lane: a ring the producer alone writes, a reader page the consumer alone writes, and a bell. Publishing and waking, checked by herd7 and TLA+; who makes a connection; what happens when a lane is full. Draws on: design-domains.md, Connections, design-domains.md, Introduction and connection down, design-domains.md, Flow control.
- The funnel. The receiver’s one reader of its lanes copies a frame
out first, checks it in order, tells a malformed frame from one over
a limit, and is checked against
binary_to_term/1. Draws on: design-domains.md, The funnel,liberx/erx_funnel.c. - Exports: a pid is a name, not an authority. Why a pid cannot be the authority; the three writers of a name’s holders; what a peer cannot do; Safe Erlang compared. Draws on: design-domains.md, Exports, design-domains.md, Alternatives considered.
- The origin envelope. Every message from another domain arrives as
{erx_origin, D, Msg}, with no exception, and only each protocol’s owner takes the envelope off its own shapes. Draws on: design-domains.md, The peer. - One message, end to end. One round trip between the system
domain and a domain named
web, through every part of the chapter. - Links, monitors, exits and spawns across a wall. A process
exits; killing a process in another domain; spawning into another
domain; a domain dies and its peers see
{domain_down, D, Info}. Draws on: design-domains.md, Links, monitors and exits, design-domains.md, Spawning. - The system domain and the domain library. Three layers: the domain service in Erlang, the domain library in C, the root service; creating and ending a domain. Draws on: design-system.md, The domain library, design-system.md, Creating a domain, design-system.md, Ending a domain.
- What fails, and what it reaches. No failure in the chapter reaches past the domain or connection it began in, and the limits of the system as built are stated plainly.
- Design choices. A router between domains, a table kept by the sender, revocable pid exports, synchronous calls between domains: why each was not chosen.
Figures. 10.1: two domains and their connection, each side one memory object only its own domain writes. 10.2: a pid’s parts and the export lookup. 10.3: one round trip between two domains. 10.4: a sandbox crash, from fault to the linked process.
Chapter 11. Authority and Security
About 20 pages. Who may do what, and why: the threat model, trust placement, the four forms of authority, principals, machine code, keys and untrusted text, and what a hostile app could do on the system as built.
Background needed. Chapter 2 for capabilities, the confused deputy and the trusted computing base; chapter 8 for handles, the root service and what Fiasco gives; chapter 6 for exports, the funnel and the origin envelope.
Background boxes. Threat models. Branch-predictor attacks. Signatures and a key vault. Terminal escape sequences.
Status. Built: the walls between domains, which rest on Fiasco and the root service; handles, exports, facets and file grants; the refusal of a mapping that is writable and executable; the three runtime images; and the escaping of untrusted text. Designed: rights on handles, budgets that refuse, CPU bands, devices behind an IOMMU, a safe decode of other domains’ messages in the system domain, login with a passphrase, and the key vault.
- The threat model. Adversaries and assets, the untrusted inputs, and what the threat model leaves out, branch-predictor attacks among it. Draws on: design.md, Threat model, design.md, Residual risks.
- Trust placement. Two things keep domains apart, Fiasco and the root service; the trusted base; why the system domain may stop the machine; why no verifier. Draws on: design.md, Trust placement, design.md, Why no verifier.
- Four forms of authority. Handles, exports, facets and file grants: each has a holder and a checker, never the same party.
- Handles: authority over kernel objects. What a handle protects, what it does not (rights, and a number reused after its own close), and the design that closes both. Draws on: design-fiasco.md, Rights.
- Exports and the funnel: authority over processes. The export table and the funnel; a missing right looks like a dead process; the envelope. Draws on: design-domains.md, Authority, design-domains.md, The funnel.
- Facets: authority over services. One grant, one grantee, one process; what every facet checks; why a process and not a list, and how that answers the confused deputy. Draws on: design-domains.md, Facets.
- File grants: authority over files. A namespace of grants, and narrowing it for a program. Draws on: design-files.md, Who a grant is for.
- A worked example: who may set the clock. The grants, one set followed end to end, three refusals and one gap. Draws on: design-time.md, Who may set it.
- Principals, sessions and quotas. What a principal holds, and which limits hold and which are only records. Draws on: design-domains.md, Principals, sessions and quotas.
- Machine code. No mapping writable and executable at once, and who may make code; a page can still be writable through one mapping and executable through another. Draws on: design.md, Machine code.
- Keys, the vault and cryptography. OpenSSL in two of the three runtime images, and the vault with its key facets, which is designed and not built. Draws on: design-crypto.md, Three runtime images, and OpenSSL in two of them, design-system.md, The vault.
- Text from untrusted sources. One escape per sink, generated from one table, and what is not escaped: a program’s own output. Draws on: design-live.md, Escape rules.
- Trusted code only, and what a hostile app can do. The boot banner’s promise, a table of what is built and what is designed, and what a hostile app could do on the system as built. Draws on: design-fiasco.md, What a hostile app can do.
- When a guard fails. Each check fails visibly and reaches only what it guards.
- Design choices. User ids against held authority, checks at the receiver against a broker in between, escaping at the sink against cleaning at the source.
Figures. 11.1: the trust map, as designed. 11.2: a facet between a domain and a service. 11.3: the escape at each text sink.
Chapter 12. Memory and Budgets
About 13 pages. Each domain’s memory is charged to its budgets, and ending a budget ends everything charged to it; the root service alone maps memory, from one pool, through one range table.
Background needed. Chapter 1 for virtual memory (section 1.4); chapter 8 for the root service, handles, memory objects and reserve, commit and map. Chapter 5 introduced budgets.
Background boxes. Page tables and faults, recalled. Space banks. Copy-on-write, recalled.
Status. Built: the page pool and the range table, with commit, map, protect and revoke over Fiasco, shared objects and the JIT alias; budgets that end everything charged to them, each domain’s own and outer budgets, and admission arithmetic; sub-spaces that end with their creator; eager copies; one machine-wide figure reported as partial. Designed: budgets as walls over kernel memory, pages and the root’s tables; rights recorded per handle; per-budget reports; a refused map that fails only the domain’s call; the mapper for shared file mappings.
- Budgets as space banks. A budget is an account with a guarantee and a cap, a wall or elastic; ending a budget ends what was charged to it. Draws on: design-fiasco.md, Memory and budgets.
- Own and outer budgets. What a domain alone maps dies with it; what others map for it, each connection through a child budget, and its crash area outlive it. Draws on: design-system.md, Ending a domain.
- The pool. One pool of pages in the root’s window, and the contiguous pool for devices. Draws on: design-fiasco.md, The pool.
- Walls: what is built, what is designed. Accounts without walls today, and the three parts a wall needs. Draws on: design-fiasco.md, Memory and budgets.
- The range table. Three facts (runs, mapping records, the page index); plans, then effects; one table of steps; a mapping keeps the rights it was made with; trimming never meets sharing. Draws on: design-fiasco.md, The range table, design-fiasco.md, Protection.
- One commit, end to end. One page of one ERTS commit through every layer.
- Sub-spaces and shared objects. A sub-space spends its creator’s memory; shared objects and the JIT alias. Draws on: design-fiasco.md, Spaces, parents and exit records, design-fiasco.md, Shared objects and the JIT alias.
- Deleted memory. What a destroy does to a mapping, and why no server maps what a client can delete: the session’s mapper for shared file mappings, designed and not built. Draws on: design-files.md, Shared mappings.
- Reporting a budget, and what is not charged. What is reported, as exact, partial or unsupported, and the memory nothing charges. Draws on: design-fiasco.md, Reporting a budget.
- When memory runs out. The pool, a domain’s wall, Fiasco’s kernel memory: what each running out does.
- Design choices. Commit at once or fill on demand, one mapper or a pager per domain, walls or elastic quotas everywhere, eager copies.
Figures. 12.1: own and outer budgets of two connected domains. 12.2: a private page and a mapping in a task’s range table.
Chapter 13. Native Programs, Pipes and POSIX-lite
About 16 pages. UNIX programs where they pay: a native program is a port of its session, running in a sub-space with only what its starter handed it, and pipes live in user space.
Background needed. Chapter 1 for exit status and signals, file descriptors, pipes and memory ordering; chapter 3 for processes, monitors and ports; chapter 5 for sessions and sub-spaces; chapter 8 for fault gates and doorbells.
Background boxes. Executable files. Signals and job control. POSIX.
Status. Built: a C program from the session’s release runs in a
sub-space of its session and talks to one Erlang process, its handler;
pipes in user space, made by the session’s pipe server; programs in
command-shell pipelines beside Erlang commands; files, poll, dup
and pipe; a kill that reaches only the program itself; FreeBSD’s
signal and errno numbers. Designed: programs starting programs
(posix_spawn, wait), executable files as images, catchable signals
through vCPU upcalls, job control that pauses a program, terminal input
and sockets for programs, toybox, the second tier, and the POSIX-lite
personality as a whole.
- A program is a port. What a program consists of (anchor, image, budget, channel, table, data area) and why a port. Draws on: design-programs.md, A program is a port.
- The channel and the handler. The channel and its credit; the handler, which never waits on the program or on a request’s work; the command server; what a program can start. Draws on: design-programs.md, The channel, design-programs.md, The handler, design-programs.md, What a program can start.
- Starting a program. From the shell’s line to
main. Draws on: design-programs.md, The cost of a program. - Pipes in user space. Why not in the kernel; one ring per writer; shared memory a stranger can write; ends and their holders; doorbells and lost wakeups, checked by herd7. Draws on: design-pipes.md, The ring, design-pipes.md, The lock page, design-pipes.md, Ends and their holders.
- A pipeline, end to end.
echo piped text | rcio upcase, through shell pipes, pumps and rings to$status. - Signals, job control and FreeBSD’s numbers. What a program can
do with a signal;
killreaches only the program itself;Ctrl-Cand the session’s job control; signals through vCPU upcalls, designed and not built; one signal numbering. Draws on: design-programs.md, Signals in liberx, design-programs.md, Job control. - The POSIX-lite personality. What a recompiled program gets, why
posix_spawnand notfork, and toybox beside the Erlang file utilities; designed and not built as a whole. Draws on: design-programs.md, The POSIX-lite personality, design-programs.md, toybox. - The second tier.
fork,exec, shared libraries and terminals for programs that cannot be changed, without giving up the first tier’s rules; designed and not built. Draws on: design-programs.md, The second tier. - When a part fails. Every failure of a program ends in one exit status, given to its starter.
- Design choices. A program as a domain, Linux binary compatibility, a handler per program.
Figures. 13.1: a session with two programs, their handlers and a
pipe between them. 13.2: the life of echo piped text | rcio upcase.
Part IV. The I/O System
Chapter 14. Files, Namespaces and Service Trees
About 36 pages. Plan 9’s idea, kept whole: each domain sees files through its own table of file servers, a path is the visible face of a capability, and every server speaks 9P2000.L.
Background needed. Chapters 1, 2, 10 and 11.
Background boxes. The UNIX file model: paths, mounts, descriptors. Plan 9 in a page: per-process namespaces and 9P. RAM file systems.
Status. Built: per-domain namespaces, file servers in other domains
reached through file grants and 9P2000.L, the erx_fs behaviour, one
path resolver for every runtime, a program’s namespace narrowed from its
starter’s, stream files whose waiting reads a keeper holds, /tmp on a
RAM file system in Erlang, /proc as a read-only view of the inspector,
the host link’s 9P bridge, and the session’s export. Designed: entries
on the export in a program’s table, shared mappings of files, a
sysctl(3) reader for programs, and an app’s own tree at /svc/NAME.
- A path is the face of a capability. There is no global root: a domain’s namespace is a private table from paths to grants, and each file server roots a grant at its subtree, so a path never reaches a server the domain does not hold. Draws on: design-files.md, Namespaces, design-files.md, Resolving a path.
- One file protocol. 9P2000.L between domains, its codec fuzzed;
the
erx_fsbehaviour that every Erlang file server implements; Erlang clients throughprim_fileand native clients throughliberx. Draws on: design-files.md, The file protocol, design-files.md, Serving: theerx_fsbehaviour. - Narrowing for a program, and shared mappings. What a program’s namespace holds and how its starter narrows it; how a session maps one file into two programs with one memory object and writes back only the pages the kernel reports stored. Draws on: design-files.md, Narrowing for a program, design-files.md, Shared mappings.
/tmp: a file system in Erlang. A small, complete file server, as the reader’s first look at theerx_fsbehaviour from the inside. Draws on: design-tmp.md.- State is a file to read, control is a file to write. Plan 9’s rule
applied to services: what the system observes is an inspection path
shown at
/proc; what an app says about itself or what someone commands is a file tree its owner serves, with astatusto read and actlfor one-shot control. Where FreeBSD hassysctl, system values are one read-only tree with many views. Draws on: design-app-trees.md, The tree, design-app-trees.md, Commanding:ctl, design-live.md, The /proc tree. - Familiar only where true. The section closes with the review
question every familiar surface faces (“what does this promise that it
does not do?”) applied to
/proc,/devand/etc/passwd. Draws on: design-app-trees.md, What FreeBSD does.
Figures. 14.1: two domains’ namespaces over the same file servers,
showing what each can and cannot reach. 14.2: a 9P2000.L walk, open and
read between domains. 14.3: an app’s /svc/NAME tree beside /proc.
Chapter 15. Devices and Drivers
About 52 pages. A driver is a supervised Erlang process over a thin device shim; the shim’s C is a functional core whose every plan is checked before it touches the hardware.
Background needed. Chapters 1 (interrupts, DMA, caches), 2, 9 and 10.
Background boxes. DMA, cache coherence and IOMMUs, before the DMA check. Memory-mapped I/O, interrupt controllers and message-signalled interrupts. PCIe in a page. USB in a page: hubs, descriptors, transfers. Device trees and firmware. virtio.
Status. Built: drivers as supervised Erlang processes (virtio devices
on QEMU virt; the SD card, GENET Ethernet and the firmware
framebuffer on the Raspberry Pi 4), the device NIF with the virtio and
GENET descriptor forms and the check of every DMA plan, the device tree
reader’s DMA facts, the unit rule, the driver-control facet and both
drivers’ restarts. Built, and unexercised because no PCIe domain runs:
the give of a PCIe function. Built as pure code with host tests: the
PCIe enumerator’s parts, USB’s cores and decoders, the TRB descriptor
form, and the check of the Pi 4’s header map. Designed: the running PCIe
and USB domains, MSI delivery, virtio over PCIe, the SD controller’s
ADMA2 form, the Pi 4’s header drivers and /proc/sensors.
- A driver is a process. Kill the network driver’s process mid-transfer from the shell; its supervisor restarts it, the device is reset, and the shell never notices. The section explains why drivers are Erlang (principle 1), what stays in C (the device shim and the per-byte primitives) and the driver table that lists every native exception with its reason. Draws on: design-system.md, Drivers, design-system.md, The driver table, design.md, Drivers and DMA.
- The device NIF. One module,
erx_device, through which a driver reaches registers, descriptors, DMA buffers and interrupts. Its C parses nothing a device wrote; each descriptor form (the virtio split ring, GENET’s register ring, ADMA2 tables) is a pure round function, and one effect loop performs reads and checked plans. Draws on: design-runtime.md, The device NIF, design-device-dma.md, The descriptor forms. - DMA by devices that are not cache-coherent. The Pi 4’s Ethernet and SD controllers read memory without looking in the CPU’s caches. Memory changes hands in whole cache-writeback granules, and at any moment either the CPU or the device is a region’s writer, never both; cache cleaning, invalidation and barriers follow from that handoff in one sequencer; and the device writes over zeros, so a completion never returns an earlier request’s bytes. Draws on: design-device-dma.md, The construction: handoffs of granules, design-arch-aarch64.md, Cache maintenance.
- The plan and its check. The method at its clearest. A
descriptor form’s core returns either the reads it needs or a decision:
a plan and the queue’s next state.
erx_device_dma_checkchecks the whole decision before any effect: the state change is one the call allows, every write lands in a region the CPU owns, every device address and the bytes reachable from it lie inside the domain’s DMA memory as the device sees it, and every register written is one the NIF owns. A plan that fails is a bug in the core, and the call isbadargwith nothing applied. On a platform with no IOMMU, which QEMUvirtand the Raspberry Pi 4 both are, this check is what stops a driver bug from aiming a device at kernel memory; the driving domain still counts in the trusted base. Draws on: design-device-dma.md, Calls, observations and plans, design-device-dma.md, Device addresses and reach, design-device-dma.md, Register ownership,otp/erts/emulator/nifs/erxos/erx_device_dma.c. - Facts the firmware owns. The device tree the firmware hands over is read where it left it: coherence and DMA ranges come from it, and a device whose facts are absent or malformed gets no DMA memory rather than a guess. Draws on: design-device-tree.md, design-fiasco.md, Devices.
- Devices that share. A register page or an interrupt line shared by two devices: the unit rule, and the console. Draws on: design-shared-devices.md, The unit rule.
- PCIe and USB. The PCIe domain alone reaches configuration space and holds nothing else; its enumerator, in Erlang, plans the bus whole (sizing, placement, interrupts) before it writes. USB’s construction: every port is a hub port, hotplug ends at the controller’s domain, a device’s name dies at once and its hardware is freed only on the controller’s report. Draws on: design-pcie.md, The construction, design-pcie.md, The PCIe domain, design-usb.md, The construction.
- The Raspberry Pi 4’s own devices. The 40-pin header (GPIO, I2C,
SPI, PWM) and the firmware’s sensor readings at
/proc/sensors, as examples of small drivers under the same rules. Draws on: design-header-io.md, design-board-sensors.md.
Figures. 15.1: a driver domain: supervisor, driver process, device NIF, device, interrupts. 15.2: a call through the device NIF: rounds of observation and decision, the check, the sequencer’s maintenance and barriers, the effects. 15.3: a region’s writer passing between CPU and device across a post and a completion. 15.4: the PCIe domain and the domains it gives devices to.
Chapter 16. Storage and Updates
About 44 pages. The system ships as read-only images checked block by block, data lives in a file system written in Erlang, and an update leaves a bootable system whatever happens.
Background needed. Chapters 1, 14 and 15.
Background boxes. Block devices, partitions and the SD card. File-system design: logs, flash translation and why flash wears (F2FS, littlefs, EROFS). A/B updates and chains of hashes from boot to system image.
Status. Built: the storage domain serving /data from littlefs over
the virtio block driver and the Pi 4’s SD card, restarting cleanly after
a driver is killed mid-transfer; each user’s home served from /data;
the medium’s layout of two slots, the vault’s partition and the data
partition; the boot-selection record the packer writes and the QEMU
harness reads; the system partition as an EROFS image with a dm-verity
hash tree, read only through blocks that pass the tree, and the
partition’s packages served as the system tree; the pure FAT reader
for root’s login keys on the card; the F2FS codec and mount core, tested
on the host. Designed: the slot writer, updates with a trial boot and a
commit, the reset count, the vault’s storage domain, the FAT server, and
erXos’s F2FS as a mounted file system with its power-cut harness.
- Storage as a service. Storage runs in a domain of its own, in Erlang, over the block driver; a bug in a file system crashes a process, not the machine. Draws on: design-storage.md, Overview.
- Read-only system images. A slot’s system partition is an EROFS
image with a dm-verity hash tree after it; every block the reader
parses is checked against the tree’s root hash first, and a block
that does not match reads as an error.
Draws on: design-storage.md, Reading EROFS images,
design-storage.md, The
systemtree. - littlefs in the storage domain. The file system that holds
/data: where it runs, what it holds, what input reaches it, what its failure ends and which fuzz targets cover it. Draws on: design-stopgap.md, Release contract. - erXos’s F2FS. A read-write file system in Erlang, designed: the durable view is never overwritten, one process owns a volume, and recovery replays from a checkpoint. Crash states serve as the oracle, and a power-cut harness checks them. Draws on: design-f2fs.md, The construction, design-f2fs.md, The crash states as the oracle, design-f2fs.md, The power-cut harness.
- Removable media and the SD controller. FAT for removable media and the card’s boot volume; the SD host controller driver on the Pi 4. Draws on: design-storage.md, Removable media: FAT, design-storage.md, The SD host controller.
- Updates through two slots. An update’s whole state is the
boot-selection record:
normal(C)boots the committed slot and lets the spare be written;trial(S)boots the spare once, after which the machine returns to C unless S is committed. A spare is bootable only if it verifies at boot, so a half-written spare needs no record of its own. Draws on: design-storage.md, System updates, design-platforms.md, Slots and the boot-selection record.
Figures. 16.1: the card’s partitions (record, boot and system slots, vault, data). 16.2: the boot-selection record as a state machine. 13.3: an EROFS read checked against the hash tree.
Part V. Communication
Chapter 17. The Network Stack
About 50 pages. A TCP/IP stack modelled on FreeBSD’s default stack and designed afresh as idiomatic Erlang.
Background needed. Chapters 1, 3, 6 and 15.
Background boxes. TCP/IP in one chapter’s worth: layers, Ethernet, ARP, IP, ICMP, UDP, TCP and DNS. How TCP works: windows, retransmission and congestion control. DHCP and NTP.
Status. Built: erx_net in a network domain (Ethernet, ARP, IPv4,
IPv6 neighbour discovery, ICMP, UDP, TCP, DHCP) over virtio-net and the
Pi 4’s GENET, ifconfig, ping, host, gen_tcp and gen_udp in any
domain that holds the net facet, bounded queues and per-domain
limits, merged receives, an exhaustive test of the TCP core, fuzz tests
of the decoders, and NTP. Designed: two speed-ups that work only in host
tests (checksums in the copy, many segments as one unit), a leaner
per-segment step, sockets for native programs, packet scripts run
against the stack, generated admission code, ntpd enabled
from /etc/start.conf, and WiFi as an interface.
- Why FreeBSD’s stack, and why not a translation. Why the stack is modelled on FreeBSD’s default stack, and what a naive Erlang stack gets wrong (timers per connection, unbounded queues, bytes parsed again at every layer). Draws on: design-network-stack.md, Why FreeBSD’s default stack, design-network-stack.md, What a naive Erlang stack gets wrong.
- The construction. One owner, one step; deadlines, not timers; credited queues; packets become checked records once, and sequence numbers are integers. Each rule and the class of bug it removes. Draws on: design-network-stack.md, The construction.
- The packet path. The process tree, receiving and classifying a frame, transmitting, framing, and checksums computed in the copy. Draws on: design-network-stack.md, Processes and the packet path.
- The TCP core. A pure state machine with the control block, events and the step; output, timers, the send log with SACK and reassembly, loss detection and congestion control; checked by exhaustive exploration and by packet scripts. Draws on: design-network-stack.md, The TCP core, design-network-stack.md, Packet scripts against FreeBSD.
- The fast path. What was measured, why the cost is architectural, and the construction that answers it: runs of segments, merged receives, batching and the lean step. Draws on: design-network-stack.md, The fast path.
- Sockets act for one domain. Who holds the network device, the
grant, ports and the demultiplexing table; the interface towards
Erlang (
socket,gen_tcp, name resolution) and towards POSIX programs; and why the trusted base parses no network traffic. Draws on: design-network-stack.md, Authority and admission, design-network-stack.md, Sockets act for one domain. - Time from the network. NTP and the clock service: the approximate start, the real-time clock at boot, and who may set the clock. Draws on: design-time.md, The construction.
Figures. 17.1: the network domain’s process tree. 17.2: a received segment from driver to socket, with the copies counted. 17.3: the TCP core as state, event, step and actions.
Chapter 18. WiFi
About 12 pages. The Raspberry Pi 4’s WiFi chip, driven from Erlang, as a study in pure protocol cores.
Background needed. Chapters 3 and 6 (state machines as pure cores), chapter 15 (drivers and devices), chapter 17 (the network stack).
Status. Built as pure Erlang cores with host tests, driving no
radio: the station’s choice of a network, the join machine, the failure
memory, erx_wifi_keys, the public half of SAE, the EAPOL-Key codec
and the WPA key arithmetic. Designed: the SDIO host, the chip process
and its firmware, the station’s process, the key holder, SAE’s three
functions in crypto, the Transition Disable report, the wlan0
medium and its tools. No WiFi driver runs on a board.
- WiFi on the CYW43455. What the chip offers, where its firmware is trusted and where it is not, and the driver’s place beside the network stack. Draws on: design-wifi.md.
- The join machine. Scanning, authentication, association and the states between them as a pure state machine that returns the next action. Draws on: design-wifi.md.
- SAE and the key exchange. WPA3’s password exchange and the EAPOL-Key handshake as pure cores, and the keys they leave. Draws on: design-wifi.md, research-wifi-iwd.md.
- Testing a radio without a lab. Host tests of the cores against recorded exchanges, and what only a board can tell. Draws on: research-wifi-pi4.md.
Background boxes. How WiFi works: access points, networks, scanning, association and the frames that carry them. WPA2 and WPA3 in one page: what a pairwise key is and what the four-way handshake proves. Why a chip’s firmware is a trusted blob.
Figures. 18.1: the WiFi join machine. 18.2: the key exchange as a sequence between station and access point.
Chapter 19. Bluetooth
About 12 pages. The Raspberry Pi 4’s Bluetooth controller, driven from Erlang, with Low Energy and the peripheral role first.
Background needed. Chapters 3 and 6 (supervised processes, pure cores), chapter 15 (drivers and devices).
Status. Built, with host tests, but run on no board: the H4 framing
and HCI codec, the controller process with its supervision, the patch
reader, advertising data and the identity address, and the line tree
with its keeper. Designed: running the domain on the Pi 4, the
controller’s power cycle, serving the line to a file grant, the GATT
server and its gate, the ble facet, the serial service, pairing and
bonding, and tests against a Bumble peer.
- Bluetooth Low Energy, peripheral first. What the role is, what erXos offers a phone that connects to it, and what it does not offer. Draws on: design-bluetooth.md.
- The HCI codec. The commands, events and data packets between host and controller, parsed once into checked records. Draws on: design-bluetooth.md.
- The controller process. The line to the chip, links, the patch reader that loads the controller’s firmware, and the supervision of all three. Draws on: design-bluetooth.md.
- Testing a radio without a lab. Host tests of the cores, a Bumble test peer, and what only a board can tell. Draws on: design-bluetooth.md, research-wifi-pi4.md.
Background boxes. How Bluetooth Low Energy works: advertising, connections, roles and the attribute protocol. The host controller interface: why a host talks to a controller chip over a serial line.
Figures. 19.1: the Bluetooth controller process and its line to the chip. 19.2: advertising, connecting and one attribute read as a sequence.
Part VI. The User’s Machine
Chapter 20. The Command Shell and the UNIX Surface
About 36 pages. The system feels like UNIX, as QNX and Plan 9 do, wherever that is also how it works.
Background needed. Chapters 1, 3, 13 and 14.
Background boxes. What a shell does: parse, expand, start, connect with pipes. Terminals, ttys and the line discipline. SSH in a page: keys and channels. The rc shell of Plan 9.
Status. Built: the command shell, rc’s language in Erlang with a
bounded parser and job control of Erlang jobs; Erlang commands that
follow FreeBSD’s output (ls, cat, grep, sort, ps, top and
others); text pipes between Erlang commands and native programs; the
line discipline erx_tty, a pure core with host tests, run by each
session tty’s terminal relay; the console’s lines and Ctrl-G menu,
which connects to a session or starts one; the start table with its
settings in /data/etc/rc.conf, and service; root’s SSH login to rc
in development builds; the session’s side of the SSH domain’s line and
the login facet’s decoder. Designed: /etc/start.conf and start flags;
interrupting and killing a session from the menu; terminal input to
native programs through /dev/tty; stopping a native program with
Ctrl-Z, the Ctrl-T status line, within, rc as a program for
system and popen, toybox, the secure attribute and password logins
at the console, the SSH domain itself with its host key in the vault,
and evi, the screen editor run as vi.
- UNIX look and feel, honestly. What a UNIX user already knows is kept wherever it fits; where UNIX systems differ, erXos follows FreeBSD; and nothing wears the name or shape of something it is not. The section shows the rule deciding concrete cases. Draws on: design.md, Decisions, overview.md, Familiar tools in a different foundation.
- rc in Erlang. rc’s language, the command namespace, pipelines of
text between Erlang commands and native programs, job control, and
erlone command away. Draws on: design-shell.md, The language, design-shell.md, Pipelines, design-shell.md, Job control. - UNIX tools over live Erlang state.
ps -o pid,rss,comm | sort -rn -k2 | headlists Erlang processes and programs together in the familiar text,-where a value is not measured;grep -r error /proc/domainsreads the inspector’s state as files. Pipes carry text; records live inerland the inspector. Draws on: design-shell.md, Text commands, design.md, Signature experiences. - Ttys and the console. A terminal is a tty; the line discipline
(cooked editing, signals for the foreground group,
termios) is a pure core; the console’s lines and the secure line that alone opens the system shell. Draws on: design-live.md, The tty, design-live.md, The console. - Starting services. The start table says what starts
without a person, and
serviceenables, starts and stops it. Draws on: design-apps.md, The start table. - An SSH connection is a line. SSH logins land on the same kind of line as the console: the SSH domain, host keys, login keys, channel and algorithm policy. Draws on: design-ssh.md, An SSH connection is a line, design-ssh.md, The SSH domain.
- evi. A screen editor as a program of the session, and what it shows about the terminal stack. Draws on: design-vi.md.
Figures. 20.1: a pipeline mixing Erlang commands and a native program. 20.2: a tty between a terminal and a session’s shells. 20.3: an SSH login’s path to a shell.
Chapter 21. The Live Machine
About 36 pages. Genera’s legacy: one inspector shows CPUs, kernel objects, domains, budgets and processes as live records, each value marked with who reported it, and code loads into the running machine.
Background needed. Chapters 3, 5, 9 and 11.
Background boxes. How the BEAM traces a process. Module identity and what a code hash names. Logging and crash dumps.
Status. Built: erx:i/0,1 and erx:inspect/2, with a reporter and
a status on every value, unsupported and partial ones on Fiasco
included; inspection agents and inspect facets; /proc, dmesg, ps
and top; erx:source/1 within the caller’s runtime; erx:trace/1
over the system domain’s processes and the root service’s trace ring;
the host link with tools/erx load, watch, files and web; the
code store, loading into the system domain; the log service to the
console; the report filter; the crash record on the console. Designed:
tracing other domains and the frames between them; the host link’s
inspect, trace, source and console commands; loading code into other
domains, with overlays; refusing code outside installed storage; log
files under /var/log; a crash record that survives the reset; domain
crash dumps under /proc; the host’s tree mounted in the machine.
- The machine is an object.
erx:i()shows CPUs, threads, groups, domains, budgets and processes;erx:source(Pid)opens the line a process executes; each value carries its reporter. Draws on: design-live.md, Inspection, design-live.md, Inspection records. - Never fabricate. Unknown, partial, unsupported and timed-out values are shown as such, each with a test. On Fiasco, another domain’s group and budget read as unsupported, and say who could not answer. Draws on: design-live.md, The /proc tree, design-fiasco.md, Reporting kernel objects.
- Tracing. Processes, messages between domains and kernel events on one timeline. Draws on: design-live.md, Tracing.
- Code from the host, named by its hash.
tools/erx watchrecompiles and hot-loads a module into the running machine; the code store names a version by its hash, so sharing and rollback fall out and a changed byte fails the load. Draws on: design-live.md, Code from the host, design-live.md, The code store, design-live.md, The host link. - Logs and crash reports. The system log renders every record as one attributed, escaped line; crash reports carry no term, so a report never leaks process state; the kernel’s crash record survives to the next boot. Draws on: design-log.md, Reports carry no term, design-system.md, The crash record.
- Attributed text. Text from a source carries its source, and every sink escapes it; the console and the host terminal are the first sinks to protect. Draws on: design-live.md, Attributed text.
Figures. 21.1: an erx:i() screen with reporters marked. 21.2: the
host link: the Mac, the frames, the machine’s code store. 21.3: a trace
timeline across two domains.
Chapter 22. Displays, Windows and the Web
About 48 pages. Windows are files, as in Plan 9’s rio; an application describes its interface as a term, and every display, a screen, a panel or a browser, renders the same stream of changes in its own idiom.
Background needed. Chapters 1, 11, 14 and 15.
Background boxes. Framebuffers, pixel formats and compositing. GPUs: command streams, shaders and their page tables. Event-driven GUI toolkits and widget trees. A browser as a display: WebSocket and canvas.
Status. Built: the display domain, whose window server serves each
window as a directory of files; the pixel area and present, by the
copy path; Plan 9’s draw protocol, executed in the client’s session, and
/dev/draw for a program started with a window; the virtio-gpu,
virtio-input and Raspberry Pi framebuffer drivers in Erlang; the input
router; the single window policy; the widget server with its LVGL
renderer; the theme; the kiosk dashboard on QEMU and the Raspberry Pi 4;
Doom as an example program; the terminal emulator’s pure core; the
browser demo; and tools/erx web. Built as pure cores with host tests,
run by no guest: the overlapping window policy and the desktop programs’
cores. Built and run in a window: erx_desk with its clock command.
Designed: the terminal window and presentations in it, the Erlang and
browser renderers, the web’s network front end, the desktop session,
find and act, the direct path to the device, and graphics hardware.
- Windows are files. A display serves each window as a directory of
files; a program finds its window at
/dev/drawand draws with Plan 9’s draw protocol. A window’s pixels are a buffer its client writes, and the display composes and interprets nothing. Draws on: design-display.md, Windows and the window system, design-display.md, The draw connection. - Input routing and the terminal. The input router, the terminal emulator as a program of its session, and output linked to the values behind it, as Genera’s dynamic windows did. Draws on: design-display.md, The input router, design-display.md, The terminal.
- One interface, every medium. An Erlang application never draws: it sends a tree that says what the widgets are, what they hold and which events they raise, and receives events as messages. A screen renders it as pixels, a panel as LVGL widgets, a browser as a web page; nothing is mirrored. Draws on: design-display.md, Widget trees, design-web.md, UIs: the browser renderer.
- LVGL inside apps. LVGL runs as a native program confined to a buffer; what it may reach, what its failure ends, and what it is fuzzed against. Draws on: design-display.md, LVGL’s release contract.
- Kiosks, the desktop and a modern look. A kiosk is an app on a display with the single policy; the desktop’s programs, the shared theme, coverage masks for corners and shadows, and the rio-style window manager. Draws on: design-display.md, Kiosks and glasses, design-desktop-programs.md.
- Find and act. The desktop session’s palette: routes from a value on the screen to an rc line. Draws on: design-find-act.md, The construction.
- erXos in a browser. The website runs erXos under an AArch64 emulator compiled to WebAssembly, with a menu of hardware faults the emulator makes; and the web client, a browser display speaking the same protocol as a screen. Draws on: design-web.md, The protocol, site/README.md.
- Graphics hardware. What the design asks of a GPU, and the V3D on the Pi 4. Draws on: design-display.md, Graphics hardware, research-gpu.md.
Figures. 22.1: a window’s directory of files. 22.2: one widget tree rendered three ways: screen, LVGL panel, browser. 22.3: the kiosk on the Pi 4 (a screenshot). 22.4: the browser demo’s layers: page, emulator, erXos.
Chapter 23. Apps and Packages
About 28 pages. An app is a principal with a session template, installed at run time as one file with its needs declared and checked.
Background needed. Chapters 8, 11, 14 and 22.
Background boxes. Package managers and manifests. Declared permissions in app systems, as on a phone.
Status. Built: the start table and service, with the settings in
/data/etc/rc.conf. erx-app pack, which makes one signed .app file
of a header, an EROFS image and its hash tree. The SDK and its
public-function check. The image builder’s packages: placed in the
system partition, then checked, admitted and started at boot, as the
kiosk QEMU suite shows. The decoder, file check, admission and recovery,
with host tests. An app’s session holding only the packages and
system grants and the services its app names. erx_ui_host. Designed:
installing, updating and removing through /apps, mount of a package,
service NAME purge, /proc/packages, the app’s budget as a wall
carved at each session’s start, the flags field, the SDK’s host tools,
erx-app test, and /svc/NAME.
- An app is a principal. It gets a session from a template, its own budget and its own grants; a user adds apps without editing the board’s manifest or rebuilding. Draws on: design-apps.md, An app.
- One file per app. The package: a signed header, an EROFS image of the app’s files and a dm-verity tree; what the declaration names (memory, code, the grants it needs). Draws on: design-packaging.md, The package, design-packaging.md, The declaration.
- Admission. How a package’s declared needs are judged against the
appsrow that bounds them, before it runs. Draws on: design-packaging.md, Admission. - Installing, updating and removing. The package lifetime, boot and recovery. Draws on: design-packaging.md, Installing, updating and removing, design-packaging.md, Boot and recovery.
- The SDK. Building an app outside the tree, running its interface on the development host under stock OTP, and two examples: the kiosk dashboard and Doom. Draws on: design-packaging.md, The SDK, design-packaging.md, Examples.
Figures. 23.1: a package’s layout. 23.2: from erx-app pack to a
running app’s session.
Part VII. System Operation
Chapter 24. Startup, Configuration and Shutdown
About 38 pages. From the firmware’s handover to a login prompt, from one system manifest, and back down to a reset.
Background needed. Chapters 1, 2, 8, 15 and 16.
Background boxes. Booting an AArch64 machine: firmware, boot loader, exception levels. Initial images and boot archives. Watchdog timers.
Status. Built: booting on QEMU virt (1 to 8 CPUs, under HVF and
TCG) and on the Raspberry Pi 4 (QEMU’s raspi4b model and a real 4 GB
board); the bootable file in both forms, with every check on the boot
image made before a byte moves; the root task’s first steps and ERTS
booting the system release; the domain service starting the manifest’s
domains in boot order and restarting them within their intensity;
root’s automatic session at the console in a development build; the
root service’s watchdog, petted by erx_watchdog, and on the Pi 4 the
code that arms the board’s hardware watchdog; one end path with a
crash record on the console; reboot and halt; seeding from the Pi
4’s hardware random generator; reproducible images. Designed: a typed
login at the console; the manifest’s partial boot; a watchdog pet that
waits on the services’ answers; the ordered end’s deadline, the
vault’s quiesce and the storage freeze; the reset count and the rescue
configuration; the trial request; a release that refuses to boot on a
weak seed; signed slots; the Raspberry Pi 5.
- The bootable file and the boot image. The one file a loader boots
(Fiasco’s
bootstrap) with the boot image inside it: an archive of everything the machine starts with, accepted from one startup plan that the kernel checks before it starts the root task. Draws on: design-kernel.md, The bootable file, design-kernel.md, The boot image, design-fiasco.md, Boot handoff. - The root task’s first steps.
liberx’s root-task entry takes the root capabilities, builds the system domain and starts ERTS, which boots OTP and the system release: the domain service, the device service, drivers, storage, the network, the console. Draws on: design-system.md, The root task, design-system.md, Boot. - The system manifest at boot. How the manifest proved at build time (chapter 7) drives what starts: every configured domain created by the domain service, every device given to the domain the manifest names, ordering and partial boots. Draws on: design-system.md, The manifest, design-abi.md, Ordering and partial boots.
- Platforms. QEMU
virtunder HVF and TCG, the Raspberry Pi 4 on Fiasco and its firmware, and the Pi 5 to come; the platform layer that keeps ports mechanical. Architecture terms stay in this section and in the architecture binding. Draws on: design-platforms.md, QEMU virt, design-platforms.md, Raspberry Pi 4 Model B, design-platforms.md, The Raspberry Pi 4 on Fiasco. - The watchdog, crash records and ending the machine. The kernel’s
one watchdog, petted by the system domain once the services it lists
answer; the crash record written before the machine ends and shown at
the next boot; ordered shutdown,
rebootandhalt. Draws on: design-fiasco.md, The watchdog, design-system.md, Shutting down, design-platforms.md, Ending the machine. - Trust in the boot. What the boot trusts, entropy at boot and the device secret, and the trial request an update makes of the firmware. Draws on: design-platforms.md, Trust in the boot, design-platforms.md, Entropy sources.
Figures. 24.1: the boot sequence from firmware to login prompt, with the build-time manifest check drawn beside it. 24.2: the boot image’s layout. 24.3: the watchdog’s chain of answers.
Chapter 25. Checking the System
About 40 pages. How erXos knows it is right: answers that come from outside its own code.
Background needed. The whole book; chapters 6 and 7 most.
Background boxes. Test oracles and differential testing. Coverage-guided fuzzing and sanitizers. Model checking (TLA+) and bounded proofs (CBMC). Reproducible builds.
Status. Built: the oracles: make check-oracle, which runs OTP’s
emulator, kernel, stdlib and compiler suites on erXos and on stock OTP
and compares them case by case, then runs musl’s
libc-test; the funnel’s differential test against
binary_to_term/1; and the command shell against plan9port’s rc and
the terminal emulator against libvterm. Also built are the fuzz
targets and their budgets, the sanitizer runs, the chokepoint checks
and the stack proof. The models and bounded proofs are built: the TLA+
models of the lane, connections and links, the herd7 litmus tests and
the CBMC harnesses. So are the exhaustive explorations of the TCP
core, the USB hub machine, the pipe server and the host link’s 9P
bridge; reproducible boot images and the deterministic-mode
transcript; the QEMU suites; and the seeded faults of each of these.
Designed: the TLA+ models of the update protocol and of the host
link’s sessions; the kernel interface’s conformance cases (the harness
is built, its roster holds one pending case); the full suites of
hostile programs; the fuzz campaign against Fiasco’s system calls; and
the further oracles section 25.1.4 lists.
- Agreement with ground truth. Behaviour that disagrees with an independent oracle is a bug until it is explained and pinned by a test. OTP’s own suites run on erXos and on stock OTP on the development host, compared case by case; musl’s tests run against erXos’s C library. Draws on: design.md, Testing and the oracle, design-host-oracle.md.
- Seeded faults. A check proves itself by failing on a planted fault: a removed bound, an off-by-one comparison, a dropped fence. The section shows seeds for a chokepoint, a TLA+ model, a litmus test and a mutated copy of the Fiasco backend. Draws on: README.md, Correctness and testing.
- Fuzzing every parser of untrusted input. Each parser lands with its fuzz target: the boot image, executable headers, the funnel, the 9P codec, the shell’s parser, device rings, and more. The section explains how a target is built from the same sources for the host. Draws on: design-verification.md.
- Models and bounded proofs. TLA+ checks protocol models with TLC; herd7 checks the lane’s wake handshake and the pipe’s lock under Arm’s memory model and RC11; CBMC checks C17 code for every input within stated bounds. What each covers and what it does not. Draws on: design-verification.md, Models and implementations, design-verification.md, CBMC harnesses and runner.
- Exhaustive tests of protocol state machines. Pure cores built for the host and explored exhaustively: the TCP core, the USB hub and port machine, the window manager. Draws on: design-network-stack.md, The TCP core: exhaustive exploration.
- Determinism and reproducibility. The same inputs give byte-identical boot images; a boot in deterministic mode, under single-threaded TCG with instruction counting, gives a byte-identical transcript. Draws on: design-kernel.md, Deterministic mode, design-build.md, Processes and evidence.
- Fast checks and slow checks. Every commit passes
make checkand the QEMU suites it touches; the oracle, the sanitizers, the fuzz budgets, the models and the full QEMU suite run less often, off the path of each commit. Why iteration speed comes first. Draws on: design-build.md, What the gates run.
Figures. 25.1: the oracles and what each checks. 25.2: a seeded fault and the check that catches it. 25.3: the fast and the slow checks.
Appendix A. Glossary
About 14 pages. Every term a newcomer meets, in one line, with a
pointer to the chapter or Background box that explains it. It covers
the terms of operating systems (page table, DMA, interrupt, system
call), of microkernels (capability, IPC, rendezvous, root task), of
Erlang and OTP (supervisor, gen_server, link, node), of each
chapter’s Background box (TCP window, GATT, ELF, 9P), and erXos’s own
(domain, facet, lane, budget, funnel), from
design.md, The model and the terms each chapter
introduces. A reader who meets a term before its chapter finds it here,
in alphabetical order, with the chapter to turn to. The first version,
a-glossary.md, lists the terms of chapters 1 to 25
and the appendices.
Appendix B. Reading the Design Documents
About 8 pages. A reading order for a reader new to the system, what each design document is for, the common section layout (context, requirements, construction, invariants, failure, refusals, verification), where decisions are recorded, and which chapter leads into which documents. Draws on: design.md, Document map.
Appendix C. Building and Booting erXos
About 8 pages. Host setup, fetching pinned sources, the oracle
bootstrap, make check, booting in QEMU and making a Raspberry Pi 4
card, and hosted mode.
Status. Built: everything this appendix tells you to run. The host
setup on macOS and FreeBSD, make fetch against the pins of
deps.lock, make setup and the oracle bootstrap, make check and
the slower gates, tools/erx boot on QEMU, the QEMU medium and the
Raspberry Pi 4 card from make pack, netbooting a board from a Mac,
and hosted mode on macOS.
Draws on: README.md, Getting started, guide/README.md, design-build.md.
Appendix D. Dependencies and Licenses
About 6 pages. The dependency table, why each third-party component pays for itself, and how licenses and notices travel with every image.
Status. Built: the pins of deps.lock and their check at fetch; an
ORIGIN.md beside every import, with the change-notice check that
make check runs; the packer’s refusal of any file of unknown
provenance; the license rule LicenseCheck and its closed table; and
the NOTICES file that every packed system image carries as
system/NOTICES. Designed: a copy of NOTICES beside the boot image
in each slot’s boot partition, and the rows marked as designed.
Draws on: design.md, Dependencies, design.md, Licenses and notices.
Appendix E. Lineage
About 6 pages. The lineage of the system’s ideas: the systems and papers erXos draws on, each with the idea taken and the chapter where it appears. Draws on: research-prior-art.md, design.md, References.
Decisions
- Where the book sits among the documents. The book is listed in the document roles of CLAUDE.md and in the document map of design.md as the reader’s introduction: it cites the design documents and decides nothing. It sits between overview.md and the design documents, so a review of a design change asks whether a chapter needs the same edit.
- How the status boxes stay true. When README.md’s “Current status” changes, each chapter’s status box it touches is checked against it in the same commit as the status list. A chapter whose box is stale is a documentation bug.
- Figures. Figures are Mermaid where GitHub renders it (sequence and state diagrams) and plain-text diagrams in code blocks otherwise. Screenshots appear only for the display. The book keeps no drawing tool’s source files.
- Overview and book. overview.md covers the ground of chapters 4 and 5 in short. It stays the short introduction, and chapter 4 links it.
- The reader without background. Every chapter is written for a reader who can program and has no Erlang, operating-system or microkernel background. Part I teaches those three subjects; a chapter states the background it needs, explains in a Background box what only it uses, and never uses a term of those subjects before one of them has explained it. Appendix A lists every such term with its place. A chapter that needs the reader to know something Part I and its boxes do not give is incomplete.
- WiFi and Bluetooth have a chapter each. They share a board and a test approach but no protocol, no core and no design document, so chapters 18 and 19 stand apart and each carries its own status box.