■ erXos · operating system

What if every Erlang process were an operating-system process?

erXos is an operating system built on that idea. Its drivers, file systems and services are Erlang processes, supervised and isolated by a small capability microkernel, and its shell feels like UNIX.

             __  __
   ___  _ __ \ \/ /  ___   ___
  / _ \| '__| \  /  / _ \ / __|
 |  __/| |    /  \ | (_) |\__ \
  \___||_|   /_/\_\ \___/ |___/

erXos
development build: trusted code only
Erlang/OTP 29 [erts-17.1.1] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1]

rc: the command shell; erl starts the Erlang shell
% # The command shell runs rc's language, written in Erlang.
% ls /
NOTICES
bin
lib
manifest
root
% echo hello from erXos > /tmp/hi
% cat /tmp/hi
hello from erXos
% # How many processes run? Each is an Erlang process.
% ps | wc -l
      62
% erl
Eshell V17.1.1 (type help(). for help)

% Crash a process that divides by zero, and catch it
% with a monitor. The shell carries on.
1> {_, R} = spawn_monitor(fun() -> 1 / list_to_integer("0") end).
=ERROR REPORT==== 1-Jan-1970::00:00:02.651025 ===
Error in process <0.150.0> with exit value:
{badarith,[{erlang,'/',[1,0],[{error_info,#{module => erl_erts_errors}}]}]}

{<0.150.0>,#Ref<0.3910384898.1.60465>}
2> receive {'DOWN', R, _, _, {E, _}} -> {crashed, E} end.
{crashed,badarith}

% A ring of N processes passes a token round it Laps times.
3> Ring = fun(N, Laps) ->
..     Last = lists:foldl(fun(_, Next) ->
..         spawn(fun Loop() -> receive 0 -> Next ! 0;
..                   T -> Next ! T - 1, Loop() end end)
..       end, self(), lists:seq(1, N)),
..     Alive = erlang:system_info(process_count),
..     Last ! N * Laps,
..     (fun Wait() -> receive 0 -> ok;
..                    T -> Last ! T - 1, Wait() end end)(),
..     {Alive, N * Laps}
..   end.
#Fun<erl_eval.41.130099583>

% 5,000 processes alive at once, 100,000 messages.
4> Ring(5000, 20).
{5061,100000}
erXos Recorded session
01 / The idea

One machine, one Erlang system.

Drivers, file systems and system services are Erlang processes, supervised like any other. When one crashes, its supervisor restarts it.

The machine stays open. From an Erlang shell you inspect, trace and change it, all the way down to kernel objects. That is the goal, and the design is built around it: a kernel that knows nothing of Erlang, and an Erlang system that runs the machine.

In erXos, Erlang's process is the operating system's process: one process space, one message system and one view of identity, supervision and inspection span every runtime.

G1

Secure by construction

Authority is a capability someone handed you. Knowing a pid, a path or a node name grants nothing.

G2

Millions of processes

Erlang processes are cheap to create and cheap to switch. erXos keeps them cheap.

G3

Crashes are routine

A driver wedges its device. It fails alone, its supervisor restarts it, and the machine carries on.

G4

A live machine

Upgrade a driver in place. Follow any process to the line of source it is executing.

G5

One kernel, many runtimes

The kernel hands out CPU time. ERTS comes first. C programs share the machine on the same terms.

G6

Cross-platform

AArch64 on QEMU first, then the Raspberry Pi 4 and 5. Other architectures are ports, not redesigns.

02 / Principles

Four rules shape every change.

P1

Erlang wherever practical

Erlang runs the drivers, the file systems, the network stack and the shell. C is for the kernel, device shims and per-byte work. Practical is the key word.

P2

Beauty and simplicity

A few concepts, used everywhere. A little speed is a fair price for a lot of simplicity.

P3

Declarative where it simplifies

One manifest says what the machine runs. The build checks it. Where a description is simpler than code, it replaces the code.

P4

UNIX look and feel

Shell, files, pipes, ps. FreeBSD programs, recompiled from source. UNIX is how the system looks, not what it is built on.

03 / Architecture

A small kernel. Everything else is a process.

The kernel knows threads, memory, capabilities and interrupts. It knows nothing of Erlang. Everything else runs in user mode, the runtimes included.

A domain is an address space with its own runtime. It is also an Erlang node. Processes in different domains link, monitor and message each other as Erlang processes always have. A crashed domain looks like a dead node.

Surface
UNIXshell · files · pipes · terminals
Domains
System domainERTS · domain service · console · services
Network domainERTS · network drivers · TCP/IP stack
Storage domainERTS · storage drivers · file systems
DomainERTS · your Erlang application
DomainERTS · another application, isolated from the first
Domainnative programs on musl
Interface
liberxthe kernel interface, one ABI for every runtime
user mode ↑ · ↓ kernel mode
Kernel
Capability microkernelthreads · memory · capabilities · interrupts · CPU grants
Machine
AArch64QEMU virt · Raspberry Pi 4 and 5 to follow
A1

Two-level scheduling

The kernel grants CPU time. Each runtime schedules its own processes within that grant, without a kernel call.

A2

A budget for everything

Every kernel allocation is charged to a budget. Delete the budget, and everything charged to it goes too.

A3

Private namespaces

Straight from Plan 9. There is no global root, so a path is the visible face of a capability.

erXos stands on the shoulders of giants. Erlang/OTP gave it processes and supervision; L4 and Fiasco, the kernel; Plan 9, namespaces and files; QNX, UNIX on a microkernel; Genera, a whole machine open to inspection. What is new is how the pieces meet.

04 / Status

Early, but it boots.

Phase 1: erXos runs on a modified Fiasco behind erXos's own kernel interface; erXos's own kernel replaces it in Phase 2.

Running

ONQEMU virt

AArch64, on 1 to 8 CPUs.

ONRaspberry Pi 4

Boots from an SD card, with Ethernet, HDMI and a serial console.

ONIn the browser

The console above, on an AArch64 emulator.

ONDomains

Storage, network, display and each session run their own Erlang runtime, in user mode.

ONDrivers in Erlang

Console, virtio block, net, gpu and input, and the Pi's SD card, Ethernet and framebuffer.

ONNetwork

A TCP/IP stack in Erlang, with DHCP, HTTPS and SSH.

ONGraphics

Windows, Plan 9's draw protocol and LVGL widgets; the kiosk shows live quotes.

ONFiles

Private namespaces, littlefs on /data, and EROFS checked by dm-verity.

ONShell

rc, written in Erlang, with job control and pipes.

ONInspector

Processes, domains and kernel objects at /proc.

ONNetwork time

ntpd sets the clock at boot.

ONChecked

OTP's own test suites, against stock OTP.

Next

NEXTerXos's kernel

Begun as host-tested C cores.

NEXTPrograms

Native programs run in pipelines with Erlang commands; terminal input and sockets come next.

NEXTF2FS

The data file system. Its on-disk reader is written; the file system is next.

NEXTPCIe

An enumerator in Erlang, in a domain of its own, on QEMU virt and the Pi 4.

NEXTUpdates

A new system goes to the spare slot and commits after a healthy trial boot.

NEXTRaspberry Pi 5

The next board.

NEXTUSB, Bluetooth and WiFi

Protocol cores written; drivers next.

05 / Apps

Two example apps.

The Pi 4 card and the QEMU image carry two example apps, each in a session of its own. Screenshots from QEMU virt.

The kiosk: an analog clock and the date on the left, live Bitcoin, Ether and Solana prices with 24-hour charts on the right. E1

Kiosk

The screen a board boots to: a clock and live prices, fetched over HTTPS through erXos's TCP/IP stack. Written in Erlang, drawn with LVGL.

Doom running on erXos under QEMU, with Freedoom's graphics. E2

Doom

A native program: type doom in rc and it opens a window over the kiosk. doomgeneric with Freedoom's data.

06 / Correctness

Checked against answers the code did not write.

A test only proves what it checks. So erXos compares its code with stock OTP, a bounded model checker and a weak-memory simulator, and each check says what it covered.

Most checks come with a deliberately broken twin or a seeded defect that they must catch, so a check that cannot fail gets noticed. These checks have limits. A model checker checks a model, not machine code. A litmus test checks a small program, not the compiled lane. A race detector finds data races but cannot show that a protocol is right.

C1

Stock OTP as the oracle

OTP 29.1.2's emulator, kernel, stdlib and compiler test suites run on erXos and on stock OTP, and the results are compared case by case. erXos's C library also runs musl's own tests.

C2

Weak memory, under herd7

Litmus tests of the lane's wake handshake and ring positions run under Arm's AArch64 model and RC11. With its fences the protocol must never reach the bad outcome. Without them it must.

C3

TLA+ models

TLC checks models of the lane, connections and links, each with seeded defects it must catch. Models for the host link, updates and the scheduler are planned and not written.

C4

CBMC

Bounded model checking of the budget arithmetic and the capability table's lookup, rights and slot closing, for every input within stated bounds.

C5

Fuzzing and sanitizers

The rule is a fuzz target for every parser of untrusted input, and the known gaps are tracked. Host tests of the C code run under AddressSanitizer and UndefinedBehaviorSanitizer, and concurrency tests under ThreadSanitizer.

C6

Rules the build enforces

Rules of the form “only X may do Y” fail the build on a new site. The stack depth of the C17 cores is computed from their call graph, and two clean builds of the production boot image must match byte for byte.

07 / Try it

The real thing, in your browser.

The console at the top of this page runs erXos itself. Press Boot erXos and your browser downloads the same boot image that QEMU runs, then starts it on v64, an AArch64 machine emulated in WebAssembly. The kernel, the system domain and the command shell all come up inside the page, in a few seconds.

Nothing runs on a server, and nothing you type leaves your machine. Reset starts the machine again from the same image.

Kernel
Capability microkernel
System
Erlang/OTP
Surface
UNIX
Target
AArch64 · QEMU virt
Stage
Phase 1 · early
T1

The command shell first

The console opens in the command shell, rc's language written in Erlang. Write a file to the RAM file system and read it back: echo hello > /tmp/hi, then cat /tmp/hi.

T2

Then Erlang

erl starts the Erlang shell. q(). ends it and brings back the % prompt.

T3

Look around

ls / lists the root of the namespace. ps lists the processes, which are Erlang processes. top shows them live under uptime, memory and run queue; Ctrl-C ends it. dmesg prints the system log from boot on.

T4

On the network

ifconfig shows the network device and the address it got by DHCP. ping -c 3 8.8.8.8 sends real ICMP through erXos's Erlang stack and driver. A small router emulated in this page answers every echo itself: no packet leaves your browser.

08 / Download

Run it on your own machine.

D1 Board

Raspberry Pi 4

Download card .img.xz · 45.8 MB

Write it to a microSD card with dd or Raspberry Pi Imager.

Serial console at 115200 baud; needs a bootloader from 2022-12-01 or later.

# diskN: the card in diskutil list; everything on it is erased
diskutil unmountDisk /dev/diskN
xz -dc sdcard-232e44d7d49297a8.img.xz \
  | sudo dd of=/dev/rdiskN bs=4m
diskutil eject /dev/diskN
SHA-256
b1a6c47b8abae34ccf649f9b7d512c2384fe4d1ab2177ddb97522cc111fdddfa
Source
eb38b35853fa · build record
D2 Emulator

QEMU virt

Download image .boot.gz · 44.0 MB

The image the console above runs. Boot it with QEMU; Ctrl-A X quits.

qemu-system-aarch64 -machine virt,gic-version=3 \
  -cpu cortex-a76 -m 256 -nographic -nic none \
  -global virtio-mmio.force-legacy=false \
  -netdev user,id=net0 \
  -device virtio-net-device,netdev=net0,bus=virtio-mmio-bus.23 \
  -kernel image-43b63ac6a9904857.boot.gz
SHA-256
c72b9e6c9bff4d1787e24d8e69c2da552a3872d8a2844b840b4d6e1c788d5c6b
Source
eb38b35853fa

Read the release notes first: 0.1 is a development preview that runs trusted code only. The console above runs on v64: license, third-party code.