erXos: An Operating System in ErlangChapter 9Processes, Runtimes and Two-Level Scheduling
Contents
  1. 1Operating Systems from the Ground Up
  2. 2Microkernels and Capabilities
  3. 3Erlang and OTP
  4. 4History and Goals
  5. 5Design Overview of erXos
  6. 6Correct by Construction
  7. 7One Source for Every Fact
  8. 8The Kernel Interface and Fiasco
  9. 9Processes, Runtimes and Two-Level Scheduling
    1. 9.1The Erlang process is the system’s process
    2. 9.2How Fiasco shares the processors
    3. 9.3The runtime contract
    4. 9.4ERTS as a parts bin
    5. 9.5Code: the interpreter, BeamAsm and shared code
    6. 9.6The C library seam and hosted mode
    7. 9.7Two schedulers you can watch as one
    8. 9.8What happens when a part fails
    9. 9.9Design choices
    10. Summary
    11. Where to read next
  10. 10Domains: A Wall Around Some Processes
  11. 11Authority and Security
  12. 12Memory and Budgets
  13. 13Native Programs, Pipes and POSIX-lite
  14. 14Files, Namespaces and Service Trees
  15. 15Devices and Drivers
  16. 16Storage and Updates
  17. 17The Network Stack
  18. 18WiFi
  19. 19Bluetooth
  20. 20The Command Shell and the UNIX Surface
  21. 21The Live Machine
  22. 22Displays, Windows and the Web
  23. 23Apps and Packages
  24. 24Startup, Configuration and Shutdown
  25. 25Checking the System
  26. AGlossary
  27. BReading the Design Documents
  28. CBuilding and Booting erXos
  29. DDependencies and Licenses
  30. ELineage

Chapter 9Processes, Runtimes and Two-Level Scheduling

Background needed. Chapter 8 for liberx, the root service, threads and waits; section 3.5.2 for schedulers and reductions.

An Erlang process in the network domain enters an endless loop. On the same machine the display domain must draw the next frame, and the storage domain has a write half done. Something must take the processor from the loop, give it to the others, and give it back. When the network domain falls idle, its threads must sleep without spending a cycle and wake when the next packet arrives. A wakeup that comes while a thread is deciding to sleep must not be lost; a lost one leaves a packet unread until something else wakes the domain.

On a UNIX system the kernel’s scheduler meets the first half of this problem. An Erlang system there is one process, beam.smp, with one scheduler thread per processor. The kernel sees those few threads and nothing of the thousands of Erlang processes inside them; ERTS shares each thread among its processes by its own rules. The second half, sleeping without losing a wakeup, ERTS solves for itself with a spin, then a wait on a pthread condition variable or in kqueue.

erXos keeps the two levels and makes them the system’s design: Fiasco schedules threads, and each domain’s runtime, ERTS, schedules its Erlang processes on its own threads. The kernel never calls into a runtime.

One picture fixes this chapter’s words. Take a network domain whose ERTS has two scheduler threads and one dirty I/O thread, the helper that runs long C calls (section 3.5.4).

  • The two schedulers are executors: threads that run Erlang processes in turn and never block except to park, that is, to sleep until a wakeup.
  • The dirty I/O thread is a worker: a thread that may block.
  • The executors make up the domain’s group (chapter 8), and each holds a slot in it, its entry in the group’s table, by which another executor can wake it by name.
  • A packet’s arrival rings a doorbell bound to the group.
  • Every wakeup advances a counter, the epoch. An executor reads the epoch before it looks for work, and its park returns at once if the epoch has moved since, so a wakeup that races with the park cancels it.

9.1The Erlang process is the system’s process

Ask a UNIX machine what is running, and ps lists UNIX processes. Ask erXos, and ps lists Erlang processes, because those are the system’s processes. A pid, a mailbox, links and monitors make the unit of execution and of failure for the whole machine (design.md, The model). The kernel has address spaces and threads but no processes of its own (section 3.9.1).

9.1.1What a user and a programmer see

Type ps at the shell, rc, and it prints FreeBSD’s default columns, PID TT STAT TIME COMMAND, one row for each Erlang process of the runtime the session runs in, which is the system domain’s (erlang/apps/erx_shell/src/erx_cmd_ps.erl). Each row is named by the process’s registered name, or the label it set for itself through OTP’s proc_lib, or the function it started in. This is Built. The TIME and %CPU columns print -: the runtime does not measure a process’s processor time in a form ps reads, and erXos prints “not known” rather than a guess. Native programs are designed to appear in the same table, as rows of their own; that is not built.

The same processes appear in the Erlang shell’s erx:i(), under /proc, and in the inspector (chapter 21). No second, hidden kind of process lies beneath them.

An Erlang programmer writes the same code as for OTP anywhere else.

9.1.2Two levels in one picture

Figure 9.1 shows the two schedulers. Fiasco sees threads and picks which one runs on each processor. Inside each domain, ERTS sees its Erlang processes and picks which one each executor runs.

network domaindisplay domainFiasco: picks a thread for each CPUexecutor 1Erlang processesexecutor 2Erlang processesworker: dirty I/O threadexecutor 1Erlang processesworker
Figure 9.1.The two levels. Fiasco schedules threads and knows nothing of Erlang; each runtime switches among its own processes on its own executors without entering the kernel.

The split puts each decision where the information is. ERTS knows which process holds a lock, which has a message waiting and which runs at high priority; Fiasco knows which thread of which domain is ready on which processor. A switch between two Erlang processes needs no kernel entry, and a busy domain makes a great many of them, which is why the kernel does not.

9.2How Fiasco shares the processors

What keeps one domain from taking all the processor time? On erXos the answer is Fiasco’s fixed-priority scheduler and each domain’s place in it.

9.2.1One priority, round robin

Fiasco gives each thread a priority from 1 to 255 and always runs the highest-priority ready thread on each processor. Threads of equal priority take turns, a time slice each (design-fiasco.md, Scheduling). In erXos every thread but one runs at priority 1 (SERVICE_THREAD_PRIORITY in liberx/backend/fiasco/service.c). The exception is the root service’s watchdog, at 255 (watchdog_thread.c), which takes no lock. This is Built.

A spinning domain cannot shut the others out, but nothing gives the display domain precedence either. A thread that yields goes behind the others of its priority (section 8.5.1).

Fiasco has no load balancer. A thread stays on the processor where it was placed until erx_thread_set_affinity moves it, and a domain’s threads started with no processor named share one processor (design-fiasco.md, Scheduling). ERTS names no processor for its schedulers. So a domain’s executors share one processor and take turns on it; they do not run in parallel, whatever figure 9.1 suggests. The cost is unmeasured and is on the performance watch list.

9.2.2Bands

A band is a run of Fiasco priorities given to a domain by its row in the system manifest; every thread of the domain starts at its band’s priority (design-fiasco.md, Bands). The design has six bands, one priority each, listed in table 9.1 from the highest priority down.

BandHolds
1the watchdog, at 255
2the root service
3the system domain
4driver and device domains
5interactive domains: display, web, sessions
6apps and programs

Table 9.1. The six bands. A domain that spins delays only its own band and those below it.

A band orders domains; it does not divide time among them. A spinning app, in the lowest band, gets what every other thread on its processor leaves. Inside a band, threads still take turns.

A band measures no share: a driver domain in a busy loop takes its processor from every interactive domain there, for as long as it loops. It bounds no latency. And it belongs to a domain, not to a request. When an app asks the storage service to write a file, the write runs on the storage domain’s threads, at that domain’s band, not at the app’s.

Bands are Designed and not built.

9.2.3CPU sets and thread limits

A domain’s CPU set names the processors its threads may run on, and its thread limit caps how many threads it may have across its tasks. Both come from the same manifest row as the band (design-fiasco.md, Threads, bands and stops). The root creates every thread and hands Fiasco each thread’s processor mask through one call. In the design, one function first narrows the mask to the domain’s CPU set and the processors online. Its result has a type of its own that only that call takes, so a mask that skipped the narrowing does not build. A thread past the limit gets ERX_ELIMIT. CPU sets and thread limits are Designed and not built.

9.3The runtime contract

What may a runtime ask of the kernel, and what must it promise? Section 4.2 told why two-level schedulers usually failed: the kernel had to call into the user-level scheduler whenever a thread blocked, at whatever point it had reached. erXos’s contract (section 4.2.3) removes the cause; this section shows it on Fiasco.

9.3.1Executors and workers

In ERTS the normal and dirty CPU schedulers are executors; the dirty I/O schedulers, ERTS’s helper threads and the thread that waits for fault records are workers (design-runtime.md, Scheduling). The class is fixed at creation: erx_thread_create takes it, and an executor gets its slot then (liberx/backend/fiasco/thread.c). ERTS asks for the class through one option of its thread library that only erts_start_schedulers sets (otp/erts/include/internal/ethread.h).

liberx refuses a worker what only an executor may do: erx_park and erx_group_take_ready answer ERX_ETYPE to a thread with no slot (liberx/backend/fiasco/park.c). This is Built.

The design also refuses the other direction (design-scheduler.md, The runtime contract). An executor may make two kinds of blocking call: a park, and a umtx wait, a wait on a word of its domain’s private memory (section 8.5). Every other blocking call answers ERX_EWOULDBLOCK. That refusal is Designed and not built. It is a rule of liberx, not of Fiasco: a runtime that bypassed liberx could block its own executors, which harms only its own domain.

The umtx wait is allowed because ERTS builds its locks on it, and forbidding it would mean rewriting ERTS’s locking. The word lies in memory only the domain’s own threads touch, so an executor in a umtx wait waits for a thread of its own domain. Locks are held for a few instructions, so the wait is usually short; it is long only when Fiasco has preempted the holder (section 9.3.4). Either way it waits on its own domain, never on another.

9.3.2The lost wakeup

Every sleeping scheme has the same race. An executor finds no work and decides to sleep. Between the look and the sleep, a message arrives and its sender rings the doorbell. The wakeup goes to a thread that is not asleep, and the executor then sleeps with work waiting. The top half of figure 9.2 shows it.

The epoch closes the race. The executor reads the epoch before it looks for work and passes the value to the park. Under a lock, the park compares the current epoch with the value read. If anything advanced it in between, the park returns at once and the executor looks again. If not, the executor joins the parked list under the same lock, and any later wakeup finds it there.

look: no workwake: nobody parked, wake lostparkread epoch = 7look: no workwake: epoch 7 -> 8park(7)epoch is 8: return at oncelook again: work foundasleep with work waitingwith an epochexecutorpark recordsender
Figure 9.2.Park and wake, drawn for one epoch. The wakeup that falls between the look and the park advances the epoch, so the park that would miss it returns at once.

There are two epochs, because there are two kinds of wakeup. A doorbell wakes “some executor of the group” and advances the group’s epoch. ERTS also wakes one executor by name, when it puts a process on a sleeping scheduler’s run queue: erx_executor_wake advances that slot’s epoch. The park returns at once if either changed (design-scheduler.md, Park and wake).

9.3.3Park and wake on Fiasco

Fiasco has no park call, so liberx builds one from a semaphore per thread, on which every wait in erXos ends (section 8.5.2), and a dispatcher, one thread in each task (Fiasco’s address space) that receives the task’s doorbell deliveries (design-fiasco.md, The park protocol). The group’s ready list holds the doorbells that have fired and not yet been handled. Follow a request into a parked storage domain.

  1. A process in a session sends the storage service a request. The session’s runtime copies the frame into a lane, the one-way ring that carries messages between two domains, and rings the lane’s doorbell (section 8.7.1).
  2. Fiasco delivers the doorbell to the storage domain’s dispatcher.
  3. The dispatcher takes the park lock and runs the signal event: it puts the doorbell on the ready list unless it is listed already, advances the group’s epoch, and takes the longest-parked executor off the parked list. It releases the lock and raises that executor’s semaphore.
  4. The executor returns from its wait. At its safe point, between two Erlang processes, it takes the ready list with erx_group_take_ready, which returns each doorbell’s label once, however often it fired, and dispatches each label (section 9.4.4).

One doorbell wakes one executor. When none is parked, a running executor finds the doorbell at its next safe point.

The protocol’s decisions are one pure function, erx_backend_park_step (liberx/backend/core/park_table.c), which takes the park record and an event (a park, a wake, a doorbell, a timeout, a thread’s end and a few more) and returns at most one semaphore to raise or wait on. It calls no kernel, so host tests drive it (section 6.3). Parking is Built. Its cost is one thread switch: a wakeup of an idle domain goes through the dispatcher, so it takes two switches where a kernel with a park call would take one.

9.3.4What a runtime does not get

On Fiasco a runtime gets threads, park and wake, and nothing else from the scheduler (design-fiasco.md, The scheduler on Fiasco).

  • No upcall. The kernel never enters a runtime at a point the runtime did not choose. erXos arms Fiasco’s one upcall interface, its vCPU mode, for no runtime thread; the design arms it only for native programs’ workers, never an executor (section 8.5.6).
  • No urgency. A runtime cannot change one thread’s priority: erx_thread_set_urgency answers ERX_ENOTSUP (liberx/backend/core/refused.c).
  • No share. Fiasco charges each thread for its own slices and keeps no account per domain (section 5.6.5).
  • No grace period. A slice ends where it ends, even inside a run-queue lock: a thread that holds a lock gets no time past its slice, and an executor that wants the lock sleeps in its umtx wait until the holder runs again.

Each absence buys simplicity: a runtime never entered unexpectedly needs no code to recover, and a kernel that takes no hints cannot be misled by a runtime that lies.

9.4ERTS as a parts bin

Why does erXos carry its own ERTS instead of running an unmodified one on a compatibility layer? Much of ERTS only adapts it to UNIX, such as its poll set over file descriptors, and where erXos already has a mechanism for a job, keeping both would give a reader two models to learn (section 3.9.5).

9.4.1One mechanism per job

Each job has one mechanism, ERTS’s or erXos’s. Table 9.2 shows the scheduling rows.

JobMechanism
choosing which process runsERTS’s run queues, priorities, reductions and stealing
choosing which thread runsFiasco
waiting for workerx_park with epochs
readinessdoorbells on the ready list, through the event table
timeoutsERTS’s timer wheel per scheduler, which files each Erlang timer by when it expires; a park’s deadline is the wheel’s next expiry
long C callsdirty schedulers: dirty CPU schedulers are executors, dirty I/O ones workers

Table 9.2. Scheduling in ERTS on erXos (design-runtime.md, One mechanism per job).

ERTS’s scheduler stays whole, because it has run production Erlang systems for decades. Its sleep and readiness are redone: the poll set assumes file descriptors, which a domain does not have, and the spin spends time that another domain at the same priority could have used.

9.4.2The inside of ERTS on erXos

ERTS’s scheduler, garbage collector, loader and distribution sit on an adapter, the directory otp/erts/emulator/sys/erxos/, which holds what ERTS needs from the system beneath it and is the only part of ERTS that includes a liberx header (design-runtime.md, The adapter). Figure 9.3 shows the layers.

Erlang code: OTP, services, driversERTS: scheduler, GC, loader, distributionadapter, sys/erxos: park and wake, event table, lane driver, crash areamusl: threads, malloc, stringsliberxFiasco, and the root service
Figure 9.3.ERTS on erXos. Only the adapter and musl call liberx, and only liberx reaches Fiasco or the root service. The lane driver is the port that carries frames between domains; the crash area is where a dying runtime leaves its dump (section 9.8).

ERTS’s files outside the adapter carry some seventy numbered edits (design-runtime.md, The edits outside sys/erxos). Two matter here. scheduler_wait, the step where a scheduler with nothing to run spins and then sleeps, parks with erx_park instead of a thread event or the poll set. wake_scheduler wakes with erx_executor_wake.

9.4.3The scheduler’s sleep, in code

sched_park, in otp/erts/emulator/beam/erl_process.c, stands where stock ERTS called erts_tse_twait:

/* Answers as erts_tse_twait does: 0 at a wake, which may be spurious, or ETIMEDOUT. */
static int
sched_park(ErtsParkEpochs epochs, Sint64 timeout)
{
    const uint64_t deadline = timeout < 0 ? ERX_WAIT_FOREVER : erx_deadline_after((uint64_t) timeout);

    switch (erx_park(epochs.group, epochs.slot, deadline)) {
    case ERX_OK:
        return 0;
    case ERX_ETIMEDOUT:
        return ETIMEDOUT;
    default:
        erts_exit(ERTS_ABORT_EXIT, "%s:%d: Internal error\n", __FILE__, __LINE__);
        return 0;
    }
}

Two things matter. The function answers as the call it replaces did, so the rest of scheduler_wait, its flags and its loop, is ERTS’s own. And the epochs arrive as an argument, ErtsParkEpochs, the group’s and the slot’s values, which the caller read before it decided to sleep. The timeout is the next expiry of the scheduler’s timer wheel, so a parked scheduler wakes for its next Erlang timer and no sooner. A dirty I/O scheduler, which has no slot, keeps ERTS’s thread event, a umtx wait on erXos.

9.4.4The event table: readiness without file descriptors

On UNIX, ERTS learns that a socket has data from kqueue or poll, which report file descriptors. A domain has none. It has doorbells: each connection’s lane, each device line and each native program’s channel (the pipe between ERTS and the program, chapter 13) has one, bound to the domain’s group with a label, a number chosen by the code that bound it. The ready list returns labels, and the event table says what each names (design-runtime.md, The event table). A label leads to one of three sinks:

  • a connection sink, a lane from another domain: the lane port’s input task is scheduled, and a normal scheduler copies the frames out;
  • a port sink, such as a native program’s channel: ERTS’s own input task for that port is scheduled, so the port’s work runs one piece at a time, as every port’s does;
  • a process sink, such as a device’s interrupt: the term registered with the label is sent to an Erlang process, as {erx_irq, Device, Line} reaches a driver.

A label whose sink has gone is dropped. The table’s core, otp/erts/emulator/sys/erxos/erx_event_table.c, knows nothing of ERTS and calls no kernel, so host tests drive every path; erx_event.c holds its effects. ERTS on erXos runs no poll thread. The event table is Built.

9.5Code: the interpreter, BeamAsm and shared code

Which code a domain may run is a security decision as much as a speed one. A just-in-time compiler, or JIT, turns a program into machine code while it runs: it writes code and then runs it, which is what an attacker who finds a memory bug wants to do. erXos never maps a page writable and executable through one mapping (section 6.2.6), so a JIT writes through one mapping and runs through another.

9.5.1Three runtime images

erXos builds ERTS three ways, listed in table 9.3 (design-runtime.md, BeamAsm).

ImageWhat it isWho runs it
system imagethe interpreter, without BeamAsm, its C++ runtime or OpenSSLthe system domain
interpreter imagethe same build with OpenSSLcommand domains (where a shell command with less authority than its session runs), every domain in deterministic mode, and any domain whose manifest row chooses it
BeamAsm imageERTS with BeamAsm, OTP’s JITevery other domain

Table 9.3. The three runtime images.

Which image a domain runs follows from three facts fixed at boot: whether it is the system domain, one field of its manifest row, and whether the machine boots in deterministic mode (erx.det=1, a mode whose runs repeat exactly, in which BeamAsm’s output would vary with addresses and hash order). No domain chooses at run time. Both the interpreter and BeamAsm are Built; the manifests in config/manifest/ choose the interpreter for every domain (section 5.1.3).

The system domain interprets because it is the trusted base, and its image holds no JIT and no C++. OTP’s own notes put BeamAsm at “anything from no difference to up to four times as fast” as the interpreter. The system domain gives that up for a trusted base with no code generator to attack.

In a BeamAsm domain, each block of JIT code is a memory object mapped twice, once writable for BeamAsm and once executable for the processor, so one page is writable through one mapping and executable through the other. Only a domain whose image is marked for JIT gets an address space that allows such objects (otp/erts/emulator/sys/erxos/erx_jit_memory.c).

9.5.2Code by hash

The system domain’s code service loads modules into the system domain’s runtime by hash: each module is named with the SHA-256 of its file, and the service checks the bytes against it before ERTS’s loader reads them. A changed byte fails the load with {error, {hash_mismatch, Name}} (design-runtime.md, Code in the system domain). The code service and its check are Built (erlang/apps/erx_code/src/erx_code.erl).

The system domain’s runtime is designed to accept a module only if its bytes lie in installed storage: the pages of its release, or an arena only the code service writes. It refuses any other with {error, not_installed}, however the binary travelled. That refusal is designed and not built.

9.5.3Shared code

Every ERTS domain loads its own copy of lists, gen_server and the rest of kernel and stdlib, charged to its own budget; the design document measures loaded code at about 10 of the 16 MB the system domain reports at its shell (design-shared-code.md). The design builds one read-only image of the common modules into the boot image, and every interpreter domain maps it at the same addresses, so one copy serves the machine (design-shared-code.md, The central construction). The shared-code image is Designed and not built.

9.6The C library seam and hosted mode

ERTS is a C program, and it gets memory, threads and the clock from a C library.

9.6.1One seam

musl is the C library of every runtime and native program. No trap instruction is left in it. Every place where musl would make a system call lands in one function, __erx_syscall, a switch on musl’s Linux system-call names (SYS_futex, SYS_mmap and the rest) that carries each call onto liberx; a call it does not handle answers ENOSYS (design-fiasco.md, musl over the backend).

musl needserXos gives
pages for mallocerx_vm_reserve, and erx_space_protect, which commits what it makes accessible
threadserx_thread_create, erx_thread_exit
locks and condition variablesumtx waits on private memory
clocksFiasco’s clock for CLOCK_MONOTONIC, the root’s wall clock for CLOCK_REALTIME
random byteserx_entropy_take

This is Built. The design deletes the Linux names, so that a call not ported fails when the program is linked rather than when it runs (design-runtime.md, The C library seam); that deletion is designed and not built. A runtime uses no file descriptor but standard output and error; native programs get files, pipes and signals from a POSIX layer over liberx (chapter 13).

The errno and signal numbers musl returns are FreeBSD’s, generated from one table in the ABI (section 7.3.1), so a program written for FreeBSD prints the errno a FreeBSD user expects. This is Built.

pthread_cancel acts at the target thread’s next cancellation point, as POSIX allows: Fiasco’s backend does not stop a thread from outside (liberx/backend/core/refused.c). ERTS never calls it.

9.6.2Hosted mode

A crash inside ERTS on QEMU is slow to debug. Hosted mode runs the same runtime as a macOS process, on liberx’s hosted backend, under AddressSanitizer, the other sanitizers and lldb (design-runtime.md, Hosted mode). It is Built. QEMU stays the authority: a test that passes only hosted is not a pass.

A hosted runtime links macOS’s libSystem, not musl: each C library finds its per-thread data through the one processor register that points at the current thread, and in a macOS process libSystem owns it. Above the C library nothing differs, and build checks fail on a file of sys/erxos that names __APPLE__, Linux or a backend.

9.7Two schedulers you can watch as one

When a domain is slow, which scheduler is to blame? On UNIX, top shows the kernel’s view and a program’s own tools show its view, rarely both at once. erXos merges them.

9.7.1One timeline

erx:trace(Spec), typed at the Erlang shell, prints one stream of events in the order of one clock (design-live.md, Tracing). From ERTS it takes OTP’s own trace of each time an Erlang process is scheduled in or out. From the root service it takes the trace ring, 4,096 records that liberx writes as it carries out parks, wakes and doorbell deliveries (design-fiasco.md, The trace ring). The trace process merges both in time order (erlang/apps/erx/src/erx_trace.erl). It is Built.

So one stream shows an executor of the system domain park, the dispatcher deliver a doorbell, the executor wake, and the Erlang process that ran next. Only the root’s own threads write the ring, so a park in another domain does not appear, though that domain’s process events do.

9.7.2A worked example

A file service runs in domain A. Session B runs a build that reads thousands of small files; session C runs an editor whose user has pressed save. Each session has a lane into A, and in A a facet process, which receives that session’s requests and checks them (section 7.7; design-scheduler.md, A worked example).

  1. B’s processes send read requests as fast as they can. When B’s lane is full, B’s senders wait inside B’s own runtime, so A is asked for no more than the lane holds.
  2. A’s executors run B’s facet and the processes that serve B’s reads. That time is A’s, on A’s threads: A pays for the work it does for B.
  3. C rings its doorbell. A’s dispatcher lists it and wakes a parked executor of A; if every executor is running, one finds it at its next safe point.
  4. A’s ERTS runs C’s facet beside B’s, sharing A’s executors by priority and reductions. C’s request is decoded within a few of A’s scheduling rounds, not behind B’s whole backlog, because that backlog waits in B’s lane, not in A’s run queues.
  5. If all of A’s threads are busy, A takes turns on its processor with every other ready thread; nothing lends A the time of B or C.

The design adds two parts. Bands order A, B and C by their manifest rows. And each facet admits its client’s requests within a share of reductions per period, so B’s facet stops taking requests once B’s share is spent, while C’s are taken at once (design-scheduler.md, Service work). Both are designed and not built: a facet admits each request whose operation it lists (erlang/apps/erx/src/erx_facet.erl).

9.8What happens when a part fails

Each level fails in its own way, and each failure stays where it started.

An Erlang process loops. Its reductions run out at the end of its turn, and ERTS runs the next process. Other domains do not see it.

A NIF loops, or an executor blocks. C code called from Erlang is not counted in reductions, and a NIF that waits on a server blocks its executor. Either way that executor runs no Erlang process until the call returns, and the domain’s other executors steal its queued processes. Only its own domain is affected. A long C call belongs on a dirty scheduler (section 3.5.4).

A domain spins. Its threads take turns with every other thread on their processors, at one priority, so other domains slow down but run.

An executor dies. The park protocol’s release event frees the slot, advances the group’s epoch and wakes the longest-parked executor, so a wakeup meant for the dead one is not lost with it.

A domain ends. Its runtime writes a dump to the domain’s crash area and stops the domain. Every thread of every task ends with it, its group and park record end with it, and the processes linked to its processes get exit signals (design-runtime.md, Crash handling).

The machine stalls. Once a second the watchdog thread, at priority 255, checks that the root service has reported in; a missed report resets the machine. On the Raspberry Pi 4 it also reloads the hardware watchdog, so a hang of Fiasco itself ends in a reset (design-fiasco.md, The watchdog).

9.9Design choices

Every Erlang process a kernel thread. The kernel could schedule each Erlang process itself, as a 1:1 thread library does. But every spawn would be a request to the root and every switch a kernel entry. Erlang processes are cheap because they never enter the kernel (section 4.2.1).

Scheduler activations. The kernel could tell the runtime whenever one of its threads blocks. That needs the kernel to enter the runtime at an arbitrary point, and every system that tried it for general C code gave it up (section 4.2.2). ERTS does not need it, because an Erlang process never blocks its thread: a receive with no message parks the process, and the executor runs another.

Shares instead of priorities. Fiasco could give each domain a share of processor time, a percentage of a core. That needs a scheduling class Fiasco does not have, a large change to its scheduler, accounting and switch path (design-fiasco.md, The scheduler on Fiasco). Bands give an order between domains for one number each; the cost is that a busy domain in a high band can starve every lower one on its processors.

Lending time to a service. A client could donate its processor time to the server working for it, so that a server with no time of its own runs on its callers’ time (research-hierarchical-scheduling.md). erXos does not: a service runs on its own threads, and nothing must work out who pays. The price is that a saturated service slows all its clients together (section 5.6.5).

Stock ERTS on a POSIX layer. erXos could give ERTS a full POSIX emulation and leave ERTS untouched. Every wakeup would then pass through an emulated poll, and erXos would carry two models of readiness. erXos judges some seventy numbered edits to ERTS cheaper to maintain than an emulation of the system ERTS was written for.

Summary

Every Erlang process is a process of the system. Fiasco runs every thread but the watchdog at one priority, round robin, and a domain’s executors share one processor; bands, CPU sets and thread limits are designed. Executors only park, workers may block, and a park against two epochs cannot lose a wakeup. ERTS keeps its scheduler and replaces its sleep and readiness with erx_park and an event table. The system domain interprets; musl reaches liberx through one function; the same runtime runs hosted on macOS; and erx:trace shows both schedulers in one timeline.

The scheduler decides which thread runs; the next chapter turns to where those threads live, the domain, and the wall around it.

  • design-fiasco.md, The scheduler on Fiasco: the scheduler design on Fiasco, the bands and the park protocol.
  • design-scheduler.md: the full two-level design.
  • design-runtime.md: ERTS on erXos, from the adapter and the event table to BeamAsm, the C library seam and hosted mode.
  • research-hierarchical-scheduling.md: the record of M:N threading and why it was abandoned.
  • Thomas Anderson, Brian Bershad, Edward Lazowska and Henry Levy, “Scheduler Activations: Effective Kernel Support for the User-Level Management of Parallelism”, SOSP 1991: the design whose failure the runtime contract answers.
  • OTP, erts/emulator/internal_doc/BeamAsm.md: BeamAsm and its speed against the interpreter.