erXos: An Operating System in ErlangChapter 15Devices and Drivers
Contents
  1. 1Operating Systems from the Ground Up
  2. 2Microkernels and Capabilities
  3. 3Erlang and OTP
  4. 4History and Goals
  5. 5Design Overview of erXos
  6. 6Correct by Construction
  7. 7One Source for Every Fact
  8. 8The Kernel Interface and Fiasco
  9. 9Processes, Runtimes and Two-Level Scheduling
  10. 10Domains: A Wall Around Some Processes
  11. 11Authority and Security
  12. 12Memory and Budgets
  13. 13Native Programs, Pipes and POSIX-lite
  14. 14Files, Namespaces and Service Trees
  15. 15Devices and Drivers
    1. 15.1A driver is a process
    2. 15.2How a device reaches its driver
    3. 15.3The device NIF
    4. 15.4DMA: the plan and its check
    5. 15.5Devices that share a page or a line
    6. 15.6PCIe
    7. 15.7USB
    8. 15.8The Raspberry Pi 4’s small devices
    9. 15.9What fails, and what it reaches
    10. 15.10Design choices
    11. Summary
    12. Where to read next
  16. 16Storage and Updates
  17. 17The Network Stack
  18. 18WiFi
  19. 19Bluetooth
  20. 20The Command Shell and the UNIX Surface
  21. 21The Live Machine
  22. 22Displays, Windows and the Web
  23. 23Apps and Packages
  24. 24Startup, Configuration and Shutdown
  25. 25Checking the System
  26. AGlossary
  27. BReading the Design Documents
  28. CBuilding and Booting erXos
  29. DDependencies and Licenses
  30. ELineage

Chapter 15Devices and Drivers

Background needed. Chapter 1, section 1.5: registers, interrupts, DMA, caches and the device tree. Chapter 3: processes, supervisors, NIFs and BIFs. Chapter 6, section 6.4: one DMA post, traced. Chapter 8, section 8.7: IRQ objects and the objects a device is given. Chapter 10: domains.

A driver bug that writes one wrong address into a network card can make the card overwrite any page of memory, and on most machines nothing in the processor notices. Drivers are also the largest and least tested part of most kernels.

A driver that runs in kernel mode programs descriptor rings with addresses it computes, places each cache flush and barrier by hand, and parses whatever the device wrote. A wrong pointer anywhere in this is a kernel panic, or silent corruption.

erXos runs each driver as a supervised Erlang process, in a domain that holds its device. Under the process sits one small piece of C, the device NIF, which reaches registers, interrupts and DMA memory for the driver and parses nothing the device wrote. Where a device must be told an address, a pure function computes a plan, and an independent check verifies it whole before anything reaches the hardware.

Section 15.1 restarts a driver under a live connection. Sections 15.2 to 15.5 follow a device from the manifest to its driver; 15.6 to 15.8 cover PCIe, USB and the Pi 4’s small devices; 15.9 and 15.10 cover failure and the choices.

15.1A driver is a process

Open a TCP connection from root’s Erlang shell to an echo server outside the guest, restart the network driver under it, and the connection carries on. The restart is one line in the system domain’s shell:

erx_driver_ctl:restart(erx:cap(network_ctl), net).

The line goes through the network domain’s driver-control facet (chapter 11), through which the system domain stops, starts or restarts a domain’s drivers. The driver process ends. A new one resets the virtio network device and reinstalls its queue, and the interface and its DHCP client start again over it. Bytes sent over the old connection still come back: the connection lives in the network stack’s processes, which never stopped. This is Built.

15.1.1What a driver’s end reaches

A driver’s supervisors, not the driver, decide what its end reaches. In the network domain each interface has a link supervisor whose first child is the driver. The driver is the link’s significant child: in OTP’s terms, its end shuts the link supervisor down, and the supervisor above starts the link again whole. The driver comes first and resets the device; then the interface and the DHCP client, which lose only soft state. A crash and a restart through the facet end the same way. A driver lives at least a second, so a dead cable restarts its link once a second rather than in a tight loop.

The storage domain is arranged the other way, on purpose. Its block driver’s supervisor restarts nothing: every child is permanent, under a restart intensity of zero. So the block driver’s end ends the storage domain, and the domain service starts it again with its device. The reason is littlefs above the driver: a file system that ran on across a driver restart would have to reason about half-done writes, while a fresh domain mounts the volume as of its last sync. A block driver killed mid-transfer leaves every closed file intact and the file being written cut at a sync. Both arrangements are Built.

15.1.2Why drivers are Erlang

A driver is mostly protocol: reset the device, negotiate features, queue requests, match completions to callers, recover from a stall. Erlang processes and gen_server were built for that. The rest is parsing what the device wrote, which Erlang does with binary pattern matching. A field that does not match crashes the one process that read it, and its supervisor starts it again; in C the same mistake reads past a buffer.

What stays in C is small and named. The device shim is the device NIF’s C: register access, interrupt binding, queues, and the DMA plan with its check. The per-byte primitives, such as checksums and bulk copies, are C because a per-byte loop costs too much in Erlang. Any other native driver code would need a row of the driver table (design-system.md, The driver table), naming the device and the measurement that showed Erlang too slow for it. On Fiasco, a measurement admits none, because it would time the kernel’s interrupt path rather than the driver, and the table has no rows. Every driver erXos runs is Erlang.

Figure 15.1 shows the parts of one driver domain.

messageserx_device callsregisters, queuesline fires{erx_irq, Dev, Line}supervisordriver process (Erlang)clients: network stack or file systemdevice NIF (C)deviceIRQ object
Figure 15.1.A driver domain. The driver is one Erlang process among others; only the device NIF touches the device, and an interrupt reaches the driver as a message.

15.2How a device reaches its driver

No driver in erXos probes for its device at run time. A device belongs to the domain the system manifest gives it to, the checked file that lists the machine’s domains and what each holds (chapter 7). So which domain may touch a device is settled before the machine boots, and the device reaches its driver as a record.

15.2.1Units and holders

The manifest gives devices out by unit: a set of devices that share a register page or an interrupt line, given whole to one domain, its holder. The Raspberry Pi 4’s manifest gives the SD unit to the storage domain in three lines:

unit block
  placement "sd"
  holder @domain.storage

The placement sd names an entry of the board’s platform record, the file that lists the board’s devices, their addresses and its pins. It holds the EMMC2 controller, which drives the SD card, and the board’s older SD controller, which shares EMMC2’s interrupt line (section 15.5). The unit is named block on QEMU virt too, so the system shell’s erx_driver_ctl:restart(erx:cap(storage_ctl), block) means the same on both machines.

Each time the holder starts, the system domain’s domain library makes the unit’s kernel objects for it, an act called a give: a physical memory object over each run of register pages, a contiguous memory object for DMA, and an IRQ object for each line (chapter 8). A device’s DMA memory is charged to the system domain’s budget, not the holder’s, and is kept across the holder’s restarts, so a restarted holder gets the same memory back. The device service, the system domain’s service that hands each holder its devices, then sends the holder each device’s record: the handles of its objects, its peripheral’s name in the CMSIS-SVD sources, its device-tree path, whether it is cache-coherent, and its descriptor form (design-runtime.md, The device NIF). A driver passes the record to erx_device:open/1. All of this is Built.

15.2.2Facts the firmware owns

Two facts decide whether a device’s DMA is safe, and neither is known when the image is built: whether the device sees the processor’s caches, and which addresses it uses for which memory. On the Pi 4 the SD controller sees the first GiB of RAM at bus address 0xC0000000, while GENET sees RAM at the processor’s own addresses (design-platforms.md).

erXos takes both from the device tree the firmware hands over, read by the root at boot before any device object is made (design-device-tree.md). The coherent flag comes from the node’s dma-coherent property: QEMU virt marks each virtio node, and the BCM2711’s tree marks none. The DMA range comes from the dma-ranges properties above the node: the device sees processor address dma_cpu + x at bus address `dma_bus

  • x`. The reader answers each fact with a value or with an absence. A device whose facts are missing or malformed gets no DMA memory at all: the reader never fills in a default, and no table written at build time holds a range that could disagree with the board that boots. The reader is Built. It cannot check that the hardware does what the tree says: the firmware is trusted as part of the machine.

15.3The device NIF

A driver reaches its device through one Erlang module, erx_device, and nothing else. A build check fails the build if a module outside the driver applications and the console’s UART drivers names it; similar checks guard the NIF’s other narrow calls.

15.3.1A driver, start to finish

The virtio block driver shows the whole interface in a few lines. Its start_device/1 brings the device up:

start_device(Spec) ->
    maybe
        {ok, Dev} ?= step(open, erx_device:open(Spec)),
        ok ?= erx_virtio:identify(fun(Register) -> erx_device:read(Dev, Register) end,
                                  ?ERX_VIRTIO_ID_BLOCK),
        {ok, Queue} ?= step(queue_create, erx_device:queue_create(Dev, ?QUEUE, ?QUEUE_SIZE,
                                                                  erx_blk_request:buffer_size())),
        {ok, Features} ?= step(negotiate, negotiate(Dev)),
        {ok, Irq} ?= step(attach, erx_irq_line:attach(Dev, ?IRQ_LINE)),
        {ok, Capacity} ?= capacity(Dev, ?GENERATION_TRIES),
        {ok, #state{dev = Dev, irq = Irq, queue = Queue, capacity = Capacity, features = Features}}
    end.

Notice two things. Registers are named, not addressed: read takes an atom such as config_generation. And the driver writes no ring address; it asks for a queue of a size, and negotiate installs it.

15.3.2Registers by name

Each register name resolves through a table generated from the device’s CMSIS-SVD file, the XML description of its registers (chapter 7). The table gives each register’s offset, width, access and fields, so a driver cannot write a read-only register. The reads and writes are BIFs inside ERTS, not NIF calls, because a driver makes them one at a time and a BIF costs less to call.

Some registers can aim the device: a ring’s base address, a DMA length, the bit that marks a descriptor valid. The SVD source marks each such register or field ownedByNif, and the NIF refuses, with badarg, a driver’s write that would change one, and writes those itself from its own queues (design-device-dma.md, Register ownership). This is Built; the virtio-mmio, GENET and SD controller descriptions carry the marks.

The Pi 4’s SD driver shows the rule at its sharpest. Some SD commands start a DMA transfer, so the NIF keeps the command register: erx_device:command/2 is the one way to write it, and it refuses any command word that would start DMA, because the NIF arms no DMA queue for the SD controller. So the SD driver moves every byte through the data register, by programmed I/O.

15.3.3Interrupts are messages

A driver binds its device’s line through erx_irq_line, and each interrupt then arrives as an ordinary Erlang message. The virtio block driver’s handler:

handle_info({erx_irq, Dev, ?IRQ_LINE}, State = #state{dev = Dev}) ->
    Status = erx_device:read(Dev, interrupt_status),
    ok = erx_device:write(Dev, interrupt_ack, Status band 3),
    ok = erx_irq_line:ack(State#state.irq),
    drain(State);

The line stays masked from its delivery until erx_irq_line:ack, so a device that holds its line raised cannot flood the domain. The driver acknowledges before it drains the completions, so one that lands during the drain fires the line again instead of being missed. No thread blocks waiting: the domain’s dispatcher (chapter 8) waits in the kernel for all its IRQ objects. Every delivery is a hint to look, so a late, repeated or spurious interrupt costs one status read and a drain that finds nothing.

15.3.4Queues outlive the driver

A queue, and the pool of buffers behind it, belongs to the device’s record in the NIF, never to the driver process, so a driver’s crash leaves the buffers it posted posted, where the device may still write them. A restarted driver that asks for the same queue gets it back. Its negotiate resets the device first, and only once the device reports idle does the NIF take back the old buffers and install the queue again. So nothing a device may still hold an address for is given out again. This is Built.

15.4DMA: the plan and its check

DMA must get three things right on every transfer: every address the device is given lies in memory its domain holds, the processor and the device never write the same memory at once, and every cache operation and barrier is in place for a device that does not see the caches (section 1.5.5). Chapter 6 followed a post; this section gives the structure and follows a completion.

15.4.1Forms, plans and one check

The device NIF holds a descriptor form for each kind of device it drives, the code that knows that device’s ring layout. A form’s core is pure: it reads no hardware and no memory the device can see. When it needs a value the device wrote, such as a ring’s index, it asks the effect layer to read it and is called again with the answer. So a queue call, such as a post or a completion, runs in rounds, each ending in more reads, in a plan and the queue’s next state, or in the call’s end (design-device-dma.md, Calls, observations and plans).

One independent check, erx_device_dma_check, stands between every plan and the hardware. It verifies that every device address and length the plan writes stays inside the queue’s DMA memory, as the device sees it, and that no region, a span of that memory such as a ring or a buffer, has two writers. The effect layer calls the check and applies only a plan it accepted; a refused plan is a bug in the form, and the call fails with nothing applied. Figure 15.2 shows one call.

complete(Dev, Queue)requestreads needed: used indexread used indexvalues readplan and next statecheck planacceptedcache maintenance, barriers, reads, copies{ok, [{Tag, Bytes}]}driver processform core (pure)effect layercheckdevice
Figure 15.2.A call through the device NIF. The form decides and the effect layer acts; the check sees each plan whole before anything is applied, and the cache maintenance comes from neither the form nor the driver.

Two forms are Built and drive devices: the virtio split ring, for every virtio driver on QEMU virt, and GENET’s register ring on the Pi 4. GENET’s descriptors are device registers, and the NIF writes their addresses once, when the queue is installed. A third form, the TRB ring of a USB host controller, is built as pure code with host tests, drives no device and is not part of the NIF. The SD controller’s ADMA2 form is designed and not built, which is why the SD driver uses programmed I/O.

15.4.2Writers, and the maintenance that follows

A form has no way to say “clean this line” or “issue a barrier”. Instead each region’s writer is a function of the queue’s state: a buffer’s writer is the device from its post until its completion, and the processor otherwise (Figure 15.3).

queue installedpost (zero the rest, clean, barrier)completion seen (clean and invalidate, barrier)device resetcpudevice
Figure 15.3.A buffer’s writer passing between the processor and the device. Every cache operation sits on a change of writer, and no plan names one.

Code outside every form compares the states before and after a plan, finds the regions whose writer changed, and derives the maintenance from that alone (design-device-dma.md, The construction: handoffs of granules):

  1. Zeroing. A buffer passing to the device has every byte the request does not fill set to zero.
  2. Handing over. Before the device uses a region, the processor cleans it, so no dirty line can later overwrite the device’s data.
  3. Taking back. After the device is done, the processor cleans and invalidates the region before reading, discarding any line fetched by speculation meanwhile.

Regions start and end on whole granules, the unit a cache writes back at once, so maintaining one region never touches another’s bytes. For a coherent device, such as every virtio device on QEMU virt, only the barriers stay; for GENET, which is not coherent, every rule acts. A build check keeps cache maintenance and barrier instructions in the effect layer alone.

15.4.3A completion, traced

Suppose the virtio block driver has posted a read of one sector, and the device has finished it and raised its line. The driver’s handle_info acknowledges and calls erx_device:complete(Dev, Queue):

  1. The core asks for the used ring’s index; the effect layer reads it, with the barrier that keeps later reads behind it.
  2. The core asks for the used entries the index covers, each naming a descriptor chain and the bytes the device wrote.
  3. The core returns a plan and a next state in which the buffer’s writer is the processor again; the check confirms both.
  4. From the change of writer, the buffer is cleaned and invalidated where needed, and the effect layer copies the bytes, up to the length the device reported, into a fresh Erlang binary.
  5. The buffer returns to the pool, and the call answers {ok, [{Tag, Bytes}]}, which the driver passes to the caller.

Two properties follow with no code in the driver. The binary is a copy, so no Erlang term aliases memory the device can write. And the buffer was zeroed at its post, so a device that reports more bytes than it wrote returns zeros, never an earlier request’s bytes, which might belong to another file.

15.4.4What the check does not protect

An IOMMU, the translation unit between devices and memory (chapter 8), could confine each device to its own buffers. Neither machine erXos runs on confines the devices it drives: QEMU’s IOMMU serves only PCIe, and the Pi 4 has none. There the check stops a bug in a form from aiming a device at the kernel’s memory, but not a driver domain that has been taken over, which can program its device through registers in ways no check foresaw. So every domain that holds a DMA device is in the trusted base (chapter 11).

15.5Devices that share a page or a line

On QEMU virt each virtio device has a 512-byte register window, eight to a 4 KiB page. On the Pi 4 the mini UART, the console, shares a register page with two SPI controllers, and EMMC2 shares its interrupt line with the older SD controller. The kernel maps memory in whole pages and binds one IRQ object to a line, so neither a page nor a line can be split between two domains.

15.5.1The unit rule

erXos makes this a fact of configuration rather than a case to handle (design-shared-devices.md, The unit rule). A unit is closed under “shares a page” and “shares a line”: devices that share either are in one unit, given whole to one domain. The manifest check refuses a manifest that splits a unit, and the give makes one register window per run of pages and one IRQ object per line, so the kernel never meets a page or a line claimed twice. The rule is Built, in the manifest checker, the domain library and the QEMU harness, which places the virtio devices by unit.

15.5.2Sharing inside one domain

Inside a domain, records whose windows share a page share the NIF’s one mapping of it, and each still reaches only its own registers, since every name resolves inside its device’s window. A shared line needs one more rule, because one line delivers one message. The line has a line owner, a process that binds it once, and the drivers of the devices that name it join the line owner instead of binding. The storage domain on the Pi 4 starts the line owner of EMMC2’s line before the block driver. No driver runs the older SD controller: the domain holds it as part of the unit and opens nothing of it. Only erx_irq_line binds or acknowledges a line. The console, in the system domain, joins the line owner of its line the same way (design-shared-devices.md, The console). All of this is Built.

15.6PCIe

Whoever writes configuration space decides where each function’s registers appear and whether it may master DMA, so erXos gives that power to one place alone.

15.6.1The PCIe domain

In the design, the PCIe domain holds each host bridge’s configuration window and no other device, because it is in the trusted base whole. Its one process, the enumerator, walks the bus, places each function’s BARs inside the bridge’s windows, and reports each function to the device service (design-pcie.md, The PCIe domain). A function then goes to its manifest holder like any device: the give carves its register windows from its placed BARs and writes its interrupt route. Only the PCIe domain touches configuration space; the holder asks for what it needs through the device service.

15.6.2A plan before any write

The enumerator does not write configuration space as it discovers things. Each function has a target, kept in the domain library: its addresses, fixed by the boot’s first walk of its bridge, and the mode its holder chose, off, decoding or mastering, with an interrupt route (design-pcie.md, The construction). A pure function, erx_pcie_plan:plan/2, takes the function’s configuration as read and its target, and returns at most three stages of writes: lower the switches as far as needed, write the data (BAR addresses, the MSI address), raise the switches to the target. So a BAR never moves while the function decodes it, and bus mastering is raised last and lowered first. Host tests check those properties over every switch state, mode and route.

The holder chooses the mode through the device service’s pci facet. A driver opens its function, which turns decoding on and bus mastering off; resets the controller through its own registers; and only then asks for master. The plan refuses mastering for a function that does not read as its holder left it. So a function the firmware or boot loader left running cannot master DMA before its driver has reset it.

15.6.3What is built

No image erXos boots runs the PCIe domain; that is designed and not built, as are MSI delivery, virtio over PCIe and every PCIe driver. The enumerator’s parts, from the configuration-space decoder to the plan and the BCM2711 bridge’s bring-up core, are Built as pure code with host tests. The domain library’s intake of a PCIe report and its give of a function are Built in C, wired into the system domain and unexercised, since no PCIe domain sends a report.

15.7USB

USB breaks the manifest’s assumption that devices are fixed: a user plugs in a keyboard, unplugs it, and plugs a disk into the same port.

15.7.1Hotplug ends at the controller’s domain

In the design no USB device enters the manifest or crosses a domain (design-usb.md, The construction). A USB device is reached only through transfers its controller runs, so it belongs to the controller’s domain. Other domains see places, fixed when the domains are configured. Input numbers hid0 to hid7 are files of input records in the controller domain’s input tree, which the display domain reads. Drives da0 to da3 let a file server ask for a partition of whatever medium a drive holds. A keyboard takes the lowest free input number when it appears and frees it when it goes. No file grant, facet or message is added when a device arrives: a reader’s file grants are made once, when its domain is created.

15.7.2A name dies at once, hardware later

A function process never names its device by its xHCI slot number or hub port, which the controller reuses. The bus process gives each attached device a fresh Erlang reference, and every transfer names a pipe (USB’s word, not a UNIX pipe): that reference and an endpoint. Detach removes the reference at once, so a transfer on a stale pipe answers {error, gone}, whatever device later takes the slot: chapter 8’s rule for handles, applied to USB.

The hardware cannot be freed at once, because the controller may still be using a detached device’s rings and buffers. They return to the pool, and the slot number to use, only when the controller reports that it holds nothing of them.

15.7.3One kind of port

The bus core knows one kind of port, a hub’s: read status, reset, clear a change, power on and off. The controller process answers these for its root ports by translating them into registers, so a device on a root port and one three hubs down run the same states. Three kinds of process own all the state, each deciding by a pure step: one for the controller, one for the bus, and one for each claimed interface.

Both machines’ controllers sit on PCIe (qemu-xhci, and the Pi 4’s VL805 chip), so USB needs the PCIe domain. The cores, the descriptor and HID report decoders and the input tree are Built as pure code, with host tests and fuzz targets. No USB driver runs on a booted machine; that is designed and not built.

15.8The Raspberry Pi 4’s small devices

The Pi 4’s 40-pin header carries pins that can be GPIO, I2C, SPI, UART or PWM signals, and its firmware reports the chip’s temperature and clocks. Both follow the rules of every other device.

15.8.1One map of the header

Which pin does what is one table, the header map in the platform record (design-header-io.md, The construction). For each of GPIO 0 to 27 it gives a use: plain gpio, a device’s signal with its function and pull, the JTAG debug port, or reserved. The map is checked against a table of the chip’s pins from the BCM2711 datasheet, so it cannot give the console’s pins to SPI. That check is Built. In the design, drivers serve /dev/gpio, /dev/i2c1, /dev/spi0 and /dev/pwm0, each from the domain that holds its controller; they are designed and not built.

15.8.2Readings from the firmware

The Pi 4’s firmware answers requests through a mailbox, and some answers are sensor readings. In the design a reading is one more row of the platform record, sent by the device service’s one firmware worker through the kernel call that already carries the firmware’s notifications, so there is no second route to the firmware (design-board-sensors.md). A sensor driver shows the readings at /proc/sensors, which is designed and not built.

15.9What fails, and what it reaches

What failsWhat happensWhat the user sees
a network driver crashes or is restartedthe link ends and the interfaces supervisor starts it again: driver (resetting the device), interface, DHCP clientTCP connections carry on
the block driver crashesthe storage domain ends and the domain service starts it again/data returns, each file as of its last sync
a device writes nonsense into a bufferthe driver’s pattern match fails and the process crashesas for that driver’s crash
a form returns an unsafe planthe check refuses it; nothing is appliedthe driver’s call fails
a device stops answering, so a register access abortsusually the driver’s domain ends; an error the kernel takes late can end another domainas for that domain’s restart
a device’s DMA facts are absentthe device gets no DMA memoryits driver’s queue_create is refused, and the driver does not start
a driver domain is taken overon these machines its device can write any memorynothing stops it; the domain is in the trusted base

Three limits deserve plain words. A fault in the device NIF, native code in the runtime, ends its whole domain rather than one process. A device error that the processor reports while it runs in the kernel is charged to whichever thread next enters user mode, so it can end another domain’s task. A device that misbehaves can hang or panic the kernel, which stops the whole machine. And a device the firmware or boot loader left running can write memory before its driver’s first reset (design-platforms.md).

15.10Design choices

Drivers in Erlang, not in C. A driver is split from the policy beside it, after sDDF, the seL4 Device Driver Framework, and written in Erlang for the reasons of section 15.1.2. The cost is an interrupt’s path through the dispatcher and the runtime’s scheduler.

A plan and a check, not careful code. A kernel’s DMA interface can hand a driver the cache operations, but the driver still places each one. In erXos forms cannot express maintenance, and the check is a second, independent statement of the rules, so a bug must pass two pieces of code written to different specifications.

One domain for configuration space. Running the enumerator in the system domain would save a domain, but would put a parser of device-written data in the code whose fault resets the machine.

Summary

A driver in erXos is an Erlang process in a domain that holds its device. Its supervisors decide what its end reaches: a network driver restarts under a live TCP connection, while the block driver’s end restarts the storage domain whole. The only C beneath a driver is the device NIF. It reaches registers by generated name, refuses writes that could aim the device, turns interrupts into messages, and keeps queues in the device’s record, so a crash frees nothing the device may still use. Where a device must be told an address, a pure form returns a plan and one independent check verifies it; cache maintenance follows from a region’s change of writer. Devices reach domains by unit, through the manifest, with DMA facts from the device tree or none at all. In the design of PCIe and USB, one domain alone touches configuration space, and USB hotplug never leaves its controller’s domain; their cores are built.