Eio_fs_chaosSourceFilesystem faults and power cuts, for tests of Eio programs and of Eio filesystems.
Eio_fs_chaos has two checks, one function each:
run checks that an Eio program copes with a filesystem that fails. It runs the program many times on in-memory filesystems that fail, and each time starts it again on what the fault left, as a machine restarts a service.conformance checks that an Eio filesystem answers as Linux or macOS does.Both raise Failure with a report when the check fails, as an assertion does, so they are the body of a test in any test runner. run is Eio_chaos.run on one device, disk: a program that also needs a network is tested with Eio_chaos.run [ disk (); Eio_net_chaos.network () ]. A test that wants one fault at one place builds it from the building blocks: wrap for errors, Power for a power cut by hand, ack for a file the program has promised to keep.
run, conformance and Power.v take ?os, as Eio_fs_mem.v does: `Linux, the default, `Macos or `Windows. It picks the filesystem's answers and what a power cut keeps of each kind of sync. The choice comes from the argument, never from the host, so a test checks the same thing on every machine.
conformance replays scenarios, each a short sequence of operations recorded on each kernel. A report of run names its runs as scenarios too: a fault at a place of the first run.
How a check picks its faults, the contracts a program meets, its reports and replay, its runs and the disk model a power cut follows come after.
The type for what a program or a filesystem under test runs with: the capabilities of Eio.Stdenv, each simulated, so code written against the accessors of Eio.Stdenv takes it unchanged. Eio_chaos.env says what each is.
fs is the root of a fresh in-memory filesystem, the one that fails, and cwd the same directory, confined to itself as eio_posix confines a process's (an absolute path or link and a ".." above it are refused with Permission_denied). Both run as Eio_fs_mem.root, and Eio_fs_mem.as_user runs them as another user. net refuses each operation, unless another device gives it, as the network of eio-net does;clock and mono_clock start at 0., and a wait takes no time (runs);secure_random and the default Random state draw from a fixed seed, whatever ~seed: the first run and each run with a fault draw the same, and a restart draws others, as a new process does;stdin is empty, and stdout, stderr and Eio.traceln are kept in memory: a report shows the last 2 KiB the program printed;domain_mgr runs its work in the calling fiber, on the run's own scheduler;process_mgr starts nothing: spawn and pipe raise Eio.Io with Not_available;debug is Eio's, backend_id is "chaos", and chaos is the run's Eio_chaos.t, for a device the program builds over others.A program that reaches the system other than through env, by Unix, the standard library's channels, Random.self_init, Domain.spawn or Sys.command, escapes the check. Built with hermetic (open! Hermetic), such a call does not compile, so what differs between two runs can only be state the program kept in memory from an earlier one. Without it, a direct call escapes, and when it makes two runs with no fault differ in the kinds of operation they make, the check fails with a report that names the first difference and its likely causes.
val run :
?os:[< Eio_chaos.os ] ->
?budget:int ->
?seed:int ->
?replay:string ->
?setup:(env -> unit) ->
?inspect:(env -> unit) ->
(env -> unit) ->
unitrun ~os ~budget ~seed ~replay ~setup ~inspect app runs app once with no fault on an in-memory filesystem that answers as os (defaults to `Linux) does and holds what setup (defaults to doing nothing) made there. It then runs app again many times, each time from that state, with one fault:
EACCES or EROFS on an open that may write or a directory change, and ENOSPC on one that makes a file or an entry, EACCES or EIO on an open for reading or a stat, ENOSPC or EIO on a write, EIO on a read, a truncation or a sync;EIO and the page cache drops the pages it could not write: a later read of them sees what the disk holds;ENOSPC;After each of those runs, run runs app again with no fault, on what the filesystem then holds, as a machine restarts a service. After a power cut, that is what the disk kept.
How the faults are picked. run tries every fault above at every operation of its first run, but of the many reads and writes of one file it takes the first three, the last and one of each size. Then it draws more runs from seed (defaults to 0) within budget (defaults to 20_000), as Exploration says.
Fault.t.Swap_symlink is not among the faults: it fails ordinary code that stats then opens a path in a directory no attacker can write. Security-sensitive code asks for it with wrap.
When it passes. run passes iff:
Eio.Io, an assertion of app included. A power cut ends the run where it falls;inspect (defaults to doing nothing), called on the filesystem as a fresh process, returns;An Eio.Io error that escapes a run with a fault, from one fiber or several, neither passes nor fails it: the disk did fail, so what decides is the state the run left, which the last two rules check. A restart alone cannot see lost data, as the program writes it again: ack is how the program says what it promised, and inspect checks what only the test knows, such as the records a log must still hold.
The filesystems live in memory and their clocks jump to the next timer, so a check of a small program takes milliseconds.
disk () is the disk of run as a device of Eio_chaos.run, to test a program on it and other simulated devices at once: an in-memory filesystem that answers as the ~os of the check does, with the faults and power cuts above. run app is Eio_chaos.run [ disk () ] app, but for the name its reports give.
val conformance :
?os:[< Eio_chaos.os ] ->
?user:Eio_fs_mem.user ->
?under:Eio_chaos.device list ->
?budget:int ->
?seed:int ->
?replay:string ->
(sw:Eio.Switch.t -> env -> _ Eio.Path.t) ->
unitconformance ~os ~user dir runs each scenario recorded on the kernel of os (defaults to `Linux) in a directory of mode 0o755 it makes in a fresh, empty directory dir ~sw env, opened as a sandbox, as the recorder ran it below its current directory, and compares each answer with the kernel's. sw ends with the scenario, so what dir opens under it is closed there.
Each scenario runs in an event loop of its own, as a run of run does, given an env whose fs is a fresh in-memory filesystem that answers as os does. A filesystem built on other capabilities, such as a wrapper over a directory, takes them from that env, so its runs are as deterministic as a program's. A directory on the host works too: its calls run one at a time in the scenario's loop.
let check_mem () =
Eio_fs_chaos.conformance (fun ~sw:_ _env -> Eio_fs_mem.v ())A scenario is a short sequence of Eio.Path operations: names that differ by case or Unicode form, names and paths at their length limits, opens of each kind, symbolic links and their chains, renames, removals, and the permission rules a user meets. An answer is a result (the bytes read, the entries listed, a file's kind, size or mode) or an error, which compares by the class Eio gives it (Not_found, Already_exists, Permission_denied and so on), else by its errno.
A report writes each operation as the traces do, a verb and its arguments, paths relative to the scenario's directory:
write p d saves d in p, creating or truncating it, and read p loads p; readd p is one read of up to 16 bytes, and pwa p d off writes d at off in p opened for appending;open p flags opens p for reading with r, else for writing, with x exclusive, t truncating, c creating if missing and a appending, and not following a final link with n;opendir p, lsd p (lists p opened as a directory) and ls p;mkdir p [mode], unlink p, rmdir p, rename a b, symlink t p, readlink p, chmod p mode, lchmod p mode and chgrp p, which gives p the user's supplementary group;stat p and lstat p answer the kind and size, mode p the mode, and group p whether the group is the user's own, the supplementary one or another;chain n makes n links l1 to ln, each to the one before, the first to f;hold p and holddir p keep p open as a file or a directory until close p or the scenario's end, cd p opens p as a directory that confines the operations after it, and mkdirs p is Eio.Path.mkdirs;p{s}n is p followed by s repeated n times, \NNN the byte of decimal code NNN, and @p is p under the scenario's directory by its absolute path.The answers were recorded as a user other than root, with the umask 0o022 and a supplementary group. user is the Eio_fs_mem.user dir's operations run as: with it, every scenario runs, and a scenario that changes a file's group changes it to the first of user's supplementary groups. Without it, the scenarios that need a user other than root, or a supplementary group, are left out. Eio_fs_mem.v passes for the os it is given.
A scenario that names a path by its absolute spelling, as a link's target, takes it from Eio.Path.native of its directory; where that has none, as on Eio_fs_mem but under `Windows, the scenario is left out.
On Windows only eio_windows was recorded, as OCaml's Unix reaches the C runtime there, not the kernel: the answers are eio_windows 1.7's on NTFS, as an administrator without user, and as a basic user, who may not make a symbolic link, with it.
Where Eio defines an answer the kernel cannot give, as for a path Eio resolves one component at a time, the expected answer is the one eio_posix gave on that kernel. eio_posix itself does not pass: it resolves a trailing slash, "." and ".." itself, where the kernel answers otherwise, and differs on 13 of the 115 scenarios on Linux and 21 on macOS. A filesystem that wraps eio_posix differs there too.
conformance runs every scenario, whatever the earlier ones answered. In a scenario it stops at the first answer that differs, as the later ones depend on it. It blocks its caller, as run does.
Over other devices. A filesystem built over other devices, as a remote filesystem over a network, takes them from its env: under (defaults to none) are the devices it holds, with a quiet disk as its fs unless one of them is a disk. Its scenarios first run as above, each on healthy devices. Then, if under is not empty, conformance is Eio_chaos.run of a program that runs every scenario in turn, in one event loop, with budget, seed and replay as run takes them, and the faults the devices of under allow, but no power cut. It passes iff each run does:
Eio_chaos.faulted), an answer is the kernel's, or an error eio-fs injects, which the fault below may explain: a missing file, a refused access, a read-only or full filesystem, or an input/output error. At the first such error the scenario ends, as the later answers depend on whether the operation took effect, and so does one whose directory cannot be made;So a hang fails, an error of another device that leaks, such as Connection_reset or End_of_file, fails, and so does a success that differs from the kernel's, as a short read given as the whole file. A device that turns a harmless fault below into an error of its own passes. A report is that of run, named after conformance, and its replay line runs that case alone.
type Eio.Exn.Backend.t += | Injected_fault of stringThe backend code carried by the errors a wrapper raises, the Not_found and Permission_denied of a refused access included: the errno's name and the path, as "EIO /f". So an injected error is an Eio.Io error of the shape a disk's error has.
wrap f base is base with f asked about each operation before it runs. The operation meets the fault f answers. f may also act: count, log, change base, or cut the power with Power.cut. That is how a test measures the reads a scan makes, or forces an interleaving at one operation. base may be in memory or on disk. Files and directories opened through the wrapper stay wrapped.
Injected errors are ordinary Eio.Io exceptions: the code under test cannot tell them from those of a disk.
ack env file records that the program has promised that file is durable as it is now, as a database acknowledges a commit. file is a path of env#fs or env#cwd. After each fault, before the restart, run compares each acknowledged file with what it held at its last ack. A file the run opened for writing, wrote, truncated, renamed or removed since, or whose directory it renamed or removed, is not compared: inspect judges it. An ack after a power cut does nothing.
let app env =
let config = Eio.Path.(env#fs / "config") in
Eio.Path.save ~create:(`Or_truncate 0o644) config "new";
Eio_fs_chaos.ack env configThis app syncs neither config nor its directory, so it fails the check: a power cut loses the write, and config no longer holds "new".
A check runs the program once with no fault and records its operations. Its scenarios are positions in that run, in two tiers:
A check never passes on a partial linear tier. One that needs more runs than its budget fails and says so: shrink the program or raise ~budget. The first run stops as soon as the tier needs more, so a program that polls for ever costs no more than the budget allows, and the report shows its trace up to there.
At most two faults per run, or one lasting condition. A program that tries an operation three times gets past two faults, but no retry gets past a lasting condition: app must accept that any operation fails, and may demand only that the program handles it.
A lasting condition reaches every operation. A filesystem that turns read-only, full, or denies access does so for every operation from then on, the retries included. A program that retries at once spins, and a run fails as a hang when, with no success in between and its clock still but for the time operations take:
A batch that tries each item up to three times and drops what fails passes, and so does a pool that tries up to 1000 names for each name its first run made. No check can tell a spin from a loop that ends later, so one that tries more often than that is taken for a spin.
A restart is a fresh process. It runs on the directory as the fault or the disk left it, and so does inspect: neither may read state an earlier run left in memory.
Each run makes the same kinds of operation. A check aims a fault at an operation by its kind and its rank among the operations of that kind, so the names a program gives its files may change from run to run, and a report says where two runs with no fault named different files. Two runs with no fault that differ in the kinds of operation they make, or in their order, fail the check: env says what can make them differ.
Do not catch Eio.Cancel.Cancelled. A power cut raises it from the operation it falls before, and from every one after, as in a cancelled fiber, and once every fiber waits it cancels them all, as nothing runs after a power cut. A run that makes 1000 more operations after the cut fails as a hang.
A check is deterministic. A failing run raises Failure with:
Eio_fs_chaos.run ~replay:"..." app call that runs that scenario again, which passes ~os:`Macos for a check under macOS.A report's size is bounded by the program's structure, not by how long it spun:
A replay:
The report of a check run inside another is text in the outer one.
The sampled tier draws its scenarios from a seed, the same one on every check unless ~seed gives another, so a check meets the same faults each time. A replay names its scenario, not the draw that found it, so it needs no seed.
A check reads no environment variable and prints nothing. A test that wants a larger budget at night, or other faults in the sampled tier, reads its own:
let check app =
let int name = Option.map int_of_string (Sys.getenv_opt name) in
Eio_fs_chaos.run ?budget:(int "CHAOS_BUDGET") ?seed:(int "CHAOS_SEED") appEach run, with its restart, has its own event loop, inside or outside another one. A check runs to its end before it returns, so it blocks the event loop it is called from: the other fibers of that loop wait for it.
Each run's clocks start at 0. and move to the next timer when every fiber waits, so a wait takes no time. Each turn, a fiber resumed by a yield, another fiber or a timer, also moves them by 1 us, as CPU time passes between two. Five guards stop a run that does not end:
An operation takes time, and other fibers run meanwhile: a sync of a file or a directory 1 ms, any other operation 1 us, and one that fails none. Its effect lands at the end of that time. A sync covers the changes made before it: a write, a change of mode or owner, or an entry another fiber makes while it runs stays unsynced: Linux's fsync(2) promises nothing of them.
A cut leaves a state ext4 with data=writeback may leave. Under ~os:`Macos the same rules hold, but for the syncs Power lists.
Lengths.
Pages and sectors.
Stale data.
'7' when no block was freed.Failed syncs. A failed sync loses the file's unsynced writes, though some of their sectors may already be on disk. The page cache may keep them or evict them: before the cut, a read after a failed sync sees them as kept, or, in a scenario of its own, as the disk holds them.