Module Eio_net_chaosSource

Network faults, for tests of Eio programs and of Eio networks.

Eio_net_chaos has two checks, one function each:

Both raise Failure with a report when the check fails, as an assertion does, so they are the body of a test in any test runner. run is Eio_chaos.run on one device, network: a program that also needs a disk is tested with Eio_chaos.run [ Eio_fs_chaos.disk (); network () ]. A test that wants one fault at one place builds it from the building blocks: wrap for a network, pair for two connected streams.

run, conformance and wrap take ?os, as Eio_net_mem.v does: `Linux, the default, `Macos or `Windows. It picks the answers expected and the kernel's timings the faults follow. The choice comes from the argument, never from the host, so a test checks the same thing on every machine.

conformance replays two kinds of recording: sequences, each an order of a few socket operations, and scenarios, longer programs written by hand, such as the reuse of a port or a name lookup. A report of run names its runs as scenarios too: a fault at a place of the first run.

How a check picks its faults, the contracts a program meets, its reports and replay and its runs come after.

The type for what a program or a network under test runs with: the capabilities of Eio.Stdenv, each simulated, so code written against the accessors of Eio.Stdenv takes it unchanged. Eio_chaos.env says what each is.

  • net is an in-memory network of Eio_net_mem, the one that fails, and fs and cwd refuse each operation, unless another device gives them, as the disk of eio-fs does;
  • clock and mono_clock start at 0., and a wait takes no time (runs);
  • secure_random and the default Random state draw from a fixed seed, whatever ~seed: the first run and each run with a fault draw the same, and a restart draws others, as a new process does;
  • stdin is empty, and stdout, stderr and Eio.traceln are kept in memory: a report shows the last 2 KiB the program printed;
  • domain_mgr runs its work in the calling fiber, on the run's own scheduler;
  • process_mgr starts nothing: spawn and pipe raise Eio.Io with Not_available;
  • debug is Eio's, backend_id is "chaos", and chaos is the run's Eio_chaos.t, for a device the program builds over others.

A program that reaches the system other than through env, by Unix, the standard library's channels, Random.self_init, Domain.spawn or Sys.command, escapes the check. Built with hermetic (open! Hermetic), such a call does not compile, so what differs between two runs can only be state the program kept in memory from an earlier one. Without it, a direct call escapes, and when it makes two runs with no fault differ in the kinds of operation they make, the check fails with a report that names the first difference and its likely causes.

Testing a program

Sourceval run : ?os:[< Eio_chaos.os ] -> ?budget:int -> ?seed:int -> ?replay:string -> ?setup:(env -> unit) -> ?inspect:(env -> unit) -> (env -> unit) -> unit

run ~os ~budget ~seed ~replay ~setup ~inspect app runs app once with no fault on an in-memory network that answers as os (defaults to `Linux) does and holds what setup (defaults to doing nothing) left there. It then runs app again many times, each time on a new network prepared the same way, with one fault:

  • a connection is reset at both ends, as on TCP, or one end closes it in the middle of a message, or it goes silent for good, as when the peer's host loses power;
  • a connect is refused, times out after 133 seconds, or finds no route to its host; an accepted connection is reset before its first byte, as when the client left first;
  • a datagram is lost, duplicated, held back behind the next, replaced with noise from the same sender, or delivered 50 ms late;
  • for the whole run, every read returns one byte, or every write takes one byte, until its end has moved 4 KiB; or every read waits 5 ms and takes what several writes sent; or every connection queues 32 KiB each way, or the kernel's default for its family if that is smaller, so a writer whose reader does not read blocks;
  • the outage, for the whole run: every connect is refused and every datagram lost. is_outage is then true.

app is everything on the network: a client under test comes with the server it talks to, and a server with a client. Every one of them meets the faults. A Unix-domain socket meets only what AF_UNIX can do, as it is reliable and ordered (unix(7)): a connect may be refused, but neither times out nor finds no route; a peer may close, crash or hang, and its reader still gets every byte whose write returned; reads may take what several writes sent, and a write of up to 32 KiB arrives whole. No datagram is lost, duplicated, reordered or garbled, and the outage refuses connects only.

After each of those runs, run runs app again with no fault, on the same network, healed, as a machine restarts a service. The network keeps what the run left: a port still held after a close, in TIME_WAIT.

How the faults are picked. run learns the operations of app from its first run. It tries every fault above at every operation of that run, but for the many connections alike, of which it takes the first three, the last and one of each shape, and the many reads, writes and datagrams of one end, of which it takes the first three, the last and one of each size. A cut in the middle of a stream falls where the peer's writes end. Then it spends a fortieth of budget (defaults to 20_000), or half as many runs as it has run if that is more, on runs drawn at random from seed (defaults to 0): the other connections, reads, writes and datagrams, and two faults in a run, the second anywhere in the run the first one changed, such as the second connection after a reset. Another seed draws other runs. Exploration says more.

Fault.t.No_descriptor is not among the faults: Eio.Net.run_server stops on it, so a test that wants it asks for it with wrap.

When it passes. run passes iff:

  • the first run returns, and a second run with no fault makes the same kinds of operation in the same order. Where only their connections differ, the check goes on, and a report says where;
  • no run hangs or raises an exception other than Eio.Io, End_of_file and Eio.Time.Timeout, alone or together in an Eio.Exn.Multiple, an assertion of app included. A program with no timeout hangs on a silent peer;
  • after each fault, inspect (defaults to doing nothing), called on the healed network, returns;
  • each restart returns: the program can listen and connect again.

An Eio.Io error, an End_of_file or an Eio.Time.Timeout that escapes a run with a fault neither passes nor fails it: the network did fail, so what decides is whether the program works again once the network does, which the restart checks. A program that must succeed despite a fault says so with its own assertion, and is_outage tells it when it may give up.

Under `Macos a connect that times out takes 75 seconds, TIME_WAIT lasts 30, and a Unix-domain crash reads as end of file at both ends, as macOS reports no reset on AF_UNIX. Under `Windows it takes 21 seconds, as Windows documents its two SYN retransmissions, which no trace records, and there is no TIME_WAIT. The networks live in memory and their clocks jump to the next timer, so a check of a small program takes milliseconds.

  • raises Failure

    with a report of the first failing run: the fault, the operations of the run, the error, what the program printed, and the line to paste that runs that case again, as Eio_net_chaos.run ~replay:"net/85d85f49/connect:1.timeout" app. replay reruns that one case, and its restart, whatever the seed that found it. To find it, a replay may first run app a few times, so the failing run is the last one.

  • raises Invalid_argument

    if budget is not positive, or replay is not a replay string of run.

Sourceval network : unit -> Eio_chaos.device

network () is the network of run as a device of Eio_chaos.run, to test a program on it and other simulated devices at once: an in-memory host with a loopback interface that answers as the ~os of the check does, with the faults above. A restart keeps the ports held in TIME_WAIT; a power cut, which another device may bring, keeps nothing. run app is Eio_chaos.run [ network () ] app, but for the name its reports give.

Sourceval is_outage : env -> bool

is_outage env is true iff env is that of the outage run of run. A program that must succeed under any single fault may give up only then. It is false in every other run, in a restart, in inspect, and for an env no run of run made.

Testing a network

Sourceval conformance : ?os:[< Eio_chaos.os ] -> ?under:Eio_chaos.device list -> ?budget:int -> ?seed:int -> ?replay:string -> (sw:Eio.Switch.t -> env -> _ Eio.Net.t) -> unit

conformance ~os net runs each sequence of socket operations recorded on the kernel of os (defaults to `Linux) on a fresh network net ~sw env, and compares each answer with the kernel's. sw ends with the sequence, so what net holds under it is released there.

Each sequence runs in an event loop of its own, as a run of run does, with a env of its own whose devices are simulated: a network built on capabilities, such as a userspace stack that needs a clock or entropy for its ports and sequence numbers, takes them from it, and its answers are the same on every run. Its net is an in-memory network that follows os, for a network that carries its traffic over another one, as a tunnel does. The host's own network does not pass through conformance: only the host's event loop drives it.

let check_mem () =
  Eio_net_chaos.conformance (fun ~sw env ->
      Eio_net_mem.v ~clock:env#mono_clock ~sw ())

A network is a whole Eio.Net.t: the two ends of each connection are made by its own listen and connect, so a sequence sees what a close at one end does at the other. The sequences cover, over TCP, UDP and Unix-domain sockets, on IPv4 and IPv6: every order of up to four of listen, connect, accept, read, write, shutdown and close, and of up to three datagram operations; then binds, backlogs and the reuse of an address after a close, Unix paths, name resolution and socket options.

An answer is a result, "would wait" for an operation that does not complete at once, or an error. Each step has the set of answers os allows: the one the kernel gave and, where the answer depends on the host's configuration (its addresses, name server, services database or host name) or on how the kernel counts a buffer's memory, the answers other hosts give. An answer passes iff it is in the set. Errors compare by the class Eio gives them (Connection_reset, Connection_failure, Not_found), else by their errno, and an error of a backend that carries no errno is any error. Eio_net_mem.v passes for the os it is given.

conformance runs every sequence, whatever the earlier ones answered. Of a sequence it reports the first answer outside its set only, as the later ones depend on it. It leaves out the sequences only a prepared host can answer: those that wait 60 seconds for a port to be free again, those that use up every port, and those that send off the loopback interface; and, under `Macos, those whose answer depends on timing, of which the traces hold one answer: 21 scenarios, and the TCP sequences that read or write after an end shut its receive side while bytes or an end of stream were on their way to it, or after bytes went to an end that had shut it or closed, as whether the reset macOS then sends has arrived depends on when the operation runs; and, under `Windows, the scenarios eio-net.mem does not model (socket options' values, the addresses a connect or a datagram reaches across families, the edge addresses and the names getnameinfo gives). Under `Windows an answer where eio_windows breaks Eio's contract (a refused connect that never returns, a read that waits after a receive shutdown, an unclassified WSAESHUTDOWN, a datagram receive that raises EMSGSIZE, a write of no bytes on a closed flow, a raw Unix.Unix_error, lookups, abstract Unix paths and reuse_addr on a Unix path) is allowed the answer the contract gives, and the report names each such defect it allowed. Unix paths are made in the current directory.

An operation is "would wait" if it has not completed after 64 turns of the event loop, as the recorder took it, and a sequence that has not ended after 10,000 turns answers "timeout". Each turn takes 1 us of a simulated clock, so eio-net.mem passes in about three seconds.

Over other devices. A network built over other devices, as a tunnel over another network, takes them from its env: under (defaults to none) are the devices it holds, with a quiet network as its net unless one of them is a network. Its sequences and scenarios first run as above, each on healthy devices. Then, if under is not empty, conformance is Eio_chaos.run of a program that runs 32 of the scenarios, drawn from seed (defaults to 0), in one event loop, as the sequences and scenarios are too many to replay at each of its runs, with budget and replay as run takes them, and the faults the devices of under allow. It passes iff each run does: until a device fails, every answer is the kernel's; once one failed (Eio_chaos.faulted), an answer may differ, as the faults eio-net injects also lose, duplicate, reorder and garble datagrams and silence peers, but it names no error a network does not give, such as Not_found or Invalid_argument, and the run does not hang. The restart replays nothing, as a network keeps its ports across it and the scenarios name fixed ports. This checks less than with no fault: a network that loses what it was given passes there.

  • raises Failure

    if an answer is outside its set, with a report (Eio_chaos.fail): a table with a row for each step and answers that differ, the step, the first sequence that gave them, its operations up to the one that differs, and how many more did, the answer allowed, or the set of them, and the one given, as connect, tcp4 accept; close S; close S; connect (and 21,360 more), ok and Refused; then how many passed, differed and were left out, and why. Past the first hundred rows it gives the count alone. A step names its end of the connection: C the one that connected, S the one that accepted.

Building blocks

Sourcetype Eio.Exn.Backend.t +=
  1. | Injected_fault
    (*

    The backend code carried by the connection errors a wrapper raises, so an injected Connection_reset or Connection_failure is an Eio.Io error of the shape a socket raises.

    *)
Sourcemodule Op : sig ... end

The operations a wrapper sees.

Sourcemodule Fault : sig ... end

The faults a wrapper injects.

Sourceval wrap : ?os:[< Eio_chaos.os ] -> clock:_ Eio.Time.clock -> (Op.t -> Fault.t option) -> _ Eio.Net.t -> [ `Generic ] Eio.Net.ty Eio.Resource.t

wrap ~os ~clock f net is net with f asked about each operation before it runs, on the connections and the datagram sockets it makes. The operation meets the fault f answers; clock times stalls and timeouts. f may also act: count or log. getaddrinfo, getnameinfo and datagram sends pass through. os (defaults to `Linux) is the kernel whose rules the faults follow, as run says: pass the one net follows.

The network, its connections and its datagram sockets may be used from several domains at once. f is called by one of them at a time, in the order of Op.t.index, while a lock is held: it must not wait, and must not use the network. If f raises, the operation raises that exception and does nothing else: an accept leaves its connection queued for the next, and a receive its datagram. The operation's index and its socket number stay used.

A fault reaches both ends of a connection when net is an Eio_net_mem network, which tells the wrapper which accepted socket a connect made. On another network the wrapper sees the two ends of a connection as two sockets, numbered apart, and a fault shuts down the socket of the end that meets it: a reset on a read gives the real peer end of file, and a reset on an accept gives the real client ECONNRESET once it sends. An end of stream on a read shuts this end's receive side, and the real peer sees nothing: on Linux this end reads what is queued, then end of file, and on macOS it reads end of file at once, as macOS drops what is queued.

After the same operation fails 10,000 times on the same target while clock stands still, with no success there in between, the next attempt raises Failure, as the program that makes them spins for ever, and after 20,000 it waits for ever. An end of file counts only against the connection it ended: a read of it again and again spins, a reconnect after each does not.

A read or a write after the flow's own close raises Invalid_argument before f is asked, as Eio's own sockets do, and a read under way as another fiber closes the flow hands up what arrives, as on eio_posix. A read takes from the socket no more than it returns, so the bytes not yet read stay in the socket's queue, where a shutdown or a close finds them. A close drops the datagrams a socket holds back, and so does the end of the switch a socket was made under, which closes it. With an f that answers None, each operation does what it does on net, but for that guard.

Sourceval pair : ?os:[< Eio_chaos.os ] -> clock:_ Eio.Time.clock -> (Op.t -> Fault.t option) -> [ `Generic ] Eio.Net.stream_socket_ty Eio.Resource.t * [ `Generic ] Eio.Net.stream_socket_ty Eio.Resource.t

pair ~os ~clock f is a connected pair of in-memory streams, the client end first, with f asked about each read and write of either end, as wrap asks: socket is 1. Its streams follow os (defaults to `Linux), as Eio_net_mem.pair takes it. A fuzzer drives it from its input.

Exploration

A check runs the program once with no fault and records its operations: its connects, accepts, reads, writes and receives, in order. Its scenarios are places in that run, chosen from its structure.

The structural elements of a sequence are its first three, its last, and the first of each class. The length class of a length is that length rounded down to a power of two.

The scenarios come in two tiers:

A check never passes on a partial linear tier. One that needs more runs than its budget fails and says so: shrink the program or raise ~budget. The first run stops as soon as the tier needs more, so a program that polls for ever costs no more than the budget allows, and the report shows its trace up to there.

Contracts

At most two faults per run. A program that tries three times always succeeds, so app may demand success. A broker that fans a message out to three subscribers may demand that one of them gets everything: two faults reach two subscribers at most. The second fault of a pair may be a mild one, a read or a write that moves one byte, or a connect or a datagram held back for 50 ms, and it counts as one of the two. A lasting condition runs alone and counts as one fault, though it reaches every connection of the run.

Every program on the network meets the faults, the test's own peers included. A connection may go silent for good, as when a host loses power, and no close or reset then ends a wait on it, only the kernel's keepalive or retransmission timeouts: give each read a timeout, and each write that may fill a socket's buffer, in test servers and clients alike. Write test servers with Eio.Net.run_server ~on_error:ignore, and parse strictly: a peer that takes a partial line for a whole one turns an injected end of stream into a wrong answer.

Let time pass between retries. A program that reconnects at once, with no sleep, spins under the outage, and fails as a busy loop or a hang (Runs).

Ports are held as on Linux. The end of a connection that closes first holds its port for 60 seconds of the run's clock, in TIME_WAIT. A server that listens again on the same port, within a run or in the restart that follows it, fails with an address in use, as it would on Linux: listen with ~reuse_addr:true, as a real server does.

A restart is a fresh process. It runs on the network the fault left, healed, and so does inspect: neither may read state an earlier run left in memory. They share the run's event loop, its clock and its limits (Runs).

Each run makes the same kinds of operation. A check aims a fault at an operation by its kind and its rank among the operations of that kind, never by an address, so the ports and socket paths a program draws may change from run to run, and a report says where two runs with no fault used other connections, on its diverged line. Two runs with no fault that differ in the kinds of operation they make, or in their order, fail the check: env says what can make them differ.

Assert latency to catch a missing timeout. A connect that times out takes 133 seconds, as on Linux, and then fails: a program without a connect timeout waits for it and still succeeds. A test that bounds its program's latency reads env#clock before and after.

What a check does not catch. A bug that needs a datagram delayed with others, or for longer than 50 ms, may pass: the linear tier delays one datagram at a time. A program that never fails the calls still waiting on a connection that died passes unless the program waits for every call and checks how each ended. A garbled datagram is random bytes, so a parser that rejects most noise but accepts some may pass.

Reports and replay

A failing run raises Failure with the fault, the operations of the failing run, how it ended when its inspect or restart failed, what it printed, and the Eio_net_chaos.run ~replay:"..." app call that runs that scenario alone, which passes ~os:`Macos for a check under macOS; a failing run with no fault, with its operations. Each operation shows the fault it met, and a fault inside a read or a write says after how many of its bytes it fell. Repeated operations are folded and a long report is cut around its first fault.

The scenario is found by the operation it names and its fault, so a replay still runs after a fix that changes the program's first run: its report then says the scenario was re-matched, and a re-matched replay that passes returns as any passing check does, saying nothing. A replay whose scenario the check no longer has fails, and one whose scenario is not of the form a report names is refused.

The sampled tier draws its scenarios from a seed, the same one on every check unless ~seed gives another, so a check meets the same faults each time. A replay names its scenario, not the draw that found it, and the program's entropy does not depend on ~seed: the replay line needs no ~seed.

A check reads no environment variable and prints nothing. A test that wants a larger budget at night, or other faults in the sampled tier, reads its own:

let check app =
  let int name = Option.map int_of_string (Sys.getenv_opt name) in
  Eio_net_chaos.run ?budget:(int "CHAOS_BUDGET") ?seed:(int "CHAOS_SEED")
    app

Runs

Each run, with its restart, has its own event loop, inside or outside another one. A check runs to its end before it returns, so it blocks the event loop it is called from: the other fibers of that loop wait for it.

Each run's clocks start at 0. and move to the next timer when every fiber waits, so a wait takes no time. Each turn, a fiber resumed by a yield, an I/O completion or a timer, also moves them by 1 us, as CPU time passes between two, and so does each network operation, one that never waits included: a listen, a send, a lookup or a close. A call of Eio_unix.run_in_systhread runs its function at once, in the run's own thread, so the run stays deterministic. Five guards stop a run that does not end: