Windtrap 0.2.0 is a complete rewrite of Windtrap. It keeps the shape of 0.1.0's API and rebuilds it on a stronger design: one runner for every kind of test, one seed for every generated value in a run, and failures kept as data that the report renders. It also fixes many correctness problems, listed below.
The main additions are stateful testing and mutation testing.
~domains, the middle of each sequence runs on several domains at once.< into <= or && into ||, and reports each change that no test fails on. Each such change points at a gap in the suite.Windtrap now runs five kinds of test:
expect;expect_file;It measures a suite in two ways: coverage, as with Bisect_ppx, and mutation testing.
The core of a suite reads as before: test, group, equal and witnesses such as int and list keep their meaning, and so do let%expect_test and [%expect]. Around that core, 0.2.0 makes substantial changes, improvements and additions to the API. The API is flat, so lib/windtrap.mli documents all of it in one file.
See doc/manual/migrating-from-0.1.md for how to migrate from 0.1.0.
These changes compile without an error and change what a 0.1 suite does. Each one is also listed under its area below.
"disabled" runs, where 0.1 skipped it. skip () in its body, or --exclude-tag disabled on the command line, leaves it out.Random state is seeded from the path of each test, where 0.1 started every test from Random.init 137.float eps and float_rel, NaN equals nothing, where 0.1 found NaN equal to NaN, and float_rel no longer finds an infinity equal to every float.expect or [%expect] records the failure and returns, where 0.1 raised, so the rest of the test runs.exit inside a test fails that test, and the run continues.output () fails the test under --stream and raises Invalid_argument outside a test, where 0.1 returned "".(test) executable that links the library.-f and -e add up when repeated, where 0.1 kept the last of each.CI and GITHUB_ACTIONS count as unset when they are empty or 0, false, no, n or off.--junit PATH treats a PATH without .xml as a directory and writes <suite>.xml in it.WINDTRAP_COVERAGE_FILE names the dump file itself, where 0.1 used it as a prefix for generated file names.WINDTRAP_UPDATE, WINDTRAP_COVERAGE_LOG and the inline-test=drop cookie are not read.run : ?argv:string array -> string -> test list -> int returns the exit code instead of calling exit; a suite ends on let () = exit (run "mylib" tests) (see Reading the exit code).run's arguments ~quick, ~bail, ~fail_fast, ~filter, ~exclude, ~tags, ~exclude_tags, ~failed, ~list_only, ~seed, ~timeout, ~prop_count, ~format, ~junit, ~stream, ~output_dir, ~update and ~snapshot_dir are removed, and so is type format; the flags, their mirrors or a synthetic ~argv configure a run.unit, where 0.1 ignored the body's result; a body that computes a value ends on an assertion or on ignore.?pos and ?here are replaced by ?__POS__ on every constructor and every verb, and type here is removed; ~__POS__ passes the call's position.Tag is removed; tags are a string list, as in ~tags:[ "net" ], and a group's tags are added to those of every test under it.slow name fn is test with the tag "slow", which --exclude-tag slow leaves out; a test under a slow group can no longer be marked quick."disabled" runs, where 0.1 skipped it, and no tag has a meaning for selection; skip () in the body skips the test, and --exclude-tag disabled leaves it out.group's ~setup, ~teardown, ~before_each and ~after_each are removed; bracket and scoped give each test a resource, and fixture shares one across the run (see Giving each test its own resource).test, group, slow, cases, bracket and scoped raise Invalid_argument when applied if ~timeout is not finite and positive or ~retries is negative, where 0.1 ran ~timeout:0. as a one-second alarm.cases ~name base inputs fn takes a naming function in place of a witness; each child is the test name input under the group base.ftest and fgroup are removed; focus t focuses any test or group, and the tests outside the focus have no row and no count (see Focusing on one test).run refuses a suite that holds a focus under CI, prints windtrap: focused tests committed (focus at <file:line>); remove focus to run under CI and returns 1.run raises Invalid_argument when called while a run executes, as from a test body; two runs in a row in one process are allowed.~__POS__, a declaration or an assertion is located from the call stack, which needs -g, dune's default (see Locating a failing assertion).cases child is a test that -f selects alone, and two inputs with one name make run refuse the suite.test, group, slow, cases, bracket and scoped take ?__POS__, ?tags, ?timeout and ?retries; on a group, timeout and retries apply to each test under it that declares none, as in group ~timeout:30. "integration" tests.xfail ?reason t marks a test or group as expected to fail; its failure leaves the exit code, -x and --failed alone, and a pass fails with expected to fail (<reason>), but the test passed (see Keeping a known bug in the suite).testable ~pp ?equal ?gen ?check (), Testable.gen, Testable.with_gen, Testable.check and Testable.check_result are removed; Testable.make ~pp ~equal takes a required equality and no (), and Testable.structural ~pp compares with Stdlib.( = ).of_equal and contramap are removed from the top level; Testable.of_equal and Testable.contramap replace them.seq, lazy_t, small_int and nat are removed; Testable.contramap List.of_seq (list t) and Testable.contramap Lazy.force t replace the first two, and int the others.Pp is removed from the interface; a printer has type 'a printer, which is Format.formatter -> 'a -> unit.some, ok, error and no_raise are removed; some t e v is equal (option t) (Some e) v, ok t e r is equal t e (require_ok r), error t e r is equal t e (require_error r), and no_raise fn is fn ().raises_invalid_arg m fn and raises_failure m fn are removed; raises (Invalid_argument m) fn and raises (Failure m) fn replace them.float eps and float_rel, NaN equals nothing, where 0.1 found NaN equal to NaN.float eps raises Invalid_argument unless eps > 0.; exact equality is float_exact.float_rel raises Invalid_argument when a tolerance is negative or NaN, or when both are zero.float_rel no longer finds an infinity equal to every float.float eps and float_rel print as float_exact does, where 0.1 printed six significant digits, so a failure never shows two unequal floats alike.Testable.of_equal prints <abstract>, where 0.1 printed <opaque>.slist prints both sides sorted, where 0.1 printed them as given.raises and raises_match let a skip, a timeout, an intercepted exit and assume pass through; 0.1's raises reported a skip inside it as Wrong exception raised, and a raises_match that accepted any exception swallowed it.raises and raises_match report a wrong exception with the raised one's backtrace.raises expects an Invalid_argument, Failure or Sys_error and gets the same constructor with another message, the failure diffs the two messages.expected and actual and marks what changed for every witness, from the two printed values: the changed spans of one-line values, and a unified diff with @@ hunks for multi-line ones (see Comparing two values).~ line marks it unless the report is coloured on a terminal.both sides render as: <v>.~msg prints a line above the values, where 0.1 printed it in place of the headline; the headlines, such as Values are not equal, are removed.file:line, then the source line of the call.uncaught exception: with its backtrace, which run records without OCAMLRUNPARAM=b.Dune__exe__ prefix.float_exact compares floats bit for bit, orders -0. below 0. as its equality tells them apart, and prints the shortest decimal that round-trips.text is a string witness printed verbatim, so two multi-line texts fail with a line diff.less, at_most, greater and at_least compare under the witness's order, which the base witnesses carry and Testable.with_compare gives; a witness without an order raises Invalid_argument.require_some, require_ok, require_error and require_match assert a shape and return its payload (see Unwrapping an option or a result).is_none, is_ok and is_error take ?pp and print the payload they did not want, as <abstract> without it.starts_with ~affix, ends_with ~affix, contains ~sub, not_contains ~sub and in_order ~subs search a string, and a failure says where the needle was or was not found in an excerpt of the haystack.satisfies ?claim t pred v asserts pred v, and mem t x xs asserts that xs holds x; both print the values with t's printer.Exn.invalid_arg, Exn.failure and Exn.sys_error are predicates for raises_match, each with ?substring.Law asserts seventeen textbook laws, such as Law.associative w op (a, b, c), and a failure names the law, states its equation and prints every term it computed; prop "…" Gen.(triple g g g) (Law.associative w op) checks one over drawn values (see Stating a textbook law).prop ?__POS__ ?tags ?timeout ?count ?max_discard ?examples name gen law takes a Gen.t and a law 'a -> unit that asserts with the verbs; a 0.1 boolean law becomes fun l -> is_true (law l).prop', prop2, prop3 and prop4 are removed; prop' is prop, and several inputs are one tuple from Gen.pair, Gen.triple or Gen.quad.~config is removed; ?count, ?max_discard and --seed replace it.Gen.t is abstract and built from the combinators alone; Gen.make_primitive, Gen.no_shrink, Gen.add_shrink_invariant and Gen.find are removed.Gen.oneof, Gen.oneofl and Gen.pure are renamed Gen.one_of, Gen.of_list and Gen.constant; Gen.list_size sg g is Gen.list ~size:sg g, and Gen.string_size sg cg is Gen.string_of ~size:sg cg.Gen.of_list ~pp and Gen.constant ~pp take the printer of the values they list, so a list of chosen values prints without Gen.with_pp.Gen.sized is removed; Gen.bind Gen.nat f replaces Gen.sized f.( >>= ), ( >|= ) and Gen.ap are removed; let*, let+ and and+ replace them.Gen.fix and Gen.delay are removed; a recursive type is generated by a let rec generator over a depth (see Generating a recursive type).Gen.int32_range and Gen.int64_range are removed; Gen.map Int32.of_int (Gen.int_range lo hi) replaces the first, and likewise for Int64.?origin of Gen.int_range, Gen.float_range and Gen.char_range is removed; a range shrinks toward its point closest to 0, or closest to 'a' for Gen.char_range.?ratio of Gen.option, Gen.result and Gen.either is removed; Gen.frequency weighs choices.Gen.float draws finite floats only, where 0.1 also drew NaN and infinities.Gen.float_range low high includes high, where 0.1 excluded it, and sampling it raises Invalid_argument when a bound is not finite.--seed and WINDTRAP_SEED take the token s1: and 16 lowercase hexadecimal digits; any other spelling is a usage error.cover ~label ~at_least cond is cover label cond, which fails the property with never covered: "label" when no passing case marked label. Coverage is judged once every case has run, so a property that fails on a case or gives up lists no label as never covered.(seed s1:…) when a property is selected.counterexample (case N, shrunk K steps): <value> and shows the law's own failure with its diff. A report whose failures include a property or a stateful test has one replay: <command> --seed <token> line above the summary, which reruns the run's selection on the values it drew.Gen.map or Gen.bind prints as computed from <pre-image>.max_shrink defaulted to 100.counterexample may not be minimal.skip or timeout inside a law is no longer shrunk as a counterexample.max_discard cases are discarded, twice the effective count by default.collect and classify labels print as a table in a failing property's block, and under -v for a passing one."prop", and a stateful test "prop" and "stateful".~examples:[ v1; v2 ] runs the law on those inputs first on every run, unshrunk (see Keeping a counterexample as a regression).Gen.such_that p gen draws until p holds, at most 100 times, and then discards the case.Gen.with_pp pp gen gives a generator its printer.Gen.array takes ?size, and Gen.bytes_of generates bytes.Gen.list, Gen.array, Gen.string_of and Gen.bytes_of without ~size draw a length below 64 and about 5 on average; ~size sets a longer one.Gen.int, Gen.int_range, Gen.int32, Gen.int64 and Gen.nativeint draw a corner case with probability 0.1: a range's bounds, its point closest to 0 and that point's neighbours, or a type's 0, 1, -1 and extremes.stateful name commands runs generated programs of calls on a system and on a reference, a model written for the test or another implementation, which judges each outcome of the system, and fails at the first call whose outcomes differ (see Writing a stateful test).command name signature reference system is one operation of the API; a signature takes g @-> … for an argument drawn from g, t ^-> … for a value of t that an earlier call made, and ends in returns w, makes t or chooses w.abstract prefix declares a type whose values only calls make, named q1, q2 under abstract "q"; ?pp prints a value's reference side in a failing program, ?invariant r s runs on every value after every call, and ?release s releases a system side when a program ends (see Releasing what a program made).chooses w compares an outcome the API leaves open: the reference receives the system's outcome and returns or raises the one it accepts (see Checking an outcome the API leaves open).?pre is asked of the reference's arguments when the program runs, and a call it refuses is skipped on both sides and absent from the report.?count programs (default 100) of at most ?steps calls (default 20).~pp (see Reading a failing program).Assert_failure or Match_failure is never an outcome: in a system function it fails the case at that call, in a reference function, like anything a ~pre raises, it breaks the reference, whose failure shrinks apart from the system's and prints as reference of call N of N (~pre of call N of N for a ~pre), and in a chooses reference it is the system's mismatch (see Reading a failure of the model).assume or reject in a command fails the case with assume or reject in a command; a call's legality is its ~pre.push: argument 2 has no printer; attach one with Gen.with_pp.stateful ~domains:n runs the middle of each program on n domains at once, 50 times, and fails when no order of the calls replayed on the reference gives the outcomes the system gave; the test carries the tag parallel (see Testing on several domains).~pre runs only before the parallel calls; stateful ~domains above 1 raises Invalid_argument when every command makes a value or has a ~pre.domain and a result column, then no order of the calls gives these results and the closest order, as the closest order, 2 then 3, differs at call 4: length q1.a replay of the reference differs from this run; the reference must behave the same from run to run.--mutate and --arm a stateful test on several domains runs each program once on the test's domain, so a kill does not depend on a schedule.cannot spawn a worker domain: <message>.never called: "pop" (over 100 passing cases); a call runs only where its arguments resolve and its ~pre holds when a command that a passing case could draw was called by none; a command listed twice is one command, and ~count:0 judges nothing (see Writing a stateful test).snapshot, snapshot_pp and snapshotf are removed, and no file lives under __snapshots__; expect s @@ __POS_OF__ {|…|} or expect_file s "test/name.expected" replaces them.expect and expect_exact take the produced text and a __POS_OF__ literal, as in expect (output ()) @@ __POS_OF__ {|…|}; a failure is located at the literal, and a correcting run rewrites it.capture and capture_exact are removed; a test compares output () with any verb.output () fails the test under --stream with this test requires capture; rerun without --stream, and raises Invalid_argument outside a test, where 0.1 returned "" in both cases.[%expect] node records the test's failure and returns, where 0.1 raised; one run reports every stale expectation of a test, and one correcting run corrects them.WINDTRAP_UPDATE is not read; a (test) stanza runs %{test} --corrected, which writes <file>.corrected beside each file, and diff?s them for dune promote (see Running expectations under dune).WINDTRAP_SNAPSHOT_DIR, WINDTRAP_SNAPSHOT_DIFF_CONTEXT, WINDTRAP_SNAPSHOT_MAX_BYTES and WINDTRAP_SNAPSHOT_REPORT are not read.-u is refused under CI (exit 1), and -u with --corrected is a usage error.[%%run_tests] is removed, and inline tests in an (executable) no longer run at exit; the tests belong in a library with (inline_tests).(executable) that registers inline tests exits 2 with windtrap: registered inline tests were never driven.(test) executable or program that links the library.[%expect], [%expect_exact] and [%expect.output] compile only inside a let%expect_test body.[%expect.unreachable] was in 0.1; an attribute such as [@@expect.uncaught_exn], which 0.1 dropped silently, fails with attribute expect.uncaught_exn is not supported by ppx_windtrap; catch and print the exception before an [%expect].inline-test=drop is not read; dune's inline_tests cookie drops the test forms.-u rewrites each literal and each expect_file file in place, and every accepted test passes.output () and the captured log include what C code wrote with printf, since capture flushes the C stdout and stderr buffers before reading.expect: mismatch, expect: no baseline or expect_file "<path>": no baseline. Under dune its block ends on accept: dune promote <file>; a run by hand ends on one accept: <command> -u line above the summary, over the run's selection.corrections (N):.[%expect] keeps its node.expect_exact correction whose text holds a CR is a quoted literal with the CR written \r, so it passes once promoted.let%expect_test body writes after its last node fails the test as expect: no baseline, and its correction appends ; and an [%expect] node that holds it, as ppx_expect does; blank output passes (see Writing expect tests inside a library).[%expect] or [%expect_exact] node that a run of its test never reaches fails the test with the body returned without reaching this node, and the test keeps no correction; each functor instance is judged alone, and an Expect_test_config.run that never calls the body fails at the body's first node.negative-tests/trailing.ml byte for byte as upstream (15 corrections, 8 byte-identical), and the corrected negative-tests/escaped_strings.ml passes (see test/conformance/RESULTS.md).run --corrected under the suite <lib>/<file>, and exits 1 when a test failed outside its kept corrections. (2).module%test keeps the module's attributes.expect_file actual path compares actual with the file at path, relative to the project root; a missing file is a mismatch that -u corrects by writing the file, and under --corrected it fails the run (see Keeping a baseline in a file).WINDTRAP_PROJECT_ROOT, else the parent of dune's build directory, else the working directory; no marker file is consulted./ does, so reports print paths relative to the root.Expect_test_config, from ppx_windtrap.config, wraps expect-test bodies with run and rewrites captured output with sanitize, and a local module of that name overrides it (see Masking what changes between runs).fixture ?teardown create is acquired by the first call inside a test and released after the last test of the run; calling it outside a test raises Invalid_argument (see Sharing one resource across the run).exit inside a test fails that test with the test called exit and was intercepted; a test must return or raise, never exit the process, and the run continues.Random state is seeded from the test's path, where 0.1 started every test from Random.init 137.bracket's teardown raises is a [teardown] failure listed beside the body's, where 0.1 replaced the body's failure with Fun.Finally_raised.bracket's setup raises is a [setup] failure.Stack_overflow fails its test, and only Sys.Break and Out_of_memory end the run.scoped scope name fn is the test whose body receives the resource a scope such as In_channel.with_open_text path provides; a scope that swallows the body's failure cannot pass the test.subtest name fn runs a named part of the current test and records its failure while the rest runs. Inside a property's law or a stateful test's function, its failure fails the case, which shrinks, and the report names the subtest under the counterexample.current_test () returns the running test's path.temp_dir () and temp_file () make scratch paths the runner removes when the attempt ends.setenv and chdir change the environment and the working directory for the running test, and the runner restores both when the attempt ends (see Using files, variables and a working directory).windtrap: line and the usage and exits 2, where 0.1 exited 1.--format, WINDTRAP_FORMAT, the TAP report and the dot reporter are removed; the report has two verbosities, the default and -v.<suite>: no tests ran: filter "servr" matched none of 15 tests. and exits 2, where 0.1 exited 0.-f and -e repeat, where 0.1 kept the last of each, and a test is kept when its path holds an -f pattern and no -e pattern.--, is an -f pattern.-q/--quick and --bail N are removed; --exclude-tag slow leaves out the slow tests, and -x stops at the first failure and counts the rest as N not run.-l prints the paths of the selection in declaration order, where 0.1 printed every path sorted.-l makes the checks a run makes, so duplicate paths or a focus under CI exit 1.--failed keeps a record per suite, and with nothing recorded it prints windtrap: no recorded failures match the current suite and exits 2.--junit PATH writes the file beside the terminal report, and treats a PATH without .xml as a directory that gets <suite>.xml.CI and GITHUB_ACTIONS count as unset when empty or 0, false, no, n or off.--failed record, and a record it cannot read counts as empty.windtrap: unknown option '--juint'; did you mean '--junit'?.storage: 12 passed, 2 skipped, 1 expected failure in 2.8ms., whose duration is wall-clock time (see Running every suite).output () call, its block closes on the last 10 lines of that output and full log: <path>, where 0.1 showed no captured output.-v prints one row per test, with its status (PASS, FAIL, SKIP with its reason, or XFAIL), path and duration, and no group header lines.[3/15] <path>… names the running test, with or without -v.--slow-threshold SECONDS (default 1) lists under slow tests every test not tagged slow that ran that long; Slowest tests: is removed.flaky tests and counted as (N flaky).WINDTRAP_TAIL_ERRORS and WINDTRAP_COLUMNS, which changed nothing in 0.1, are not read.--color auto honours NO_COLOR and TERM=dumb, and --color takes any case.windtrap: warning: could not write JUnit report to <file>: <reason> and leaves the exit code alone.<log dir>/<suite>/<groups>/<test>.output, overwritten by each run, with no run directories, latest links or Test output saved to line.::error annotation after the group, and the summary is the last line.windtrap: interrupted in <path> and the summary, release the fixtures, and end the process by the same signal.timed out after 0.5s.windtrap:.\xNN.--shard K/N keeps bucket K of N of the selection, by a hash of each test path.WINDTRAP_VERBOSE, WINDTRAP_JUNIT, WINDTRAP_OUTPUT, WINDTRAP_SHARD and WINDTRAP_SLOW_THRESHOLD are new mirrors.--help lists each flag with its mirror.run stopped after <test>: a call on another domain outlived the test's limit above the summary; on Windows, where no limit is enforced, it hangs the run.output, expect, expect_exact, expect_file, collect, classify, cover, a fixture's accessor and the functions of the running test raise a failure when called from a domain other than the one that called run, which fails the running test when it reaches the test's domain, as through Domain.join.(instrumentation (backend ppx_windtrap)) and --instrument-with ppx_windtrap are now ppx_windtrap.coverage, and dune rejects ppx_windtrap as a backend (see Instrumenting a library).windtrap coverage reports coverage.windtrap coverage flags --summary-only, -C/--context, --skip-covered, --coverage-path, --source-path and -j are removed; a positional PATH replaces --coverage-path, and --json replaces -j.--json drops source_available and uncovered_offsets.<build dir>/_coverage/windtrap-<hash>/, and the runner's -o no longer moves them (see Where the dumps are).PATH or an unreadable, corrupt or 0.1 dump makes windtrap coverage exit 1, and a usage error exits 2 behind windtrap:.WINDTRAP_COVERAGE_FILE names the dump itself, where 0.1 took it as the prefix of generated file names.WINDTRAP_COVERAGE_LOG is not read; a dump that cannot be written is one windtrap: warning: line on stderr.windtrap coverage finds the dumps from any subdirectory.coverage: 71.4% (312/437 points); -u keeps the table and adds the uncovered source.--min PCT exits 1 below PCT, and the last line then reads coverage: 71.4% (312/437 points), minimum 80%: FAILED.--lcov prints an LCOV tracefile, which genhtml, Codecov, Coveralls and editor gutters read; GitLab's merge-request view reads neither format windtrap writes.--expect PATH exits 1 unless every source under PATH has coverage data, and --do-not-expect PATH exempts a file or directory from it.--expect names each source without coverage data once, spelled with /.[@@coverage off] also excludes a module binding.(instrumentation (backend ppx_windtrap.mutate)) with --instrument-with ppx_windtrap.mutate compiles every mutant of a library into the build, each off until armed (see Instrumenting a library for mutation).&& and ||, or swaps + and -, and is named <file>:<line>:<col>:<rewrite>.[@mutate off "reason"] and its [@@…] and [@@@…] forms dismiss equivalent mutants; a bare [@mutate off] dismisses every mutant of its expression, in later releases too, and a payload that names a rewrite is refused (see Dismissing an equivalent mutant).--mutate[=PREFIX,…] runs the suite, then each reached mutant in a child process with the tests that reached it, prints each survivor with those tests, and exits 0 (see Testing a suite's mutants).--arm ID runs the suite with one mutant armed and says whether it was killed.windtrap mutants merges the verdict files of the executables and exits 1 when a mutant survived every executable that reached it.--mutate is refused (exit 1) on Windows, in a process that spawned a domain, when the dry run fails or has no mutant in scope, and when the determinism probe disagrees with the dry run.not evaluated with the arm: command that tests it in a new process, and the merge keeps it not evaluated unless an executable killed it (see What a mutation run runs).evaluated outside tests, apart from the lines never reached (see What a mutation run runs).windtrap.prop (Windtrap_prop, with Arbitrary and Prop.check), windtrap.myers and windtrap.clock are removed; windtrap holds Gen and prop.windtrap.coverage (Windtrap_coverage) is replaced by windtrap.runtime, which depends on the stdlib alone.Windtrap.Ppx_runtime is removed, and ppx_windtrap.runtime holds the inline-test runtime.ppx_windtrap declares windtrap as a runtime library, so a stanza preprocessed with it need not list windtrap in (libraries …).ppx_windtrap.coverage and ppx_windtrap.mutate are the instrumentation backends, and ppx_windtrap.config holds Expect_test_config.windtrap binary has the commands coverage and mutants, and an unknown command exits 2.Clock and Myers to an executable.Windtrap is an all-in-one OCaml testing framework that unifies unit tests, property-based tests, snapshot tests, and expect tests under a single API. Instead of juggling multiple testing libraries, Windtrap gives you one cohesive package with a PPX for inline expect tests (ppx_windtrap).
ppx_windtrap with automatic correction.bisect_ppx integration.