owl-rllib

Reinforcement learning in OCaml, built on Owl.

owl-rllib aims to be small, readable and reproducible rather than large: one file per algorithm, explicit random number generators, and every environment and loss checked against a reference implementation (Gymnasium, PyTorch). DQN and PPO match CleanRL's published results on its benchmark environments (see bench/).

Contents

Installation

owl-rllib needs Owl 1.3 or later, which needs OpenBLAS and LAPACKE (e.g. libopenblas-dev liblapacke-dev on Debian and Ubuntu, openblas on Homebrew).

opam

opam install owl-rllib

Docker

The image matrixanger/owl-rllib (tag latest) has Owl's main branch and owl-rllib built, tested and installed, together with the Python packages for the Gymnasium / MuJoCo / VizDoom bridge. It is rebuilt whenever this repository or the Owl image (matrixanger/owl:main) changes.

docker run -it matrixanger/owl-rllib
cd /root/owl-rllib
dune exec examples/classic/a7_actor_critic.exe

Other projects in the container can use (libraries owl-rllib) directly.

From source

git clone https://github.com/owlbarn/owl-rllib.git
cd owl-rllib
opam install . --deps-only --with-test   # Owl and, for the tests, alcotest
dune build
dune test

The API documentation is generated with dune build @doc.

Python, for the bridge

owl-rllib.bridges runs Gymnasium or VizDoom in a Python child process, so it needs Python with those packages; nothing else does. The interpreter is $OWL_RLLIB_PYTHON, by default python3, and tests that need the packages are skipped when they are missing.

The Docker image already has them: Python 3.11 in /opt/venv, with OWL_RLLIB_PYTHON set, so the bridge examples and the MuJoCo benchmarks run as they are.

Elsewhere, install the versions in python-requirements.txt, the ones the tests' reference data was recorded with. VizDoom 1.2.3 needs Python 3.12 or older; with uv:

uv venv --python 3.11 ~/.venvs/owl-rllib
uv pip install --python ~/.venvs/owl-rllib/bin/python -r python-requirements.txt
export OWL_RLLIB_PYTHON=~/.venvs/owl-rllib/bin/python
dune test   # now also runs the bridge tests

Box2D is needed for LunarLander, MuJoCo and imageio for Hopper and the other MuJoCo tasks, and VizDoom for examples/classic/a8_vizdoom.ml; install only what you use.

Quick start

The package has three libraries: owl-rllib (module Owl_rllib), owl-rllib.llm (Owl_rllib_llm) and owl-rllib.bridges (Owl_rllib_bridges). With dune:

(executable (name main) (libraries owl-rllib))
open Owl_rllib
module N = Neural.Make (Owl.Neural.S.Graph)

let () =
  let rngs = Rng.create 0 in
  let policy = N.mlp ~seed:rngs.network_seed ~shape:[| 4 |] ~hidden:[ 64; 64 ] ~outputs:2 in
  let critic = N.mlp ~seed:(rngs.network_seed + 1) ~shape:[| 4 |] ~hidden:[ 64; 64 ] ~outputs:1 in
  let env = Vec_env.create (Array.init 8 (fun _ -> Cartpole.make ~rng:rngs.env ())) in
  let returns = Ppo.S.train ~updates:100 ~rngs ~policy ~critic ~distribution:Policy.S.categorical env in
  Printf.printf "last episode: %.0f\n" returns.(Array.length returns - 1)
OPENBLAS_NUM_THREADS=1 dune exec examples/classic/a7_actor_critic.exe

Set OPENBLAS_NUM_THREADS=1: the networks here are small, and OpenBLAS's default thread count slows them down.

Examples

examples/ holds runnable programs in two tracks, each building on the last:

Each file's header gives the command to run it and what it shows.

Reproducibility

Every random choice goes through an explicit Random.State.t (see Rng): the same seed on the same machine gives bit-identical results (another machine or BLAS build, including the Docker image, gives different but equally valid ones). One exception: training a convolutional network in Owl occasionally differs by one unit in the last place between runs, so the VizDoom example (a8) is not bit-for-bit reproducible.

License

MIT, see LICENSE.