Module Lm.MakeSource

Parameters

module Graph : Owl_neural_graph_sig.Sig

Signature

include module type of struct include Owl_rllib.Neural.Make(Graph) end

Tensors

val scalar : float -> Algodiff.t
val column : float array -> Algodiff.t

A [n; 1] matrix.

val batch : Algodiff.A.arr array -> Algodiff.t

Stack observations along a new batch axis.

val one_hot : int -> int array -> Algodiff.t

one_hot n actions is a [batch; n] matrix.

val values : Algodiff.A.arr -> float array

All elements, in row-major order.

val predict : Graph.network -> Algodiff.A.arr -> float array

The network's output for one observation.

val n_actions : (Algodiff.A.arr, int) Owl_rllib.Ndenv.t -> int

Raises Invalid_argument unless actions are discrete.

Training

val adam : float -> Params.typ

Adam with the given learning rate, betas 0.9 and 0.999.

val update : Graph.network -> Stepper.state -> Algodiff.t -> unit

One optimiser step on a loss computed from Graph.forward.

Networks

Every constructor seeds Owl's generator with seed before initialising the weights, so a seed always gives the same network.

val mlp : seed:int -> shape:int array -> hidden:int list -> outputs:int -> Graph.network

ReLU hidden layers and a linear output, with Owl's default initialisation.

type init = Owl_rllib.Neural.Make(Graph).init =
  1. | Default
    (*

    Owl's default initialisation

    *)
  2. | Orthogonal of float
    (*

    orthogonal weights, gain sqrt 2 for hidden layers and the given gain for the output layer (as in CleanRL's PPO)

    *)
val orthogonal : float -> Init.typ

Orthogonal weights scaled by a gain, like PyTorch's orthogonal_.

val mlp_with : activation:Activation.typ -> init:init -> seed:int -> shape:int array -> hidden:int list -> outputs:int -> Graph.network
val gaussian_mlp : activation:Activation.typ -> init:init -> seed:int -> shape:int array -> hidden:int list -> actions:int -> Graph.network

A network for Owl_rllib.Policy.Make.gaussian whose log_std does not depend on the state: one learned vector, initially 0.

val cnn : seed:int -> shape:int array -> convs:(int * int * int) list -> hidden:int list -> outputs:int -> Graph.network

For NHWC images [height; width; channels]: each (kernel, stride, channels) is a VALID ReLU convolution, followed by ReLU hidden layers on the flattened features and a linear output.

Sourceval vocab : int
Sourceval window : int

Context length: the longest prompt plus every answer token but the last.

Sourceval input_size : int
Sourceval encode : int array -> float array

A context as its left-padded one-hot window, row-major [position; token].

Sourceval features : int array array -> Algodiff.t

Encode a batch of contexts.

Sourceval create : ?hidden:int list -> ?outputs:int -> seed:int -> unit -> Graph.network

Hidden layers default to [128; 128]; outputs (default vocab) can be 1 for a critic.

Sourceval token_log_probs : Algodiff.t -> int array -> Algodiff.t

log pi(action | context) for every row, differentiable in the logits.

Sourcetype sample = {
  1. prompt : int array;
  2. tokens : int array;
    (*

    generated tokens, ending with eos unless cut off

    *)
  3. log_probs : float array;
    (*

    log pi(token) under the generating network

    *)
}
Sourceval generate : ?greedy:bool -> ?max_len:int -> rng:Random.State.t -> Graph.network -> int array array -> sample array

Generate up to max_len tokens for every prompt (default Owl_rllib_llm.Lm_task.max_answer), one batched forward pass per position; sampling by default, the most likely token with ~greedy:true.

Sourceval rows : (int array * int array) array -> int array array * int array * int array

For (prompt, tokens) pairs, one row per token: the context before it, the token, and the index of its pair.

Sourceval accuracy : ?sample:bool -> ?task:Lm_task.task -> Graph.network -> Lm_task.problem array -> float

The fraction of problems answered right, with greedy decoding or one sample per problem; the sampling stream is fixed, so evaluations repeat.

Sourceval sft : ?batch_size:int -> ?learning_rate:float -> ?report:(int -> float -> unit) -> ?task:Lm_task.task -> steps:int -> rng:Random.State.t -> Graph.network -> Lm_task.problem array -> unit

Supervised fine-tuning with teacher forcing: minimise the cross-entropy of every target token given the tokens before it. report step loss is called after every step.