Module Owl_rllib_llm.Lm_rlSource

RL for language models, following the progression REINFORCE, RLOO, PPO, GRPO and GRPO's variants (Dr. GRPO, DAPO, GSPO, CISPO).

A batch holds group_size completions for each of several prompts, group-major: completion i belongs to prompt i / group_size. Losses are sums over token rows with one weight per row; the weights decide how per-token terms are aggregated (aggregation).

Advantages from outcome rewards, one per completion

Sourceval mean : float array -> float
Sourceval groups : group_size:int -> 'a array -> 'a array array

Split into consecutive groups. Raises Invalid_argument unless the length is a multiple of group_size.

Sourceval batch_mean_baseline : float array -> float array

Rewards minus the batch mean.

Sourceval rloo : group_size:int -> float array -> float array

RLOO: each completion's baseline is the mean reward of the others in its group.

Sourceval grpo : ?eps:float -> group_size:int -> float array -> float array

GRPO: standardise within the group with the sample standard deviation (eps defaults to 1e-4).

Sourceval centred : group_size:int -> float array -> float array

Dr. GRPO: only subtract the group mean.

Sourceval zero_groups : group_size:int -> 'a array -> float

The fraction of groups whose rewards are all equal (zero advantage).

Sourceval clip_fraction : clip_low:float -> clip_high:float -> log_ratios:float array -> advantages:float array -> float

The fraction of ratios outside [1 - clip_low, 1 + clip_high] on the side of their advantage, where the clipped surrogate has no gradient.

KL estimators from one token sampled from pi

Sourceval k1 : log_prob:float -> ref_log_prob:float -> float

Unbiased; can be negative.

Sourceval k3 : log_prob:float -> ref_log_prob:float -> float

Unbiased and never negative; usually lower variance when the policies are close.

Aggregation

Sourcetype aggregation =
  1. | Sequence_sum
    (*

    sum over a completion's tokens, mean over completions

    *)
  2. | Sequence_mean
    (*

    mean over a completion's tokens, mean over completions

    *)
  3. | Token_mean
    (*

    mean over all tokens in the batch

    *)
  4. | Fixed_length of int
    (*

    Dr. GRPO: a completion's sum divided by a constant

    *)
Sourceval row_weights : aggregation -> lengths:int array -> int array -> float array

row_weights a ~lengths sequence: the weight of each row, given every completion's length and the completion each row belongs to.

Training

Sourcemodule Make (Graph : Owl_neural_graph_sig.Sig) : sig ... end
Sourcemodule S : sig ... end
Sourcemodule D : sig ... end