Owl_rllib_llm.Lm_rlSourceRL for language models, following the progression REINFORCE, RLOO, PPO, GRPO and GRPO's variants (Dr. GRPO, DAPO, GSPO, CISPO).
A batch holds group_size completions for each of several prompts, group-major: completion i belongs to prompt i / group_size. Losses are sums over token rows with one weight per row; the weights decide how per-token terms are aggregated (aggregation).
Split into consecutive groups. Raises Invalid_argument unless the length is a multiple of group_size.
Rewards minus the batch mean.
RLOO: each completion's baseline is the mean reward of the others in its group.
GRPO: standardise within the group with the sample standard deviation (eps defaults to 1e-4).
Dr. GRPO: only subtract the group mean.
The fraction of groups whose rewards are all equal (zero advantage).
val clip_fraction :
clip_low:float ->
clip_high:float ->
log_ratios:float array ->
advantages:float array ->
floatThe fraction of ratios outside [1 - clip_low, 1 + clip_high] on the side of their advantage, where the clipped surrogate has no gradient.
piUnbiased; can be negative.
Unbiased and never negative; usually lower variance when the policies are close.
row_weights a ~lengths sequence: the weight of each row, given every completion's length and the completion each row belongs to.