Module Cascade.LexerSource

Stage 2 stream: characters -> Token.t (CSS Syntax section 4).

Wraps a Reader.t and produces Token.t values via the section 4.3 tokenization algorithm. Exposes the uniform next / peek / reconsume triple used across parse stages.

The input is already-decoded UTF-8 text. CSS Syntax section 3.2 byte-stream decoding is outside this layer.

Sourcetype t

A lexer stream: a character cursor plus one-token pushback.

Sourceval of_reader : Reader.t -> t

of_reader r wraps an existing character reader.

Sourceval of_string : ?enforce_spec:bool -> string -> t

of_string s builds a fresh reader from an already-decoded UTF-8 string and wraps it. enforce_spec is passed to Reader.of_string.

Sourceval source : t -> string

source t is the full input string the underlying reader was built from.

Sourceval next : t -> Token.t

next t consumes the next token. Returns Token.kind.Eof at end of input. Honours any token pushed back by reconsume.

Sourceval peek : t -> Token.t

peek t is the next token without consuming it. A subsequent peek or next returns the same token.

Sourceval reconsume : t -> Token.t -> unit

reconsume t tok pushes tok back so the next next returns it. The pushback buffer is unbounded -- multiple reconsume calls stack.

Sourceval save : t -> unit

save t records the current position. A subsequent restore replays every token consumed since this save so the next next returns the same sequence again. save/restore/commit stack: nested saves are independent.

Sourceval restore : t -> unit

restore t replays the tokens consumed since the most recent save. Pops one entry off the save stack.

Sourceval commit : t -> unit

commit t discards the most recent save without rewinding. The replay log is folded into the parent save (if any) so an outer restore still sees the consumed tokens.

Sourceval is_done : t -> bool

is_done t is true when no more tokens remain.

Sourceval spec_non_ascii_ident_cp : int -> bool

spec_non_ascii_ident_cp cp is the CSS Syntax section 4.2 predicate: is cp in that section's range list of non-ASCII ident code points? Exposed for serialisers, which hex-escape anything outside it; an escape is read by every parser, so emission stays on this list even though reading accepts any code point >= U+0080 unless ~enforce_spec:true.