Module Cascade.TokenSource

CSS Syntax Module Level 3 section 4.2: token taxonomy.

Types only; the section 4.3 tokenization algorithm lives in Lexer. Every token carries the source Loc.t it was read from.

Sourcetype hash_flag =
  1. | Id
  2. | Unrestricted

Whether a Hash starts an identifier (#abc) or is unrestricted (#123). Only id-flag hashes are valid as ID selectors.

Sourcetype number_flag =
  1. | Integer
  2. | Number

Whether a number was written as an integer or not.

Sourcetype number = {
  1. value : float;
  2. repr : string;
  3. number_flag : number_flag;
}

The parsed numeric value, its original textual representation, and the integer-vs-number flag.

Sourcetype bracket =
  1. | Curly
    (*

    \{ ... \}

    *)
  2. | Paren
    (*

    ( ... )

    *)
  3. | Square
    (*

    [ ... ]

    *)

The three kinds of balanced bracket character used in CSS blocks.

Sourcetype unicode_range_form =
  1. | Single of {
    1. width : int;
    }
  2. | Range of {
    1. start_width : int;
    2. end_width : int;
    }
  3. | Wildcard of {
    1. prefix_width : int;
    2. wildcards : int;
    }
Sourcetype kind =
  1. | Ident of string
  2. | Function of string
    (*

    Ident immediately followed by (.

    *)
  3. | At_keyword of string
    (*

    @ followed by an ident.

    *)
  4. | Hash of {
    1. value : string;
    2. hash_flag : hash_flag;
    }
  5. | String of {
    1. value : string;
    2. quote : char;
    3. terminated : bool;
    }
    (*

    String literal. quote is the opening quote character (double or single); the spec treats both as equivalent delimiters (CSS Syntax section 4.3.5) but we record it for quote-sensitive rules (e.g. @charset per CSS Syntax section 8.2) and to round-trip the input style. terminated is false when the lexer reached EOF without seeing the closing quote (CSS Syntax section 4.3.5 returns the string token but flags a parse error); the serializer omits the closing quote so the original byte sequence is preserved.

    *)
  6. | Bad_string
    (*

    Unterminated string (newline or EOF before the closing quote).

    *)
  7. | Url of string
  8. | Bad_url
    (*

    Malformed url(...) body (unquoted content with invalid chars).

    *)
  9. | Delim of string
    (*

    Any single Unicode code point not consumed by another token rule. Stored as the UTF-8 byte sequence (1 to 4 bytes).

    *)
  10. | Number_tok of number
  11. | Percentage of number
  12. | Dimension of {
    1. number : number;
    2. unit_ : string;
    }
  13. | Whitespace
    (*

    Any run of whitespace characters.

    *)
  14. | Unicode_range of {
    1. start_value : int;
    2. end_value : int;
    3. form : unicode_range_form;
    }
    (*

    U+XXXX / U+XXXX-YYYY / U+XX?? (CSS Syntax section 4.3.14). The three syntactic forms are normalised to the [start_value, end_value] inclusive range; start_value = end_value for the single-codepoint form. form keeps the typed token shape for non-minified fidelity.

    *)
  15. | Cdo
    (*

    <!-- at top level.

    *)
  16. | Cdc
    (*

    --> at top level.

    *)
  17. | Colon
  18. | Semicolon
  19. | Comma
  20. | Open of bracket
    (*

    Opening bracket of a balanced group.

    *)
  21. | Close of bracket
    (*

    Closing bracket of a balanced group.

    *)
  22. | Eof

Token payload: the section 4.2 variants without the location wrapper.

Sourcetype t = {
  1. kind : kind;
  2. loc : Loc.t;
}

A located token: the section 4.2 payload plus the source range it covers.

Sourceval equal_hash_flag : hash_flag -> hash_flag -> bool

equal_hash_flag a b tests hash token flags for equality.

Sourceval equal_number_flag : number_flag -> number_flag -> bool

equal_number_flag a b tests number token flags for equality.

Sourceval equal_bracket : bracket -> bracket -> bool

equal_bracket a b tests bracket kinds for equality.

Sourceval equal_kind : kind -> kind -> bool

equal_kind a b tests token payloads for structural equality.

Sourceval v : kind:kind -> loc:Loc.t -> t

v ~kind ~loc is a token with the given payload and location.

Sourceval synthetic : kind -> t

synthetic k is a token with payload k and Loc.dummy - for test fixtures and synthetic values that don't come from a real input.

Sourceval pp_kind : kind Pp.t

pp_kind renders just the payload, e.g. <ident foo>.

Sourceval pp : t Pp.t

pp renders a located token, e.g. <ident foo>@[3-6].

Sourceval to_string : t -> string

to_string token renders a located token.