A simple format.

bcfg is essentially based on the format described here: scfg. There is only one construct, the directive, and a directive is made of three things:

• a name, the first word of the line (service, listen, group);
• a list of parameters, the words which follow it on the same line (www.example.org and public);
• a list of children, the directives written between { and }, which are directives in their own right.

A configuration file is therefore a tree of directives, and nothing else. In particular there are no primitive types: 128 and true are strings, exactly like /run/git.sock is. The format does not decide for you whether a number is an integer, a float or a byte count, which is a feature: the program which reads the file is the only one which knows.

The format is line-driven: a line break is part of the syntax. It is not possible to write service www.example.org { listen 80 } on a single line; the opening brace must be followed by a line break, and so must the closing one. Indentation, on the other hand, is free (an issue that has frustrated generations of developers). A comment starts with # and runs to the end of the line.

The distinction between a name and a parameter is deliberately left to the author of the file. In

username dinosaure

the name characterises the value, whereas in

dinosaure as_github_username as_login

the name is the value and the parameters characterise it. bcfg does not impose either reading, and, as we will see, bcfg query is able to walk both.

Words, quoting and non UTF-8 content.

A word which contains a space, a quote, a brace or a backslash must be quoted, with '' or with "". Inside quotes, \xNN denotes an arbitrary byte, which is how a value that is not valid UTF-8 is written down. This is the first difference with scfg:

$ cat >iso8859.cfg<<EOF
> foo {
>   "\xe9ternuer"
> }
> EOF
$ bcfg query 'foo.*' iso8859.cfg \
  | xargs printf "%b\n" \
  | iconv -f ISO-8859-1 -t UTF-8
éternuer

An unquoted word must be valid UTF-8, and bcfg refuses to print a value which is not (it quotes it and escapes the offending bytes instead).

The second difference is the ability to fold a long value over several lines, following RFC 822. This is what the --margin option of bcfg lint uses:

$ bcfg lint --margin 42 long.cfg
foo {
  "a really long long long long long long \
   long long long long directive..."
}

The two forms above are lexical: they disappear once the file is parsed, and the value is a single string in both cases.

Querying a configuration: bcfg query.

bcfg query is to configuration files what jq is to JSON. Its manual page (bcfg query --help) is the reference; this section walks through the same material with the file above.

The mental model is worth stating first, because everything follows from it. A query starts from the list of the top-level directives and every step of the query turns a list of directives into another list of directives. The result is printed as a configuration file again. There is no other kind of value: even a port number comes out as a directive whose name is 80.

Selecting.

A word selects the directives of the current level which bear that name, * selects them all, and . descends into the children:

$ bcfg query default services.cfg
default www.example.org
$ bcfg query 'group.member' services.cfg
member www.example.org
member git.example.org
$ bcfg query 'service.upstream.*' services.cfg
kind tcp
host 10.0.0.2
port 8080
kind unix
path /run/git.sock
kind tcp
host 10.0.0.3
port 8081

Note that a query never descends by itself: group.member does not look into the nested staging group, whereas group.group.member does.

Taking a value.

[N] keeps the N-th parameter of each selected directive (counting from 0). The parameter becomes the name of the result, and the children are kept, so [0] is both "give me the value" and "drop the keyword":

$ bcfg query 'service.listen' services.cfg
listen 80
listen 443
listen 22
listen 80
$ bcfg query 'service.listen[0]' services.cfg
80
443
22
80
$ bcfg query 'group.group.member[0]' services.cfg
static.example.org

Filtering on parameters.

A filter written in parentheses keeps or discards directives; it does not descend and it does not change what is selected, so it can appear anywhere in a query, and several times:

$ bcfg query 'service(public).listen[0]' services.cfg
80
443
80
$ bcfg query 'service(^public).listen[0]' services.cfg
22
$ bcfg query 'service(public)(:^tls).listen[0]' services.cfg
80

foo(pat) keeps the directives having at least one parameter which matches pat, while foo(^pat) is the anti-join: it keeps those having no parameter matching it. The distinction matters as soon as a negation is involved. service(!public) reads as "some parameter of this service is not public", and since every service has a host name, it selects them all:

$ bcfg query 'service(!public).listen[0]' services.cfg
80
443
22
80

"This service is not public" is service(^public).

Filtering on children.

The same two filters exist for the children, with a leading :. They look at the names of the children:

$ bcfg query 'service(:tls).listen[0]' services.cfg
80
443
$ bcfg query 'service(:^tls).listen[0]' services.cfg
22
80

which reads as "the services which define a tls block" and "the services which do not".

Filtering on names.

A filter can also be written before what it applies to, in which case it filters the names:

$ bcfg query '(&,!service,!group)' services.cfg
default www.example.org
$ bcfg query 'group.(!member)*' services.cfg
group staging {
  member static.example.org
}

Patterns.

Wherever a filter expects something to match, it accepts a pattern. A pattern is a word (compared with a plain string equality, there is no globbing), *, or a combination of them:

• !p negates,
• p|q and p&q are the disjunction and the conjunction,
• (|,a,b,c) and (&,a,b,c) are their n-ary forms, which are more readable when there are more than two of them,
• parentheses group, as in service((public|internal)&!internal).

$ bcfg query 'service(internal|public).listen[0]' services.cfg
80
443
22
80
$ bcfg query "service(&,!internal,'www.example.org').listen[0]" services.cfg
80
443

Substitutions.

A pattern may also be computed from the file itself. @(query) evaluates query against the whole document and matches any of the values it produces (its first parameter, or its name when it has none, exactly as query[0] would print it).

$ bcfg query 'service(@(default[0])).listen[0]' services.cfg
80
443
$ bcfg query 'service(@(group.group.member[0])).upstream.port[0]' services.cfg
8081

The first query reads as "the service designated by the default directive", the second as "the port of the services which belong to the staging group". Since a substitution is a pattern, it composes with the operators above: service(!@(group.member[0])) selects the services which no group mentions.

$(query) is an alias of @(query), for those used to jq. @ is preferred because it is not special inside shell double quotes.

Quoting.

Words in a query follow the lexical rules of the format itself. A value which contains a character that is meaningful to the query language (a dot, a bracket, a parenthesis, !, |, &, :, ^, *, @) must therefore be quoted:

$ bcfg query "service('www.example.org').tls.certificate[0]" services.cfg
/etc/ssl/www.example.org.pem

Unquoted, www.example.org would be read as three names separated by dots. Single quotes are handy inside shell double quotes, and the other way around.

Producing JSON.

bcfg query can print its result as JSON, which is the shortest path to jq and to any language with a JSON parser:

$ bcfg query -o json 'group' services.cfg
{
  "group": {
    "$params": "production",
    "member": [
      "www.example.org",
      "git.example.org"
    ],
    "group": {
      "$params": "staging",
      "member": "static.example.org"
    }
  }
}

A list of directives becomes an object keyed by names, a name which appears several times becomes an array, and the parameters of a directive which has children are kept under the $params key. Note that the shape of the output depends on the file: one member gives a string, two give an array. Values are always strings, since the format has no number and no boolean. A small to_entries in jq gives, for instance, a CSV summary:

$ bcfg query -o json 'service[0]' services.cfg \
  | jq -r 'to_entries[]
           | [ .key, ([.value.listen] | flatten | join(" ")),
               (.value.tls.certificate // "no tls") ]
           | @csv'
"www.example.org","80 443","/etc/ssl/www.example.org.pem"
"git.example.org","22","no tls"
"static.example.org","80","no tls"

Large files.

A query which contains no substitution never looks outside the top-level directive which is being examined. Such a query is evaluated in a streaming fashion, one top-level directive at a time, and bcfg query never holds the whole document in memory. As soon as an @(...) appears, the file has to be fully parsed first, because the sub-query may refer to any part of it.

The other tools.

bcfg provides three other commands, which are all built on the same library.

bcfg validate checks that a file is lexically and syntactically correct. It is the tool to reach for in a Makefile or in a CI job:

$ bcfg validate services.cfg
$ echo $?
0

Its real value is the error report. The parser is generated by Menhir, whose ability to introspect the state of the automaton lets us say what was expected and where:

$ bcfg validate - <<EOF
> foo {
> EOF
Error at l2.0-0:
> Missing a subdirective.
 1 foo {
>  EOF

This machinery is available to your own programs: Bcfg.Error exposes the state of the automaton and the last symbol it recognised, and Bcfg.Txtloc re-reads the offending lines. bcfg validate is nothing more than a pattern matching over these two, and its whole implementation fits in bin/bcfg_validate.ml.

bcfg iso checks a property of our own implementation: whatever it is able to parse, it must be able to write back without altering it. It parses a file, re-encodes the value, parses the result again and compares the two:

$ bcfg iso services.cfg
$ echo $?
0

bcfg lint re-indents a file. Indentation carries no meaning in the format, but a project may still want a single style, and the tool follows the spirit of vim's expandtab and shiftwidth:

$ bcfg lint --indent tab --margin 42 services.cfg

Since comments and blank lines are not part of the parsed tree, they do not survive bcfg lint. Keep that in mind before using --in-place on a file whose comments matter.

Reading a configuration from OCaml.

Two libraries are available. bcfg gives the tree as it is, and bcfgt maps that tree onto your own types.

The tree: Bcfg.

Bcfg.parser turns a Lexing.lexbuf into a Bcfg.t, that is a list of Bcfg.directive. A directive is a record with three fields, so everything can be done with pattern matching:

let hostnames (t : Bcfg.t) =
  let fn = function
    | { Bcfg.name = "service"; parameters = hostname :: _; _ } -> Some hostname
    | _ -> None
  in
  List.filter_map fn t

The library depends only on Menhir and the OCaml runtime, so it can be used in restricted contexts such as unikernels, where a file may not even exist: Bcfg.parser only wants a lexing buffer. The other direction is Bcfg.emitter, which renders a tree as a sequence of strings according to an output configuration (Bcfg.Out.config: margin, indentation, escaping). Finally, Bcfg.Stream offers a SAX-like view of the format, for documents which are too large to be held in memory.

The values: bcfgt.

Writing the projection by hand quickly becomes tedious: every field has to be looked up, every number parsed, every error message written. bcfgt does it from a single description, which works in both directions. It is directly inspired by jsont.

Say we want these types for the file at the top of this page:

type level = Quiet | Info | Debug
type tcp = { host : string; port : int }
type upstream = Tcp of tcp | Unix of string
type tls = { certificate : string; key : string option }

type service = {
  hostname : string;
  public : bool;
  internal : bool;
  memory : int; (* in bytes *)
  weight : float;
  log : level;
  listen : int list;
  upstream : upstream;
  tls : tls option;
}

A codec is built by declaring, in order, what the directive contains:

let level = Bcfgt.enum [ ("quiet", Quiet); ("info", Info); ("debug", Debug) ]

let mebibytes =
  let dec mb = mb * 1024 * 1024 in
  let enc bytes = bytes / (1024 * 1024) in
  Bcfgt.map ~dec ~enc Bcfgt.int

let tls =
  let open Bcfgt in
  directive ~name:"tls" (fun certificate key -> { certificate; key })
  |> field "certificate" string (fun t -> t.certificate)
  |> opt "key" string ~get:(fun t -> t.key)
  |> uniq

let upstream =
  let open Bcfgt in
  cases ~tag:"kind" string
    [
      case "tcp"
        (directive (fun host port -> { host; port })
        |> field "host" string (fun t -> t.host)
        |> field "port" int (fun t -> t.port))
        ~inject:(fun tcp -> Tcp tcp)
        ~project:(function Tcp tcp -> Some tcp | _ -> None);
      case "unix"
        (directive Fun.id |> field "path" string Fun.id)
        ~inject:(fun path -> Unix path)
        ~project:(function Unix path -> Some path | _ -> None);
    ]

let service =
  let open Bcfgt in
  let fn hostname public internal memory weight log listen upstream tls =
    { hostname; public; internal; memory; weight; log; listen; upstream; tls }
  in
  directive ~name:"service" fn
  |> req string (fun t -> t.hostname)
  |> flag "public" (fun t -> t.public)
  |> flag "internal" (fun t -> t.internal)
  |> field "memory" mebibytes (fun t -> t.memory)
  |> field "weight" float (fun t -> t.weight)
  |> field "log" level (fun t -> t.log)
  |> field "listen" (list int) (fun t -> t.listen)
  |> field "upstream" upstream (fun t -> t.upstream)
  |> opt "tls" tls ~get:(fun t -> t.tls)
  |> some

Each combinator says one thing:

• req reads a positional parameter, the one written next to the name of the directive;
• flag reads a marker, a parameter which is only there to be present or absent (public, internal). It does not occupy a positional slot, so service www.example.org public and service public www.example.org are read alike;
• field reads a required child, opt an optional one, and list a repeated one (listen appears twice for the first service);
• enum and map adapt a scalar: the former restricts it to a set of words, the latter converts it (here mebibytes into bytes);
• cases describes a sum type, whose case is chosen by a tag child (kind);
• uniq closes a record which appears once, some a record which appears any number of times. That is why service has type service list Bcfgt.t: it describes every top-level service directive of the file.

A tree-shaped configuration, such as our nested groups, is expressed with fix:

type group = { name : string; members : string list; groups : group list }

let group =
  let open Bcfgt in
  fix @@ fun group ->
  directive ~name:"group" (fun name members groups -> { name; members; groups })
  |> req string (fun t -> t.name)
  |> field "member" (list string) (fun t -> t.members)
  |> field "group" (list group) (fun t -> t.groups)
  |> uniq

Reading the file is then a matter of parsing it and decoding it:

let ( let* ) = Result.bind

let load filename =
  let ic = open_in_bin filename in
  let finally () = close_in ic in
  Fun.protect ~finally @@ fun () ->
  let* cfg =
    Result.map_error
      (fun err -> `Msg (Format.asprintf "%a" Bcfg.pp_error_for_human err))
      (Bcfg.parser (Lexing.from_channel ic))
  in
  let* services = Bcfgt.decode service cfg in
  let* groups = Bcfgt.decode (Bcfgt.list group) cfg in
  Ok (services, groups)

and writing it uses the very same values, which is the whole point of the exercise:

let save filename services =
  let oc = open_out_bin filename in
  let finally () = close_out oc in
  Fun.protect ~finally @@ fun () ->
  Bcfgt.encode service services |> Bcfg.emitter |> Seq.iter (output_string oc)

When the file does not say what the codec expects, decoding fails with a message which points at the culprit. The parsed tree carries no location, so the path is structural rather than a line number:

service > #0 > upstream > port: "http" is not a valid integer
service > #0 > log: "verbose" is not one of: quiet, info, debug
service > #1 > memory: missing field "memory"

The whole program, with the printing left out here, is available as example/services.ml in the distribution and is run by the test suite, so it cannot drift away from the library.

Syntax.

Here is the syntactic definition of bcfg, in the ABNF notation described by RFC 5234:

config     = *newline *directive
directive  = word parameters children
parameters = *(1*wsp word)
children   = "{" newline *directive "}" newline
           / newline

Where newline is one or more line breaks, possibly preceded by a comment:

newline = 1*(*wsp [comment] LF)
comment = *wsp "#" *(%x00-09 / %x0b-FF)
wsp     = SP / HTAB

Lexically, a word is either a bare word, or a word delimited by single or double quotes:

word      = 1*(pchar / escape)
          / DQUOTE *(dqchar / escape / hexadecimal / folding) DQUOTE
          / "'" *(qchar / escape / hexadecimal) "'"

pchar     = %x21 / %x23-26 / %x28-5B / %x5D-7A / %x7C / %x7E / utf-8
qchar     = %x21-26 / %x28-5B / %x5D-7E / utf-8
dqchar    = %x21 / %x23-5B / %x5D-7E / utf-8

escape    = "\" ( "\" / "'" / DQUOTE / SP / "#" / "{" / "}"
                / "a" / "b" / "t" / "n" / "v" / "f" / "r" )
hexadecimal = "\x" 2HEXDIG
folding     = "\" *wsp [comment] LF 1*wsp

A bare word therefore excludes the space, the two quotes, the backslash and the braces, all of which have a meaning in the format, and it must be valid UTF-8 (utf-8 is the encoding described by RFC 3629). Everything else can be written inside quotes, byte by byte if needed. Note that folding (the RFC 822 continuation) only exists inside double quotes.

About scfg and its OCaml implementation.

The project arose from the idea of having a simple configuration file. The scfg format provides a good foundation, and the project initially began as an effort to improve scfg so that it uses ocamllex rather than sedlex and offers an interface similar to jsont for decoding and encoding configuration files in OCaml.

As the work involved was quite substantial, we decided to make it a project in its own right, even though the credit goes to Léo Andrès, the original author of scfg. The format has then been extended, notably with the ability to fold values according to RFC 822 (drawing on skills acquired in email processing) and to carry content which is not valid UTF-8. These extensions are not part of the scfg format.