Scripts

Script Introduction

The build system has first-class support for Lua as a scripting language.

Lua scripts are processed by the build system in a couple places:

• REGULAR SCRIPT: In values.lua or *.values.lua files in the same include directories (-I) as the Values (values.json[c] and *.values.json[c]) files
• EMBEDDED SCRIPT: Embedded in comments at the top of single-file scripts.

For example, a regular script may be:

-- file: values.lua
SomeRule = require('SomeLibrary_Std.SomeRule')
SomeRule = SomeRule.at('1.0.0') -- this should be on the same line except bug with OCaml Lua parser
SomeRule:Executable {
  id='OurTest_Std.OurMain@2.3.4',
  files={
    glob={
      origin='someorigin',
      patterns={'**/*.ml', '**/*.mli'},
      exclude={'tests/**'}
    }
  }
}

while embedded in an OCaml single-file script the Lua script is inside the !dk comment:

#!/usr/bin/env
let () = print_endline "In the beginning ..."
let () = print_endline "We ran this inside our executable."
(*
SomeRule = require('SomeLibrary_Std.SomeRule')
SomeRule = SomeRule.at('1.0.0')
SomeRule:Executable {
  id=build.me.id,
  files=build.me.asset
}
!dk!p *)

All scripts in a running build share the same Lua state, and Lua is interpreted serially. That means two things:

• All Lua scripts must be fast. The "continuation" mechanism, described in a later subsection, lets scripts give parallelizable work to the build engine through subshells.
• Lua scripts must be written to minimize use of global variables.

Script Phases

A Lua script is scanned once but evaluated (ie. interpreted) twice.

The first evaluation does a quick scan in a very restrictive sandbox to find which dependencies the script needs and what modules and rules the script exports. This first evaluation happens in the the VALUESCAN phase documented later in the specification.

The second evaluation runs the script conventionally. This second evaluation happens in the the VALUELOAD phase documented later in the specification. All Lua functions behave as documented later in this specification.

Care is needed so that the script completes without errors in the sandbox of the first VALUESCAN evaluation. The VALUESCAN sandbox does the following:

1. require('Mod.X_Y_Z') (preferred) or require(dependency).at(version) (deprecated) will capture the name and version of the dependency, but not load the dependency.
2. assert(...) and error(...) continue to do Lua conventional error checking
3. build.is_building will return a false-y value (ie. nil, or false if the Lua implementation version is modern)
4. All other built-in functions (ex. print(), table.unpack) are defined to return a sensible Lua value but do nothing.
5. The fallback for reading an unknown key from a table (ex. print(a.b.some_unknown_field)) is to return nil (conventionally it would error).
6. The fallback for writing an unknown key to a table (ex. a.b.some_unknown_field = 1) is to return nil (conventionally it would error).
7. The fallback for unknown functions (ex. some_unknown_function()) is a function that returns nil (conventionally it would error).

Lua Specification

The overall design goal is to maintain conventional Lua behavior as much as possible. The end user, to the extent possible, should be able to use their favorite Lua IDEs to edit their Lua build scripts.

---

The build system uses Lua 2.5 for its syntax (no for loops) and its data model (no metatables), but uses functions available from Lua 5.1+ (ex. require).

Historical note: Lua 2.5 was published in 1996 and lacks several features of modern-day Lua: for loops, metaprogramming for metatables, and coroutines. However, rules are mostly configuration, and a full programming language makes hermetic, bounded-time builds difficult or impossible. So even if a future specification uses a later Lua version, several features will be disabled.

---

To support Lua IDEs:

• Lua 5.1+: The Lua convention is one module exported by script. So unlike value.json[c], a [*.]values.lua script only has one module. To export functions and rules from the module, the module returns a Lua table per the Lua 5.2+ convention (and compatible with Lua 5.1).

---

Lua names (aka identifiers), for maximum portability, use the Lua 2.5 lexical conventions:

Identifiers can be any string of letters, digits, and underscores, not beginning with a digit.

and the Lua 5.4 reserved words:

• and
• break
• do
• else
• elseif
• end
• false
• for
• function
• goto
• if
• in
• local
• nil
• not
• or
• repeat
• return
• then
• true
• until
• while

Lua Global Variables

Lua Global Variable - arg

arg

A table defined only when Lua is run embedded or a file is run directly as a Lua script.

Before running any code, lua collects all command-line arguments in a global table called arg. The script name goes to index 0, the first argument after the script name goes to index 1, and so on. Any arguments before the script name (that is, the interpreter name plus its options) go to negative indices. For example, in the call:

${dk_build_system} lua -la b.lua t1 t2

the table is like this:

arg = { [-2] = "lua", [-1] = "-la",
        [0] = "b.lua",
        [1] = "t1", [2] = "t2" }

If there is no script in the call, the interpreter name goes to index 0, followed by the other arguments. For instance, the call

${dk_build_system} lua -e "print(arg[1])"

will print -e. If there is a script, the script is called with arguments arg[1], ···, arg[#arg].

Lua Global Variable - loadstring

loadstring (string [, chunkname])

Compiles the string.

If there are no errors, returns the compiled chunk as a function; otherwise, returns nil plus the error message. The environment of the returned function is the global environment.

chunk = assert(loadstring(s))
chunk()

When absent, chunkname defaults to the given string or an abbrevation of it.

Compatibility: Lua 5.1, 5.2, 5.3 but removed from 5.4.

Lua Global Variable - next

next (table, index)

This function allows a program to traverse all fields of a table. Its first argument is a table and its second argument is an index in this table. It returns the next index of the table and the value associated with the index. When called with nil as its second argument, the function returns the first index of the table (and its associated value). When called with the last index, or with nil in an empty table, it returns nil. In Lua there is no declaration of fields; semantically, there is no difference between a field not present in a table or a field with value nil. Therefore, the function only considers fields with non nil values. The order in which the indices are enumerated is not specified, even for numeric indices. If the table is modified in any way during a traversal, the semantics of next is undefined.

Lua Global Variable - tostring

tostring (e)

This function receives an argument of any type and converts it to a string in a reasonable format.

Table contents are not converted. See jsondk.encode to show inside of a table.

Lua Global Variable - print

print (e1, e2, ...)

This function receives any number of arguments, and prints their values in a reasonable format. Each value is printed in a new line. This function is not intended for formatted output, but as a quick way to show a value, for instance for error messages or debugging.

See printf for functions for formatted output.

See jsondk.encode to print tables.

Lua Global Variable - printf

printf("format", ...)

This function performs like its C counterpart, printing a formatted string.

It is equivalent to this Lua code:

print(string.format(format, unpack(arg))

without the newline inserted by print.

format is a formatting string containing C printf() style formatting codes. It is followed by a list of arguments to be substituted into the format string.

This function was borrowed from Premake's printf.

Lua Global Variable - tonumber

tonumber (e)

This function receives one argument, and tries to convert it to a number. If the argument is already a number or a string convertible to a number (see Section 4.2), then it returns that number; otherwise, it returns nil.

Lua Global Variable - type

type (v)

This function allows Lua to test the type of a value. It receives one argument, and returns its type, coded as a string. The possible results of this function are "nil" (a string, not the value nil), "number", "string", "table", "function" (returned both for C functions and Lua functions), and "userdata".

Lua 5.1+ compatibility: Unlike Lua 2.5, the type function does not return a "tag" as a second result.

Lua Global Variable - assert

assert (v [, message])

Raises an error if the value of its argument v is false (i.e., nil or in a future specification false); otherwise, returns all its arguments. In case of error, message is the error object; when absent, it defaults to assertion failed!

Compatible with Lua 5.1.

Lua Global Variable - error

error (message)

This function issues an error message and terminates the last called function from the library. It never returns.

Lua 5.1+ compatibility: The "level" argument in error (message, [level]) is ignored.

Lua build library

build is a Lua table with access to the running build.

build.newrules

local M = { id = '...' }
fnrules = build.newrules(M)
function fnrules.SomeRule(command,request)
  -- ...
end
return M

-- or if interactive user interface rules are needed ...

local M = { id = '...' }
fnrules, uirules = build.newrules(M)
function fnrules.SomeRule(command,request)
  -- ...
end
function uirules.SomeRuleThatCanTakeOverConsole(command,request)
  -- ...
end
return M

build.newrules(M) creates a fnrules and uirules field inside the module table M.

The fnrules and uirules fields will both be empty tables, and those empty tables are returned.

• fnrules are function rules that can be used in values.json[c] files or invoked by the end-user.
• uirules are interactive rules that can only be invoked by the end-user.

See Custom Lua Rules for a detailed explanation of the difference between fnrules and uirules.

Lua jsondk library

The jsondk library is embedded into the build system (nothing needs to be downloaded) but it must be accessed through jsondk = require('jsondk'). That keeps with the design goal to maintain Lua conventions.

There is a popular, unrelated Lua library dkjson for parsing JSON. Since the chance for confusion is high, jsondk is broadly compatible with dkjson.

jsondk.encode

jsondk = require('jsondk')
tbl = {
  animals = { "dog", "cat", "aardvark" },
  instruments = { "violin", "trombone", "theremin" }
}
str = jsondk.encode (tbl, { indent = true })

-- or
jsondk = require('jsondk')
str = jsondk.encode (tbl)

Converts a Lua value to JSON:

• nil values are not printed
• jsondk.null values are encoded as JSON null

If indent is truthy then the JSON is pretty-printed.

jsondk.decode

jsondk = require('jsondk')
str = '{"animals":["dog","cat","aardvark"],"bugs":null}'
jsondk.decode (str)

-- or
jsondk = require('jsondk')
value, errmsg, errrendered, sb, sl, sc, eb, el, ec = jsondk.decode (str)

Converts JSON to a Lua value:

• Large numbers are converted to floating-point numbers with a possible loss of precision. If outside the floating-point range, an error is raised.
• JSON nulls are converted to jsondk.null Lua values

If the JSON could be converted, the result is the first return value.

Otherwise:

• value is nil
• errmsg is a brief error message
• errrendered is a prerendered error
• sb, sl, and sc are the starting byte offset (zero-based), line and column (1-based)
• eb, el, and ec are the ending byte offset (zero-based), line and column (1-based)

jsondk.null

jsondk = require('jsondk')
tbl = {
  animals = { "dog", "cat", "aardvark" },
  bugs = jsondk.null,
  trees = nil
}
str = jsondk.encode (tbl)

-- {
--   "animals":["dog","cat","aardvark"],
--   "bugs":null
-- }

The jsondk.null Lua value represents JSON null.

Lua math library

This Lua 5.4 compatible library provides basic mathematical functions. It provides all its functions and constants inside the table math. Functions with the annotation "integer/float" give integer results for integer arguments and float results for non-integer arguments. The rounding functions math.ceil, math.floor, and math.modf return an integer when the result fits in the range of an integer, or a float otherwise.

math.abs

math.abs (x)

Returns the maximum value between x and -x. (integer/float)

math.acos

math.acos (x)

Returns the arc cosine of x (in radians).

math.asin

math.asin (x)

Returns the arc sine of x (in radians).

math.atan

math.atan (y [, x])

Returns the arc tangent of y/x (in radians), using the signs of both arguments to find the quadrant of the result. It also handles correctly the case of x being zero.

The default value for x is 1, so that the call math.atan(y) returns the arc tangent of y.

math.ceil

math.ceil (x)

Returns the smallest integral value greater than or equal to x.

math.cos

math.cos (x)

Returns the cosine of x (assumed to be in radians).

math.deg

math.deg (x)

Converts the angle x from radians to degrees.

math.exp

math.exp (x)

Returns the value eˣ (where e is the base of natural logarithms).

math.floor

math.floor (x)

Returns the largest integral value less than or equal to x.

math.fmod

math.fmod (x, y)

Returns the remainder of the division of x by y that rounds the quotient towards zero. (integer/float)

math.huge

math.huge

The float value HUGE_VAL, a value greater than any other numeric value.

math.log

math.log (x [, base])

Returns the logarithm of x in the given base. The default for base is e (so that the function returns the natural logarithm of x).

math.max

math.max (x, ···)

Returns the argument with the maximum value according to the Lua operator <.

math.maxinteger

math.maxinteger

An integer with the maximum value for an integer.

math.min

math.min (x, ···)

Returns the argument with the minimum value, according to the Lua operator <.

math.mininteger

math.mininteger

An integer with the minimum value for an integer.

math.modf

math.modf (x)

Returns the integral part of x and the fractional part of x. Its second result is always a float.

math.pi

math.pi

The value of π.

math.rad

math.rad (x)

Converts the angle x from degrees to radians.

math.sin

math.sin (x)

Returns the sine of x (assumed to be in radians).

math.sqrt

math.sqrt (x)

Returns the square root of x. (You can also use the expression x^0.5 to compute this value.)

math.tan

math.tan (x)

Returns the tangent of x (assumed to be in radians).

math.tointeger

math.tointeger (x)

If the value x is convertible to an integer, returns that integer. Otherwise, returns fail.

math.type

math.type (x)

Returns "integer" if x is an integer, "float" if it is a float, or fail if x is not a number.

math.ult

math.ult (m, n)

Returns the string t (ie. a boolean true) if and only if integer m is below integer n when they are compared as unsigned integers.

Lua package library

The package library provides basic facilities for loading modules in Lua. It exports one function directly in the global environment: require. Everything else is exported in the table package.

require

require (modname)

Loads the given module.

If the modname is a standard module id (ex. MyLibrary_Std.A.B.MyModule - tbd: document this) a task is added to the task graph to search for it. The section Custom Lua Modules describes how to create your own modules.

As of the writing of this specification, only standard modules may be loaded.

Standard modules must be required with a specific version using one of two equivalent forms:

• Version-encoded form (preferred): require('Lib_Std.Mod.X_Y_Z') where X_Y_Z is the semver version string with every . and - replaced by _. Example: require('CommonsBase_Std.Extract.0_2_0') loads version 0.2.0. Note: the version suffix starting with a digit is valid as a require argument string even though it is not a valid bare Lua identifier.
• .at() form (deprecated): require('Lib_Std.Mod').at('X.Y.Z'). Example: require('CommonsBase_Std.Extract').at('0.2.0').

Once imported with require, standard modules are enriched with constants as per Lua 5.1 module() convention and Lua module versioning conventions and a _build field:

Field	Example
_NAME	MyModule._NAME would be MyLibrary_Std.A.B.MyModule
_PACKAGE	MyModule._PACKAGE would be MyLibrary_Std.A.B
_VERSION	MyModule._VERSION would be 1.0.0
_M	(may be removed) MyModule._M would be a Lua reference to MyModule
_build	described later in Custom Lua Rules

Historical note: Even though the implementation of module() is deprecated after Lua 5.1, its conventions were never deprecated.

package.registrykey

package.registrykey

A opaque variable holding a key to an internal table of packages that are loaded.

Lua request.rule library

This library is available through the request.rule field to function rules and to UI Rules.

For example:

local M = { id = '...' }
rules = build.newrules(M)
function rules.SomeRule(command,request)
  if command == "declareoutput" then
    -- use the [rule] library
    local id = request.rule.generatesymbol()
  end
end
return M

request.rule.generatesymbol

request.rule.generatesymbol(arg1, arg2, ...)

Generates a deterministic standard namespace term. For example, it may generate X6pro7j57evsyymo36mehvpabhy from a constant X followed by a lowercase base32-encoding of the BLAKE2s 128-bit digest of:

1. The rule's MODULE string from its id = "MODULE@VERSION"
2. The rule's VERSION string from its id = "MODULE@VERSION"
3. The request.user table
4. The arguments arg1, arg2, ...

Each Lua value has its digest calculated according to:

• nil: The 0x00 byte
• number: The little-endian IEEE 754 double-precision float representation of the number (even for integers).
• string: The bytes of the string
• function: An error is raised.
• userdata: An error is raised.
• a table with number or string keys:
  • Any number key is converted to an integer or it raises an error; then the integer is converted to a string.
  • The keys are lexographically sorted
  • The digest is calculated with depth-first traversal. That is the string KEY1 then the Lua value VALUE1, then KEY2 and VALUE2, until there are no more key values.

Lua request.execution library

This library is available only to UI Rules through the request.execution field.

An example for an execution specific UI rule:

local M = { id = '...' }
rules, uirules = build.newrules(M)
function uirules.SomeRule(command,request)
  if command == "submit" then
    -- use the [request.execution] library
    local osfamily = request.execution.OSFamily
    if osfamily == "macos" then
      print("Howdy mac users!")
    end
  end
end
return M

Tip

Function rules do not have request.execution because function rule outputs must stay reproducible across execution platforms. When a function rule needs platform specific outputs, use declareoutput.execution_slot together with slot expansions like ${SLOTNAME.Release.execution_abi}.

An example for an execution specific function rule that does not use request.execution:

function rules.F_Build(command, request)
  if command == "declareoutput" then
    return {
      declareoutput = {
        return_objects = {
          id = "UserLibrary_Std.A.B.FreeRule.OutputObject@1.0.0",
          slots = { "Release.Windows_x86_64", "Release.Darwin_arm64" },
          execution_slot = "Release.execution_abi"
        }
      }
    }
  elseif command == "submit" then
    return {
      values = {
        schema_version = { major = 1, minor = 0 },
        forms = {
          {
            id = p.outputid,
            function_ = {
              commands = {
                -- using one named slot (ex. SLOT.Release.Windows_x86_64) restricts the command to only Release.Windows_x86_64.
                -- when [env -u SOMETHING -- ...] runs, it delegates to the "..." part
                {
                    "$(get-object CommonsBase_Std.Coreutils@0.6.0 -s ${SLOTNAME.Release.execution_abi} -m ./coreutils.exe -f coreutils.exe -e '*')",
                    "env", "-u", "${SLOT.Release.Windows_x86_64}", "--",
                    "cmd", "/c", "\"call build.bat msvc64 & exit /b %ERRORLEVEL%\""
                },
                {
                    "$(get-object CommonsBase_Std.Coreutils@0.6.0 -s ${SLOTNAME.Release.execution_abi} -m ./coreutils.exe -f coreutils.exe -e '*')",
                    "env", "-u", "${SLOT.Release.Darwin_arm64}", "--",
                    "/bin/sh", "-c", "./configure && make && make install"
                }
              },
            },
            outputs = {
              assets = {
                -- files in common to all slots
                {
                  slots = { "Release.Windows_x86_64", "Release.Darwin_arm64" },
                  paths = { "LICENSE.txt" }
                },
                -- files specific to each ABI
                {
                  slots = { "Release.Windows_x86_64" },
                  paths = { "sample.exe" }
                },
                {
                  slots = { "Release.Darwin_arm64" },
                  paths = { "sample" }
                },
              }
            }
          }
        }
      }
    }
  end
end

request.execution.OSFamily

request.execution.OSFamily

The OSFamily of the execution platform. Values include windows and macos; they are defined in OSFamily.

request.execution.ABIv3

request.execution.ABIv3

The third version of the DkML ABI of the execution platform described in ${SLOTNAME.SlotName}. Examples include Windows_x86_64 and Darwin_arm64.

request.execution.OSv3

request.execution.OSv3

The third version of the DkML OS of the execution platform. The set of OSv3 values is:

• UnknownOS
• Android
• DragonFly
• FreeBSD
• IOS
• Linux
• NetBSD
• OpenBSD
• OSX
• Windows

Lua request.io library

This library is available to function rules and UI Rules through the request.io field.

For example:

local M = { id = '...' }
rules = build.newrules(M)
function rules.SomeRule(command,request)
  if command == "submit" then
    -- use the [request.io] library
    local file = request.io.open("a/b/somefile", "w")
  end
end
return M

Capabilities are restricted so that:

• File objects can be created only to write to files in a directory unique to the rule and output key. The intent for these writable file objects is to allow creating assets and nothing else.
• File objects for reading can be obtained from value shell expressions given in response to Function Rule Command - submit

Some build system implementations may sandbox the I/O operations.

request.io.open

file_or_directory = request.io.open(filename_or_dirname, mode)

This function opens a file or directory in the mode specified in the string mode. It returns a new file descriptor, or, in case of errors, nil plus an error message.

The mode string can be any of the following:

• "r" read-only mode.
• "w" write mode. filename_or_dirname must be a filename. Any parent directories required by filename will be created.

Unlike the C library function fopen, the file will be opened in binary mode rather than text mode. (Text mode adds CRLF on Windows systems and is non-reproducible when cross-compiling.)

The filename_or_dirname must be a strictly relative path:

• An absolute path will raise an error.
• After the path is normalized, any path segments that start with .. will raise an error.
• After the path is normalized, any path segments that contain a forward or backward slash will raise an error. For example, Unix filenames can contain backslashes, but they will raise errors.

Directory operations:

• For the r read-only mode when filename_or_dirname is a directory, the usable functions are request.io.list and request.io.toasset.

The file may be closed after the request is finished (ie. the run-function command is finished), but it is the author's responsibility to close the file with request.io.close or with request.io.toasset.

request.io.read

request.io.read(file, format1, ...)

Reads the file file according to the given formats format1, ... which specify what to read. For each format, the function returns a string or a number with the characters read, or nil if it cannot read data with the specified format. (In this latter case, the function does not read subsequent formats.) When called without arguments, it uses a default format that reads the next line (see below).

The available formats are

• a, all or *all: reads the whole file, starting at the current position. On end of file, it returns the empty string; this format never fails unless the file does not exist or is unreadable
• l, line or *line: reads the next line skipping the end of line, returning nil on end of file. This is the default format.
• L: reads the next line keeping the end-of-line character (if present), returning nil on end of file.
• number: reads a string with up to this number of bytes, returning nil on end of file. If number is zero, it reads nothing and returns an empty string, or nil on end of file.

The formats l and L should be used only for text files.

This function behaves Lua 5.4 io.read except the format n is not supported.

request.io.write

request.io.write(file, value1, ...)

Writes the value of each of its arguments to file file. The arguments must be strings or numbers. To write other values, use tostring or string.format or jsondk.encode.

request.io.list

request.io.list(dir, format1, ...)

List the contents of directory dir according to the given formats format1, ... which specify what to list. For each format, the function returns a table (see below) with the directory contents read, or nil if it cannot list the directory with the specified format. (In this latter case, the function does not list with subsequent formats.) When called without arguments, it uses a default format that lists the whole directory (see below).

The available formats are

• a or all: list the entire directory, starting at the current position. On the end of directory, it returns the empty table; this format never fails unless the directory does not exist or is unreadable

The directory contents table has:

• keys that are index numbers: 1, 2, etc.
• values that are lazily-opened readonly file or subdirectories. Lazy-open means you do not need to close it unless you read from it.

Use request.io.isfile and request.io.isdir to check the type of the directory entry.

request.io.isfile

request.io.isfile(file)

A truthy value if and only if file is a file object. Any other Lua value will return a falsy value.

request.io.isdir

request.io.isdir(dir)

A truthy-value if and only if dir is a directory object. Any other Lua value will return a falsy value.

request.io.realpath

request.io.realpath(file [, { relative = 1 }])
request.io.realpath(dir [, { ... }])

The path to the file or directory object.

The validity is only guaranteed inside:

• a submit command until the next continuation.
• a ui command until the ui command is finished

In particular:

• The path may not exist immediately after request.io.realpath. A hermetic implementation is allowed to:

  1. Return dangling symlinks as the return value of request.io.realpath
  2. Bind those symlinks (ex. ln -s -f on Unix) to correct locations after the Lua rule function is finished but immediately before running rule expressions
  3. Bind those symlinks to dangling locations after the rule expressions are finished

If relative is truthy, the path is returned as a relative path from the project base directory. Since, especially on Windows, not all paths can be made relative, the conversion to a relative path is best-effort.

Without relative, at the discretion of the build system implementation the returned path may be an absolute path or a relative path.

request.io.toasset

local origin, asset = request.io.toasset(file_or_dir, {
  path = "some/asset/path",
  origin_name = "..."
})

Converts the file or directory to an asset and closes the file.

In the options only path is mandatory.

path: Two assets at the same path is an error. Each path must be strictly relative or an error will be raised.

origin_name: The name of the origin. The origin is a label used to invalidate assets. Many assets can share the same origin.

The origin return value will be a table unique to the request. Multiple calls to request.io.toasset in the same request will give the same origin table:

{
  name = "SOME_IDENTIFIER",
  mirrors = { "selfasset://SOME_IDENTIFIER" }
}

The asset return value will be another table:

{
  -- from the `origin` table
  origin = "SOME_IDENTIFIER",
  -- from the request.io.toasset(file, {path}) argument
  path = "some/asset/path",
  -- the rest is calculated from the file
  size = 151,
  checksum = {
    sha256 = "0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"
  }
}

Security note: There is no protection against two request.io.toasset with the same path and origin_name. However, implementations are required to calculate the SHA256 checksum in the asset return value from the request.io.write rather than the filesystem. That means in a race multiple assets may be placed in the valuestore, but all will be valid assets and at most one will be the checksum recorded in the tracestore.

request.io.flush

request.io.flush()

Flushes the standard output that was buffered from print and printf.

request.io.close

request.io.close(file)
request.io.close(directory)

Closes the file or directory.

Lua request.submit library

This library is available to function rules and UI Rules through the request.submit field.

For example:

local M = { id = '...' }
rules = build.newrules(M)
function rules.SomeRule(command,request)
  if command == "submit" then
    -- use the [submit] library
    local id = request.submit.outputid
  end
end
return M

request.submit.outputid

request.submit.outputid
-- example: OurTest_Std.A.X6pro7j57evsyymo36mehvpabhy@0.1.0

This string is the form or asset identifier declared in the "declareoutput" command.

It is only available to function rules.

request.submit.outputmodule

request.submit.outputmodule
-- example: OurTest_Std.A.X6pro7j57evsyymo36mehvpabhy

This string is the module (MODULE) of the form or asset MODULE@VERSION declared in the "declareoutput" command.

It is only available to function rules.

request.submit.outputversion

request.submit.outputmodule
-- example: 0.1.0

This string is the version (VERSION) of the form or asset MODULE@VERSION declared in the "declareoutput" command.

It is only available to function rules.

Lua request.ui library

This library is available to UI Rules through the request.ui field.

For example:

local M = { id = '...' }
rules, uirules = build.newrules(M)
function uirules.SomeRule(command,request)
  if command == "ui" then
    -- use the [request.ui] library
    local bundle, getbundle, getasset = request.ui.glob { -[[ ... ]] }
  end
end
return M

request.ui.glob

bundle, getbundle, getasset = request.ui.glob {
  patterns = {"src/**/*.c"}
  [, cell = "root"]
  [, project = "OurProject_Std@0.1.0"]
  [, excludes = {"src/**/test*.c"} ]
  [, trace = 1]
}

Creates a bundle of files from a project source directory for use when constructing a values inside a Custom Lua Rule.

The design intent is to allow user influenced change detection and reproducibility for project files:

• User-influenced change detection: In large projects (ex. monorepos), the project tree can be broken into smaller bundles. Only parts of the build that depend on smaller project bundles will be rebuilt when a project source file changes.
• Reproducibility: A build user does not access the project files directly; the project files are always checksummed and made available through this request.ui library.

The project argument is the identifier and version for the end-user's project. It defaults to OurProject_Std@0.1.0. The UI rule may be used by several projects, so the project argument is intended to be supplied by the end-user as a request parameter to the UI rule. It may be a library id and version (ex. OurProject_Std@1.0.0) or a standard module id and version (ex. OurProject_Std.A.B.SomeModule@1.0.0). The project must belong to the distribution package and version if the project is distributed. Using the Our vendor namespace means the project cannot be distributed, but the project does not need to have a distribution with keys and version ranges.

The cell argument is the name of the cell from the project structure. It defaults to root. The generated bundle identifier is PROJECT_MODULE.Cells.Xyyyyyyy@PROJECT_VERSION where PROJECT_MODULE is the module or library identifier from project, PROJECT_VERSION is the project version from project, and yyyy is the lowercase, no-padding, base32-encoded SHA256 checksum of cell.

The patterns and excludes are glob expressions on project files that conform to Language Server Protocol 3.18 patterns:

• * to match zero or more characters in a path segment
• ? to match on one character in a path segment
• ** to match any number of path segments, including none
• {} to group conditions (e.g. **​/*.{ts,js} matches all TypeScript and JavaScript files)
• [] to declare a range of characters to match in a path segment (e.g., example.[0-9] to match on example.0, example.1, …)
• [!...] to negate a range of characters to match in a path segment (e.g., example.[!0-9] to match on example.a, example.b, but not example.0)

excludes exclude files after they have been found by patterns.

The same project file may belong to different assets.

The specification does not mandate how change detection is implemented. An implementation may scan all the globs at startup, or cache the globbed files and only update them when an invalidation is given to the build system.

The project directory structure will be maintained in the asset. For example, given the project:

src/
  main.c
  media/
    player.c
  db/
    sql.c
  platforms/
    windows.asm
    linux.s
    macos.s
test/
  test-db.c

and patterns = {"src/**/*.c"}, the asset will have the structure:

src/
  main.c
  media/
    player.c
  db/
    sql.c

The return values are the bundle, partial get-bundle command and the partial get-asset command:

• bundle: The bundle. For example:

  {
    id = "PROJECT_MODULE.Sources.Xyyyyyyy@PROJECT_VERSION", -- derived bundle id
    listing = {
      origins = {
        {
          name = "...origin...", -- from `request.ui.glob {origin}` argument
          mirrors = { "." } -- `.` is the project directory
        }
      }
    },
    assets = {
      -- one asset per file matched by the glob patterns
      {
        origin = "...origin...",
        path = "<relative path to project file>",
        size = 123000, -- replaced with real size of project file
        checksum = {
          -- replaced with real SHA256
          sha256 = "0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"
        }
      }
    }
  }

• partial get-bundle command: The partially complete value shell command get-bundle MODULE@VERSION with MODULE@VERSION replaced with a real value. To use the command in subshells, the -d : must be added to complete the value shell command.

• partial get-asset command: The partially complete value shell command get-asset MODULE@VERSION with MODULE@VERSION replaced with a real value. To use the command in subshells, the -p PROJECT_SOURCE_FILE -f :file:BASENAME must be added to complete the value shell command.

Performance consideration: Using the partial get-asset command will almost always be more efficient for single file access than partial get-bundle command, as the latter may zip up the bundle and then unzip more files than are needed.

Using the bundle could look like the following, where the source code is given as an argument to a compiler:

function uirules.MyRule(command, request)
  if command == "submit" and continue_ == "start" then
    local bundle, getbundle = request.ui.glob {
      patterns = { "src/**/*.c" }
    }
    return {
      submit = {
        values = {
          forms = {
            {
              id = request.submit.outputid,
              -- ...
              function_ = {
                commands = {
                  "some-programming-language-compiler",
                  -- let's pretend that there is a `-c DIR` option
                  -- to compile everything in a directory
                  "-c",
                  -- using `getbundle` will copy/link all the globbed
                  -- files into an isolated directory
                  "$(" .. getbundle .. " -d :)"
                }
              }
            }
          },
          bundles = {
            bundle
          }
        }
      }
    }
  end

request.ui.spawn

request.ui.spawn {
  program = "/bin/echo"
  [, args = { "arg1", "arg2", "..." }]
  [, cwd = "/some/dir"]
  [, envmods = { "+DOTNET_ROOT=/some/dir", "..." }]
}

Runs the program with the arguments args.... The program will have its environment modified by envmods in accordance to Environment Modifications.

The default cwd is the user's working directory.

Build system implementations are required to have security controls.

The caller is expected to check the return values. Using the Lua convention assert(request.ui.spawn { ... }) is sufficient to pass only on exit code zero. The return values are:

• If the user rejected giving permission to the spawn, the three (3) return values are nil, an error mesage, and the string denied.
• On exit code 0, the two (2) return values are a truthy value and the number 0.
• On any other exit code, the four (4) return values are nil, an error message, the string exit, and the exit code number.
• If terminated due to a signal, the four (4) return values are nil, an error message, the string signal, and the signal number.
• If stopped due to a signal, the four (4) return values are nil, an error message, the string stop, and the signal number.

That is:

nil, "The request to trust the rule to launch the program was denied", "denied"
"t", 0
nil, "The program exited with code 55", "exit"
nil, "The program terminated due to signal 15", "signal", 15
nil, "The program stopped due to signal 19", "stop", 19

request.ui.capture

result = request.ui.capture {
  program = "gh"
  [, args = { "auth", "status" }]
  [, cwd = "/some/dir"]
  [, envmods = { "+GH_HOST=github.com", "..." }]
  [, max_output_bytes = 16777211]
}

Runs the program with the arguments args..., captures stdout and stderr, and returns a result table instead of streaming output to the terminal. The program will have its environment modified by envmods in accordance to Environment Modifications.

program may be a .cmd or .bat file on Windows.

The default cwd is the user's working directory.

The default maximum captured size is 16777211 bytes for each stream. On process start failure, the three return values are nil, an error message, and the string error. On captured output limit failure, the three return values are nil, an error message, and the string output-limit.

On process completion, one table is returned:

{
  status = "exit" | "signal" | "stop",
  code = 0,
  stdout = "...",
  stderr = "..."
}

request.ui.checksum

metadata = request.ui.checksum {
  path = "relative/project/file"
}

Calculates metadata for a project-local file. The path field must be a strictly relative project path.

On success, returns a single table:

{
  sha256 = "...",
  size = 123
}

If the file does not exist, the three (3) return values are nil, an error message, and the string absent, so a caller can map absence to request.ui.writefile's expected_sha256 = false. Using the Lua convention local meta = request.ui.checksum { path = ... } treats a missing file as meta == nil. On any other failure (a path that is not strictly relative or an I/O error), the three (3) return values are nil, an error message, and the string error.

request.ui.readfile

content = request.ui.readfile {
  path = "relative/project/file"
}

Reads a project-local file and returns its entire contents as a string. The path field must be a strictly relative project path, resolved against the user's project directory, the same base as request.ui.checksum and request.ui.writefile, and not the UI rule's sandbox. Unlike request.io.read, which reads the rule's discarded working area, request.ui.readfile reads the checked-in project tree.

The contents are returned verbatim as bytes; no newline translation is performed.

The caller is expected to check the return values. Using the Lua convention assert(request.ui.readfile { ... }) is sufficient. The return values are:

• On success, the single return value is the file contents as a string.
• If the file does not exist or is a directory, the three (3) return values are nil, an error message, and the string absent.
• On any other failure (a path that is not strictly relative or escapes the project, exceeding the size cap, or an I/O error), the three (3) return values are nil, an error message, and the string error.

An optional max_bytes field caps the number of bytes read (default 16777211).

request.ui.writefile

request.ui.writefile {
  path = "relative/project/file",
  content = "file contents",
  expected_sha256 = "e3b0c44298fc1c14..."   -- or false to require the file is absent
}

Conditionally writes content to a project-local file. This is a compare-and-swap: the write succeeds only if the file's current on-disk content matches the caller's stated expectation, guarding against a concurrent writer that changed the file since the caller last inspected it.

path must be a strictly relative project path, resolved against the user's project directory, the same base as request.ui.checksum and request.ui.signify. Missing parent directories are created. Unlike request.io.write, which stages bytes in the rule's discarded working area, request.ui.writefile publishes into the project tree.

The expected_sha256 field is required; there is no unconditional write:

• A hex SHA-256 string requires the file to currently exist with exactly that SHA-256. Obtain it beforehand from request.ui.checksum.
• The boolean false or the literal string false requires the file to not currently exist (a create).

content is written verbatim as bytes (encoded UTF-8); no newline translation is performed, so the caller controls line endings.

The caller is expected to check the return values. Using the Lua convention assert(request.ui.writefile { ... }) is sufficient. The return values are:

• If the user rejected giving permission to write the file, the three (3) return values are nil, an error message, and the string denied.
• If the current file does not satisfy expected_sha256 (it exists with a different SHA-256, exists when false was required, or is absent when a SHA-256 was required), the three (3) return values are nil, an error message, and the string conflict.
• On success, the three (3) return values are a truthy value, the strictly relative project path written, and the SHA-256 of the newly written content.
• On any other failure (a path that is not strictly relative or escapes the project, or an I/O error), the three (3) return values are nil, an error message, and the string error.

request.ui.signify

signed = request.ui.signify {
  operation = "sign",
  message = "INDEX",
  signature = "INDEX.sig"
}

verified = request.ui.signify {
  operation = "verify",
  message = "INDEX",
  signature = "INDEX.sig"
}

Signs or verifies a project-local file using the OpenBSD signify build keys local to your host and possibly shared with your team.

All path fields must be strictly relative project paths. On signing, the signature is written to the signature path. On verify failure, the three return values are nil, an error message, and the string verify.

request.ui.sleep

request.ui.sleep { seconds = 10 }

Suspends the UI rule for the requested number of seconds.

request.ui.buildpubkey

local key = request.ui.buildpubkey

The contents of the build public key, read in binary mode. This is the same public key used by request.ui.signify to verify signatures.

Lua string library

This mostly Lua 5.4 compatible library provides generic functions for string manipulation, such as finding and extracting substrings, and pattern matching. When indexing a string in Lua, the first character is at position 1 (not at 0, as in C). Indices are allowed to be negative and are interpreted as indexing backwards, from the end of the string. Thus, the last character is at position -1, and so on.

The string library provides all its functions inside the table string. Unlike Lua 5.1+, it does not sets a metatable for strings where the __index field points to the string table. Therefore, you cannot use the string functions in object-oriented style. For instance, string.byte(s,i) cannot be written as s:byte(i).

The string library assumes one-byte character encodings.

string.byte

string.byte (s [, i [, j]])

Returns the internal numeric codes of the characters s[i], s[i+1], ..., s[j]. The default value for i is 1; the default value for j is i. These indices are corrected following the same rules of function string.sub.

Numeric codes are not necessarily portable across platforms.

string.find

string.find (s, pattern [, init [, plain]])

The pattern is a Lua 2.5 pattern; see Lua 2.5 §6.2 Patterns.

Looks for the first match of pattern in the string s. If it finds a match, then find returns the indices of s where this occurrence starts and ends; otherwise, it returns fail. A third, optional numeric argument init specifies where to start the search; its default value is 1 and can be negative. A true as a fourth, optional argument plain turns off the pattern matching facilities, so the function does a plain "find substring" operation, with no characters in pattern being considered magic.

If the pattern has captures, then in a successful match the captured values are also returned, after the two indices.

string.format

string.format (formatstring, ···)

Returns a formatted version of its variable number of arguments following the description given in its first argument, which must be a string. The format string follows the same rules as the ISO C function sprintf. The only differences are that the conversion specifiers and modifiers F, n, *, h, L, and l are not supported and that there is an extra specifier, q. Both width and precision, when present, are limited to two digits.

The specifier q formats booleans, nil, numbers, and strings in a way that the result is a valid constant in Lua source code. Booleans and nil are written in the obvious way (true, false, nil). Floats are written in hexadecimal, to preserve full precision. A string is written between double quotes, using escape sequences when necessary to ensure that it can safely be read back by the Lua interpreter. For instance, the call

     string.format('%q', 'a string with "quotes" and \n new line')

may produce the string:

     "a string with \"quotes\" and \
      new line"

This specifier does not support modifiers (flags, width, precision).

The conversion specifiers A, a, E, e, f, G, and g all expect a number as argument. The specifiers c, d, i, o, u, X, and x expect an integer.

The specifier s expects a string; if its argument is not a string, it is converted to one following the same rules of tostring. If the specifier has any modifier, the corresponding string argument should not contain embedded zeros.

The specifier p formats the pointer returned by lua_topointer in Lua 5.1+, but in this specification an error is raised.

string.len

string.len (s)

Receives a string and returns its length. The empty string "" has length 0. Embedded zeros are counted, so "a\000bc\000" has length 5.

string.lower

string.lower (s)

Receives a string and returns a copy of this string with all ASCII uppercase letters changed to lowercase. All other characters are left unchanged.

string.rep

string.rep (s, n [, sep])

Returns a string that is the concatenation of n copies of the string s separated by the string sep. The default value for sep is the empty string (that is, no separator). Returns the empty string if n is not positive.

(Note that it is very easy to exhaust the memory of your machine with a single call to this function.)

string.sub

string.sub (s, i [, j])

Returns the substring of s that starts at i and continues until j; i and j can be negative. If j is absent, then it is assumed to be equal to -1 (which is the same as the string length). In particular, the call string.sub(s,1,j) returns a prefix of s with length j, and string.sub(s, -i) (for a positive i) returns a suffix of s with length i.

If, after the translation of negative indices, i is less than 1, it is corrected to 1. If j is greater than the string length, it is corrected to that length. If, after these corrections, i is greater than j, the function returns the empty string.

string.upper

string.upper (s)

Receives a string and returns a copy of this string with all ASCII lowercase letters changed to uppercase. All other characters are left unchanged.

Lua stringdk library

stringdk.quote_posix_shell

stringdk.quote_posix_shell (s)

Returns a string that is quoted as a single word in a POSIX shell.

stringdk.quote_value_shell_literal

stringdk.quote_value_shell_literal (s)

Returns a string that is quoted as one literal word in the Value Shell Language. The returned word evaluates to the exact text s, so any ${...} sequences inside s are treated as data rather than as VSL expansions.

Examples:

• stringdk.quote_value_shell_literal ("hello world") returns 'hello world'
• stringdk.quote_value_shell_literal ("abc${/}def") returns 'abc${/}def'

stringdk.quote_value_shell_evalable

stringdk.quote_value_shell_evalable (s)

Treats s as VSL code. It parses s as exactly one evalable Value Shell Language term and then re-emits that term in canonical single-word VSL form.

This means the function validates the syntax of s, preserves expandable constructs such as variables and subshells, and rejects strings that are not one valid evalable VSL term. If the input is already a canonical evalable term, the output may be unchanged.

Examples:

• stringdk.quote_value_shell_evalable ("abc${/}def") returns abc${/}def
• stringdk.quote_value_shell_evalable ("${SRC}") returns ${SRC}
• stringdk.quote_value_shell_evalable ("hello world") returns two values: nil and a string explaining why it is not one evalable VSL term
• stringdk.quote_value_shell_evalable ("${SRC") returns two values: nil and a string indicating the VSL syntax is invalid

stringdk.quote_windows_batch

stringdk.quote_windows_batch (s)

Returns a string that can be inserted verbatim as one whitespace-delimited word in a Windows cmd.exe command line or Windows batch script command that launches an executable. In other words, cmd.exe should parse the result as exactly one command word whose value is s.

This is intended for building command invocations such as executable paths and arguments. It is not a general-purpose escaping function for every batch-file construct such as if, set, or other contexts with additional cmd.exe parsing rules.

stringdk.sanitizesubpath

stringdk.sanitizesubpath (s)

Returns a normalization of the subpath s if and only if s is a strict subpath; that is:

• s, after file path normalization, does not begin with ..
• s has no unsafe file characters

If s can't be sanitized, a nil and an error string is returned.

This function supports conventional Lua assert checks: local sanitized = assert(stringdk.sanitizesubpath (s)).

Lua table library

This library provides generic functions for table manipulation. It provides all its functions inside the table table.

Remember that, whenever an operation needs the length of a table, all caveats about the length operator apply (see §3.4.7). All functions ignore non-numeric keys in the tables given as arguments.

table.concat

table.concat (list [, sep [, i [, j]]])

Given a list where all elements are strings or numbers, returns the string list[i]..sep..list[i+1] ··· sep..list[j]. The default value for sep is the empty string, the default for i is 1, and the default for j is the length of the list (ie. #list from Lua 5.1+). If i is greater than j, returns the empty string.

table.getn

table.getn (table)

CAUTION This Lua 5.0 function will be removed at a later date when the conventional Lua 5.1 table length operator # is introduced. Be prepared to change your Lua modules and rules when this happens.

Returns the size of a table, when seen as a list. If the table has an n field with a numeric value, this value is the size of the table. Otherwise, the size is one less the first integer index with a nil value.

Deprecated in Lua 5.1.

table.insert

table.insert (list, [pos,] value)

Inserts element value at position pos in list, shifting up the elements list[pos], list[pos+1], ···, list[#list]. The default value for pos is #list+1, so that a call table.insert(t,x) inserts x at the end of the list t.

table.move

table.move (a1, f, e, t [,a2])

Moves elements from the table a1 to the table a2, performing the equivalent to the following multiple assignment: a2[t],··· = a1[f],···,a1[e]. The default for a2 is a1. The destination range can overlap with the source range. The number of elements to be moved must fit in a Lua integer.

Returns the destination table a2.

Introduced in Lua 5.3.

table.pack

table.pack (···)

Returns a new table with all arguments stored into keys 1, 2, etc. and with a field "n" with the total number of arguments. Note that the resulting table may not be a sequence, if some arguments are nil.

table.remove

table.remove (list [, pos])

Removes from list the element at position pos, returning the value of the removed element. When pos is an integer between 1 and #list, it shifts down the elements list[pos+1], list[pos+2], ···, list[#list] and erases element list[#list]; The index pos can also be 0 when #list is 0, or #list + 1.

The default value for pos is #list, so that a call table.remove(l) removes the last element of the list l.

table.unpack

table.unpack (list [, i [, j]])

Returns the elements from the given list. This function is equivalent to

     return list[i], list[i+1], ···, list[j]

By default, i is 1 and j is #list.

Lua unified library

This library provides the functions available to unified scripts.

Almost all library functions return a type constant as their first return value. That constant helps text or graphical user interfaces decide how to print the remaining return values.

unified.existingoutput

% unified.existingoutput {}
'whatever is here like this number'
78
'is printed again including multiline table constructors like'
{
  a=1,
  b=2
}

unified.existingoutput {} is the contents of the output.

unified.existingoutput {}:

• reads the contents of the output as a sequence of Lua values
• returns the values from each Lua chunk

where each value is subject to the restrictions:

• there is at most one value per line
• each Lua value must be a data constructor (ie. nil, a string, a number or a table constructor)

In the example above, there are four (4) return values. The first is a Lua string, the second is a Lua number, the third is a Lua string, and the fourth is a Lua table.

Any Lua function can use unified.existingoutput {} to have a persistent memory of what happened the last time the Lua function was called.

Or any Lua function can use unified.existingoutput {} to read what was placed manually entered (ex. configuration values) in the output block.

unified.sections

# A literate script example - optional, ignored
## OurLibrary_Std.A.B.C@1.0.0

% unified.sections {}
'sections'
'A literate script example - optional, ignored'
'OurLibrary_Std.A.B.C'

unified.sections {} is the type constant section followed by the ATX headers.

unified.scriptmodver

# A literate script example - optional, ignored
## OurLibrary_Std.A.B.C@1.0.0

% unified.scriptmodver {}
'modver'
'OurLibrary_Std.A.B.C'
'1.0.0'

scriptmodver (the script module version) is three values:

1. The type constant modver.
2. The identifier for the script. Example: OurLibrary_Std.A.B.C
3. The version of the script. Example: 1.0.0

The script module version comes from the previous level 2 ATX header:

# A literate script example - optional, ignored

## OurLibrary_Std.A.B.C@1.0.0

... more of the unified script

## OurLibrary_Std.D.E.F@2.0.0

... more of the script that belongs to
... another package in the same library

The first word in the level 2 ATX header that is a valid MODULE@VERSION becomes the scriptmodver.

If there is no valid MODULE@VERSION then scriptmodver will be the values nil and an error message.

unified.asset

# A literate script example - optional, ignored
## OurLibrary_Std.A.B.C@1.0.0

Assumes that the script is .../*.u/run.u and that
.../*.u/data/gawk-5.3.1.tar.gz exists.
  % unified.asset { name="GawkTarball", file="data/gawk-5.3.1.tar.gz" }
  'asset'
  '6264553'
  { 'sha256:fa41b3a85413af87fb5e3a7d9c8fa8d4a20728c67651185bb49c38a7f9382b1e',
    'sha1:b82ae461dcc46e4aaa3381d9ada2a093e9aa1b49' }

Assumes that the script is .../*.u/run.u and that
.../*.u/user/share/gawk directory exists.
  % unified.asset { name="GawkShare", dir="usr/share/gawk" }
  'asset'
  '123456'
  { 'sha256:0000003812089120bc2a5d84f9e65cd0c25e4a4d724c80075c357239c74ae904',
    'sha1:99baee504a1fe91a07bc66b6900bd39874191889' }

unified.asset loads the local file or directory, and makes a singleton bundle from the asset. The local file or directory must be a strictly relative path.

unified.asset is available in any unified script that has a section name <standard module id>@<semantic version>, including the workspace script.

• name must be a standard namespace term (ie. begins with a capital letter)
• the bundle id is <scriptid>.<name>@<scriptver> where scriptid and scriptver are from unified.scriptmodver
• the origin is named the library id of scriptid (ex. OurLibrary_Std if scriptid = OurLibrary_Std.A.B.C) and has mirrors set to the library cell (ex. cell://OurLibrary_Std).

Returns nil and an error message, or three values:

1. The type constant asset.
2. The size of the asset.
3. A numbered table of the checksums.

The singleton bundle in the above example would be:

{
    "id": "OurLibrary_Std.A.B.C.GawkShare@1.0.0",
    "listing": {
        "origins": [
            {
                "name": "OurLibrary_Std",
                "mirrors": [
                    "cell://OurLibrary_Std"
                ]
            }
        ]
    },
    "assets": [
        {
            "origin": "OurLibrary_Std",
            "path": "data/gawk-5.3.1.tar.gz",
            "checksum": {
                "sha256": "fa41b3a85413af87fb5e3a7d9c8fa8d4a20728c67651185bb49c38a7f9382b1e"
            },
            "size": 6264553
        }
    ]
}

unified.envmod

Not implemented yet.

# A literate script example - optional, ignored
## OurLibrary_Std.A.B.C@1.0.0
### function!

% unified.envmod "+YACC=$(get-object CommonsBase_GNU.Bison@3.8.2
          -s Release.execution_abi -m ./bin/yacc -e bin/yacc -f yacc)"

unified.envmod adds an environment modification to the function OurLibrary_Std.A.B.C.Fx@1.0.0 (the first valid word in the previous level 2 ATX header, with Fx added).

The level 3 ATX header must be function!.

unified.output

Not implemented yet.

# A literate script example - optional, ignored
## OurLibrary_Std.A.B.C@1.0.0
### function!

% unified.output {}
[Release.Darwin_Arm64 Release.Darwin_x86_64 Release.Windows_x86_64]
include/gawkapi.h

[Release.Darwin_Arm64 Release.Darwin_x86_64]
bin/awk bin/gawk "etc/profile.d/gawk.sh"
"lib/gawk/time.so" "libexec/awk/grcat"

[Release.Windows_x86_64]
bin/awk.exe bin/gawk.exe "lib/gawk/time.dll"

unified.output defines the files that will be generated in the slot directores by the function OurLibrary_Std.A.B.C.Fx@1.0.0 (the first valid word in the previous level 2 ATX header, with Fx added).

The example above would be the equivalent of the form output in JSON:

"assets": [
  {
    "slots": [
      "Release.Darwin_arm64", "Release.Darwin_x86_64",
      "Release.Windows_x86_64"
    ],
    "paths": [
      "include/gawkapi.h"
    ]
  },
  {
    "slots": [
      "Release.Darwin_arm64", "Release.Darwin_x86_64"
    ],
    "paths": [
      "bin/awk", "bin/gawk",
      "etc/profile.d/gawk.sh",
      "lib/gawk/time.so", "libexec/awk/grcat"
    ]
  },
  {
    "slots": [
      "Release.Windows_x86_64"
    ],
    "paths": [
      "bin/awk.exe", "bin/gawk.exe",
      "lib/gawk/time.dll"
    ]
  }
]

The level 3 ATX header must be function!.

The shell output is the list of files. That is, the output function slurps (reads all of) the shell output, partitions the output based on [Slot1 ... SlotN] sections, and then splits each section into whitespace separated words.

Double quotes are required for any filename that has either whitespace or a double quote. An embedded double quote must be escaped with a second double quote. That is, "there is a middle "" double quote" is the filename there is a middle " double quote.

The return values are either a nil and an error message, or the values:

1. The type constant outputpaths.
2. A table whose keys are the slots and whose values are path lists. The key is a lexographically ordered list of slots (ex. {"Release.Darwin_arm64", "Release.Darwin_x86_64"}). The value is a path list (ex. {"bin/awk", "bin/gawk"}).

unified.assign

Not implemented yet.

# A literate script example - optional, ignored
## recipes!
% unified.assign { "coreutils",
    "$(get-object CommonsBase_Std.Coreutils@0.2.2 -s Release.Windows_x86_64 -m ./coreutils.exe -f : -e '*')"
  }
VAR:coreutils:=...

unified.assign { "xyz": "a-value" } sets the variable xyz to the value a-value such that in subsequent shell commands in the recipes! section like:

## recipes!
...
### print-value
Prints the value of xyz
$ echo The value is ${VAR:xyz}.
The value is a-value.

the ${VAR:xyz} expands to a-value.

The level 2 ATX header must be recipes!.

unified package target

Not implemented yet.

# A literate script example - optional, ignored
## OurTargets_Std.TheName@0.1.0
### targets!
% local ml = require("NotInriaCaml_Std.Build").at("1.0.0")
NotInriaCaml_Std.Build@1.0.0
% ml.jsonschema["library"]
url
https://www.schemastore.org/ocaml-library.json
% ml.library { name="something", ... }

The ml.library { name="something", ... } is interpreted as:

ml.target_library(command, request, continue_)

The .target_library behaves just like a Function Rule Function except it does not need to be attached to "rules" in rules, uirules = build.newrules(M).

The initial command is submit and the initial request is the table:

{
  user = {
    context = {
      sections={ "A literate script example", "OurTargets_Std.TheName@0.1.0", "targets!"},
      targetid="OurTargets_Std.TheName",
      targetver="0.1.0"
    },
    doc = { name="something", ... }
  }
}

If .jsonschema is present and the build implementation supports it, the inputs to the action (ex. .library) will be validated during interpretation with the JSON schema .jsonschema["library"]. The first return value is either url or inline.

If .luadefinition is present and the build implementation supports it, an IDE (ex. a Visual Studio Code extension based on Markdown Language Server and Lua Language Server) can load .luadefinition into a Lua .d.lua definition file to provide auto-complete.

Lua workspace globals

Global functions are available to unified scripts in the workspace section.

These functions return a type constant as their first return value. That constant helps text or graphical user interfaces decide how to print the remaining return values.

import

Imports a distribution.

## workspace

  %% import {
  ..   type="TYPE",
  ..   ... depends on TYPE ... }
  'import'
  'TYPE'
  { { 'CommonsBase_Std', '2.5.202603190707', {
      'blake2b-256:9d956430ebb347d46e0037e8094bb92b1fcbfa52603394b643685c40b489f7f0',
      'sha256:8e37f1d16259b643fbd3ce447d53e97ef321d96555bf9c3ba23a328108848ec6',
      'sha1:b82ae461dcc46e4aaa3381d9ada2a093e9aa1b49'
    } } }

The import will:

• download the distribution metadata
• validate the metadata
• place distribution metadata in the trace store (deprecated; https://github.com/diskuv/dk/issues/101)
• place distribution metadata in the source tree

The return values are either nil and an error message, or three values:

1. the type constant 'import'

2. the value 'TYPE' from import { type="TYPE", ... }

3. a table value whose:

   • keys are a versioned library that was distributed in the GitHub release
   • values are a numbered table of the checksums of the distribution metadata (.../<LIBRARY>-<VERSION>.values.json)

However, if there is existing output (ie. CommonsBase_Std@2.5.202603190707) and all of the LIBRARY@VERSION are present in the trace store or source tree, then the import command is skipped.

📢 The import's LIBRARY@VERSION outputs in the workspace section, along with the distribution metadata in the source tree, behave like lock files in package managers like npm and cargo.

An implementation may also place:

1. lazy value files in the value store by default to avoid the time and space to download every binary artifact from the distribution

import type=github-l2

Imports a distribution from a GitHub release.

## workspace

  %% import {
  ..   type="github-l2",
  ..   repo="OWNER/REPO",
  ..   host="",
  ..   tag="" }
  'import'
  'github-l2'
  { { 'CommonsBase_Std', '2.5.202603190707', {
      'blake2b-256:9d956430ebb347d46e0037e8094bb92b1fcbfa52603394b643685c40b489f7f0',
      'sha256:8e37f1d16259b643fbd3ce447d53e97ef321d96555bf9c3ba23a328108848ec6',
      'sha1:b82ae461dcc46e4aaa3381d9ada2a093e9aa1b49'
    } } }

• host defaults to github.com.
• repo are the two URL segments after the host. For example, the https://github.com/diskuv/dk.git GitHub project has repo=diskuv/dk.
• tag, if unspecified, is the latest release tag.

github-l2 will validate the release using GitHub's SLSA Level 2 attestations.

Custom Lua Modules

Any Lua script that returns a table is a Lua module that can be imported by other Lua scripts.

The simplest module is:

-- values.lua
local M = { id='MyLibrary_Std.A.B.MyModule@1.0.0' }
function M.somefunc()
  print('inside somefunc()')
end
return M

The id field is required, and is the same MODULE@VERSION used throughout the build system.

The module above can be imported in another values.lua script as follows:

MyModule = require('MyLibrary_Std.A.B.MyModule')
MyModule = MyModule.at('1.0.0')
if build.is_building then MyModule.somefunc() end

See Script Phases for why if build.is_building then ... end is required to guard expressions.

To avoid conflicts with other modules, an error will be raised if a field is exported that is a standard namespace term (ex. SomeModule).

Keeping your exports lowercased (or at least the first letter is lowercase) is sufficient to satify this restriction.

Introduction to Custom Lua Rules

Lua rules are Lua functions inside modules that dynamically build other values.

A simple rule MyRule is:

-- values.lua
local M = { id='MyLibrary_Std.A.B.MyModule@1.0.0' }
rules, _ignore_ui_rules = build.newrules(M)
function rules.MyRule(command, request, continue_)
  print('ok')
end
return M

The rule above can be run from the command line:

${dk_build_system} run-function MyLibrary_Std.A.B.MyModule.MyRule@1.0.0 -s Some.Slot -- a=1 b=2

or from a subshell in a values.json build file:

{
  // ...
  "forms": [
    "function": {
      "commands": [
        "echo",
        "$(run-function MyLibrary_Std.A.B.MyModule.MyRule@1.0.0 -s Some.Slot -- a=1 b=2)"
      ]
    }
  ]
}

or imported from another values.lua script:

MyRule = require('MyLibrary_Std.A.B.MyModule.MyRule')
MyRule = MyRule.at('1.0.0')
MyRule.use { a=1, b=2 }

Function Rules

Function rules (ie. M.fnrules) are rules that are free to be used everywhere: in values.json files and directly by the end-user.

Function rules should be pure functions (ie. repeat and get the same results on a different machine) so they do not have direct access to changeable project source code directories.

When a distribution script runs a function rule from a *.values.lua scriptmodule, the whole scriptmodule is included in the distributed package. That is the normal way to distribute Lua-only rule modules that do not also ship separate *.values.json[c] declarations.

By convention, function rule names are prefixed with F_ (for example F_Run or F_Untar). That keeps function rules distinct from interactive rule names like Run in the same scriptmodule.

The form of a function rule function named YourFreeRule is:

local M = { id='MyLibrary_Std.A.B.MyModule@1.0.0' }
rules = build.newrules(M)
function rules.YourFreeRule(command, request, continue_)
  -- your rule here
end
return M

The response to a function rule function must match the dk-rule-response.json schema.

The sequence of commands given to the function rule is:

   [command == "declareoutput"]
             |
             |
             v
     [command == "submit"]

The next sections describe what each command does.

Function Rule Command - declareoutput

The declareoutput command is the build system asking the function rule to declare the output keys and any static input dependencies before the rule adds tasks to the task graph.

The output keys can be object keys:

function rules.YourFreeRule(command, request)
  if command == "declareoutput" then
    return {
      -- "$schema" = "https://diskuv.com/dk/schema/dk-rule-response-1.0.json",
      declareoutput = {
        return_objects = {
          -- parse [request.user] to calculate `id` and `slots`
          id = "UserLibrary_Std.A.B.UserModule.OutputObject@1.0.0",
          slots = { "Release.Windows_x86_64", "Release.Darwin_arm64" },
          execution_slot = "Release.execution_abi"
        }
      }
    }
  end
end

or an asset key:

function rules.YourFreeRule(command, request)
  if command == "declareoutput" then
    return {
      -- "$schema" = "https://diskuv.com/dk/schema/dk-rule-response-1.0.json",
      declareoutput = {
        return_asset = {
          -- parse [request.asst] to calculate `id` and `path`
          id = "UserLibrary_Std.A.B.UserModule.OutputAsset@1.0.0",
          path = "some/file"
        }
      }
    }
  end
end

Static rule inputs are declared in the same declareoutput table:

function rules.YourFreeRule(command, request)
  if command == "declareoutput" then
    return {
      -- "$schema" = "https://diskuv.com/dk/schema/dk-rule-response-1.0.json",
      declareoutput = {
        return_asset = {
          id = "UserLibrary_Std.A.B.UserModule.OutputAsset@1.0.0",
          path = "some/file"
        },
        input_bundles = {
          { id = "UserLibrary_Std.A.B.SourceBundle@1.0.0" }
        },
        input_assets = {
          { id = "UserLibrary_Std.A.B.SourceBundle@1.0.0", path = "src.tar.gz" }
        },
        input_objects = {
          {
            id = "UserLibrary_Std.A.B.Tool@1.0.0",
            slots = { "Release.Windows_x86_64", "Release.Linux_x86_64" },
            execution_slot = "Release.execution_abi"
          }
        }
      }
    }
  end
end

For objects, the execution_slot is a literal or wildcard slot that translates the current execution platform to one of the return_object slots. Build system implementations will schedule the running of a task for the object that corresponds to the execution_slot.

In a distribution, the responsibility of the CI system is to ensure the graph is built on all execution platforms so all slots are available. Consider, for example, a C language build. The CI system must provide virtual machines or cross-compilers so that C binary artifacts (executables and libraries) are created for all the ABIs.

input_bundles, input_assets, and input_objects are the static dependencies of the rule itself.

Historical Note: This pattern of declaring the output before doing the building was inspired by Buck2's dynamic dependencies.

Function Rule Command - submit

The submit command is the entry point for the function rule to build artifacts by:

• add values to the valuestore and tasks to the task graph
• ask the build system for more information

All responses must set the field submit. That is:

return { submit = { values = ..., expressions = ..., commands = ..., andthen = ... } }

The fields that go into submit.values and submit.andthen are enumerated in the authoritative dk-rule-response.json schema.

The forms[].function field may be written as forms[].function_ to avoid conflicts with the Lua function keyword.

Consider the following response to a submit command:

return {
  submit = {
    values = {
      forms = {
        {
          id = "OurExample_Std.SomeModule@0.1.2",
          function_ = {
            commands = {
              "$(get-object CommonsBase_Std.Coreutils@0.2.2 -s ${SLOTNAME.Release.execution_abi} -m ./coreutils.exe -f coreutils.exe -e '*')",
              "sort",
              "--output",
              "${SLOT.request}/sorted-file",
              assert(request.user.filename, "please provide `filename=FILENAME`")
            }
          }
        }
      }
    },
    expressions = {
      files = {
        sorted_file =
          "$(get-object OurExample_Std.SomeModule@0.1.2 -s ${SLOTNAME.Release.execution_abi} -m ./sorted-file -f :file)"
      }
    },
    commands = {
      {"run-function", "OurExample_Std.SomeRule@3.4.5"}
    },
    andthen = {
      continue_ = {
        state = "have-sorted-file",
        passthrough = { someconstant = "the constant" }
      },
    }
  }
}

When the build system sees that response, the following sequence occurs:

1. the values are treated as if it were a new values.json file
2. all the expressions.strings are evaluated and will be made available as Lua strings in request.continued
3. all the expressions.files are evaluated and will be made available as readable file objects in request.continued
4. all the expressions.dirs are evaluated and will be made available as readable directory objects in request.continued
5. all the commands are evaluated
6. the rule function will get a callback (ie. andthen)

All four steps (values, expressions, commands, andthen) were optional.

The build system will perform the callback of the andthen with the following parameters:

-- YourFreeRule(command, request, continue_)
YourFreeRule(
  -- command
  "submit",
  -- request
  {
    user = ..., -- request.user table
    io = ..., -- request.io library
    continued = {
      -- anything in 'andthen.continue_.passthrough' is given literally to the rule
      someconstant = "the constant",
      -- anything in 'expressions.files', 'expresionss.dirs' and `expressions.strings`
      -- is evaluated and their responses given to the rule
      sorted_file = "...path to sorted-file..."
    }
  },
  -- continue_
  "have-sorted-file"
)

UI Rules

UI rules (ie. uirules) are rules that:

• only an end-user can run these rules; using UI rules inside a values.json[c] file will fail the build
• the dialog subcommand runs UI rules, while run-function is reserved for function rules
• interact with the end-user through a console or a graphical user interface
• only one UI rule may run at a time even if the build system implementation parallelizes noninteractive rules
• have access to the project source code directories through the request.ui library

UI rules are impure functions that have outputs that are not reproducible because they have direct access to changing project source code:

• request.ui.glob reads from the project source tree
• request.ui.spawn and request.ui.capture default to running in the user's working directory unless a cwd is supplied

Because UI rules are impure, UI rules are never cached. With request.ui.glob these impure UI rules can take immutable snapshots of the project source code (ie. assets); these immutable assets can be used directly or passed to pure function rules.

The form of a UI rule function named YourUiRule is:

local M = { id='MyLibrary_Std.A.B.MyModule@1.0.0' }
rules, uirules = build.newrules(M)
function uirules.YourUiRule(command, request, continue_)
  -- your rule here
end
return M

Please see Function Rules for a description of the command, request and continue_ arguments.

The response to a UI rule function must match the dk-rule-response.json schema.

The sequence of commands given to the UI rule is:

   [command == "submit"]
             |
             |
             v
     [command == "ui"]

The next sections describe what each command does.

UI Rule Command - submit

The submit command is the entry point for the UI rule to build artifacts by:

• adding values to the valuestore and tasks to the task graph
• ask the build system for more information

The responses to the command are the same as Function Rule Command - submit.

UI Rule Command - ui

The ui command is executed after the submit command.

Its purpose is to let UI rules do something with the built artifacts from the submit command.

A typical action would be to run the built artifact or display a summary of the artifacts.

Rule Argument - request

The details about the build request will be available as follows:

Field	Commands Applicable To	What
request.user	Function Rule submit	Rule request document translated from the arguments to run-function
	UI Rule submit	... The request document is described later in the Rule Request Documents section.
	UI Rule ui
	but not Embedded File Scripts
request.rule	Function Rule declareoutput	request.rule
	Function Rule submit
	UI Rule submit
	UI Rule ui
request.execution	UI Rule submit	request.execution
	UI Rule ui
request.io	Function Rule submit	request.io
	UI Rule submit
	UI Rule ui
request.continued	Function Rule submit	The last continuation. See continue_ argument
	UI Rule submit
	UI Rule ui
	Embedded File Scripts
request.ui	UI Rule submit	request.ui
	UI Rule ui
request.ui	UI Rule submit	request.ui
	UI Rule ui
request.srcfile.id	Embedded File Scripts	Asset id of the Lua is embedded in it, if any
request.srcfile.bundle	Embedded File Scripts	Bundle of the Lua is embedded in it, if any
request.srcfile.getasset	Embedded File Scripts	The shell command get-asset
		to get the asset in request.srcfile.bundle.
		The -f BASENAME or -f : argument must be added.
request.submit.outputid	Function Rule submit	MODULE@VERSION given by Function Rule declareoutput
request.submit.outputmodule	Function Rule submit	MODULE in MODULE@VERSION given by Function Rule declareoutput
request.submit.outputversion	Function Rule submit	VERSION in MODULE@VERSION given by Function Rule declareoutput

It is important to check whether user provided arguments have been provided. Consider using expressions like the following to check that they are set:

assert(request.user.filename, "Please provide `filename=FILENAME`")

Rule Argument - continue_

The continue_ argument is the state of a request. A request's boundaries is the start and stop of a single run-function command submitted by a user or a precommand or a subshell.

The continue_ value will be:

Value	Commands Applicable To	Notes
start	Function Rule submit	The first submit command will begin
	UI Rule submit	in the start state.
	Embedded File Scripts
last value	Function Rule submit	Subsequent submit commands for the
	UI Rule submit	same request will use the last submit
	Embedded File Scripts	response's submit.andthen.continue_.state
		field
Lua nil	UI Rule ui

Since rules block the build system, rules must be fast and often rules are broken into small steps using a state machine.

For example, a rule may need to sort a large file. It would be terrible for performance if a build system capable of parallelism was blocked to sort a file. Instead, the following states can be used:

       [start]
          |
          |
          v
   [have-sorted-file]
          |
          |
          v
       [done]

When the rule sees continue_=="start", it can return subshell expression to the build system to fetch a sort tool from the uutils coreutils project. Something like:

if command == "declareoutput" then
  local symbol = request.rule.generatesymbol()
  return {
    declareoutput = {
      return_asset = {
        id = "OurTest_Exec." .. symbol .. "@1.0.0",
        path = "SHA256.sig"
      }
    }
  }
elseif command == "submit" && continue_ == "start" then
  return {
    -- "$schema" = "https://diskuv.com/dk/schema/dk-rule-response-1.0.json",
    submit = {
      values = {
        forms = {
          {
            id = request.submit.outputid,
            function_ = {
              commands = {
                "$(get-object CommonsBase_Std.Coreutils@0.2.2 -s ${SLOTNAME.Release.execution_abi} -m ./coreutils.exe -f :exe:coreutils.exe)",
                "sort",
                "--output",
                "${SLOT.request}/sorted-file",
                -- the file to sort is provided by the user
                assert(request.user.filename, "provide `filename=FILENAME` on the command line")
              }
            }
          }
        }
      },
      expressions = {
        files = {
          sorted_file =
            "$(get-object " .. form_id .. " -s ${SLOTNAME.Release.execution_abi} -m ./sorted-file -f :file)"
        }
      },
      andthen = {
        continue_ = {
          state = "have-sorted-file"
        },
      }
    }
  }
end

The Function Rule Command - submit section describes in detail how the build system interprets the response. The salient part of the example is the andthen is a signal to the build system to call the rule function again.

The rule function will be called back with continue_ = "have-sorted-file"; when that happens, the rule should do something useful with the sorted file. For this example we just print the file.

if continue_ == "start" then
  -- ...
elseif continue_ == "have-sorted-file" then
  printf("The sorted file is at: %s\n", request.continued.sorted_file)
  return {
    -- an empty submit table means the rule is done.
    submit = { }
  }
end

Rule Request Documents

The request to Custom Lua Rules is always a JSON document.

In the introduction example of Custom Lua Rules:

MyRule = require('MyLibrary_Std.A.B.MyModule.MyRule')
MyRule = MyRule.at('1.0.0')
MyRule.use { a=1, b=2 }

the JSON document was converted from the Lua table { a=1, b=2 } into:

{ "a": 1, "b": 2 }

See jsondk.encode for how Lua values are converted to JSON. However, for rule requests the jsondk.null value is never encoded. That means a Lua nil is considered equivalent to a missing value.

The introduction example also submitted a request to a rule through the command line:

${dk_build_system} run-function MyLibrary_Std.A.B.MyModule.MyRule@1.0.0 -s Some.Slot -- a=1 b=2

Those command line arguments a=1 b=2 get converted into the same JSON document as before:

{ "a": 1, "b": 2 }

The conversion of command line arguments follows the withdrawn but still useful W3C HTML JSON Forms specification:

• ... -- name=Jane creates the request document {"name":"Jane"}
• ... -- pet[species]=Dahut kids[0]=Ashley creates the request document {"pets":{"species":"Dahut"},"kids":["Ashley"]}

Build systems are free to accept the form document directly from a HTML form as defined in W3C HTML JSON Forms specification or directly from a JSON document.

Embedded File Scripts

The body of a Lua UI rule function (called the guest Lua script) may be embedded as comments at the bottom of a larger file (called the host script).

Consider the following C# host script:

Console.WriteLine("This is running in C#!");
// local X = require("CommonsBase_Dotnet.SDK"); X = X.at("10.0.100-rc.2.25502.107")
// return X.run { ctx = ctx }
// !dk!s

It contains the guest Lua script extracted the bottom comments:

local X = require("CommonsBase_Dotnet.SDK"); X = X.at("10.0.100-rc.2.25502.107")
return X.run { ctx = ctx }

The guest Lua script is located using the !dk! marker that must appear on one of the last two nonblank lines within 16K of the end of the file.

Aside: The s in !dk!g means to extract the guest Lua script from the surrounding slash (//) comments. It is one of many possible codes to integrate with the most popular programming languages (see Embedded Language Codes section).

The build system is responsible for:

1. Creating an asset from the host script.

2. Converting the guest Lua script into a UI rule function:

   local M = { id = "... some generated id ..." }
   _rules, uirules = build.newrules(M)
   function uirules.EmbeddedFileScript(command, request, continue_)
     request.srcfile = { --[[ ... the host script asset ... ]] }
     local ctx = { command = command, request = request, continue_ = continue_, arg = arg }
     -- this is the guest Lua script
     local X = require("CommonsBase_Dotnet.SDK"); X = X.at("10.0.100-rc.2.25502.107")
     return X.run { ctx = ctx }
   end
   return M

   The Behavior of Embedded Lua describes request.srcfile in more detail.

3. Running the above UI rule function.

Behavior of Embedded Lua

The guast Lua script script will have the variables:

Global	What
arg	Command line arguments (see below)
command	At first submit and then ui
request	The request table (see below)
continue_	The state. See Rule Argument - continue_
ctx	The table:
	{ command=command, request=request, continue_=continue_, arg=arg }

The command line arguments, if any, will be the global table named arg that conforms to the Lua 5.4 "arg" library.

The request table is available as:

• request.io: the request.io

• request.submit: This table will be empty.

• request.user: This table will be empty. This is in contrast to the non-embedded UI rule where the command line arguments would be converted into Rule Request Documents.

• request.srcfile: An information table about the source file that contains the embedded Lua.

• request.srcfile.id: An asset identifer unique to the source file.

• request.srcfile.basename: The basename of the source file.

• request.srcfile.bundle: The bundle. For example:

  {
    id = "... value of request.srcfile.id ...",
    listing = {
      origins = {
        {
          name = "run",
          mirrors = { "selfasset://run" }
        }
      }
    },
    assets = {
      {
        origin = "run",
        path = "<basename_of_source_file>-<short_hash_of_source_file>",
        size = 151, -- replaced with real size
        checksum = {
          -- replaced with real SHA256
          sha256 = "0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"
        }
      }
    }
  }

• request.srcfile.getasset: The partially complete value shell command get-asset MODULE@VERSION -p PATH with MODULE@VERSION and PATH replaced with real values. To use the command in subshells, the -f BASENAME or -f : must be added to complete the value shell command.

The algorithm is:

1. Add a values.lua file to the valuestore that is a wrapper around the recognized embedded Lua:

   -- TheUniqueId replaced with a string based on the SHA-256 of the file
   -- that contains the embedded Lua.
   local M = { id = 'OurScript_Std.XTheUniqueId@0.1.0' }
   _rules, uirules = build.newrules(M)
   function uirules.Run(command,request,continue_)
     request.srcfile = request.srcfile or {}
     request.srcfile.id = "..."
     request.srcfile.bundle = {} -- ... it is populated
     -- embedded Lua goes here
   end
   return M

2. Add the arg table as a global variable populated with the actual command line arguments.

3. Add libraries and statements given to the Lua interpreter (-e and -l options)

4. Run the equivalent of run-function OurScript_Std.XTheIdentifier.Run@0.1.0 with no arguments. That runs the rule.

5. If -i given to the Lua interpreter, start a REPL.

Embedded Language Codes

The code table below is organized with the following character meanings:

• vertical: S is start, I is interior, E is end
• horizontal: S is start, E is end

<sp> means the space (ASCII 32) character. <sp 6> means six (6) spaces. (*) means no whitespace at start of pattern.

Codes are named after the patterns not the programming language unless the latter is unambiguous or historically pervasive.

Code	Languages	SS	IS	EE
h	PowerShell		#<sp>
	+ Python + Perl
	+ POSIX shell
	+ Ruby + R
	+ Assembly
	+ PHP
h	same as above		#
p	OCaml, Pascal	(*		*)
jd	Java, C, C++	/*	*<sp>	*/
	+ JavaScript
	+ Typescript
c	C, C++, C#, Java	/*		*/
	+ JavaScript
	+ Go + SQL
	+ Rust + Swift
	+ Dart
	+ Assembly
	+ PHP + Kotlin
s	C#		///
s	C, C++, C#, Java		//<sp>
	+ JavaScript
	+ Go + Dart
	+ Rust + Swift
	+ PHP + Kotlin
s	same as above		//
db	Lua	--[[		--]]
d	Lua, SQL, Ada		--<sp>
	+ Haskell
d	same as above		--
hs	Haskell	{-		-}
tq	Python	"""		"""
tq	Python	'''		'''
a	Visual Basic		'<sp>
a	Visual Basic		'
dos	Visual Basic		Rem<sp>
	+ Windows batch
dos	Windows batch		@REM<sp>
dos	Windows batch		::<sp>
dos	Windows batch		::
pod	Perl	(*) =begin<sp>		(*) =end<sp>
pod	Perl	(*) =for<sp>		(*) =cut
rb	Ruby	(*) =begin		(*) =end
x	XML + HTML	<!--		-->
	+ Markdown
sc	Assembly		;<sp>
sc	Assembly		;
e	Fortran		!<sp>
e	Fortran		!
sg	COBOL		(*) <sp 6>*<sp>
sg	COBOL		(*) <sp 6>*
sg	COBOL		(*) <sp 6>/<sp>
sg	COBOL		(*) <sp 6>/
sg	COBOL		*><sp>
sg	COBOL		*>
ps	PowerShell	<#		#>
sh	POSIX shell	: <<'END'		(*) END

Pascal curly brace comment delimiters, Fortran 77 fixed source delimiters, and COBOL inline comments are not supported. Nested comments are also not supported.

Recognizing Embedded Lua

The algorithm to recognize embedded Lua is:

• Strip any blank lines from the end of the file.
• For each line L of the source code, starting from the last line and going backwards to the second-last line (that is, only consider last two non-blank lines):
  • Let Lafter be the line after L (that is, Lafter is one line nearer to the end of the file), or the empty line if there L is the last line.
  • NEXT ROW: For each row R in the code table:
    • Let N(s) be a recognizer that matches leading whitespace followed by the literal IS(R) anchored to the beginning of the line s.
    • Let O(s) be a recognizer that matches the literal EE(R). If there is a (*) marker with EE(R) the match must be anchored to the beginning of the line s.
    • Let P(s) be a recognizer that matches the concatenation of !dk! and the Code(R) cellanywhere in the lines`.
    • Let Q(s) be a recognizer that matches the literal SS(R). If there is a (*) marker with SS(R) the match must be anchored to the beginning of the line s.
    • RECOGNIZE BLOCK: Does P recognize L and does O recognize either S or Lafter? That is, is P(L) && (O(L) || O(Lafter)) true?
      • Let Lq be the line before L that is recognized by Q (that is, Lq is one line or more nearer to the start of the file). If no Lq found within 16KB of L, skip to RECOGNIZE LINES.
      • All lines prior to L but after the Lq are considered the rule body.
        • If IS(R) is set in the code table with (*) marker, the literal IS(R) is removed if at the start of the line.
        • If IS(R) is set in the code table without (*) marker, the lines are trimmed of leading whitespace and the literal IS(R) is removed if at the start of the line.
        • The lines are also stripped of a trailing carriage return, if any.
        • DONE.
    • RECOGNIZE LINES: Does N recognize L? That is, is N(L) true?
      • Let Lnotn be the line before L that is not recognized by N (that is, Lnotn is one line or more nearer to the start of the file). If no Lnotn found within 16KB, continue to NEXT ROW.
      • All lines after Lnotn before L are considered the rule body.
        • If IS(R) has (*) marker, the literal IS(R) is removed from the start of the line.
        • If IS(R) has no (*) marker, the lines are trimmed of leading whitespace and the literal IS(R) from the start of the line.
        • The lines are also stripped of a trailing carriage return, if any.
        • DONE.
• If not DONE at this point, the file does not have an embedded Lua UI function body.

This evaluation strategy can dedent one (1) space when there are multiple rows with the same code. For example, the code h (POSIX shell, etc.) has a row with IS=#<sp> and a lower row with IS=#.

That means the embedded Lua in the file:

#!/bin/sh
echo 'In the beginning'
exit 0
# return {}
# !dk!h

is the following line which has been correctly dedented by one space:

return {}

Writing Lua Rules

Rule Requirements

A - RULE NAMING

Rule names must be standard namespace terms so they can be appended to the module id to create a new, still-valid module id.

Keeping the rule names with the first letter capitalized and no underscores is sufficient to satify this restriction.

B - RETURNED FIELDS

The basic syntax for making a conventional Lua module is:

local M = {}

-- you: add things to "M". For example,
--   rules, uirules = build.newrules(M)

return M

The M.id field is required for all modules.

The M.fnrules field is populated by build.newrules, and is required for all modules that export free rules. By convention the local variable is named "rules".

Likewise, the M.uirules field is populated by build.newrules, and is required for all modules that export interactive rules. By convention the local variable is named "uirules".

C - PERFORMANCE

Rules must be fast as they block the build system. Use continuations to delegate all the I/O intensive work to the build system.

D - LEXICAL STRUCTURE

During the VALUESCAN phase the values.lua files are scanned for rules in the procedure described in the Script Phases.

The practical implications are that the global scope should only be used for:

1. require function calls

2. defining functions:

   local M = {}
   function M.somefunc() print('VALUESCAN does not execute code inside functions') end

3. the final return statement

Anything that does not fit the above pattern should be guarded in a build.is_building condition:

if build.is_building then
  -- VALUESCAN will not execute code inside this code block
end

Error Handling in Rules

Lua has both assert(condition) and error "message" to raise errors. While using these functions are okay, especially for serious errors, these will expose Lua stack traces to your end-user.

The conventional way to indicate an error does not print a Lua stack trace. The convention is to return two values from a rule, the first of which is a nil and the second is the error message:

local M = {}
rules = build.define_rules(M)
function rules.MyRule(command,request)
  -- an error happened
  return nil, "this is the error message"
end
return M

Best Practices:

• Use nil, "error ..." for errors where the user was at fault (the person who submitted a request to the rule). The user forgetting to provide a required field is an example.
• Use assert and error defensively in preconditions, invariants and postconditions to catch programming errors. The resulting stack trace can be copy-pasted into a bug report by the user so you can fix the programmer error.

Form Document

🚧 This section is still under construction.

Information is supplied to a rule as a JSON document.

The primary way today to supply this JSON document is through the command line syntax run-function MODULE@VERSION -- CLI_FORM_DOC, where CLI_FORM_DOC is a CLI-based recipe to construct a JSON document.

The form has a options JSON object to describe how the JSON document submitted to a form maps to command line options, arguments and variables. nit: This should be "command==queryschema" given to rule ... it has nothing to do with the misnamed 'form' object in values.json!

The top-level fields of the form document are available in variables:

• ${PARAM.fieldname} is the text of the form field named fieldname, but it will error if the field is not a JSON string
• ${PARAMFILE.fieldname} is the file path to the JSON value of the form field named fieldname

The form document also contributes to the command line invocation of the form's function, if it has one.

Key Concept: The group is a layout of command line options and arguments that covers both the order of options and arguments, and also breaks like -- or subcommand names in the command line.

Form Command Line

The command is one array item of the form [<program>, <arg1>...] in "function": { "commands": [ [program1, args1...], [program2, args2...], ... ] }. Each command line is constructed as the concatenation of:

1. The <program>, <arg1>... for a single command.
2. The {"options": "fields": [...]} without any group field
3. The arguments in groups[0] (if any)
4. The {"options": "fields": [...]} with a group: 0 field (if any)
5. The arguments in groups[1] (if any)
6. The {"options": "fields": [...]} with a group: 1 field (if any)
7. ... and so on up to and including group 9
8. If {"options": "document": {...}} is present, an option and a location of a file containing the entire JSON form document

If <program> is a relative path, it is resolved against the function working directory before spawning on all platforms.

On Windows, that command line is rendered using cmd.exe command-word quoting and spacing rules because CreateProcessW takes one command-line string rather than a native argv array. The ["--cmd.exe", "/c", "<string>"] special form described above is the exception: it builds one explicit cmd.exe /c "..." payload from the supplied string with its own inner quoting rules.

Option Groups

Groups are necessary when you want some options and arguments to go before or after a -- seperator:

cmake -E rm -f -- file1 file2

or if you want some options and arguments to go before or after a subcommand:

git -C some_directory log --oneline

or if you need to order some options like how -L is required to be first:

find /home/user -L -name "*.log" -type f -exec rm {} \;

Since Windows especially but all operating systems have limits on the size of the command line arguments, the schema may specify a responsefile which consolidates all of the command line arguments at the end into a single file that can be read by the program (the first argument of the function args). Both MSVC and clang support these responsefiles.