Values

Value Shell Language (VSL)

All encodings of VSL are UTF-8 unless explicitly noted as different.

There is a POSIX shell / PowerShell styled language to query for objects and assets. For example, the "command":

get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe

will get the object with the id OurStd_Std.Build.Clang@1.0.0 and place it in the clang.exe file.

There are two ways to run these shell commands:

1. Directly from the command line. For example, ${dk_build_system} get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe where ${dk_build_system} could be the dk0 reference implementation.

2. Embedded as "precommands" in a values file. For example,

   { // ...
    "precommands": {
      "private": [
        "get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe"
      ]
    },
    // ...
   }

All commands have a output path (ex. -f echo.exe). Most command have two forms:

• -f FILE (ex. -f echo.exe)
• -d DIR (ex. -d target)

but some commands may only have the -d DIR directory output.

The best practice for relative paths is to use the forward slash (/) as directory separator in the output paths for readability (no escaping in JSON) and portability (backslashes don't work on Unix). However, when the path is an absolute directory, use the native format, including UNC paths on Windows (ex. \\Server2\Share\Test\Foo.txt).

For security, the commands may be evaluated in a sandbox or a chroot environment. Do not use .. path segments or they may fail to resolve in sandboxes.

VSL Lexical Rules

A value shell command is a command line that is split into words.

In a JSONC (JSON with comments) values file, each precommand is a command line:

"precommands": {
  "private": [
    // command line 1
    "get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe",
    "...",
    // command line N
    "get-object OurStd_Std.Build.GCC@1.0.0 -s Release.Agnostic -f gcc.c"
  ]
}

The get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe command line has the words get-object, OurStd_Std.Build.Clang@1.0.0, -s, Release.Agnostic, -f, and clang.exe.

Words are directly used in other places in the values file:

"function": {
    "commands": [
      // word 1
      "sh",
      // word 2
      "-c",
      // word 3
      "\"find . > ${SLOT.Release.Agnostic}/some-file\""
    ],
    "envmods": [
      // word 4: ${CACHE}/dkcoder
      "+DKCODER_CTX_CACHE_DIR=${CACHE}/dkcoder"
    ]
}

so the individual words can be assembled into a command line.

Each function argument in args and each environment modification value in envmods must be one value shell word. In word 3 we used double quotes to squash several words into one shell word. In the next section the different ways to form words are explained.

Types of Words

The get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe from the previous section had the words:

• get-object
• OurStd_Std.Build.Clang@1.0.0
• -s
• Release.Agnostic
• -f
• clang.exe

These are examples of bare words; that is, words without any surrounding quotes.

Each bare word is one or more of the following components without any interleaving spaces:

• literals like clang.exe
• variables like ${SLOT.Release.Agnostic} that get expanded to the string value of the named variable from the next section Variables available in VSL
• subshells like $(get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -d :) that get expanded to a temporary file or directory output of the subshell expression

An example of a single bare word that has all three types of components is:

$(get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -d :)/clang${.exe}

which is compromised of a subshell which could expand to some/dir/clang-1.0.0/bin, the literal /clang and the variable ${.exe} which could expand to .exe. That is, the bare word could expand to some/dir/clang-1.0.0/bin/clang.exe. No quotations were required.

The lexical rules allow for a subshell to be nested in another subshell, although nested subshells should not be required until form parameters are added. For completeness, the bare word $(get-asset MyAssets_Std.Bundle@1.0.0 -p $(get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -d :) -d :) is valid, although nonsensical.

VSL words can also have quotes surrounding them so that spaces can be handled. The three types of words are:

1. Bare words (we've already covered these) with characters that are not whitespace, single quotes (') or double quotes (").
2. Single-quoted words like 'C:\My Documents'
3. Double-quoted words like "${CONFIG}\Floor Plans\Master Bedroom.rvt"

Backticks (`) are ordinary literal characters in bare words. They do not escape the following character unless they are inside a double-quoted word.

Single-quoted words ('...') evaluate literally to the text inside the single quotes, including any whitespace and newlines.

Double-quoted words ("...") squash many bare words into a single word by:

• evaluating each "inner bare word" (each bare word inside the double-quotes) using the rules above. However, the inner bare words can contain single-quote (') characters which are treated like any ordinary character.
• keeping the whitespace between each inner bare words

The inner bare words and the whitespace between the inner words are concatenated into a single word.

Within double-quotes:

• single-quote (') characters are ordinary literal characters
• literal double-quote (") characters must be escaped using the ` (backtick, aka. grave accent) as the escape character
• literal backtick (`) characters must be escaped using the ` (backtick, aka. grave accent) as the escape character

Historical reasoning: Backticks were chosen for compatibility with Windows paths and familiarity with PowerShell; carets ^ were rejected since in Windows Batch the caret has complex rules.

See ESCAPING.md for worked examples of simple and complex VSL escaping.

Variables available in VSL

• ${SLOT.slotname}
• ${/}
• ${SRC}
• ${HOME}
• ${DATA}
• ${CONFIG}
• ${STATE}
• ${RUNTIME}

remote UI_MODULE@VERSION COMMAND

remote UI_MODULE@VERSION [REMOTE_OPTION=VALUE...] [REPOSITORY] COMMAND...

Run a local value shell command, or a top-level test, lua, or run command, on a remote execution engine.

If UI_MODULE@VERSION is not fully qualified, short forms <NamespaceTerm>*@<Ver> are expanded to CommonsBase_Remote.<NamespaceTerm>*@<Ver> like:

remote GitHub@0.1.0 run-object MODULE@VERSION -s REQUEST_SLOT -m ./tool -- --help

into:

remote CommonsBase_Remote.GitHub@0.1.0 run-object MODULE@VERSION -s REQUEST_SLOT -m ./tool -- --help

and short forms <Vendor><Qualifier>.<NamespaceTerm>*@<Ver> are expanded to <Vendor><Qualifier>_Remote.<NamespaceTerm>*@<Ver> like:

remote BuildBuddy.Cloud@0.1.0 run-object MODULE@VERSION -s REQUEST_SLOT -m ./tool -- --help

into:

remote BuildBuddy_Remote.Cloud@0.1.0 run-object MODULE@VERSION -s REQUEST_SLOT -m ./tool -- --help

The REPOSITORY argument is optional when the remote rule can infer or has configured a repository. GitHub repositories are written in a form accepted by the remote rule, such as github.com/OWNER/REPO.

Leading REMOTE_OPTION=VALUE words are passed to the remote rule before the optional repository and inner command are parsed.

Tip

The CommonsBase_Remote.GitHub rule recognizes the following options (with defaults):

• workspace=dk.u
• sessions=4
• retention=8
• create_repo=false
• dry_run=false

The GitHub remote rule uses a workspace script, defaulting to dk.u, and a maximum session count, defaulting to 4. If create_repo=true, the rule creates a private GitHub repository when the requested repository does not already exist. It stores the age recipient and its OpenBSD signify public signature in GitHub Actions repository variables, stores the age secret key in a GitHub secret, and transfers staged local assets through GitHub prereleases. Each polling iteration for workflow runs or prereleases is reported as progress. If a git fetch followed by git rebase updates a session branch, the session is treated as contended; the implementation waits 30 seconds and tries another session.

The words after the remote options and optional repository are parsed as the inner command. For value shell commands, the inner command uses the same VSL variable and subshell expansion as a direct local invocation. In particular, run-object and run-asset command-line arguments may contain variables and subshells such as $(get-object ...). The audited command stored in the session branch is a single-line POSIX shell command, so the GitHub workflow (etc.) can pass the exact argument vector to the build tool.

Before submitting work, an implementation prints the resolved inner command in value shell syntax, using two spaces between top-level terms. This is intentionally different from shell syntax and is shown so the user can see exactly what command is being sent to the remote execution engine.

Remote value shell commands must return results in the same observable shape as local commands. For example, run-object and run-asset still produce the same captured stdout/stderr value shape, and the local implementation imports the returned valuestore/tracestore data before presenting output. Remote test, lua, and run return the remote exit status and display the remote stdout, stderr, and log output.

Note

For remote execution, CommonsBase_Remote.GitHub creates exactly one object value. That object contains a Cap'n Proto metadata file plus payload files: stdout/stderr text files or a zip holding directory contents. The single object value and its single leaf trace (that is, no dependency trace) are then exported and re-imported with the same orchestration as distribute and import-github-l2.

To avoid a single leaf trace from interfering with conventional traces, a synthetic key is used. Only the get-asset and get-bundle commands, the commands that naturally have no dependencies, do not use synthetic keys.

Local -f FILE and -d DIR extraction happens after import.

get-object MODULE@VERSION -s REQUEST_SLOT (-f FILE | -d DIR/)

Get the contents of the slot REQUEST_SLOT for the object uniquely identified by MODULE@VERSION.

Option	Description
-f FILE	Place object in FILE
-d DIR/	The object must be a zip archive, and its contents are extracted into DIR/.
-n STRIP	See Option: [-n STRIP]
-m MEMBER	See [Option: -m MEMBER]
-x GLOB	Exclude globbed files from -d DIR/. May be repeated.
-e GLOB	Make globbed files executable in -d DIR/ and -f FILE. May be repeated.

See Options: -f FILE, -d DIR, -x GLOB, and -e GLOB for output behavior.

The object ID implicitly or explicitly contains build metadata; see ID with Build Metadata.

run-object MODULE@VERSION -s REQUEST_SLOT (-c COMMAND | -m MEMBER)

Run a command from the object slot REQUEST_SLOT for the object uniquely identified by MODULE@VERSION.

Option	Description
-c COMMAND	Run the strictly relative COMMAND inside the anonymous directory tree produced by get-object -d :.
-m MEMBER	Extract only MEMBER, normalize it as a file path, and use that normalized path as the command to run.
-n STRIP	See Option: [-n STRIP]. Only valid with -c COMMAND.
-x GLOB	Exclude globbed files from the anonymous directory. May be repeated.
-e GLOB	Make globbed files executable in the anonymous directory. May be repeated.
-- [ARGS...]	Interpret ARGS as a value shell command line with variables and subshells, like function.commands words.

The command is executed with the same current working directory model as run-function, but the executable path is always resolved relative to the anonymous command directory. COMMAND and normalized MEMBER must be strictly relative file paths.

The standard output and standard error streams are captured in separate files but stored as a single x<ID> value in the valuestore.

The output and error stream files are capped at 16777211 bytes, which is the lower of the approximate Cap'n Proto Text payload limit of 512 MB and the maximum string length on 32-bit platforms.

Tip

For larger content, redirect the command's standard output or standard error to form rule output files using shell redirection or a log capture executable.

run-function MODULE@VERSION (-f FILE | -d DIR/) -- CLI_FORM_DOC

Run the Lua function rule uniquely identified by MODULE@VERSION using the JSON constructed from CLI_FORM_DOC.

Option	Description
-f FILE	Place object in FILE
-d DIR/	The object must be a zip archive, and its contents are extracted into DIR/.
-n STRIP	See Option: [-n STRIP]
-m MEMBER	See [Option: -m MEMBER]
-x GLOB	Exclude globbed files from -d DIR/. May be repeated.
-e GLOB	Make globbed files executable in -d DIR/ and -f FILE. May be repeated.

If no -f or -d option is given, the object is dumped to the standard output (ie. your console).

See Options: -f FILE, -d DIR, -x GLOB, and -e GLOB for output behavior.

See Form Document for the CLI_FORM_DOC form parameters. If there are none, the -- CLI_FORM_DOC can be left out.

The object ID implicitly or explicitly contains build metadata; see ID with Build Metadata.

enter-object MODULE@VERSION -s REQUEST_SLOT -- CLI_FORM_DOC

Enter a shell like PowerShell or /bin/bash that has the contents of the slot REQUEST_SLOT for the object uniquely identified by identifier ID.

The shell is meant only for debugging problems, and may not appear if the object ID has been successfully built.

See Form Document for form parameters. If there are none, the -- CLI_FORM_DOC can be left out.

The object MODULE@VERSION implicitly or explicitly contains build metadata; see ID with Build Metadata.

merge-object MODULE@VERSION -s REQUEST_SLOT (-f FILE | -d DIR/)

Build the object uniquely identified by MODULE@VERSION and merge the contents of the slot REQUEST_SLOT into the requested output.

Option	Description
-f FILE	Place merged object in FILE
-d DIR/	Merge contents of the zip archive into the existing directory DIR/. The object must be a zip archive.
-n STRIP	See Option: [-n STRIP]
-m MEMBER	See [Option: -m MEMBER]
-x GLOB	Exclude globbed files from -d DIR/. May be repeated.
-e GLOB	Make globbed files executable in -d DIR/ and -f FILE. May be repeated.

More than one merge-object can use the same output directory DIR.

See Options: -f FILE, -d DIR, -x GLOB, and -e GLOB for output behavior.

The object MODULE@VERSION implicitly or explicitly contains build metadata; see ID with Build Metadata.

get-asset MODULE@VERSION FILE_PATH (-f FILE | -d DIR/)

Get the contents of the asset at FILE_PATH for the bundle MODULE@VERSION.

Option	Description
-f FILE	Place asset in FILE
-d DIR/	The asset must be a zip archive, and its contents are extracted into DIR/.
-n STRIP	See Option: [-n STRIP]
-m MEMBER	See [Option: -m MEMBER]
-x GLOB	Exclude globbed files from -d DIR/. May be repeated.
-e GLOB	Make globbed files executable in -d DIR/ and -f FILE. May be repeated.

See Options: -f FILE, -d DIR, -x GLOB, and -e GLOB for output behavior.

run-asset MODULE@VERSION FILE_PATH (-c COMMAND | -m MEMBER)

Run a command from the asset FILE_PATH in the bundle MODULE@VERSION.

Option	Description
-c COMMAND	Run the strictly relative COMMAND inside the anonymous directory tree produced by get-asset -d :.
-m MEMBER	Extract only MEMBER, normalize it as a file path, and use that normalized path as the command to run.
-n STRIP	See Option: [-n STRIP]. Only valid with -c COMMAND.
-x GLOB	Exclude globbed files from the anonymous directory. May be repeated.
-e GLOB	Make globbed files executable in the anonymous directory. May be repeated.
-- [ARGS...]	Interpret ARGS as a value shell command line with variables and subshells, like function.commands words.

The command is executed with the same current working directory model as run-function, but the executable path is always resolved relative to the anonymous command directory. COMMAND and normalized MEMBER must be strictly relative file paths.

The standard output and standard error streams are captured in separate files but stored as a single x<ID> value in the valuestore.

The output and error stream files are capped at 16777211 bytes, which is the lower of the approximate Cap'n Proto Text payload limit of 512 MB and the maximum string length on 32-bit platforms.

Tip

For larger content, redirect the command's standard output or standard error to form rule output files using shell redirection or a log capture executable.

get-bundle MODULE@VERSION (-f FILE | -d DIR/)

Get the archive file for the bundle MODULE@VERSION.

Option	Description
-f FILE	Place bundle in FILE. FILE will be a zip archive with all the bundle files.
-d DIR/	All the bundle files will be extracted into DIR/.
-n STRIP	See Option: [-n STRIP]
-x GLOB	Exclude globbed files from -d DIR/. May be repeated.
-e GLOB	Make globbed files executable in -d DIR/ and -f FILE. May be repeated.

See Options: -f FILE, -d DIR, -x GLOB, and -e GLOB for output behavior.

What about the -m MEMBER option?

Use get-asset to get a specific asset. Having a -m MEMBER option would be equivalent but redundant and slightly confusing since not much about assets implies they are stored as archives.

Options: -f FILE and -d DIR and -x GLOB and -e GLOB

Each output file (a single file if -f FILE or zero or more if -d FILE) has a normalized entry name.

Any -x GLOB or -e GLOB option operates on a glob matched against the normalized entry name:

• For -f FILE the normalized entry name is the basename of FILE.
• For -d DIR the normalized entry name is zip archive entry name, with any redundant ./ and somedir/../ path segments removed, and leading segments removed if -n STRIP was used.

For example, the options -e '**/*.exe' -e 'bin/*' -x 'doc/**' will cause any .exe output file to be made executable, and any files in the output bin/ directory to be made executable, and to not output any entries (including deeply nested entries) to the doc/ directory.

When an output file is made executable:

• On Unix the file's executable bit will be set (ie. chmod a+x OUTPUTFILE)
• On macOS, if the output file is a Mach-O file that needs a code signature and the output file is *not within an Apple app or bundle, then the file will be code signed with the machine signature (ie. /usr/bin/codesign --sign - --force OUTPUTFILE). Any output file within a *.app/Contents/** file tree or *.bundle/Contents/** file tree (case-insensitive) is considered to be within an Apple app or bundle. For example, if the output file were t/x/y/SomeApp.app/Contents/bin/somefile it would not be codesigned because SomeApp is presumed to be signed already within an existing or future x/y/SomeApp.app/Contents/_CodeSignature/.

No command may write to the same output file. Specifically:

• It is an error to have more than one get-object or get-bundle or get-asset or resume-object or merge-object use the same FILE.
• It is an error to have more than one get-object or get-bundle or get-asset or resume-object use the same DIR or otherwise overlap the same DIR. Overlapping means one command can't write to the subdirectory of another command's DIR.

Use [merge-object](#merge- --s-request_slot--f-file---d-dir----cli_form_doc) when you want to write into the same directory. Even so, no merge-object may extract the same file in the same output directory.

Option: [-n STRIP]

-n STRIP defaults to zero.

STRIP is how many levels of the zip archive to strip. Many zip archives place all content under a versioned root directory:

llvmorg-19.1.3-win32/
  LICENSE.txt
  README.txt
  src/

To leave the directory structure as-is, set STRIP to 0. To strip away the top level llvmorg-19.1.3-win32 directory, set STRIP to 1.

Option: [-m MEMBER]

Gets the zip file member from the object or asset, which must be a zip archive.