Forms

A form has a request slot that must always be specified by the user as a parameter.

Form Variables

Variable Availability

Some variables are available in the Value Shell Language (VSL); see Variables available in VSL

All variables are available in .forms.function.args and .forms.function.envmods.

${SLOT.request}

The output directory for the request slot. The -s REQUEST_SLOT option (ex. get-object MODULE@VERSION -s REQUEST_SLOT) is the request slot.

If the command has no request slot (ex. get-bundle MODULE@VERSION) and you use ${SLOT.request}, an error is reported.

${SLOT.SlotName}

The output directory for the form function for the slot named SlotName.

SlotName is a period-separated list of "parts":

• lowercase context variables, and/or
• capitalized namespace terms

The namespace terms and context variables can be combined in any order.

See ${SLOTNAME.SlotName} for a description of parts and the context variables.

Output directories for the build system in install mode are the end-user installation directories, while for other modes the output directory may be a sandbox temporary directory.

Expressions are only evaluated if all the output types the expression uses are valid for the build system. For example, an expression that uses the output directory ${SLOT.Release.Darwin_arm64} will be skipped by the build system in install mode if the end-user machine's ABI is not darwin_arm64.

More generally:

Type	Expression Evaluated?	Immediate Thunk Controller
${SLOT.Release.Agnostic}	Always	A sandbox directory
${SLOT.Release.Darwin_arm64}	Always	A sandbox directory

Type	Expression Evaluated?	Install Thunk Controller
${SLOT.Release.Agnostic}	Always	The install directory
${SLOT.Release.Darwin_arm64}	Only if the end-user machine's ABI	The install directory
	is darwin_arm64

${SLOTABS.SlotName}

The absolute path of the output directory for the form function for the slot named SlotName.

CAUTION: ⚠️ Using this variable can lead to non-relocatable packages.

Prefer ${SLOT.SlotName} unless an absolute path is required.

${SLOTNAME.SlotName}

The name of the slot SlotName after expanding any context variables.

SlotName is a period-separated list of "parts":

• lowercase context variables, and/or
• capitalized namespace terms

The namespace terms and context variables can be combined in any order.

For example, ${SLOTNAME.Release.execution_abi} has two parts:

1. Release is a namespace term.
2. execution_abi is a context variable. Context variables are expanded according to the table below.

Context Variable	Example Value	Description
execution_abi	Windows_x86_64	The ABI for the execution platform.
target_abi	Windows_x86	The ABI of the generated executables and libraries.
		Defaults to the execution ABI.
request	Release.Agnostic	The request slot from the -s REQUEST_SLOT command line option
		(ex. get-object MODULE@VERSION -s Release.Agnostic)

If the command has no request slot (ex. get-bundle MODULE@VERSION) and you use the request context variable, an error is reported.

In the example we have been using, ${SLOTNAME.Release.execution_abi} will resolve to Release.Windows_x86_64 if the build is executing on a Windows 64-bit execution platform.

The list of execution_abi values is updated periodically from the t_abi enumeration (sum type) values in the dkml-c-probe project. At the time of writing, the list is:

• Android_arm32v7a
• Android_arm64v8a
• Android_x86
• Android_x86_64
• Darwin_arm64
• Darwin_x86_64
• DragonFly_x86_64
• FreeBSD_x86_64
• Linux_arm32v6
• Linux_arm32v7
• Linux_arm64
• Linux_x86
• Linux_x86_64
• NetBSD_x86_64
• OpenBSD_x86_64
• Windows_arm32
• Windows_arm64
• Windows_x86
• Windows_x86_64

${/} directory separator

The directory separator. Except for one edge case (below), it is always / even on Windows. That is, form commands can assume the / separator, which can simplify function code when the function interacts with MSYS2.

There is a special edge case for the build system in install mode: the build system in install mode will set the directory separator to \ on Windows and / on Unix. This allows installation to canonicalized UNC paths for Windows like the remote file \\Server2\Share\Test\Foo.txt or long-path capable \\?\C:\Test\Foo.txt.

${.exe.execution}

The executable suffix for the execution platform.

On a Windows execution platform it is .exe; otherwise it is empty.

${.exe.target}

The executable suffix for the target platform.

When the build system is running in "install" mode, the executable suffix will be:

• .exe on Windows
• `` on Unix

When the build system is running normally, the executable suffix will be .exe even on Unix. This behavior:

• reduces the need for seperate .precommands for Windows and Unix, and separate .function.args
• is a performance and space optimization since a common executable suffix increases the chances that non-ABI specific artifacts share the same hash across Windows and Unix.

${HOME}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the home directory to be the OS-specific home directory for the install end-user.

${CACHE}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the cache directory to be the OS-specific cache directory (ex. Temporary Internet Files on Windows, the XDG-compliant cache directory in Unix).

${DATA}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the data directory to be the OS-specific data directory (ex. LocalAppData on Windows, the XDG-compliant data directory in Unix).

${CONFIG}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the config directory to be the OS-specific config directory (ex. LocalAppData on Windows, the XDG-compliant config directory in Unix).

${STATE}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the state directory to be the OS-specific data directory (ex. LocalAppData on Windows, the XDG-compliant state directory in Unix).

${RUNTIME}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the runtime directory to be the OS-specific data directory (ex. LocalAppData on Windows, the XDG-compliant runtime directory in Unix).

${HOMEABS}, ${CACHEABS}, ${DATAABS}, ${CONFIGABS}, ${STATEABS}, ${RUNTIMEABS}

The absolute path to ${HOME}, ${CACHE}, ${DATA}, ${CONFIG}, ${STATE} or ${RUNTIME}.

Execution Constraints

Form functions may have execution constraints like the following that restricts the function to only run on a Windows execution platform:

        "execution": [
          {
            "name": "OSFamily",
            "value": "windows"
          }
        ]

These names and values follow the Platform Lexicon defined by the Bazel build tool: https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/platform.md/.

Tip: be careful! If you specify the ISA property pair, it may be ignored today but recognized in a future version.

If there is a need to constrain the execution, it is conventional to use a lookup table to map slots to execution values. The use of get-asset and the $(...) subshell will be explained in later sections. But for now, here is the convention that will restrict the execution of each of your slots to a specific OSFamily:

  "forms": [
    {
      // ...
      "function": {
        "execution": [
          {
            "name": "OSFamily",
            "value": "$(get-asset SomeWhere_Std.Lookup@1.0.0 -p osfamily -m ./${SLOTNAME.request})"
          }
        ],
        // ...
      },
      // ...
    }
  ],
  "bundles": [
    {
      "id": "SomeWhere_Std.Lookup@1.0.0",
      "listing": {
        "origins": [
          {
            "name": "table-pwsh",
            "mirrors": ["some-project-path/table"]
          }
        ]
      },
      "assets": [
        {
          "path": "osfamily",
          "origin": "table-pwsh",
          "size": 13980,
          "checksum": {
            "sha256": "5b59b7adbd5d4ccf24c51ac26e144754f0089f8c94de3b44a8bcf60ea81fb029"
          }
        }
      ]
    }
  ]

and then define in your project a folder some-project-path/table with files having the names of your slots:

• Release.Windows_x86_64 - the contents should be the OSFamily value windows
• Release.Darwin_arm64 - the contents should be macos
• etc.

Finally, use the --autofix flag to set the size and checksum fields automatically.

This is slightly complex on purpose ... the use of execution constraints is discouraged. Instead, try to make your package cross-platform.

Tip: In the future, remote execution platforms will be supported using most of the same mechanisms as distributions.

OSFamily

One of the following: windows, macos, linux (includes Android), netbsd, freebsd (includes DragonFly BSD), and openbsd.

Precommands

The precommands are a set of commands run before an form's function. It is not a sequence of commands since you cannot make assumptions about the order of the precommands until the sequential flag is enabled.

The following optimizations are allowed:

• Precommands may be run in parallel unless the "precommands": { "sequential": true, ... } flag is enabled.

Precommands will be skipped if the requested slot does not match the precommand output slot. For example, let's say you issue the command get-object THE_ID -s Release.Agnostic. Let's also say THE_ID object has two precommands:

1. get-asset ... -f ${SLOT.Release.Agnostic}
2. get-object ... -d ${SLOT.Something.Else}

Then the second precommand may be skipped because the requested slot Release.Agnostic does not match Something.Else.

Environment Modifications

The names in the environment follow the POSIX specification, except on Windows the environment names are folded to be case-insensitive. The rules are:

• The character set for the environment names is the Portable Character Set except = (<U003D>) and NUL.
• The environment name can't start with a digit
• MlFront restricts more strictly than POSIX for whitespace: the only whitespace accepted are spaces (<U0020>). Linefeeds and other non-space whitespace are not accepted.
• MlFront restricts more strictly than POSIX for control characters: the BEL (<U0007>) is not accepted.

The values in the environment are encoded as UTF-8 and may contain variables.

Because the first character of an environment modification unambiguously determines the type of modification, like + in +NAME=VALUE or - in -NAME, no escaping of environment names or values is required.

+NAME=VALUE

Add or set the environment variable with name NAME to VALUE.

The VALUE may contain variables.

You may use NAME= to set the environment variable to empty. It is preferable to use -NAME since programs have inconsistent behavior when environment variables are empty; some treat empty as an unset environment variable, while others treat empty as empty.

-NAME

Remove the environment variable with name NAME.

<NAME=VALUE

Prepends VALUE and a path separator to the environment variable with name NAME. However, the path seperator (; or : on Windows or Unix, respectively) is not added if the environment variable is empty.

The VALUE may contain variables.

For example, <PATH=C:\Windows\system32 prepends C:\Windows\system32; to the PATH on Windows and prepends C:\Windows\system32: to the PATH on Unix. In this example, the Unix prepending does not make sense, which is why the best practice is to use variables for the VALUE like <PATH=${CACHE}${/}bin so the modification is portable across operating systems.

Form Processing

Processing Order

The order of processing is as follows:

1. The form's subshells in the function args and envmods (if any) are executed, in parallel if supported by the build system.

2. The form's precommands are executed, in parallel if supported by the build system.

3. If there is a breakpoint from the enter-object command, a system shell (PowerShell, bash, etc.) is invoked.

4. Each form function command line (commands) is executed unless the requested slot does not match the command output slot. For example, let's say you issue the command get-object THE_ID -s Release.Agnostic. Let's also say commands has two commands:

   1. "get-asset", ..., "-f", "${SLOT.Release.Agnostic}"
   2. "get-object", ..., "-d", "${SLOT.Something.Else}"

   Then the second command may be skipped because the requested slot Release.Agnostic does not match Something.Else.

5. The form's output files are verified to exist.

6. The ${SLOT.slotname} that are part of the form's arguments and precommands are made available to other forms.

Execution Step Cacheing

Each precommand and each function command is a build task. A task is skipped (that is, its previous result is reused) when both of the following are unchanged from the previous run:

1. Task key: the command's argument values
2. Dependencies: the $(...) subshells. If any subshell dependency produces a different value (because the asset or object content changed), the dependencies differ and the step re-executes.

Tip

Avoid precommands that place values in intermediate directories rather than ${SLOT.*} directories. Instead use subshells. If you use intermediate directories the intermediate files won't be updated or invalidated.

// WRONG — precommand + cp = cached:
"precommands": { "private": ["get-asset MOD@VER -p path/to/script.py -f script.py"] },
"commands": [
  ["coreutils", "cp", "script.py", "${SLOT.request}/script.py"],
  ["python3", "script.py"]
]

// CORRECT — subshell creates dependency, re-runs when asset is updated or invalidated:
"commands": [
  ["python3", "$(get-asset MOD@VER -p path/to/script.py -f script.py)"]
]

Alternatively, each build system implementation has a way to update workspace asset checksums and invalidate values.

Windows command-line quoting

On Windows, function.commands are rendered into one lpCommandLine string parameter to the Windows API function call CreateProcessW. The string is assembled by cmd.exe (Windows Batch) quoting of each function.commands command (ie. commands with spaces will be surrounded with double quotes, etc.), then concatenating them all with spaces.

Windows --cmd.exe special form

When a form needs cmd.exe /c, prefer the special function.commands form whose exact shape is ["--cmd.exe", "/c", "<VSL-quoted string>"]. Spaces added for clarity:

{
  "function": {
    "commands": [
      "--cmd.exe",
      "/c",
      "\"  echo hi > `\"${SLOT.Release.Agnostic}\\some-file.txt`\"   \""
    ]
  }
}

Stripped of the VSL quoting the unquoted argument to /c is:

   echo hi > "${SLOT.Release.Agnostic}\some-file.txt"

That expands to being the lpCommandLine parameter to the Windows API function call CreateProcessW:

cmd.exe /c "   echo hi > "...\somewhere\Release.Agnostic\some-file.txt"   "