Subshells

subshell options

A subshell may have options for the build system that precede the subshell command.

Each option must:

• precede the subshell command (ie. precede get-object, etc.)
• start with a dash (-)
• be a single valueshell word

For example, $(--path=absnative -q '-x=still one word' get-object ...) has three options:

• --path=absnative
• -q
• -x=still one word

Any options that are not recognized by the build system must be ignored.

The recognized options are:

Option	Description
--path=STYLE	Sets the subshell to return a path in one of the styles:
	- rel: default. relative path, Unix style
	- absnative: absolute Windows path on Windows, Unix path on Unix
	- absunix: absolute Unix-style path (ex. /a/b, C:/a/b) if
	possible. Unchanged if not (ex. Windows UNC paths).

subshell: get-object MODULE@VERSION -s REQUEST_SLOT

Get the contents of the slot REQUEST_SLOT for the object uniquely identified by MODULE@VERSION.

Option	Description
-f : or -f BASENAME	Place object in an anonymous file and return its filepath
-d :	The object must be a zip archive, and its contents are extracted into an anonymous directory.
-n STRIP	See Option: [-n STRIP]
-m MEMBER	See [Option: -m MEMBER]
-x GLOB	Exclude globbed files from -d DIR/. May be repeated.
-e GLOB	Make globbed files executable in -d DIR/ and -f .... May be repeated.

If none of the -f :, -f BASENAME, or -d : options are specified, the contents are captured and returned with the following restrictions:

• the content may not exceed 1024 bytes
• no translation is performed on the bytes (UTF-16 is not translated to UTF-8, etc.)
• the byte 0 (ASCII NUL) may not be in the content as a security measure

subshell: run-object MODULE@VERSION -s REQUEST_SLOT (-c COMMAND | -m MEMBER)

Run the executable selected by -c COMMAND or -m MEMBER from the object slot REQUEST_SLOT in MODULE@VERSION.

The -f :, -f BASENAME, and -d : options are accepted.

The -x GLOB, -e GLOB and -n STRIP options keep the same meanings they have for the top-level run-object command when selecting the command tree.

The returned content has the same restrictions as the other content-returning subshells:

• the content may not exceed 1024 bytes
• no translation is performed on the bytes (UTF-16 is not translated to UTF-8, etc.)
• the byte 0 (ASCII NUL) may not be in the content as a security measure

subshell: run-function MODULE@VERSION -- CLI_FORM_DOC

Submit the JSON constructed from CLI_FORM_DOC to the rule uniquely identified by MODULE@VERSION.

Option	Description
-f : or -f BASENAME	Place object in an anonymous file and return its filepath
-d :	The object must be a zip archive, and its contents are extracted into an anonymous directory.
-n STRIP	See Option: [-n STRIP]
-m MEMBER	See [Option: -m MEMBER]
-x GLOB	Exclude globbed files from -d DIR/. May be repeated.
-e GLOB	Make globbed files executable in -d DIR/ and -f .... May be repeated.

See Form Document for the CLI_FORM_DOC form parameters. If there are none, the -- CLI_FORM_DOC can be left out.

If none of the -f :, -f BASENAME, or -d : options are specified, the contents are captured and returned with the following restrictions:

• the content may not exceed 1024 bytes
• no translation is performed on the bytes (UTF-16 is not translated to UTF-8, etc.)
• the byte 0 (ASCII NUL) may not be in the content as a security measure

subshell: get-asset MODULE@VERSION FILE_PATH

Get the contents of the asset at FILE_PATH for the bundle MODULE@VERSION.

Option	Description
-f : or -f BASENAME	Place asset in an anonymous file and return its filepath
-d :	The asset must be a zip archive, and its contents are extracted into an anonymous directory.
-n STRIP	See Option: [-n STRIP]
-m MEMBER	See [Option: -m MEMBER]
-x GLOB	Exclude globbed files from -d DIR/. May be repeated.
-e GLOB	Make globbed files executable in -d DIR/ and -f .... May be repeated.

If none of the -f :, -f BASENAME, or -d : options are specified, the contents are captured and returned with the following restrictions:

• the content may not exceed 1024 bytes
• no translation is performed on the bytes (UTF-16 is not translated to UTF-8, etc.)
• the byte 0 (ASCII NUL) may not be in the content as a security measure

subshell: run-asset MODULE@VERSION FILE_PATH (-c COMMAND | -m MEMBER)

Run the executable selected by -c COMMAND or -m MEMBER from the asset at FILE_PATH in MODULE@VERSION.

The -f :, -f BASENAME, and -d : options are accepted.

The -x GLOB, -e GLOB and -n STRIP options keep the same meanings they have for the top-level run-asset command when selecting the command tree.

The returned content has the same restrictions as the other content-returning subshells:

• the content may not exceed 1024 bytes
• no translation is performed on the bytes (UTF-16 is not translated to UTF-8, etc.)
• the byte 0 (ASCII NUL) may not be in the content as a security measure

Anonymous Files: -f : or -f BASENAME

Place the object or asset in an anonymous regular file and return the file path.

The file will be named BASENAME and placed in a directory with no other files.

If the simplified -f : expression is used, the file will be named a.dat.

The file will be made executable if one of the -e GLOB patterns match. The executable will:

• on Unix execution machines its executable bits are set (bitwise OR 0o111)
• on macOS execution machines it will be codesigned if it is a Mach-O file that must be signed

Conventionally if the file is to be made executable, the BASENAME includes an .exe extension so it can run on Windows. However the .exe extension is not required if the file will not run on Windows.

Anonymous Directories: -d :

Place the object or asset in an anonymous directory and return the directory path.

Object ID with Build Metadata

These rules apply to the *-object commands only:

• get-object MODULE@VERSION ...
• merge-object MODULE@VERSION ...
• run-function MODULE@VERSION ...
• enter-object MODULE@VERSION ...

The purpose of these rules is to ensure that unique builds can be uniquely and deterministically identified.

Versions can have explicit build metadata. For example, the VERSION 1.0.0+bn-20250801235901.commit-054d5983 has the two dot-separated build metadata fields: bn-20250801235901 and commit-054d5983.

If the version VERSION has explicit build metadata in the format bn-*, then the object is locked to that specific build number. In the above example the object is locked to build number 20250801235901 because that is the build metadata with format bn-*.

When the version VERSION has no explicit build metadata, or the version VERSION's build metadata does not include a bn-* field, then the first matching rule of the following rules determines what the build metadata will be:

1. If a lockfile has a build metadata reference (ex. 1.0.0 = bn-20250801235901+commit-054d5983), the build metadata is used.
2. The constructive trace store list of traces key(i), dependencies(i), result(i) is scanned. If there is a trace i where the version of key(i) matches the VERSION and where result(i) is an object value, then the build metadata of the latest such key(i) will be used.
3. The build metadata will be constructed from the -t TIMESTAMP command line option, with the bn-YYYYMMDDhhmmss format.
4. The build metadata will be bn-20250101000000.

Important: the system clock is never consulted.

In CI, the best practice is to use the -t TIMESTAMP option, and base it on either:

• the source control commit timestamp.
  • Pro: Very easy to go back to the source code.
  • Con: If you need to force a new build from existing source code, you must create an empty commit-
• a monotonically increasing build number (ex. GITHUB_RUN_NUMBER if you use GitHub Actions).
  • Pro: No empty commits.
  • Con: Depending on your CI provider, it may be hard to go from a build number back to the source code.

Here are some examples for using the source control commit timestamp:

.github/workflows/example-build.yml

# CI System: GitHub Actions
# Variable Name: github.event.head_commit-timestamp
- name: Build project
  env:
    # Any dk build system is fine. dk0 is the reference implementation.
    dk_build_system: dk0
  run: ${dk_build_system} -t "${{ github.event.head_commit-timestamp }}" ...

.gitlab-ci.yml

# CI System: GitLab CI
# Docs: https://docs.gitlab.com/ci/variables/predefined_variables/#predefined-variables
# Variable Name: CI_COMMIT_TIMESTAMP
# Variable Example: 2022-01-31T16:47:55-08:00
variables:
  # Any dk build system is fine. dk0 is the reference implementation.
  dk_build_system: dk0
job:
  script:
    - ${dk_build_system} -t "$CI_COMMIT_TIMESTAMP" ...

Here are some example of using a monotonically increasing build number:

# GitHub Actions: GITHUB_RUN_NUMBER https://docs.github.com/en/actions/reference/workflows-and-actions/variables
# GitLab CI: CI_PIPELINE_IID https://docs.gitlab.com/ci/variables/predefined_variables/
# Azure Pipelines: Build.BuildId https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml

# FILLMEIN ... wait for `-n RUN_NUMBER` option to complement `-t TIMESTAMP`
# FILLMEIN ... `-n` includes leading zeroes so lexographic comparisons work

JSON Files

The schema is at etc/jsonschema/mlfront-value.json.

On Windows the JSON files are expected to be terminated with LF not CRLF line endings.

The build system is resilient to CRLF line endings:

• The values canonical id normalizes the JSON with all carriage returns removed before calculating the canonical id
• The values checksums normalizes the JSON with all carriage returns removed before conversion to an AST

JSON Canonicalization

The form is reconstructed as JSON exactly with the following keys (and only the following keys) in the exact order:

1. assets.files.checksum.sha1
2. assets.files.checksum.sha256
3. assets.files.path
4. assets.files.size
5. assets.id
6. forms.function.args
7. forms.function.envmods
8. forms.id
9. forms.outputs.files.paths
10. forms.outputs.files.slots
11. forms.precommands.private
12. forms.precommands.public
13. schema_version.major
14. schema_version.minor

with:

• all whitespace between JSON tokens removed from the canonicalized JSON
• all non-existent array values replaced with empty arrays, and all non-existent boolean balues replaced with false
• all assets.files sorted by the ascending lexographical UTF-8 byte encoding of assets.files.path
• assets.files.checksum.sha1 removed if assets.files.checksum.sha256 is present

The above canonicalization should conform to RFC 8785; if there are any ambiguities RFC 8785 must be followed.

Of particular note is that the assets.listing.origins and assets.files.origin fields are not present in the canonicalization since the bundle path, checksum and size uniquely identifiy an asset. In other words, if you have a locally cached file with the same checksum as a remote bundle, you can substitute the locally cached file without changing identifiers in the value store.

For example, the form:

{
  "schema_version":{"major":1,"minor":0},
  "forms": [
    {
      "id": "FooBar_Baz@0.1.0",
      "precommands": {
        "private": [
          "private1"
        ],
        "public": [
          "public1"
        ]
      },
      "function": {
        "commands": [
          "arg1"
        ],
        "envmods": [
          "-PATH"
        ]
      },
      "outputs": {
        "assets": [
          {
            "paths": [
              "outpath1"
            ],
            "slots": [
              "output1"
            ]
          }
        ]
      }
    }
  ],
  "bundles": [
    {
      "id": "DkDistribution_Std.Bundle@2.4.202508011516-signed",
      "listing": {
        "origins": [
          {
            "name": "github-release",
            "mirrors": [
              "https://github.com/diskuv/dk/releases/download/2.4.202508011516-signed"
            ]
          }
        ]
      },
      "assets": [
        {
          "origin": "github-release",
          "path": "SHA256.sig",
          "size": 151,
          "checksum": {
            "sha256": "0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"
          }
        },
        {
          "origin": "github-release",
          "path": "SHA256",
          "size": 559,
          "checksum": {
            "sha256": "4bd73809eda4fb2bf7459d2e58d202282627bac816f59a848fc24b5ad6a7159e"
          }
        }
      ]
    }
  ]
}

is canonicalized to:

{"bundles":[{"assets":[{"checksum":{"sha256":"4bd73809eda4fb2bf7459d2e58d202282627bac816f59a848fc24b5ad6a7159e"},"path":"SHA256"},{"checksum":{"sha256":"0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"},"path":"SHA256.sig"}],"id":"DkDistribution_Std.Bundle@2.4.202508011516-signed"}],"forms":[{"function":{"commands":["arg1"],"envmods":["-PATH"]},"id":"FooBar_Baz@0.1.0","outputs":{"assets":[{"paths":["outpath1"],"slots":["output1"]}]},"precommands":{"private":["private1"],"public":["public1"]}}],"schema_version":{"major":1,"minor":0}}