DkSDK: the dev kit to tame the complexity of multi-platform/tier/language software

Hello Builder! Scripting is a small, free and important piece of DkSDK. Walk with me as we use the DkSDK tool "DkCoder" to write little scripts that become full-fledged programs. My hope is that within minutes you feel like your dev experience is as productive as in Python, but enhanced so you:

  1. have nothing to install except Git and optionally Visual Studio Code
  2. have safe and understandable code ("static typing")

You'll start with the mouse-click install and the basic one-liner:

Copy
let () = Tr1Stdlib_V414Io.StdIo.print_endline "Hello Builder!"

which you run with:

Copy
./dk DkRun_V0_2.Run DkHelloScript_Std.AndHello
Copy
Hello Builder!

Then you will write snippets of shell scripts and do a quick tour of the prior art.

You'll also run a small GUI example.

Along the way you'll encounter a small language made by a community who can write some very good libraries. And a software kit that gets out the way and makes good software accessible. Let's begin.

You, your co-developers and your users can start scripting in a couple clicks.

FIRST, if you don't have these installed then install Git from https://git-scm.com/downloads and Visual Studio Code from https://code.visualstudio.com/download.

Then click this link: Clone and Open DkHelloScript in Visual Studio Code.

Click to Trust the Authors and Install the Recommended Extensions!
Want to Use The Command Line Instead?
Copy
git clone https://gitlab.com/diskuv/samples/dkcoder/DkHelloScript.git
code DkHelloScript

SECOND, open src/DkHelloScript_Std/AndHello.ml in your IDE or open src/DkHelloScript_Std/AndHello.ml in your browser.

You should see:

Copy
open Tr1Stdlib_V414Io

let () = StdIo.print_endline "Hello Builder!"
Wait a minute or two for the one-time background installation, and then you'll notice the text colors have changed. Hover over the print_endline to see its API.

THIRD, open a Terminal > New Terminal and run:

Copy
./dk DkRun_V0_2.Run DkHelloScript_Std.AndHello
Copy
Hello Builder!

Checkmark DONE! Ok, one-liners are not very interesting.

But ... that should have only taken a few minutes.

And ... the same command works on Windows, macOS and GNU/Linux.

You will be prompted to accept the DkCoder licenses. The game demo has a separate license: GPL 3.0. If you want to adapt the game demo for commercial purposes you must contact the demo author at https://github.com/sanette.
We'll go over licensing in general later in this walkthrough.

Reproducibility or quick typing? Pick one

We specified a version number DkRun_V0_2.Run in our last example:

Copy
./dk DkRun_V0_2.Run DkHelloScript_Std.AndHello

You can relax your fingers by instead typing:

Copy
./dk DkHelloScript_Std.AndHello

In your everyday scripting you won't want to type DkRun_V0_2.Run. Leave it out.

However, when you are publishing documentation (like this article) you should always include the version number. You'll find it easy for your users to copy-and-paste and more importantly, the behavior of your scripts won't change in some future version of ./dk.

Focus on what you are running

Open src/DkHelloScript_Std/AndHelloAgain.ml in your IDE.

It has the same content as DkHelloScript_Std/AndHello.ml. However, it has red squiggly lines.

The red is an indication that DkCoder has not compiled the script. That is a good thing because generally you don't want to waste time compiling every script every time you make an edit in a large project.

Now run the DkHelloScript_Std.AndHelloAgain script with:

Copy
./dk DkRun_V0_2.Run DkHelloScript_Std.AndHelloAgain

Visual Studio Code should remove the red in a minute.

Faster?
You can speed up how fast Visual Studio Code recognizes the newly focused script by using the View > Command Palette menu, and then using the OCaml: Restart Language Server action.

DkCoder optimizes for rapid iterative development by:

  1. Only compiling the script that you last ran (ex. DkHelloScript_Std/AndHelloAgain.ml). If your script requires other scripts to run, those are also compiled.
  2. Compiling all the scripts in a project when you first open Visual Studio Code. That means you can browse your project when you first start your day without seeing any red. Then, when you have found a script you want to edit or a new script to add, edit and run that script repeatedly throughout the day.
    Upcoming Changes
    Since large projects may have many scripts, a future version of DkCoder will allow you to select which modules are automatically compiled at startup.

Runtime requirements

  • Windows 10 (version 1903 aka 19H1, or later), or Windows 11, either 32-bit or 64-bit.

  • macOS AMD/Intel or Apple Silicon (Big Sur 11.0 or higher).

  • Linux AMD/Intel 64-bit with glibc 2.28 or higher.

    • You will need the following system tools installed:

      • tar, wget or curl, git, unzip
      • ninja (this requirement will be removed)
      • If any are missing you will be asked if DkCoder can install them the first time you run ./dk from the command line.
    • For graphics you will need one of:

      • X11 with a direct OpenGL driver.
        • A good prereq test is running glxgears (ex. yum install glx-utils; glxgears). If that works, DkCoder graphics should work.
        • Don't expect the LIBGL_ALWAYS_INDIRECT=1 environment variable to work. That is sometimes recommended when forwarding X11 with ssh -X or into a Docker container, but it forces the protocol to be a too-old OpenGL version 1.4.
      • Wayland. Set the environment variable SDL_VIDEODRIVER=wayland to force it.
      • Linux Direct Rendering Manager. If supported, you will have a /dev/dri/card0. Set the environment variable SDL_VIDEODRIVER=kmsdrm to force it.
    • That's it. Linux desktops are complex!

Basic shell scripting

In an earlier section an OCaml programming language environment was transparently downloaded for you. You will be using OCaml in this walkthrough, even though you may find other DkSDK documentation that uses DkCoder with C and Java. You can proceed through this walkthrough without knowing OCaml.

Once you are done the DkCoder walkthrough you may want to go to the OCaml - Learn site. For this walkthrough we'll stick to ordinary OCaml open-source examples that you can replicate in a conventional OCaml environment.
In this section we'll be using the shexp library that was created by Jane Street Capital. If you are familiar with traditional shell scripts like bash you'll find shexp more powerful.

Open src/DkHelloScript_Std/B43Shell/B35Shexp/B43Countdown.ml in your IDE or open src/DkHelloScript_Std/B43Shell/B35Shexp/B43Countdown.ml in your browser.

You should see:

Copy
open Tr1Shexp_Std.Shexp_process
open Tr1Shexp_Std.BindsShexp

(* Counts down from n with a one second delay between ticks *)
let rec countdown (n : int) : unit t =
  if n > 0 then begin
    echo (string_of_int n) ;%bind
    sleep 1.0 ;%bind
    countdown (n - 1)
  end
  else echo "Done countdown. Bye Builder!"

let main_t : unit t =
  echo "Hello Builder! ..." ;%bind
  echo "------------------------------------------------------------" ;%bind
  echo "Starting countdown..." ;%bind
  countdown 5

let () = if Tr1EntryName.module_id = __MODULE_ID__ then eval main_t

And running it with:

Copy
./dk DkRun_V0_2.Run DkHelloScript_Std.B43Shell.B35Shexp.B43Countdown

gives:

Copy
Hello Builder! ...
------------------------------------------------------------
Starting countdown...
5
4
3
2
1
Done countdown. Bye Builder!

Prior Art

This is only a quick tour of the most popular alternatives to DkCoder. Most of the advantages for DkCoder boil down to it works on Windows and Unix without pre-installing other packages.

  • Shell scripts like /bin/bash. A POSIX-compatible shell script is pre-installed on almost all Unix distributions. By comparison, DkCoder has a POSIX shell script launcher that transparently installs pre-compiled binaries for macOS and the major Linux desktop distributions. However, DkCoder has expanded reach with its Windows shell script launcher that transparently installs on Windows (and Unix) machines.
  • PowerShell. The scripts are written in a full-featured programming language, and in that respect PowerShell is similar to DkCoder. Furthermore, PowerShell has an command shell that is a complete alternative to both the Command Prompt on Windows and /bin/sh on Unix. DkCoder does not have a command shell. However, DkCoder scripts work on both Windows and Unix without pre-installing the PowerShell distribution, so DkCoder may be easier to adopt.
  • Perl. CPAN gives Perl a huge standardized package registry with standardized package tools. By comparison, DkCoder has no packages (today!) in its package registry. However, DkCoder will soon have an alternative to package registries that is expected to be the main form of sharing until DkCoder is out of its infancy stage. Also, as with shell scripts, Perl is pre-installed on almost all Unix distributions. However, DkCoder has expanded reach with its Windows and Unix shell script launcher.
  • Python. Most of the advantages and disadvantages for Perl apply to Python. Python is less ubiquitous but more popular than Perl. DkCoder scripting should have a similar learning curve to Python, as evidenced by its early use by high-schoolers.
  • Fortran. The DkCoder author (me!) has never written a line of Fortran, but its "dynamic dependency" system of Fortran files containing declarations of which Fortran modules are used is similar to DkCoder. DkCoder goes a step further and drops the need to declare dependencies that are available in the DkRegistry.
  • etc.

Adding some flair with graphics

Open src/DkHelloScript_Std/B57Graphics/B43Bogue/B43Tiny.ml in your IDE or open src/DkHelloScript_Std/B57Graphics/B43Bogue/B43Tiny.ml in your browser.

You should see:

Copy
open Tr1Bogue_Std.Bogue

let () =
  if Tr1EntryName.module_id = __MODULE_ID__ then
    Widget.label "Hello world"
    |> Layout.resident
    |> Bogue.of_layout
    |> Bogue.run

Run it with:

Copy
./dk DkRun_V0_2.Run DkHelloScript_Std.B57Graphics.B43Bogue.B43Tiny

Making Your Own Script Project

FIRST, create an empty folder for your project and open it in Visual Studio Code. If your project should use Git source control, in the Terminal run:

Copy
git init

Files Created
Copy
.
└─── .git/

SECOND, in the Terminal paste the following to get four files into your project:

Copy
git clone https://gitlab.com/diskuv/dktool.git
dktool/dk user.dkml.wrapper.upgrade HERE
./dk dkml.wrapper.upgrade DONE

Files Created
Copy
.
├── .gitattributes
├── __dk.cmake
├── dk
└── dk.cmd

THIRD, create the file .vscode/extensions.json:

Copy
{
  "recommendations": ["ocamllabs.ocaml-platform"]
}

Files Created
Copy
.
└── .vscode/
    └── extensions.json

FOURTH, create the file .vscode/settings.json:

Copy
{
  "ocaml.sandbox": {
    "kind": "custom",
    "template": "${firstWorkspaceFolder}/dk DkRun_V0_2.RunQuiet --log-level ERROR -- DkDev_Std.Exec -- $prog $args"
  }
}
Files Created
Copy
.
└── .vscode/
    └── settings.json

Checkmark DONE! You can add new scripts now. Yep, expect that we will write a script to create a new project!

Parties Who Own Scripts

There are three parties of people who own DkCoder scripts:

  1. You (the first party)
  2. Us (the second party)
  3. Them (the third parties)
Copy
    <p class="block">Here is a handy reference table that summarizes the distinctions. You won't understand the table right now and that is OK. We'll cover "Us" and "Them" in later sections in a future edition of DkCoder, but it is nice to have all the information in one place. All you need to remember is if you have a question about parties, this is the table to check.</p>
Party Reference Table
Party Code Generators Terms Compiled Initially
Party Code Generators Terms Compiled Initially
You DuneIde (default), Dune Checked Yes
Us Dune No
Them DuneIde (default), Dune Checked No

In this section we'll talk about where your You scripts go.

Your scripts must be placed in specific paths. Here is an example which you've seen in this walkthrough, and the general pattern you must follow:

Copy
.
└── src/
    ├── DkHelloScript_Std/
    │   └── Example001.ml
    └── <Libraryowner><Project>_<Unit>/
        │── <Modulename1>.ml
        └── <Modulename2>/
            │── <Modulename3>.ml
            │── <Modulename3>.mli
            └── <Modulename4>/
                └── <Modulename5>.ml

In the example above the DkHelloScript_Std is a library. A library is an organization of files and directories. These directories can have subdirectories, and those subdirectories can have their own subdirectories (etc.).

Any directory under src/ that is named in the format:

<Libraryowner><Project>_<Unit>

is also a library.

Breakdown of the DkHelloScript_Std library name
Component Part Why
Component Part Why
Dk Library owner A short identifier for which party owns the source code
HelloScript Project Conventionally this is a name related to your source code repository, but it can be any organizational name you want. You might, for example, want to name it your product name.
Std Unit Conventionally if you want to create a single library in a project, you use the name Std. Once you get too much code in your project, you can use the Unit to separate your code into more manageable, smaller pieces.

We'll define the exact format of &lt;Libraryowner&gt;, &lt;Project&gt; and &lt;Unit&gt; below.

You can place your script into the library directory, or into a subdirectory (or subdirectory of a subdirectory, etc.) named according to the &lt;Modulename&gt;/ format defined below, as long as your script is named according to the &lt;Modulename&gt;.ml format. There is also an optional &lt;Modulename&gt;.mli script interface which is not covered in this article.

Rules for the parts of a library name
Part Examples Rules
Part Examples Rules
Library owner Dk, Acme, Blue123 The Library owner must start with an ASCII capital letter and have a lowercase ASCII second letter; the remaining characters (if any) must be lowercase ASCII letters and/or ASCII digits.
Project X, Ab, Widgets, WidgetsPlus The Project must start with an ASCII capital letter. The remaining characters (if any) must be ASCII letters (any case) and/or ASCII digits. Conventionally you use something similar to your source code repository name as the Project.
Unit Std, V1, X The library unit must start with an ASCII capital letter. The remaining characters (if any) must be ASCII letters (any case) and/or ASCII digits and/or underscores (`_`). Conventially the main unit for your project is named `Std`. Once you get too much code in your project, you can use the Unit to separate your code into more manageable, smaller pieces.
Carefully choose the library owner
The Library owner (ex. Acme above) should uniquely identify yourself or your organization. Pick one and use it consistently. If you decide to publish libraries to the DkRegistry (described later) you will avoid conflicts with other libraries.

You can use the regular expression ([A-Z][a-z][a-z0-9]*)([A-Z][A-Za-z0-9]*)_([A-Z][A-Za-z0-9_]*) on your favorite regex site (ex. https://regex101.com/) to check if you have a valid &lt;Libraryowner&gt;&lt;Project&gt;_&lt;Unit&gt; name.

Rules for a module name
Examples Counter Examples Rule
Acme, Acme_Two, X DkHelloScript_Std

The module name MUST NOT be a valid &lt;Libraryowner&gt;&lt;Project&gt;_&lt;Unit&gt; name.

Acme, Acme*Two, X 12345, someThing The module name must start with an ASCII capital letter. The remaining characters (if any) must be ASCII letters (any case) and/or ASCII digits and/or underscores (*).

Standard OCaml Library - Stdlib

The Stdlib standard library provides the basic operations over the built-in types (numbers, booleans, byte sequences, strings, exceptions, references, lists, arrays, input-output channels, ...) and standard library modules.

In conventional OCaml programs the Stdlib is automatically opened. That means you can type print_endline "Hi" rather than Stdlib.print_endline "Hi". However, in DkCoder access to the Stdlib is restricted: Stdlib does a bit too much. In particular, input-output channels and threading do not make sense when compiling to JavaScript for use on a web page. Even more critical is that the Stdlib.Obj module is included, which has unsafe functions that make it impossible to prove overall code safety.

You can explicitly get back the unsafe standard library by performing an open at the top of your scripts:

Copy
open Tr1Stdlib_V414

However, it is recommended to open the pieces of the standard library you actually need. That way if, for example, your script does not use files it can be compiled to JavaScript.

Splitting the OCaml Standard Library
Package Description Modules
Package Description Modules
Tr1Stdlib_V414CRuntime Modules that need a C99 runtime Arg, Callback, Filename, Format, In_channel, LargeFile, Lexing, Out_channel, Printexc, Printf, Scanf, StdExit1, Sys, Unix, UnixLabels
Tr1Stdlib_V414Io Modules for input and output. Can't be used on Android and iOS. StdIo2
Tr1Stdlib_V414Threads Modules to create and coordinate threads. Brings in Tr1Stdlib_V414CRuntime. Condition, Event, Thread
Tr1Stdlib_V414Gc Modules for garbage collection Ephemeron, Gc, Weak
Tr1Stdlib_V414Random Modules for random numbers Random
Tr1Stdlib_V414Unsafe Modules that can break type-safety Dynlink, Marshal, Obj
Tr1Stdlib_V414Base Modules safe for use on any platform. Also includes safe functions like sqrt. Array, ArrayLabels, Atomic, Bool, Buffer, Bytes, BytesLabels, Char, Complex, Digest, Either, Float, Fun, Hashtbl, Int, Int32, Int64, Lazy, List, ListLabels, Map, MoreLabels, Nativeint, Oo, Option, Parsing, Queue, Result, Seq, Set, Stack, StdBinds3, StdLabels, String, StringLabels, Uchar, Unit
Deprecated Stdlib modules
The ThreadUnix, Stream, Pervasives, and Genlex modules have been deprecated in OCaml and do not appear in any of the Tr1Stdlib_V414* packages
Footnote 1: StdExit
StdExit is not a module provided by Stdlib. It is a new module that contains all the program termination types and functions of Stdlib:
Copy
val exit : int -> 'a
(** Terminate the process, returning the given status code
   to the operating system: usually 0 to indicate no errors,
   and a small positive integer to indicate failure.
   ... *)

val at_exit : (unit -> unit) -> unit
(** Register the given function to be called at program termination
   time. ... *)

Full documentation is at Stdlib - Program termination

Footnote 2: StdIo
StdIo is not a module provided by Stdlib. It is a new module that contains all the input/output types and functions of Stdlib:
Copy
type in_channel
(** The type of input channel. *)

type out_channel
(** The type of output channel. *)

val stdin : in_channel
(** The standard input for the process. *)

val stdout : out_channel
(** The standard output for the process. *)

val stderr : out_channel
(** The standard error output for the process. *)

val print_char : char -> unit
(** Print a character on standard output. *)

val print_string : string -> unit
(** Print a string on standard output. *)

val print_bytes : bytes -> unit
(** Print a byte sequence on standard output. *)

val print_int : int -> unit
(** Print an integer, in decimal, on standard output. *)

val print_float : float -> unit
(** Print a floating-point number, in decimal, on standard output. *)

val print_endline : string -> unit
(** Print a string, followed by a newline character, on
   standard output and flush standard output. *)

val print_newline : unit -> unit
(** Print a newline character on standard output, and flush
   standard output. *)

val prerr_char : char -> unit
(** Print a character on standard error. *)

val prerr_string : string -> unit
(** Print a string on standard error. *)

val prerr_bytes : bytes -> unit
(** Print a byte sequence on standard error. *)

val prerr_int : int -> unit
(** Print an integer, in decimal, on standard error. *)

val prerr_float : float -> unit
(** Print a floating-point number, in decimal, on standard error. *)

val prerr_endline : string -> unit
(** Print a string, followed by a newline character on standard
   error and flush standard error. *)

val prerr_newline : unit -> unit
(** Print a newline character on standard error, and flush
   standard error. *)

val read_line : unit -> string
(** Flush standard output, then read characters from standard input
   until a newline character is encountered. ... *)

val read_int_opt: unit -> int option
(** Flush standard output, then read one line from standard input
   and convert it to an integer. ... *)

val read_int : unit -> int
(** Same as {!read_int_opt}, but raise [Failure "int_of_string"]
   instead of returning [None]. *)

val read_float_opt: unit -> float option
(** Flush standard output, then read one line from standard input
   and convert it to a floating-point number. ... *)

val read_float : unit -> float
(** Same as {!read_float_opt}, but raise [Failure "float_of_string"]
   instead of returning [None]. *)

type open_flag =
    Open_rdonly      (** open for reading. *)
  | Open_wronly      (** open for writing. *)
  | Open_append      (** open for appending: always write at end of file. *)
  | Open_creat       (** create the file if it does not exist. *)
  | Open_trunc       (** empty the file if it already exists. *)
  | Open_excl        (** fail if Open_creat and the file already exists. *)
  | Open_binary      (** open in binary mode (no conversion). *)
  | Open_text        (** open in text mode (may perform conversions). *)
  | Open_nonblock    (** open in non-blocking mode. *)
(** Opening modes for {!open_out_gen} and {!open_in_gen}. *)

val open_out : string -> out_channel
(** Open the named file for writing, and return a new output channel
   on that file, positioned at the beginning of the file. ... *)

val open_out_bin : string -> out_channel
(** Same as {!open_out}, but the file is opened in binary mode,
   so that no translation takes place during writes. ... *)

val open_out_gen : open_flag list -> int -> string -> out_channel
(** [open_out_gen mode perm filename] opens the named file for writing,
   as described above. ... *)

val flush : out_channel -> unit
(** Flush the buffer associated with the given output channel,
   performing all pending writes on that channel. ... *)

val flush_all : unit -> unit
(** Flush all open output channels; ignore errors. *)

val output_char : out_channel -> char -> unit
(** Write the character on the given output channel. *)

val output_string : out_channel -> string -> unit
(** Write the string on the given output channel. *)

val output_bytes : out_channel -> bytes -> unit
(** Write the byte sequence on the given output channel. *)

val output : out_channel -> bytes -> int -> int -> unit
(** [output oc buf pos len] writes [len] characters from byte sequence [buf],
   starting at offset [pos], to the given output channel [oc]. ... *)

val output_substring : out_channel -> string -> int -> int -> unit
(** Same as [output] but take a string as argument instead of
   a byte sequence. *)

val output_byte : out_channel -> int -> unit
(** Write one 8-bit integer (as the single character with that code)
   on the given output channel. ... *)

val output_binary_int : out_channel -> int -> unit
(** Write one integer in binary format (4 bytes, big-endian)
   on the given output channel. ... *)

val output_value : out_channel -> 'a -> unit
(** Write the representation of a structured value of any type
   to a channel. ... *)

val seek_out : out_channel -> int -> unit
(** [seek_out chan pos] sets the current writing position to [pos]
   for channel [chan]. ... *)

val pos_out : out_channel -> int
(** Return the current writing position for the given channel. ... *)

val out_channel_length : out_channel -> int
(** Return the size (number of characters) of the regular file
   on which the given channel is opened. ... *)

val close_out : out_channel -> unit
(** Close the given channel, flushing all buffered write operations. ... *)

val close_out_noerr : out_channel -> unit
(** Same as [close_out], but ignore all errors. *)

val set_binary_mode_out : out_channel -> bool -> unit
(** [set_binary_mode_out oc true] sets the channel [oc] to binary
   mode: no translations take place during output. ... *)

val open_in : string -> in_channel
(** Open the named file for reading, and return a new input channel
   on that file, positioned at the beginning of the file. *)

val open_in_bin : string -> in_channel
(** Same as {!open_in}, but the file is opened in binary mode,
   so that no translation takes place during reads. ... *)

val open_in_gen : open_flag list -> int -> string -> in_channel
(** [open_in_gen mode perm filename] opens the named file for reading,
   as described above. ... *)

val input_char : in_channel -> char
(** Read one character from the given input channel. ... *)

val input_line : in_channel -> string
(** Read characters from the given input channel, until a
   newline character is encountered. ... *)

val input : in_channel -> bytes -> int -> int -> int
(** [input ic buf pos len] reads up to [len] characters from
   the given channel [ic], storing them in byte sequence [buf], starting at
   character number [pos]. ... *)

val really_input : in_channel -> bytes -> int -> int -> unit
(** [really_input ic buf pos len] reads [len] characters from channel [ic],
   storing them in byte sequence [buf], starting at character number [pos].
   ... *)

val really_input_string : in_channel -> int -> string
(** [really_input_string ic len] reads [len] characters from channel [ic]
   and returns them in a new string. ...*)

val input_byte : in_channel -> int
(** Same as {!input_char}, but return the 8-bit integer representing
   the character. ... *)

val input_binary_int : in_channel -> int
(** Read an integer encoded in binary format (4 bytes, big-endian)
   from the given input channel. ... *)

val input_value : in_channel -> 'a
(** Read the representation of a structured value, as produced
   by {!output_value}, and return the corresponding value. ... *)

val seek_in : in_channel -> int -> unit
(** [seek_in chan pos] sets the current reading position to [pos]
   for channel [chan]. ... *)

val pos_in : in_channel -> int
(** Return the current reading position for the given channel. ... *)

val in_channel_length : in_channel -> int
(** Return the size (number of characters) of the regular file
    on which the given channel is opened. ... *)

val close_in : in_channel -> unit
(** Close the given channel. ... *)

val close_in_noerr : in_channel -> unit
(** Same as [close_in], but ignore all errors. *)

val set_binary_mode_in : in_channel -> bool -> unit
(** [set_binary_mode_in ic true] sets the channel [ic] to binary
   mode: no translations take place during input. ... *)

val __LOC__ : string
(** [__LOC__] returns the location at which this expression appears in
    the file currently being parsed by the compiler, with the standard
    error format of OCaml: "File %S, line %d, characters %d-%d". *)

val __FILE__ : string
(** [__FILE__] returns the name of the file currently being
    parsed by the compiler. *)

val __LOC_OF__ : 'a -> string * 'a
(** [__LOC_OF__ expr] returns a pair [(loc, expr)] where [loc] is the
    location of [expr] in the file currently being parsed by the
    compiler, with the standard error format of OCaml: "File %S, line
    %d, characters %d-%d". *)

Full documentation is at Stdlib - Input/output

Footnote 3: StdBinds
StdBinds is not a module provided by Stdlib. It contains modules that can be opened and used with the ocaml-monadic PPX macros:
Copy
(** Open to use [let%bind] with {!Result}. *)
module BindsResult = struct
  exception ResultFailed of string option

  let bind = Result.bind
  let map = Result.map
  let return = Result.ok
  let zero ?msg () = raise (ResultFailed msg)
end

(** Open to use [let%bind] with {!Option}. *)
module BindsOption = struct
  let bind = Stdlib.Option.bind
  let map = Stdlib.Option.map
  let return = Stdlib.Option.some
  let zero () = None
end

The macro documentation is at ocaml-monadic extensions

DkCoder Runtime Libraries

shexp.process

This is a library for creating possibly parallelized pipelines that represent shell scripts.

Docs: shexp README

Warning!
This library only works on 64-bit machines.

ocaml-monadic

This small macro (PPX) library provides the let%bind, if%bind and match%bind forms. These are similar in design to Jane Street's ppx_let macros but work in 32-bit architectures and will be easier to support backwards-compatibilty on OCaml 4.

This library also offers the ;%bind, let%orzero and [%guard] forms.

Docs: ocaml-monadic README

Limitations imposed by cross-platform support.

DkSDK is a family of tools that supports cross-platform development, even on embedded devices. This cross-platform support places constraints on how you structure your DkCoder projects.

  • The FAT32 filesystem is found on older Windows disks, CD drives and on embedded devices. FAT32 only supports filenames up to 255 characters. DkCoder limits the paths <Libraryowner><Libraryproject>_<Libraryunit>/<Modulename>/.../<Modulename>.mli to 240 characters. Even when the FAT32 filesystem entirely disappears from use this 240 character limit may remain since the limit encourages modular, single-focused libraries.
  • The FAT16 filesystem is the only filesystem in moderate use today that is not supported by DkCoder. FAT16 might still be used on your older USB thumbdrives and some very old embedded devices. Since FAT16 directory names are limited to 8 uppercase characters, and the minimum <Libraryowner><Libraryproject>_<Libraryunit> name is 7 characters, it is not reasonable to support FAT16.

Release Notes - Early Technical Limitations

  • Your script today needs to fully self-contained. You cannot refer to one script's code in another script.
  • The ./dk script delegates to the build tool CMake first, and then to OCaml. That design reflects the history that DkCoder was designed first for C packages. The current design means:
    1. ./dk has the performance overhead of spawning CMake on startup.
    2. ./dk has command line arguments interpreted by CMake before being interpreted by your script. In particular, nested double-quotes like ./dk ... "here is a nested `" double quote" in PowerShell won't work.
  • The GUI code was very recently ported to Windows. It has not had memory auditing so segfaults may occur. And playing sounds is known to hang. Do not use for GUIs and music for production code until we message that this has been fixed.

Security

Advanced Topic
This section is intended for security engineers.

The primary security goal of DkCoder is to allow the script user to assert, before a script is run, whether the script does not access resources like files, the console, sound, etc. Not all scripts can be asserted, but if DkCoder asserts a script does not have access to files for exmaple, then the script does not have access to files under some light assumptions. These assumptions are detailed in this section.

Access to Source Code
Security engineers, auditors and certifiers are granted free audit access to DkCoder and other DkSDK source code. The requirements are that you have worn the security hat for the majority of the past three months, have a publicly discoverable telephone switchboard, and reside in a country having no United States export controls. Contact information to request an audit account is available at the bottom of this article.
Status
# ETA What is Implemented
# ETA What is Implemented
Alpha Now Split the OCaml Standard Library by resource type
Beta TBD Implementation of the Technical Requirements below
After GA TBD Allow users to write AWS Cedar policies to restrict access to a) resources used by running scripts and b) which scripts can be downloaded and c) data submitted to DkCoder services.

Assumption A1 - A superset of modules used by a script can be obtained

This assumption is based on the OCaml 4.14 environment model Env.t defined in typing/env.mli. These Env.t have values, modules, types, module types, classes and class types produced during compilation. Only add operations like add_value are present, so by the end of compilation we have a superset of statically compiled modules used by the compiled script.

However,

  1. Some of the modules may be functors; that is, new modules can be created at runtime of a specified module type. That leads to a technical requirement:

    If a script uses a functor module, transitively or not, then the script is unassertable (aka. "tainted").

How it will be implemented: The codept analysis tool parses the source and gathers module information, including whether a module is a functor. Using the --log-level DEBUG shows the modules used by the scripts.

Assumption A2 - Access to resources can be gated through modules

By definition, resources are either values like input channels or are accessed through values like function calls (including external C function calls).

In OCaml, values are present in a toplevel Env.t or in a (possibly nested) module and/or class.

  1. Since there is no tool like codept to analyze OCaml classes, there is a technical requirement:

    If a script uses a class, transitively or not, then the script is unassertable (aka. "tainted").

  2. Since the standard OCaml toplevel environment is the contents of the Stdlib module (ie. there is an implicit open Stdlib at the top of each module), and Stdlib contains toplevel values like print_endline, there is a technical requirement:

    All Stdlib toplevel values are annotated with OCaml alerts that fail the compilation when the toplevel values are used.

  3. Since alerts can be locally overridden, there is a technical requirement:

    If a script uses a local alert override, transitively or not, then the script is unassertable (aka. "tainted").

To repeat: In OCaml, values are present in a toplevel Env.t or in a (possibly nested) module and/or class. What remains after implementing the technical requirements above is that resource values necessary for safety assertions are only present in (possibly nested) modules.

How it will be implemented: A PPX can be run that injects a new empty module module Dk__Tainted = struct end whenever a class or local alert override is used. The module analysis by codept will discover the Dk__Tainted, and the analysis phase can fail.

Assumption A3 - Violating the correctness of OCaml type checking can be detected at compile time

The type-safe violations fall into these categories:

  1. Using an external C function declaration
  2. Using the Obj module
  3. Using the Marshal module

This is a weak assumption as it requires expert knowledge of OCaml.

Since the presence of modules can be trivially detected with assumption A1, the technical requirement is:

If a script uses an external C function declaration, transitively or not, then the script is unassertable (aka. "tainted").

How it will be implemented: Any external declaration can be caught by a PPX which injects a module Dk__Tainted = struct end. The presence of any of Dk__Tainted, Obj or Marshal during the codept analysis will taint the script.

Getting in touch

I'm a recovering Luddite when it comes to social media; I deactivated Facebook and stopped using LinkedIn years ago. That will change. In the meantime, it is best to post OCaml questions on discuss.ocaml.org. And you can follow or DM me on the low-volume @Diskuv on 𝕏.