Intro to Scripting

DkSDK: the dev kit to tame the complexity of multi-platform/tier/language software

Hello Builder! Scripting is a small, free and important piece of DkSDK. Walk with me as we use the DkSDK tool "DkCoder" to write little scripts that become full-fledged programs. My hope is that within minutes you feel like your dev experience is as productive as in Python, but enhanced so you:

1. have nothing to install except Git and optionally Visual Studio Code
2. have safe and understandable code ("static typing")

You'll start with the mouse-click install and the basic one-liner:

let () = Tr1Stdlib_V414Io.StdIo.print_endline "Hello Builder!"

which you run with:

./dk DkRun_V0_2.Run -- DkHelloScript_Std.AndHello

Hello Builder!

Then you will write snippets of shell scripts and do a quick tour of the prior art.

You'll also run a small GUI example.

Along the way you'll encounter a small language made by a community who can write some very good libraries. And a software kit that gets out the way and makes good software accessible. Let's begin.

FIRST, if you don't have these installed then install Git from https://git-scm.com/downloads and Visual Studio Code from https://code.visualstudio.com/download.

Then click this link: Clone and Open DkHelloScript in Visual Studio Code.

git clone https://gitlab.com/diskuv/samples/dkcoder/DkHelloScript.git
code DkHelloScript

SECOND, open src/DkHelloScript_Std/AndHello.ml in your IDE or open src/DkHelloScript_Std/AndHello.ml in your browser.

You should see:

open Tr1Stdlib_V414Io

let () = StdIo.print_endline "Hello Builder!"

THIRD, open a Terminal > New Terminal and run:

./dk DkRun_V0_2.Run -- DkHelloScript_Std.AndHello

Hello Builder!

DONE! Ok, one-liners are not very interesting.

But ... that should have only taken a few minutes.

And ... the same command works on Windows, macOS and GNU/Linux.

We specified a version number and a double dash separator (DkRun_V0_2.Run --) in our last example:

./dk DkRun_V0_2.Run -- DkHelloScript_Std.AndHello

You can relax your fingers by instead typing:

./dk DkHelloScript_Std.AndHello

In your everyday scripting you won't want to type DkRun_V0_2.Run --. Leave it out.

However, when you are publishing documentation (like this article) you should always include the version number. You'll find it easy for your users to copy-and-paste and more importantly, the behavior of your scripts won't change in some future version of ./dk.

Open src/DkHelloScript_Std/AndHelloAgain.ml in your IDE.

It has the same content as DkHelloScript_Std/AndHello.ml. However, it has red squiggly lines.

The red is an indication that DkCoder has not compiled the script. That is a good thing because generally you don't want to waste time compiling every script every time you make an edit in a large project.

Now run the DkHelloScript_Std.AndHelloAgain script with:

./dk DkRun_V0_2.Run -- DkHelloScript_Std.AndHelloAgain

Visual Studio Code should remove the red in a minute.

DkCoder optimizes for rapid iterative development by:

1. Only compiling the script that you last ran (ex. DkHelloScript_Std/AndHelloAgain.ml). If your script requires other scripts to run, those are also compiled.
2. Compiling all the scripts in a project when you first open Visual Studio Code. That means you can browse your project when you first start your day without seeing any red. Then, when you have found a script you want to edit or a new script to add, edit and run that script repeatedly throughout the day.
   Upcoming Changes

   Since large projects may have many scripts, a future version of DkCoder will allow you to select which modules are automatically compiled at startup.

• Windows 10 (version 1903 aka 19H1, or later), or Windows 11, either 32-bit or 64-bit.

• macOS AMD/Intel or Apple Silicon (Big Sur 11.0 or higher).

• Linux AMD/Intel 64-bit with glibc 2.28 or higher.

  • You will need the following system tools installed:

    • tar, wget or curl, git, unzip
    • ninja (this requirement will be removed)
    • If any are missing you will be asked if DkCoder can install them the first time you run ./dk from the command line.

  • For graphics you will need one of:

    • X11 with a direct OpenGL driver.
      • A good prereq test is running glxgears (ex. yum install glx-utils; glxgears). If that works, DkCoder graphics should work.
      • Don't expect the LIBGL_ALWAYS_INDIRECT=1 environment variable to work. That is sometimes recommended when forwarding X11 with ssh -X or into a Docker container, but it forces the protocol to be a too-old OpenGL version 1.4.
    • Wayland. Set the environment variable SDL_VIDEODRIVER=wayland to force it.
    • Linux Direct Rendering Manager. If supported, you will have a /dev/dri/card0. Set the environment variable SDL_VIDEODRIVER=kmsdrm to force it.

  • That's it. Linux desktops are complex!

In an earlier section an OCaml programming language environment was transparently downloaded for you. You will be using OCaml in this walkthrough, even though you may find other DkSDK documentation that uses DkCoder with C and Java. You can proceed through this walkthrough without knowing OCaml.

Open src/DkHelloScript_Std/B43Shell/B35Shexp/B43Countdown.ml in your IDE or open src/DkHelloScript_Std/B43Shell/B35Shexp/B43Countdown.ml in your browser.

You should see:

open Tr1Shexp_Std.Shexp_process
open Tr1Shexp_Std.BindsShexp

(* Counts down from n with a one second delay between ticks *)
let rec countdown (n : int) : unit t =
  if n > 0 then begin
    echo (string_of_int n) ;%bind
    sleep 1.0 ;%bind
    countdown (n - 1)
  end
  else echo "Done countdown. Bye Builder!"

let main_t : unit t =
  echo "Hello Builder! ..." ;%bind
  echo "------------------------------------------------------------" ;%bind
  echo "Starting countdown..." ;%bind
  countdown 5

let () = if Tr1EntryName.module_id = __MODULE_ID__ then eval main_t

And running it with:

./dk DkRun_V0_2.Run -- DkHelloScript_Std.B43Shell.B35Shexp.B43Countdown

gives:

Hello Builder! ...
------------------------------------------------------------
Starting countdown...
5
4
3
2
1
Done countdown. Bye Builder!

This is only a quick tour of the most popular alternatives to DkCoder. Most of the advantages for DkCoder boil down to it works on Windows and Unix without pre-installing other packages.

• Shell scripts like /bin/bash. A POSIX-compatible shell script is pre-installed on almost all Unix distributions. By comparison, DkCoder has a POSIX shell script launcher that transparently installs pre-compiled binaries for macOS and the major Linux desktop distributions. However, DkCoder has expanded reach with its Windows shell script launcher that transparently installs on Windows (and Unix) machines.
• PowerShell. The scripts are written in a full-featured programming language, and in that respect PowerShell is similar to DkCoder. Furthermore, PowerShell has an command shell that is a complete alternative to both the Command Prompt on Windows and /bin/sh on Unix. DkCoder does not have a command shell. However, DkCoder scripts work on both Windows and Unix without pre-installing the PowerShell distribution, so DkCoder may be easier to adopt.
• Perl. CPAN gives Perl a huge standardized package registry with standardized package tools. By comparison, DkCoder has no packages (today!) in its package registry. However, DkCoder will soon have an alternative to package registries that is expected to be the main form of sharing until DkCoder is out of its infancy stage. Also, as with shell scripts, Perl is pre-installed on almost all Unix distributions. However, DkCoder has expanded reach with its Windows and Unix shell script launcher.
• Python. Most of the advantages and disadvantages for Perl apply to Python. Python is less ubiquitous but more popular than Perl. DkCoder scripting should have a similar learning curve to Python, as evidenced by its early use by high-schoolers.
• Fortran. The DkCoder author (me!) has never written a line of Fortran, but its "dynamic dependency" system of Fortran files containing declarations of which Fortran modules are used is similar to DkCoder. DkCoder goes a step further and drops the need to declare dependencies that are available in the DkRegistry.
• etc.

Open src/DkHelloScript_Std/B57Graphics/B43Bogue/B43Tiny.ml in your IDE or open src/DkHelloScript_Std/B57Graphics/B43Bogue/B43Tiny.ml in your browser.

You should see:

open Tr1Bogue_Std.Bogue

let () =
  if Tr1EntryName.module_id = __MODULE_ID__ then
    Widget.label "Hello world"
    |> Layout.resident
    |> Bogue.of_layout
    |> Bogue.run

Run it with:

./dk DkRun_V0_2.Run -- DkHelloScript_Std.B57Graphics.B43Bogue.B43Tiny

FIRST, create an empty folder for your project and open it in Visual Studio Code. If your project should use Git source control, in the Terminal run:

git init

.
└─── .git/

SECOND, in the Terminal paste the following to get four files into your project:

git clone https://gitlab.com/diskuv/dktool.git
dktool/dk user.dkml.wrapper.upgrade HERE
./dk dkml.wrapper.upgrade DONE

.
├── .gitattributes
├── __dk.cmake
├── dk
└── dk.cmd

THIRD, create the file .vscode/extensions.json:

{
  "recommendations": ["ocamllabs.ocaml-platform"]
}

.
└── .vscode/
    └── extensions.json

FOURTH, create the file .vscode/settings.json:

{
  "ocaml.sandbox": {
    "kind": "custom",
    "template": "${firstWorkspaceFolder}/dk DkRun_V0_2.RunQuiet --log-level ERROR -- DkDev_Std.Exec -- $prog $args"
  }
}

.
└── .vscode/
    └── settings.json

DONE! You can add new scripts now. Yep, expect that we will write a script to create a new project!

There are three parties of people who own DkCoder scripts:

1. You (the first party)
2. Us (the second party)
3. Them (the third parties)

    <p class="block">Here is a handy reference table that summarizes the distinctions. You won't understand the table right now and that is OK. We'll cover "Us" and "Them" in later sections in a future edition of DkCoder, but it is nice to have all the information in one place. All you need to remember is if you have a question about parties, this is the table to check.</p>

In this section we'll talk about where your You scripts go.

Your scripts must be placed in specific paths. Here is an example which you've seen in this walkthrough, and the general pattern you must follow:

.
└── src/
    ├── DkHelloScript_Std/
    │   └── Example001.ml
    └── <Libraryowner><Project>_<Unit>/
        │── <Modulename1>.ml
        └── <Modulename2>/
            │── <Modulename3>.ml
            │── <Modulename3>.mli
            └── <Modulename4>/
                └── <Modulename5>.ml

In the example above the DkHelloScript_Std is a library. A library is an organization of files and directories. These directories can have subdirectories, and those subdirectories can have their own subdirectories (etc.).

Any directory under src/ that is named in the format:

<Libraryowner><Project>_<Unit>

is also a library.

We'll define the exact format of <Libraryowner>, <Project> and <Unit> below.

You can place your script into the library directory, or into a subdirectory (or subdirectory of a subdirectory, etc.) named according to the <Modulename>/ format defined below, as long as your script is named according to the <Modulename>.ml format. There is also an optional <Modulename>.mli script interface which is not covered in this article.

You can use the regular expression ([A-Z][a-z][a-z0-9]*)([A-Z][A-Za-z0-9]*)_([A-Z][A-Za-z0-9_]*) on your favorite regex site (ex. https://regex101.com/) to check if you have a valid <Libraryowner><Project>_<Unit> name.

The Stdlib standard library provides the basic operations over the built-in types (numbers, booleans, byte sequences, strings, exceptions, references, lists, arrays, input-output channels, ...) and standard library modules.

In conventional OCaml programs the Stdlib is automatically opened. That means you can type print_endline "Hi" rather than Stdlib.print_endline "Hi". However, in DkCoder access to the Stdlib is restricted: Stdlib does a bit too much. In particular, input-output channels and threading do not make sense when compiling to JavaScript for use on a web page. Even more critical is that the Stdlib.Obj module is included, which has unsafe functions that make it impossible to prove overall code safety.

You can explicitly get back the unsafe standard library by performing an open at the top of your scripts:

open Tr1Stdlib_V414

However, it is recommended to open the pieces of the standard library you actually need. That way if, for example, your script does not use files it can be compiled to JavaScript.

val exit : int -> 'a
(** Terminate the process, returning the given status code
   to the operating system: usually 0 to indicate no errors,
   and a small positive integer to indicate failure.
   ... *)

val at_exit : (unit -> unit) -> unit
(** Register the given function to be called at program termination
   time. ... *)

type in_channel
(** The type of input channel. *)

type out_channel
(** The type of output channel. *)

val stdin : in_channel
(** The standard input for the process. *)

val stdout : out_channel
(** The standard output for the process. *)

val stderr : out_channel
(** The standard error output for the process. *)

val print_char : char -> unit
(** Print a character on standard output. *)

val print_string : string -> unit
(** Print a string on standard output. *)

val print_bytes : bytes -> unit
(** Print a byte sequence on standard output. *)

val print_int : int -> unit
(** Print an integer, in decimal, on standard output. *)

val print_float : float -> unit
(** Print a floating-point number, in decimal, on standard output. *)

val print_endline : string -> unit
(** Print a string, followed by a newline character, on
   standard output and flush standard output. *)

val print_newline : unit -> unit
(** Print a newline character on standard output, and flush
   standard output. *)

val prerr_char : char -> unit
(** Print a character on standard error. *)

val prerr_string : string -> unit
(** Print a string on standard error. *)

val prerr_bytes : bytes -> unit
(** Print a byte sequence on standard error. *)

val prerr_int : int -> unit
(** Print an integer, in decimal, on standard error. *)

val prerr_float : float -> unit
(** Print a floating-point number, in decimal, on standard error. *)

val prerr_endline : string -> unit
(** Print a string, followed by a newline character on standard
   error and flush standard error. *)

val prerr_newline : unit -> unit
(** Print a newline character on standard error, and flush
   standard error. *)

val read_line : unit -> string
(** Flush standard output, then read characters from standard input
   until a newline character is encountered. ... *)

val read_int_opt: unit -> int option
(** Flush standard output, then read one line from standard input
   and convert it to an integer. ... *)

val read_int : unit -> int
(** Same as {!read_int_opt}, but raise [Failure "int_of_string"]
   instead of returning [None]. *)

val read_float_opt: unit -> float option
(** Flush standard output, then read one line from standard input
   and convert it to a floating-point number. ... *)

val read_float : unit -> float
(** Same as {!read_float_opt}, but raise [Failure "float_of_string"]
   instead of returning [None]. *)

type open_flag =
    Open_rdonly      (** open for reading. *)
  | Open_wronly      (** open for writing. *)
  | Open_append      (** open for appending: always write at end of file. *)
  | Open_creat       (** create the file if it does not exist. *)
  | Open_trunc       (** empty the file if it already exists. *)
  | Open_excl        (** fail if Open_creat and the file already exists. *)
  | Open_binary      (** open in binary mode (no conversion). *)
  | Open_text        (** open in text mode (may perform conversions). *)
  | Open_nonblock    (** open in non-blocking mode. *)
(** Opening modes for {!open_out_gen} and {!open_in_gen}. *)

val open_out : string -> out_channel
(** Open the named file for writing, and return a new output channel
   on that file, positioned at the beginning of the file. ... *)

val open_out_bin : string -> out_channel
(** Same as {!open_out}, but the file is opened in binary mode,
   so that no translation takes place during writes. ... *)

val open_out_gen : open_flag list -> int -> string -> out_channel
(** [open_out_gen mode perm filename] opens the named file for writing,
   as described above. ... *)

val flush : out_channel -> unit
(** Flush the buffer associated with the given output channel,
   performing all pending writes on that channel. ... *)

val flush_all : unit -> unit
(** Flush all open output channels; ignore errors. *)

val output_char : out_channel -> char -> unit
(** Write the character on the given output channel. *)

val output_string : out_channel -> string -> unit
(** Write the string on the given output channel. *)

val output_bytes : out_channel -> bytes -> unit
(** Write the byte sequence on the given output channel. *)

val output : out_channel -> bytes -> int -> int -> unit
(** [output oc buf pos len] writes [len] characters from byte sequence [buf],
   starting at offset [pos], to the given output channel [oc]. ... *)

val output_substring : out_channel -> string -> int -> int -> unit
(** Same as [output] but take a string as argument instead of
   a byte sequence. *)

val output_byte : out_channel -> int -> unit
(** Write one 8-bit integer (as the single character with that code)
   on the given output channel. ... *)

val output_binary_int : out_channel -> int -> unit
(** Write one integer in binary format (4 bytes, big-endian)
   on the given output channel. ... *)

val output_value : out_channel -> 'a -> unit
(** Write the representation of a structured value of any type
   to a channel. ... *)

val seek_out : out_channel -> int -> unit
(** [seek_out chan pos] sets the current writing position to [pos]
   for channel [chan]. ... *)

val pos_out : out_channel -> int
(** Return the current writing position for the given channel. ... *)

val out_channel_length : out_channel -> int
(** Return the size (number of characters) of the regular file
   on which the given channel is opened. ... *)

val close_out : out_channel -> unit
(** Close the given channel, flushing all buffered write operations. ... *)

val close_out_noerr : out_channel -> unit
(** Same as [close_out], but ignore all errors. *)

val set_binary_mode_out : out_channel -> bool -> unit
(** [set_binary_mode_out oc true] sets the channel [oc] to binary
   mode: no translations take place during output. ... *)

val open_in : string -> in_channel
(** Open the named file for reading, and return a new input channel
   on that file, positioned at the beginning of the file. *)

val open_in_bin : string -> in_channel
(** Same as {!open_in}, but the file is opened in binary mode,
   so that no translation takes place during reads. ... *)

val open_in_gen : open_flag list -> int -> string -> in_channel
(** [open_in_gen mode perm filename] opens the named file for reading,
   as described above. ... *)

val input_char : in_channel -> char
(** Read one character from the given input channel. ... *)

val input_line : in_channel -> string
(** Read characters from the given input channel, until a
   newline character is encountered. ... *)

val input : in_channel -> bytes -> int -> int -> int
(** [input ic buf pos len] reads up to [len] characters from
   the given channel [ic], storing them in byte sequence [buf], starting at
   character number [pos]. ... *)

val really_input : in_channel -> bytes -> int -> int -> unit
(** [really_input ic buf pos len] reads [len] characters from channel [ic],
   storing them in byte sequence [buf], starting at character number [pos].
   ... *)

val really_input_string : in_channel -> int -> string
(** [really_input_string ic len] reads [len] characters from channel [ic]
   and returns them in a new string. ...*)

val input_byte : in_channel -> int
(** Same as {!input_char}, but return the 8-bit integer representing
   the character. ... *)

val input_binary_int : in_channel -> int
(** Read an integer encoded in binary format (4 bytes, big-endian)
   from the given input channel. ... *)

val input_value : in_channel -> 'a
(** Read the representation of a structured value, as produced
   by {!output_value}, and return the corresponding value. ... *)

val seek_in : in_channel -> int -> unit
(** [seek_in chan pos] sets the current reading position to [pos]
   for channel [chan]. ... *)

val pos_in : in_channel -> int
(** Return the current reading position for the given channel. ... *)

val in_channel_length : in_channel -> int
(** Return the size (number of characters) of the regular file
    on which the given channel is opened. ... *)

val close_in : in_channel -> unit
(** Close the given channel. ... *)

val close_in_noerr : in_channel -> unit
(** Same as [close_in], but ignore all errors. *)

val set_binary_mode_in : in_channel -> bool -> unit
(** [set_binary_mode_in ic true] sets the channel [ic] to binary
   mode: no translations take place during input. ... *)

val __LOC__ : string
(** [__LOC__] returns the location at which this expression appears in
    the file currently being parsed by the compiler, with the standard
    error format of OCaml: "File %S, line %d, characters %d-%d". *)

val __FILE__ : string
(** [__FILE__] returns the name of the file currently being
    parsed by the compiler. *)

val __LOC_OF__ : 'a -> string * 'a
(** [__LOC_OF__ expr] returns a pair [(loc, expr)] where [loc] is the
    location of [expr] in the file currently being parsed by the
    compiler, with the standard error format of OCaml: "File %S, line
    %d, characters %d-%d". *)

(** Open to use [let%bind] with {!Result}. *)
module BindsResult = struct
  exception ResultFailed of string option

  let bind = Result.bind
  let map = Result.map
  let return = Result.ok
  let zero ?msg () = raise (ResultFailed msg)
end

(** Open to use [let%bind] with {!Option}. *)
module BindsOption = struct
  let bind = Stdlib.Option.bind
  let map = Stdlib.Option.map
  let return = Stdlib.Option.some
  let zero () = None
end

shexp.process

This is a library for creating possibly parallelized pipelines that represent shell scripts.

Docs: shexp README

ocaml-monadic

This small macro (PPX) library provides the let%bind, if%bind and match%bind forms. These are similar in design to Jane Street's ppx_let macros but work in 32-bit architectures and will be easier to support backwards-compatibilty on OCaml 4.

This library also offers the ;%bind, let%orzero and [%guard] forms.

Docs: ocaml-monadic README

DkSDK is a family of tools that supports cross-platform development, even on embedded devices. This cross-platform support places constraints on how you structure your DkCoder projects.

• The FAT32 filesystem is found on older Windows disks, CD drives and on embedded devices. FAT32 only supports filenames up to 255 characters. DkCoder limits the paths <Libraryowner><Libraryproject>_<Libraryunit>/<Modulename>/.../<Modulename>.mli to 240 characters. Even when the FAT32 filesystem entirely disappears from use this 240 character limit may remain since the limit encourages modular, single-focused libraries.
• The FAT16 filesystem is the only filesystem in moderate use today that is not supported by DkCoder. FAT16 might still be used on your older USB thumbdrives and some very old embedded devices. Since FAT16 directory names are limited to 8 uppercase characters, and the minimum <Libraryowner><Libraryproject>_<Libraryunit> name is 7 characters, it is not reasonable to support FAT16.

The primary security goal of DkCoder is to allow the script user to assert, before a script is run, whether the script does not access resources like files, the console, sound, etc. Not all scripts can be asserted, but if DkCoder asserts a script does not have access to files for exmaple, then the script does not have access to files under some light assumptions. These assumptions are detailed in this section.

Assumption A1 - A superset of modules used by a script can be obtained

This assumption is based on the OCaml 4.14 environment model Env.t defined in typing/env.mli. These Env.t have values, modules, types, module types, classes and class types produced during compilation. Only add operations like add_value are present, so by the end of compilation we have a superset of statically compiled modules used by the compiled script.

However,

1. Some of the modules may be functors; that is, new modules can be created at runtime of a specified module type. That leads to a technical requirement:

   If a script uses a functor module, transitively or not, then the script is unassertable (aka. "tainted").

How it will be implemented: The codept analysis tool parses the source and gathers module information, including whether a module is a functor. Using the --log-level DEBUG shows the modules used by the scripts.

Assumption A2 - Access to resources can be gated through modules

By definition, resources are either values like input channels or are accessed through values like function calls (including external C function calls).

In OCaml, values are present in a toplevel Env.t or in a (possibly nested) module and/or class.

1. Since there is no tool like codept to analyze OCaml classes, there is a technical requirement:

   If a script uses a class, transitively or not, then the script is unassertable (aka. "tainted").

2. Since the standard OCaml toplevel environment is the contents of the Stdlib module (ie. there is an implicit open Stdlib at the top of each module), and Stdlib contains toplevel values like print_endline, there is a technical requirement:

   All Stdlib toplevel values are annotated with OCaml alerts that fail the compilation when the toplevel values are used.

3. Since alerts can be locally overridden, there is a technical requirement:

   If a script uses a local alert override, transitively or not, then the script is unassertable (aka. "tainted").

To repeat: In OCaml, values are present in a toplevel Env.t or in a (possibly nested) module and/or class. What remains after implementing the technical requirements above is that resource values necessary for safety assertions are only present in (possibly nested) modules.

How it will be implemented: A PPX can be run that injects a new empty module module Dk__Tainted = struct end whenever a class or local alert override is used. The module analysis by codept will discover the Dk__Tainted, and the analysis phase can fail.

Assumption A3 - Violating the correctness of OCaml type checking can be detected at compile time

The type-safe violations fall into these categories:

1. Using an external C function declaration
2. Using the Obj module
3. Using the Marshal module

This is a weak assumption as it requires expert knowledge of OCaml.

Since the presence of modules can be trivially detected with assumption A1, the technical requirement is:

If a script uses an external C function declaration, transitively or not, then the script is unassertable (aka. "tainted").

How it will be implemented: Any external declaration can be caught by a PPX which injects a module Dk__Tainted = struct end. The presence of any of Dk__Tainted, Obj or Marshal during the codept analysis will taint the script.