An overview of MlFront. Part 4 - MlFront_Config and MlFront_Archive

Dec 9, 2024

Recap

So far I have been describing MlFront as a Java-like packaging addition to OCaml. And that is true, but it is quite incomplete.

MlFront’s underlying formalism is a function that can take one or more source files and:

a) make modifications of those source files either in memory or indirectly using OCaml compiler flags
b) generate other source files

The adding of module aliases at the top of source files, a special case of “a)”, is how the Java-like packaging structure is layered on top of OCaml source code:

This Part 4 article today concerns “b)” - the generation of other source files.

Summary

I have an old opam package DkSDKFFIOCaml_Std that is a low-level bridge between OCaml and other programming languages. It can be extraordinarily difficult to build, so I made it a mix of pure OCaml source code and prebuilt library downloads. Yes, with DkCoder (and with MlFront when it is ready) I have a nice scripting framework to automate that difficult build process. But at the end of the day my use of scripting has just been hiding complexity. Today I’ll describe how embedded OCaml dependencies like the following simplifies the build process:

module _ = DkSDKFFI_OCaml
(** The bridge between OCaml and other programming languages.

    {[ `v1 [
          `sec [ `scheme "dkcoder" ];
          `blib ["https://gitlab.com/api/v4/projects/62703194/packages/generic/@DKML_TARGET_ABI@/2.1.4/@DKML_TARGET_ABI@-4.14.2-DkSDKFFI_OCaml-2.1.4-none.blib.zip"];
          `clib ["https://gitlab.com/api/v4/projects/62703194/packages/generic/@DKML_TARGET_ABI@/2.1.4/@DKML_TARGET_ABI@-4.14.2-DkSDKFFI_OCaml-2.1.4-none.clib.zip"]
        ] ]} *)

(* And use what you imported ... *)
let () =
   ignore (DkSDKFFI_OCaml.Com.create_c ())

Context

Let me walk through the backstory first. There is an Android application I’ve been working on with high school robotics students and mentors. Android uses Java or Kotlin as its native programming language, so my library DkSDKFFIOCaml_Std provides the glue to write OCaml source code and run it within an Android/Java application. These users have Windows PCs and laptops. With the Android Studio IDE running on Windows, they can spin up a Android device emulator inside the IDE, and they can plug in their physical Android test devices (mobile phones and tablets) and debug the mobile application over USB.

They need to download and install Qt, Android Studio, Python, OCaml, JDKs, Gradle, MSYS2, WSL2, CMake, and have it all configured and built. Naturally I have this scripted with DkCoder so all these users need to do is copy and paste a couple command line invocations. Assuming nothing goes wrong, that will take a couple hours to perform the install of all that software. I test my scripts inside Windows Sandbox … a clean-room version of Windows. Inevitably there will be something wrong with somebody’s PC; perhaps there was a conflicting global version of Python, or some credentials they had were stale. I get a stack trace when that happens, and almost always I can figure out a solution from just the stack trace. OCaml-ers take that for granted, but you don’t get stack traces with conventional shell scripts and PowerShell. And once I have the solution, I can adjust the scripts and through type-safety be reasonably confident that I am not breaking something else. I can’t overemphasize how important OCaml stack traces and type-safety has been for scripting.

Here is the real problem. OCaml can’t cross-compile from Windows to Android (or Windows to any other operating system). Now OCaml can cross-compile from Linux to Android. So one solution could have been using WSL 2. And I did do that initially, and ran into problems that weren’t obvious at first:

Having to teach Windows users a different operating system (Linux). On the face of it this doesn’t sound so bad because in robotics the deployment environment might be some variant of Linux. The thing that is not obvious to those comfortable with Unix is the unfamiliarity many Windows devs have using the command line. Windows has a GUI culture … point and click … and even deploying software to a robot that runs Linux is normally point-and-click from within an IDE. This gap is quite surmountable, but it takes time.
WSL2 has to support nested virtualization so that Android Studio can launch a device emulator (which itself is a virtual machine). This is not the default configuration because sometimes the Windows hardware + OS version doesn’t support nested virtualization. More pernicious are the memory requirements for WSL2. The minimum recommendation is 8GB system RAM so that WSL2 can take 4GB for itself. On top of that you are going to need about 3GB to run the Android device emulator. With nested virtualization the very commonplace 8GB Windows PC does not have enough RAM.
The WSL2 built-in X11 graphics layer is unstable. It would work for months at a time without issue, and then consistently give you swirling lines. See this and having to reboot is a productivity killer: I got around the swirls by buying a 3rd party X server https://x410.dev/. It wasn’t as integrated into Windows as WSL2 built-in graphics but at least it was stable.
Finally, and most importantly, WSL2 has no direct access to USB. There are hacky solutions to this like tunneling USB over IP, but fundamentally debugging physical Android devices in WSL2 is painful.

All that pain I described was a direct result of OCaml not being able to cross-compile from Windows to Android. OCaml is a long way from having first-class Windows support. I refocused my effort on getting Android Studio running natively on Windows. Android Studio has no problem using the C build tool cmake on Windows to cross-compile to the four Google-supported Android ABI targets. What I did was make a native Windows proxy for cmake that would invoke a dual build inside WSL 2; the proxy would build a cross-compiling OCaml compiler, then compile the OCaml project source code, and then copy the cross-compiled artifacts back to Windows. And mostly that worked. But Android Studio still had one sharp edge: during Android Studio indexing of a project, it would need to scan several intermediate build files, and those scans would be conducted over the incredibly slow WSL2 Windows/Linux file system boundary.

So in a nutshell, to get a reasonably smooth development process on Windows with OCaml and Android Studio requires a high degree of architecture complexity and slowness during key Android Studio operations.

And that brings us all the way back to:

module _ = DkSDKFFI_OCaml
(** The bridge between OCaml and other programming languages.

    {[ `v1 [
          `sec [ `scheme "dkcoder" ];
          `blib ["https://gitlab.com/api/v4/projects/62703194/packages/generic/@DKML_TARGET_ABI@/2.1.4/@DKML_TARGET_ABI@-4.14.2-DkSDKFFI_OCaml-2.1.4-none.blib.zip"];
          `clib ["https://gitlab.com/api/v4/projects/62703194/packages/generic/@DKML_TARGET_ABI@/2.1.4/@DKML_TARGET_ABI@-4.14.2-DkSDKFFI_OCaml-2.1.4-none.clib.zip"]
        ] ]} *)

(* And use what you imported ... *)
let () =
   ignore (DkSDKFFI_OCaml.Com.create_c ())

What if we used bytecode?

Imagine if instead of compiling OCaml source code into native Android libraries with Android Studio’s clang cross-compiling C compiler, we compiled OCaml source code into OCaml bytecode with the ocamlc bytecode compiler. The WSL2 technical requirement for cross-compiling OCaml goes away. Well, it almost goes away … we still need a tool to upconvert an OCaml bytecode library into a nativecode (ex. Android) shared library so that Android Studio can use it. That upconversion will be the subject of a later post. But for now, no WSL2 means less architectural complexity, less time to install, faster builds, and more Windows PCs able to develop the applications.

OCaml bytecode is similar to Java bytecode in that any machine can compile into portable bytecode. Unlike Java which has *.jar files, there is no standard packaging mechanism for OCaml bytecode. OCaml does have a bytecode archive file format called .cma files, but they can’t be used independently without .cmi files and C libraries called “stub” libraries. For those interested, those files are described in detail in the OCaml manual.

One set of designs I created are the MlFront_Archive package formats:

*.blib.zip - This is the bytecode archive. It is a zip file containing .cma, .cmi and some other critical metadata.
*.clib.zip - This is the C library archive. It is a zip file containing .so or .dylib or .dll shared libraries, and also the corresponding static libraries.
*.nlib.zip - This is the nativecode archive. It has the native code variants of the bytecode files. This is not under active development, but for completeness is included.
*.src.zip - This is a source archive. This is not under active development since the OCaml package manager opam and now the conventional build tool dune pkg have a well-defined concept of source archive. However conventional OCaml source archives are not hermetic so I might put some formalism into src.zip to achieve hermeticity at a later date.

The important concept is that *.blib.zip and *.clib.zip for OCaml are analogous to *.jar files for Java. The design is available at:

Format of packages: https://gitlab.com/diskuv/registries/public-packages#generic-registry-layout
Binaries to unpack the packages: https://gitlab.com/dkml/build-tools/MlFront_Archive/-/releases

Expressing dependencies

The other design is how to express a dependency on these JAR-analogs (the blib.zip and clib.zip) within a script. In a Java world we’d add the dependencies to our project’s Maven metadata or within our Gradle build scripts, and in Python we might use a requirements spec file. However, I really wanted dependencies to be embedded inside OCaml source code.

One programming language that does dependency embedding is Go, where we can do import "rsc.io/quote" inside a .go file. I rarely use Go, but my mental model of how it works is:

rsc.io/quote is expanded to https://rsc.io/quote
The META tag <meta name="go-import" content="rsc.io/quote git https://github.com/rsc/quote"> if fetched from https://rsc.io/quote
The git project https://github.com/rsc/quote is cloned which contains a go.mod describing the remote module.

That is a huge simplifier for dependency specification. The idea was great, but I didn’t want to adopt a HTTP-first design for embedding dependencies. HTTP-first, while a natural fit for Go, is a little heavy for use cases centered on scripting. What I expect in scripting is that all the metadata can be self-contained inside the script. In other words, the metadata expressed by Go with <meta name="go-import" content="rsc.io/quote git https://github.com/rsc/quote"> should be inside the script.

That leads to the inclusion of the following in our scripts:

`blib ["https://gitlab.com/api/v4/projects/62703194/packages/generic/@DKML_TARGET_ABI@/2.1.4/@DKML_TARGET_ABI@-4.14.2-DkSDKFFI_OCaml-2.1.4-none.blib.zip"];

And overall my preference is not to add anything that is not syntactically valid OCaml. So introducing the remote metadata using the OCaml idiom:

module _ = DkSDKFFI_OCaml

is valid OCaml while gracefully degrading to a no-op in executed code.

The last design questions were where to place the metadata, and in which format. I settled on putting the metadata into the ocamldoc as a valid OCaml expression because I wanted to have the metadata be a core part of the documentation for a script. That is, if I ran odoc on the script, I could see exactly where the remote libraries were with nice syntax highlighting without having special PPX or annotation processors. But this is probably the squishiest area of the design.

Anyway, we end up with

module _ = DkSDKFFI_OCaml
(** Some description of why you are importing this, and perhaps
    where it comes from. 

    {[ `v1 [ ... ] ]} *)

where `v1 [ ...] is a valid OCaml “remote specification” expression.

The remote specification design is in the MlFront_Config library:

What is the status?

The specs are present, there is an “blib” exporting tool available in DkCoder, and I have enough of the blib downloading + import available in DkCoder that I can edit that problematic Android OCaml source code in VS Code without going through the WSL 2 architectural complexity. There is still much work to do, so I’ll leave the DkCoder announcement until it is ready for public consumption. But certainly MlFront_Config (the remote specification interface) can be reviewed and integrated into your favorite scripting framework today.