@thi.ng/wasm-api

This project is part of the @thi.ng/umbrella monorepo.

• About
• Custom API modules
• String handling
• Memory allocations
  • API module auto-initialization
• Object indices & handles
• Using the Zig build system
• Naming & structural conventions
• Status
• Support packages
• Installation
• Dependencies
• Usage examples
• API
• Basic usage example
  • Zig version
  • C version
• Authors
• License

About

Generic, modular, extensible API bridge and infrastructure for hybrid JS & WebAssembly projects

This package provides the following:

1. A small WasmBridge class as generic interop basis and much reduced boilerplate for hybrid JS/WebAssembly applications.
2. A minimal core API for debug output, string/pointer/typedarray accessors for 8/16/32/64 bit (u)ints and 32/64 bit floats. Additionally, a number of support modules for DOM manipulation, scheduled function execution, WebGL, WebGPU, WebAudio etc. is being actively worked on.
3. Different types of memory-mapped (UTF-8) string abstractions (slice or pointer based)
4. Shared (opt-in) memory allocation mechanism, also accessible from JS/TS side
5. Simple registration & dependency-order initialization for child WASM API modules
6. Include files for Zig, and C/C++ defining glue code for the TypeScript core API defined by this package
7. Zig build files to simplify using hybrid TS/Zig packages with the built-in build system
8. Extensible shared datatype code generator infrastructure for (currently) Zig & TypeScript and C11. For TS fully type checked and memory-mapped (zero-copy) accessors of WASM-side data are generated. In principle, all languages with a WASM target are supported, however currently only bindings for these mentioned langs are included.
9. CLI frontend/utility for the code generator(s)

Custom API modules

The WasmBridge can be extented via custom defined API modules. Such API extensions will consist of a collection of JS/TS functions & variables, their related counterparts (import definitions) for the WASM target and (optionally) some shared data types (bindings for which can be generated by this package too).

On the JS side, custom API modules can be easily integrated via the IWasmAPI interface. The following example provides a brief overview:

import { IWasmAPI, WasmBridge } from "@thi.ng/wasm-api";

export class CustomAPI implements IWasmAPI {
    // Unique API module identifier to group WASM imports,
    // must match ID used by native code (see further below).
    readonly id = "custom";
    // optionally list IDs of other API modules this module depends on
    // these are used to infer the correct initialization order
    readonly dependencies = [];

    parent!: WasmBridge;

    async init(parent: WasmBridge) {
        this.parent = parent;
        this.parent.logger.debug("initializing custom API");

        // any other tasks you might need to do...

        return true;
    }

    /**
     * Returns object of functions to import as externals into the
     * WASM module during instantiation. These imports are merged
     * into a larger imports object alongside the bridge's core API...
     */
    getImports(): WebAssembly.Imports {
        return {
            /**
             * Writes `num` random float32 numbers from given address
             */
            fillRandom: (addr: number, num: number) => {
                addr >>>= 2;
                while(num-- > 0) this.parent.f32[addr++] = Math.random();
            }
        };
    }
}

Now we can supply this custom API when creating the main WASM bridge:

export const bridge = new WasmBridge([new CustomAPI()]);

In Zig (or any other language of your choice) we can then utilize this custom API like so (Please also see example projects & other example snippets in this readme):

Bindings file / lib:

//! custom.zig - extern definitions of custom JS API

/// JS external to fill a slice w/ random values
/// Note: Each API module uses a separate import object to avoid naming clashes
/// Here we declare an external binding belonging to the "custom" import group
///
/// The bridge core API uses "wasmapi" as reserved import group name
extern "custom" fn fillRandom(addr: [*]f32, num: usize) void;

Main Zig file:

// Import JS core API
const js = @import("wasm-api");
const custom = @import("custom.zig");

export fn test_randomVec4() void {
    var foo = [4]f32{ 1, 2, 3, 4 };

    // print original
    js.printF32Array(foo[0..]);

    // populate foo with random numbers
    custom.fillRandom(&foo, foo.len);

    // print result
    js.printF32Array(foo[0..]);
}

String handling

Most low-level languages deal with strings very differently and alas there's no general standard. Some have UTF-8/16 support, others don't. In some languages (incl. C & Zig), strings are stored as zero terminated, in others they aren't... It's outside the scope of this package to provide an allround out-of-the-box solution. The WasmBridge provides read & write accessors to obtain JS strings from UTF-8 encoded WASM memory. See getString() and setString() for details.

Furthermore, the package provides these string wrapper types:

• WasmStringPtr
• WasmStringSlice

Finally, see more information in the @thi.ng/wasm-api-bindgen package readme.

Memory allocations

If explicitly enabled on the WASM side, the WasmBridge includes support for malloc/free-style allocations (within the linear WASM memory) from the JS side.

The actual allocator is implementation specific and suitable generic mechanisms are defined for both the included Zig & C bindings. Please see for further reference:

• /zig/lib.zig: comments about WASM-side allocator handling in Zig
• /include/wasmapi.h: comments about WASM-side allocator handling in C/C++
• WasmBridge.allocate(): allocating memory from JS side
• WasmBridge.free(): freeing previously allocated memory from JS side

Note: The provided Zig library supports the idiomatic (Zig) pattern of working with multiple allocators in different parts of the application and supports dynamic assignments/swapping of the exposed allocator. See comments in source file and tests for more details...

try {
    // allocate 256 bytes of memory for passing a string to WASM side
    // the function returns a tuple of `[address, len]`
    const [addr, len] = bridge.allocate(256);

    // write zero terminated string to reserved memory (max. `len` bytes)
    // function returns number of bytes written (excl. sentinel)
    const num = bridge.setString("hello WASM world!", addr, len, true);

    // call WASM function doing something w/ the string
    bridge.exports.doSomethingWithString(addr, num);

    // cleanup
    bridge.free([addr, len]);
} catch(e) {
    // deal with allocation error
    // ...
}

API module auto-initialization

The supplied child APIs (wasm-api-dom, wasm-api-schedule etc.) use an auto-intialization hook related to the above WASM_ALLOCATOR mechanism: If that allocator is available, the WASM side of these modules will auto initialize and thus reduce boilerplate. However, if no such central allocator is defined and/or a custom allocator should be used, then these API modules will be have to be initialized manually.

Object indices & handles

Since only numeric values can be exchanged between the WASM module and the JS host, any JS native objects the WASM side might want to be working with must be managed manually in JS. For this purpose the ObjectIndex class can be used by API modules to handle ID generation (incl. recycling, using @thi.ng/idgen) and the indexing of different types of JS objects/values. Only the numeric IDs (handles) will then need to be exchanged with the WASM module...

import { ObjectIndex } from "@thi.ng/wasm-api";

const canvases = new ObjectIndex<HTMLCanvasElement>({ name: "canvas" });

// index item and assign new ID
canvases.add(document.createElement("canvas"));
// 0

// look up item by ID
canvases.get(0);
// <canvas ...>

// work w/ retrieved item
canvases.get(0).id = "foo";

// check if item for ID exists (O(1))
canvases.has(1)
// false

// by default invalid IDs throw error
canvases.get(1)
// Uncaught Error: Assertion failed: missing canvas for ID: 2

// error can be disabled via 2nd arg
canvases.get(1, false)
// undefined

// find ID using custom predicate (same failure behavior as .get())
canvases.find((x) => x.id == "bar")
// Uncaught Error: Assertion failed: given predicate matched no canvas

canvases.delete(0);
// true

The supplied Zig core library also includes a ManagedIndex for similar resource management on the Zig side of the application. For example, in the @thi.ng/wasm-api-dom & @thi.ng/wasm-api-schedule packages this is used to manage Zig-side event listeners.

Using the Zig build system

This package provides utilities to simplify using hybrid TS/Zig WASM API modules which are distributed as NPM packages. Using these utils, a build file for Zig's built-in build system is as simple as:

const std = @import("std");

pub fn build(b: *std.build.Builder) void {
    // obtain a standard std.build.LibExeObjStep, pre-configured w/ given options
    // see source comments in imported build.zig for further details...
    var lib = @import("node_modules/@thi.ng/wasm-api/zig/build.zig").wasmLib(b, .{
        // Declare extra WASM API packages to use
        // Each package can also declare dependencies to other such packages
        // (wasm-api and wasm-api-bindgen are made available everywhere)
        .packages = &.{
            .{ .id = "wasm-api-dom", .path = "@thi.ng/wasm-api-dom/zig/lib.zig" },
            .{ .id = "wasm-api-schedule", .path = "@thi.ng/wasm-api-schedule/zig/lib.zig" },
        },
        // (optional) build mode override
        // if commented out, we can pass CLI args to choose build mode (default: .Debug)
        .mode = .ReleaseSmall,
    });
    // optionally, add further custom configuration
    // ...

    // finally trigger build
    lib.install();
}

All bundled example projects (see list below) are being built via this script. Please find more details/options in the commented source code: /zig/build.zig

Naming & structural conventions

To avoid guesswork about the internals of any of the supplied WASM API modules, please also consult the information in #368.

Status

ALPHA - bleeding edge / work-in-progress

Search or submit any issues for this package

Support packages

• @thi.ng/wasm-api-bindgen - Polyglot bindings code generators for hybrid JS & WebAssembly projects
• @thi.ng/wasm-api-dom - Browser DOM bridge API for hybrid TypeScript & WASM (Zig) applications
• @thi.ng/wasm-api-schedule - Delayed & scheduled function execution (via setTimeout() etc.) for hybrid WASM apps

Installation

yarn add @thi.ng/wasm-api

ES module import:

<script type="module" src="https://cdn.skypack.dev/@thi.ng/wasm-api"></script>

Skypack documentation

For Node.js REPL:

const wasmApi = await import("@thi.ng/wasm-api");

Package sizes (brotli'd, pre-treeshake): ESM: 2.47 KB

Dependencies

• @thi.ng/api
• @thi.ng/arrays
• @thi.ng/checks
• @thi.ng/errors
• @thi.ng/hex
• @thi.ng/idgen
• @thi.ng/logger

Usage examples

Several demos in this repo's /examples directory are using this package.

A selection:

Screenshot	Description	Live demo	Source
	Zig-based DOM creation & canvas drawing app	Demo	Source
	Zig-based 2D multi-behavior cellular automata	Demo	Source
	Simple Zig/WASM click counter DOM component	Demo	Source
	Zig-based To-Do list, DOM creation, local storage task persistence	Demo	Source

API

Generated API docs

Basic usage example

import { WasmBridge, WasmExports } from "@thi.ng/wasm-api";
import { readFileSync } from "fs";

// WASM exports from our dummy module (below)
interface App extends WasmExports {
    start: () => void;
}

(async () => {
    // new API bridge with defaults
    // (i.e. no child API modules and using console logger)
    const bridge = new WasmBridge<App>();

    // instantiate WASM module using imports provided by the bridge
    // this also initializes any bindings & bridge child APIs (if any)
    // (also accepts a fetch() `Response` as input)
    await bridge.instantiate(readFileSync("hello.wasm"));

    // call an exported WASM function
    bridge.exports.start();
})();

Zig version

Requires Zig to be installed:

//! Example Zig application (hello.zig)

/// import externals
/// see build command for configuration
const js = @import("wasm-api");

export fn start() void {
    js.printStr("hello world!");
}

The WASM binary can be built using the following command (or for more complex scenarios add the supplied .zig file(s) to your build.zig and/or source folder):

# compile WASM binary
zig build-lib \
    --pkg-begin wasm-api node_modules/@thi.ng/wasm-api/zig/lib.zig --pkg-end \
    -target wasm32-freestanding \
    -O ReleaseSmall -dynamic \
    hello.zig

# disassemble WASM
wasm-dis -o hello.wast hello.wasm

The resulting WASM:

(module
 (type $i32_i32_=>_none (func (param i32 i32)))
 (type $none_=>_none (func))
 (type $i32_=>_i32 (func (param i32) (result i32)))
 (import "wasmapi" "_printStr" (func $fimport$0 (param i32 i32)))
 (global $global$0 (mut i32) (i32.const 1048576))
 (memory $0 17)
 (data (i32.const 1048576) "hello world!\00")
 (export "memory" (memory $0))
 (export "start" (func $0))
 (export "_wasm_allocate" (func $1))
 (export "_wasm_free" (func $2))
 (func $0
  (call $fimport$0
   (i32.const 1048576)
   (i32.const 12)
  )
 )
 (func $1 (param $0 i32) (result i32)
  (i32.const 0)
 )
 (func $2 (param $0 i32) (param $1 i32)
 )
)

C version

Requires Emscripten to be installed:

#include <wasmapi.h>

void WASMAPI_KEEP start() {
    wasm_printStrZ("hello world!");
}

Building the WASM module:

emcc -Os -Inode_modules/@thi.ng/wasm-api/include \
  -sERROR_ON_UNDEFINED_SYMBOLS=0 --no-entry \
  -o hello.wasm hello.c

Authors

• Karsten Schmidt

If this project contributes to an academic publication, please cite it as:

@misc{thing-wasm-api,
  title = "@thi.ng/wasm-api",
  author = "Karsten Schmidt",
  note = "https://thi.ng/wasm-api",
  year = 2022
}

License

© 2022 Karsten Schmidt // Apache License 2.0