Compartment mapper
The compartment mapper builds compartment maps for Node.js style
applications, finding their dependencies and describing how to create
Compartments for each package in the application.
Creating a compartment map for a Node.js application allows us to harness
the SES module loader to encapsulate each dependency and grant the least
necessary authority to each third-party package, mitigating prototype pollution
attacks and some supply chain attacks.
Since most Node.js packages do not modify objects in global scope,
many libraries and applications work in Compartments without modification.
The importLocation
function runs a compartmentalized application off the file
system.
The globals
are properties to add to the globalThis
in the global scope
of the application's main package compartment.
The modules
are built-in modules to grant the application's main package
compartment.
import fs from "fs";
import { importLocation } from "@endo/compartment-mapper";
const modules = { fs };
const globals = { console };
const read = async location =>
fs.promises.readFile(new URL(location).pathname);
const { namespace } = await importLocation(
read,
moduleLocation,
{
globals,
modules
}
);
The compartment mapper does nothing to arrange for the realm to be frozen.
The application using the compartment mapper is responsible for applying the
[SES] shim (if necessary) and calling lockdown
to freeze the realm (if
necessary).
The compartment mapper is also not coupled specifically to Node.js IO and does
not import any powerful modules like fs
.
The user must provide read
and write
functions from whatever IO powers they
have. These powers can be provided as individual functions or as objects
carrying functions. ReadPowers
has optional functions which can be used to
unlock compatibility features. When fileURLToPath
is available, __dirname
and __filename
will be provided to CJS modules. If requireResolve
is
available, it will be called whenever a CJS module calls require.resolve()
.
type ReadPowers = {
read: (location: string) => Promise<Uint8Array>,
canonical: (location: string) => Promise<string>,
computeSha512: { (bytes: Uint8Array) => string }?,
fileURLToPath: { (url: string | URL) => string }?,
pathToFileURL: { (path: string) => URL }?,
requireResolve: { (from: string, request: string, options?: {}) => string }?
}
TODO
A future version will allow application authors to distribute their choices
of globals and built-in modules to third-party packages within the
application, as with LavaMoat.
The importLocation
function uses loadLocation
.
Using loadLocation
directly allows for deferred execution or multiple runs
with different globals or modules in the same process.
Calling loadLocation
returns an Application
object with an
import({ globals?, modules? })
method.
Use writeArchive
to capture an application in an archival format.
Archives are zip
files with a compartment-map.json
manifest file.
import fs from "fs";
import { writeArchive } from "@endo/compartment-mapper";
const read = async location =>
fs.promises.readFile(new URL(location).pathname);
const write = async (location, content) =>
fs.promises.writeFile(new URL(location).pathname, content);
await writeArchive(
write,
read,
new URL('app.zip', import.meta.url).toString(),
new URL('app.js', import.meta.url).toString()
);
The writeArchive
function uses makeArchive
.
Using makeArchive
directly gives you the archive bytes.
Use importArchive
to run an application from an archive.
Note the similarity to importLocation
.
import fs from "fs";
import { importArchive } from "@endo/compartment-mapper";
const modules = { fs };
const globals = { console };
const read = async location =>
fs.promises.readFile(new URL(location).pathname);
const { namespace } = await importArchive(
read,
archiveLocation,
{
globals,
modules
}
);
The importArchive
function composes loadArchive
and parseArchive
.
Use loadArchive
to defer execution or run multiple times with varying
globals.
Use parseArchive
to construct a runner from the bytes of an archive.
The loadArchive
and parseArchive
functions return an Application
object with an import({ globals?, modules? })
method.
loadArchive
and parseArchive
do not run the archived program,
so they can be used to check the hash of a program without running it.
Package Descriptors
The compartment mapper uses Compartments, one for each Node.js package your
application needs.
The compartment mapper generates a compartment graph from Node.js packaged
module descriptors: the package.json
files of the application and all its
dependencies.
Consequently, an application must have a package.json
.
Each package has its own descriptor, package.json
.
Some standard properties of the descriptor are relevant and used by a
compartment map.
name
type
main
exports
browser
dependencies
files
The compartment map will contain one compartment for each package.json
necessary to build the application.
Like Node.js, the compartment mapper trusts the package manager to arrange the
packages such that a satisfactory version of every package's dependencies rests
in a parent directory, under node_modules
.
The main
, browser
, and exports
properties determine the modules each
package exports to other compartments.
The exports
property describes package entry points and can be influenced
by build tags.
Currently, the only tag supported by the compartment mapper is import
, which
indicates that the module map should use ESM modules over CommonJS modules or
other variants.
TODO
A future version may reveal other tags like browser
to prepare an
application for use in a web client.
For this case, the compartment mapper would prepare a JSON manifest like an
importmap
(if not precisely an importmap
).
The "compartment map" would be consistent except when the dependency graph
changes so updates could be automated with a postinstall
script.
Preparing a web application for production would follow a process similar to
creating an archive, but with the browser
build tag.
The browser
and require
tags are well-known but not yet supported.
The browser
tag will apply for compartment maps generated for use on the web.
The require
tag is a fallback for environments that do not support ESM and
will never apply.
If no exports
apply to the root of the compartment namespace ("."
),
the main
property serves as a default.
TODO
A future version may also respect the imports
property.
TODO
A future version may also respect wildcard patterns in exports
and
imports
.
The files
property indicates all of the files in the package that
should be vended out to applications.
The file set implicitly includes all **.js
, **.mjs
, and **.cjs
files.
The file set implicitly excludes anything under node_modules
.
With the compartment mapper, just as in Node.js, a module specifier that has no
extension may refer either to the file with the js
extension, or if that file
does not exist, to the index.js
file in the directory with the same name.
TODO
The compartment mapper does not yet do anything with the files
globs but a
future version of the compartment mapper will collect these in archives.
The compartment mapper should eventually provide the means for any
compartment to access its own files using an attenuated fs
module or
fetch
global, in conjunction with usable values for import.meta.url
in
ECMAScript modules or __dirname
and __filename
in CommonJS modules.
Officially beginning with Node.js 14, Node.js treats .mjs
files as ECMAScript
modules and .cjs
files as CommonJS modules.
The .js
extension indicates a CommonJS module by default, to maintain
backward compatibility.
However, packages that have a type
property that explicitly says module
will treat a .js
file as an ECMAScript module.
This unforunately conflicts with packages written to work with the ECMAScript
module system emulator in the esm
package on npm, which allows every file
with the js
extension to be an ECMAScript module that presents itself to
Node.js as a CommonJS module.
To overcome such obstacles, the compartment mapper will accept a non-standard
parsers
property in package.json
that maps file extensions, specifically
js
to the corresponding language name, one of mjs
for ECMAScript modules,
cjs
for CommonJS modules, and json
for JSON modules.
All other language names are reserved and the defaults for files with the
extensions cjs
, mjs
, json
, text
, and bytes
default to the language of
the same name unless overridden.
JSON modules export a default object resulting from the conventional JSON.parse
of the module's UTF-8 encoded bytes.
Text modules export a default string from the module's UTF-8 encoded bytes.
Bytes modules export a default ArrayBuffer capturing the module's bytes.
If compartment mapper sees parsers
, it ignores type
, so these can
contradict where using the esm
emulator requires.
{
"parsers": {"js": "mjs"}
}
Many Node.js applications using CommonJS modules expect to be able to require
a JSON file like package.json
.
The compartment mapper supports loading JSON modules from any type of module.
As of Node.js 14, Node does not support importing JSON using ECMAScript
import
directives, so using this feature may limit compatibility with the
Node.js platform.
The compartment mapper supports loading CommonJS modules from ECMAScript
modules as well as ECMAScript modules importing CommonJS modules.
This presumes that the CommonJS modules exclusively use require
calls with a
single string argument, where require
is not lexically bound, to declare
their shallow dependencies, so that these modules and their transitive
dependencies can be loaded before any module executes.
As of Node.js 14, Node does not support loading ECMAScript modules from
CommonJS modules, so using this feature may limit compatibility with the
Node.js platform.
TODO A future version may introduce language plugins, so a package may state
that files with a particular extension are either parsed or linked with
another module.
TODO
The compartment mapper may elect to respect some properties specified for
import maps.
TODO
A future version of the compartment mapper may add support for
source-to-source translation in the scope of a package or compartment.
This would be expressed in package.json
using a property like
translate
that would contain a map from file extension
to a module that exports a suitable translator.
For browser applications, the compartment mapper would use the translator
modules in two modes.
During development, the compartment mapper would be able to load the
translator in the client, with the browser
tag.
The compartment mapper would also be able to run the translator in a separate
non-browser compartment during bundling, so the translator can be excluded
from the production application and archived applications.
TODO
The compartment mapper may also add support for compartment map plugins that
would recognize packages in devDependencies
that need to introduce globals.
For example, packages that use JSX and a virtual DOM would be able to add a
module-to-module translator and endow the compartment with the h
the
translated modules need.
Source Maps
The makeArchive
, makeAndHashArchive
, and writeArchive
tools can receive a
sourceMapHook
as one of its options.
The sourceMapHook
receives a source map string
for every module it
archives, along with details compartment
, module
, location
, and sha512
.
The compartment
is the fully-qualified file URL of the package root.
The module
is the package-relative module specifier.
The location
is the fully-qualified file URL of the module file.
The sha512
, if present, was generated with the computeSha512
power from the
generated module bytes.
The functions importArchive
, loadArchive
, and parseArchive
tools can receive a computeSourceMapLocation
option that recives the same
details as above and must return a URL.
These will be appended to each module from the archive, for debugging purposes.
The @endo/bundle-source
and @endo/import-bundle
tools integrate source maps
for an end-to-end debugging experience.
Design
Each of the workflows the compartment mapper executes a portion of one sequence
of underlying internals.
- search (
search.js
): Scan the parent directories of a given moduleLocation
until successfully finding and reading a package.json
for the containing
application. - map compartments from Node.js packages (
node-modules.js
): Find and gather
all the package.json
files for the application's transitive dependencies.
Use these to construct a compartment map describing how to construct a
Compartment
for each application package and how to link the modules each
exports in the compartments that import them. - load compartments (
archive.js
): Using compartment.load
, or
implicitly through compartment.import
, create a module graph for the
application's entire working set.
When creating an archive, this does not execute any of the modules.
The compartment mapper uses the compartments and a special importHook
that
records the text of every module the main module needed. - import modules (
import.js
, import-archive.js
): Actually execute the
working set.
Around this sequence, we can enter late or depart early to store or retrieve an
archive.
The compartment mapper provides workflows that use read
and write
hooks
when interacting with a filesystem or work with the archive bytes directly.
This diagram represents the the workflows of each of the public methods like
importLocation
.
Each column of pipes |
is a workflow from top to bottom.
Each asterisk *
denotes a step that is taken by that workflow.
The dotted lines .'. : '.'
indicate carrying an archive file from the end of
one workflow to the beginning of another, either as bytes or a location.
In the diagram, "powers" refer to globals and built-in modules that may provide
capabilities to a compartment graph.
For writeArchive
and makeArchive
, these may be provided but will be ignored
since the application does not execute.
loadLocation writeArchive
importLocation | | makeArchive
| | | |
| | | | parseArchive
| | | | | loadArchive
| | | | | | importArchive
| | | | | | |...
search -> * * * * | |'| . '
map compartments -> * * * * .'.| | |' : :
read archive -> | | | ' | * * : :
unpack archive -> | | | : * * * : :
assemble compartments -> * * * : * : : <- powers
load compartments -> * * * : * : :
import modules -> * | | : * : :
pack archive -> * * ' : :
write archive -> * '.' <- data : :
'..............' : <- files
'...............'
Compartment maps
The compartment mapper works by generating a compartment map from your
application workspace and all of the node_modules
it needs.
A compartment map is similar to a lock file because it collects information
from all of the installed modules.
A compartment map describes how to construct compartments for each
package in your application and link their module namespaces.
The compartment map shape:
type CompartmentMap = {
tags: Tags,
entry: Entry,
compartments: Record<CompartmentName, Compartment>,
realms: Record<RealmName, Realm>,
};
type Tags = Array<Tag>;
type Tag = string;
type Entry = CompartmentModule;
type CompartmentName = string;
type Compartment = {
location: Location,
modules: ModuleMap,
parsers: ParserMap,
types: ModuleParserMap,
scopes: ScopeMap,
realm: RealmName?
};
type Location = string;
type ModuleMap = Record<InternalModuleSpecifier, Module>;
type Module = CompartmentModule | FileModule | ExitModule;
type CompartmentModule = {
compartment: CompartmentName?,
module: ExternalModuleSpecifier?,
};
type FileLocation = string
type FileModule = {
location: FileLocation,
parser: Parser,
};
type ExitName = string;
type ExitModule = {
exit: ExitName
};
type InternalModuleSpecifier = string;
type ExternalModuleSpecifier = string;
type ParserMap = Record<Extension, Parser>;
type Extension = string;
type Parser = "mjs" | "cjs" | "json";
type ModuleParserMap = Record<InternalModuleSpecifier, Parser>;
>> When the compartment mapper creates an archive, it captures all of the Modules
>> explicitly and erases the scopes entry.
type ScopeMap = Record<InternalModuleSpecifier, Scope>;
type Scope = {
compartment: CompartmentName
};
type Realm = {
};
type RealmName = string;
type ModuleParameter = string;
Compartment map policy
The policy
option accepted by the compartment-mapper API methods provides means to narrow down the endowments passed to each compartment independently.
The rules defined by policy get preserved in the compartment map and enforced in the application. To explore how policies work, see Policy Demo.
The shape of the policy
object is based on policy.json
from LavaMoat. MetaMask's LavaMoat generates a policy.json
file that serves the same purposes, using a tool called TOFU: trust on first use.
TODO
Endo policy support is intended to reach parity with LavaMoat's policy.json.
Policy generation may be ported to Endo.