@appland/appmap-agent-js

appmap-agent-js

JavaScript agent for the AppMap framework.

To install:

npm install @appland/appmap-agent-js

To run:

mkdir tmp
mkdir tmp/appmap
echo 'enabled: true' > appmap.yml
npx appmap-agent-js -- main.mjs argv0 argv1
cat tmp/appmap/main.appmap.json

Requirements

• unix-like os
• git
• node 12.x || >= 14.0.0 (ie any major node version that is still maintained)
• curl >= 7.55.0 (because of --data @- CLI option)
• mocha >= 8.0.0 (because of root hooks)

CLI (Automated Recording)

The agent's CLI is essentially a test runner for node processes which is capable of recording appmaps. More precisely, the agent spawns child processes based on configuration data and coordinates their recording via a server-client communication.

By default, the agent will read configuration data on the file ./appmap.yml. A custom location for the configuration file can be provided with the --rc-file CLI argument. At the moment, the presence of a configuration file is mandatory but this might change in the future.

Below, we present the three strategies currently supported by the agent to record node processes.

Normal Recorder

The normal recorder will create a single appmap file for the entire spawn child process. It is the default recorder used by the agent. For instance using the fork format:

# appmap.yml
enabled: true
children:
  - type: fork
    recorder: normal # superfluous
    main: lib/main.js
    argv: [argv0, argv1]

Globbing is also supported:

# appmap.yml
enabled: true
children:
  - type: fork
    recorder: normal # Superfluous
    globbing: true   # False by default
    main: lib/*.js   # Will spawn as many children as there
                     # is js files in the lib directory
    argv: [argv0, argv1]

There exists another format called "spawn" which provides more flexibility but does not support globbing:

# appmap.yml
enabled: true
children:
  - type: spawn
    recorder: normal # superfluous
    exec: node
    argv: [main1.js, argv0, argv1]
  # A simpler array format is also supported:
  - [node, main2.js, argv0, argv1]
  # Even direct string:
  - 'node main3.js argv0 argb1'
  # NB: Strings are normalized into:
  - type: spawn
    exec: /bin/sh
    argv: [-c, 'node main3.js argv0 argv1']

Spawning children can also be done via CLI arguments:

npx appmap-agent-js
  --enabled true
  # Multiple children can be provided with the string format
  --children 'node main1.js argv0 argv1'
  --children 'node main2.js argv0 argv2'
  # The array format can be provided as positional arguments:
  -- node main3.js argv0 argv2

An important difference between the fork and spawn format is that: the fork format will not propagate recording to grand-child processes while the spawn format will. This is because, under the hood, the fork format uses command line arguments while the spawn format uses environment variables. A typical scenario where this matters consists in executing a node package with the npx command. There is at least two processes at play: the npx child process and the package grand child process. To fine tune the per-process recording, the enabled option accepts other values than boolean. For instance:

# appmap.yml
enabled:
  # Only record node processes whose main module is
  # (recursively) located inside the cwd. Hence the
  # npx executable will not be recorded (if its not
  # in the current directory).
  - path: .
    recursive: true
children:
  - type: spawn
    recorder: normal # superfluous
    exec: npx
    argv: [package-name, argv0, argv1]

The normal recorder is suitable to record the test suites of test frameworks such as node-tap, tape, and ava. For instance with node-tap:

enabled:
  # Do not record the tap test runner
  # but only its child processes.
  - path: test/
    recursive: true
children:
  - type: spawn
    recorder: normal # superfluous
    exec: npx tap
    # Disable code coverage so that
    # the appmap will not show code
    # generated by istanbul.
    argv: [test/**/*.js, --no-coverage]

It is also possible to directly use the agent's runner:

enabled: true
children:
  - type: fork
    globbing: true
    main: test/**/*.js
    recorder: normal # superfluous

Mocha Recorder

The mocha recorder will create an appmap file for each test case (ie it calls) of the entire test suite (ie every describe calls on every test file).

// lib/abs.mjs
exports const abs = (x) => x < 0 ? -x : x;

// test/abs.mjs
import {abs} from "../lib/abs.mjs";
import {strict as Assert} from "assert";
describe("abs", function () {
  // will generate tmp/appmap/abs-0.appmap.json
  it("should return a positive number", function () {
    Assert.ok(abs(-3) > 0);
  });
  // will generate tmp/appmap/abs-1.appmap.json
  it("should also return a positive number", function () {
    Assert.ok(abs(3) > 0);
  });
});

# appmap.yml #
enabled:
  # Avoid recording the npx command
  - path: test/
    recursive: true
packages:
  # Instrument all files (recursively) located in lib
  - path: lib/
    recursive: true
children:
  # Use the globally installed mocha package
  - type: spawn
    recorder: mocha
    exec: mocha
    argv: [test/abs.mjs]
  # Alternatively, use a command to execute the mocha package
  - type: spawn
    recorder: mocha
    exec: [npx mocha]
    argv: [test/abs.mjs]
  # This will *not* work:
  - type: spawn
    recorder: mocha
    exec: npx
    argv: [mocha, test/abs.mjs]
# NB: Forked children can not be recorded via the mocha recorder

npx appmap-agent-js
cat tmp/appmap/abs-0.appmap.json
cat tmp/appmap/abs-1.appmap.json

Empty Recorder

By itself, the empty recorder will not generate any appmap file. It is useful for increasing the capabilities of manual recording or for using the agent as a regular test runner. More information here.

# appmap.yml
enabled: true
children:
  - type: spawn
    recorder: null
    exec: node
    argv: [main.js]
  - type: fork
    recorder: null
    main: main.js

API (Manual Recording)

An API is provided to perform manual recording.

import {makeAppmap} from "appmap-agent-js";
// Prepare the process for recording
// NB: Only a single concurrent appmap is allowed per process
const appmap = makeAppmap(appmap_configuration);
// Start recording events
// NB: An appmap can create multiple (concurrent) recordings
const recording = appmap.start(recording_configuration);
// Stop recording events
recording.pause();
// Restart recording events
recording.play();
// Terminate the recording and write the appmap file
recording.stop();

An asynchronous api is also provided:

import {makeAppmapAsync} from "appmap-agent-js";
((async () => {
  const appmap = await makeAppmapAsync(appmap_configuration);
  const recording = await appmap.startAsync(recording_configuration);
  await recording.pauseAsync();
  await recording.playAsync();
  await recording.stopAsync();
}) ());

Note that the synchronous and asynchronous API's can be intertwined freely . In clear, a synchronously created appmap can perform asynchronous methods and vice-versa.

Under the hood, the API chooses between two modes. In remote mode, the API detected that it has been required within a process that has been spawned by the agent's CLI (with the empty recorder) and start communicating with the agent process. In inline mode, the API will embed the logic that is normally located on the agent process. Below we summarize the pros and cons of inline mode vs remote mode.

	Inline Mode	Remote Mode
Native module recording	Function calls/returns cannot be recorded within native modules. This is because instrumentation of native module can only be done via the --experimental-loader CLI option.
Communication overhead	Eliminated.
Recording Saving	Can be prevented by the program under recording with operations such as: process.removeAllListeners('exit'). Conflict resolution of appmap files span across a single appmap instance.	Is guaranteed to happen.
Appmap files conflict resolution	Spans across a single appmap instance.	Spans across all the spawned child processes.
Proper Layering	The program under recording can mess with the inlined logic by modifying the global object -- eg: delete String.prototype.substring.	Resilient to modification of the global object.

Class Map

At the base of the class map tree, the file structure is mirrored by package code objects. That is that each directory and each file will be represented by a package code object. Within a file, there are two types of nodes that are eligible to be part of the class map: class-like nodes and function-like nodes.

Class-like Nodes

These nodes are the JavaScript (most common) means to implement classes. As expected, they are each represented by a class code object. There are three types of class-like nodes:

• Class Declaration

class Counter {
  constructor () { this.state = 0; }
  increment () { this.state++; }
}
const counter = new Counter();

• Class Expression

const Counter = class {
  constructor () { this.state = 0; }
  increment () { this.state++; }
};
const counter = new Counter();

• Object Expression In javascript, object literals are used to implement multiple concepts. The most common usages are:
  • Singleton: an object literal can be used to implement a class with a single instance. For instance:

    const counter = {
      state: 0,
      increment () { this.state++; }
    };
    

  • Prototype: an object can be used to embed the sharable fields / methods of a class. For instance:

    const prototype = {
      increment () { this.state++; }
    };
    const counter = {
      __proto__: prototype,
      state: 0
    };
    

  • Mapping: an object can be used to map strings / symbols to values of any type. In principle, this usage should prevent the object literal from appearing in the class map. Unfortunately, there is no general solution to tell this usage apart from the others.

The class code object of class-like node will contain the code object of all the eligible nodes that occur as its property value. In that case, we say that the node is bound to the class-like node.

const isBound (node) => (
  (
    (
      node.parent.type === "MethodDefinition" &&
      node.parent.parent.type === "ClassBody") ||
    (
      node.parent.type === "Property" &&
      node.parent.parent.type === "ObjectExpression")) &&
  node.parent.value === node
);

Bound nodes will be named according to the following algorithm:

const getBoundName = (node) => {
  node = node.parent;
  if (node.type === "MethodDefinition") {
    if (node.kind === "constructor") {
      console.assert(!node.static);
      return "constructor";
    }
    if (node.kind === "method") {
      return `${node.static ? "static " : ""}${getKeyName(node)}`;
    }
    console.assert(node.kind === "get" || node.kind === "set");
    return `${node.static ? "static " : ""}${node.kind} ${getKeyName(node)}`;
  }
  if (node.type === "Property") {
    if (node.kind === "init") {
      return getKeyName(node);
    }
    console.assert(node.kind === "get" || node.kind === "set");
    return `${node.kind} ${getKeyName(node)}`;
  }
  console.assert(false);
};

const getKeyName = (node) => {
  console.assert(node.type === "MethodDefinition" || node.type === "Property");
  if (node.computed) {
    return "[#computed]";
  }
  if (node.key.type === "Identifier") {
    return node.key.name;
  }
  if (node.key.type === "Literal") {
    console.assert(typeof node.key.value === "string");
    return JSON.stringify(node.key.value);
  }
  console.assert(false);
};

For instance:

• var o = { f: function g () {} });: f
• ({ "f": function g () {} });: "f"
• ({ [f]: function g () {} });: [#computed]
• ({ m () {} }): m
• ({ get x () {} });: get x
• (class { constructor () {} });: constructor
• (class { m () {} });: m
• (class { get x () {} });: get x
• (class { static m () {} });: static m
• (class { static get x () {} });: static get x

Function-like Nodes

These nodes are the JavaScript (most common) means to implement functions. They are each represented by a class code object which is guaranteed to includes one and only one function code object named (). This trick is necessary because function code objects are not allowed to contain children in the appmap specification whereas nesting functions and classes inside other functions is one of the key aspect of JavaScript. There are three types of function-like nodes:

• Function Declaration: function f () {};
• Function Expression: const f = function () {};
• Arrow Function Expression: const a = () => {};

The class code object of a function-like node will contain all the eligible nodes that are not bound to a class-like node. Also, they will be named based on the ECMAScript static naming algorithm and prepended by the @ character. If the node has no static name, @anonymous is provided. For instance:

• Declaration:
  • function f () {}: @f
  • class c () {}: @c
  • export default function () {}: @default
  • export default class {}: @default
• Simple Variable Initializer:
  • var f = function g () {};: @g
  • var c = class d () {};: @d
  • var f = function () {};: @f
  • var c = class () {};: @c
  • var o = {}: @o
  • var a = () => {}: @a
• Simple Right-Hand Side:
  • (f = function g () {});: @g
  • (c = class d () {}): @d
  • (f = function () {});: @f
  • (c = class () {}): @c
  • (o = {}): @o
  • (a = () => {}): @a
• Anonymous:
  • var {o} = {{}}: @anonymous
  • (o += {});: @anonymous
  • o.f = function () {}: @anonymous

Configuration

The actual format requirements for configuration can be found as a json-schema here.

Agent-related Options

These options define the behavior of the agent as a test runner.

• protocol <string> Communication protocol between the server process and its client processes by which the agent process and its children processes should communicate. Default: "messaging". Valid values are:
  • "messaging": Simple TCP messaging protocol (faster than http1 and http2).
  • "http1" and "http2": Standard http communication. Will be useful in the future to support browser recording and recording of node processes located on a remote host.
  • "inline": This protocol indicates the agent to inline its logic into the client process. This removes some communication overhead but comes at the cost of blurring the separation between the recording and the program under recording. For instance, the program under test may mess with the recording in the following ways:
    • Removing hooks to write the appmap before exiting the process -- eg: process.removeAllListeners('exit').
    • Modifying the global object -- eg: global.String.prototype.substring = null.
• port <number> | <string>: Port through which the agent process and its children processes should communicate. A string indicates a path to a unix domain socket which is faster. Default: 0 which will use a random available port.
• concurrency <number> | <string>: Set the maximum number of concurrent children. If it is a string, it should be formatted as /[0-9]+%/ and it is interpreted as a percentage of the host's number of logical core. For instance "50%" in machine with 4 logical cores will result in maximum 2 concurrent children. Default: 1 (sequential children spawning).
• children <[]>: List of children to spawn. Valid children types are:
  • <string>: Command to pass to /bin/sh. For instance "node $HOME/main.js" is equivalent to {type:"spawn", exec:"/bin/sh", argv:["-c", "node $HOME/main.js"]}. Note that the command is actually parsed by bash (eg environment variables will be be substituted).
  • <string[]>: A parsed command. For instance ["node", "./main.js"] is equivalent to {type:"spawn", exec:"node", argv:["./main.js"]}. Note that every elements is provided as literal to bash (eg environment variables will not be substituted).
  • <object>: Object describing the spawning of a child process whose structure is largely inspired from require('child_process').spawn.
    • type "spawn"
    • exec <string> | <string[]>: Executable to run. An array of string indicates the usage of an actual executable to run a pseudo executable. Currently, this is only useful for the mocha recorder to indicate where to insert hooks. So {type:"spawn", exec:"npx", argv:["mocha", "test/*.js"]} will not work as expected where {type:"spawn", exec:["npx", "mocha"], argv:["test/*.js"]} will.
    • argv <string[]>: List of command line arguments to pass to the child process.
    • options <object>: Options object
      • require('child_process').spawn options.
      • cwd <string>: Current working directory of the child process. Default: process.cwd().
      • env <object>: Mapping from environment variables to their respective value. This object will be used to extend the environment of the agent process. So the actual environment given to the child process will be: {...process.env, ...child.options.env}. Default {}
      • stdio <string> | [string, string]: Child's stdio configuration.
      • encoding: Encoding of all the child's stdio streams.
      • timeout <number>: The maximum number of millisecond the child process is allowed to run before being killed. Default: 0 (no timeout).
      • killSignal <string>: The signal used to kill the child process when it runs out of time. Default: "SIGTERM".
    • configuration <object>: The child process will use a configuration that is .
    • recorder <string> | null: Name of the runtime that should be used to record the child process. Default: "normal". Valid values are:
      • "normal"
      • "mocha"
      • null
  • <object>: An object describing the spawning of a child process whose structure is largely inspired from require('child_process').fork. Note that when globbing is enabled, this object may actually spawn multiple child processes at runtime.
    • type "fork"
    • globbing <boolean>: Indicates whether the main property should be interpreted as a glob. Default: true.
    • main <string>: Path to the main module.
    • argv <string[]>: List of command line arguments to pass to the main module.
    • options <object>
      • execPath: path to a node executable
      • execArgv: list of command line arguments to pass to the node executable.
      • ``
    • configuration Configuration
    • recorder <string> | null: name of the runtime that should be use to record the child process. Default: "normal". Valid values are:
      • "normal"
      • null

Appmap-related Options

Each appmap may have

• enabled <boolean> | <Specifier[]>: Indicates whether a node process should be recorded based on its main module's path. A boolean
• hook-cjs <boolean>: Indicates whether commonjs modules should be instrumented to record call/return events. Default: true.
• hook-esm <boolean>: Indicates whether native modules should be instrumented to record call/return events. Default: true.
• hook-http <boolean>: Indicates whether the native modules http and https should be monkey-patched to record http traffic. Default: true.
• escape-prefix: The prefix of the variables used to store recording-related data. If variables from the program under recording starts with this prefix, the instrumentation will fail. Default: "APPMAP".
• packages:
• exclude <string[]>: A list of name to always exclude from function

Recording-related Options

• recording
• class-map-pruning <boolean>: Indicates whether all the code entities (ie elements of the classMap array) of a file should be pruned off if the file did not produce any call/return event. Default: false.
• event-pruning <boolean>: Indicates whether call/return events should be pruned off if they are not located .
• output <string> | <object>: Path of the directory where appmap files should be written. The name of the file is based on the map-name option if it is present. Else, it is based on the path of the main module. It is also possible to explicitly set the name of the file using the object form: {"directory": "path/to/output", "file-name": "my-file-name"}. Note that .appmap.json will appened to the provided

If it is an object is may contain two string properties: directory and file-name.

• base <string> | <object>: Path of the directory to which paths from the recording should be express relatively. If it is an object it should be of the form {"directory": "path/to/base"}. "path/to/base" is the same as {directory:"path/to/base/"}. Default: current working directory of the agent process.
• app-name <string>: Name of the app that is being recorded.
• map-name <string>: Name of the recording.
• name <string>: Synonym to map-name.