@vercel/devlow-bench

devlow-bench

DEVeloper workfLOW BENCHmarking tool

Installation

npm install devlow-bench

Usage

Usage: devlow-bench [options] <scenario files>
## Selecting scenarios
  --scenario=<filter>, -s=<filter>   Only run the scenario with the given name
  --interactive, -i                  Select scenarios and variants interactively
  --<prop>=<value>                   Filter by any variant property defined in scenarios
## Output
  --json=<path>, -j=<path>           Write the results to the given path as JSON
  --console                          Print the results to the console
  --datadog[=<hostname>]             Upload the results to Datadog
                                     (requires DATADOG_API_KEY environment variables)
## Help
  --help, -h, -?                     Show this help

Scenarios

A scenario file is similar to a test case file. It can contain one or multiple scenarios by using the describe() method to define them.

import { describe } from "devlow-bench";

describe(
  "my scenario",
  {
    /* property options */
  },
  async (
    {
      /* property values */
    }
  ) => {
    // run the scenario
  }
);

The describe() method takes three arguments:

• name: The name of the scenario
• props: An object with possible property values for the scenario.
• fn: The function that runs the scenario. It is passed an object with the property values as the first argument.

The props object can contain any number of properties. The key is the name of the property. The value must either be an array of possible values (number, string, boolean), or it can be true as shortcut for [true, false] resp. false for [false, true]. The scenario will run for every possible combination of the property values, if not specified otherwise.

Example

import { describe } from "devlow-bench";

describe(
  "my scenario",
  {
    myProperty: [1, 2, 3],
    myOtherProperty: true,
  },
  async ({ myProperty, myOtherProperty }) => {
    console.log(myProperty, myOtherProperty);
  }
);

// will print:
// 1 true
// 2 true
// 3 true
// 1 false
// 2 false
// 3 false

Reporting measurements

import { measureTime, reportMeasurement } from "devlow-bench";

// Measure a time
await measureTime("name of the timing", {
  /* optional options */
});

// Report some other measurement
await reportMeasurement("name of the measurement", value, unit, {
  /* optional options */
});

Options:

• relativeTo: measure time/value relative to some other measurement.
• scenario: override the reported scenario name (to make measurement independent of scenario name)
• props: override the reported scenario properties (to make measurement independent of scenario properties, object is merged with original props, to remove a prop use null value)

Browser operations

The devlow-bench package provides a few helper functions to run operations in the browser.

import { newBrowserSession } from "devlow-bench/browser";

const session = await newBrowserSession({
  // options
});
await session.hardNavigation("metric name", "https://example.com");
await session.reload("metric name");
await session.softNavigationByClick("metric name", ".selector-to-click");
await session.close();

Run with BROWSER_OUTPUT=1 to show the output of the browser.

Run with HEADLESS=false to show the actual browser window.

Shell operations

The devlow-bench package provides a few helper functions to run operations in the shell.

import { command } from 'devlow-bench/shell';

const shell = await command("pnpm", ["run", "build"], {
  env: { /* optional env vars */ }
  cwd: "/optional/path/to/directory"
});

// Wait for successful exit
await shell.ok();

// Wait for exit
const exitCode = await shell.end();

// Wait for specific output
const [match, world] = await shell.waitForOutput(/hello (world)/);

// Report memory usage or the process tree as metric
await shell.reportMemUsage("metric name", { /* optional options */ });

shell.stdout, shell.stderr

// merged output
shell.output

// Kill the process tree
await shell.kill();

Run with SHELL_OUTPUT=1 to show the output of the shell commands.

File operations

The devlow-bench package provides a few helper functions to run operations on the file system.

import { waitForFile } from "devlow-bench/file";

// wait for file to exist
await waitForFile("/path/to/file", /* timeout = */ 30000);