flexi-bench

FlexiBench

FlexiBench is a flexible benchmarking library for JavaScript and TypeScript. It is designed to be simple to use, but also powerful and flexible. It is inspired by common testing framework APIs, and aims to provide a similar experience for benchmarking.

Installation

npm install --save-dev flexi-bench

yarn add --dev flexi-bench

pnpm add --save-dev flexi-bench

Features

• Variations: Run the same benchmark with different configurations.
• Setup and Teardown: Run setup and teardown code before and after the benchmark.
• Commands: Run simple commands as benchmarks.

Usage

Runner API

The runner API is the simplest way to run benchmarks. It allows running a single benchmark or a suite of benchmarks.

const { suite, benchmark, setup, teardown } = require('flexi-bench');

// A `suite` call at the top level will be evaluated as a whole suite.
suite('My Suite', () => {
  // Nested `benchmark` calls will not be evaluated until the parent `suite` is evaluated.
  benchmark('My Benchmark', (b) => {
    setup(() => {
      // some setup to run before the entire benchmark
    });

    teardown(() => {
      // some teardown to run after the entire benchmark
    });

    b.withAction(() => {
      // some action to benchmark
    });
  });
});

// Top-level `benchmark` calls will be evaluated immediately.
benchmark('My Benchmark', (b) => {
  b.withIterations(10).withAction(() => {
    // some action to benchmark
  });
});

Within the callbacks for each suite or benchmark you can utilize the builder API for full customization of the benchmark. Variations can either be added directly via the builder API, or by nesting a variation call within the benchmark call, or at the suite level to apply the variation to all benchmarks in the suite.

const { suite, benchmark, variation } = require('flexi-bench');

suite('My Suite', () => {
  benchmark('My Benchmark', (b) => {
    b.withIterations(10).withAction(() => {
      // some action to benchmark
    });

    variation('with NO_DAEMON', (v) =>
      v.withEnvironmentVariable('NO_DAEMON', 'true'),
    );
  });
});

Basic Benchmarks

More detailed documentation will come soon. For now, here is a simple example:

const { Benchmark } = require('flexi-bench');

const benchmark = new Benchmark('My Benchmark', {
  iterations: 10,
  action: () => {
    // some action to benchmark
  },
});

await benchmark.run();

Most options for the benchmark can also be provided by a builder API:

const { Benchmark } = require('flexi-bench');

const benchmark = new Benchmark('My Benchmark')
  .withIterations(10)
  .withSetup(() => {
    // some setup to run before the entire benchmark
  })
  .withAction(() => {
    // some action to benchmark
  });

Setup and Teardown

Some benchmarks will require some work before the benchmark to setup the environment, and some work after the benchmark to clean up. This can be done with the setup/setupEach and teardown/teardownEach methods:

benchmark('My Benchmark', (b) => {
  setup(() => {
    // some setup to run before the entire benchmark
  });

  setupEach(() => {
    // some setup that is ran before each iteration of the benchmark
  });

  teardown(() => {
    // some teardown to run after the entire benchmark
  });

  teardownEach(() => {
    // some teardown that is ran after each iteration of the benchmark
  });
});

These can also be set using the builder API:

const { Benchmark } = require('flexi-bench');

const benchmark = new Benchmark('My Benchmark')
  .withIterations(10)
  .withSetup(() => {
    // some setup to run before the entire benchmark
  })
  .withSetupEach(() => {
    // some setup that is ran before each iteration of the benchmark
  })
  .withTeardown(() => {
    // some teardown to run after the entire benchmark
  })
  .withTeardownEach(() => {
    // some teardown that is ran after each iteration of the benchmark
  });

Understanding Actions

The action for a benchmark provides the actual event that is being benchmarked. FlexiBench can currently benchmark 2 types of actions:

• Callbacks
• Commands

An action can be specified via the action property of the benchmark or variation constructor, or via .withAction on the builder API. Actions specified on the running variation will override the action specified on the parent benchmark.

For example, the following code will run the action specified on the variation, and not the action specified on the benchmark. This would result in the output bar being printed 10 times, instead of foo:

const { Benchmark } = require('flexi-bench');

const benchmark = new Benchmark('My Benchmark', {
  iterations: 10,
  action: () => {
    console.log('foo');
  },
}).withVariation('with NO_DAEMON', (v) =>
  v.withAction(() => {
    console.log('bar');
  }),
);

Callbacks

If the process you are benchmarking is available as a JavaScript function, you can pass it directly to the action property or execute it within the action callback. For example, if you are benchmarking a function that sorts an array, you can do the following:

const { benchmark, setup } = require('flexi-bench');

benchmark('My Benchmark', (b) => {
  let array;

  setup(() => {
    array = Array.from({ length: 1000 }, () => Math.random());
  });

  b.withIterations(10).withAction(() => {
    array.sort();
  });
});

Commands

If the process you are benchmarking is not available as a JavaScript function, you can run it as a command. This can be done by passing a string to the action property. For example, if you are benchmarking the runtime of a cli command, you can do the following:

const { Benchmark } = require('flexi-bench');

const benchmark = new Benchmark('My Benchmark', {
  iterations: 10,
  action: 'echo "Hello, World!"',
});

await benchmark.run();

While it would be possible to write an action that runs a command using child_process methods, using the syntactic sugar provided by flexi-bench is more convenient and provides a few benefits.

Utilizing the syntactic sugar also means that flexi-bench is aware that you are indeed running a command. This sounds obvious, but opens up some neat possibilities since we are running the command from within flexi-bench. For example, we can add variations based on tailoring CLI options which would not be possible if you ran your command directly using child_process methods on your own.

const { Benchmark } = require('flexi-bench');

const benchmark = new Benchmark('My Benchmark', {
  iterations: 10,
  action: 'echo',
})
  .withVariation('with argument', (v) => v.withArgument('Hello, Earth!'))
  .withVariation('with argument', (v) => v.withArgument('Hello, Mars!'));

await benchmark.run();

When running commands via the syntactic sugar, the command is invoked with a function similar to below:

const child = spawn(action, variation.cliArgs, {
  shell: true,
  windowsHide: true,
});
child.on('exit', (code) => {
  if (code === 0) {
    resolve();
  } else {
    reject(`Action failed with code ${code}`);
  }
});

For more information on the simple command API, see the example: ./examples/simple-command.ts

Suites

A suite is a collection of benchmarks that can be run together:

const { Benchmark, Suite } = require('flexi-bench');

const suite = new Suite('My Suite')
  .addBenchmark(
    new Benchmark('My Benchmark 1', {
      iterations: 10,
      action: () => {
        // some action to benchmark
      },
    }),
  )
  .addBenchmark(
    new Benchmark('My Benchmark 2', {
      iterations: 10,
      action: () => {
        // some action to benchmark
      },
    }),
  );

Suites can also be created using the runner API:

const { suite, benchmark } = require('flexi-bench');

suite('My Suite', () => {
  benchmark('My Benchmark 1', (b) => {
    b.withIterations(10).withAction(() => {
      // some action to benchmark
    });
  });

  benchmark('My Benchmark 2', (b) => {
    b.withIterations(10).withAction(() => {
      // some action to benchmark
    });
  });
});

Variations

Variations allow running the same benchmark with different configurations:

const { Benchmark, Variation } = require('flexi-bench');

const benchmark = new Benchmark('My Benchmark', {
  iterations: 10,
  action: () => {
    // some action to benchmark
  },
}).withVariation('with NO_DAEMON', (v) =>
  v.withEnvironmentVariable('NO_DAEMON', 'true'),
);

Variations can do most things that the main benchmark can do, including having their own setup and teardown functions, or even a custom action.

Some helper functions are provided on the Variation class to make it easier to set up variations:

const { Benchmark, Variation } = require('flexi-bench');

const benchmark = new Benchmark('My Benchmark', {
  iterations: 10,
  action: () => {
    // some action to benchmark
  },
}).withVariations(
  // Adds 4 variations with all possible combinations of the given environment variables
  Variation.FromEnvironmentVariables([
    ['NO_DAEMON', ['true', 'false']],
    ['OTHER_VAR', ['value1', 'value2']],
  ]),
);

Variations can also be added to suites. Variations added to a suite will be applied to all benchmarks in the suite.

For example, the below suite would run each benchmark with 'NO_DAEMON' set to true, and then with 'OTHER_VAR' set to 'value1' for a total of 4 benchmark runs in the suite:

const { Benchmark, Suite, Variation } = require('flexi-bench');

const suite = new Suite('My Suite')
  .addBenchmark(
    new Benchmark('My Benchmark 1', {
      iterations: 10,
      action: () => {
        // some action to benchmark
      },
    }),
  )
  .addBenchmark(
    new Benchmark('My Benchmark 2', {
      iterations: 10,
      action: () => {
        // some action to benchmark
      },
    }),
  )
  .withVariation('with NO_DAEMON', (v) =>
    v.withEnvironmentVariable('NO_DAEMON', 'true'),
  )
  .withVariation('with OTHER_VAR', (v) =>
    v.withEnvironmentVariable('OTHER_VAR', 'value1'),
  );

Examples

See examples folder.

• ./examples/benchmark.ts is the motivation for this project. It benchmarks the performance of Nx commands with and without a daemon.
• ./examples/performance-observer.ts is a simple example of how to use the PerformanceObserver API to measure the performance of a function.
• ./examples/simple-command.ts demonstrates how to benchmark a simple command.