iso-bench

iso-bench

iso-bench is a small benchmark library focused on avoiding optimization/deoptimization pollution between tests by isolating them.

Motivation

I've always used benchmark.js for my benchmark tests, but I noticed that changing the tests order also changed the performance outcome. They were getting polluted between them with V8 and memory optimizations/deoptimizations. After this, I decided to take advantage of forking to do tests in completely separated processes with their own V8 instances, memory and so on, to avoid present and future optimization/deoptimization pollution.

All single threaded benchmark libraries, like benny or benchmark.js have this problem, so you may had this pollution on your tests and you didn't even notice, just thinking that one test was faster than the other. This happened to me, and when I noticed the problem I had to redo some PacoPack code ☹️.

Pollution examples

Running this test on benchmark.js, it will return different outcomes. Note how I rerun the very same first test again:

const Benchmark = require("benchmark");
const functions = {
    method: function(buf:Buffer) {
        return buf.readUint8(0);
    },
    direct: function(buf:Buffer) {
        return buf[0];
    },
    method_again: function(buf:Buffer) {
        return buf.readUint8(0);
    }
};
const buffers = new Array(1000).fill(0).map(() => {
    const buf = Buffer.allocUnsafe(1);
    buf[0] = Math.floor(Math.random() * 0xFF);
    return buf;
});
const suite = new Benchmark.Suite();
for (const [type, fn] of Object.entries(functions)) {
    suite.add(`${type}`, () => {
        for (let i = 0; i < buffers.length; i++) {
            fn(buffers[i]);
        }
    });
}
suite.on("cycle", event => {
    console.log(String(event.target));
}).run({
    async: true
});

Which yields the next results:

method       x 314,830 ops/sec
direct       x 300,522 ops/sec
method_again x 187,985 ops/sec // SLOWER THAN "method" WHICH IS THE SAME ONE??

And if I run the direct test first, it is even worse:

direct       x 1,601,246 ops/sec // 5 TIMES FASTER THAN BEFORE??
method       x 183,015 ops/sec // This test already got deoptimized
method_again x 183,956 ops/sec

On iso-bench this is not possible, as every test will run in a completely different process. No matter the order, the outcome will be equally stable. This is the very same test on iso-bench:

import { IsoBench } from "..";
const bench = new IsoBench();
const functions = {
  method: function(buf:Buffer) {
      return buf.readUint8(0);
  },
  direct: function(buf:Buffer) {
      return buf[0];
  },
  method_again: function(buf:Buffer) {
      return buf.readUint8(0);
  }
};
const buffers = new Array(1000).fill(0).map(() => {
  const buf = Buffer.allocUnsafe(1);
  buf[0] = Math.floor(Math.random() * 0xFF);
  return buf;
});
for (const [type, fn] of Object.entries(functions)) {
  bench.add(`${type}`, () => {
      for (let i = 0; i < buffers.length; i++) {
          fn(buffers[i]);
      }
  });
}
bench.consoleLog().run();

Which yields these results with zero pollution:

method       - 1.714.953 op/s in 3140 ms. 1.009x (BEST)
direct       - 1.712.045 op/s in 3032 ms. 1.008x
method_again - 1.699.022 op/s in 3128 ms. 1.000x (WORSE)

Installation

npm install iso-bench

Usage

Example code:

import { IsoBench } from "iso-bench";

const bench = new IsoBench("My bench");
bench.add("indexOf", () => {
    "thisisastring".indexOf("a") > -1;
})
.add("RegExp", () => {
    /a/.test("thisisastring");
})
.consoleLog()
.run();

Documentation

new IsoBench(name, options?);

Creates a new IsoBench to add tests.

• name: The name of this IsoBench instance. Optional.
• options: Object:
  • parallel: The amount of parallel tests to run. Defaults to 1.
  • time: The minimum time (in milliseconds) to invest on each test. The library will automatically increase the amount of cycles to reach a minimum of ms between tests to take samples. Defaults to 3000.
  • samples: Amount of samples to get. Defaults to 1.
  • warmUpTime: The minimum time (in milliseconds) to pre-run the tests, so the JavaScript engine optimizer kicks-in before initializing the timer. Defaults to 500.

---

bench.add(name, test):this;

Adds new test.

• name: The name of this test.
• test: The test function to run. Returns the IsoBench instance, to concatenate new tests easily.

---

bench.consoleLog():this;

Adds a built-in Processor that outputs the result in the console.

---

bench.addProcessor(processor:Processor):this;

Adds a custom Processor that must implement the Processor interface.

---

bench.run():Promise<Result>;

Runs the tests and returns a Promise that will resolve when all the tests are completed. It will return a Result instance.

Result

This is the result of the benchmark. It will contain a list of the tests executed. Note that inside the forked processes, this result will not contain any test (getTests() will return null), as the main process should be the only one processing the results.

---

result.getTests():Test[]|null;

Returns an array of test results in the main process or null in a child process. Always check for null and do nothing if it is null. Only the master process should work with the result.

Processor

Processors will receive the benchmark events to process them. They must implement the Processor interface:

export interface Processor {
    end?(result:Result):void;
}

Processor methods:

end(result:Result):void;

Will be called with a Result instance when the benchmark ends. Optional.

---

Custom Processor example:

import { Processor, Result } from "iso-bench";
class MyProcessor implements Processor {
    end(result:Result) {
        const tests = result.getTests();
        if (tests) {
            console.log(tests);
            // TODO: Work with the tests[] array
        }
    }
}