arbitrator

Arbitrator

Generative property testing for JavaScript.

arbitrator is a library for generative testing of program properties, ala QuickCheck.

By providing a specification of the JavaScript program in the form of properties, the properties can be tested to remain true for a large number of randomly generated cases. In the case of a test failure, the smallest possible test case is found.

Atop the shoulders of giants

arbitrator is based on Clojure's test.check which is inspired by Haskell's QuickCheck. Many gracious thanks goes to all of the brilliance and hard work enabling this project to exist.

Note: arbitrator is a fork of testcheck-js. All credit goes to Lee Byron for getting this thing off the ground.

Getting started

Install arbitrator using npm

npm install arbitrator

Then require it into your testing environment and start testing.

import {Generator, Property} from 'arbitrator';

const result =
  Property.forAll(Generator.int, x => x - x === 0).check();

API

All API documentation is contained within the type definition file, arbitrator.d.ts.

Defining properties

A property is simply a function which is expected to always return true, we might also call these properties "assumptions" or "expectations".

For example, say we wanted to test the assumption that any number subtracted from itself will be 0, we could define this property as:

function (x) {
  return x - x === 0;
}

Or as another example, let's determine that sorting an array is stable and idempotent, which is to say that sorting a sorted array shouldn't do anything. We could write:

function (arr) {
  var arrCopy = arr.slice();
  return deepEqual(arrCopy.sort(), arr.sort().sort());
}

That's really it! The only thing special about this property function is that it is pure, e.g. it relies only on the provided arguments to determine its return value (no other reading or writing!).

If you can start to describe your program in terms of its properties, then arbitrator can test them for you.

Generating test cases

Once we've defined some properties, we generate test cases for each properties by describing the types of values for each argument.

For testing our first property, we need numbers:

Generator.int

For the second, we need arrays of numbers

Generator.int.array()

There are a wide variety of value generators, we've only scratched the surface. We can pick amongst a set of values with Generator.fromOneOf, nested arrays with ints Generator.int.nested(x => x.array()) and much more. You can even define your own generators with #map, #chain and #sized.

Checking the properties

Finally, we check our properties using our test case generator (in this case, up to 1000 different tests before concluding).

const result =
  Property.forAll(Generator.int, (x) => x - x === 0)
    .check({times: 1000});

check runs through random cases looking for failure, and when it doesn't find any failures, it returns:

{ result: true, 'num-tests': 1000, seed: 1406779597155 }

Smallest failing test

Let's try another property: the sum of two integers is the same or larger than either of the integers alone.

Property.forAll2(Generator.int, Generator.int, (a, b) => {
  return a + b >= a && a + b >= b;
}).check();

check runs through random cases again. This time it found a failing case, so it returns:

{ result: false,
  'failing-size': 2,
  'num-tests': 3,
  fail: [ 2, -1 ],
  shrunk:
   { 'total-nodes-visited': 2,
     depth: 1,
     result: false,
     smallest: [ 0, -1 ] } }

Something is wrong. Either:

1. Our assumption is wrong (e.g. bug in our software).
2. The test code is wrong.
3. The generated test data is too broad.

In this case, our problem is that our generated data is too broad for our assumption. What's going on?

We can see that the fail case 2, -1 would in fact not be correct, but it might not be immediately clear why. This is where test case shrinking comes in handy. The shrunk key provides information about the shrinking process and most importantly, the smallest values that still fail: 0, -1.

We forgot about an edge case! If one of the integers is negative, then the sum will not be larger. This shrunken test case illustrated this much better than the original failing test did. Now we know that we can either improve our property or make the test data more specific:

Property.forAll2(Generator.posInt, Generator.posInt, (a, b) => {
  return a + b >= a && a + b >= b;
}).check();

With our correction, our property passes all tests.

Thinking in random distributions

It's important to remember that your test is only as good as the data being provided. While arbitrator provides tools to generate random data, thinking about what that data looks like may help you write better tests. Also, because the data generated is random, a test may pass which simply failed to uncover a corner case.

"Testing shows the presence, not the absence of bugs"

— Dijkstra, 1969

Sampling Test Data

Visualizing the data check generates may help diagnose the quality of a test. Use sample to get a look at what a generator produces:

Generator.int.sample()
// [ 0, 0, 2, -1, 3, 5, -4, 0, 3, 5 ]

The Size of Test Data

Test data generators have an implicit size property, which could be used to determine the maximum value for a generated integer or the max length of a generated array. arbitrator begins by generating small test cases and gradually increases the size.

So if you wish to test very large numbers or extremely long arrays, running check the default 100 times with maxSize of 200, you may not get what you expect.

Data relationships

Let's test an assumption that should clearly be wrong: a string split by another string always returns an array of length 1.

Property.forAll2(
  Generator.string.notEmpty(),
  Generator.string.notEmpty(),
  (str, separator) => {
    return str.split(separator).length === 1;
  }
).check();

Unless you got lucky, you probably saw this check pass. This is because we're testing for a relationship between these strings. If separator is not found in str, then this test passes. The second random string is very unlikely to be found within the first random string.

We could change the test to be aware of this relationship such that the separator is always contained within the str.

Property.forAll3(
  Generator.string.notEmpty(),
  Generator.posInt,
  Generator.strictPosInt,
  (str, start, length) => {
    var separator = str.substr(start % str.length, length);
    return str.split(separator).length === 1;
  }
).check();

Now separator is a random substring of str and the test fails with the smallest failing arguments: '0', 0, 1.

Contribution

Use Github issues for requests.

Pull requests actively welcomed. Learn how to contribute.