Fast Check
Yet another property based testing framework BUT written in TypeScript
Getting started
Usage
Documentation [Generated TypeDoc]
Tips
Why should I migrate to fast-check?
Issues found by fast-check in famous packages
Examples - Examples against REST API - More examples
Getting started
Install the module with: npm install fast-check --save-dev
Running examples in RunKit: https://runkit.com/dubzzz/fast-check-basic-examples
Property based testing explained:
If you want to synchronize and build the code locally:
git clone https://github.com/dubzzz/fast-check.git && cd fast-check
npm install
npm run prebuild
npm run build
npm run test
npm run e2e
Usage
In mocha or jasmine
Using fast-check with mocha is really straightfoward.
It can be used directly in describe
, it
blocks with no extra care.
The following snippets written in Javascript shows an example featuring two properties:
const fc = require('fast-check');
const contains = (text, pattern) => text.indexOf(pattern) >= 0;
describe('properties', () => {
it('should always contain itself', () => {
fc.assert(fc.property(fc.string(), text => contains(text, text)));
});
it('should always contain its substrings', () => {
fc.assert(fc.property(fc.string(), fc.string(), fc.string(), (a,b,c) => contains(a+b+c, b)));
});
});
In case of failure, the tests would raise a red flag and the output should help you to diagnose what went wrong in your implementation (example with a failing implementation of contain):
1) should always contain its substrings
Error: Property failed after 1 tests (seed: 1527422598337, path: 0:0): ["","",""]
Shrunk 1 time(s)
Got error: Property failed by returning false
Hint: Enable verbose mode in order to have the list of all failing values encountered during the run
More on integration in:
ava,
jasmine,
jest,
mocha
and
tape.
In a web-page
In order to use fast-check from a web-page (for instance with QUnit or other testing tools), you have to reference the web-aware script as follow:
<script src="https://cdn.jsdelivr.net/npm/fast-check/lib/bundle.js"></script>
You can also reference a precise version by setting the version you want in the url:
<script src="https://cdn.jsdelivr.net/npm/fast-check@0.0.11/lib/bundle.js"></script>
Once it has been included, fast-check becomes accessible directly by calling fastcheck
(in window.fastcheck
). I highly recommend you to alias it by fc
whenever possible by running const fc = fastcheck
at the beginning of the scripts using it.
Documentation
Arbitraries
Arbitraries are responsible for the random generation (but deterministic) and shrink of datatypes.
They can be combined together to build more complex datatypes.
Boolean (:boolean)
fc.boolean()
either true
or false
Numeric (:number)
Integer values:
fc.integer()
all possible integers ie. from -2147483648 (included) to 2147483647 (included)fc.integer(max: number)
all possible integers between -2147483648 (included) and max (included)fc.integer(min: number, max: number)
all possible integers between min (included) and max (included)fc.nat()
all possible positive integers ie. from 0 (included) to 2147483647 (included)fc.nat(max: number)
all possible positive integers between 0 (included) and max (included)
Floating point numbers:
fc.float()
uniformly distributed float
value between 0.0 (included) and 1.0 (excluded)fc.float(max: number)
uniformly distributed float
value between 0.0 (included) and max (excluded)fc.float(min: number, max: number)
uniformly distributed float
value between min (included) and max (excluded)fc.double()
uniformly distributed double
value between 0.0 (included) and 1.0 (excluded)fc.double(max: number)
uniformly distributed double
value between 0.0 (included) and max (excluded)fc.double(min: number, max: number)
uniformly distributed double
value between min (included) and max (excluded)
String (:string)
Single character only:
fc.hexa()
one character in 0123456789abcdef
(lower case)fc.base64()
one character in A-Z
, a-z
, 0-9
, +
or /
fc.char()
between 0x20 (included) and 0x7e (included) , corresponding to printable characters (see https://www.ascii-code.com/)fc.ascii()
between 0x00 (included) and 0x7f (included)fc.unicode()
between 0x0000 (included) and 0xffff (included) but excluding surrogate pairs (between 0xd800 and 0xdfff). Generate any character of UCS-2 which is a subset of UTF-16 (restricted to BMP plan)fc.char16bits()
between 0x0000 (included) and 0xffff (included). Generate any 16 bits character. Be aware the values within 0xd800 and 0xdfff which constitutes the surrogate pair characters are also generated meaning that some generated characters might appear invalid regarding UCS-2 and UTF-16 encodingfc.fullUnicode()
between 0x0000 (included) and 0x10ffff (included) but excluding surrogate pairs (between 0xd800 and 0xdfff). Its length can be greater than one as it potentially contains multiple UTF-16 characters for a single glyph
Multiple characters:
fc.hexaString()
, fc.hexaString(maxLength: number)
or fc.hexaString(minLength: number, maxLength: number)
string based on characters generated by fc.hexa()
fc.base64String()
, fc.base64String(maxLength: number)
or fc.base64String(minLength: number, maxLength: number)
string based on characters generated by fc.base64()
. Provide valid base64 strings: length always multiple of 4 padded with '=' characters. When using minLength
and maxLength
make sure that they are compatible together. For instance: asking for minLength=2
and maxLength=4
is impossible for base64 strings as produced by the frameworkfc.string()
, fc.string(maxLength: number)
or fc.string(minLength: number, maxLength: number)
string based on characters generated by fc.char()
fc.asciiString()
, fc.asciiString(maxLength: number)
or fc.asciiString(minLength: number, maxLength: number)
string based on characters generated by fc.ascii()
fc.unicodeString()
, fc.unicodeString(maxLength: number)
or fc.unicodeString(minLength: number, maxLength: number)
string based on characters generated by fc.unicode()
fc.string16bits()
, fc.string16bits(maxLength: number)
or fc.string16bits(minLength: number, maxLength: number)
string based on characters generated by fc.char16bits()
. Be aware that the generated string might appear invalid regarding the unicode standard as it might contain incomplete pairs of surrogatefc.fullUnicodeString()
, fc.fullUnicodeString(maxLength: number)
or fc.fullUnicodeString(minLength: number, maxLength: number)
string based on characters generated by fc.fullUnicode()
. Be aware that the length is considered in terms of the number of glyphs in the string and not the number of UTF-16 charactersfc.stringOf(charArb: Arbitrary<string>)
, fc.stringOf(charArb: Arbitrary<string>, maxLength: number)
or fc.stringOf(charArb: Arbitrary<string>, minLength: number, maxLength: number)
string based on characters generated by charArb
. minLength
and maxLength
define bounds of the number of elements generated using charArb
that can be found in the final string
Strings that mimic real strings, with words and sentences:
json()
or json(maxDepth: number)
json strings having keys generated using fc.string()
. String values are also produced by fc.string()
unicodeJson()
or unicodeJson(maxDepth: number)
json strings having keys generated using fc.unicodeString()
. String values are also produced by fc.unicodeString()
fc.lorem()
, fc.lorem(maxWordsCount: number)
or fc.lorem(maxWordsCount: number, sentencesMode: boolean)
lorem ipsum strings. Generator can be configured by giving it a maximum number of characters by using maxWordsCount
or switching the mode to sentences by setting sentencesMode
to true
in which case maxWordsCount
is used to cap the number of sentences allowed. This arbitrary is not shrinkable
Combinors of arbitraries (:T)
fc.constant<T>(value: T): Arbitrary<T>
constant arbitrary only able to produce value: T
fc.constantFrom<T>(...values: T[]): Arbitrary<T>
randomly chooses among the values provided. It considers the first value as the default value so that in case of failure it will shrink to itfc.oneof<T>(...arbs: Arbitrary<T>[]): Arbitrary<T>
randomly chooses an arbitrary at each new generation. Should be provided with at least one arbitrary. All arbitraries are equally probable and shrink is still working for the selected arbitrary. fc.oneof
is able to shrink inside the failing arbitrary but not accross arbitraries (contrary to fc.constantFrom
when dealing with constant arbitraries)fc.option<T>(arb: Arbitrary<T>): Arbitrary<T | null>
or fc.option<T>(arb: Arbitrary<T>, freq: number): Arbitrary<T | null>
arbitrary able to nullify its generated value. When provided a custom freq
value it changes the frequency of null
values so that they occur one time over freq
tries (eg.: freq=5
means that 20% of generated values will be null
and 80% would be produced through arb
). By default: freq=5
fc.array<T>(arb: Arbitrary<T>): Arbitrary<T[]>
, fc.array<T>(arb: Arbitrary<T>, maxLength: number): Arbitrary<T[]>
or fc.array<T>(arb: Arbitrary<T>, minLength: number, maxLength: number): Arbitrary<T[]>
array of random length containing values generated by arb
. By setting the parameters minLength
and/or maxLength
, the user can change the minimal (resp. maximal) size allowed for the generated array. By default: minLength=0
and maxLength=10
fc.set<T>(arb: Arbitrary<T>): Arbitrary<T[]>
, fc.set<T>(arb: Arbitrary<T>, maxLength: number): Arbitrary<T[]>
or fc.set<T>(arb: Arbitrary<T>, minLength: number, maxLength: number): Arbitrary<T[]>
set of random length containing unique values generated by arb
. All the values in the set are unique given the default comparator = (a: T, b: T) => a === b
which can be overriden by giving another comparator function as the last argument on previous signaturesfc.tuple<T1,T2,...>(arb1: Arbitrary<T1>, arb2: Arbitrary<T2>, ...): Arbitrary<[T1,T2,...]>
tuple generated by aggregating the values of arbX
like generate: () => [arb1.generate(), arb2.generate(), ...]
. This arbitrary perfectly handle shrinks and is able to shink on all the generatorsfc.dictionary<T>(keyArb: Arbitrary<string>, valueArb: Arbitrary<T>): Arbitrary<{[Key:string]:T}>
dictionary containing keys generated using keyArb
and values gneerated by valueArb
fc.record<T>(recordModel: {[Key:string]: Arbitrary<T>}): Arbitrary<{[Key:string]: T}>
or fc.record<T>(recordModel: {[Key:string]: Arbitrary<T>}, constraints: RecordConstraints): Arbitrary<{[Key:string]: T}>
record using the incoming arbitraries to generate its values. It comes very useful when dealing with settings. It takes an optional parameter of type RecordConstraints
to configure some of its properties. The setting withDeletedKeys=true
instructs the record generator that it can omit some keys
Objects (:any)
The framework is able to generate totally random objects in order to adapt to programs that do not requires any specific data structure. All those custom types can be parametrized using ObjectConstraints.Settings
.
export module ObjectConstraints {
export interface Settings {
maxDepth?: number;
key?: Arbitrary<string>;
values?: Arbitrary<any>[];
};
};
Default for key
is: fc.string()
.
Default for values
are: fc.boolean()
, fc.integer()
, fc.double()
, fc.string()
and constants among null
, undefined
, Number.NaN
, +0
, -0
, Number.EPSILON
, Number.MIN_VALUE
, Number.MAX_VALUE
, Number.MIN_SAFE_INTEGER
, Number.MAX_SAFE_INTEGER
, Number.POSITIVE_INFINITY
or Number.NEGATIVE_INFINITY
.
fc.anything()
or fc.anything(settings: ObjectConstraints.Settings)
generate a possible values coming from Settings and all objects or arrays derived from those same settingsfc.object()
or fc.object(settings: ObjectConstraints.Settings)
generate an objectfc.jsonObject()
or fc.jsonObject(maxDepth: number)
generate an object that is eligible to be stringified and parsed back to itself (object compatible with json stringify)fc.unicodeJsonObject()
or fc.unicodeJsonObject(maxDepth: number)
generate an object with potentially unicode characters that is eligible to be stringified and parsed back to itself (object compatible with json stringify)
Custom arbitraries
Derive existing arbitraries
All generated arbitraries inherit from the same base class: Arbitrary.
It cames with two useful methods: filter(predicate: (t: T) => boolean): Arbitrary<T>
and map<U>(mapper: (t: T) => U): Arbitrary<U>
. These methods are used internally by the framework to derive some Arbitraries from existing ones.
Additionaly it comes with noShrink()
which derives an existing Arbitrary<T>
into the same Arbitrary<T>
without the shrink option.
Filter values
filter(predicate: (t: T) => boolean): Arbitrary<T>
can be used to filter undesirable values from the generated ones. It can be used as some kind of pre-requisite for the parameters required for your algorithm. For instance, you might need to generate two ordered integer values. One approach can be to use filter as follow:
const minMax = fc.tuple(fc.integer(), fc.integer())
.filter(t => t[0] < t[1]);
But be aware that using filter
may highly impact the time required to generate a valid entry. In the previous example, half of the generated tuples will be rejected. It can nontheless be a very useful and powerful tool to derive your arbitraries quickly and easily.
Transform values
map<U>(mapper: (t: T) => U): Arbitrary<U>
in its side does not filter any of the generated entries. It take one entry (generated or shrinked) and map it to another.
For instance the previous example could have been refactored as follow:
const minMax = fc.tuple(fc.integer(), fc.integer())
.map(t => t[0] < t[1] ? [t[0], t[1]] : [t[1], t[0]]);
Another example would be to derive fc.integer()
and fc.array()
to build fc.char()
and fc.string()
:
const char = () => fc.integer(0x20, 0x7e).map(String.fromCharCode);
const string = () => fc.array(fc.char()).map(arr => arr.join(''));
Most of the built-in arbitraries use this trick to define themselves.
Remove the shrinker
Calling noShrink()
on an Arbitrary<T>
just remove the shrinker of the Arbitrary<T>
. For instance, the following code will produce an Arbitrary<number>
without shrinking operation.
const intNoShrink = fc.integer().noShrink();
Build your own
You can also fully customize your arbitrary and by not deriving it from any of the buit-in arbitraries. What you have to do is to derive from Arbitrary and implement generate(mrng: Random): Shrinkable<T>
.
generate
is responsable for the generation of one new random entity of type T
(see signature above). In order to fulfill it in a deterministic way it received a mrng: Random
. It comes with multiple built-in helpers to generate values:
next(n: number): number
: uniformly distributed n bits value (max value of n = 31)nextBoolean(): boolean
: uniformly distributed boolean valuenextInt(): number
: uniformly distributed integer valuenextInt(from: number, to: number): number
: uniformly distributed integer value between from (inclusive) and to (inclusive)nextDouble(): number
: uniformly distributed double value between 0.0 (included) and 1.0 (not included)
The generated value also came with a shrink method able to derive smaller values in case of failure. It can be ignored making the arbitrary not shrinkable.
Once again the built-in types can be very helpful if you need an example.
Properties
fc.property
: define a new property ie. a list of arbitraries and a test function to assess the success
The predicate would be considered falsy if its throws or if output
evaluates to false
.
function property<T1>(
arb1: Arbitrary<T1>,
predicate: (t1:T1) => (boolean|void)): Property<[T1]>;
function property<T1,T2>(
arb1: Arbitrary<T1>, arb2: Arbitrary<T2>,
predicate: (t1:T1,t2:T2) => (boolean|void)): Property<[T1,T2]>;
...
fc.asyncProperty
: define a new property ie. a list of arbitraries and an asynchronous test function to assess the success
The predicate would be considered falsy if its throws or if output
evaluates to false
(after await
).
function asyncProperty<T1>(
arb1: Arbitrary<T1>,
predicate: (t1:T1) => Promise<boolean|void>): AsyncProperty<[T1]>;
function asyncProperty<T1,T2>(
arb1: Arbitrary<T1>, arb2: Arbitrary<T2>,
predicate: (t1:T1,t2:T2) => Promise<boolean|void>): AsyncProperty<[T1,T2]>;
...
WARNING:
The predicate function must not change the inputs it received. If it needs to, it has to clone them before going on. Impacting the inputs might led to bad shrinking and wrong display on error.
Nonetheless a failing property will still be a failing property.
Runners
fc.assert
: run the property and throws in case of failure
This function has to be awaited in case it is called on an asynchronous property.
This function is ideal to be called in describe
, it
blocks.
It does not return anything in case of success.
It can be parametrized using its second argument.
export interface Parameters {
seed?: number;
numRuns?: number;
timeout?: number;
path?: string;
logger?: (v: string) => void;
unbiased?: boolean;
verbose?: boolean;
}
function assert<Ts>(property: IProperty<Ts>, params?: Parameters);
fc.check
: run the property and return an object containing the test status along with other useful details
This function has to be awaited in case it is called on an asynchronous property.
It should never throw whatever the status of the test.
It can be parametrized with the same parameters than fc.assert
.
The details returned by fc.check
are the following:
interface RunDetails<Ts> {
failed: boolean,
numRuns: number,
numShrinks: number,
seed: number,
counterexample: Ts|null,
counterexamplePath: string|null,
error: string|null,
}
function check<Ts>(property: IProperty<Ts>, params?: Parameters);
fc.sample
: sample generated values of an Arbitrary<T>
or Property<T>
It builds an array containing all the values that would have been generated for the equivalent test.
It also accept Parameters
as configuration in order to help you diagnose the shape of the inputs that will be received by your property.
type Generator<Ts> = Arbitrary<Ts> | IProperty<Ts>;
function sample<Ts>(generator: Generator<Ts>): Ts[];
function sample<Ts>(generator: Generator<Ts>, params: Parameters): Ts[];
function sample<Ts>(generator: Generator<Ts>, numGenerated: number): Ts[];
fc.statistics
: classify the values produced by an Arbitrary<T>
or Property<T>
It provides useful statistics concerning generated values.
In order to be able to gather those statistics it has to be provided with a classifier function that can classify the generated value in zero, one or more categories (free labels).
It also accept Parameters
as configuration in order to help you diagnose the shape of the inputs that will be received by your property.
Statistics are dumped into console.log
but can be redirected to another source by modifying the logger
key in Parameters
.
type Generator<Ts> = Arbitrary<Ts> | IProperty<Ts>;
type Classifier<Ts> = ((v: Ts) => string) | ((v: Ts) => string[]);
function statistics<Ts>(generator: Generator<Ts>, classify: Classifier<Ts>): void;
function statistics<Ts>(generator: Generator<Ts>, classify: Classifier<Ts>, params: Parameters): void;
function statistics<Ts>(generator: Generator<Ts>, classify: Classifier<Ts>, numGenerated: number): void;
Tips
Opt for verbose failures
By default, the failures reported by fast-check
feature most relevant data:
- seed
- path towards the minimal counterexample
- number of tries before the first failure
- depth of the shrink
- minimal counterexample
fast-check
comes with a verbose mode, which can help users while trying to dig into a failure.
For instance, let's suppose the folowwing property failed:
fc.assert(
fc.property(
fc.string(), fc.string(), fc.string(),
(a,b,c) => contains(a+b+c, b)));
The output will look something like:
Error: Property failed after 1 tests (seed: 1527423434693, path: 0:0:0): ["","",""]
Shrunk 1 time(s)
Got error: Property failed by returning false
Hint: Enable verbose mode in order to have the list of all failing values encountered during the run
In order to enable the verbose
mode, we just need to give a second parameter to fc.assert
as follow:
fc.assert(
fc.property(
fc.string(), fc.string(), fc.string(),
(a,b,c) => contains(a+b+c, b)),
{verbose: true});
Verbose logs give more details on the error as they will contain all the counterexamples encountered while shrinking the inputs. The example above results in:
Error: Property failed after 1 tests (seed: 1527423434693, path: 0:0:0): ["","",""]
Shrunk 2 time(s)
Got error: Property failed by returning false
Encountered failures were:
- ["","JeXPqIQ6",">q"]
- ["","",">q"]
- ["","",""]
With that output, we notice that our contains
implementation seems to fail when the pattern
we are looking for is the beginning of the string we are looking in.
Preview generated values
Before writing down your test, it might be great to confirm that the arbitrary you will be using produce the values you want.
This can be done very easily by using either fc.sample
or fc.statistics
.
The following code constructs an array containing the first 10 values that would have been generated by the arbitrary fc.anything()
if used inside a fc.assert
or fc.check
:
fc.sample(
fc.anything(),
10
);
In some cases, having a sample is not enough and we want more insights about the generated data.
For instance, I might be interested by the share of even numbers generated by fc.nat()
.
For that purpose I can use fc.statistics
as follow:
fc.statistics(
fc.nat(),
n => n % 2 === 0 ? 'Even number' : 'Odd number',
10000
);
Replay after failure
fast-check
comes with a must have feature: replay a failing case immediately given its seed and path (seed only to replay all).
Whenever fc.assert
encounters a failure, it displays an error log featuring both the seed and the path to replay it. For instance, in the output below the seed is 1525890375951 and the path 0:0.
Error: Property failed after 1 tests (seed: 1525890375951, path: 0:0): [0]
Shrunk 1 time(s)
Got error: Property failed by returning false
In order to replay the failure on the counterexample - [0]
, you have to change your code as follow:
fc.assert(
fc.property(
fc.nat(),
checkEverythingIsOk
)
);
fc.assert(
fc.property(
fc.nat(),
checkEverythingIsOk
),
{
seed: 1525890375951,
path: "0:0"
}
);
Why should I migrate to fast-check?
fast-check
has initially be designed in an attempt to cope with limitations I encountered while using other property based testing frameworks designed for JavaScript:
- strong and up-to-date types - thanks to TypeScript
- ability to shrink on
fc.oneof
- unfortunately it is often not the case - easy
map
method to derive existing arbitraries while keeping shrink - some frameworks ask the user to provide both a->b and b->a mappings in order to keep a shrinker - biased by default - by default it generates both small and large values, making it easier to dig into counterexamples without having to tweak a size parameter manually
- verbose mode - easier troubleshooting with verbose mode enabled
- replay directly on the minimal counterexample
Issues found by fast-check in famous packages
fast-check
has been able to find some unexpected behaviour among famous npm packages. Here are some of the errors detected using fast-check
:
Issue detected: enabling !!int: binary
style when dumping negative integers produces invalid content [more]
Code example: yaml.dump({toto: -10}, {styles:{'!!int':'binary'}})
produces toto: 0b-1010
not toto: -0b1010
Issue detected: unicode characters outside of the BMP plan are not handled consistently [more]
Code example:
leftPad('a\u{1f431}b', 4, 'x')
leftPad('abc', 4, '\u{1f431}')
Issue detected: enabling the bracket
setting when exporting arrays containing null values produces an invalid output for the parser [more]
Code example:
m.stringify({bar: ['a', null, 'b']}, {arrayFormat: 'bracket'})
m.parse('bar[]=a&bar&bar[]=b', {arrayFormat: 'bracket'})