testable-utils

Testable Script Utilities

A set of utility APIs for use while running Testable scenarios (Webdriver.io or Node.js script).

• Custom Metrics
• Stopwatch
• Logging
• CSV
• Execution Info
• Webdriver.io Custom Commands

Installation

Available on the public NPM registry as testable-utils.

npm install testable-utils --save

Local Testing

When you run your script locally, any calls to Testable APIs will print to the console. During actual test execution via Testable test runners the API calls integrate as expected.

APIs

Capture Metrics

Capture custom metrics during your test. Testable supports 3 types of metrics. See our custom metrics documentation for more details. Note that these utils do not support traces yet.

Counter

results([resource], [url]).counter(name, [increment], [units])

Keep track of a counter across test execution. Increment defaults to 1. Resource and url default to blank and are included with the "overall results".

For example:

var results = require('testable-utils').results;

results().counter('slowRequests', 1, 'requests');

Timing

results([resource], [url]).timing(name, timing, [units])

Capture a timing. Units defaults to ms. Resource and url default to blank and are included with the "overall results". Testable will calculate various aggergations like min, max, average, standard deviation, and the percentiles defined in your test configuration.

For example:

var results = require('testable-utils').results;

results('Google Homepage', 'https://www.google.com').timing('pageLoadMs', 1294);

Histogram

results([resource], [url]).histogram(name, bucket, [increment])

Capture a histogram. Increment defaults to 1. Resource and url default to blank and are included with the "overall results".

For example:

var results = require('testable-utils').results;

results().histogram('httpResponseCodes', 200);

Stopwatch

Utility API to time how long a piece of code takes to execute and capture as a Testable metric.

For example:

var stopwatch = require('testable-utils').stopwatch;

stopwatch(function(done) {
  // some code we want to time here
  // call done() once it is done, can be async
  done();
}, 'myCustomTimer').then(function() {
	// called after done() is invoked
});

Logging

By default all Webdriver.io output to stdout and stderr is logged at the debug level during test execution. To log at other log levels use this API.

Examples:

var log = require('testable-utils').log;

log.trace("This will only be captured during a smoke test or when run locally");
log.debug("Some useful debug");
log.info("Some useful info");
log.error("Whoops something went wrong");
log.fatal("Something went really wrong, lets abort the entire test");

Execution Info

Information about the test context in which this test is executing. See the Testable docs for more details.

When executed locally the info is set to dummy values.

Accessible at:

// Information on chunk, concurrent client, execution, data stores, iteration, etc
var info = require('testable-utils').info;

// true when run locally, false when run on testable
var isLocal = require('testable-utils').isLocal;

CSV

Read one or more rows from a CSV file. When run locally it looks for the CSV file on the local filesystem. When run on Testable, make sure you upload the CSV to the scenario.

The API is as described in the Testable documentation.

For the below examples we will use a data.csv file:

Symbol,Price
MSFT,100
IBM,101
GOOG,600

See the Testable documentation for full details of the options.

Get row by index

Gets a row from the CSV file by index. Indices start at 1. get() return a Promise.

var dataTable = require('testable-utils').dataTable;
dataTable
	.open('data.csv')
	.get(1)
	.then(function(result) {
		// Example:
		// { index: 1, data: { Symbol: 'MSFT', Price: '100' }, indexed: [ 'MSFT', '100' ] }
		console.log('Symbol: ' + result.data['Symbol']);
	});

Get random row

Gets a random row from the CSV file. random() return a Promise.

var dataTable = require('testable-utils').dataTable;
dataTable
	.open('data.csv')
	.random()
	.then(function(result) {
		// Example:
		// { index: 1, data: { Symbol: 'MSFT', Price: '100' }, indexed: [ 'MSFT', '100' ] }
		console.log('Symbol: ' + result.data['Symbol']);
	});

Iterate CSV

Iterate over the CSV file, retrieving 1 or more rows. The iterator is global across the entire test execution. See Testable documentation for full set of options. next() return a Promise.

var dataTable = require('testable-utils').dataTable;
dataTable
	.open('data.csv')
	.next()
	.then(function(results) {
		// Example:
		// [ { index: 1, data: { Symbol: 'MSFT', Price: '100' }, indexed: [ 'MSFT', '100' ] } ]
		console.log('Symbol: ' + results[0].data['Symbol']);
	});

Webdriver.io Commands

All of the API calls above are registered as custom commands with Webdriver.io.

Simply include var testableUtils = require('testable-utils'); in you test spec or in one of the configuration file hooks like onPrepare() and all the below custom commands will automatically get registered.

Note that all the Webdriver.io commands can be used in a synchronous fashion.

Screenshots

One command that has no testable-utils equivalent is browser.testableScreenshot(name). This command takes a screenshot and puts it in the output directory to be collected as part of the test results. It also includes a prefix to make it easier to identify: [region]-[chunk]-[user]-[iteration]-[name].png. Tests are broken up into chunks, and within each chunk users and iterations are numbered starting at 0. So for example us-east-1-123-0-0-MyHomePage.png would be chunk id 123, first user, first iteration, image name MyHomePage.

Command Mappings

Webdriver.io	testable-utils
browser.testableInfo()	info
var result = browser.testableCsvGet(csvFile, index);	dataTable .open(csvFile) .get(index) .then(function(result) { ... }
var result = browser.testableCsvRandom(csvFile);	dataTable .open(csvFile) .random() .then(function(result) { ... }
var results = browser.testableCsvNext(csvFile[, options]);	dataTable .open(csvFile) .random([options]) .then(function(results) { ... }
var result = browser.testableResult([resource], [url]); browser.testableCounter( result, name, [increment], [units]);	results([resource], [url]) .counter(name, [increment], [units]);
var result = browser.testableResult([resource], [url]); browser.testableTiming( result, name, timing, [units]);	results([resource], [url]) .timing(name, timing, [units]);
var result = browser.testableResult([resource], [url]); browser.testableHistogram( result, name, bucket, [increment]);	results([resource], [url]) .histogram(name, bucket, [increment]);
browser.testableLogTrace(msg);	log.trace(msg);
browser.testableLogDebug(msg);	log.debug(msg);
browser.testableLogInfo(msg);	log.info(msg);
browser.testableLogError(msg);	log.error(msg);
browser.testableLogFatal(msg);	log.fatal(msg);
// blocks until done() is called browser.testableStopwatch(function(done) { // some code to time done(); }, metricName, [resource]);	// returns Promise immediately stopwatch(function(done) { // some code to time done(); }, metricName, [resource]);