exframe-testing

Exframe testing module

Module that runs unit and contract tests, collects coverage information and stitches it together into a consolidated report. It uses the service's docker-compose.yml file to bring up the necessary environment, then it replaces the service itself with a version that records the code coverage before running the tests.

Use:

The Exframe testing module makes use of exframe-configuration and npm scripts to run tests and collect code coverage. Specifically, exframe-testing uses the following npm scripts:

unit-tests (runs the unit tests)

contract-tests (runs the contract tests)

combine-coverage (merges the coverage.json files output by the above scripts and produces reports of itemized and total code coverage)

In addition, the microservice's configuration must provide a script for running the service with code coverage enabled. This value should be stored in config.default.coverageScript. If not provided, the script defaults to:

./node_modules/.bin/nyc --handle-sigint --reporter lcov --report-dir ./documentation/contract-tests --temp-dir ./documentation/contract-tests node index.js'

The testing framework can be initiated from a service by:

node ./node_modules/exframe-testing/index.js

It is recommended to make the above the npm test script in the package.json of the service.

Executing the above script will run both the unit tests and the contract tests, then merge the coverage objects of both test suites into a single report. You can run either test suite individually with:

node ./node_modules/exframe-testing/index.js unit

or

node ./node_modules/exframe-testing/index.js contract

##Example

package.json:

{
  "name": "my-service",
  "version": "0.0.1",
  "description": "My Service",
  "main": "index.js",
  "repository": {
    "type": "git",
    "url": ""
  },
    "config": {
      "reporter": "spec"
  },
  "scripts": {
    "start": "node index.js",
    "test": "./node_modules/.bin/exframe-testing",
    "unit-tests": "nyc -r lcov --report-dir ./documentation/coverage -t ./documentation/coverage -x index.js -x '**/routing/**' _mocha -- -R $npm_package_config_reporter -c",
    "contract-tests": "./mocha --reporter $npm_package_config_reporter -- -c  ./test-contract",
    "combine-coverage": "nyc merge ./documentation/coverage ./.nyc_output/coverage-unit.json && nyc merge ./documentation/contract-tests ./.nyc_output/coverage-contract.json && nyc report --r lcov --report-dir ./documentation/combined"
  },
  "author": "Exzeo",
  "license": "ISC",
  "dependencies": {},
  "devDependencies": {
    "nyc": "^15.1.0",
    "chai": "^3.5.0",
    "mocha": "^3.2.0"
  }
}

./test-contract/tests/mytest.test.js:

/* eslint arrow-body-style: 0 */
/* eslint no-unused-expressions: 0 */
/* eslint no-multi-spaces: 0 */
/* eslint indent: 0 */


'use strict';

const expect = require('chai').expect;
const axios = require('axios');
const framework = require('exframe-testing');

describe('POST /something', () => {
  describe('Basic happy path', () => {
    const fields = ['description',             'data',   'expectedStatus'];
    const values = [
                   ['Does stuff with thing 1', 'thing1', 200]
                   ['Does stuff with thing2',  'thing2', 400]
                   ]
    const testCases = framework.build(fields, values);
    const test = testCase => {
      it(testCase.description, () => {
        const options = {
          method: 'POST',
          url: `${config.default.service.url}:${config.default.service.port}/something`,
          data: testCase.data
        };
        return axios(options)
          .catch(err => err.response)
          .then(response => {
            expect(response.status).to.equal(testCase.expectedStatus);
          });
      });
    };
    testCases.forEach(testCase => test(testCase));
  });
});

Utility Methods

waitFor(action: () -> Promise, condition: -> bool, period, maxTimeout) -> Promise

Perform an action once every {period} milliseconds for no more than {maxTimeout} milliseconds. Evaluate the result with the given condition callback until {condition} returns true or timesout. Returns a promise that resolves with the result of a successful action.

• action - The action to perform
• condition - The condition to evaluate against the action result
• period - The period in milliseconds to wait between action attempts
• maxTimeout - The max time in milliseconds that action will continue to be attempted. Timeouts will reject the promise with a 'Wait timeout' error.

const response = await waitFor(() => axios({...}), response => response.status === 200);

checkDependency(dependency, [port], [options]) -> Promise

Waits a period of time (10 seconds by default) for the service dependency to be healthy, or reject if the timeout is reached.

• dependency (Object or String): Service dependency that must be healthy before the promise resolves.
  • A String value that matches a property of the default config object in the project's config file will use the underlying value if it's an object containing url and optional port properties. If no matching service is found in the config, the string will be used as the url directly.
  • An object value must contain a url property and optional port property.
• port (String): Optional port to use in the health check to the dependency. *Note: this is only used if dependency is a string containing a raw url, otherwise it is ignored.
• options (Object): Optionally configure various settings of the dependency checks.
  Note: this parameter is currently only supported for objects in the dependencies array.
• options.timeoutMs (Number) Number of milliseconds to wait for each service to become healthy before rejecting.

  await checkDependency('http://myservice', 80, {
    timeoutMs: 30000
  });

checkDependencies(dependencies, [options]) -> Promise

Waits a period of time (10 seconds by default) for all service dependencies to be healthy, or reject if the timeout is reached.

• dependencies (Array of Objects and/or Strings): List of service dependencies that must be healthy before the promise resolves.
  • A String value that matches a property of the default config object in the project's config file will use the underlying value if it's an object containing url and optional port properties. If no matching service is found in the config, the string will be used as the url directly.
  • An object value must contain a url property and optional port property.
• options (Object): Optionally configure various settings of the dependency checks.
  Note: this parameter is currently only supported for objects in the dependencies array.
• options.timeoutMs (Number) Number of milliseconds to wait for each service to become healthy before rejecting.

  await checkDependencies([
    config.harmonyData,
    config.eventing
  ], {
    timeoutMs: 30000
  });

FAQ

When I run the tests, there is an error telling me that the mocha module cannot be found, but mocha is in my package.json. Why can't the test framework find it?

mocha is likely a dev depedency and NODE_ENV=production is probably set in your Dockerfile. Legacy versions of exframe-logger required NODE_ENV=production to be set in order for the logger to send logs to Sematext Logsene. We have changed exframe-logger to send logs to Sematext Logsene only if a token is supplied. So, you may safely remove the following line from your Dockerfile:

ENV NODE_ENV=production

if you do this, you must upgrade to exframe-logger ^2.0.0. Otherwise, your microservice will no longer send logs to Sematext logsene. Please also verify that you are only providing exframe-logger with a valid Sematext token when creating your logger. Many microservices have used the following code:

const logger = require('exframe-logger').create(process.env.LOGSENE_TOKEN || 'token');

With exframe-logger ^2.0.0, there is no longer any need to provide a string as a token when process.env.LOGSENE_TOKEN is undefined. If you do, exframe-logger will continuously attempt to send logs to Sematext Logsene with the invalid token. So, please remove it:

const logger = require('exframe-logger').create(process.env.LOGSENE_TOKEN);

When I run the tests, my service comes up but the tests time out and say that the service is not running

You probably do not have a health check for your service. By default, the testing framework polls /health in order to determine that the service is healthy before it starts running the tests. If your microservice lacks a health check, you can easily add one using the exframe-health framework.

My service is compiled with babel. When I run npm test, the framework fails because it is attempting to run the service with the base code rather than the compiled code.

The default configuration for starting the service with code coverage is:

./node_modules/.bin/nyc --handle-sigint -reporter lcov --report-dir ./documentation/contract-tests --temp-dir ./documentation/contract-tests node index.js'

First, we strongly suggest you refactor your service so that it does not need to use babel. If this is impractical, there is a configuration option you can set to tell the testing framework to run the service with the compiled code. In the service's default configuration, set the coverageScript to point to the compiled code. For example in ./config/default.yml

default:
  service:
    port: "80"
    url: "http://file-index"
  coverageScript: ./node_modules/.bin/nyc --handle-sigint --reporter lcov --report-dir ./documentation/contract-tests --temp-dir ./documentation/contract-tests node ./dist/bootservice.js'

This points the framework to ./dist/bootservice.js and runs the compiled code to start the service.

I want to be able to seed my test database with data using the mongo-seed container

In order for testing to know that you are adding database seeding to your test runs, the framework will need to use a naming standard to determine if the seed container is running.

When setting up your docker-compose to add the mongo-seed container, use the container_name property set to 'mongo-seed'. The framework uses that name to lookup if the container is running and calls the health check on the container to determine when it is done so that contract tests can begin.

  mongo-seed:
    image: exzeo/integration-seed:latest
    container_name: mongo-seed
    depends_on:
      - mongo