attester

Attester

attester is a command line tool allowing to run JavaScript tests in multiple web browsers at once.

It starts an internal web server, then starts a set of web browsers, makes them execute the tests, and finally writes test reports.

It is written in JavaScript, to be run with node.js.

Features

• Supports multiple browser instances to run tests in parallel
• Includes instrumentation for code coverage with node-coverage
• Supports PhantomJS for fully headless tests
• Compatible with most other web browsers
• Test results output formats:
  • Json file
  • JUnit-style single XML file
  • JUnit-style set of files, format accepted by Sonar
• Code coverage output formats:
  • node-coverage json file
  • lcov file, accepted by Sonar (currently only for line coverage)
• Supports Aria Templates unit tests and Mocha with either Chai or Expect.js.
• Adding support for other test frameworks is as simple as adding an adapter for that test framework.

License

Apache License 2.0

Dependencies & installation

You'll need node.js. Attester is available in npm repository; you can use npm install attester in your console, or clone this repository and then issue npm install.

If you want to make use of PhantomJS headless testing, you'll additionally need to install PhantomJS and make sure it's in your PATH.

Installing PhantomJS on Windows

1. Download phantomjs-1.9.8-windows.zip and extract it.
2. Move the contents of phantomjs-1.9.8-windows to C:\bin\phantomjs
3. Add C:\bin\phantomjs to PATH
4. Check that it works by issuing phantomjs --version in cmd

Installing PhantomJS on Ubuntu

Quick setup on 64bit Ubuntu:

cd /usr/local/share/
sudo wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-x86_64.tar.bz2
sudo tar jxvf phantomjs-1.9.8-linux-x86_64.tar.bz2
sudo ln -s /usr/local/share/phantomjs-1.9.8-linux-x86_64/ /usr/local/share/phantomjs
sudo ln -s /usr/local/share/phantomjs/bin/phantomjs /usr/local/bin/phantomjs

If you have 32bit version, replace x86_64 with i686 in the commands above.

Check that it works by issuing phantomjs --version in console.

Usage

attester [<options>] [<configuration file>]

Configuration file

The configuration file describes the test campaign to execute. It can be either in the YAML (with a .yml or .yaml extension) or in the JSON format (with a .json extension).

Here is a sample .yml configuration file with explanations about each setting:

resources:  # Specifies which files will be accessible through the web server:
 '/': # This configures resources for the root of the web server:
  - 'src/main/js'   # It is possible to specify one or more directories for each path.
  - 'src/tests'     # When requesting a file, the first directory which contains it is used.
 '/aria': # This configures resources for the /aria path on the server:
  - 'libraries/aria'
tests:
# Describes test configuration for each type of test
# Only Aria Templates tests are supported currently, but other types of tests will be added in the future
 aria-templates:
  # There are two ways to specify which tests have to be executed:
  # either by their classpath or by their file path
  # It is possible to combine both.
  # With file paths, it is possible to use patterns.
  classpaths:
   includes:
    - MainTestSuite
   excludes:
    - test.sample.MyUnfinishedTest
   browserExcludes: # Allows to specify some excluded classpaths specific to each browser
     PhantomJS: # Each browser listed in browserExcludes must exactly match the name of a browser in the browsers section
      - test.sample.MyTestFailingInPhantomJS
  files:
   rootDirectory: 'src/tests'
   includes:
    - 'test/example/*TestCase.js'
   excludes:
    - 'test/example/*SpecialTestCase.js'
   browserExcludes: # Allows to specify some exclusion patterns specific to each browser
    PhantomJS: # Each browser listed in browserExcludes must exactly match the name of a browser in the browsers section
     - 'test/example/*NoPhantomTestCase.js'
  bootstrap : '/aria/bootstrap.js' # Path to the bootstrap file of Aria Templates. This is the default value.
  extraScripts: # Path to extra scripts to be inserted in the test page just after the Aria Templates bootstrap file.
   - '/test/testEnvConfig.js'
  rootFolderPath : '/' # Root folder path (Aria.rootFolderPath variable) This is the default value.
  debug : true # Enables or disables Aria Templates debug mode (Aria.debug variable). This is the default value.
  memCheckMode : true # Enables or disables memory check mode (Aria.memCheckMode variable). This is the default value.
 mocha:
  files:
   includes:
    - 'test/example/*.js'
   excludes:
    - 'test/example/*.conf.js'
   browserExcludes: # Allows to specify some exclusion patterns specific to each browser
    PhantomJS: # Each browser listed in browserExcludes must exactly match the name of a browser in the browsers section
     - 'test/example/notWorkingInPhantomJS/*.js'
  ui: 'bdd' # User-interface (bdd|tdd|exports). Defaults to bdd
  ignoreLeaks: false # Optional to ignore global variable leaks
  globals: # Optional, allow the given list of global [names]
   - 'myGlobal'
  assertion: 'expect' # Assertion library to be included. It could be either expect or chai or any external url
coverage:
 files: # Specifies which files will be instrumented for code coverage
  rootDirectory: 'src/main/js'
  includes:
   - '**/*.js'
test-reports: # Path for each test report type:
  json-file: 'report.json'
  xml-file: 'report.xml'
  xml-directory: 'reports'
coverage-reports: # Path for each coverage test report type:
  json-file: 'coverage.json'
  lcov-file: 'coverage.lcov'
browsers:
  # If the browsers section is not present, attester will not make any difference between browsers
  # and run each test only once (in whatever browser is connected)
  # However, if the browsers section is present, each test will be run once in each browser listed here.
  # (and browsers not listed here will have nothing to do if they are connected)
  - 'PhantomJS'
  - 'Opera'
  # It's possible to distinguish browsers by operating systems, read more below
  - 'Safari on Mac OS X'
  - 'Chrome on Windows 7'
  - 'Chrome on Ubuntu'
  - 'Chrome on Android'
  # It is also possible to distinguish several versions of the same browser:
  - 'IE 7'
  - 'IE 8'
  - 'IE 9'
  - 'IE 10'
  # Browser version can be also a semver-compliant string; see https://github.com/isaacs/node-semver
  - 'Firefox 3.6'
  - 'Firefox >=20'
  # In the logs, browser will be identified using the name, version and OS as provided. If you want to change that, add an alias at the end using 'as':
  - 'Chrome 28 as Chrome Stable 28'
  - 'Chrome 30 as Chrome Canary 30'
  # You can mix all of that options, and have whitespace as you see fit.
  - 'Firefox >=25 on Windows 7 as Firefox/Windows'
  - 'Firefox >=25 on Ubuntu    as Firefox/Ubuntu'
  # You can also add flags with the 'with' keyword to specify an environment which cannot be automatically detected.
  # Flags have to be explicitly set in the URL when connecting a browser (e.g.: http://localhost:7777/__attester__/slave.html?flags=JAWS).
  - 'IE 11 on Windows with JAWS as IE-JAWS'
  - 'IE 11 on Windows with mySpecialFlag'

Attester uses ua-parser-js for browser and operating system detection and supports values as returned by ua-parser-js.

Environment Variables It is possible to build a special portion of the configuration object from an external file using --env option.

attester --env package.json

This puts the content of package.json inside the env section of attester configuration. It is then possible to reference such values with simple templates.

<%= prop.subprop %> Expands to the value of prop.subprop in the config. Such templates can only be used inside string values, either in arrays or objects

{
  "resources": {
    "/": "src"
  },
  "tests": {
    "mocha": {
      "files": {
        "includes": ["spec/**/*"]
      }
    }
  },
  "coverage": {
    "files": {
      "rootDirectory": "src",
      "includes": ["**/*.js"]
    }
  },
  "coverage-reports": {
    "json-file": "/reports/coverage-<%= env.version %>.json"
  }
}

The configuration above generates a coverage report file called coverage-x.y.z.json where x.y.z is the value taken from package.json or any other file passed referenced by --env.

Template options can be used both in yaml and json file and the environment file can be of any of the two format.

Usual options:

--phantomjs-path <path> Path to the PhantomJS executable. Attester does not install PhantomJS automatically. The path to the PhantomJS executable can be specified either in the PHANTOMJS_PATH environment variable, or through this command line option. If none are specified, the phantomjs binary should be in your PATH for Attester to find it.

--phantomjs-instances <number> Number of instances of PhantomJS to start (default: 0). Additionally, a string auto can be passed to let the program use the optimal number of instances for best performance (max. 1 per CPU thread).

--run-browser <path> Path to any browser executable to execute the tests. Can be repeated multiple times to start multiple browsers or multiple instances of the same browser. Each browser is started with one parameter: the URL to open to start tests. At the end of the tests, all started processes are killed.

--launcher-config <launcher config> Path to an attester-launcher configuration file. Browsers configured in this file are automatically started and stopped when needed. Can be repeated multiple times to pass multiple configuration files to attester-launcher.

--env <path> Path to a yaml or json file containing environment options. See section above.

--ignore-errors When enabled, test errors (not including failures) will not cause this process to return a non-zero value.

--ignore-failures When enabled, test failures (anticipated errors) will not cause this process to return a non-zero value.

--live-results When enabled (which is the default, disable with --no-live-results), it is possible to connect to attester to get live test results. Note that, when enabled, this feature keeps the full log of test results in memory, which increases the memory consumption of attester.

--host <host> Host used for the internal web server.

--port <number> Port used for the internal web server. If set to 0 (default), an available port is automatically selected.

--server-only Only starts the web server, and configure it for the test campaign but do not start the campaign. This is useful to run tests manually.

--max-task-restarts <number> Maximum number of times a task can be restarted after being interrupted by a browser disconnection (default: 5).

--task-restart-on-failure When enabled, failing tasks are also automatically restarted as if the browser was disconnected. This option is especially useful for test campaigns which are not very stable, containing tests which can randomly fail (even if, of course, having randomly failing tests in a test suite is not recommended!).

--log-level <number> Level of logging: integer from 0 (no logging) to 5 (trace).

--colors Uses colors (disable with --no-colors).

--help Displays a help message and exits.

--version Displays the version number and exits.

Advanced options

--json-console When enabled, JSON objects will be sent to stdout to provide information about tests. This is used by the junit bridge.

--heartbeats Delay (in ms) for heartbeats messages sent when --json-console is enabled. Use 0 to disable them.

--task-timeout Maximum duration (in ms) for a test task.

--shutdown-on-campaign-end <boolean> When set to false, will not shut the server down nor exit the process upon finishing the campaign. Might be useful for debugging.

--predictable-urls <boolean> If true, resources served by the campaign have predictable URLs (campaign1, campaign2...). Otherwise, the campaign part in the URL is campaign ID.

Configuration file options

Any option configurable through the configuration file can also be specified from the command line with the --config prefix. For example, to configure resources, it is possible to use:

attester --config.resources./ path1/root1 --config.resources./ path2/root2 --config.resources./aria path/to/aria

Test Page

It's important to understand exactly what is served by attester to a connected browser.

A slave browser receives a page with a single test and the files necessary to run it. The test page has access to anything that is listed in resources.

From the configuration file tests is used to build a list of tests for each testing framework resources simply defines what is accessible by the page, none of these files is automatically added to the page

Inside test there are two parameters that deal with files

• files used to generate the list of test files
• extraScripts resources to be included in the test page

It's worth noticing that extraScripts must be an absolute path from the server root (what is configured in the resources).

Let's see one example

{
  "resources" : {
    "/" : ["src"],
    "/test" : ["spec"]
  }
}

The block above configures attester to serve any file inside src folder from the server root, and files from spec from /test.

{
  "resources" : {
    "/" : ["src"],
    "/test" : ["spec"]
  },
  "tests" : {
    "mocha" : {
      "files" : {
        "includes" : ["spec/**/*.js"]
      }
    }
  }
}

This block tells attester that each JavaScript file in the spec folder is a mocha test. When dispatching them to connected browsers, each will receive a page containing

• mocha
• an attester adapter to mocha tests
• a single test file

Including source files it's up to the test that can use script loaders like noder, curl, head.js or similar. Using a script loader allows to include in the tested page only what's strictly necessary for the test.

It's also possible to configure attester to include other scripts then the test through extraScripts.

{
  "resources": {
    "/": ["src"],
    "/test": ["spec"]
  },
  "tests": {
    "mocha": {
      "files": {
        "includes": ["spec/**/*.js"]
      },
      "extraScripts": ["/sourceCode.js", "/utilities.js"]
    }
  }
}

The paths specified in extraScripts are resources URL. In the example above sourceCode.js must be physically located inside src folder.

The files specified in extraScripts are included in each and every test.