cypress-multithreaded-runner

cypress-multithreaded-runner

A lightweight method to run Cypress spec files in multiple threads, with built-in support for Allure report generation.

When used out of the box, Cypress will only run your spec files in a single thread. It also won't create any kind of easy-to-read report once the tests have completed. By making use of this module, you'll be able to adapt your Cypress project to run much faster than before, while also generating a comprehensive Allure report for every run.

NOTE: Currently, generating the Allure report is needed in order for this module to function correctly, but this may change later.

Dependencies

• node 18 or above
• A Cypress project (tested on versions 12.10 and 13.7)
• pip (optional: only needed if you wish to combine the Allure report into a single file)
• @cypress/grep (optional: only needed if you want to use Cypress grep features)

Setup

In its current form, you can import the runner into a simple node application, initialising it like so:

const runner = require("cypress-multithreaded-runner");

const cypressConfigOverride = {
  reportDir: "put-the-reports-here",
};

runner({
  phaseDefaults: {
    cypressConfig: {
      filepath: "src/cypress/configurations/my-generic-cypress.config.js",
      object: cypressConfigOverride,
    },
    grep: "my-test",
    grepTags: "my-tag",
  },
  phases: [
    {
      specsDir: "src/cypress/tests/phase1",
    },
    {
      specsDir: "src/cypress/tests/phase2",
    },
  ],
  allureReportDir: "i-would-rather-save-the-allure-report-here",
  reportDir: cypressConfigOverride.reportDir,
  waitForFileExist: {
    filepath: "a-file-that-I-expect-to-exist-before-subsequent-threads-run.txt",
    timeout: 60,
    deleteAfterCompletion: true,
  },
});

By default, every spec file within each phase's specsDir will spawn a unique instance of Cypress. Once everything's been tested, the Allure report will be generated.

A benchmark file will also be generated. This will dictate the optimal order in which the threads should spawn in future. Some identifiers from your config are used to determine which order should be used. That way, you can run your Cypress tests with different arguments (for example, different grep arguments) and the results will not interfere with other benchmarks.

Should tests in any phase fail, all threads from subsequent phases will stop immediately. If you don't wish this to happen, you'll want to just use one phase.

Generating the Allure report

To generate Allure reports, you will need to add the following import to your support file:

import "cypress-multithreaded-runner/allure";

In addition, add the following import to your config file:

const allureWriter = require("cypress-multithreaded-runner/allure/writer");

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      allureWriter(on, config);
      return config;
    },
  },
});

The imports above are simple wrappers for @mmisty/cypress-allure-adapter, so refer to that module's readme should you wish to customise the plugin's behaviour.

If either of the imports are missing, you may find that an empty Allure report is generated after running tests.

Optional: Grep

If you want to make use of @cypress/grep, add it as a dependency to your project and follow the official readme to see how to import it. Refer to the options table to see what features are supported by cypress-multithreaded-runner out of the box.

Using grep in conjunction with onlyRunSpecFilesIncludingAnyText and onlyRunSpecFilesIncludingAllText

Grep can be a little slow if you have a lot of spec files, as Cypress will still try to run each of them before evaluating whether they satisfy the grep rules.

To save time, you may want to use utilise onlyRunSpecFilesIncludingAnyText and/or onlyRunSpecFilesIncludingAllText in addition to grep. These properties will ensure that spec files will only be processed if the given string(s) are found within them. Unlike grep, the filtering will occur before Cypress is launched, which in practice will mean you can expect a significant speed boost in many scenarios. However, as it's only checking for the presence of strings, you shouldn't pass through advanced grep rules to onlyRunSpecFilesIncludingAnyText and/or onlyRunSpecFilesIncludingAllText.

While it may be a common use case, you don't need to use grep in order for onlyRunSpecFilesIncludingAnyText or onlyRunSpecFilesIncludingAllText to function. The values you assign to these properties do not need to be the same as the ones you give to grep or grepTags, as it works completely independently.

For basic grep rules, the following example may represent a common use case:

const runner = require("cypress-multithreaded-runner");

const cypressConfigOverride = {
  reportDir: "put-the-reports-here",
};

runner({
  cypressConfig: {
    filepath: "src/cypress/configurations/my-generic-cypress.config.js",
    object: cypressConfigOverride,
  },
  reportDir: cypressConfigOverride.reportDir,
  specsDir: "src/cypress/tests",
  grep: "my-test",
  grepTags: "my-tag",
  onlyRunSpecFilesIncludingAllText: ["my-test", "my-tag"],
  ignoreCliOverrides: ["grep", "grepTags"],
});

Optional: Prevent the parent process from ending when tests fail

By default, this module will end the parent process (with exit code 1) should the tests complete with one or more failures. Bypassing this will therefore allow your app to continue running every time the tests complete. You can set endProcessIfTestsFail to false to achieve this.

Get the exit code

Unless an error unrelated to one of your tests occurs, you can get the aforementioned exit code by running this module in async mode. To achieve this, import and use the module like so:

const asyncRunner = require("cypress-multithreaded-runner/async"); // different import

const cypressConfigOverride = {
  reportDir: "put-the-reports-here",
};

(async () => {
  const exitCode = await runner({
    cypressConfig: {
      filepath: "src/cypress/configurations/my-generic-cypress.config.js",
      object: cypressConfigOverride,
    },
    reportDir: cypressConfigOverride.reportDir,
    specsDir: "src/cypress/tests",
    endProcessIfTestsFail: false, // if this isn't set to false, your app will stop running if any tests fail!
  });

  console.log("exitCode: ", exitCode);
})();

The exit code will be 0 if all tests pass and 1 if any test fails.

General guidance

Pay attention to the performance of each thread when adding a new test. It is a good idea to try making the accumulation of tests in each thread take roughly the same amount of time to finish running, as any thread that takes significantly longer than the others will be a bottleneck! Therefore, try to add any new tests to threads which aren't oversaturated. You can have a look at how each thread is performing by viewing the "Thread Performance Summary" & other logs, all of which will be added to the bottom of the Allure report as well as in separate text files.

Diminishing returns will be observable the more threads you add, as most computers have a relatively low number of threads that can be actively used at any given time. When too many threads are added you may notice a large increase in the frequency of failing tests, and you may also actually notice that other threads are just running slower in general. It's all a bit of a balancing act if you want to get it all just right.

None of the Cypress threads can "talk" to each other, so your project may need a bit of extra configuration if this is needed. For example, let's say your first thread logs in to a website and you'd like this session to be maintained across your other threads. You could have the first Cypress thread log in and then save a file containing all of the cookies that the other threads need. You can make use of the waitForFileExist option to help make such functionality possible for your project. cypress-wordpress-session is an example of a Cypress addon that can be used to create & load a cookies file for a persistent session across multiple threads.

Overriding options in the command line

The module makes use of argv to allow options to be overriden via the command line. Arguments passed in via the command line will take precedence over any options set via your node application. For example, let's say you want to override the reportDir option. You can run your node application with a standard kebab-case argument (eg. --report-dir="my-report-dir") and the reportDir will be overriden.

Options

Name	Type	Default value	Description
reportDir	string	null	The location to save the full report to. This currently needs to be the same directory that's passed through to the Cypress instances.
allureReportDir	string	<reportDir>/allure-report	The location to save just the Allure report
allureReportHeading	string	null	A custom heading to add to the top of the Allure report
generateAllure	boolean	true	Generate the Allure report. Overrides the openAllure, combineAllure & hostAllure options. NOTE: It's not currently possible to completely disable this feature. For now, setting this to false will generate only what's required for this module to function correctly.
openAllure	boolean	false	Open the Allure report after it's generated. If combineAllure is also passed in, it'll open the combined Allure report instead. If hostAllure is also passed in, openAllure will be ignored.
open	boolean	false	Alias for openAllure
combineAllure	boolean	false	Combine the Allure report into a single file (complete.html). Requires pip to be installed.
combine	boolean	false	Alias for combineAllure
hostAllure	boolean	false	Spin up a localhost for the Allure report after it's generated
host	boolean	false	Alias for hostAllure
ignoreCliOverrides	arrayOf(string)	null	A list of keys of properties that you don't want the CLI to override when you run an instance of cypress-multithreaded-runner. This will enable your node app to do a custom override of that uses a combination of the CLI and itself. See here for an example
specFiles	arrayOf(string)	null	An array of one or more spec files to filter. Only spec files present in the array will be tested. This will apply to every phase, so you may find some phases are skipped entirely. The files must all exist within the given specsDir directory.
logMode	oneOf([1,2,3,4])	1	The method by which you want each Cypress thread to print logs to the console. 1: Only print logs from one thread at a time, in order of when each thread began. Only when the first thread completes will the logs from the second thread be printed, then the third and so on. 2: Only print logs from one thread at a time, but allow non-chronology. Should any other thread complete before the current one does, these logs will be "queued" and then print when the current one completes. For example, let's say thread 3 completes before thread 1 completes, with thread 2 completing at the end. In this scenario, you can expect to see all of the output for thread 1, then all of thread 3, then thread 2. 3: Print any log from any thread as soon as it manifests. This will likely mean that you see a mix of logs from each thread, and can be difficult to understand the continuity in the logs the more threads you have. 4: Identical to mode 3, except that when all threads complete, print all logs again but separate out all of the threads.
waitForFileExist	object	null	Wait for a specific file to exist (and larger than 0 bytes in size) before subsequent threads begin. For specific options, see the table below.
maxThreadRestarts	number	5	Should an instance of Cypress crash, it may be restarted up to this many times until all threads complete successfully. Note that any spec file that fails in a beforeEach hook will be considered a crash. This behaviour may be amended in a future version of this module.
maxConcurrentThreads	number	Half the number of available threads	The maximum number of threads that'll run at any one time. By default, this will be the number of threads the client's CPU is reporting divided by two. It's recommended not to use more threads than a CPU can provide, otherwise performance is likely to be negatively affected. If not enough threads are available, cypress-multithreaded-runner will wait for any of the currently running Cypress instances to complete before starting another one. This process will repeat until all tests are completed.
threadDelay	number	30	The amount seconds to wait before starting the next thread, unless the current Cypress instance has already started running. If waitForFileExist has been set, the 2nd thread will continue waiting until the given file exists. For more info, see the table below.
threadMode	oneOf([1,2])	1	The method by which you want each Cypress thread to be scheduled logs to the console. 1: Every spec file runs in its own Cypress instance. This is recommended for most projects. If there's a lot of variance in how fast each spec file takes to run, then this is the best solution for you. 2: A thread will be created for every top-level directory within your specsDir. For example, 8 top-level folders will be 8 threads. There's a small time cost in spinning up a new Cypress instance, so if you want to spend time manually shuffling the spec files around to balance each thread, you may find a performance gain using this instead of mode 1. NOTE: If you make use of the specFiles option, threadMode will always be set to mode 1.
onlyPhaseNo	number	null	Set this to the value of any of the phases to run just the threads within that phase. Phases count from a value of 1, not 0.
startingPhaseNo	number	null	Set this to the value of any of the phases to run just the threads within and after that phase. Phases count from a value of 1, not 0.
endingPhaseNo	number	null	Set this to the value of any of the phases to run just the threads before and within that phase. Phases count from a value of 1, not 0.
threadInactivityTimeout	number	600	The maximum amount of seconds to wait for a thread to respond before it's considered a crash. Every time any thread logs something, the timeout will be reset. Set to 0 to have no timeout at all.
threadTimeLimit	number	1800	The maximum amount of seconds to wait for a thread to complete. This is the total allocated time per thread. It won't reset should the thread restart due to errors. Set to 0 to have no time limit at all.
orderThreadsByBenchmark	boolean	true	Honour the order of the threads as set in the thread benchmark file. If no such file exists, it'll fallback to alphabetical order. Any threads that don't exist in the benchmark file will run at the start of the phase.
saveThreadBenchmark	boolean	true	Update the thread benchmark file with a new order for the threads, with the slowest thread at the start and the fastest thread at the end.
threadBenchmarkFilepath	string	cmr-benchmarks.json	The file where the thread benchmark will be saved.
benchmarkDescription	string	null	An optional string to add to the benchmark such that its identifier is easily understandable. The value of this string will also be encoded into the identifier. For example, if you run your suite of tests two times and the only argument that's different is the benchmarkDescription, the results from the first run won't overwrite the second.
phaseDefaults	object	null	Default properties you wish to set for every object in the phases array. For more info, see table below.
phases	arrayOf(object)	null	Phases of Cypress test threads. Any phase can have any number of threads. Every object will override equivalent property keys set in phaseDefaults. Should tests in any phase fail, all threads from subsequent phases will stop immediately. For example, you may want to run high priority tests first, then medium priority, then low priority. If the high priority tests fail, the medium & low priority tests will stop running. For more info, see table below.
repeat	number	1	The number of times all phases should repeat. Make use of this to stress test your system and identify any unstable tests.
endProcessIfTestsFail	boolean	true	When set to true, the parent process will stop running (with exit code 1) if any test fails.
maxConcurrentThreadsExperiment	object	null	An experimental feature which can be used to determine the optimum number of threads that can be used to run Cypress tests on your machine. It does this by running all of your tests with a different value set for maxConcurrentThreads and then comparing the time taken for each setting. For specific options, see table below. NOTE: This feature will not work if the module is run in the async mode.

phases

Name	Type	Default value	Description
cypressConfig	object	null	The config that'll be passed through to every Cypress instance. For specific options, see the table below.
specsDir	string	null	The top-level directory containing all Cypress spec files are
browser	string	null	The web browser you want the spec files to use
grep	string	null	grep arg to be passed through as an environment variable to each Cypress instance. See here for more information.
grepTags	string	null	grepTags arg to be passed through as an environment variable to each Cypress instance. See here for more information.
grepUntagged	boolean	false	grepUntagged arg to be passed through as an environment variable to each Cypress instance. See here for more information.
onlyRunSpecFilesIncludingAnyText	oneOfType(string, arrayOf(string))	null	Only run spec files in each Cypress thread if ANY of the strings in the given array are found within the spec file. If a single string is provided instead of an array, the entire string must be found. Case insensitive. See here for more information.
onlyRunSpecFilesIncludingAllText	oneOfType(string, arrayOf(string))	null	Only run spec files in each Cypress thread if ALL of the strings in the given array are found within the spec file. Case insensitive. See here for more information.
passthroughEnvArgs	string	null	Additional Cypress environment arguments to be passed through to each Cypress instance. No preprocessing will be done to this string

cypressConfig

Name	Type	Default value	Description
filepath	string	null	A config file to be passed through to every Cypress instance
object	object	null	A config object to be passed through to every Cypress instance. Any overlapping options will override those set via the filepath

waitForFileExist

Name	Type	Default value	Description
filepath	string	null	The file you wish all threads except the first one to wait for
minSize	number	2	The minimum acceptable size (in bytes) for the file. Default is therefore 2 bytes. A null file will be 0 bytes. If the file exists but has a size smaller than this value, it'll be treated as if it doesn't exist.
timeout	number	60	The maximum amount of seconds to wait for the file to exist. The threads will begin when the time elapses, regardless of whether the file exists or not.
deleteAfterCompletion	boolean	false	Delete the file once all threads have completed
stopWaitingWhenFirstThreadCompletes	boolean	true	When set to true (recommended), subsequent threads will start running when the first thread completes, even if the file doesn't exist.

maxConcurrentThreadsExperiment

Name	Type	Default value	Description
maxConcurrentThreadsArray	arrayOf(number)	null	An array of values you wish to set & compare for maxConcurrentThreads
repeat	number	3	The number of times to repeat each experiment