Monocart Coverage Reports
Code coverage tool to generate native V8 reports or Istanbul reports.
Usage
const MCR = require('monocart-coverage-reports');
const coverageOptions = {
name: 'My Coverage Report - 2024-02-28',
outputDir: './coverage-reports',
reports: ["v8", "console-details"]
}
const coverageReport = MCR(coverageOptions);
coverageReport.cleanCache();
await coverageReport.add(coverageData1);
await coverageReport.add(coverageData2);
await coverageReport.generate();
Default Options
Available Reports
V8 build-in reports (V8 data only):
Istanbul build-in reports (both V8 and istanbul data):
clover
cobertura
html
html-spa
json
json-summary
lcov
lcovonly
none
teamcity
text
text-lcov
text-summary
Other build-in reports (both V8 and istanbul data):
console-details
Show file coverage and uncovered lines in the console. Like text
, but for V8. For Github actions, we can enforce color with env: FORCE_COLOR: true
.
-
raw
only keep all original data, which can be used for other reports input with inputDir
-
Custom Reporter
{
reports: [
[path.resolve('./test/custom-istanbul-reporter.js'), {
type: 'istanbul',
file: 'custom-istanbul-coverage.text'
}],
[path.resolve('./test/custom-v8-reporter.js'), {
type: 'v8',
outputFile: 'custom-v8-coverage.json'
}],
[path.resolve('./test/custom-v8-reporter.mjs'), {
type: 'both'
}]
]
}
example: ./test/custom-istanbul-reporter.js, see istanbul built-in reporters' implementation for reference.
example: ./test/custom-v8-reporter.js
Multiple Reports:
const MCR = require('monocart-coverage-reports');
const coverageOptions = {
outputDir: './coverage-reports',
reports: [
['console-summary'],
['v8'],
['html', {
subdir: 'istanbul'
}],
['json', {
file: 'my-json-file.json'
}],
'lcovonly',
["custom-reporter-1"],
["custom-reporter-2", {
type: "istanbul",
key: "value"
}],
['/absolute/path/to/custom-reporter.js']
]
}
const coverageReport = MCR(coverageOptions);
coverageReport.cleanCache();
Using entryFilter
and sourceFilter
to filter the results for V8 report
When V8 coverage data collected, it actually contains the data of all entry files, for example:
dist/main.js
dist/vendor.js
dist/something-else.js
We can use entryFilter
to filter the entry files. For example, we should remove vendor.js
and something-else.js
if they are not in our coverage scope.
dist/main.js
When inline or linked sourcemap exists to the entry file, the source files will be extracted from the sourcemap for the entry file, and the entry file will be removed if logging
is not debug
.
> src/index.js
> src/components/app.js
> node_modules/dependency/dist/dependency.js
We can use sourceFilter
to filter the source files. For example, we should remove dependency.js
if it is not in our coverage scope.
> src/index.js
> src/components/app.js
For example:
const coverageOptions = {
entryFilter: (entry) => entry.url.indexOf("main.js") !== -1,
sourceFilter: (sourcePath) => sourcePath.search(/src\//) !== -1
};
Or using minimatch
pattern:
const coverageOptions = {
entryFilter: "**/main.js",
sourceFilter: "**/src/**"
};
const coverageOptions = {
entryFilter: {
'**/vendor.js': false,
'**/main.js': true
},
sourceFilter: {
'**/src/**': true
}
};
onEnd Hook
For example, checking thresholds:
const EC = require('eight-colors');
const coverageOptions = {
name: 'My Coverage Report',
outputDir: './coverage-reports',
onEnd: (coverageResults) => {
const thresholds = {
bytes: 80,
lines: 60
};
console.log('check thresholds ...', thresholds);
const errors = [];
const { summary } = coverageResults;
Object.keys(thresholds).forEach((k) => {
const pct = summary[k].pct;
if (pct < thresholds[k]) {
errors.push(`Coverage threshold for ${k} (${pct} %) not met: ${thresholds[k]} %`);
}
});
if (errors.length) {
const errMsg = errors.join('\n');
console.log(EC.red(errMsg));
}
}
}
mcr
CLI
The CLI will run the program as a child process with NODE_V8_COVERAGE=dir
until it exits gracefully, and generate the coverage report with the coverage data from the dir
.
npm i monocart-coverage-reports -g
mcr node ./test/test-node-env.js -r v8,console-summary --lcov
npm i monocart-coverage-reports
npx mcr node ./test/test-node-env.js -r v8,console-summary --lcov
Usage: mcr [options] <command>
CLI to generate coverage reports
Arguments:
command command to execute
Options:
-V, --version output the version number
-c, --config <path> custom config path
-o, --outputDir <dir> output dir for reports
-r, --reports <name[,name]> coverage reports to use
-n, --name <name> report name for title
-i, --inputDir <dir> input dir for merging raw files
--entryFilter <pattern> entry url filter
--sourceFilter <pattern> source path filter
--outputFile <path> output file for v8 report
--inline inline html for v8 report
--assetsPath <path> assets path if not inline
--lcov generate lcov.info file
--logging <logging> off, error, info, debug
-h, --help display help for command
- Loading default configuration file
.mcrrc
mcr.config.json
mcr.config.mjs
mcr.config.cjs
mcr.config.js
- Or loading custom configuration file with
-c
or --config
mcr node ./test.js -c path-to/my-custom-config.js
Compare Reports
Compare Workflows
-
Istanbul Workflows
-
V8 Workflows
Collecting Istanbul Coverage Data
- Instrumenting source code
Before collecting Istanbul coverage data, It requires your source code is instrumented with Istanbul
- Browser
- Node.js
- Collecting coverage data from
global.__coverage__
Collecting V8 Coverage Data
- For source code: enable
sourcemap
and do not compress/minify:
- Browser (Chromium Only)
Collecting coverage data with Chromium Coverage API:
- Node.js
Manually Resolve the Sourcemap
Sometimes, the sourcemap file cannot be successfully loaded with the sourceMappingURL
, you can try to manually read the sourcemap file before the coverage data is added to the report.
const jsCoverage = await page.coverage.stopJSCoverage();
jsCoverage.forEach((entry) => {
if (entry.url.endsWith('my-dist.js')) {
entry.sourceMap = JSON.parse(fs.readFileSync('dist/my-dist.js.map').toString('utf-8'));
}
});
await MCR(coverageOptions).add(jsCoverage);
Collecting Raw V8 Coverage Data with Puppeteer
Puppeteer does not provide raw v8 coverage data by default. A simple conversion is required, see example: ./test/test-puppeteer.js
await Promise.all([
page.coverage.startJSCoverage({
resetOnNavigation: false,
includeRawScriptCoverage: true
}),
page.coverage.startCSSCoverage({
resetOnNavigation: false
})
]);
await page.goto(url);
const [jsCoverage, cssCoverage] = await Promise.all([
page.coverage.stopJSCoverage(),
page.coverage.stopCSSCoverage()
]);
const coverageData = [... jsCoverage.map((it) => {
return {
source: it.text,
... it.rawScriptCoverage
};
}), ... cssCoverage];
Node.js V8 Coverage Report for Server Side
Possible solutions:
-
NODE_V8_COVERAGE=dir
- Sets Node.js env
NODE_V8_COVERAGE
=dir
before the program running, the coverage data will be saved to the dir
after the program exits gracefully. - Read the JSON file(s) from the
dir
and generate coverage report. - Example:
cross-env NODE_V8_COVERAGE=.temp/v8-coverage-env
node ./test/test-node-env.js && node ./test/generate-report.js
-
V8 API + NODE_V8_COVERAGE
- Writing the coverage started by NODE_V8_COVERAGE to disk on demand with
v8.takeCoverage()
, it does not require waiting until the program exits gracefully. - Example:
cross-env NODE_V8_COVERAGE=.temp/v8-coverage-api
node ./test/test-node-api.js
-
Inspector API
- Connecting to the V8 inspector and enable V8 coverage.
- Taking coverage data and adding it to the report.
- Example:
node ./test/test-node-ins.js
-
CDP API
- Enabling Node Debugging.
- Collecting coverage data with CDP API.
- Example:
node --inspect=9229 ./test/test-node-cdp.js
-
Node Debugging + CDP + NODE_V8_COVERAGE + V8 API
- When the program starts a server, it will not exit on its own, thus requiring a manual invocation of the
v8.takeCoverage()
interface to manually collect coverage data. Remote invocation of the v8.takeCoverage()
interface can be accomplished through the Runtime.evaluate
of the CDP. - Example for koa web server:
node ./test/test-node-koa.js
-
Child Process + NODE_V8_COVERAGE
Multiprocessing Support
The data will be added to [outputDir]/.cache
, After the generation of the report, this data will be removed unless debugging has been enabled or a raw report has been used, see Debug for Coverage and Sourcemap
- Main process, before the start of testing
const MCR = require('monocart-coverage-reports');
const coverageOptions = require('path-to/same-options.js');
const coverageReport = MCR(coverageOptions);
coverageReport.cleanCache();
- Sub process 1, testing stage 1
const MCR = require('monocart-coverage-reports');
const coverageOptions = require('path-to/same-options.js');
const coverageReport = MCR(coverageOptions);
await coverageReport.add(coverageData1);
- Sub process 2, testing stage 2
const MCR = require('monocart-coverage-reports');
const coverageOptions = require('path-to/same-options.js');
const coverageReport = MCR(coverageOptions);
await coverageReport.add(coverageData2);
- Main process, after the completion of testing
const MCR = require('monocart-coverage-reports');
const coverageOptions = require('path-to/same-options.js');
const coverageReport = MCR(coverageOptions);
await coverageReport.generate();
Merge Coverage Reports
The following usage scenarios may require merging coverage reports:
- When the code is executed in different environments, like Node.js Server Side and browser Client Side (Next.js for instance). Each environment may generate its own coverage report. Merging them can give a more comprehensive view of the test coverage. see example nextjs-with-playwright for automatic report merging.
- When the code is subjected to different kinds of testing. For example, unit tests with Jest might cover certain parts of the code, while end-to-end tests with Playwright might cover other parts. Merging these different coverage reports can provide a holistic view of what code has been tested.
- When tests are run on different machines or different shards, each might produce its own coverage report. Merging these can give a complete picture of the test coverage across all machines or shards.
If the reports cannot be merged automatically, then here is how to manually merge the reports.
First, using the raw
report to export the original coverage data to the specified directory.
const coverageOptions = {
name: 'My Unit Test Coverage Report',
outputDir: "./coverage-reports/unit",
reports: [
['raw', {
outputDir: "raw"
}],
['v8'],
['console-summary']
]
};
Then, after all the tests are completed, generate a merged report with option inputDir
:
import fs from "fs";
import { CoverageReport } from 'monocart-coverage-reports';
const coverageOptions = {
name: 'My Merged Coverage Report',
inputDir: [
'./coverage-reports/unit/raw',
'./coverage-reports/e2e/raw'
],
outputDir: './coverage-reports/merged',
reports: [
['v8'],
['console-summary']
],
onEnd: () => {
fs.rmSync('./coverage-reports/unit/raw', {
recursive: true,
force: true
})
}
};
await new CoverageReport(coverageOptions).generate();
Resolve sourcePath
for the Source Files
If the source file comes from the sourcemap, then its path is a virtual path. Using the sourcePath
option to resolve a custom path.
For example, we have tested multiple dist files, which contain some common files. We hope to merge the coverage of the same files, so we need to unify the sourcePath
in order to be able to merge the coverage data.
const coverageOptions = {
sourcePath: (filePath) => {
const list = ['my-dist-file1/', 'my-dist-file2/'];
for (const str of list) {
if (filePath.startsWith(str)) {
return filePath.slice(str.length);
}
}
return filePath;
}
};
It also supports simple key/value replacement:
const coverageOptions = {
sourcePath: {
'my-dist-file1/': '',
'my-dist-file2/': ''
}
};
Adding Empty Coverage for Untested Files
By default the untested files will not be included in the coverage report, we can add empty coverage data for all files with option all
, the untested files will show 0% coverage.
const coverageOptions = {
all: {
dir: ['./src'],
filter: (filePath) => {
return true;
}
}
};
The filter also supports minimatch
pattern:
const coverageOptions = {
all: {
dir: ['./src'],
filter: '**/*.js'
}
};
const coverageOptions = {
all: {
dir: ['./src'],
filter: {
'**/ignored-*.js': false,
'**/*.html': false,
'**/*.ts': false,
'**/*.scss': "css",
'**/*': true
}
}
};
Ignoring Uncovered Codes
To ignore codes, use the special comment which starts with v8 ignore
:
function uncovered() {
}
- Ignoring the next line or next N lines
const os = platform === 'wind32' ? 'Windows' : 'Other';
const os = platform === 'wind32' ? 'Windows' : 'Other';
if (platform === 'linux') {
console.log('hello linux');
}
Chromium Coverage API
V8 Coverage Data Format
export interface CoverageRange {
startOffset: integer;
endOffset: integer;
count: integer;
}
export interface FunctionCoverage {
functionName: string;
ranges: CoverageRange[];
isBlockCoverage: boolean;
}
export interface ScriptCoverage {
scriptId: Runtime.ScriptId;
url: string;
functions: FunctionCoverage[];
}
export type V8CoverageData = ScriptCoverage[];
see devtools-protocol ScriptCoverage and v8-coverage
How to convert V8 to Istanbul
It is a popular library which is used to convert V8 coverage format to istanbul's coverage format. Most test frameworks are using it, such as Jest, Vitest, but it has two major problems:
- 1, The source mapping does not work well if the position is between the two consecutive mappings. for example:
const a = tf ? 'true' : 'false';
^ ^ ^
m1 p m2
m1
and m2
are two consecutive mappings, p
is the position we looking for. However, we can only get the position of the m1
or m2
if we don't fix it to p
. Especially the generated code is different from the original code, such as the code was minified, compressed or converted, it is difficult to find the exact position.
- 2, The coverage of functions and branches is incorrect. V8 only provided coverage at functions and it's blocks. But if a function is uncovered (count = 0), there is no information for it's blocks and sub-level functions. And also there are some problems about counting the functions and branches.
How Monocart Works
We implemented new converter:
- 1, Trying to fix the middle position if not found the exact mapping for the position.
- 2, Finding all functions, statements and branches by parsing the source code AST. However, there's a small issue, which is the V8 cannot provide effective branch coverage information for
AssignmentPattern
.
AST | V8 |
---|
AssignmentPattern | 🛇 Not Support |
ConditionalExpression | ✔ |
IfStatement | ✔ |
LogicalExpression | ✔ |
SwitchStatement | ✔ |
Debug for Coverage and Sourcemap
Sometimes, the coverage is not what we expect. The next step is to figure out why, and we can easily find out the answer step by step through debugging.
- Start debugging for v8 report with option
logging: 'debug'
const coverageOptions = {
logging: 'debug',
reports: [
['v8'],
['console-summary']
]
};
When logging
is debug
, the raw report data will be preserved in [outputDir]/.cache
or [outputDir]/raw
if raw
report is used. And the dist file will be preserved in the V8 list, and by opening the browser's devtool, it makes data verification visualization effortless.
Integration
const coverageOptions = {
outputDir: "./coverage-reports",
reports: [
['codecov']
]
};
- name: Codecov
uses: codecov/codecov-action@v3
with:
files: ./coverage-reports/codecov.json
const coverageOptions = {
outputDir: "./coverage-reports",
lcov: true
};
- name: Coveralls
uses: coverallsapp/github-action@v2
with:
files: ./coverage-reports/lcov.info
- Using
lcov
report. Github actions example:
- name: Analyze with SonarCloud
uses: sonarsource/sonarcloud-github-action@master
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
with:
projectBaseDir: ./
args: >
-Dsonar.organization=cenfun
-Dsonar.projectKey=monocart-coverage-reports
-Dsonar.projectName=monocart-coverage-reports
-Dsonar.javascript.lcov.reportPaths=docs/mcr/lcov.info
-Dsonar.sources=lib
-Dsonar.tests=test
-Dsonar.exclusions=dist/*,packages/*
Integration with Any Testing Framework
- Collecting coverage data when any stage of the test is completed, and adding the coverage data to the coverage reporter.
- Generating the coverage reports after the completion of all tests.
- see Multiprocessing Support
VSCode Extension
- Coverage Gutters - Display test coverage generated by lcov or xml in VSCode editor.
Thanks