monocart-coverage-reports

Monocart Coverage Reports

Code coverage tool to generate native V8 reports or Istanbul reports.

• Usage
• Default Options
• Available Reports
• Using entryFilter and sourceFilter to filter the results for V8 report
• onEnd Hook
• mcr CLI
• Compare Reports
• Compare Workflows
• Collecting Istanbul Coverage Data
• Collecting V8 Coverage Data
• Node.js V8 Coverage Report for Server Side
• Multiprocessing Support
• Integration
• Ignoring Uncovered Codes
• Chromium Coverage API
• V8 Coverage Data Format
• How to convert V8 to Istanbul
  • Using v8-to-istanbul
  • How Monocart Works
• Istanbul Introduction
• Thanks

Usage

const MCR = require('monocart-coverage-reports');
const options = {
    outputDir: './coverage-reports',
    reports: "v8"
}
const coverageReport = MCR(options);
await coverageReport.add(coverageData1);
await coverageReport.add(coverageData2);
const coverageResults = await coverageReport.generate();
console.log(coverageResults.summary);

• example v8
• example istanbul

Default Options

• lib/default/options.js

Available Reports

• v8 (V8 data only)
  • Test in browser
    • Build with webpack: V8 and V8 Minify
    • Build with Rollup and Esbuild
    • Collect with puppeteer
    • anonymous and css
  • Test in Node.js
    • Collect with env, and also V8 API, Inspector and CDP
    • Web server example: koa

• v8-json (V8 data only)

  • V8 coverage-report.json

• console-summary shows coverage summary in console

  

Following are istanbul reports

• clover
• cobertura
• html
  • Istanbul html
  • V8 to Istanbul
• html-spa
  • Istanbul html-spa
• json
• json-summary
• lcov
• lcovonly
  • V8 lcov.info
  • Istanbul lcov.info
• none
• teamcity
• text
• text-lcov
• text-summary

const MCR = require('monocart-coverage-reports');
const options = {
    outputDir: './coverage-reports',
    reports: [
        ['console-summary'],
        ['v8'],
        ['html', {
            subdir: 'istanbul'
        }],
        ['json', {
            file: 'my-json-file.json'
        }],
        'lcovonly'
    ]
}
const coverageReport = MCR(options);

Using entryFilter and sourceFilter to filter the results for V8 report

When V8 coverage data collected, it actually contains the data of all entry files, for example:

dist/main.js
dist/vendor.js
dist/something-else.js

We can use entryFilter to filter the entry files. For example, we should remove vendor.js and something-else.js if they are not in our coverage scope.

dist/main.js

When inline or linked sourcemap exists to the entry file, the source files will be extracted from the sourcemap for the entry file, and the entry file will be removed if logging is not debug.

> src/index.js
> src/components/app.js
> node_modules/dependency/dist/dependency.js

We can use sourceFilter to filter the source files. For example, we should remove dependency.js if it is not in our coverage scope.

> src/index.js
> src/components/app.js

Example:

const coverageOptions = {
    entryFilter: (entry) => entry.url.indexOf("main.js") !== -1,
    sourceFilter: (sourcePath) => sourcePath.search(/src\//) !== -1
};

onEnd Hook

For example, checking thresholds:

const EC = require('eight-colors');
const coverageOptions = {
    name: 'My Coverage Report',
    outputDir: './coverage-reports',
    onEnd: (coverageResults) => {
        const thresholds = {
            bytes: 80,
            lines: 60
        };
        console.log('check thresholds ...', thresholds);
        const errors = [];
        const { summary } = coverageResults;
        Object.keys(thresholds).forEach((k) => {
            const pct = summary[k].pct;
            if (pct < thresholds[k]) {
                errors.push(`Coverage threshold for ${k} (${pct} %) not met: ${thresholds[k]} %`);
            }
        });
        if (errors.length) {
            const errMsg = errors.join('\n');
            console.log(EC.red(errMsg));
            // throw new Error(errMsg);
            // process.exit(1);
        }
    }
}

mcr CLI

The CLI will run the program as a child process with NODE_V8_COVERAGE=dir until it exits gracefully, and generate the coverage report with the coverage data from the dir.

• Global mode

npm i monocart-coverage-reports -g
mcr "node ./test/test-node-env.js" -r v8,console-summary --lcov

• Current working directory mode

npm i monocart-coverage-reports
npx mcr "node ./test/test-node-env.js" -r v8,console-summary --lcov

• CLI Options

Usage: mcr [options] <command>

CLI to generate coverage reports

Arguments:
  command                      command to execute

Options:
  -V, --version                output the version number
  -c, --config <path>          config path for options
  -o, --outputDir <dir>        output dir for reports
  -r, --reports <name[,name]>  coverage reports to use
  -n, --name <name>            report name for title
  --outputFile <path>          output file for v8 report
  --inline                     inline html for v8 report
  --assetsPath <path>          assets path if not inline
  --lcov                       generate lcov.info file
  -h, --help                   display help for command

• Using configuration file for more options

mcr "node ./test/test-node-env.js" -c test/cli-options.js

Compare Reports

	Istanbul	V8	V8 to Istanbul
Coverage data	Istanbul (Object)	V8 (Array)	V8 (Array)
Output	Istanbul reports	V8 reports	Istanbul reports
- Bytes	❌	✅	❌
- Statements	✅	❌	☑️❔
- Branches	✅	☑️❔	☑️❔
- Functions	✅	✅	✅
- Lines	✅	✅	✅
- Execution counts	✅	✅	✅
CSS coverage	❌	✅	✅
Minified code	❌	✅	❌

❔ - Partial or conditional support

Compare Workflows

• Istanbul Workflows

  • 1, Collecting Istanbul coverage data
  • 2, Adding coverage data and generating coverage report

• V8 Workflows

  • 1, Collecting V8 coverage data
  • 3, Adding coverage data and generating coverage report

Collecting Istanbul Coverage Data

• Instrumenting source code

  Before collecting Istanbul coverage data, It requires your source code is instrumented with Istanbul

  • webpack: babel-plugin-istanbul, example: webpack.config-istanbul.js
  • rollup: rollup-plugin-istanbul
  • vite: vite-plugin-istanbul
• Browser
  • Collecting coverage data from window.__coverage__, example: test-istanbul.js
• Node.js
  • Collecting coverage data from global.__coverage__

Collecting V8 Coverage Data

• For source code: enable sourcemap and do not compress/minify:
  • webpack: devtool: source-map and mode: development, example webpack.config-v8.js
  • rollup: sourcemap: true
  • vite: sourcemap: true and minify: false
  • esbuild: sourcemap: true and minify: false
• Browser (Chromium Only)

  Collecting coverage data with Chromium Coverage API:

  • Playwright example, and anonymous, css
  • Puppeteer example
• Node.js
  • see following Node.js V8 Coverage Report for Server Side

Node.js V8 Coverage Report for Server Side

Possible solutions:

• NODE_V8_COVERAGE=dir

  • Sets Node.js env NODE_V8_COVERAGE=dir before the program running, the coverage data will be saved to the dir after the program exits gracefully.
  • Read the JSON file(s) from the dir and generate coverage report.
  • Example:

  cross-env NODE_V8_COVERAGE=.temp/v8-coverage-env node ./test/test-node-env.js && node ./test/generate-node-report.js

• V8 API + NODE_V8_COVERAGE

  • Writing the coverage started by NODE_V8_COVERAGE to disk on demand with v8.takeCoverage(), it does not require waiting until the program exits gracefully.
  • Example:

  cross-env NODE_V8_COVERAGE=.temp/v8-coverage-api node ./test/test-node-api.js

• Inspector API

  • Connecting to the V8 inspector and enable V8 coverage.
  • Taking coverage data and adding it to the report.
  • Example:

  node ./test/test-node-ins.js

• CDP API

  • Enabling Node Debugging.
  • Collecting coverage data with CDP API.
  • Example:

  node --inspect=9229 ./test/test-node-cdp.js

• Node Debugging + CDP + NODE_V8_COVERAGE + V8 API

  • When the program starts a server, it will not exit on its own, thus requiring a manual invocation of the v8.takeCoverage() interface to manually collect coverage data. Remote invocation of the v8.takeCoverage() interface can be accomplished through the Runtime.evaluate of the CDP.
  • Example for koa web server:

  node ./test/test-node-koa.js

• Child Process + NODE_V8_COVERAGE

  • see mcr CLI

Multiprocessing Support

The data will be added to [outputDir]/.cache, and the cache will be removed after reports generated.

• sub process 1

const MCR = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = MCR(options);
await coverageReport.add(coverageData1);

• sub process 2

const MCR = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = MCR(options);
await coverageReport.add(coverageData2);

• main process

// after all sub processes finished
const MCR = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = MCR(options);
const coverageResults = await coverageReport.generate();
console.log(coverageResults.summary);

Integration

• monocart-reporter - Test reporter for Playwright
• vitest-monocart-coverage - Integration with Vitest coverage

Ignoring Uncovered Codes

To ignore codes, use the special comment which starts with v8 ignore :

• Ignoring all until stop

/* v8 ignore start */
function uncovered() {
}
/* v8 ignore stop */

• Ignoring the next line or next N lines

/* v8 ignore next */
const os = platform === 'wind32' ? 'Windows' : 'Other';

const os = platform === 'wind32' ? 'Windows' /* v8 ignore next */ : 'Other';

// v8 ignore next 3
if (platform === 'linux') {
    console.info('hello linux');
}

Chromium Coverage API

• V8 coverage report - Native support for JavaScript code coverage to V8. (Chromium only)
• Playwright Coverage Class
• Puppeteer Coverage class
• DevTools Protocol for Coverage

V8 Coverage Data Format

// Coverage data for a source range.
export interface CoverageRange {
    // JavaScript script source offset for the range start.
    startOffset: integer;
    // JavaScript script source offset for the range end.
    endOffset: integer;
    // Collected execution count of the source range.
    count: integer;
}

// Coverage data for a JavaScript function.
/**
 * @functionName can be an empty string.
 * @ranges is always non-empty. The first range is called the "root range".
 * @isBlockCoverage indicates if the function has block coverage information.
    If this is false, it usually means that the functions was never called.
    It seems to be equivalent to ranges.length === 1 && ranges[0].count === 0.
*/
export interface FunctionCoverage {
    // JavaScript function name.
    functionName: string;
    // Source ranges inside the function with coverage data.
    ranges: CoverageRange[];
    // Whether coverage data for this function has block granularity.
    isBlockCoverage: boolean;
}

// Coverage data for a JavaScript script.
export interface ScriptCoverage {
    // JavaScript script id.
    scriptId: Runtime.ScriptId;
    // JavaScript script name or url.
    url: string;
    // Functions contained in the script that has coverage data.
    functions: FunctionCoverage[];
}

export type V8CoverageData = ScriptCoverage[];

see devtools-protocol ScriptCoverage and v8-coverage

Collect raw v8 coverage data with Puppeteer

await page.coverage.startJSCoverage({
    // provide raw v8 coverage data
    includeRawScriptCoverage: true
});

await page.goto(url);

const jsCoverage = await page.coverage.stopJSCoverage();
const rawV8CoverageData = jsCoverage.map((it) => {
    // Convert to raw v8 coverage format
    return {
        source: it.text,
        ... it.rawScriptCoverage
    };
}

see example: ./test/test-puppeteer.js

How to convert V8 to Istanbul

Using v8-to-istanbul

It is a popular library which is used to convert V8 coverage format to istanbul's coverage format. Most test frameworks are using it, such as Jest, Vitest, but it has two major problems:

• 1, The source mapping does not work well if the position is between the two consecutive mappings. for example:

const a = tf ? 'true' : 'false';
               ^     ^  ^
              m1     p  m2

m1 and m2 are two consecutive mappings, p is the position we looking for. However, we can only get the position of the m1 or m2 if we don't fix it to p. Especially the generated code is different from the original code, such as the code was minified, compressed or converted, it is difficult to find the exact position.

• 2, The coverage of functions and branches is incorrect. V8 only provided coverage at functions and it's blocks. But if a function is uncovered (count = 0), there is no information for it's blocks and sub-level functions. And also there are some problems about counting the functions and branches:

// Problem: When the function or parent-level function is uncovered, then its sub-level functions will never be counted.
functions.forEach(block => {
    block.ranges.forEach((range, i) => {
        if (block.isBlockCoverage) {
            // v8-to-istanbul: new CovBranch() 
            // Problem: not every block is branch, and the first block could be function.
            if (block.functionName && i === 0) {
                // v8-to-istanbul: new CovFunction()
                // Problem: no anonymous function
            }
        } else if (block.functionName) {
            // v8-to-istanbul: new CovFunction()
            // Problem: no anonymous function
        }
    }
});

see source code v8-to-istanbul.js

How Monocart Works

We implemented new converter instead of v8-to-istanbul:

• 1, Trying to fix the middle position if not found the exact mapping for the position.
• 2, Finding all functions and branches by parsing the source code AST, however the V8 cannot provide effective branch coverage information, so the branches is still not perfect but close to reality.

AST	V8
AssignmentPattern	🛇 Not Support
ConditionalExpression	✔ Not Perfect
IfStatement	✔ Not Perfect
LogicalExpression	✔ Not Perfect
SwitchStatement	✔ Not Perfect

Istanbul Introduction

• Istanbul coverage report - Instrumenting source codes and generating coverage reports
• babel-plugin-istanbul
• istanbul-reports
• Code Coverage Introduction

Thanks

• Special thanks to @edumserrano