monocart-coverage-reports

Monocart Coverage Reports

Code coverage tool to generate native V8 reports or Istanbul reports.

• Usage
• Default Options
• Available Reports
• Multiprocessing Support
• onEnd Hook
• mcr CLI
• Compare Reports
• Compare Workflows
• Collecting Istanbul Coverage Data
• Collecting V8 Coverage Data
• Node.js V8 Coverage Report for Server Side
• Using entryFilter and sourceFilter to filter the results for V8 report
• How to convert V8 to Istanbul
  • Using v8-to-istanbul
  • How Monocart Works
• Ignoring Uncovered Lines
• Chromium Coverage API
• V8 Coverage Data Format
• Istanbul Introduction
• Thanks

Usage

const MCR = require('monocart-coverage-reports');
const options = {
    outputDir: './coverage-reports',
    reports: "v8"
}
const coverageReport = MCR(options);
await coverageReport.add(coverageData1);
await coverageReport.add(coverageData2);
const coverageResults = await coverageReport.generate();
console.log(coverageResults.summary);

• example v8
• example istanbul

Default Options

• lib/default/options.js

Available Reports

• v8 (V8 data only)
  • Test in browser
    • Build with webpack: V8 and V8 Minify
    • Build with Rollup and Esbuild
    • Collect with puppeteer
  • Test in Node.js
    • Collect with env, and also V8 API, Inspector and CDP
    • Web server example: koa

• v8-json (V8 data only)

  • V8 coverage-report.json

• console-summary shows coverage summary in console

  

Following are istanbul reports

• clover
• cobertura
• html
  • Istanbul html
  • V8 to Istanbul
• html-spa
  • Istanbul html-spa
• json
• json-summary
• lcov
• lcovonly
  • V8 lcov.info
  • Istanbul lcov.info
• none
• teamcity
• text
• text-lcov
• text-summary

const MCR = require('monocart-coverage-reports');
const options = {
    outputDir: './coverage-reports',
    reports: [
        ['console-summary'],
        ['v8'],
        ['html', {
            subdir: 'istanbul'
        }],
        ['json', {
            file: 'my-json-file.json'
        }],
        'lcovonly'
    ]
}
const coverageReport = MCR(options);

Integration

• monocart-reporter - Test reporter for Playwright
• vitest-monocart-coverage - Integration with Vitest coverage

Multiprocessing Support

The data will be added to [outputDir]/.cache, and the cache will be removed after reports generated.

• sub process 1

const MCR = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = MCR(options);
await coverageReport.add(coverageData1);

• sub process 2

const MCR = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = MCR(options);
await coverageReport.add(coverageData2);

• main process

// after all sub processes finished
const MCR = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = MCR(options);
const coverageResults = await coverageReport.generate();
console.log(coverageResults.summary);

onEnd Hook

For example, checking thresholds:

const EC = require('eight-colors');
const coverageOptions = {
    name: 'My Coverage Report',
    outputDir: './coverage-reports',
    onEnd: (coverageResults) => {
        const thresholds = {
            bytes: 80,
            lines: 60
        };
        console.log('check thresholds ...', thresholds);
        const errors = [];
        const { summary } = coverageResults;
        Object.keys(thresholds).forEach((k) => {
            const pct = summary[k].pct;
            if (pct < thresholds[k]) {
                errors.push(`Coverage threshold for ${k} (${pct} %) not met: ${thresholds[k]} %`);
            }
        });
        if (errors.length) {
            const errMsg = errors.join('\n');
            console.log(EC.red(errMsg));
            // throw new Error(errMsg);
            // process.exit(1);
        }
    }
}

mcr CLI

• Global mode

npm i monocart-coverage-reports -g
mcr "node ./test/test-node-env.js" -r v8,console-summary --lcov

• Current working directory mode

npm i monocart-coverage-reports
npx mcr "node ./test/test-node-env.js" -r v8,console-summary --lcov

• CLI Options

Usage: mcr [options] <command>

CLI to generate coverage reports

Arguments:
  command                      command to execute

Options:
  -V, --version                output the version number
  -c, --config <path>          config path for options
  -o, --outputDir <dir>        output dir for reports
  -r, --reports <name[,name]>  coverage reports to use
  -n, --name <name>            report name for title
  --outputFile <path>          output file for v8 report
  --inline                     inline html for v8 report
  --assetsPath <path>          assets path if not inline
  --lcov                       generate lcov.info file
  -h, --help                   display help for command

• Using configuration file for more options

mcr "node ./test/test-node-env.js" -c test/cli-options.js

Compare Reports

	Istanbul	V8	V8 to Istanbul
Coverage data	Istanbul (Object)	V8 (Array)	V8 (Array)
Output	Istanbul reports	V8 reports	Istanbul reports
- Bytes	❌	✅	❌
- Statements	✅	❌	☑️❔
- Branches	✅	☑️❔	☑️❔
- Functions	✅	✅	✅
- Lines	✅	✅	✅
- Execution counts	✅	✅	✅
CSS coverage	❌	✅	✅
Minified code	❌	✅	❌

❔ - Partial or conditional support

Compare Workflows

• Istanbul Workflows

  • 1, Collecting Istanbul coverage data
  • 2, Adding coverage data and generating coverage report

• V8 Workflows

  • 1, Collecting V8 coverage data
  • 3, Adding coverage data and generating coverage report

Collecting Istanbul Coverage Data

• Instrumenting source code

  Before collecting Istanbul coverage data, It requires your source code is instrumented with Istanbul

  • Webpack: babel-plugin-istanbul, example: webpack.config-istanbul.js
  • Rollup: rollup-plugin-istanbul
  • Vite: vite-plugin-istanbul
• Browser

  Collecting coverage data from window.__coverage__, example: test-istanbul.js

• Node.js

  Collecting coverage data from global.__coverage__

Collecting V8 Coverage Data

• For source code: enable sourcemap and do not compress/minify:
  • Webpack: build with source-map devtool and development mode, example webpack.config-v8.js
  • Rollup: build with options sourcemap: true
  • Vite: build with options sourcemap: true and minify: false
• Browser (Chromium Only)

  Collecting coverage data with Chromium Coverage API, see playwright example or puppeteer example

• Node.js

  see Node.js V8 Coverage Report for Server Side

Node.js V8 Coverage Report for Server Side

Possible solutions:

• Node.js env NODE_V8_COVERAGE=dir

  • Before running your Node.js application, set env NODE_V8_COVERAGE=dir. After the application runs and exits, the coverage data will be saved to the dir directory in JSON file format
  • Read the json file(s) from the dir and generate coverage report
  • example:

  cross-env NODE_V8_COVERAGE=.temp/v8-coverage-env node ./test/test-node-env.js && node ./test/generate-node-report.js

• V8 API + NODE_V8_COVERAGE

  • Writing the coverage started by NODE_V8_COVERAGE to disk on demand with v8.takeCoverage() and stopping with v8.stopCoverage().
  • example:

  cross-env NODE_V8_COVERAGE=.temp/v8-coverage-api node ./test/test-node-api.js

• Inspector API (or module collect-v8-coverage)

  • Connecting to the V8 inspector and enable V8 coverage.
  • Taking coverage data and adding to the report after your application runs.
  • example: ./test/test-node-ins.js

• CDP API

  • Enabling Node Debugging
  • Collecting coverage data with CDP API.
  • example: ./test/test-node-cdp.js

• Debugger + NODE_V8_COVERAGE + CDP + V8 API

  cross-env NODE_V8_COVERAGE=.temp/v8-coverage-koa node --inspect=9229 ./test/koa.js

  • example:

  node ./test/test-node-koa.js

Using entryFilter and sourceFilter to filter the results for V8 report

When V8 coverage data collected, it actually contains the data of all entry files, for example:

dist/main.js
dist/vendor.js
dist/something-else.js

We can use entryFilter to filter the entry files. For example, we should remove vendor.js and something-else.js if they are not in our coverage scope.

dist/main.js

When inline or linked sourcemap exists to the entry file, the source files will be extracted from the sourcemap for the entry file, and the entry file will be removed if logging is not debug.

> src/index.js
> src/components/app.js
> node_modules/dependency/dist/dependency.js

We can use sourceFilter to filter the source files. For example, we should remove dependency.js if it is not in our coverage scope.

> src/index.js
> src/components/app.js

Example:

const coverageOptions = {
    entryFilter: (entry) => entry.url.indexOf("main.js") !== -1,
    sourceFilter: (sourcePath) => sourcePath.search(/src\//) !== -1
};

How to convert V8 to Istanbul

Using v8-to-istanbul

It is a popular library which is used to convert V8 coverage format to istanbul's coverage format. Most test frameworks are using it, such as Jest, Vitest, but it has two major problems:

• 1, The source mapping does not work well if the position is between the two consecutive mappings. for example:

const a = tf ? 'true' : 'false';
               ^     ^  ^
              m1     p  m2

m1 and m2 are two consecutive mappings, p is the position we looking for. However, we can only get the position of the m1 if we don't fix it to p. Especially the generated code is different from the original code, such as minified, compressed or converted, then it becomes very difficult to find the middle position between two mappings.

• 2, The coverage of functions and branches is incorrect. V8 only provided coverage at functions and it's blocks. But if a function is uncovered (count = 0), there is no information for it's blocks and sub-level functions. And also there are some problems about counting the functions and branches:

functions.forEach(block => {
    block.ranges.forEach((range, i) => {
        if (block.isBlockCoverage) {
            // v8-to-istanbul: new CovBranch() 
            // Problem: not every block is branch, and the first block is actually function.
            if (block.functionName && i === 0) {
                // v8-to-istanbul: new CovFunction()
                // Problem: no anonymous function
            }
        } else if (block.functionName) {
            // v8-to-istanbul: new CovFunction()
            // Problem: no anonymous function
        }
    }
});
// Problem: When the function or parent-level function is uncovered, then its sub-level functions will never be counted.

see source code v8-to-istanbul.js

How Monocart Works

We have removed v8-to-istanbul because of the two major problems and implemented new converter:

• 1, Trying to fix the middle position if not found the exact mapping.
• 2, Finding all functions and branches by parsing the source code AST, however the V8 cannot provide effective branch coverage information, so the branches is still not perfect.

AST	V8
AssignmentPattern	🛇 Not Support
ConditionalExpression	✔ Not Perfect
IfStatement	✔ Not Perfect
LogicalExpression	✔ Not Perfect
SwitchStatement	✔ Not Perfect

Ignoring Uncovered Codes

To ignore codes, use the special comment which starts with v8 ignore :

• ignoring all until told

/* v8 ignore start */
function uncovered() {
}
/* v8 ignore stop */

• ignoring the next line or next N lines

/* v8 ignore next */
const os = platform === 'wind32' ? 'Windows' : 'Other';

const os = platform === 'wind32' ? 'Windows' /* v8 ignore next */ : 'Other';

// v8 ignore next 3
if (platform === 'linux') {
    console.info('hello linux');
}

Chromium Coverage API

• V8 coverage report - Native support for JavaScript code coverage to V8. (Chromium only)
• Playwright Coverage Class
• Puppeteer Coverage class
• DevTools Protocol for Coverage

V8 Coverage Data Format

// Coverage data for a source range.
export interface CoverageRange {
    // JavaScript script source offset for the range start.
    startOffset: integer;
    // JavaScript script source offset for the range end.
    endOffset: integer;
    // Collected execution count of the source range.
    count: integer;
}

// Coverage data for a JavaScript function.
/**
 * @functionName can be an empty string.
 * @ranges is always non-empty. The first range is called the "root range".
 * @isBlockCoverage indicates if the function has block coverage information.
    If this is false, it usually means that the functions was never called.
    It seems to be equivalent to ranges.length === 1 && ranges[0].count === 0.
*/
export interface FunctionCoverage {
    // JavaScript function name.
    functionName: string;
    // Source ranges inside the function with coverage data.
    ranges: CoverageRange[];
    // Whether coverage data for this function has block granularity.
    isBlockCoverage: boolean;
}

// Coverage data for a JavaScript script.
export interface ScriptCoverage {
    // JavaScript script id.
    scriptId: Runtime.ScriptId;
    // JavaScript script name or url.
    url: string;
    // Functions contained in the script that has coverage data.
    functions: FunctionCoverage[];
}

export type V8CoverageData = ScriptCoverage[];

see devtools-protocol ScriptCoverage and v8-coverage

Collect raw v8 coverage data with Puppeteer

await page.coverage.startJSCoverage({
    // provide raw v8 coverage data
    includeRawScriptCoverage: true
});

await page.goto(url);

const jsCoverage = await page.coverage.stopJSCoverage();
const rawV8CoverageData = jsCoverage.map((it) => {
    // Convert to raw v8 coverage format
    return {
        source: it.text,
        ... it.rawScriptCoverage
    };
}

see example: ./test/test-puppeteer.js

Istanbul Introduction

• Istanbul coverage report - Instrumenting source codes and generating coverage reports
• babel-plugin-istanbul
• istanbul-reports
• Code Coverage Introduction

Thanks

• Special thanks to @edumserrano