monocart-coverage-reports

Monocart Coverage Reports

Code coverage tool to generate native V8 reports or Istanbul reports.

• Usage
• Default Options
• Available Reports
• Multiprocessing Support
• onEnd Hook
• Compare Reports
• Compare Workflows
• Collecting Istanbul Coverage Data
• Collecting V8 Coverage Data
• Node.js V8 Coverage Report for Server Side
• Using entryFilter and sourceFilter to filter the results for V8 report
• How to convert V8 to Istanbul
  • Using v8-to-istanbul
  • How Monocart Works
• Ignoring Uncovered Lines
• Chromium Coverage API
• V8 Coverage Data Format
• Istanbul Introduction
• Thanks

Usage

const CoverageReport = require('monocart-coverage-reports');
const options = {
    outputDir: './coverage-reports',
    reports: "v8"
}
const coverageReport = new CoverageReport(options);
await coverageReport.add(coverageData1);
await coverageReport.add(coverageData2);
const coverageResults = await coverageReport.generate();
console.log(coverageResults.summary);

• example v8
• example istanbul

Default Options

• lib/default/options.js

Available Reports

• v8 (V8 data only)
  • V8 html - Example for browser, build with webpack, and also Rollup and Esbuild
  • V8 Node - Example for Node.js using env, and also V8 API, Inspector and CDP
  • V8 Minify

• v8-json (V8 data only)

  • V8 coverage-report.json

• console-summary shows coverage summary in console

  

Following are istanbul reports

• clover
• cobertura
• html
  • Istanbul html
  • V8 to Istanbul
• html-spa
  • Istanbul html-spa
• json
• json-summary
• lcov
• lcovonly
  • V8 lcov.info
  • Istanbul lcov.info
• none
• teamcity
• text
• text-lcov
• text-summary

const CoverageReport = require('monocart-coverage-reports');
const options = {
    outputDir: './coverage-reports',
    reports: [
        ['console-summary'],
        ['v8'],
        ['html', {
            subdir: 'istanbul'
        }],
        ['json', {
            file: 'my-json-file.json'
        }],
        'lcovonly'
    ]
}
const coverageReport = new CoverageReport(options);

Integration

• monocart-reporter - Test reporter for Playwright
• vitest-monocart-coverage - Integration with Vitest coverage

Multiprocessing Support

The data will be added to [outputDir]/.cache, and the cache will be removed after reports generated.

• sub process 1

const CoverageReport = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = new CoverageReport(options);
await coverageReport.add(coverageData1);

• sub process 2

const CoverageReport = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = new CoverageReport(options);
await coverageReport.add(coverageData2);

• main process

// after all sub processes finished
const CoverageReport = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = new CoverageReport(options);
const coverageResults = await coverageReport.generate();
console.log(coverageResults.summary);

onEnd Hook

For example, checking thresholds:

const EC = require('eight-colors');
const coverageOptions = {
    name: 'My Coverage Report',
    outputDir: './coverage-reports',
    onEnd: (coverageResults) => {
        const thresholds = {
            bytes: 80,
            lines: 60
        };
        console.log('check thresholds ...', thresholds);
        const errors = [];
        const { summary } = coverageResults;
        Object.keys(thresholds).forEach((k) => {
            const pct = summary[k].pct;
            if (pct < thresholds[k]) {
                errors.push(`Coverage threshold for ${k} (${pct} %) not met: ${thresholds[k]} %`);
            }
        });
        if (errors.length) {
            const errMsg = errors.join('\n');
            console.log(EC.red(errMsg));
            // throw new Error(errMsg);
            // process.exit(1);
        }
    }
}

Compare Reports

	Istanbul	V8	V8 to Istanbul
Coverage data	Istanbul (Object)	V8 (Array)	V8 (Array)
Output	Istanbul reports	V8 reports	Istanbul reports
- Bytes	❌	✅	❌
- Statements	✅	❌	☑️❔
- Branches	✅	☑️❔	☑️❔
- Functions	✅	✅	✅
- Lines	✅	✅	✅
- Execution counts	✅	✅	✅
CSS coverage	❌	✅	✅
Minified code	❌	✅	❌

❔ - Partial or conditional support

Compare Workflows

• Istanbul Workflows

  • 1, Collecting Istanbul coverage data
  • 2, Adding coverage data and generating coverage report

• V8 Workflows

  • 1, Collecting V8 coverage data
  • 3, Adding coverage data and generating coverage report

Collecting Istanbul Coverage Data

• Instrumenting source code

  Before collecting Istanbul coverage data, It requires your source code is instrumented with Istanbul

  • Webpack: babel-plugin-istanbul, example: webpack.config-istanbul.js
  • Rollup: rollup-plugin-istanbul
  • Vite: vite-plugin-istanbul
• Browser

  Collecting coverage data from window.__coverage__, example: test-istanbul.js

• Node.js

  Collecting coverage data from global.__coverage__

Collecting V8 Coverage Data

• For source code: enable sourcemap and do not compress/minify:
  • Webpack: build with source-map devtool and development mode, example webpack.config-v8.js
  • Rollup: build with options sourcemap: true
  • Vite: build with options sourcemap: true and minify: false
• Browser (Chromium Only)

  Collecting coverage data with Chromium Coverage API, see example

• Node.js

  see Node.js V8 Coverage Report for Server Side

Node.js V8 Coverage Report for Server Side

• Using Node.js env NODE_V8_COVERAGE=dir

  • Before running your Node.js application, set env NODE_V8_COVERAGE=dir. After the application runs and exits, the coverage data will be saved to the dir directory in JSON file format
  • Read the json file(s) from the dir and generate coverage report
  • example:

  cross-env NODE_V8_COVERAGE=.temp/v8-coverage-env node ./test/test-node-env.js && node ./test/generate-node-report.js

• Using V8 API + NODE_V8_COVERAGE

  • Writing the coverage started by NODE_V8_COVERAGE to disk on demand with v8.takeCoverage() and stopping with v8.stopCoverage().
  • example:

  cross-env NODE_V8_COVERAGE=.temp/v8-coverage-api node ./test/test-node-api.js

• Using foreground-child + NODE_V8_COVERAGE

  • example: ./test/test-node-fgc.js

• Using Inspector API (or module collect-v8-coverage)

  • Connecting to the V8 inspector and enable V8 coverage.
  • Taking coverage data and adding to the report after your application runs.
  • example: ./test/test-node-ins.js

• Using CDP API

  • Enabling Node Debugging
  • Collecting coverage data with CDP API.
  • example: ./test/test-node-cdp.js

Using entryFilter and sourceFilter to filter the results for V8 report

When V8 coverage data collected, it actually contains the data of all entry files, for example:

dist/main.js
dist/vendor.js
dist/something-else.js

We can use entryFilter to filter the entry files. For example, we should remove vendor.js and something-else.js if they are not in our coverage scope.

dist/main.js

When inline or linked sourcemap exists to the entry file, the source files will be extracted from the sourcemap for the entry file, and the entry file will be removed if logging is not debug.

> src/index.js
> src/components/app.js
> node_modules/dependency/dist/dependency.js

We can use sourceFilter to filter the source files. For example, we should remove dependency.js if it is not in our coverage scope.

> src/index.js
> src/components/app.js

Example:

const coverageOptions = {
    entryFilter: (entry) => entry.url.indexOf("main.js") !== -1,
    sourceFilter: (sourcePath) => sourcePath.search(/src\//) !== -1
};

How to convert V8 to Istanbul

Using v8-to-istanbul

It is a popular library which is used to convert V8 coverage format to istanbul's coverage format. Most test frameworks are using it, such as Jest, Vitest, but it has two major problems:

• 1, The source mapping does not work well if the position is between the two consecutive mappings. for example:

const a = tf ? 'true' : 'false';
               ^     ^  ^
              m1     p  m2

m1 and m2 are two consecutive mappings, p is the position we looking for. However, we can only get the position of the m1 if we don't fix it to p. Especially the generated code is different from the original code, such as minified, compressed or converted, then it becomes very difficult to find the middle position between two mappings.

• 2, The coverage of functions and branches is incorrect. V8 only provided coverage at functions and it's blocks. But if a function is uncovered (count = 0), there is no information for it's blocks and sub-level functions. And also there are some problems about counting the functions and branches:

functions.forEach(block => {
    block.ranges.forEach((range, i) => {
        if (block.isBlockCoverage) {
            // v8-to-istanbul: new CovBranch() 
            // Problem: not every block is branch, and the first block is actually function.
            if (block.functionName && i === 0) {
                // v8-to-istanbul: new CovFunction()
                // Problem: no anonymous function
            }
        } else if (block.functionName) {
            // v8-to-istanbul: new CovFunction()
            // Problem: no anonymous function
        }
    }
});
// Problem: When the function or parent-level function is uncovered, then its sub-level functions will never be counted.

see source code v8-to-istanbul.js

How Monocart Works

We have removed v8-to-istanbul because of the two major problems and implemented new converter:

• 1, Trying to fix the middle position if not found the exact mapping.
• 2, Finding all functions and branches by parsing the source code AST, however the V8 cannot provide effective branch coverage information, so the branches is still not perfect.

AST	V8
AssignmentPattern	🛇 Not Support
ConditionalExpression	✔ Not Perfect
IfStatement	✔ Not Perfect
LogicalExpression	✔ Not Perfect
SwitchStatement	✔ Not Perfect

Ignoring Uncovered Codes

To ignore codes, use the special comment which starts with v8 ignore :

• ignoring all until told

/* v8 ignore start */
function uncovered() {
}
/* v8 ignore stop */

• ignoring the next line or next N lines

/* v8 ignore next */
const os = platform === 'wind32' ? 'Windows' : 'Other';

const os = platform === 'wind32' ? 'Windows' /* v8 ignore next */ : 'Other';

/* v8 ignore next 3 */
if (platform === 'linux') {
    console.info('hello linux');
}

Chromium Coverage API

• V8 coverage report - Native support for JavaScript code coverage to V8. (Chromium only)
• Puppeteer Coverage Class
• Playwright Coverage Class
• DevTools Protocol for Coverage

V8 Coverage Data Format

// Coverage data for a source range.
export interface CoverageRange {
    // JavaScript script source offset for the range start.
    startOffset: integer;
    // JavaScript script source offset for the range end.
    endOffset: integer;
    // Collected execution count of the source range.
    count: integer;
}

// Coverage data for a JavaScript function.
/**
 * @functionName can be an empty string.
 * @ranges is always non-empty. The first range is called the "root range".
 * @isBlockCoverage indicates if the function has block coverage information.
    If this is false, it usually means that the functions was never called.
    It seems to be equivalent to ranges.length === 1 && ranges[0].count === 0.
*/
export interface FunctionCoverage {
    // JavaScript function name.
    functionName: string;
    // Source ranges inside the function with coverage data.
    ranges: CoverageRange[];
    // Whether coverage data for this function has block granularity.
    isBlockCoverage: boolean;
}

// Coverage data for a JavaScript script.
export interface ScriptCoverage {
    // JavaScript script id.
    scriptId: Runtime.ScriptId;
    // JavaScript script name or url.
    url: string;
    // Functions contained in the script that has coverage data.
    functions: FunctionCoverage[];
}

export type V8CoverageData = ScriptCoverage[];

see devtools-protocol ScriptCoverage

Istanbul Introduction

• Istanbul coverage report - Instrumenting source codes and generating coverage reports
• babel-plugin-istanbul
• istanbul-reports
• Code Coverage Introduction

Thanks

• Special thanks to @edumserrano