monocart-coverage-reports

Monocart Coverage Reports

Code coverage tool to generate native V8 reports or Istanbul reports.

• Usage
• Default Options
• Available Reports
• Multiprocessing Support
• Compare Reports
• Compare Workflows
• Collecting Istanbul Coverage Data
• Collecting V8 Coverage Data
• Node.js V8 Coverage Report for Server Side
• Using entryFilter and sourceFilter to filter the results for V8 report
• How to convert V8 to Istanbul
  • Using v8-to-istanbul
  • How Monocart Works
• Chromium Coverage API
• Istanbul Introduction
• Thanks

Usage

const CoverageReport = require('monocart-coverage-reports');
const options = {
    outputDir: './coverage-reports',
    reports: "v8"
}
const coverageReport = new CoverageReport(options);
await coverageReport.add(coverageData1);
await coverageReport.add(coverageData2);
const coverageResults = await coverageReport.generate();
console.log(coverageResults.summary);

• example v8
• example istanbul

Default Options

• lib/default/options.js

Available Reports

• v8 (V8 data only)
  • V8 html - Example for browser, build with webpack, and also Rollup and Esbuild
  • V8 Node - Example for Node.js using env, and also V8 API, Inspector and CDP
  • V8 Minify

• v8-json (V8 data only)

  • V8 coverage-report.json

• console-summary shows coverage summary in console

  

Following are istanbul reports

• clover
• cobertura
• html
  • Istanbul html
  • V8 to Istanbul
• html-spa
  • Istanbul html-spa
• json
• json-summary
• lcov
• lcovonly
  • V8 lcov.info
  • Istanbul lcov.info
• none
• teamcity
• text
• text-lcov
• text-summary

const CoverageReport = require('monocart-coverage-reports');
const options = {
    outputDir: './coverage-reports',
    reports: [
        ['console-summary'],
        ['v8'],
        ['html', {
            subdir: 'istanbul'
        }],
        ['json', {
            file: 'my-json-file.json'
        }],
        'lcovonly'
    ]
}
const coverageReport = new CoverageReport(options);

Integration

• monocart-reporter - Test reporter for Playwright
• vitest-monocart-coverage - Integration with Vitest coverage

Multiprocessing Support

The data will be added to [outputDir]/.cache, and the cache will be removed after reports generated.

• sub process 1

const CoverageReport = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = new CoverageReport(options);
await coverageReport.add(coverageData1);

• sub process 2

const CoverageReport = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = new CoverageReport(options);
await coverageReport.add(coverageData2);

• main process

// after all sub processes finished
const CoverageReport = require('monocart-coverage-reports');
const options = require('path-to/same-options.js');
const coverageReport = new CoverageReport(options);
const coverageResults = await coverageReport.generate();
console.log(coverageResults.summary);

Compare Reports

	Istanbul	V8	V8 to Istanbul
Coverage data	Istanbul (Object)	V8 (Array)	V8 (Array)
Output	Istanbul reports	V8 reports	Istanbul reports
- Bytes	❌	✅	❌
- Statements	✅	❌	☑️❔
- Branches	✅	☑️❔	☑️❔
- Functions	✅	✅	✅
- Lines	✅	✅	✅
- Execution counts	✅	✅	✅
CSS coverage	❌	✅	✅
Minified code	❌	✅	❌

❔ - Partial or conditional support

Compare Workflows

• Istanbul Workflows

  • 1, Collecting Istanbul coverage data
  • 2, Adding coverage data and generating coverage report

• V8 Workflows

  • 1, Collecting V8 coverage data
  • 3, Adding coverage data and generating coverage report

Collecting Istanbul Coverage Data

• Instrumenting source code

  Before collecting Istanbul coverage data, It requires your source code is instrumented with Istanbul

  • Webpack: babel-plugin-istanbul, example: webpack.config-istanbul.js
  • Rollup: rollup-plugin-istanbul
  • Vite: vite-plugin-istanbul
• Browser

  Collecting coverage data from window.__coverage__, example: test-istanbul.js

• Node.js

  Collecting coverage data from global.__coverage__

Collecting V8 Coverage Data

• For source code: enable sourcemap and do not compress/minify:
  • Webpack: with source-map devtool and development mode, example webpack.config-v8.js
  • Rollup: options sourcemap: true
• Browser (Chromium Only)

  Collecting coverage data with Chromium Coverage API, see example

• Node.js

  see Node.js V8 Coverage Report for Server Side

Node.js V8 Coverage Report for Server Side

• Using Node.js env NODE_V8_COVERAGE=dir

  • Before running your Node.js application, set env NODE_V8_COVERAGE=dir. After the application runs and exits, the coverage data will be saved to the dir directory in JSON file format
  • Read the json file(s) from the dir and generate coverage report
  • example:

  cross-env NODE_V8_COVERAGE=.temp/v8-coverage node ./test/test-node-env.js && node ./test/generate-node-report.js

• Using V8 API

  • Writing the coverage started by NODE_V8_COVERAGE to disk on demand with v8.takeCoverage() and stopping with v8.stopCoverage().
  • example:

  cross-env NODE_V8_COVERAGE=.temp/v8-coverage node ./test/test-node-api.js

• Using Inspector API (or module collect-v8-coverage)

  • Connecting to the V8 inspector and enable V8 coverage.
  • Taking coverage data and adding to the report after your application runs.
  • example: ./test/test-node-ins.js

• Using CDP API

  • Enabling Node Debugging
  • Collecting coverage data with CDP API.
  • example: ./test/test-node-cdp.js

Using entryFilter and sourceFilter to filter the results for V8 report

When V8 coverage data collected, it actually contains the data of all entry files, for example:

1, dist/main.js
2, dist/vendor.js
3, dist/something-else.js

We can use entryFilter to filter the entry files. For example, we should remove vendor.js and something-else.js if they are not in our coverage scope.

1, dist/main.js

When inline or linked sourcemap exists to the entry file, the source files will be extracted from the sourcemap for the entry file, and the entry file will be removed if logging is not debug.

1, src/index.js
2, src/components/app.js
3, node_modules/dependency/dist/dependency.js

We can use sourceFilter to filter the source files. For example, we should remove dependency.js if it is not in our coverage scope.

1, src/index.js
2, src/components/app.js

Example:

const coverageOptions = {
    entryFilter: (entry) => entry.url.indexOf("main.js") !== -1,
    sourceFilter: (sourcePath) => sourcePath.search(/src\//) !== -1
};

How to convert V8 to Istanbul

Using v8-to-istanbul

It is a popular library which is used to convert V8 coverage format to istanbul's coverage format. Most test frameworks are using it, such as Jest, Vitest, but it has two major problems:

• 1, The source mapping does not work well if the position is between the two consecutive mappings. for example:

const a = tf ? 'true' : 'false';
               ^     ^  ^
              m1     p  m2

m1 and m2 are two consecutive mappings, p is the position we looking for. However, we can only get the position of the m1 if we don't fix it to p. Especially the generated code is different from the original code, such as minified, compressed or converted, then it becomes very difficult to find the middle position between two mappings.

• 2, The coverage of functions and branches is incorrect. V8 only provided coverage at functions and it's blocks. But if a function is uncovered (count = 0), there is no information for it's blocks and sub-level functions. And also there are some problems about counting the functions and branches:

functions.forEach(block => {
    block.ranges.forEach((range, i) => {
        if (block.isBlockCoverage) {
            // v8-to-istanbul: count range as branch here. 
            // Problem: not every block is branch, and the first block is actually function.
            if (block.functionName && i === 0) {
                // v8-to-istanbul: count range as function here. 
                // Problem: there is no function name if the function is anonymous or expression.
            }
        } else if (block.functionName) {
            // v8-to-istanbul: count range as function here. 
            // Problem: same as above function.
        }
    }
});
// Problem: When the function or parent-level function is uncovered, then its sub-level functions will never be counted.

see source code v8-to-istanbul.js

How Monocart Works

We have removed v8-to-istanbul because of the two major problems and implemented new converter:

• 1, Trying to fix the middle position if not found the exact mapping.
• 2, Finding all functions and branches by parsing the source code AST, however the V8 cannot provide effective branch coverage information, so the branches is still not perfect.

AST	V8
AssignmentPattern	🛇 Not Support
ConditionalExpression	✔ Not Perfect
IfStatement	✔ Not Perfect
LogicalExpression	✔ Not Perfect
SwitchStatement	✔ Not Perfect

Chromium Coverage API

• V8 coverage report - Native support for JavaScript code coverage to V8. (Chromium only)
• Puppeteer Coverage Class
• Playwright Coverage Class
• DevTools Protocol for Coverage

Istanbul Introduction

• Istanbul coverage report - Instrumenting source codes and generating coverage reports
• babel-plugin-istanbul
• istanbul-reports
• Code Coverage Introduction

Thanks

• Special thanks to @edumserrano