solidity-coverage

What is solidity-coverage?

solidity-coverage is an npm package that provides code coverage metrics for Solidity smart contracts. It helps developers ensure that their tests cover all possible execution paths in their smart contracts, which is crucial for identifying potential vulnerabilities and ensuring the robustness of the code.

What are solidity-coverage's main functionalities?

Code Coverage Report

This feature allows you to generate a code coverage report for your Solidity smart contracts. By integrating solidity-coverage with your Truffle configuration, you can run your tests and get a detailed report on which lines of code were executed.

const coverage = require('solidity-coverage');
module.exports = {
  networks: {
    development: {
      host: '127.0.0.1',
      port: 8545,
      network_id: '*',
    },
  },
  plugins: ['solidity-coverage'],
};

Integration with Truffle

solidity-coverage integrates seamlessly with the Truffle framework, allowing you to run your tests and generate coverage reports without additional setup. This integration simplifies the process of ensuring your smart contracts are thoroughly tested.

const coverage = require('solidity-coverage');
module.exports = {
  networks: {
    development: {
      host: '127.0.0.1',
      port: 8545,
      network_id: '*',
    },
  },
  plugins: ['solidity-coverage'],
};

Command Line Interface

You can use the command line interface to run solidity-coverage directly from your terminal. This command will execute your tests and generate a coverage report, making it easy to incorporate into your CI/CD pipeline.

npx solidity-coverage

Other packages similar to solidity-coverage

truffle

Truffle is a development environment, testing framework, and asset pipeline for Ethereum. While it does not provide code coverage out of the box, it can be used in conjunction with solidity-coverage to achieve similar functionality. Truffle focuses more on the overall development and deployment process of smart contracts.

hardhat

Hardhat is a development environment for Ethereum software. It provides a flexible and extensible way to manage and automate the recurring tasks inherent to developing smart contracts and dApps. Hardhat has its own coverage plugin, hardhat-coverage, which offers similar functionality to solidity-coverage but is designed to work specifically with the Hardhat environment.

embark

Embark is a framework for serverless Decentralized Applications using Ethereum, IPFS, and other platforms. It includes a testing framework and can be extended with plugins to provide code coverage. Embark is more comprehensive in scope, covering the entire dApp development lifecycle.

README

solidity-coverage

Code coverage for Solidity testing

• For more details about what this is, how it works and potential limitations, see the accompanying article.
• solidity-coverage is in development and its accuracy is unknown. If you find discrepancies between the coverage report and your suite's behavior, please open an issue.
• solidity-coverage is Solcover

Install

$ npm install --save-dev solidity-coverage

Run

$ ./node_modules/.bin/solidity-coverage

Tests run signficantly slower while coverage is being generated. A 1 to 2 minute delay between the end of Truffle compilation and the beginning of test execution is possible if your test suite is large. Large Solidity files can also take a while to instrument.

Network Configuration

By default, solidity-coverage generates a stub truffle.js that accomodates its special gas needs and connects to a modified version of testrpc on port 8555. If your tests will run on the development network using a standard truffle.js and testrpc instance, you shouldn't have to do any configuration. If your tests depend on logic or special options added to truffle.js you should declare a coverage network there following the example below.

Example

module.exports = {
  networks: {
    development: {
      host: "localhost",
      port: 8545,
      network_id: "*"
    },
    coverage: {
      host: "localhost",
      network_id: "*",
      port: 8555,         // <-- If you change this, also set the port option in .solcover.js.
      gas: 0xfffffffffff, // <-- Use this high gas value
      gasPrice: 0x01      // <-- Use this low gas price
    },
    ...etc...
  }
};

Options

You can also create a .solcover.js config file in the root directory of your project and specify additional options if necessary:

Example:

module.exports = {
    port: 6545,
    testrpcOptions: '-p 6545 -u 0x54fd80d6ae7584d8e9a19fe1df43f04e5282cc43',
    testCommand: 'mocha --timeout 5000',
    norpc: true,
    dir: './secretDirectory',
    skipFiles: ['Routers/EtherRouter.sol']
};

Option	Type	Default	Description
accounts	Number	35	Number of accounts to launch testrpc with.
port	Number	8555	Port to run testrpc on / have truffle connect to
norpc	Boolean	false	Prevent solidity-coverage from launching its own testrpc. Useful if you are managing a complex test suite with a shell script
testCommand	String	truffle test	Run an arbitrary test command. ex: mocha --timeout 5000. NB: Also set the port option to whatever your tests require (probably 8545).
testrpcOptions	String	--port 8555	options to append to a command line invocation of testrpc. NB: Using this overwrites the defaults so always specify a port in this string and in the port option
copyNodeModules	Boolean	false	Copies node_modules into the coverage environment. May significantly increase the time for coverage to complete if enabled. Useful if your contracts import solidity files from an npm installed package.
skipFiles	Array	['Migrations.sol']	Array of contracts (with paths expressed relative to the contracts directory) that should be skipped when doing instrumentation. Migrations.sol is skipped by default, and does not need to be added to this configuration option if it is used.
dir	String	.	Solidity-coverage copies all the assets in your root directory (except node_modules) to a special folder where it instruments the contracts and executes the tests. dir allows you to define a relative path from the root directory to those assets. Useful if your contracts & tests are within their own folder as part of a larger project.

FAQ

Solutions to common issues people run into using this tool:

• Running out of gas
• Running out of memory (locally and in CI)
• Running out of time (in mocha)
• Running on windows
• Running testrpc-sc on its own
• Using alongside HDWalletProvider
• Integrating into CI
• Why are asserts and requires highlighted as branch points?
• Why are send and transfer throwing in my tests?

Example reports

• metacoin (Istanbul HTML)
• zeppelin-solidity (Coveralls)
• gnosis-contracts (Codecov)

Contribution Guidelines

Contributions are welcome! If you're opening a PR that adds features please consider writing some unit tests for them. You could also lint your submission with npm run lint. Bugs can be reported in the issues.

Contributors

• @area
• @cgewecke
• @adriamb
• @cag
• @maurelian
• @rudolfix
• @phiferd