swagger-coverage-cli
Advanced tools
A Node.js CLI tool to measure test coverage of Swagger/OpenAPI specs using Postman collections or Newman run reports. Features smart endpoint mapping with intelligent status code prioritization and enhanced path matching.
A comprehensive command-line utility to analyze API test coverage across multiple protocols: OpenAPI/Swagger (REST), gRPC Protocol Buffers, and GraphQL schemas. Generates unified HTML reports with protocol-specific insights. Check out the Example!**
swagger-coverage-cli is a comprehensive tool that helps you measure API test coverage across multiple protocols. It analyzes how much of your documented APIs are actually covered by your Postman tests. The tool supports:
.proto) files with service definitions.graphql, .gql) files with queries, mutations, and subscriptionsThe tool supports processing multiple API specifications in a single run, making it ideal for organizations managing microservices with diverse protocols. It calculates unified coverage percentages and produces detailed HTML reports with protocol-specific insights.
swagger-coverage-cli provides comprehensive support for modern API ecosystems with multiple protocols, enabling unified coverage analysis across your entire technology stack.
# Single protocol APIs
swagger-coverage-cli api.yaml collection.json # OpenAPI/REST
swagger-coverage-cli service.proto collection.json # gRPC
swagger-coverage-cli schema.graphql collection.json # GraphQL
swagger-coverage-cli api-docs.csv collection.json # CSV
# Mixed protocol APIs (Enterprise-ready)
swagger-coverage-cli "api.yaml,service.proto,schema.graphql" collection.json
# All existing options work across protocols
swagger-coverage-cli "api.yaml,service.proto" collection.json --verbose --strict-body
/users/{id} vs /users/{userId}).proto file analysis/package.service/method)application/grpc and variants.graphql and .gql files/graphql endpoint--disable-spec-validation for legacy APIs or specs with reference issuescoverage-report.html with protocol identificationflowchart LR
A[OpenAPI/Swagger Spec] --> B[Load & Parse Spec]
C[Postman Collection] --> D[Load & Parse Collection]
B --> E[Match Operations]
D --> E
E --> F[Calculate Coverage]
F --> G[Generate HTML & Console Report]
swagger-coverage-cli now supports processing multiple Swagger/OpenAPI specifications in a single run, making it perfect for organizations managing multiple APIs or microservices.
To analyze multiple APIs, provide comma-separated file paths:
swagger-coverage-cli "api1.yaml,api2.yaml,api3.json" collection.json
When processing multiple APIs, the console output will show:
=== Swagger Coverage Report ===
APIs analyzed: User API, Product API, Order API
Total operations in spec(s): 24
Matched operations in Postman: 18
Coverage: 75.00%
Unmatched Spec operations:
- [User API] [GET] /users/{id}/profile (statusCode=404)
- [Product API] [POST] /products (statusCode=400)
- [Order API] [DELETE] /orders/{id} (statusCode=204)
The HTML report will include:
npm install -g swagger-coverage-cli
You will need:
openapi.yaml or swagger.json) OR CSV documentation file following the specified format.Note:
- If using a Postman collection, make sure it includes actual test scripts that assert or check specific status codes (e.g.,
pm.response.to.have.status(200)).- Folder-level tests are fully supported: tests defined at the folder level in your Postman collection will be automatically applied to all requests within that folder, ensuring comprehensive coverage calculation.
- If using a Newman report, the tool will extract actual response codes and test results from the execution data.
Use the following command:
npm swagger-coverage-cli <swaggerFile> <postmanCollectionOrNewmanReport> [options]
For Multiple APIs:
npm swagger-coverage-cli "api1.yaml,api2.yaml,api3.json" collection.json [options]
Examples:
With Postman Collection:
npm swagger-coverage-cli openapi.yaml collection.json --verbose --strict-query --strict-body
With Newman Report:
npm swagger-coverage-cli openapi.yaml newman-report.json --newman --verbose
Multiple APIs with Newman Report:
npm swagger-coverage-cli "users-api.yaml,products-api.yaml" newman-report.json --newman --output multi-api-report.html
Options:
--verbose: Display additional logs (helpful for debugging).--newman: Treat input file as Newman run report instead of Postman collection.--strict-query: Enforce strict checks on query parameters (e.g., required params, enum, pattern, etc.).--strict-body: Verify that application/json request bodies in the spec match raw JSON bodies in Postman requests.--disable-spec-validation: Disable OpenAPI/Swagger spec validation (useful for specs with validation or reference issues).--output <file>: Customize the name of the HTML report file (default is coverage-report.html).npm swagger-coverage-cli -- <swaggerFile> <postmanCollectionOrNewmanReport> [options]
After execution, you will see:
=== Swagger Coverage Report ===
Total operations in spec: 12
Matched operations in Postman: 9
Coverage: 75.00%
Unmatched operations:
- [DELETE] /items/{id} (statusCode=204)
- [PUT] /items/{id} (statusCode=400)
...
coverage-report.html (or the name you provided with --output) is generated.
Use standard OpenAPI/Swagger files in YAML or JSON format:
# Single OpenAPI specification
swagger-coverage-cli api-spec.yaml collection.json
# Multiple REST APIs
swagger-coverage-cli "api-v1.yaml,api-v2.yaml,legacy.json" collection.json
# With strict validation
swagger-coverage-cli openapi.yaml collection.json --strict-query --strict-body
Supported OpenAPI features:
/users/{id}, /users/{userId})Analyze Protocol Buffer service definitions:
# Single gRPC service
swagger-coverage-cli user-service.proto collection.json
# Multiple gRPC services
swagger-coverage-cli "user.proto,order.proto,payment.proto" collection.json
# Mixed with OpenAPI
swagger-coverage-cli "rest-api.yaml,grpc-service.proto" collection.json
gRPC-specific features:
.proto files/package.service/method)application/grpc, application/grpc+proto)company.api.v1.UserService)Example Postman request for gRPC:
{
"method": "POST",
"url": "{{grpcUrl}}/user.v1.UserService/GetUser",
"header": [
{ "key": "Content-Type", "value": "application/grpc" }
],
"body": {
"mode": "raw",
"raw": "{\"user_id\": \"123\"}"
}
}
Analyze GraphQL schema definitions:
# Single GraphQL API
swagger-coverage-cli schema.graphql collection.json
# Multiple GraphQL schemas
swagger-coverage-cli "user-schema.gql,product-schema.graphql" collection.json
# Full stack coverage
swagger-coverage-cli "api.yaml,service.proto,schema.graphql" collection.json
GraphQL-specific features:
Example Postman request for GraphQL:
{
"method": "POST",
"url": "{{apiUrl}}/graphql",
"header": [
{ "key": "Content-Type", "value": "application/json" }
],
"body": {
"mode": "raw",
"raw": "{\"query\": \"query GetUser($id: ID!) { user(id: $id) { id name email } }\", \"variables\": {\"id\": \"123\"}}"
}
}
Use CSV format for flexible API documentation:
# CSV-based API documentation
swagger-coverage-cli api-docs.csv collection.json
# Mixed with other formats
swagger-coverage-cli "api.yaml,docs.csv,service.proto" collection.json
CSV format columns:
method: HTTP method (GET, POST, etc.)path: API endpoint pathstatusCode: Expected response status codedescription: Operation descriptiontags: Comma-separated tags for groupingMicroservices Architecture:
# Complete microservices stack
swagger-coverage-cli "gateway.yaml,user-service.proto,analytics.graphql,docs.csv" tests.json
# Per-team analysis
swagger-coverage-cli "team-a-api.yaml,team-b-service.proto" team-tests.json
CI/CD Integration:
# Production coverage check
swagger-coverage-cli "$(find apis -name '*.yaml' -o -name '*.proto' -o -name '*.graphql' | tr '\n' ',')" collection.json --output coverage-$(date +%Y%m%d).html
swagger-coverage-cli uses precise mathematical formulas to calculate API test coverage. Understanding these formulas helps you interpret coverage reports and set appropriate coverage targets.
The core coverage calculation is based on the ratio of matched operations to total operations in your API specification:
Coverage (%) = (Matched Operations / Total Operations) × 100
Where:
An operation is considered matched when ALL of the following criteria are satisfied:
specOperation.method === postmanRequest.method--strict-query enabled): Required parameters are present and valid--strict-body enabled): JSON body structure matches specificationStandard Coverage = Matched Operations / Total Spec Operations × 100
When using Postman collections, coverage is calculated based on:
pm.response.to.have.status(code) assertionsWhen using Newman reports, coverage calculation includes:
Newman Coverage = (Executed Operations with Passing Tests / Total Spec Operations) × 100
For multiple API specifications, coverage is calculated as:
Combined Coverage = (Sum of Matched Operations across all APIs / Sum of Total Operations across all APIs) × 100
Each API operation is identified by: {API_Name}:{Method}:{Path}:{StatusCode}
Coverage can also be calculated per tag/group:
Tag Coverage = (Matched Operations in Tag / Total Operations in Tag) × 100
Beyond basic percentage, consider these quality indicators:
Sometimes API specifications may have validation errors or broken references, especially in legacy systems or during development. The --disable-spec-validation flag allows you to analyze coverage even when specs have issues.
Use --disable-spec-validation when:
$ref references that can't be resolved# Standard usage (validation enabled - will fail on invalid specs)
swagger-coverage-cli api.yaml collection.json
# Disable validation for specs with issues
swagger-coverage-cli api.yaml collection.json --disable-spec-validation
# Works with all other flags
swagger-coverage-cli api.yaml collection.json --disable-spec-validation --verbose --strict-body
Without the flag (validation enabled):
$ swagger-coverage-cli broken-spec.yaml collection.json
Error: Token "NonExistentSchema" does not exist.
With the flag (validation disabled):
$ swagger-coverage-cli broken-spec.yaml collection.json --disable-spec-validation
=== Swagger Coverage Report ===
Total operations in spec(s): 12
Matched operations in Postman/Newman: 9
Coverage: 75.00%
HTML report saved to: coverage-report.html
swagger-coverage-cli tries to match each operation from the spec with a request in Postman. An operation is considered covered if:
HTTP Method matches exactly (GET, POST, PUT, DELETE, etc.).
Path:
/users/{id}) is converted to a regex (like ^/users/[^/]+$).{{baseUrl}}) must match that regex.Status Code:
200, 404), the CLI checks the Postman test scripts to see if that status code is asserted (e.g., pm.response.to.have.status(200)).Query Parameters (only if --strict-query is enabled):
enum, pattern, type), the tool verifies that the Postman request includes that parameter and meets the constraints.Request Body (only if --strict-body is enabled):
requestBody includes application/json, the CLI checks if the Postman request body is raw JSON and can be parsed without errors.If all criteria are satisfied, the operation is matched (covered). Otherwise, it’s reported as unmatched.
Smart endpoint mapping is an advanced feature that significantly improves coverage accuracy by using intelligent algorithms to match endpoints. It is enabled by default in all operations.
# Smart mapping is enabled by default
swagger-coverage-cli api-spec.yaml collection.json --verbose
# Output shows smart mapping statistics:
# Smart mapping: 6 primary matches, 3 secondary matches
# Coverage: 50.00%
Status Code Intelligence:
# API defines multiple status codes
GET /users:
responses:
'200': { description: Success }
'400': { description: Bad Request }
'500': { description: Server Error }
# Postman only tests success case
pm.test("Status code is 200", function () {
pm.response.to.have.status(200);
});
# Smart mapping result:
# ✅ Primary Match: GET /users (200) - Matched
# ❌ Secondary: GET /users (400, 500) - Unmatched but deprioritized
Enhanced Path Matching:
# API Spec: /users/{userId}/profile
# Postman: /users/123/profile
# Result: ✅ Intelligent parameter matching (confidence: 1.0)
For comprehensive examples, use cases, and implementation details, see: 📖 Smart Mapping Examples & Use Cases
This document covers:
Swagger/OpenAPI/.csv:
Postman:
Newman Reports:
newman run collection.json --reporters json --reporter-json-export newman-report.jsonThe tool supports two types of input for test data:
Recommendation: Use Newman reports when possible for more accurate coverage analysis, especially in CI/CD pipelines where collections are actually executed.
swagger-coverage-cli fully supports folder-level tests in Postman collections. Tests defined at the folder level are automatically applied to all requests within that folder and its subfolders, making it easy to apply common test assertions across multiple endpoints.
When you define tests at the folder level in your Postman collection:
Consider this Postman collection structure:
{
"name": "Users API",
"item": [
{
"name": "Get User",
"request": { "method": "GET", "url": "/users/1" }
},
{
"name": "Create User",
"request": { "method": "POST", "url": "/users" }
}
],
"event": [
{
"listen": "test",
"script": {
"exec": [
"pm.test('Status code is 200 or 201', function () {",
" pm.expect(pm.response.code).to.be.oneOf([200, 201]);",
"});"
]
}
}
]
}
In this example:
Folder-level tests support the same patterns as request-level tests:
pm.response.to.have.status(200)pm.expect(pm.response.code).to.eql(201)pm.expect(pm.response.code).to.be.oneOf([200, 201, 204])The folder-level test feature handles various edge cases:
In addition to traditional OpenAPI/Swagger specifications, swagger-coverage-cli supports API documentation provided in a CSV format. This allows for a more flexible and easily editable documentation process, especially for teams that prefer spreadsheet-based documentation.
Your CSV file should adhere to the following structure to ensure compatibility with swagger-coverage-cli:
METHOD,URI,NAME,STATUS CODE,BODY,TAGS
METHOD
GET, POST, PUT, DELETE).URI
/api/products).NAME
getProducts).STATUS CODE
200, 400).BODY
TAGS
products).Below is an example of how your documentation.csv should be structured:
METHOD,URI,NAME,STATUS CODE,BODY,TAGS
GET,/api/products,getProducts,200,"{\"products\":\"updated\"}","products"
GET,/api/products,getProducts,400,"{\"error\":\"Invalid query parameters\"}","products"
Contributions are welcome! Please:
git checkout -b feature/something)Feel free to add tests in the test/ folder to cover any new logic.
This project is licensed under the MIT License.
Happy testing! Feel free to open issues or submit pull requests for any improvements.
FAQs
A Node.js CLI tool to measure test coverage of Swagger/OpenAPI specs using Postman collections or Newman run reports. Features smart endpoint mapping with intelligent status code prioritization and enhanced path matching.
The npm package swagger-coverage-cli receives a total of 26 weekly downloads. As such, swagger-coverage-cli popularity was classified as not popular.
We found that swagger-coverage-cli demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
/Security News
Socket researchers discovered nine malicious NuGet packages that use time-delayed payloads to crash applications and corrupt industrial control systems.
Security News
Socket CTO Ahmad Nassri discusses why supply chain attacks now target developer machines and what AI means for the future of enterprise security.
Security News
Learn the essential steps every developer should take to stay secure on npm and reduce exposure to supply chain attacks.