Generate AI summaries of test results using a wide range of AI models like OpenAI, Anthropic, Gemini, Mistral, Grok, DeepSeek, Azure, Perplexity, and OpenRouter
AI Test Reporter is a powerful tool that generates intelligent summaries of test results using a wide range of AI models. With access to over 300 models through various providers (OpenAI, Anthropic Claude, Google Gemini, Mistral, Perplexity, OpenRouter, and more), it analyzes failing tests and provides actionable insights about what went wrong and how to fix it.
You can support the project with a follow and a star
Maintained by Matthew Thomas
Contributions are very welcome!
Explore more integrations
[!TIP] The following integrations are available with free tiers so are great to try out AI test reporter:
- Google Gemini: Offers a free tier with lower rate limits. Pricing details
- Mistral AI: Provides a free API tier to explore the service. Tier details
You can use any of the models supported by the following providers:
You use your own API keys for the models you select.
Generate a CTRF report using your testing framework. CTRF reporters are available for most testing frameworks and easy to install.
No CTRF reporter? No problem!
Use junit-to-ctrf to convert a JUnit report to CTRF
Run the following command:
npx ai-ctrf openai <path-to-ctrf-report>
An AI summary for each failed test will be added to your test report.
The package interacts with the OpenAI API, you must set OPENAI_API_KEY environment variable.
You will be responsible for any charges incurred from using your selected OpenAI model. Make sure you are aware of the associated cost.
A message is sent to OpenAI for each failed test.
--model: OpenAI model to use (default: gpt-3.5-turbo).
--systemPrompt: Custom system prompt to guide the AI response.
--frequencyPenalty: OpenAI frequency penalty parameter (default: 0).
--maxTokens: Maximum number of tokens for the response.
--presencePenalty: OpenAI presence penalty parameter (default: 0).
--temperature: Sampling temperature (conflicts with topP).
--topP: Top-p sampling parameter (conflicts with temperature).
--log: Whether to log the AI responses to the console (default: true).
--maxMessages: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)
consolidate: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)
Run the following command:
npx ai-ctrf azure-openai <path-to-ctrf-report>
An AI summary for each failed test will be added to your test report.
The package interacts with the Azure OpenAI API, you must set AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, and AZURE_OPENAI_DEPLOYMENT_NAME environment variable or provide them as arguments.
You will be responsible for any charges incurred from using your selected Azure OpenAI model. Make sure you are aware of the associated cost.
A message is sent to Azure OpenAI for each failed test.
--model: OpenAI model to use (default: gpt-3.5-turbo).
--systemPrompt: Custom system prompt to guide the AI response.
--frequencyPenalty: OpenAI frequency penalty parameter (default: 0).
--maxTokens: Maximum number of tokens for the response.
--presencePenalty: OpenAI presence penalty parameter (default: 0).
--temperature: Sampling temperature (conflicts with topP).
--topP: Top-p sampling parameter (conflicts with temperature).
--log: Whether to log the AI responses to the console (default: true).
--maxMessages: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)
consolidate: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)
Run the following command:
npx ai-ctrf claude <path-to-ctrf-report>
An AI summary for each failed test will be added to your test report.
The package interacts with the Anthropic API, you must set ANTHROPIC_API_KEY environment variable.
You will be responsible for any charges incurred from using your selected Claude model. Make sure you are aware of the associated costs.
A message is sent to Claude for each failed test.
--model: Claude model to use (default: claude-3-5-sonnet-20240620).
--systemPrompt: Custom system prompt to guide the AI response.
--maxTokens: Maximum number of tokens for the response.
--temperature: Sampling temperature.
--log: Whether to log the AI responses to the console (default: true).
--maxMessages: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)
consolidate: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)
Run the following command:
npx ai-ctrf grok <path-to-ctrf-report>
An AI summary for each failed test will be added to your test report.
The package interacts with the Grok API, you must set GROK_API_KEY environment variable.
You will be responsible for any charges incurred from using your selected Grok model. Make sure you are aware of the associated cost.
A message is sent to Grok for each failed test.
--model: Grok model to use (default: grok-1).
--systemPrompt: Custom system prompt to guide the AI response.
--maxTokens: Maximum number of tokens for the response.
--temperature: Sampling temperature.
--log: Whether to log the AI responses to the console (default: true).
--maxMessages: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)
consolidate: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)
Run the following command:
npx ai-ctrf deepseek <path-to-ctrf-report>
An AI summary for each failed test will be added to your test report.
The package interacts with the DeepSeek API, you must set DEEPSEEK_API_KEY environment variable.
You will be responsible for any charges incurred from using your selected DeepSeek model. Make sure you are aware of the associated cost.
A message is sent to DeepSeek for each failed test.
--model: DeepSeek model to use (default: deepseek-coder).
--systemPrompt: Custom system prompt to guide the AI response.
--maxTokens: Maximum number of tokens for the response.
--temperature: Sampling temperature.
--log: Whether to log the AI responses to the console (default: true).
--maxMessages: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)
consolidate: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)
Run the following command:
npx ai-ctrf mistral <path-to-ctrf-report>
An AI summary for each failed test will be added to your test report.
The package interacts with the Mistral API, you must set MISTRAL_API_KEY environment variable.
You will be responsible for any charges incurred from using your selected Mistral model. Make sure you are aware of the associated cost.
A message is sent to Mistral for each failed test.
Mistral offers free API credits upon signup, making it a great option to test the tool without immediate costs.
--model: Mistral model to use (default: mistral-medium).
--systemPrompt: Custom system prompt to guide the AI response.
--maxTokens: Maximum number of tokens for the response.
--temperature: Sampling temperature.
--topP: Top-p sampling parameter.
--log: Whether to log the AI responses to the console (default: true).
--maxMessages: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)
consolidate: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)
Run the following command:
npx ai-ctrf gemini <path-to-ctrf-report>
An AI summary for each failed test will be added to your test report.
The package interacts with the Google Gemini API, you must set GOOGLE_API_KEY environment variable.
You will be responsible for any charges incurred from using your selected Gemini model. Make sure you are aware of the associated cost.
A message is sent to Gemini for each failed test.
Google offers free API credits for Gemini, providing a cost-effective way to try out the tool.
--model: Gemini model to use (default: gemini-pro).
--systemPrompt: Custom system prompt to guide the AI response.
--maxTokens: Maximum number of tokens for the response.
--temperature: Sampling temperature.
--topP: Top-p sampling parameter.
--log: Whether to log the AI responses to the console (default: true).
--maxMessages: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)
consolidate: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)
Run the following command:
npx ai-ctrf perplexity <path-to-ctrf-report>
An AI summary for each failed test will be added to your test report.
The package interacts with the Perplexity API, you must set PERPLEXITY_API_KEY environment variable.
You will be responsible for any charges incurred from using your selected Perplexity model. Make sure you are aware of the associated cost.
A message is sent to Perplexity for each failed test.
--model: Perplexity model to use (default: pplx-7b-online).
--systemPrompt: Custom system prompt to guide the AI response.
--maxTokens: Maximum number of tokens for the response.
--temperature: Sampling temperature.
--topP: Top-p sampling parameter.
--log: Whether to log the AI responses to the console (default: true).
--maxMessages: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)
consolidate: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)
Run the following command:
npx ai-ctrf openrouter <path-to-ctrf-report>
An AI summary for each failed test will be added to your test report.
The package interacts with the OpenRouter API, you must set OPENROUTER_API_KEY environment variable.
OpenRouter provides access to multiple AI models through a single API, including models from Anthropic, OpenAI, Google, Meta, and more.
You will be responsible for any charges incurred from using your selected OpenRouter model. Make sure you are aware of the associated cost.
A message is sent to OpenRouter for each failed test.
--model: OpenRouter model to use (default: anthropic/claude-3-opus). Available models include:
--systemPrompt: Custom system prompt to guide the AI response.
--maxTokens: Maximum number of tokens for the response.
--temperature: Sampling temperature.
--topP: Top-p sampling parameter.
--frequencyPenalty: Frequency penalty parameter.
--presencePenalty: Presence penalty parameter.
--log: Whether to log the AI responses to the console (default: true).
--maxMessages: Limit the number of failing tests to send for summarization in the LLM request. This helps avoid overwhelming the model when dealing with reports that have many failing tests. (default: 10)
consolidate: Consolidate and summarize multiple AI summaries into a higher-level overview (default: true)
The AI model analyzes information to:
When consolidation is enabled (--consolidate), the AI analyzes all test failures AI summaries together to provide a high-level summary of issues.
For each failing test, the AI receives the complete test object.
The complete environment details from the environment object in the CTRF report is provided to the AI model.
The complete tool details from the tool object in the CTRF report is provided to the AI model.
You are responsible for any charges incurred from using the AI models. Make sure you are aware of the associated costs.
{
"results": {
"tool": {
"name": "AnyFramework"
},
"summary": {
"tests": 1,
"passed": 0,
"failed": 1,
"pending": 0,
"skipped": 0,
"other": 1,
"start": 1722511783500,
"stop": 1722511804528
},
"tests": [
{
"name": "should display profile information",
"status": "failed",
"duration": 800,
"message": "Assertion Failure: profile mismatch",
"trace": "ProfileTest.js:45...",
"ai": "The test failed because there was a profile mismatch at line 45 of the ProfileTest.js file. To resolve this issue, review the code at line 45 to ensure that the expected profile information matches the actual data being displayed. Check for any discrepancies and make necessary adjustments to align the expected and actual profile information."
},
]
}
}
View AI summaries in directly in the Github Actions workflow:
Add a Pull Request comment with your AI summary:
Send a Slack message with your AI test summary:
If you find this project useful, consider giving it a GitHub star ⭐ It means a lot to us.
FAQs
Generate AI summaries of test results using a wide range of AI models like OpenAI, Anthropic, Gemini, Mistral, Grok, DeepSeek, Azure, Perplexity, and OpenRouter
The npm package ai-ctrf receives a total of 0 weekly downloads. As such, ai-ctrf popularity was classified as not popular.
We found that ai-ctrf demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
require(esm) backported to Node.js 20, easing the transition to ESM-only packages and reducing complexity for developers as Node 18 nears end-of-life.
Security News
PyPI now supports iOS and Android wheels, making it easier for Python developers to distribute mobile packages.
Security News
Create React App is officially deprecated due to React 19 issues and lack of maintenance—developers should switch to Vite or other modern alternatives.