mattermost-ctrf

MatterMost Test Results Notification

Send MatterMost message with test result from popular testing frameworks

Help us grow CTRF

⭐ If you find this project useful, please consider following the CTRF organisation and giving this repository a star ⭐

It means a lot to us and helps us grow this open source library.

Features

• Send Test Results to MatterMost: Automatically send test results to a MatterMost channel.
• Send Flaky Test Details to MatterMost: Automatically send flaky test details to a MatterMost channel.
• Conditional Notifications: Use the --onFailOnly option to send notifications only if tests fail.

Setup

You'll need a CTRF report generated by your testing framework. CTRF reporters are available for most testing frameworks and easy to install.

No CTRF reporter? No problem!

Use junit-to-ctrf to convert a JUnit report to CTRF.

Create a MatterMost Incoming Webhook

Go to the MatterMost API: Incoming Webhooks page and follows instructions to setup an incoming webhook.

Set the Environment Variable

Set the webhook URL as an environment variable in your shell or CI environment:

export MATTERMOST_WEBHOOK_URL='https://your/webhook/url'

Make sure to replace 'https://your/webhook/url' with your actual webhook URL.

You might want to store the webhook URL as a secret.

Usage

To send the test results summary to MatterMost:

npx mattermost-ctrf results /path/to/ctrf-report.json

To send flaky test report to MatterMost:

npx mattermost-ctrf flaky /path/to/ctrf-report.json

Send Only on Failures

To send the test results summary to MatterMost only if there are failed tests, use the --onFailOnly option:

npx mattermost-ctrf results /path/to/ctrf-file.json --onFailOnly

Options

• --onFailOnly, -f: Send notification only if there are failed tests.

Merge reports

You can merge reports if your chosen reporter generates multiple reports through design, parallelisation or otherwise.

The ctrf-cli package provides a method to merge multiple ctrf json files into a single file.

After executing your tests, use the following command:

npx ctrf merge <directory>

Replace directory with the path to the directory containing the CTRF reports you want to merge.

What is CTRF?

CTRF is a universal JSON test report schema that addresses the lack of a standardized format for JSON test reports.

Consistency Across Tools: Different testing tools and frameworks often produce reports in varied formats. CTRF ensures a uniform structure, making it easier to understand and compare reports, regardless of the testing tool used.

Language and Framework Agnostic: It provides a universal reporting schema that works seamlessly with any programming language and testing framework.

Facilitates Better Analysis: With a standardized format, programatically analyzing test outcomes across multiple platforms becomes more straightforward.

Support Us

If you find this project useful, consider giving it a GitHub star ⭐ It means a lot to us.