Security News
The Changelog Podcast: Practical Steps to Stay Safe on npm
Learn the essential steps every developer should take to stay secure on npm and reduce exposure to supply chain attacks.
By Sarah Gooding -  Oct 31, 2025
🚀 DAY 5 OF LAUNCH WEEK: Introducing Socket Firewall Enterprise.Learn more →
pytest-railflow-testrail-reporter
Advanced tools
pytest-railflow-testrail-reporter
Generate json reports along with specified metadata defined in test markers.
- Version
- 0.2.5
- Maintainers
- 1
pytest-railflow-testrail-reporter
pytest-railflow-testrail-reporter is a custom reporter which generates JSON report file containing some TestRail-related data.
The reporter helps you to integrate your pytest test with TestRail easily by producing JSON test report files which can be uploaded into TestRail by using powerful Railflow NPM CLI tool or native plugins for Jenkins and TeamCity.
Requirements
In order to use pytest-railflow-testrail-reporter plugin, following prerequisites should be met.
- Python 2.7, 3.4 or greater
- Pytest
Installation
Using Pip
To install the pytest-railflow-testrail-reporter plugin using pip,run the following command in terminal:
pip install pytest-railflow-testrail-reporter
This will install the plugin to the system.
Usage
To add Railflow markers to the tests '@pytest.mark.railflow' marker is used.
The plugin supports the following attributes:
Railflow configuration attributes description
| Attribute Name | Description |
|---|---|
| title | Name of the test suite or test case |
| case_type | Case type in TestRail, e.g.: Automated, Compatibility, etc... |
| case_priority | Case type in TestRail, e.g.: Critical, High, etc... |
| case_fields | Values for custom case fields in TestRail in the following format: ['field1=value1','field2=value2'] |
| result_fields | Values for result fields in TestRail in the following format: ['field1=value1','field2=value2'] |
| jira_ids | Jira IDs.These values will be populated as a case field 'refs'. Should input as an array of strings, e.g.: ['jid1','jid2'] |
| testrail_ids | IDs of test cases in TestRail. Should input as an array of integers, e.g.: [1,2,3] |
| smart_failure_assignment | Array of TestRail users to automatically assign failed test cases. Should input as a string array, e.g.: ['aaa@test.com','bbb@test.com'] |
These attributes can be either used in class level or function level.
| Function level attributes | Class level attributes |
|---|---|
| jira_ids | case_fields |
| case_fields | result_fields |
| result_fields | case_type |
| testrail_ids | case_priority |
| case_type | smart_failure_assignment |
| case_priority | title |
| title |
To run the test, execute the following command in the terminal from the test directory:
pytest --jsonfile output.json
Adding arbitrary attachments into test report
The testrail_add_screenshot fixture can be used to add arbitrary attachments into the test report:
def test_add(testrail_add_screenshot):
a = 3
b = 2
c = a + b
assert c == 5
testrail_add_screenshot("screenshot path")
Adding test steps to test report
The testrail_add_test_step fixture can be used to add test steps into test report:
The fixture has the following parameters:
| Order | Description | Is Mandatory |
|---|---|---|
| 1 | Step Name | Yes |
| 2 | Status Name | Yes |
| 3 | Actual Value | No |
| 4 | Expected Value | No |
def test_multiply(testrail_add_test_step):
res = 8 * 4
testrail_add_test_step("multiply 8 and 4", "PASSED", res, "32")
Examples
Example 1 : pytest tests within a class
- Install 'pytest-railflow-testrail-reporter' .
$ pip install pytest-railflow-testrail-reporter
- Add pytest test with Railfllow marker on class level.
test_calculation.py
import pytest
@pytest.mark.railflow(
jira_ids=["301219"],
case_fields=[
{
"name": "Required text field",
"value": "Class"
}
],
result_fields=[
{
"name": "Custom field",
"value": "Result from class"
}
],
case_type="Automated",
case_priority="Critical",
smart_failure_assignment=["user1@gmail.com, user2@gmail.com"]
)
class TestClass:
def test_add(self):
a = 3
b = 2
c = a + b
assert c == 5
def test_subtract(self):
a = 3
b = 2
c = a - b
assert c == 0
@pytest.mark.railflow(
title = "method title",
jira_ids=["11111"],
case_fields=[
{
"name": "Required text field",
"value": "method"
}
],
result_fields=[
{
"name": "Custom fieLD",
"value": "Result from method"
}
],
case_type="Compatibility",
case_priority="High",
smart_failure_assignment=["user3@gmail.com"]
)
@pytest.mark.parametrize("a,b,c", [(22, 11, 2), (64, 8, 8), (9, 3, 3)])
def test_divide(self, a, b, c):
assert a / b == c
- Run tests and generate report
pytest --jsonfile output.json test_calculation.py
- View report file
Report file generated at the same directory where the test executed.
output.json
[
{
"class_name": "TestClass",
"markers": "",
"file_name": "test_sample2",
"attachments": [],
"tests": [
{
"test_name": "test_add",
"details": null,
"markers": "",
"result": "PASSED",
"duration": 0.00012004900054307655,
"timestamp": "2022-03-09T15:53:53",
"message": null
},
{
"test_name": "test_subtract",
"details": null,
"markers": "",
"result": "FAILED",
"duration": 0.00017888799993670546,
"timestamp": "2022-03-09T15:53:53",
"message": "self = <test_sample2.TestClass object at 0x7f659f8af310>\n\n def test_subtract(self):\n a = 3\n b = 2\n c = a - b\n> assert c == 0\nE assert 1 == 0\n\ntest_sample2.py:33: AssertionError"
},
{
"test_name": "test_divide",
"details": null,
"markers": "parametrize",
"parameters": [
{
"name": "a",
"value": 22
},
{
"name": "b",
"value": 11
},
{
"name": "c",
"value": 2
}
],
"result": "PASSED",
"duration": 0.0001058529996953439,
"timestamp": "2022-03-09T15:53:53",
"message": null,
"railflow_test_attributes": {
"title": "method title",
"jira_ids": [
"11111"
],
"case_fields": [
{
"name": "Required text field",
"value": "method"
}
],
"result_fields": [
{
"name": "Custom field",
"value": "Result from method"
}
],
"case_type": "Compatibility",
"case_priority": "High",
"smart_failure_assignment": [
"user3@gmail.com"
]
}
},
{
"test_name": "test_divide",
"details": null,
"markers": "parametrize",
"parameters": [
{
"name": "a",
"value": 64
},
{
"name": "b",
"value": 8
},
{
"name": "c",
"value": 8
}
],
"result": "PASSED",
"duration": 0.00013493100050254725,
"timestamp": "2022-03-09T15:53:53",
"message": null,
"railflow_test_attributes": {
"title": "method title",
"jira_ids": [
"11111"
],
"case_fields": [
{
"name": "Required text field",
"value": "method"
}
],
"result_fields": [
{
"name": "Custom field",
"value": "Result from method"
}
],
"case_type": "Compatibility",
"case_priority": "High",
"smart_failure_assignment": [
"user3@gmail.com"
]
}
},
{
"test_name": "test_divide",
"details": null,
"markers": "parametrize",
"parameters": [
{
"name": "a",
"value": 9
},
{
"name": "b",
"value": 3
},
{
"name": "c",
"value": 3
}
],
"result": "PASSED",
"duration": 0.00020506200235104188,
"timestamp": "2022-03-09T15:53:53",
"message": null,
"railflow_test_attributes": {
"title": "method title",
"jira_ids": [
"11111"
],
"case_fields": [
{
"name": "Required text field",
"value": "method"
}
],
"result_fields": [
{
"name": "Custom field",
"value": "Result from method"
}
],
"case_type": "Compatibility",
"case_priority": "High",
"smart_failure_assignment": [
"user3@gmail.com"
]
}
}
],
"railflow_test_attributes": {
"jira_ids": [
"301219"
],
"case_fields": [
{
"name": "Required text field",
"value": "Class"
}
],
"result_fields": [
{
"name": "Custom field",
"value": "Result from class"
}
],
"case_type": "Automated",
"case_priority": "Critical",
"smart_failure_assignment": [
"user1@gmail.com, user2@gmail.com"
]
}
}
]
- Install Railflow CLI
npm install railflow
- Run Railflow CLI and upload test results into TestRail
npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path section1/section2 -f pytest-railflow -r pytest_example/*.json -sm path
Where:
| Key | Description | Example |
|---|---|---|
| -k, --key | (online activation) License key. Can be set with RAILFLOW_LICENSE environment variable | -k XXXXX-XXXXX-XXXXX-XXXXX |
| -url, --url | TestRail instance URL | -url https://example.testrail.io |
| -u, --username | TestRail username. Can be set with RAILFLOW_TR_USER environment variable | -u test-username |
| -p, --password | TestRail password or API Key. Can be set with RAILFLOW_TR_PASSWORD environment variable | -p XtpHXiPLEODyhF |
| -pr, --project | TestRail project name | -pr "example project" |
| -path, --test-path | TestRail test cases path | -path "Section1/subsection2/ShoppingCart |
| -f, --format | Report format: 'pytest-railflow' (case insensitive) | -f junit |
| -r, --report-files | The file path(s) to the test report file(s) generated during the build. User can pass multiple values separated with spaces. Ant-style patterns such as **/reports/*.xml can be used. E.g. use target/reports/*.xml to capture all XML files in target/reports directory. | -r target/surefire-reports/*.xml target/failsafe-reports/*.xml |
| -sm, --search-mode | Specifies the test case lookup algorithm. name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified -path path: search for test case matching the name within the specified -path. If test case found, update the test case. If test case not found, create a new test case within specified -path | -sm path |
Please see Railflow NPM documentation for the all the details about Railflow CLI options.
- View results in TestRail
Test run
Test result details
Parameterized tests details
Example 2 : pytest tests without a class (using pytest splinter)
- Install 'pytest-splinter' .
$ pip install pytest-splinter
Also, make sure that chromedriver or Firefox driver is installed in your system for running splinter.
For Firefox driver: [Splinter Documentation on Firefox Driver](https://splinter.readthedocs.io/en/latest/drivers/firefox.html)
For Chrome driver : [SPlinter Documentation on Chrome Driver](https://splinter.readthedocs.io/en/latest/drivers/chrome.html)
2. Add pytest test.
test_browser.py
import pytest
@pytest.mark.railflow(
jira_ids=["301219"],
case_fields=[
{
"name": "Required text field",
"value": "Class"
}
],
result_fields=[
{
"name": "Custom field",
"value": "Result from class"
}
],
case_type="Automated",
case_priority="Critical",
smart_failure_assignment=["user1@gmail.com, user2@gmail.com"]
)
def test_google(browser):
"""Test using real browser."""
url = "https://www.google.com"
browser.visit(url)
browser.fill('q', 'splinter - python acceptance testing for web applications')
# Find and click the 'search' button
button = browser.find_by_name('btnK')
# Interact with elements
button.click()
assert browser.is_text_present('splinter.cobrateam.info'), "splinter.cobrateam.info wasn't found... We need to"
' improve our SEO techniques'
- Run tests and generate report (using chrome driver) if '--splinter-webdriver' argument is not provided , it will use firefox as default web driver
pytest --splinter-webdriver chrome --jsonfile output.json test_browser.py
- View report file
output.json
[
{
"class_name": null,
"test_name": "test_google",
"details": "Test using real browser.",
"markers": "",
"result": "FAILED",
"duration": 8.737954783000077,
"timestamp": "2022-03-06T10:51:51",
"message": "browser = <splinter.driver.webdriver.chrome.WebDriver object at 0x7f55c6f50670>\n\n @pytest.mark.railflow(\n jira_ids=[\"301219\"],\n case_fields=[\n {\n \"name\": \"ReQuired text field\",\n \"value\": \"Class\"\n }\n ],\n result_fields=[\n {\n \"name\": \"Custom fIeLD\",\n \"value\": \"Result from class\"\n }\n ],\n case_type=\"Railflow\",\n case_priority=\"Critical\",\n smart_failure_assignment=[\"user1@gmail.com, user2@gmail.com\"]\n )\n def test_google(browser):\n \"\"\"Test using real browser.\"\"\"\n url = \"https://www.google.com\"\n browser.visit(url)\n browser.fill('q', 'splinter - python acceptance testing for web applications')\n # Find and click the 'search' button\n button = browser.find_by_name('btnK')\n # Interact with elements\n button.click()\n> assert browser.is_text_present('splinter.cobrateam.info'), \"splinter.cobrateam.info wasn't found... We need to\"\nE AssertionError: splinter.cobrateam.info wasn't found... We need to\nE assert False\nE + where False = <bound method BaseWebDriver.is_text_present of <splinter.driver.webdriver.chrome.WebDriver object at 0x7f55c6f50670>>('splinter.cobrateam.info')\nE + where <bound method BaseWebDriver.is_text_present of <splinter.driver.webdriver.chrome.WebDriver object at 0x7f55c6f50670>> = <splinter.driver.webdriver.chrome.WebDriver object at 0x7f55c6f50670>.is_text_present\n\ntest_browser.py:30: AssertionError",
"file_name": "test_browser",
"attachments": [
" /home/projects/pytest/example_tests/railflow_pytest_examples/railflow_pytest_examples.test_browser/test_google-browser.png"
],
"railflow_test_attributes": {
"jira_ids": [
"301219"
],
"case_fields": [
{
"name": "Required text field",
"value": "Class"
}
],
"result_fields": [
{
"name": "Custom field",
"value": "Result from class"
}
],
"case_type": "Railflow",
"case_priority": "Critical",
"smart_failure_assignment": [
"user1@gmail.com, user2@gmail.com"
]
}
}
]
- Run Railflow CLI and upload test results into TestRail
npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path section1/section2 -f pytest-railflow -r pytest_example/*.json
Please check examples for more information and sample tests.
License
This software is licensed under the MIT license
See License file for more information.
FAQs
Generate json reports along with specified metadata defined in test markers.
We found that pytest-railflow-testrail-reporter demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Related posts
Security News
Security Community Slams MIT-linked Report Claiming AI Powers 80% of Ransomware
Experts push back on new claims about AI-driven ransomware, warning that hype and sponsored research are distorting how the threat is understood.
By Sarah Gooding -  Oct 30, 2025
Security News
Ruby Core Team Assumes Stewardship of RubyGems and Bundler, Former Maintainers Offer to Transfer All Rights to Matz
Ruby's creator Matz assumes control of RubyGems and Bundler repositories while former maintainers agree to step back and transfer all rights to end the dispute.
By Sarah Gooding -  Oct 29, 2025