cucumber_testrail

CucumberTestrail

Overview

This gem helps you synchronise your Cucumber test suites into Testrail. It uses tags to link Testrail projects, test suites, testcases and write reports for testruns into Testrail. It will also create new testcases if it can't find them in Testrail and update your feature files to add the tag to the scenario. It writes the step definition into the testcase and keeps it updated.

The central idea is that tags in the features files are used to link your scenarios to Testrail. The identifiers used are 'project','suite','testcase','testrun' they are all looked up from tags on the scenario, tags on the feature or environment variables. E.g. on the scenario you might have a tag @testcase_99, this will link to the Testrail tescase with id=99.

In Testrail, tests are instances of testcases within the context of a testrun. Therefore to report on a test instance you need a combination of testcase and testrun. The testrun can be one of a tag at scenario level, a tag at feature level or and environment level, e.g. a tag @testrun_5 or environment variable TESTRUN=5. The combination of testcase and testrun form a Testrail test and the result of the senario is add to a list of results for the Testrail test. How you allocate test suites, testcases and testruns is entirely up to you.

Installation

The gem is hosted our own gemserver at http://ec2-54-77-102-63.eu-west-1.compute.amazonaws.com:9292 so you need to add these lines to your Gemfile

Add this line to your application's Gemfile:

source "http://ec2-54-77-102-63.eu-west-1.compute.amazonaws.com:9292"
gem 'cucumber_testrail',"~>1.1.0" #whatever is the latest version

$ bundle

Basic usage

Once you've installed the gem and bundled it, you need to add the methods in the gem to the World object. In env.rb add

 World CucumberTestrail

Then you need to add the code to After scenario hooks

 send_result(scenario)

Then state which testrun you're operating in. This can be either a tag e.g. @testrun_5 at the scenario or feature level, or in the environment TESTRUN=5

Set up environment variables for the Testrail URL, username and password

 export TESTRAIL__USER="your-testrail-user"
 export TESTRAIL_PASSWORD="your-testrail-password"
 export TESTRAIL_BASE_URL="your-testrail-server"

What if I don't have a testcase id?

The gem will try to find one that matches the title and if it can't it'll create a new testcase. It then takes the testcase id and adds a tag to your feature file. That requires at least one line in between each scenario, but you do that anyway.

To create the testcase it needs to know the project id suite id and the sub section id. Those should be added as a tags at feature level e.g. @project_2 @suite_10 @sub_section_14

Outlines!!

Each line in the Examples section of a Scenario Outline will be mapped to a unique testcase using the scenario title, and the named variables in the examples. e.g.

 Scenario Outline: A title
   Given some steps
   Examples: some example lines
    |heading 1 | heading 2 |
    | ex1      | ex2       |

Will be mapped to a testcase with title "A title : heading 1='ex1', heading 2='ex2'". Because we will have many testcases for a single scenario outline then the gem doesn't add testcase tags

What if I don't want to send a report to Testrail

There's a special tag @ignore_testrail and environment variable IGNORE_TESTRAIL=true. So you can specify one scenario should not be linked to Testrail, one feature file, or the whole run.

What if I want to tag it with a jira ticket number?

Add the tag @jira_STORE-123 for example. This will then appear in the defect section of the test report as STORE-123. You can add multiple jira tags to a scenario

Synchronising Testrail testcases to Cucumber scenarios?

The gem relies first and foremost on the tag @testcase_id. If that tag is present it will use that id without further checks. When it writes the result of the test it also updates the title of the testcase to match the title of the scenario and the contents of the testcase steps to match the scenario steps. It overwrites these without any checks. That means if you have a Testrail testcase 5, with title 'Check that the sun comes up in the morning' and you give a tag @testcase_5 to a scenario titled 'The user should be logged in', then at the end of the run that Testrail testcase will now have the title taken from the scenario.