@alex_neo/playwright-azure-reporter

Playwright Azure Reporter

A must read!

Since version 1.5.0 reporter allows using configurationIds to publish results for different configurations e.g. different browsers Necessarily defining testRun.configurationIds or/and testPointMapper function in reporter config, otherwise reporter will be publishing results for all configurations

Since version 1.9.0 reporter allows you to use test tags as Playwright it implemented in version 1.42.0 You can define test cases ids in new format, but you still can use old format with test case id in test name

Example:

test.describe('Test suite', () => {
  test('Test name @tag1 @tag2', {
    tag: ['@[1]'] // <<-- test case id
  } () => {
    expect(true).toBe(true);
  });
});

but you should define your Azure DevOps test case id in format @[1] where 1 is your test case id in square brackets and @ is required prefix for playwright to recognize tags

How to integrate

Install package

npm install @alex_neo/playwright-azure-reporter

or

yarn add @alex_neo/playwright-azure-reporter

Usage

You must register an ID already existing test cases from Azure DevOps before running tests.

You need write testCaseId wraped in square brackets at the test name.

You can define multiple test cases for a single test with next format:

• [1] Test name - single test case
• [1,2,3] Test name - multiple test cases
• [16, 17, 18] Test name - multiple test cases with spaces
• [1, 2, 3] Test name [4] Test name [5][6] Test name - with combined format

For example:

describe('Test suite', () => {
  test('[1] First Test', () => {
    expect(true).toBe(true);
  });

  test('Correct test [3]', () => {
    expect(true).toBe(true);
  });

  test.skip('Skipped test [4]', () => {
    expect(true).toBe(true);
  });

  test('[6] Failed test', () => {
    expect(true).toBe(false);
  });

  test('[7] Test seven [8] Test eight [9] Test nine', () => {
    expect(true).toBe(true);
  });

  test('[10,11,12] Test ten, eleven, twelve', () => {
    expect(true).toBe(true);
  });

  test('[13, 14, 15] Test thirteen, fourteen, fifteen', () => {
    expect(true).toBe(true);
  });

  test('[16, 17, 18] Test sixteen, seventeen, eighteen [19] Test nineteen', () => {
    expect(true).toBe(true);
  });
});

Or you can use tags to define test cases ids (since v1.9.0) (read more here):

test.describe('Test suite', () => {
  test('Test name', {
    tag: ['@[1]', '@smoke', '@slow']
  } () => {
    expect(true).toBe(true);
  });
});

Configure Playwright Azure Reporter with playwright-azure-reporter package.

playwright.config.ts

import { PlaywrightTestConfig } from '@playwright/test';
import { AzureReporterOptions } from '@alex_neo/playwright-azure-reporter/dist/playwright-azure-reporter';

/**
 * See https://playwright.dev/docs/test-configuration.
 */
const config: PlaywrightTestConfig = {
  testDir: './tests',
  timeout: 30 * 1000,
  expect: {
    timeout: 5000,
  },
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  reporter: [
    ['list'],
    [
      '@alex_neo/playwright-azure-reporter',
      {
        orgUrl: 'https://dev.azure.com/your-organization-name',
        token: process.env.AZURE_TOKEN,
        planId: 44,
        projectName: 'SampleSample',
        environment: 'AQA',
        logging: true,
        testRunTitle: 'Playwright Test Run',
        publishTestResultsMode: 'testRun',
        uploadAttachments: true,
        attachmentsType: ['screenshot', 'video', 'trace'],
        testCaseIdMatcher: /@\[(\d+)\]/,
        testRunConfig: {
          owner: {
            displayName: 'Alex Neo',
          },
          comment: 'Playwright Test Run',
          // the configuration ids of this test run, use
          // https://dev.azure.com/{organization}/{project}/_apis/test/configurations to get the ids of  your project.
          // if multiple configuration ids are used in one run a testPointMapper should be used to pick the correct one,
          // otherwise the results are pushed to all.
          configurationIds: [1],
        },
      } as AzureReporterOptions,
    ],
  ],
  use: {
    screenshot: 'only-on-failure',
    actionTimeout: 0,
    trace: 'on-first-retry',
  },
};

export default config;

Configuration

Reporter options (* - required):

• *token - Azure DevOps token, you can find more information here

• *orgUrl - Full url for your organization space. Example: https://dev.azure.com/your-organization-name

  Note: some API's (e.g. ProfileApi) can't be hit at the org level, and has to be hit at the deployment level, so url should be structured like https://vssps.dev.azure.com/{yourorgname}

• *projectName - Name of your project (also can be got from run URL). Example: https://dev.azure.com/alex-neo/SampleProject/ - SampleProject

• *planId - Id of test plan. You can find it in test plan URL. Example: https://dev.azure.com/alex-neo/SampleProject/_testPlans/execute?planId=4&suiteId=6 - planId=4

• environment - Any string that will be used as environment name. Will be used as prefix for all test runs. Default: undefined. Example: QA

• logging [true/false] - Enabled debug logging from reporter or not. Default: false.

• uploadAttachments [true/false] - Uploading attachments (screenshot/video) after test ended. Default: false.

• attachmentsType - List of attachments types or a RegEx to match the name of the attachment that will be uploaded. Default: ['screenshot']

• isDisabled [true/false] - Disable reporter. Default: false.

• testRunTitle - Title of test run using to create new test run. Default: Playwright Test Run.

• testRunConfig - Extra data to pass when Test Run creating. Read doc from more information. Default: empty.

• testPointMapper - A callback to map the test runs to test configurations, e.g. by browser

  import { TestCase } from '@playwright/test/reporter'
  import { TestPoint } from 'azure-devops-node-api/interfaces/TestInterfaces'

  testPointMapper: async (testCase: TestCase, testPoints: TestPoint[]) => {
    switch(testCase.parent.project()?.use.browserName) {
      case 'chromium':
        return testPoints.filter((testPoint) => testPoint.configuration.id === '3');
      case 'firefox':
        return testPoints.filter((testPoint) => testPoint.configuration.id === '4');
      case 'webkit':
        return testPoints.filter((testPoint) => testPoint.configuration.id === '5');
      default:
        throw new Error("invalid test configuration!");
    }
  }

• publishTestResultsMode - Mode of publishing test results. Default: 'testResult'. Available options:

  • testResult - Published results of tests, at the end of each test, parallel to test run..
  • testRun - Published test results to test run, at the end of test run.

    Note: If you use testRun mode and using same test cases in different tests (yes i know it sounds funny), it will be overwritten with last test result.

• isExistingTestRun [true/false] - Published test results to the existing test run. In this mode test results only added to the existing test run without its creation and completion. Default: false.

  Note: If you use isExistingTestRun mode, testRunId should be specified.

• testRunId [number] - Id of test run. Used only for existingTestRun publishing mode. Also can be set by AZURE_PW_TEST_RUN_ID environment variable. Default: undefined.

  Note: If you set existing test run ID from reporter options and from environment variable - reporter options will be used

  Note: If you use isExistingTestRun mode, test run doesn't complete automatically. You should complete it manually.

• testCaseIdMatcher [string|RegExp|string[]|RegExp[]] - A string or a regular expression to match the name of the test case to extract the test case id. Default: /\[([\d,\s]+)\]/

  Example Test Titles

  • Test title: Test case @tag1=123

    • testCaseIdMatcher: /@tag1=(\d+)/
    • Extracted tags: ['123']

  • Test title: Test case @TestCase=123 [@TestCase=456]

    • testCaseIdMatcher: /@TestCase=(\d+)/
    • Extracted tags: ['123', '456']

  • Test title: Test case test123 TEST456

    • testCaseIdMatcher: [/[a-z]+(\d+)/, /[A-Z]+(\d+)/]
    • Extracted tags: ['123', '456']

  • Test title: Test case @tag1=123 @tag2=456

    • testCaseIdMatcher: ['@tag1=(\\d+)', '@tag2=(\\d+)']
    • Extracted tags: ['123', '456']

  Error Handling

  If an invalid testCaseIdMatcher is provided, an error will be thrown. For example:

   reporter: [
    ['list'],
    [
      '@alex_neo/playwright-azure-reporter',
      {
        orgUrl: 'http://localhost:4000',
        projectName: 'SampleProject',
        planId: 4,
        token: 'your-token',
        isDisabled: false,
        testCaseIdMatcher: 1234, // Invalid pattern
      }
    ],
  // This will throw an error: "Invalid testCaseIdMatcher. Must be a string or RegExp. Actual: 1234"
  

• testCaseIdZone [string] - Specifies where to look for the test case IDs. It can be either 'title' or 'annotation'. When set to 'title', the reporter will extract test case IDs from the test title and tag test section also. When set to 'annotation', it will extract test case IDs only from the test annotations. Default: 'title'.

  Pay attention that if you use testCaseIdZone: 'annotation' and testCaseIdMatcher is not defined, the reporter will not extract test case IDs from the test annotations. You should define testCaseIdMatcher to extract test case IDs from the test annotations. Matcher should match the annotation type not the annotation description!

  Example Usage

  • Test title: Test case [12345]

    • testCaseIdZone: 'title'
    • Extracted tags: ['12345']

  • Test annotations:

    test('Test case', { annotations: [{ type: 'TestCase', description: '12345' }] }, () => {
      expect(true).toBe(true);
    });
    

    • testCaseIdZone: 'annotation'
    • testCaseIdMatcher: /(TestCase)/
    • Extracted tags: ['12345']]

Usefulness

• AZURE_PW_TEST_RUN_ID - Id of current test run. It will be set in environment variables after test run created. Can be accessed by process.env.AZURE_PW_TEST_RUN_ID. Pay attention what publishTestResultsMode configuration you use. If you use testResult mode - this variable will be set when test run created, at the start of tests execution, if you use testRun mode - this variable will be set when test run completed, at the end of tests execution.

  Since version 1.10.0 you have access to AZURE_PW_TEST_RUN_ID environment variable in your ADO pipeline. You can get it from the Task Variables.

  Example of usage in Azure DevOps pipeline:

  - script: npx playwright test
    displayName: 'Run Playwright tests'
    name: 'playwright'
    env:
      CI: 'true'
  
  - script: echo $(playwright.AZURE_PW_TEST_RUN_ID)
    displayName: 'Print test run id'
  

• AZUREPWDEBUG - Enable debug logging from reporter 0 - disabled, 1 - enabled. Default: 0.

  Example of usage in Azure DevOps pipeline:

  - script: npx playwright test
    displayName: 'Run Playwright tests'
    name: 'playwright'
    env:
      CI: 'true'
      AZUREPWDEBUG: '1'