@spscommerce/e2e-core

e2e-core

UI test automation core framework which is based on Playwright and created specifically for SPS Commerce internal UI projects.

Connect e2e-core subproject to an exisitng UI project

1. Create a new folder for an e2e subproject (e.g. tests-e2e/) with a separate package.json file there.
2. Add e2e-core and all other packages that you need for the e2e testing into package.json.
3. Go into the tests-e2e/ directory and install node modules from package.json.
4. Create .credentials.sh file in the tests-e2e/ directory with the external SPS user credentials.

export USER_EMAIL="EXAMPLE_USER"
export USER_PASSWORD="EXAMPLE_PASSWORD"
# Note: if USER_PASSWORD contains \ or " symbols, please add \ symbol before it.

Attention: Internal SPS user login flow requires MFA step. Accounts with MFA cannot be fully automated, and need manual intervention. Follow the discussion: https://github.com/SPSCommerce/sps-atlas/discussions/111

1. Create .env file in the tests-e2e/ directory with an appropriate process.env variables listed below in the table. E.g.:

TEST_TIMEOUT=75000
ACTION_TIMEOUT=45000
WORKERS=6
REPO_NAME=fulfillment-ui-v2
VIEWPORT={"width": 1920, "height": 1080}

Variable	description	default	possible values
HEADLESS	launch browser in headless/headful mode	true	true, false
BROWSER	use + as a separator, e.g edge+firefox	chromium	chromium, firefox, webkit, edge
ENV	auth state is invoked for the defined env	test	test, prod
TEST_TIMEOUT	global timeout for each test	30000	[number_in_ms]
ACTION_TIMEOUT	global timeout for each PW action, like click()	30000	[number_in_ms]
WORKERS	number of workers (test threads)	1	[number]
TRACER	create PW actions trace file per each failed test	false	true, false
VIEWPORT	emulates consistent viewport for each page	{"width": 1536, "height": 960}	[string]
CI	used in CI: forbidOnly and maxFailures are applied	false	true, false
SUITE	used in CI: report file name is created per suite	TEST-results.xml	[string]
TEST_DIR	used in package.json: tests directory path	-	[string]
CONFIG_PATH	used in package.json: .env file path	-	[string]
SETUP_PATH	used in package.json: globalSetup.js file path	-	[string]
REPO_NAME	creates GH repo link on test in log files	-	[string]

1. Setup the following "scripts" commands in the package.json according to your paths:

"test": ". .credentials.sh && CONFIG_PATH=$(pwd)/.env SETUP_PATH=$(pwd)/src/globalSetup.js TEST_DIR=$(pwd) HEADLESS=false playwright test --config=node_modules/@spscommerce/e2e-core/playwright.config.js",
"test:ci": "CONFIG_PATH=$(pwd)/.env SETUP_PATH=$(pwd)/src/helpers/urlSetup.js TEST_DIR=$(pwd) CI=true playwright test --config=node_modules/@spscommerce/e2e-core/playwright.config.js"

1. Add your first test file:

import { Page, test } from '@playwright/test';
import { enableLogger, disableLogger, expect } from '@spscommerce/e2e-core';

test.describe('SPS_MyFirstTest', () => {

  test.beforeEach(async ({}, testInfo) => {
    await enableLogger(testInfo.titlePath, page);
  });

  test.afterEach(async () => {
    await disableLogger();
  });

  test('Transactions_is_opened_with_content', async () => {
    await page.goto(https://test.commerce.spscommerce.com/);
    await page.waitForTimeout(10000);
  });
});

Attention: Start all the decribe block names with SPS_ prefix. If not, created screenshots and log files won't be attached to an appropriate test.

Additional options

If CI is set to true, the forbidOnly and maxFailures options are applied.

• forbidOnly: Fail the build on CI if you accidentally left test.only in the source code
• maxFailures: Stop test suite execution after reaching 5 failed tests and skip any tests that were not executed yet.

Authentication

Authentication flow was implemented in the e2e-core library. When tests are laucnhed for the first time, it takes some time for the authentication state to be invoked. If ENV variable is not set, the storageState is created for the test environment, otherwise set URL to 'prod'. After auth state is invoked, it appears in the results-e2e/auth folder. All the next test runs will use an already created storageState.json file.

Attention: While local debugging, the storageState might get expired. User will be redirected to the login page. If it happens, please delete an auth folder in order to invoke an auth state again.

Global setup

To set something up once before running all tests, create a JS file and link it via SETUP_PATH variable in the package.json. Global setup file must export a single function that runs once before all the tests.

Results

After tests finishes, you can observe test results in the results-e2e/ folder:

• screenshot folder with screen and trace file (if TRACER=true) for each failed test
• log folder with log file for each test regardless its status
• reports folder with compatible junit xml files (Azure test results friendly format)
• auth folder with an invoked auth state

Tracer

You can observe the PW trace file in 3 ways:

• open a trace file via the terminal CLI: npx playwright show-trace [trace_file_path].zip
• use PW Trace Viewer web app: https://trace.playwright.dev/
• install "Playwright Trace Viewer for VSCode" extension: https://marketplace.visualstudio.com/items?itemName=ryanrosello-og.playwright-vscode-trace-viewer

Dockerfile

PW provides an official Docker image. These image includes all the dependencies needed to run browsers in a Docker container, and also include the browsers themselves. Here is an example of the Dockerfile:

FROM mcr.microsoft.com/playwright:v1.22.0-alpha-1652194905000-focal

ARG BROWSER

RUN mkdir e2e
WORKDIR /e2e
COPY . .
RUN npm install
RUN if [ "$BROWSER" = "msedge" ]; then npx playwright install msedge; fi

Attention: Installed Playwright package version must match the Docker image version.

Run tests in CI (Azure Pipelines)

In order to add an Azure Pipeline for your framework, head over to https://dev.azure.com/spscommerce/ and find your team. You can create a new pipeline by selecting your Git repository. For more information on getting started with Azure, visit https://github.com/SPSCommerce/azure-pipelines-config.

Attention: Do not forget to set up USER_EMAIL and USER_PASSWORD variables in Azure UI. Here is an example of a job in the yaml file:

jobs:	
  - job: TestRun	
    pool: BUILD-LINUX	
    timeoutInMinutes: 20
    steps:	
    - task: DockerCmd@7	
      displayName: 'Test execution'	
      continueOnError: true	
      inputs:	
        dockerfile: 'Dockerfile'	
        dockerBuildArgs: '--build-arg BROWSER=${{parameters.browser}}'	
        dockerImage: 'ffui/e2e'	
        imageSource: 'Dockerfile'	
        dockerRunArgs: --ipc=host --memory=6g --shm-size=1g	
        awsAssumeRole: $(BuildRole)	
        dockerEnvVars: |	
          USER_PASSWORD=$(USER_PASSWORD)
          USER_EMAIL=$(USER_EMAIL)
          BROWSER=${{parameters.browser}}
          URL=${{parameters.url}}
          SUITE=$(suite)
          CI=true
        dockerCmd: |
          npx playwright install
          xvfb-run --auto-servernum npm run test:ci $(suite)
        mountCustomVolume: true	
        externalMountLocation: '$(Build.ArtifactStagingDirectory)/results'	
        internalMountLocation: '/e2e/results-e2e'
    - task: PublishTestResults@2 
      displayName: 'Publish test results'
      inputs:
        testResultsFiles: '**/*.xml'
        searchFolder: '$(Build.ArtifactStagingDirectory)/results/reports/'
    - task: TestingInspector@1
      displayName: 'Publish log files and failed test screenshots'
      inputs:
        screenshotsFolder: '$(Build.ArtifactStagingDirectory)/results/screenshots'
        logsFolder: '$(Build.ArtifactStagingDirectory)/results/logs'
    - task: PublishBuildArtifacts@1
      displayName: 'Upload test results artifacts'
      condition: always()
      inputs:
        pathtoPublish: '$(Build.ArtifactStagingDirectory)/results'
        artifactName: 'results'
        publishLocation: 'Container'
    - task: SlackNotification@1
      displayName: Slack notification (qa channel)
      condition: in(variables['Build.Reason'],'Manual')
      inputs:
        channel: 'azure-autotests-private'
        defaultMessage: true

Notes:

• --ipc=host is used since Chrome can run out of memory without this flag Docker doc
• --shm-size=1g is used since Docker runs a container with a /dev/shm shared memory space 64MB which is typically too small for Chromium and will cause Chromium to crash when rendering large pages
• --auto-servernum usage is recommended in order to run command with a random display number
• xvfb-run is used on Linux agents for headed execution since it requires Xvfb to be installed. Playwright official Docker image has Xvfb pre-installed

TestingInspector@1 - Azure DevOps extension used in CI pipeline to upload screenshot and log files into the test results report.

Q&A

If you have any questions or difficulties with this package, reach out to us in #ffqa on Slack. Additional info over this framework could be found on Confluence. Put in a PR against this repo if you fix a bug or come up with an improvement!

Used tools

• @alex_neo/jest-expect-message - gives ability to add a custom error message to the assertions. We use this forked lib since there is a bug in the original one.
• @playwright/test - browser automation and test runner tool
• @typescript-eslint/eslint-plugin - ESLint plugin which provides lint rules for TypeScript codebases
• @typescript-eslint/parser - ESLint parser which leverages TypeScript ESTree to allow for ESLint to lint TypeScript source code
• dotenv - loads environment variables from a .env file into process.env
• eslint - static code analysis tool
• jest-extended - adds additional matchers to default ones making it easy to test everything 🙌
• typescript - language for application-scale JavaScript that adds optional types