@prodperfect/cli
Advanced tools
Readme
A command-line client for ProdPerfect's public Test API
The New & Improved™️ P2 CLI replaces the existing prodperfectqa-runner CLI and will enable customers to interact with our new remote test harness infrastructure. It is, at least in this first incarnation, simply a command-line interface for a curated subset of the Test API, allowing users to
In-depth technical documentation can be found at our documentation site:
https://prodperfect.github.io/cli
The CLI itself will
Why?
Why?
A standard develop-review-merge workflow applies here, nothing special and nothing that needs going over. Deployment and distribution is outlined later.
Quality assurance will be encapsulated through a combination of approval testing, a style which requires manual approval anytime a test case produces different output than the previously approved one; and end-to-end testing using our own dogfed test suites as targets. When a feature PR is opened, a GitHub workflow will be triggered which runs our approval tests; the build will fail if any of the tests fail approval, at which point the developer must either fix what they broke, or run the tests on their local machine to approve the new behavior, and commit the approval to their feature branch.
If deemed necessary, such as when changing the way the CLI interacts with the underlying Test API it consumes, manual end-to-end testing will be performed. The scope of such testing will be decided upon and planned out along with the relevant feature work, but once the test plan is in hand, the release candidate can be tested leveraging one or more of our in-house, dog-fed test suites such as the ones for Mission Control and Operations.
We will be leveraging GitHub workflows and an update mechanism built into the CLI itself to automate deployment and distribution.
The release process will be encapsulated by a GitHub Actions Workflow on pull requests against the main project branch:
Users will be able to download the CLI directly from our GitHub Releases repository (and/or, ideally, from our documentation site). This can be a manual process or a programmatic one via something simple like cURL.
For example, to download a specific release is simply:
$ curl -sOL https://github.com/ProdPerfect/cli/releases/download/2.7.42/cli \
&& chmod +x ./cli
To download the latest release is similarly simple:
$ curl -sOL https://github.com/ProdPerfect/cli/releases/latest/download/cli \
&& chmod +x ./cli
If we choose, it is also trivial to continue publishing our CLI to NPM under the same package name. Choosing to do so is beneficial only if it is imperative to maintain the current way in which the P2 CLI is installed in some cases, i.e. this method of installation (the incantation we currently serve to customers as copypasta) would continue to work:
$ npm install --no-save -E prodperfectqa-runner@<DIST TAG>
Note that customers will need to change their CLI usage, including CI integrations, regardless, because this would be the equivalent of a major version release and is not backwards-compatible. Choosing to continue publishing to NPM will not ensure a passive transition to the new CLI for CI integrations. See below in P2 CLI++: Tech Design for details on what that looks like.
For customers using us in CI integrations, they will first need to ensure their authentication token (issued by us to them directly) are available in their CI environment, a standard process for using any paid service and thus most clients should be familiar with it.
Once that's set, a sample integration would look like one of the following.
Super straightforward: just download the latest release asset every time to a sandbox directory (to avoid clobbering project source code) then make it executable and run it the way you want.
# in package.json
{
...
"scripts": {
"prodperfect": "mkdir -p .prodperfect && cd .prodperfect && curl -sL https://github.com/ProdPerfect/cli/releases/latest/download/cli > cli && chmod +x ./cli && ./cli suites run <SUITE ID>"
}
}
Also super straightforward: NPM installs the latest version by default if no version is specified:
# in package.json
{
...
"scripts": {
"prodperfect": "npm install --no-save -E prodperfectqa-runner && prodperfectqa-runner suites run <SUITE ID>"
}
}
For customers using us in CI integrations, they will first need to ensure their authentication token (issued by us to them directly) are available in their CI environment, a standard process for using any paid service and thus most clients should be familiar with it.
Once that's set, a sample integration would look like one of the following.
The basic version is again super straightforward: just download the specific version asset every time to a sandbox directory, make it executable, and run it the way you want.
# in package.json
{
...
"scripts": {
"prodperfect": "mkdir -p .prodperfect && cd .prodperfect && curl -sL https://github.com/ProdPerfect/cli/releases/download/2.7.42/cli > cli && chmod +x ./cli; ./cli suites run <SUITE ID>"
}
}
If we wanted to put a mild cache guard in place, we could gate the download with a basic existence+version check:
# in package.json
{
...
"scripts": {
"prodperfect": "mkdir -p .prodperfect && cd .prodperfect; [[ ! -e \"./cli\" || ! \"$(./cli -v)\" = \"2.7.42\" ]] && curl -sOL https://github.com/ProdPerfect/cli/releases/download/2.7.42/cli && chmod +x ./cli; ./cli suites run <SUITE ID>"
}
}
If we don't like the ugliness of having a long one-liner integration, the best thing to do would be to simply give them a tiny driver script, complete with embedded auth token, to put wherever they want and tell them to call it, like so:
#! /bin/env bash
AUTH_TOKEN=${AUTH_TOKEN:-<blah>}
TARGET_CLI_VERSION=2.7.42
# Bail if anything goes wrong
set -e
# Make a sandbox
mkdir -p .prodperfect
cd .prodperfect
# Install the CLI if needed
if [[ ! -e "./cli" || ! "$(./cli -v)" = "$TARGET_CLI_VERSION" ]]; then
curl -sOL https://github.com/ProdPerfect/cli/releases/download/${TARGET_CLI_VERSION}/cli
chmod +x ./cli
fi
# Run the CLI with authentication.
env AUTH_TOKEN=$AUTH_TOKEN ./cli "$@"
# in package.json
{
...
"scripts": {
"prodperfect": "path/to/install-and-execute-p2.sh suites run <SUITE ID>"
}
}
Virtually identical to installing the latest, the only difference is you just specify the version you want to install:
# in package.json
{
...
"scripts": {
"prodperfect": "npm install --no-save -E prodperfectqa-runner@<VERSION TAG> && prodperfectqa-runner suites run <SUITE ID>"
}
}
One nice thing about NPM is that it'll handle the 'smarts' of the installation process so that we don't have to.
The Sentry CLI can be used to instrument Bash scripts in similar ways as their other SDKs, and we will take full advantage of that to monitor user experience errors in our CLI.
It does not support performance monitoring, however, so we won't be able to leverage Sentry for tracking user engagement statistics; and we don't seem to have any other business subscriptions to applicable tools. So for now, we'll need to rely indirectly on the observability of the underlying Test API to build a picture of CLI usage.
As a CLI, limited maintenance is expected. The surface area for trouble is more or less limited to:
When maintenance is needed, it will follow the same workflow as general development, plus whatever additional legwork is necessary to confirm the particular issue has been addressed.
todo
FAQs
A command-line client for ProdPerfect's public Test API. It is the spiritual successor to the [`prodperfectqa-runner` CLI](https://www.npmjs.com/package/prodperfectqa-runner), which was built to automate our old testing harness; other than a shared purpos
The npm package @prodperfect/cli receives a total of 94 weekly downloads. As such, @prodperfect/cli popularity was classified as not popular.
We found that @prodperfect/cli demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In an unprecedented surge, May 2024 saw the publication of over 5,000 CVEs, marking a historic milestone in cybersecurity with an average of 164 CVEs per day, nearly double the 2023 daily average.
Security News
The White House is addressing fragmented cybersecurity regulations as CISOs report spending up to 50% of their time on compliance, aiming to harmonize requirements and improve cybersecurity outcomes.
Security News
Research
The Socket Research Team has identified a malicious Python package that is typosquatting the popular crytic-compile utility, frequently used in popular toolkits and development environments for smart contracts and crypto applications.