@devassure/cli
Advanced tools
Command-line interface for DevAssure Invisible (QA) Agent - executes end to end UI tests from natural language instructions and csv files. CLI can be used to run tests by integrating into CI/CD pipelines.
Install DevAssure CLI globally using npm:
npm install -g @devassure/cli
Or using pnpm:
pnpm add -g @devassure/cli
After installation, verify it's working:
devassure version
Authenticate with DevAssure:
devassure login
This will open your browser for OAuth2 authentication.
Alternatively, you can add an authentication token directly (recommended for CI/CD pipelines):
devassure add-token <your-token>
Initialize DevAssure configuration in your project:
devassure init
This will:
.devassure folder with configuration filesAdd test cases to the project by adding yaml files to the .devassure/tests folder in the project directory.
Example test case file:
summary: Sample test case
steps:
- Open the application url
- Verify if the page loads successfully
- Verify if there are no error messages
priority: P0
tags:
- sanity
- app-load
Run tests using the current directory as test cases directory:
devassure run-tests
# or use the alias
devassure run
Any required test data can be added to .devassure/test_data.yaml in the project directory.
Example test data file:
default:
url: 'http://localhost:3000'
users:
default:
user_name: 'user@test.com'
password: 12345678
admin:
user_name: 'admin@test.com'
password: 12345678
devassure login - Login to DevAssure using OAuth2 authenticationdevassure logout - Logout from DevAssure and clear stored tokensdevassure add-token <token_value> - Add and validate an authentication tokendevassure init - Initialize DevAssure configuration in the current directory
.devassure folder with configuration filesdevassure run-tests - Run tests using current directory as test cases directory
--path <path> - Project path (default: current directory)--csv <path> - Path to CSV file containing test cases (relative or absolute). Relative paths in order: current directory, .devassure are supported. File must exist and have .csv extension.--tag <tags> / --tags <tags> - Comma-separated tag values (e.g., --tag=tag1,tag2,tag3)--priority <priorities> / --priorities <priorities> - Comma-separated priority values (e.g., --priority=P0,P1)--folder <folders> / --folders <folders> - Comma-separated folder paths (e.g., --folder=admin/users,project/integration)--query <query> / --queries <query> - Search query string (e.g., --query="my search query")--filter <filter> / --filters <filter> - Raw filter string (takes precedence over other filter parameters, e.g., --filter="tag = tag1,tag2 && priority = P0")--archive <folder> - Archive folder path; after the run, test reports are written as devassure-results-<session-id>.zip in this folder (relative paths are resolved from current directory)--filter/--filters is provided, all other filter parameters (--tag, --priority, --folder, --query) are ignoreddevassure run (same options apply)devassure test — Run tests scoped to git code changes (branch comparison or a specific commit). Uses the same test case discovery as run-tests (current directory as test cases root), but tells the agent which branch or commit to focus on.
--path <path> — Project path (default: current directory)--head <branch name> — Source branch (changes under test); forwarded to the agent as --source. Optional when using branch mode.--base <branch name> — Target branch (baseline to compare against, usually the repo default); forwarded as --target. Optional when using branch mode.--commit <commitId> — Commit to test; forwarded as --commit-id. Use this for commit mode instead of --head / --base.--archive <folder> — Archive folder path; after the run, test reports are written as devassure-results-<session-id>.zip in this folder--environment <env> — Environment name (e.g. staging, production)--url <url> — Override default test URL (test_data.default.url and TEST DATA section)--headless <bool> — Run browser headless: true or false (omit for project default)--head, --base, or --commit, that mode is enabled. Omitting all three is equivalent to “no explicit ref”: the agent uses current branch as the source and the repository default branch as the target.devassure resume - Resume an existing test session by session ID
--session-id <session-id> - Session ID to resume (one of these or --last is required)--last - Resume the last executed sessiondevassure archive-report - Archive report results for a test session into a zip file
--output-dir <dir> - (Required) Output directory for the archived report zip--session-id <sessionId> - Session ID to archive (either this or --last is required)--last - Use the last executed session ID from the databasedevassure-agent archive-results and writes devassure-results-<session-id>.zip into the output directorydevassure open-report - Open report server for a test session
--session-id <sessionId> - Session ID to open report for--last - Open report for the last executed session--archive <path> - Path to an archived zip file (e.g. from archive-report) to open; opens the report from the zip without using the database--archive, --session-id, or --last is requireddevassure summary - Print summary for a test session
--session-id <sessionId> - Session ID to summarize (either this or --last is required)--last - Use the last executed session ID from the database--json - Output summary as JSON (session_id, environment, scenarios, score, grouped_failures, passed_validations, duration_ms, duration_string)devassure stats - Show statistics about sessions, scenarios, and storage
devassure cleanup - Clean up execution history to free up space
--retain-days <days> - Retain sessions from the last N days--retain-sessions <count> - Retain the last N sessionsdevassure version - Show version of the CLIdevassure help - Show help information--verbose - Enable verbose logging for all commandsProject files files are stored in:
~/.devassure/ (project-specific)# 1. Login
devassure login
# 2. Initialize project
devassure init
# 3. Run tests
devassure run-tests
# 4. View report for last session
devassure open-report --last
# 1. Test the diff of current branch vs default branch
devassure test
# 2. Explicit source and target branches
devassure test --head feature/login --base main
# 3. Test a specific commit
devassure test --commit abc1234567890
# Keep only sessions from last 7 days
devassure cleanup --retain-days 7
# Keep only the last 10 sessions
devassure cleanup --retain-sessions 10
# Run tests with specific tags
devassure run-tests --tag=smoke,regression
# Run tests with specific priority
devassure run-tests --priority=P0,P1
# Run tests in specific folders
devassure run-tests --folder=admin/users,project/integration
# Run tests with a search query
devassure run-tests --query="login flow"
# Combine multiple filter parameters
devassure run-tests --tag=smoke --priority=P0 --folder=admin/users
# Use raw filter string (takes precedence over other filter parameters)
devassure run-tests --filter="tag = tag1,tag2 && priority = P0"
# Use keyword search to filter tests (this will do text match on summary, steps, tags)
devassure run-tests --filter="query = dashboard"
# All parameters support plural forms
devassure run-tests --tags=smoke,regression --priorities=P0,P1 --folders=admin/users
# Resume the last session
devassure resume --last
# Resume a specific session
devassure resume --session-id=<session-id>
devassure resume --id=<session-id>
# Run tests and archive reports to a folder (zip created after run completes)
devassure run-tests --archive=./reports
devassure run --archive=/tmp/archives
# Run tests from a CSV file (relative path resolved from project path)
devassure run-tests --csv=sample-tests.csv
devassure run --csv=.devassure/sample-tests.csv
# Archive the last session's report to a directory
devassure archive-report --output-dir=./reports --last
# Archive a specific session's report
devassure archive-report --output-dir=./reports --session-id=<session-id>
# Open report from an archived zip file
devassure open-report --archive=./reports/devassure-results-<session-id>.zip
# Print summary for the last session
devassure summary --last
# Print summary for a specific session
devassure summary --session-id=<session-id>
# Output summary as JSON
devassure summary --last --json
All configuration files live under .devassure/. Below are sample contents for each file type.
App description and rules for the agent:
description: >
About my app
in multi line
rules:
- User can't create more than 3 projects
- Any data deleted can't be restored
(Optional) User personas used in tests:
normal_user:
description: No admin access can do all other operations
age_group: 18-30
gender: M
region: USA
admin:
description: Admin can add or delete other users and manage privileges
deactivated_user:
description: Login is deactivated by admin
expired_user:
description: License has expired
Default URL, users, and test data per environment:
If data keys are not present for a specific environment, the default data will be used.
default:
url: 'http://localhost:3000'
users:
default:
user_name: 'user@test.com'
password: 12345678
admin:
user_name: 'admin@test.com'
password: 12345678
stripe_data:
allowed_card:
card: 4444 4444 4444 4444
wrong_cvv:
card: 1234 1234 1234 1234
expiry: 10/28
cvv: 123
test_otp: 1122
uat:
url: 'http://uat.myapp.com'
users:
default:
user_name: 'user@uat.com'
password: 12345678
admin:
user_name: 'admin@uat.com'
password: 12345678
test_otp: 2345
Instructions for the agent during test execution:
instructions:
- Reload the app and retry if app shows warning server is busy
- Sign up a new user and proceed if any of the given logins are not working
Browser and execution preferences:
browsers:
default:
browser: chromium
resolution: 1920 x 1200
headless: true
workers: 2
Column mappings when running tests from a CSV file (used with --csv). Maps CSV column names to test case fields:
summary: Summary
steps: Steps
priority: Priority
tags: Tags
Example test case in .devassure/tests/sample-test.yaml:
summary: Sample test case
steps:
- Open the application url
- Verify if the page loads successfully
- Verify if there are no error messages
priority: High
tags:
- sample
- ui
Another example with multiple steps:
summary: Login as admin and verify dashboard
steps:
- Open the application url
- Log in with admin credentials from test data
- Verify dashboard loads and shows admin menu
- Log out
priority: P0
tags:
- smoke
- login
- admin
Actions group steps that can be reused across multiple tests. Add action YAML files under actions folder - .devassure/actions/<action_name>.yaml. To call an action, use its name as a step in a test file.
Fields in an action file:
| Field | Description |
|---|---|
name | The name of the action. Used to call the action from the test file. |
description | A short description of the action. |
steps | A list of steps that make up the action. |
Example action file (e.g. .devassure/actions/login_as_admin.yaml):
name: login_as_admin
description: Login to the app as admin using google
steps:
- Open admin portal url
- Click on Google login button
- Enter admin email and password
- If MFA is asked, enter the authenticator OTP
- If allow access is asked, click on allow access
Example 2 (e.g. .devassure/actions/add_manager_user.yaml):
name: add_manager_user
description: Add a manager user to the app
steps:
- Go to users page
- Click on Add User button
- Enter email
- Enter role as Manager
- Once team dropdown appears assign any team to the user
- Enter random user details
- Click on Add User button
Usage in test file: reference actions by name in steps:
steps:
- login_as_admin
- add_manager_user
- Open users list page
- Verify if the manager user is added
Library tools are inbuilt tools that users can include in the project by listing the required tools in a library.yaml file.
Sample library.yaml (e.g. .devassure/library.yaml):
tools:
- 'authenticator'
- 'faker:*'
Faker provides realistic synthetic data generators for tests:
| Tool-key | Description |
|---|---|
*first_name* | Generate a random first name |
*last_name* | Generate a random last name |
*full_name* | Generate a random full name |
*email* | Generate a random email address |
*phone* | Generate a random phone number |
Authenticator provides TOTP (Time-based One-Time Password) helpers:
| Tool-key | Description |
|---|---|
get_authenticator_otp | Generate a TOTP code from an authenticator secret. The authenticator secret must be passed to generate the code. |
Tools let you run custom commands or programs and consume their output. The configuration lives in .devassure/tools/index.yaml.
Requirement: tools/index.yaml is required for tools to work. It must exist at .devassure/tools/index.yaml (relative to your project path).
Tools can run any command or program; the runner executes the command and consumes its output. For custom code or scripts, put the tool code inside the .devassure/tools folder and reference the execution command in tools/index.yaml (e.g. exec: node script.js or exec: npm run myTool).
Note: You can use any programming language or program. You are responsible for setting up dependencies (e.g. Node.js, Python, venv, system binaries).
Each tool must have name, description, and exec:
| Field | Description |
|---|---|
| name | Unique identifier for the tool (e.g. used when invoking the tool). |
| description | Short description of what the tool does. |
| exec | Command(s) to run. Can be a single line or a multi-line string. Supports ${argName} substitution for args. |
Optional per-tool fields include cwd and args (see below).
Minimal example:
tools:
- name: "getProjectDetails"
description: "Get project details from api"
cwd: "api-tools"
args:
- name: projectId
type: string
exec: npm run getProjectDetails ${projectId}
.devassure/tools). If omitted, the working directory is the tools folder. Absolute paths are also supported.name and type; see Supported arg types and optional args below.settings: Default options applied to every tool run and every setup step. For all fields except env, the value on the tool or setup step is used when present; values in settings are fallback when the tool or step does not define that field. For env, the list of environment variables is merged: settings env vars and the tool/setup step env vars are combined (tool/setup entries override settings when the same key is used).
setup: Optional list of steps run once per session before scenario executions start (e.g. install dependencies). Each step can specify name, cwd, exec, and any of the settings fields; inheritance from settings applies when a step omits a field.
cwd: "api-tools" → .devassure/tools/api-tools).cwd is not provided, the working directory is the tools folder.cwd: "/opt/scripts").| Option | Applies to | Description |
|---|---|---|
| timeoutSec | Tool / setup | Maximum execution time in seconds (e.g. 10). |
| output.start / output.end | Tool / setup | Markers in stdout; only content between these markers is captured as the tool output. Strongly recommended when the process prints a lot of extra content (logs, progress). Your script should print the actual result between these markers. |
| env | Tool / setup | List of KEY: value environment variables added or overridden for the process. |
| ignore_failure | Tool / setup | If true, a non-zero exit or failure does not fail the run (e.g. optional setup). Default is false. |
When the command prints a lot of unwanted content (logs, progress, debug), use output.start and output.end in settings (or on the tool). The runner captures only the stdout between these two markers. Print the markers and the necessary output from your script.
JavaScript example: print the markers, then the result, then the end marker:
// In your script (e.g. getProjectDetails.js)
const startMarker = "__TOOL_OUTPUT_START__";
const endMarker = "__TOOL_OUTPUT_END__";
// ... do work, then:
console.log(startMarker);
console.log(JSON.stringify(result));
console.log(endMarker);
In tools/index.yaml you would set output.start and output.end to match (e.g. __TOOL_OUTPUT_START__ and __TOOL_OUTPUT_END__).
Supported types: string, number, boolean, object. Use type in each arg.
Optional args: Add optional: true to an arg. If not provided, it may be omitted or passed empty depending on how you use it in exec (e.g. "${projectName}" for an optional string).
Example with multiple types and one optional arg:
tools:
- name: "exampleTool"
description: "Example with string, number, boolean, object args"
args:
- name: id
type: string
- name: count
type: number
- name: verbose
type: boolean
- name: options
type: object
- name: projectName
type: string
optional: true
exec: node run.js ${id} ${count} ${verbose} "${options}" "${projectName}"
exec: npm run getProjectDetails ${projectId}exec: | with following lines). Useful for running several commands in sequence (e.g. warmup, main command, cleanup):exec: |
npm run warmupTestingProcess
npm run getProjectDetails ${projectId} "${projectName}"
npm run cleanupTestingProcess
The following example shows settings (timeoutSec, output markers, env, ignore_failure), setup (one step with cwd and exec), and tools with optional arg and multi-line exec:
settings:
timeoutSec: 10
output:
start: "__TOOL_OUTPUT_START__"
end: "__TOOL_OUTPUT_END__"
env:
- BUILD_ENV: "dev"
- DB_NAME: "legacy-db"
ignore_failure: false
setup:
- name: "Install dependencies"
cwd: "api-tools"
exec: "npm install"
tools:
- name: "getProjectDetails"
description: "Get project details from api"
cwd: "api-tools"
args:
- name: projectId
type: string
- name: projectName
type: string
optional: true
exec: |
npm run warmupTestingProcess
npm run getProjectDetails ${projectId} "${projectName}"
npm run cleanupTestingProcess
You can keep large or reusable test_data in separate YAML files and merge them into your scenario config using !include, so teams can share the same data across multiple scenarios/environments.
This works for both single values (strings/numbers/booleans) and arrays of values (e.g., multiple cards, multiple admin users, multiple payment methods).
# .devassure/actions/cart.yaml
- name: go_to_carts_page
description: Go to the carts page and verify the content
steps:
- Click on the carts icon
- Verify the carts page is opened
- name: add_to_cart
description: Add a product to the cart
steps:
- Click on the add to cart button
- Verify the product is added to the cart
# .devassure/actions/products.yaml
name: remove_from_cart
description: Remove a product from the cart
steps:
- Click on the remove button for the product
- Verify the product is removed from the cart
# .devassure/test_data.yaml
default:
url: http://127.0.0.1:4321
cart:
<<: !include test_data/cart.yaml
If the HTTP_PROXY or HTTPS_PROXY environment variables are set, the CLI will use them to proxy requests.
How to add new test cases?
Add YAML files under .devassure/tests/ (or test/), or use a CSV file with --csv <path> and optionally .devassure/csv_mapping.yaml to map columns to test fields.
Is test case YAML always needed or can we use CSV only?
No. You can run tests from CSV only: use devassure run-tests --csv=<path> and configure .devassure/csv_mapping.yaml so your CSV columns map to summary, steps, priority, and tags.
What are the mandatory fields for a test case?
For YAML test cases, summary and steps are required. priority and tags are optional. For CSV, the mapped columns must provide at least the equivalent of summary and steps.
How to add a new environment?
Add a new top-level key (e.g. uat) and its data in .devassure/test_data.yaml. Then run with devassure run-tests --environment=uat (or --environment uat). If you omit --environment, the default key in test_data.yaml is used.
What are the supported browsers?
Supported browsers are Chromium, Chrome, Edge. Controlled by .devassure/preferences.yaml under browsers.default.browser. The default is chromium. See the preferences.yaml section above for the structure.
How to run with headless mode?
In .devassure/preferences.yaml, set browsers.default.headless: true.
How to run the tests in Continuous Integration?
Use devassure add-token <your-token> to authenticate (no browser). Then run devassure run-tests with the options you need (e.g. --csv, --filter, --tag, --priority, --archive for saving report zips). You can retain only recent sessions with devassure cleanup --retain-days 7 or --retain-sessions 10.
For more information, visit DevAssure or run devassure help for command-specific help.
FAQs
DevAssure CLI application
The npm package @devassure/cli receives a total of 265 weekly downloads. As such, @devassure/cli popularity was classified as not popular.
We found that @devassure/cli demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Multiple high-impact npm maintainers confirm they have been targeted in the same social engineering campaign that compromised Axios.
Security News
Axios compromise traced to social engineering, showing how attacks on maintainers can bypass controls and expose the broader software supply chain.
Security News
Node.js has paused its bug bounty program after funding ended, removing payouts for vulnerability reports but keeping its security process unchanged.