@testingbot/cli

Run mobile tests on real devices in the cloud

Website · Documentation · npm

Run Espresso, XCUITest and Maestro tests on real devices in the cloud.

• Real Devices — Test on thousands of real iOS and Android devices
• Emulators & Simulators — Fast feedback with virtual devices
• Parallel Execution — Split tests across multiple devices with sharding
• CI/CD Ready — Integrates with GitHub Actions, Jenkins, and more
• Live Results — Watch tests run in real-time
• Artifacts — Download videos, screenshots, and logs

Get Started →

Installation

npm install -g @testingbot/cli

Requirements: NodeJS 20 or higher

Authentication

The CLI requires TestingBot API credentials. You can authenticate in several ways:

Browser Login (Recommended)

testingbot login

This opens your browser for authentication. After logging in, your credentials are saved to ~/.testingbot.

Other Methods

• Command-line options: --api-key and --api-secret
• Environment variables: TB_KEY and TB_SECRET
• Config file: Create ~/.testingbot with content key:secret

Commands

Maestro

Run Maestro UI tests on real devices and emulators/simulators.

testingbot maestro <app> <flows...> [options]

Arguments:

• app - Path to your app file (.apk, .ipa, .app, or .zip)
• flows - One or more paths to flow files (.yaml/.yml), directories, .zip files, or glob patterns

Device Options:

Option	Description
--device <name>	Device name (e.g., "Pixel 9", "iPhone 16")
--platform <name>	Platform: Android or iOS
--deviceVersion <version>	OS version (e.g., "14", "17.2")
--real-device	Use a real device instead of emulator/simulator
--orientation <orientation>	Screen orientation: PORTRAIT or LANDSCAPE
--device-locale <locale>	Device locale (e.g., "en_US", "de_DE")
--timezone <timezone>	Timezone (e.g., "America/New_York", "Europe/London")

Test Configuration:

Option	Description
--name <name>	Test name for dashboard identification
--build <build>	Build identifier for grouping test runs
--include-tags <tags>	Only run flows with these tags (comma-separated)
--exclude-tags <tags>	Exclude flows with these tags (comma-separated)
-e, --env <KEY=VALUE>	Environment variable for flows (can be repeated)
--maestro-version <version>	Maestro version to use (e.g., "2.0.10")

Network & Location:

Option	Description
--throttle-network <speed>	Network throttling: 4G, 3G, Edge, airplane, or disable
--geo-country-code <code>	Geographic IP location (ISO country code, e.g., "US", "DE")

Output Options:

Option	Description
--async	Start tests and exit without waiting for results
-q, --quiet	Suppress progress output
--report <format>	Download report after completion: html or junit
--report-output-dir <path>	Directory to save reports (required with --report)
--download-artifacts [mode]	Download test artifacts (logs, screenshots, video). Mode: all (default) or failed
--artifacts-output-dir <path>	Directory to save artifacts zip (defaults to current directory)

Advanced Options:

Option	Description
--shard-split <number>	Split flows into N parallel sessions for faster execution
--ignore-checksum-check	Skip checksum verification and always upload the app

CI/CD Integration:

Option	Description
--commit-sha <sha>	Git commit SHA associated with this test run
--pull-request-id <id>	Pull request ID this test run originated from
--repo-name <name>	Repository name (e.g., GitHub repo slug)
--repo-owner <owner>	Repository owner (e.g., GitHub organization or username)

Examples:

# Basic usage
testingbot maestro app.apk ./flows

# Multiple flow directories
testingbot maestro app.apk ./flows/smoke ./flows/regression ./flows/e2e

# With device selection
testingbot maestro app.apk ./flows --device "Pixel 8" --deviceVersion "14"

# Android app on real device with tags
testingbot maestro app.apk ./flows --device "Samsung Galaxy S24" --real-device --include-tags "smoke,regression"

# With environment variables
testingbot maestro app.apk ./flows -e API_URL=https://staging.example.com -e API_KEY=secret

# Download JUnit report
testingbot maestro app.apk ./flows --report junit --report-output-dir ./reports

# Download all artifacts (logs, screenshots, video)
testingbot maestro app.apk ./flows --download-artifacts --build "build-123"

# Download artifacts only for failed tests
testingbot maestro app.apk ./flows --download-artifacts failed --artifacts-output-dir ./artifacts

# Run in background (async)
testingbot maestro app.apk ./flows --async

# Split flows across 3 shards, grouping all flows over 3 parallel sessions
testingbot maestro app.apk ./flows --shard-split 3

# CI/CD integration with Git metadata
testingbot maestro app.apk ./flows \
  --commit-sha "abc123def" \
  --pull-request-id "42" \
  --repo-owner "myorg" \
  --repo-name "myapp"

Espresso

Run Android Espresso tests on real devices and emulators.

testingbot espresso [appFile] [testAppFile] [options]

Arguments:

• appFile - Path to application APK file
• testAppFile - Path to test APK file containing Espresso tests

Device Options:

Option	Description
--app <path>	Path to application APK file
--test-app <path>	Path to test APK file
--device <name>	Device name (e.g., "Pixel 6", "Samsung.*")
--platform-version <version>	Android OS version (e.g., "12", "13", "14")
--real-device	Use a real device instead of an emulator
--tablet-only	Only allocate tablet devices
--phone-only	Only allocate phone devices
--locale <locale>	Device locale (e.g., "en_US", "de_DE")
--timezone <timezone>	Timezone (e.g., "America/New_York", "Europe/London")

Test Configuration:

Option	Description
--name <name>	Test name for dashboard identification
--build <build>	Build identifier for grouping test runs
--test-runner <runner>	Custom test instrumentation runner
--language <lang>	App language (ISO 639-1 code, e.g., "en", "fr", "de")

Test Filtering:

Option	Description
--class <classes>	Run tests in specific classes (comma-separated fully qualified names)
--not-class <classes>	Exclude tests in specific classes
--package <packages>	Run tests in specific packages (comma-separated)
--not-package <packages>	Exclude tests in specific packages
--annotation <annotations>	Run tests with specific annotations (comma-separated)
--not-annotation <annotations>	Exclude tests with specific annotations
--size <sizes>	Run tests by size: small, medium, large (comma-separated)

Network & Location:

Option	Description
--throttle-network <speed>	Network throttling: 4G, 3G, Edge, or airplane
--geo-location <code>	Geographic IP location (ISO country code, e.g., "US", "DE")

Output Options:

Option	Description
--async	Start tests and exit without waiting for results
-q, --quiet	Suppress progress output
--report <format>	Download report after completion: html or junit
--report-output-dir <path>	Directory to save reports (required with --report)

Examples:

# Basic usage with positional arguments
testingbot espresso app.apk app-test.apk --device "Pixel 8"

# Using named options
testingbot espresso --app app.apk --test-app app-test.apk --device "Pixel 8"

# Real device with specific Android version
testingbot espresso app.apk app-test.apk \
  --device "Samsung Galaxy S24" \
  --platform-version "14" \
  --real-device

# Run specific test classes
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --class "com.example.LoginTest,com.example.HomeTest"

# Run tests with annotations
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --annotation "com.example.SmokeTest" \
  --size "small,medium"

# With network throttling and geolocation
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --throttle-network "3G" \
  --geo-location "DE" \
  --language "de"

# Download JUnit report
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --report junit \
  --report-output-dir ./reports

XCUITest

Run iOS XCUITest tests on real devices and simulators.

testingbot xcuitest [appFile] [testAppFile] [options]

Arguments:

• appFile - Path to application IPA file
• testAppFile - Path to test ZIP file containing XCUITests

Device Options:

Option	Description
--app <path>	Path to application IPA file
--test-app <path>	Path to test ZIP file
--device <name>	Device name (e.g., "iPhone 15", "iPad.*")
--platform-version <version>	iOS version (e.g., "17.0", "18.2")
--real-device	Use a real device instead of a simulator
--tablet-only	Only allocate tablet devices
--phone-only	Only allocate phone devices
--orientation <orientation>	Screen orientation: PORTRAIT or LANDSCAPE
--locale <locale>	Device locale (e.g., "DE", "US")
--timezone <timezone>	Timezone (e.g., "America/New_York", "Europe/London")

Test Configuration:

Option	Description
--name <name>	Test name for dashboard identification
--build <build>	Build identifier for grouping test runs
--language <lang>	App language (ISO 639-1 code, e.g., "en", "fr", "de")

Network & Location:

Option	Description
--throttle-network <speed>	Network throttling: 4G, 3G, Edge, or airplane
--geo-location <code>	Geographic IP location (ISO country code, e.g., "US", "DE")

Output Options:

Option	Description
--async	Start tests and exit without waiting for results
-q, --quiet	Suppress progress output
--report <format>	Download report after completion: html or junit
--report-output-dir <path>	Directory to save reports (required with --report)

Examples:

# Basic usage with positional arguments
testingbot xcuitest app.ipa app-test.zip --device "iPhone 16"

# Using named options
testingbot xcuitest --app app.ipa --test-app app-test.zip --device "iPhone 16"

# Real device with specific iOS version
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 15 Pro" \
  --platform-version "17.2" \
  --real-device

# iPad in landscape mode
testingbot xcuitest app.ipa app-test.zip \
  --device "iPad Pro" \
  --tablet-only \
  --orientation LANDSCAPE

# With localization settings
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --locale "DE" \
  --language "de" \
  --timezone "Europe/Berlin"

# With network throttling and geolocation
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --throttle-network "3G" \
  --geo-location "DE"

# Download HTML report
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --report html \
  --report-output-dir ./reports

# Run in background
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --async

Common Features

Real-time Progress

By default, the CLI shows real-time progress updates including:

• Test status updates with actual device names (even when using wildcards)
• Device allocation status
• Live output from Maestro flows

Use --quiet to suppress progress output.

Graceful Shutdown

Press Ctrl+C to gracefully stop running tests. The CLI will:

• Stop all active test runs on TestingBot
• Clean up resources
• Exit with appropriate status code

Press Ctrl+C twice to force exit immediately.

Report Downloads

All test frameworks support downloading reports after completion:

# JUnit XML format (for CI integration)
--report junit --report-output-dir ./reports

# HTML format (for human viewing)
--report html --report-output-dir ./reports

Artifact Downloads (Maestro only)

Download all test artifacts including logs, screenshots, and video recordings:

testingbot maestro app.apk ./flows --download-artifacts --build "my-build"

Artifacts are saved as a zip file named after the --build value (or with a timestamp if not provided).

Exit Codes

• 0 - All tests passed
• 1 - One or more tests failed or an error occurred

Documentation

For more information, visit TestingBot Documentation.

License

MIT