sonar-forge

Sonar-Forge

JMeter test orchestrator - YAML-driven JMX modification and execution.

Inspired by Blazemeter Taurus, Sonar-Forge has been built as a simplified and streamlined alternative to work within Sonar's performance testing framework and ecosystem. It enables clean, version-controlled test and tool configuration specific to each test without needting to modify JMX files and Jmeter properties, or embedding complex properties and variables within each script.

Overview

Sonar-Forge separates test configuration from test scripts by:

• Reading existing JMX files to generate an initial YAML configuration specifying test and thread properties/configuration
• Users are able to extend the configuration, or make several versions (e.g. smoke test, peak test, soak test, etc.) by updating the initial configuration without needing to modify the JMX itself.
• Executing tests with automatic result collection and error logging
• Refreshing the YAML configuration file when the JMX gets updated (e.g. new thread groups get added, etc.)

Installation

npm install -g sonar-forge

Quick Start

1. Initialize from existing JMX

sonar-forge init my-test.jmx

This creates a project structure:

my-test/
├── config/
│   └── my-test.yaml          # Test configuration
├── scripts/
│   └── my-test.jmx           # Original JMX file
├── data/                     # Test data files
├── logs/                     # JMeter logs
└── results/                  # Test results

2. Edit configuration

Edit config/my-test.yaml:

test_script: ../scripts/my-test.jmx

output:
  results_dir: ../results
  logs_dir: ../logs

jmeter:
  properties:
    jmeter.save.saveservice.output_format: csv
    jmeter.save.saveservice.timestamp_format: ms
    # ... additional JMeter properties

error_log:
  enabled: true
  filename: errors.xml

thread_groups:
  - name: Load Test
    enabled: true
    threads: 50
    rampup: 10
    iterations: -1
    duration: 300

    lifecycle:
      EnableLifeCycleController: true
      GenerateParentSample: true
      UseFixedPacing: true
      PacingInterval: 5000

3. Run your test

cd my-test
sonar-forge run config/my-test.yaml

Results are saved with timestamps:

• results/results_2025-01-15_14-30-00.jtl - CSV format with all metrics
• results/errors_2025-01-15_14-30-00.xml - XML format with full error details

Commands

init <jmx-file>

Generate project from existing JMX file.

sonar-forge init my-test.jmx
sonar-forge init my-test.jmx -o custom-dir

validate <config-file>

Validate configuration file.

sonar-forge validate config/my-test.yaml

build <config-file>

Build effective JMX without running test.

sonar-forge build config/my-test.yaml
sonar-forge build config/my-test.yaml --dry-run
sonar-forge build config/my-test.yaml --run-id "test-001"  # Set unique run ID

run <config-file>

Build and execute JMeter test.

sonar-forge run config/my-test.yaml
sonar-forge run config/my-test.yaml -v                    # Verbose output
sonar-forge run config/my-test.yaml --build-only          # Build only, don't run
sonar-forge run config/my-test.yaml --keep-jmx            # Keep effective JMX after run
sonar-forge run config/my-test.yaml --run-id "test-001"   # Set unique run ID

refresh [config-file]

Update configuration from modified JMX file.

cd my-test

# Single config file (auto-detect)
sonar-forge refresh              # Merge mode (default) - preserves config
sonar-forge refresh --replace    # Replace mode - overwrites with JMX data

# Multiple config files (specify which one)
sonar-forge refresh test1.yaml
sonar-forge refresh test2.yaml --replace

# Process all config files
sonar-forge refresh --all        # Merge mode for all configs
sonar-forge refresh --all --replace   # Replace mode for all configs

# Verbose output
sonar-forge refresh -v

Merge mode: Preserves your configuration values, adds new thread groups from JMX.

Replace mode: Completely replaces configuration with current JMX state.

--all flag: Processes all YAML config files in the config/ directory, showing a summary of successes and failures.

Configuration

Logging Level

Control JMeter's console and log file verbosity:

# JMeter logging level (DEBUG, INFO, WARN, ERROR)
log_level: WARN

Available levels (from most to least verbose):

• DEBUG - Detailed debugging information
• INFO - General informational messages (default if not specified)
• WARN - Warning messages only
• ERROR - Error messages only

Thread Groups

thread_groups:
  - name: Thread Group Name       # Must match JMX testname
    enabled: true                 # Enable/disable thread group
    threads: 50                   # Number of threads
    rampup: 10                    # Ramp-up period in seconds
    iterations: -1                # Number of iterations (-1 = infinite)
    duration: 300                 # Thread lifetime in seconds (null to disable)

Execution modes:

• Duration only: Set duration, leave iterations at -1 (runs until time limit)
• Iterations only: Set iterations, set duration to null (runs until iteration count)
• Both: Stops when either limit is reached
• Infinite: iterations: -1, duration: null (runs forever)

Lifecycle Controller

For tests using Sonar's LifecycleController:

thread_groups:
  - name: My Test
    # ... thread configuration
    lifecycle:
      EnableLifeCycleController: true
      GenerateParentSample: true
      TroubleshootingMode: false
      UseFixedPacing: true
      PacingInterval: 5000
      PacingRandomizationPct: 5

Run ID

The run_id provides a unique identifier for test runs that is automatically applied to all backend listeners. This enables tracking and correlating metrics across multiple backend systems.

Three-tier precedence (highest to lowest):

• CLI flag: --run-id "my-run-id"
• YAML config: run_id: "my-run-id"
• Auto-generated: UUID automatically generated if not specified

Examples:

# CLI flag (highest precedence)
sonar-forge run config/my-test.yaml --run-id "production-2025-01-15"

# YAML config
cat config/my-test.yaml
run_id: "test-run-001"

# Auto-generated (no configuration needed)
# Automatically generates UUID like "550e8400-e29b-41d4-a716-446655440000"

The run_id is automatically injected into the runId property of all enabled backend listeners, overriding any values configured in the YAML backend listener properties.

Backend Listeners

Sonar-Forge automatically detects and configures backend listeners for real-time test result reporting.

Supported Listeners

TimescaleDB Listener (recommended for Sonar framework):

backend_listeners:
  - name: Sonar - TimescaleDB Listener
    type: timescaledb
    enabled: true

    properties:
      dbHost: timescaledb.example.com
      dbPort: 6432
      dbName: tsdb
      dbUser: username
      dbPassword: password
      runId: test-run
      reportLabel: ""
      batchSize: 5000
      flushInterval: 4000
      poolSize: 20
      errorThreshold: 5
      sampleTableName: performance_metrics_full
      threadTableName: performance_threads_full
      recordSubSamples: true
      saveResponseBodyOfFailures: true
      responseBodyLength: 2000

InfluxDB v2 Listener:

backend_listeners:
  - name: InfluxDBv2 Backend Listener
    type: influxdb2
    enabled: false

    properties:
      testName: Test
      nodeName: Test-Node
      runId: test-run
      influxDBURL: "http://localhost:8086/"
      influxDBToken: "your-token-here"
      influxDBOrganization: performance_testing
      influxDBBucket: jmeter
      influxDBFlushInterval: 4000
      influxDBMaxBatchSize: 2000
      influxDBThresholdError: 5
      samplersList: .*
      useRegexForSamplerList: true
      recordSubSamples: true
      saveResponseBodyOfFailures: false
      responseBodyLength: 2000

Other Listeners: For other backend listener types, only name, classname, and enabled status are captured:

backend_listeners:
  - name: Custom Backend Listener
    type: other
    classname: com.example.CustomListener
    enabled: false

Warning: If no TimescaleDB or InfluxDB2 listeners are detected, a warning comment appears in the YAML:

# WARNING: No real-time reporting listeners (TimescaleDB/InfluxDB2) detected in this test
backend_listeners: []

JMeter Properties

Configure JTL output format and other JMeter settings:

jmeter:
  properties:
    jmeter.save.saveservice.output_format: csv
    jmeter.save.saveservice.time: true
    jmeter.save.saveservice.latency: true
    jmeter.save.saveservice.bytes: true
    # ... additional properties

Error Logging

Capture full request/response details for failed samples:

error_log:
  enabled: true
  filename: errors.xml

Generates timestamped XML files with complete error diagnostics.

Output Configuration

output:
  results_dir: ../results        # JTL results directory
  logs_dir: ../logs              # JMeter log directory

Workflow

Making JMX Changes

After modifying your JMX file in scripts/, sync your configuration using the refresh command.

Single Configuration File

If you have one config file, the refresh command auto-detects it:

cd my-test
sonar-forge refresh              # Merge: preserves your config values
sonar-forge refresh --replace    # Replace: overwrites with JMX values
sonar-forge refresh -v           # Verbose output shows details

Multiple Configuration Files

If you maintain multiple configurations (e.g., smoke, load, soak tests), you have several options:

Refresh a specific config:

cd my-test
sonar-forge refresh smoke-test.yaml
sonar-forge refresh load-test.yaml --replace

Refresh all configs at once:

cd my-test
sonar-forge refresh --all                 # Merge mode for all
sonar-forge refresh --all --replace       # Replace mode for all
sonar-forge refresh --all -v              # Verbose for all

The --all flag processes all YAML files in config/ and shows a summary:

Found 3 config file(s) to refresh
...
Summary
============================================================
Mode: merge
Total files: 3
✓ Successful: 2
✗ Failed: 1

What Refresh Detects

Merge mode (default):

• ✅ Adds new thread groups from JMX
• ✅ Removes thread groups deleted from JMX
• ✅ Preserves your custom configuration values (threads, duration, etc.)
• ✅ Keeps your lifecycle controller settings
• ✅ Maintains all YAML comments and formatting

Replace mode (--replace):

• Completely replaces thread group configuration with current JMX state
• Useful when you want to reset config to match JMX exactly
• ⚠️ Overwrites your custom settings

Common Scenarios

Scenario 1: Added new thread group to JMX

# Edit scripts/my-test.jmx, add new thread group "API Tests"
sonar-forge refresh              # Auto-adds "API Tests" to config
# Edit config/my-test.yaml to customize the new thread group

Scenario 2: Multiple test profiles

# You have: smoke.yaml, load.yaml, soak.yaml
# All reference scripts/my-test.jmx
# JMX was updated with new thread group

sonar-forge refresh --all        # Updates all 3 configs
# Each config preserves its custom settings (threads, duration, etc.)

Scenario 3: Reset config to match JMX

# Your config has experimental changes, want to reset
sonar-forge refresh --replace    # Overwrites config with JMX defaults

Version Control

Commit these files:

• config/*.yaml - Test configuration (source of truth)
• scripts/*.jmx - Test scripts

Add to .gitignore:

• results/ - Test results
• logs/ - JMeter logs
• data/ - Test data files (unless needed)
• *_effective.jmx - Generated effective JMX files

Upcoming Features

• Disable section that allow specific elements at the Test Plan level to be explictly enabled/disabled
• Add jmeter properties to support extending backend listener queue depth
• Improved test execution UI (similar to Taurus, but customised for Sonar framework)