evalsync

EvalSync Python Package

A Python implementation of evalsync with CLI tools for synchronizing benchmarked applications.

Installation

# Install from PyPI
pip install evalsync

# Or install from source
cd python
uv sync

CLI Usage

The package provides the evalsyncli command with demo, wrap, and client subcommands:

Demo Command

Run evalsync demo (can act as server or client):

# Run as server (manager)
evalsyncli demo --experiment-id "test-001" --num-workers 2 --duration 10 --verbose

# Run as client (worker)
evalsyncli demo --experiment-id "test-001" --role client --verbose

Wrap Command

Wrap arbitrary CLI programs with evalsync coordination:

# Basic usage
evalsyncli wrap -e "test-001" -c "client-1" -- whoami
evalsyncli wrap --verbose -e "bench-test" -- sleep 5
evalsyncli wrap -e "iperf-test" -- iperf3 -c server -t 30 -P 4

# Using environment variables
export EVALSYNC_EXPERIMENT_ID="bench-001"
export EVALSYNC_CLIENT_ID="worker-1"
export EVALSYNC_VERBOSE="true"
evalsyncli wrap -- whoami

Client Command

Basic evalsync client (no command wrapping):

evalsyncli client -e "test-001" -c "worker-1" --verbose

The wrap command automatically handles the evalsync protocol:

• Sends READY signal to manager
• Waits for START signal
• Marks measurement start
• Executes the wrapped command
• Marks measurement end
• Sends DONE signal when complete
• Cleans up resources

Signal Handling

The wrapper properly handles signals and forwards them to the wrapped command:

• SIGTERM/SIGINT: Forwarded to the child process, allowing graceful shutdown
• SIGSTOP: Cannot be caught, but child processes are in the same process group
• Process Groups: Child processes stay in the same process group as the wrapper

SIGSTOP Behavior:

The wrapper keeps child processes in the same process group, so:

evalsync -e test -- sleep 30

# To stop both wrapper and child:
kill -STOP -<process_group_id>   # Stop entire group
kill -CONT -<process_group_id>   # Resume entire group

# To terminate both:
kill -TERM <evalsync_pid>        # Graceful termination (recommended)
kill -TERM -<process_group_id>   # Terminate entire group

Finding the process group ID:

# Get process group ID from evalsync PID
ps -o pgid= -p <evalsync_pid>

# Or use pgrep to find and signal
pkill -STOP -f "evalsync.*sleep"   # Stop all matching processes
pkill -CONT -f "evalsync.*sleep"   # Resume all matching processes

Environment Variables

• EVALSYNC_EXPERIMENT_ID: Default experiment ID
• EVALSYNC_CLIENT_ID: Default client ID
• EVALSYNC_VERBOSE: Enable verbose mode ("true"/"1")
• EVALSYNC_TIMEOUT: Default timeout in seconds

Python API

from evalsync.worker import ExperimentWorker
from evalsync.manager import ExperimentManager

# Worker usage
worker = ExperimentWorker("experiment-1", "client-1", verbose=True)
worker.ready()
worker.wait_for_start()
# ... warmup ...
worker.measure_start()
# ... measured work ...
worker.measure_end()
# ... cleanup ...
worker.wait_for_stop()
worker.cleanup()

# If measurement end coincides with workload end, you can skip measure_end()
# and let wait_for_stop() transition MEASURING → DONE on END.

# Manager usage
manager = ExperimentManager("experiment-1", num_workers=2, verbose=True)
manager.wait_for_all_workers()
manager.start_all()
# ... wait for work to complete ...
manager.stop_all()
manager.cleanup()

Development

Setup

Install dependencies:

uv sync --group dev

Generate Protobuf Proto

protoc --proto_path=../proto --python_out=src/evalsync/proto --mypy_out=src/evalsync/proto ../proto/sync.proto

Running Tests

uv run pytest

Linting and Type Checking

uv run ruff check
uv run mypy src/

Submit to PyPI

uv build
uv publish