@circleci/agents

@circleci/agents

A simple toolkit for CircleCI agents, including tools for analyzing and fixing flaky tests and automating pull request workflows.

Requirements

• Node.js 22.x.x
• pnpm (recommended)

Install

pnpm install

Build

pnpm build

Usage

Analyze and Fix Flaky Tests

This script analyzes CircleCI test failures and suggests (and can apply) fixes for flaky tests.

pnpm cli fix-flaky-tests --git-url https://github.com/circleci-petri/quality-agent-target.git

you will have to setup in the target workspace the expected flaky test api output - url to fetch it would look something like this:

https://circleci.com/api/v2/insights/github/circleci-petri/quality-agent-target/flaky-tests

Environment variables required:

• GITHUB_TOKEN (GitHub personal access token for PR creation and private repo access)
• At least one AI API key:
  • ANTHROPIC_API_KEY (Anthropic Claude API key) -OR-
  • OPENAI_API_KEY (OpenAI API key for Codex fallback)

Optional environment variables:

• CCI_AGENT_OPENAI_MODEL (OpenAI model to use, defaults to 'o3')

Running Canary

To deploy and test the canary version of the flaky tests pipeline:

1. Remove Existing Pipeline Definition

• Navigate to your project settings in CircleCI
• Open browser developer tools and filter network requests for pipeline-definitions
• Delete the existing fix-flaky-tests pipeline definition

2. Create New Pipeline Definition

• Return to the pipelines page and click the "Fix Flaky Tests" button
• Once the pipeline is created, copy the POST curl request from the network tab that created the pipeline definition
• Modify the copied curl request:
  • Change the file_path parameter in the request body to:

    "file_path": "circleci-agents/configs/fix-flaky-tests/canary.yml"
    

• Execute the modified curl request to create the pipeline definition with the canary configuration

3. Verify Configuration

After completing the above steps, triggering the pipeline using the "Fix Flaky Tests" button will now use the canary configuration instead of the standard one.

4. Test Locally

testing rollbar functionality locally:

pnpm cli report-fix-flaky-tests

This should make a failed to run error in rollbar.