@gitmaxd/convex-firecrawl-agent

Convex Firecrawl Agent

Publishable Convex Component that wraps Firecrawl's async /v2/agent API as durable, reactive job resources.

Turn long-running agent workflows into something you can treat like any other Convex data model: create once, subscribe reactively, and render status/results without writing your own polling + retry + storage plumbing.

What you get

• Durable jobs: Create once; observe status and results via reactive queries.
• Webhook-first architecture: Real-time status updates via Firecrawl webhooks, with bounded reconciliation fallback for reliability.
• Retry with backoff: Kickoff failures (429, 5xx, network errors) retry up to 5 times with exponential backoff.
• Cancellation: Cancel locally and (best-effort) remotely. If Firecrawl already completed, the job ends completed (completion wins).
• Cleanup: deleteJob removes terminal jobs (and their events, plus any stored result file) to keep UIs tidy and support retention policies.
• Large result handling: Results larger than 750 KiB are stored in Convex file storage with an inline preview; use getResultDownloadUrl when resultStorageId is present.
• Observability: Optional per-job event timeline for debugging and UI timelines.

Found a bug or have a feature request? Please open an issue: https://github.com/gitmaxd/convex-firecrawl-agent/issues

Model selection

The component supports Firecrawl's agent models via the optional model parameter:

Model	Cost	Best for
spark-1-mini	60% cheaper	Most tasks (default)
spark-1-pro	Standard	Complex multi-domain research

Start with the default (spark-1-mini) for cost efficiency. Use spark-1-pro when accuracy is critical or tasks require advanced reasoning across multiple sources.

See Firecrawl model docs for detailed comparison.

Demo

See the example/ directory for a complete working demo (Convex backend + Vite React UI) that demonstrates the full job lifecycle: create → observe → cancel → delete, with live event streaming and result viewing.

Install

npm i @gitmaxd/convex-firecrawl-agent

Prerequisites

• Convex project: If you don't have one yet, run npx convex dev to create a new project (you'll be prompted to log in or create a free account).
• Firecrawl API key: Get one at firecrawl.dev, then set it in your Convex deployment:

  npx convex env set FIRECRAWL_API_KEY "<your-key>"
  

Add to your Convex app

// convex/convex.config.ts
import { defineApp } from "convex/server";
import convexFirecrawlAgent from "@gitmaxd/convex-firecrawl-agent/convex.config.js";

const app = defineApp();
app.use(convexFirecrawlAgent);

export default app;

Usage

Recommended pattern: expose an app-level API

Your Convex app should not call component functions directly from the client. Instead, define app functions that:

• authenticate the caller
• map identity → userId
• delegate to the component

Use exposeApi to generate app-level queries/mutations with your auth mapping:

// convex/firecrawlAgent.ts
import { components } from "./_generated/api";
import { exposeApi } from "@gitmaxd/convex-firecrawl-agent";

export const {
  createJob,
  getJob,
  listJobs,
  cancelJob,
  deleteJob,
  listJobEvents,
} = exposeApi(components.convexFirecrawlAgent, {
  auth: async (ctx) => {
    const userId = (await ctx.auth.getUserIdentity())?.subject;
    if (!userId) throw new Error("Unauthorized");
    return userId;
  },
  includeEvents: true,
  // Optional override for tests or custom limits:
  // resultStorageThresholdBytes: 750 * 1024,
});

Job lifecycle (create → observe → cancel → cleanup)

From your client (via your app mutations/queries) or from other Convex functions:

// create
const jobId = await ctx.runMutation(api.firecrawlAgent.createJob, {
  targetId: "doc:123",
  prompt: "Extract a short summary",
  urls: ["https://news.convex.dev"],
});

// observe (reactive when used in a client query)
const job = await ctx.runQuery(api.firecrawlAgent.getJob, { jobId });

// cancel (optional)
await ctx.runMutation(api.firecrawlAgent.cancelJob, { jobId });

// cleanup (recommended for terminal jobs)
await ctx.runMutation(api.firecrawlAgent.deleteJob, { jobId });

Guardrails & limits

To keep job documents durable and Firecrawl requests valid, the component enforces a few guardrails up front (the mutation throws and the job is not created if these fail):

• prompt length: maximum 10,000 characters. If exceeded: Prompt exceeds maximum length of 10000.
• schema must be JSON-serializable: the schema is stored on the job and forwarded as JSON, so it must round-trip through JSON.stringify(...) (no BigInt, functions, or circular references). If invalid: Schema must be JSON-serializable.
• Large-result storage threshold must be valid: resultStorageThresholdBytes (and the RESULT_STORAGE_THRESHOLD_BYTES env override used by the helper wrappers) must be a finite number ≥ 0.
  • If you pass an invalid resultStorageThresholdBytes, the wrapper throws: resultStorageThresholdBytes must be a finite number >= 0.
  • If RESULT_STORAGE_THRESHOLD_BYTES is set but invalid, it is ignored (treated as unset).
  • 0 is valid and forces results to always be stored in file storage.
• maxCredits must be valid: when provided, must be a finite integer > 0. Invalid values (non-finite, non-integer, or ≤ 0) cause the mutation to throw and no job is created.
• model selection: optional model parameter accepts "spark-1-mini" or "spark-1-pro" to select the Firecrawl agent model.

Type-safe integration (single source of truth)

import { exposeApi } from "@gitmaxd/convex-firecrawl-agent";
import { components } from "./_generated/api";

const agentApi = exposeApi(components.convexFirecrawlAgent, {
  auth: async (ctx, operation) => {
    const userId = (await ctx.auth.getUserIdentity())?.subject;
    if (!userId) throw new Error("Unauthorized");
    return userId;
  },
});

type AgentApi = typeof agentApi;
type AuthFn = Parameters<typeof exposeApi>[1]["auth"]; // (ctx, op) => Promise<string>
type CreateJobReturn = Awaited<ReturnType<AgentApi["createJob"]>>; // string jobId

Large result handling

If a job result exceeds 750 KiB, the component stores the full payload in Convex file storage and keeps an inline preview. Use getResultDownloadUrl to fetch a signed download URL when resultStorageId is present.

Observability: job events

Enable includeEvents: true in exposeApi to record a per-job event timeline. The demo UI uses this to show state transitions and retry/backoff behavior.

Troubleshooting

• Missing FIRECRAWL_API_KEY: set it with npx convex env set FIRECRAWL_API_KEY "...".
• Jobs stuck in processing: check the job's event timeline and fields like attempt, lastHttpStatus, firecrawlStatus, and error; verify your API key is valid.

See the full troubleshooting guide: docs/10-troubleshooting.md.

Documentation

• Start here: docs/02-getting-started.md
• Full doc index: docs/README.md

References

• Convex Components Authoring: https://docs.convex.dev/components/authoring
• Firecrawl Agent feature docs: https://docs.firecrawl.dev/features/agent

License

Apache-2.0. See LICENSE.