@gitmaxd/convex-firecrawl-agent

Convex Firecrawl Agent

Publishable Convex Component that wraps Firecrawl’s async /v2/agent API as durable, reactive job resources.

What you get

• Durable jobs: create once, observe status/results via queries.
• Polling engine: internal actions handle remote execution + backoff.
• Cancellation: cancel locally and (best-effort) remotely; if Firecrawl already completed, the job ends completed (completion wins).
• Cleanup: deleteJob removes completed/cancelled jobs (and their events, plus any stored result file) to keep UIs tidy and support retention policies.
• Large result handling: results larger than 750 KiB are stored in Convex file storage with an inline preview + download link.
• Observability: optional per-job event timeline.

Found a bug or have a feature request? Please open an issue: https://github.com/gitmaxd/convex-firecrawl-agent/issues

Install

npm i @gitmaxd/convex-firecrawl-agent

Add to your Convex app

// convex/convex.config.ts
import { defineApp } from "convex/server";
import convexFirecrawlAgent from "@gitmaxd/convex-firecrawl-agent/convex.config.js";

const app = defineApp();
app.use(convexFirecrawlAgent);

export default app;

Usage (job lifecycle)

In your app’s Convex functions, expose the component’s API with your auth mapping:

// convex/firecrawlAgent.ts
import { components } from "./_generated/api";
import { exposeApi } from "@gitmaxd/convex-firecrawl-agent";

export const {
  createJob,
  getJob,
  listJobs,
  cancelJob,
  deleteJob,
  listJobEvents,
} = exposeApi(components.convexFirecrawlAgent, {
  auth: async (ctx) => {
    const userId = (await ctx.auth.getUserIdentity())?.subject;
    if (!userId) throw new Error("Unauthorized");
    return userId;
  },
  includeEvents: true,
  // Optional operator override for testing or custom limits.
  // resultStorageThresholdBytes: 750 * 1024,
});

Then from your client (or other Convex functions), use the lifecycle operations:

// create
const jobId = await ctx.runMutation(api.firecrawlAgent.createJob, {
  targetId: "doc:123",
  prompt: "Extract a short summary",
  urls: ["https://news.convex.dev"],
});

// observe
const job = await ctx.runQuery(api.firecrawlAgent.getJob, { jobId });

// cancel (optional)
await ctx.runMutation(api.firecrawlAgent.cancelJob, { jobId });

// cleanup (recommended for terminal jobs)
await ctx.runMutation(api.firecrawlAgent.deleteJob, { jobId });

Guardrails & limits

To keep job documents durable and Firecrawl requests valid, the component enforces a few guardrails up front (the mutation throws and the job is not created if these fail):

• prompt length: maximum 10,000 characters. If exceeded: Prompt exceeds maximum length of 10000.
• schema must be JSON-serializable: the schema is stored on the job and forwarded as JSON, so it must round-trip through JSON.stringify(...) (no BigInt, functions, or circular references). If invalid: Schema must be JSON-serializable.
• Large-result storage threshold must be valid: resultStorageThresholdBytes (and the RESULT_STORAGE_THRESHOLD_BYTES env override used by the helper wrappers) must be a finite number ≥ 0.
  • If you pass an invalid resultStorageThresholdBytes, the wrapper throws: resultStorageThresholdBytes must be a finite number >= 0.
  • If RESULT_STORAGE_THRESHOLD_BYTES is set but invalid, it is ignored (treated as unset).
  • 0 is valid and forces results to always be stored in file storage.

API signatures (single source of truth)

import { exposeApi } from "@gitmaxd/convex-firecrawl-agent";
import { components } from "./_generated/api";

const agentApi = exposeApi(components.convexFirecrawlAgent, {
  auth: async (ctx, operation) => {
    const userId = (await ctx.auth.getUserIdentity())?.subject;
    if (!userId) throw new Error("Unauthorized");
    return userId;
  },
});

type AgentApi = typeof agentApi;
type AuthFn = Parameters<typeof exposeApi>[1]["auth"]; // (ctx, op) => Promise<string>
type CreateJobReturn = Awaited<ReturnType<AgentApi["createJob"]>>; // string jobId

Demo app

Judge Quickstart (60–90 seconds)

Prereqs: Node 18+.

• Set your Firecrawl key in your Convex dev deployment:

npx convex env set FIRECRAWL_API_KEY "<your key>"

• Run the demo (runs Convex backend + Vite frontend together):

npm ci
npm run dev

• Open the Vite URL printed in the terminal.

• Click the Extract text preset and press Create job.

Success looks like: the job appears and transitions queued → starting → processing → completed, and a Result panel appears (Summary / Raw JSON / Events) without manual polling.

The example UI demonstrates create/cancel/delete, result viewing, and an events timeline.

Docs

• Start here: docs/02-getting-started.md
• Full doc index: docs/README.md