@pdpp/local-collector

@pdpp/local-collector

Publishable PDPP local collector runtime for filesystem-class connectors.

This package is intentionally narrower than @pdpp/polyfill-connectors: it ships only the local collector runner and the device-exporter client. It contains no connector code. Browser/Patchright-backed connectors stay out of it until each has its own publishability review.

Where connector code comes from

Each supported connector (claude_code, codex, google_takeout, imessage, apple_photos) is a signed Collection Profile published by data-connectors to ghcr.io/pdp-connect/connector/<key>. This package pins one release of each by OCI digest (src/generated/collection-profile-pins.generated.ts), so the package version decides which connector code runs.

The first run (or setup sample) of a connector fetches the pinned release, verifies its Sigstore signature against the data-connectors publish workflow, and installs it under the platform state directory (${XDG_STATE_HOME:-$HOME/.local/state}/pdpp/collection-profiles on Linux, next to the outbox directory; PDPP_COLLECTION_PROFILE_ROOT overrides it). That first run needs network access to ghcr.io and to Sigstore. Later runs check the installed files against the pinned hashes and need no network; an installed copy that fails the check is moved aside and installed again.

google_messages is not available: data-connectors does not publish a Collection Profile for it yet, and the collector refuses it by name.

To move the pins to the catalog's latest releases (network required):

npm run pin:collection-profiles --workspace @pdpp/local-collector

This also rewrites the reference server's copy of the pinned profiles in reference-implementation/server/local-collector-profiles/, which the server uses as the enrollment manifest for these connectors.

For filesystem-class collectors, the local device or host supervisor decides when the process runs. The reference server owns enrollment, ingestion, state, health diagnostics, and optional desired-freshness/request-run signals, but it does not start local processes. PDPP_CONNECTION_ID is the stable connection/source identity for a specific device/account/home binding; the enrollment response currently names that value source_instance_id.

Install and upgrade

Install or upgrade to the current release. Check the affected range below first: @latest resolves to whatever npm currently marks latest, which is an affected version until a corrected release is published.

npm i -g @pdpp/local-collector@latest

Or run without installing, which always resolves the current release:

npx -y @pdpp/local-collector@latest advertise

Affected versions: 1.3.0 through 1.5.4

Every published version from 1.3.0 through 1.5.4 is unrunnable. Those tarballs contain a compiled import ... from "@pdpp/reference-contract" that is not declared as a dependency and does not exist on the npm registry. The import resolved for developers through the workspace link and failed closed on every real install, so any invocation — including --version — exits with:

Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@pdpp/reference-contract'

There is no workaround within that range; upgrading is the fix once a corrected version is published. At the time of writing no such version exists yet, so @latest still resolves to 1.5.4. Name the first fixed version here when it publishes.

Two gates run before any publish and are what detect this failure mode: validate:package rejects any bare import the manifest does not declare, and pack-install-run installs the packed tarball into a clean project and drives the real CLI against a reference server. They cover the import-resolution failures they test for; they are not a guarantee against every packaging defect.

Usage

Guided path — one command pairs the host and saves credentials, no manual JSON/env-var copying:

# @pdpp/local-collector package, npx-launched pdpp-local-collector binary
npx -y @pdpp/local-collector setup \
  --base-url https://<reference-host> \
  --code <one-time-code> \
  --connector claude_code \
  --sample 20   # bounded proof pass: verify without collecting the whole archive

# credentials are saved to ~/.config/pdpp/collectors/claude_code.env (0600);
# run resolves them automatically from --connection-id alone:
npx -y @pdpp/local-collector run --connection-id <source_instance_id>

run prints live progress to stderr as records are found (--quiet to suppress). --sample <n> works on run too, any time you want a bounded preview instead of a full collection.

connect: declare a collection horizon at pairing time

connect performs the same one-time-code exchange as setup, plus an optional collection-horizon REQUEST — no separate owner-side call needed first:

# Recent history only (defaults to 30 days if --recent is given no value):
npx -y @pdpp/local-collector connect \
  --base-url https://<reference-host> \
  --code <one-time-code> \
  --connector claude_code \
  --recent 30

# Explicit full history:
npx -y @pdpp/local-collector connect --base-url https://<reference-host> \
  --code <one-time-code> --connector codex --all

# Custom boundary — since a timestamp, and/or specific project roots:
npx -y @pdpp/local-collector connect --base-url https://<reference-host> \
  --code <one-time-code> --connector claude_code \
  --since 2026-07-01T00:00:00.000Z --source-roots ~/code/project-a,~/code/project-b

Exactly one of --recent, --all, or --since/--source-roots may be given; passing none defers entirely to the server. --since is validated locally (must parse as a date/time) and --source-roots entries that look like filesystem paths are ~-expanded, resolved to absolute paths, and must exist on this host — before any request is sent, not after a round trip that would silently collect nothing.

The scope flags are a REQUEST, never a local completeness authority: the server is the sole arbiter of the effective boundary. A device-declared scope can only NARROW whatever the server already declared (or defaulted) — a request that would WIDEN a server-declared boundary is rejected with a typed error, not silently clamped. When neither side declares a boundary the server defaults to recent history (30 days), never an implicit full pass.

If a profile already exists at the target name (default: the connector id), connect refuses to overwrite it — doing so silently would leave the OLD device credential live and un-revoked on the server with no local record left to revoke it. Pass --force to revoke the existing credential first, then connect and overwrite the profile; a failed revoke aborts before the new code is consumed, so nothing is lost on a failed retry.

Bounded collection horizon (recent history vs. full history)

The one-time --code above comes from the owner-authenticated POST /v1/owner/connections/intents call that mints it (see docs/reference/local-collector.md). That same call accepts an optional collection_scope — { since, source_roots } — declaring the boundary the collector should run within: recent history (e.g. since 30 days ago), a specific project (source_roots), or the default of no bound (full history). This is the same shape connect's scope flags above build; declaring it at intent-mint time (owner-side, before a device ever enrolls) and declaring it via connect (device-side, at enrollment) both apply to every run against that connection from then on — there is no run-time flag that can widen or override it, by design: a local flag must never be able to claim more coverage than the server actually granted.

status and doctor report the boundary currently in force for a lane under scope.active (a fingerprint string; "unscoped" means a full pass), so you can always see what a "complete" run on this connection is complete within. A run only reports coverage as committed once it has exhaustively enumerated that boundary — --sample, an interrupted run, or one stopped by the per-run scan budget always reports coverage_note as NOT committed, never a partial count read as done.

# List connector ids this build accepts.
npx -y @pdpp/local-collector connectors

# Revoke this device's credential on the reference server, then remove its
# saved local profile. Add --local-only to skip the server call (only for an
# unreachable/decommissioned server — the device token stays live otherwise).
npx -y @pdpp/local-collector logout --connector claude_code

# Preview host-local recovery for a stalled collector lane. This loads the
# enrolled local profile for the source instance and changes nothing.
npx -y @pdpp/local-collector recover --source-instance-id <source_instance_id>

# Apply recovery: requeue retryable uploads, rebuild rejected terminal commits from
# a completed pass, and retain old terminal evidence until the replacement is accepted.
npx -y @pdpp/local-collector recover --source-instance-id <source_instance_id> --apply

Low-level / scriptable primitives (unchanged, still supported):

# @pdpp/local-collector package, npx-launched pdpp-local-collector binary
npx -y @pdpp/local-collector advertise

# enroll prints the raw JSON response instead of saving a profile.
npx -y @pdpp/local-collector enroll \
  --base-url https://<reference-host> \
  --code <one-time-code>

# run with credentials supplied entirely via env vars/flags.
PDPP_LOCAL_DEVICE_ID=<device_id> \
PDPP_LOCAL_DEVICE_TOKEN=<device_token> \
PDPP_CONNECTION_ID=<source_instance_id> \
npx -y @pdpp/local-collector run \
  --base-url https://<reference-host> \
  --connector claude_code

The collector sends X-PDPP-Collector-Protocol on enrollment and every device-exporter request. The reference server rejects incompatible versions before persisting records or state.

Install globally if you prefer a persistent binary:

# @pdpp/local-collector package, installs the pdpp-local-collector binary
npm i -g @pdpp/local-collector
pdpp-local-collector advertise

device_token is write-capable for its collector lane. Store it in a secret manager or root-readable env file, and do not print it in logs, issues, or support transcripts.

For a full operator runbook, including Docker move guidance and troubleshooting for 403 after source migration and 409 collector_protocol_mismatch, see docs/reference/local-collector.md.

Connector child-process authority (current state, per D-29)

Decision D-29 records the operative assumption for every officially bundled connector this package ships: official connector source is code-trusted at the same privilege level as the runner until confinement lands. Repository separation (this package living apart from connector content) is governance and release separation, not runtime containment. Custom/untrusted connector execution stays disabled — the published CLI's run --connector accepts only the bundled connector ids.

Under that trust posture, the exact current child-process authority is:

• Environment: the child inherits the entire parent process.env, then the runtime's own reserved keys (PDPP_LOCAL_DEVICE_TOKEN, PDPP_REFERENCE_BASE_URL, PDPP_RUN_ID) are applied, then the connector's own declared env is merged in last — a connector can override the runtime's reserved keys.
• Device token: the device-exporter bearer token is forwarded to every connector child as PDPP_LOCAL_DEVICE_TOKEN.
• Filesystem and network: unrestricted, at the same privilege level as the collector process itself.
• Process execution: the runtime invokes the connector with this process's own Node binary, with node_modules/.bin under the working directory and two levels above it prepended to PATH.
• Working directory: the child's cwd is the directory that holds the installed Collection Profile's entrypoint.

This supersedes older, narrower design text that described the connector child as receiving no device token and an allowlisted-from-scratch environment — that description does not match what this runtime does today for the officially bundled connectors. Sandboxing, credential brokering (a narrower, per-connector token instead of the full device token), and a built-from-scratch allowlisted child environment remain future enablement gates, not requirements this package meets now.