@meerkly/sdk

@meerkly/sdk

A plug-and-play browser for Electron apps. Two things in one package:

• A local browser API — render any page in a hidden, hardened Chromium window and get its HTML (or raw JSON) back. No configuration, no account, no app.whenReady() sequencing.
• A Meerkly network worker — share the same browser's idle capacity with the Meerkly network and earn credits for the crawls it serves. One start() call: the SDK handles identity, enrollment, the gateway connection, job serving, and crash recovery.

npm install @meerkly/sdk

Requires Electron ≥ 28 (peer dependency) — the SDK runs in your app's main process.

Quick start

// main.ts (Electron main process)
import { MeerklySDK } from '@meerkly/sdk';

const sdk = new MeerklySDK({ workerKey: 'mk_wk_…' }); // your key from the Meerkly dashboard
void sdk.start();                                     // join the network (auto-enrolls this install)

// Local, self-hosted requests through the same engine — auth-free:
const page = await sdk.fetch('https://example.com');
console.log(page.title, page.httpStatus, page.html?.length);

That's a complete integration. You don't need to wait for app.whenReady() — the SDK does that internally. Local requests and network jobs share one engine and are serialized internally, so they never interleave. A runnable app lives in examples/basic-worker.

The local browser API

Works with new MeerklySDK({}) — no auth, no start().

const result = await sdk.fetch(url, {
  waitFor: 'stable',      // 'stable' (default) | 'domcontentloaded' | 'networkidle' | a CSS selector
  settleMs: 5000,         // cap for the stable-settle wait (0 = spec default)
  waitRules: [            // optional: first matching guard picks the target wait
    { if: '#captcha', then: '#content' }
  ],
  detectMs: 800           // how long to probe the waitRules guards
});

fetch() resolves to a FetchResult:

Field	Meaning
success	The page loaded and HTML was extracted
html	Page HTML — or the raw JSON payload when the document was a JSON media type
format	'html' or 'json' (how to read html)
finalUrl, title	After redirects
httpStatus	Main-document HTTP status (0 if not captured)
loadedMs	Wall-clock load time
waitTimedOut, matchedRule	Wait-condition outcome (matchedRule = index into waitRules, -1 = none)
error	Failure reason when success is false

Wait semantics: stable resolves once the DOM has been structurally quiet for 500 ms (ignoring attribute churn, so CSS animations don't block it), capped by settleMs; networkidle waits for zero in-flight requests for 500 ms; a CSS selector waits for the element to be visible, then settles. These are the same spec-defined semantics gateway jobs use.

Need more control? sdk.browser exposes the underlying HiddenBrowser: navigateToUrl(url), getStatus(), setVisible(true) / toggleVisible() (a debug view that shows the hidden window so you can watch it work). Never construct a second HiddenBrowser yourself — the engine serializes all work through one window and owns its session partition.

The engine is deliberately hardened: hidden window, persistent app-scoped session (never your users' real browser profile), downloads blocked, all permission prompts denied, window.open denied, sandboxed renderer, automatic recovery when a page crashes the renderer.

Joining the Meerkly network

start() needs one auth mode:

Mode	For	Behavior
workerKey: 'mk_wk_…'	Third-party apps (default)	Hard-code your key. Every end-user install enrolls itself as a device under your Meerkly account and earns credits for you. The key is enrollment-only: it grants no account reads, serves no crawls, and is never sent to the gateway. Per-key device limits cap misuse if it's extracted from your binary.
oauth: { clientId }	First-party Meerkly apps	Interactive PKCE sign-in (system browser + loopback callback) + device pairing; each end user pairs to their own account. Adds signIn() / signOut() / getAuthStatus() / getAccessToken() and the 'auth' event. Requires a Doorkeeper client with seeded loopback redirect URIs on the account service.
getDeviceToken: async () => …	Custom pairing	Bring your own device token; re-read on every (re)connect so rotation needs no restart.

Lifecycle:

const sdk = new MeerklySDK({ workerKey: 'mk_wk_…' });

sdk.on('status', (s) => console.log('worker:', s));
// idle → enrolling → connecting → connected  (workerKey mode)
// idle → waiting_for_auth → connecting → connected  (oauth mode, until signIn())
// 'paused' = terminal auth rejection (fix pairing, then reconnect());  'stopped' after stop()

sdk.on('job', ({ jobId, url, success, error }) => console.log('served', url, success));
sdk.on('error', (err) => console.warn(err.message));

await sdk.start();   // resolves identity, obtains a device token, connects, serves jobs
// …
await sdk.stop();    // leaves the network; disposes the engine only if the SDK created it

What start() does for you: waits for Electron app-ready → resolves a stable per-install machine id (random UUID persisted in stateDir, never derived from hardware) → obtains a device token (enrollment for workerKey, stored pairing for oauth) → connects to the gateway with auto-reconnect/backoff → serves fetch jobs through the shared engine. Remotely-dispatched jobs are SSRF-guarded (private/loopback/link-local targets are refused before the browser ever sees them) and bounded (max 3 pending jobs).

Options reference

All optional. Defaults are production-ready.

interface MeerklySDKOptions {
  // Auth (one of; precedence: getDeviceToken > workerKey > oauth)
  workerKey?: string;
  oauth?: { clientId: string; callbackPorts?: number[] };
  getDeviceToken?: () => Promise<string | null>;

  // Endpoints (default: production)
  gatewayUrl?: string;        // wss://gateway.meerkly.com/v1/connect ; ws:// allowed only for loopback
  accountBaseUrl?: string;    // https://account.meerkly.com

  // Identity & persistence
  stateDir?: string;          // default: app.getPath('userData')/meerkly — holds machine.json + tokens
  machineId?: string;         // explicit UUID override (the SDK won't persist it)
  workerId?: string;          // stable replica name → derived identity for volume-less containers
  tokenStore?: DeviceTokenStore;       // custom device-token persistence
  oauthTokenStore?: OAuthTokenStore;   // custom OAuth-token persistence (oauth mode)

  // Presentation & environment
  workerName?: string;        // display name in the account's device list (default: hostname)
  deviceInfo?: Partial<DeviceInfo>;    // override collected device facts (e.g. appVersion)
  platform?: 'desktop' | 'android' | 'server';  // wire platform; Electron apps are 'desktop'
  browser?: HiddenBrowser;    // inject a shared engine instead of letting the SDK create one
  logger?: BrowserLogger;     // { debug, info, warn, error } — default: silent
  allowInsecure?: boolean;    // permit http:// to a non-local account host (dev networks only)
}

Token storage prefers OS encryption (Electron safeStorage — Keychain/DPAPI/libsecret) and falls back to 0600 plaintext files where no keychain exists. Inject tokenStore/oauthTokenStore to change that policy.

OAuth mode extras

const sdk = new MeerklySDK({ oauth: { clientId: 'my-first-party-app' } });
await sdk.start();                    // 'waiting_for_auth' until the user signs in

sdk.on('auth', (status) => render(status));  // pushed on sign-in/out and session expiry

const status = await sdk.signIn();    // opens the system browser; pairs the device; connects
// { isSignedIn: true, email, deviceLinked: true }

const token = await sdk.getAccessToken();    // for your own account-API calls (e.g. GET /api/credits)
await sdk.signOut();                  // clears the session; pairing + worker connection stay up

Transient pairing failures retry automatically (30 s doubling to a 10 min cap); terminal ones (device claimed by another account, stale token scope) surface in getAuthStatus().deviceLinkError.

Good to know

• Main process only. The SDK creates BrowserWindows and uses safeStorage — don't import it in a renderer.
• One SDK instance per app. The nav chain and session-partition ownership assume it.
• The engine reports its honest identity. The User-Agent is Electron's own with only the app name token stripped (see src/browser/userAgent.ts for the measured reasoning) — don't layer UA spoofing on top; it gets detected.
• Wait/extraction semantics are spec-enforced (api-gateway/spec/, shared with the Android and headless workers). Protocol changes must update the spec + vectors + all workers together.

Development (this repo)

Part of the meerkly-desktop npm workspace; the Meerkly desktop app is the reference consumer.

npm run build -w @meerkly/sdk
npm test -w @meerkly/sdk       # needs the api-gateway repo checked out as a sibling (or SPEC_DIR set)
npm run publish:sdk            # from the repo root: build + test + publish to npm (see scripts/publish-sdk.sh)