@dpf-it/mcp-server
Advanced tools
+17
| #!/usr/bin/env node | ||
| console.error(` | ||
| @dpf-it/mcp-server has been retired in favor of a remote MCP server — no install needed. | ||
| Point your MCP client at: | ||
| https://api.dpf-it.com/mcp | ||
| VS Code, Cursor, Claude Desktop/Claude.ai, Kiro IDE, and ChatGPT Developer Mode all support | ||
| this directly with OAuth login (a browser popup handles authentication — no manual token | ||
| setup). See the setup guide: | ||
| https://dpf-it.com/integration-guide.html#mcp | ||
| If your client only supports local (stdio) MCP servers and does not yet support OAuth | ||
| (e.g. Codex CLI, Kiro CLI), there is currently no working DPF MCP option for it — check | ||
| the guide above for updates once that client adds OAuth support. | ||
| `); | ||
| process.exit(1); |
+10
-21
| { | ||
| "name": "@dpf-it/mcp-server", | ||
| "version": "0.1.11", | ||
| "version": "1.0.0", | ||
| "mcpName": "com.dpf-it/mcp-server", | ||
| "description": "MCP server for DPF - an AI-driven data integration platform built API-first for AI agents. Lets AI agents discover the DPF API, onboard new users, and run end-to-end data integration workflows.", | ||
| "description": "RETIRED — DPF now runs a remote MCP server at https://api.dpf-it.com/mcp. Installing this package prints setup instructions and exits.", | ||
| "license": "MIT", | ||
| "type": "module", | ||
| "homepage": "https://dpf-it.com", | ||
| "homepage": "https://dpf-it.com/integration-guide.html#mcp", | ||
| "bugs": { | ||
@@ -19,3 +19,4 @@ "url": "https://dpf-it.com/contact" | ||
| "ai-agent", | ||
| "iceberg" | ||
| "iceberg", | ||
| "deprecated" | ||
| ], | ||
@@ -26,7 +27,9 @@ "publishConfig": { | ||
| "bin": { | ||
| "dpf-mcp": "dist/index.js" | ||
| "dpf-mcp": "bin.js" | ||
| }, | ||
| "scripts": { | ||
| "build": "true" | ||
| }, | ||
| "files": [ | ||
| "dist", | ||
| "examples", | ||
| "bin.js", | ||
| "README.md", | ||
@@ -37,17 +40,3 @@ "LICENSE" | ||
| "node": ">=18" | ||
| }, | ||
| "scripts": { | ||
| "build": "tsc -p tsconfig.json", | ||
| "start": "node dist/index.js", | ||
| "dev": "tsx src/index.ts" | ||
| }, | ||
| "dependencies": { | ||
| "@modelcontextprotocol/sdk": "^1.12.0", | ||
| "zod": "^3.23.8" | ||
| }, | ||
| "devDependencies": { | ||
| "@types/node": "^20.14.0", | ||
| "tsx": "^4.16.0", | ||
| "typescript": "^5.5.0" | ||
| } | ||
| } |
+20
-90
@@ -1,102 +0,32 @@ | ||
| # @dpf-it/mcp-server | ||
| # @dpf-it/mcp-server — retired | ||
| MCP (Model Context Protocol) server for [DPF](https://dpf-it.com) — an AI-driven data integration platform built API-first, with every action available through both the REST API and this MCP server. This server lets AI agents and chat clients discover DPF, register new accounts, connect existing workspaces, and run end-to-end data integration workflows without leaving the conversation. | ||
| This package is retired. [DPF](https://dpf-it.com) now runs a **remote MCP server** — | ||
| no install, no npm package, no local Node.js required. | ||
| ## Quick start | ||
| ## Use the remote server instead | ||
| No account or credentials required to begin — just add the server to your MCP client config: | ||
| Point your MCP client at: | ||
| ```json | ||
| { | ||
| "mcpServers": { | ||
| "dpf": { | ||
| "command": "npx", | ||
| "args": ["-y", "@dpf-it/mcp-server"] | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| Then in a chat session, ask the agent to "get started with DPF." If credentials are already saved from a previous session it will use them and proceed straight to your task; otherwise it will ask whether you have an existing account or workspace credential, or walk you through registration (including the emailed verification code) — and it remembers your connection for future sessions. | ||
| ### Pre-authorizing tools (skip per-tool permission prompts) | ||
| Most MCP clients ask for permission before each tool runs. To approve all DPF tools up front: | ||
| **Claude Code** — add the server to the `permissions.allow` list in your project's `.claude/settings.json` (or `~/.claude/settings.json` for all projects): | ||
| ```json | ||
| { | ||
| "permissions": { | ||
| "allow": ["mcp__dpf"] | ||
| } | ||
| } | ||
| https://api.dpf-it.com/mcp | ||
| ``` | ||
| `mcp__dpf` approves every tool from this server. To be more selective, list individual tools as `mcp__dpf__<tool_name>` (e.g. `mcp__dpf__submit_query`). You can also manage this interactively with the `/permissions` command. | ||
| VS Code, Cursor, Claude Desktop/Claude.ai, Kiro IDE, and ChatGPT Developer Mode all support | ||
| this directly: connecting opens a browser login (OAuth 2.1) — your password is typed on | ||
| DPF's own page, never inside a chat — and the client stores its own session from then on, | ||
| including for brand-new accounts (the same login page handles sign-up). | ||
| **Claude Desktop / claude.ai** — when a tool first asks for permission, choose "Always allow" to stop prompts for that tool in future chats. | ||
| Full setup instructions per client: **https://dpf-it.com/integration-guide.html#mcp** | ||
| **Other clients** — most have an equivalent auto-approve setting for MCP tools (e.g. Cursor's auto-run mode); check your client's MCP settings. | ||
| ## If your client doesn't support OAuth yet | ||
| Note that pre-authorizing lets the agent call any DPF tool without pausing for confirmation — including `register_account` and `delete_data_spec`. If that's a concern, allow everything except the tools you want to confirm manually. | ||
| A small number of clients (Codex CLI, Kiro CLI) don't yet support the OAuth browser flow | ||
| that the remote server requires, and this package no longer provides a working local | ||
| alternative. Check the [integration guide](https://dpf-it.com/integration-guide.html#mcp) | ||
| for updates once those clients add OAuth support. | ||
| ## How authentication works | ||
| ## Why the change | ||
| DPF workspaces issue OAuth2 `client_credentials` pairs (`clientId` / `clientSecret`) from the workspace settings page (or via the `login_and_list_workspaces` + `setup_workspace` / `connect_workspace` tools below). This server: | ||
| 1. Stores those credentials locally at `~/.dpf/credentials.json` (file permissions `0600`), keyed by workspace. | ||
| 2. Exchanges them for short-lived bearer JWTs on demand via DPF's OAuth token endpoint, caching tokens in memory for the life of the process. | ||
| 3. Re-reads the credentials file on every startup, so once a workspace is connected, future sessions/chats need no further login. | ||
| You can connect multiple workspaces; tools accept an optional `workspaceId` and default to the first connected one if omitted. | ||
| ## Tools | ||
| **Bootstrap (no auth required)** | ||
| - `get_started` — explains DPF, reports which workspaces are already connected, and guides the agent through the right onboarding path (use saved credentials / connect existing / register new) | ||
| - `register_account` — create a brand-new DPF account (sends a 6-digit email verification code) | ||
| - `verify_otp` — confirm the emailed verification code; required before first login | ||
| - `resend_otp` — resend the verification code if it never arrived | ||
| - `login_and_list_workspaces` — log in and list the account's workspaces with the user's access role on each (the password itself is never stored) | ||
| - `setup_workspace` — connect the workspace the user chose (or create one if the account has none) and persist its credentials | ||
| - `connect_workspace` — connect an existing workspace using a `clientId`/`clientSecret` from the DPF portal | ||
| - `list_workspaces` — list locally connected workspaces | ||
| **Data operations** | ||
| - `list_data` — list a workspace's specs or jobs (pass `resource: "specs" | "jobs"`) | ||
| - `get_status` — poll a spec's analysis status or a job's data-load status (pass exactly one of `specId` or `jobId`) | ||
| - `delete_data_spec`, `submit_query` | ||
| **Skills (multi-step workflows)** | ||
| - `onboard_data_source` — create a data spec, upload sample/format files, run AI analysis, and wait for it to complete | ||
| - `update_data_spec` — change an existing spec's config and/or replace its sample/format/target-schema files, then re-run AI analysis and wait for it to complete | ||
| - `run_data_job` — create a job for an existing spec, upload data files, start processing, and wait for completion | ||
| ## Example: a hand-rolled web frontend | ||
| [`examples/auth-and-query.html`](examples/auth-and-query.html) is a self-contained, dependency-free reference page showing | ||
| how to talk to the DPF REST API directly from a browser: log in, store the JWT in a cookie, restore/expire that session | ||
| on reload, run a query, and log out. It's deliberately unstyled — the point is the auth/cookie/query mechanics, not the | ||
| UI — so copy it as a starting point and swap in your own styling or charting. | ||
| Before using it, set the `NAMESPACE` constant near the top of the script. **A namespace is always the last 12 | ||
| characters of the workspaceId** — it's a fixed derivation, not a name you choose. Find yours with: | ||
| ``` | ||
| curl -H "Authorization: Bearer <token>" https://api.dpf-it.com/iceberg/v1/namespaces | ||
| ``` | ||
| **Serve it over http(s) — don't open it as a `file://` URL.** Most browsers won't persist cookies for local | ||
| files, so login appears to succeed but every subsequent request goes out with no token and fails. From the | ||
| `examples/` directory: `npx serve` (or `python3 -m http.server`), then open the printed URL. | ||
| ## Configuration | ||
| | Env var | Default | Purpose | | ||
| |---|---|---| | ||
| | `DPF_API_BASE_URL` | `https://api.dpf-it.com` | DPF REST API base URL | | ||
| | `DPF_OAUTH_TOKEN_URL` | `${DPF_API_BASE_URL}/iceberg/v1/oauth/tokens` | OAuth2 client_credentials token endpoint | | ||
| ## License | ||
| MIT | ||
| The remote server give every client a consistent, install-free setup, real per-user | ||
| attribution (not a shared workspace credential), and is the only way for hosted clients | ||
| like ChatGPT — which can't spawn a local process at all — to use DPF's MCP server. |
| import { promises as fs } from 'node:fs'; | ||
| import path from 'node:path'; | ||
| import os from 'node:os'; | ||
| const CREDENTIALS_DIR = path.join(os.homedir(), '.dpf'); | ||
| const CREDENTIALS_PATH = path.join(CREDENTIALS_DIR, 'credentials.json'); | ||
| async function readFile() { | ||
| try { | ||
| const raw = await fs.readFile(CREDENTIALS_PATH, 'utf-8'); | ||
| const parsed = JSON.parse(raw); | ||
| if (!Array.isArray(parsed.workspaces)) | ||
| return { workspaces: [] }; | ||
| return parsed; | ||
| } | ||
| catch (err) { | ||
| if (err.code === 'ENOENT') | ||
| return { workspaces: [] }; | ||
| throw err; | ||
| } | ||
| } | ||
| async function writeFile(data) { | ||
| await fs.mkdir(CREDENTIALS_DIR, { recursive: true }); | ||
| await fs.writeFile(CREDENTIALS_PATH, JSON.stringify(data, null, 2), { | ||
| encoding: 'utf-8', | ||
| mode: 0o600, | ||
| }); | ||
| } | ||
| export async function listWorkspaceCredentials() { | ||
| const data = await readFile(); | ||
| return data.workspaces; | ||
| } | ||
| export async function getWorkspaceCredential(workspaceId) { | ||
| const data = await readFile(); | ||
| return data.workspaces.find((w) => w.workspaceId === workspaceId); | ||
| } | ||
| export async function getDefaultWorkspaceCredential() { | ||
| const data = await readFile(); | ||
| return data.workspaces[0]; | ||
| } | ||
| export async function saveWorkspaceCredential(cred) { | ||
| const data = await readFile(); | ||
| const existingIndex = data.workspaces.findIndex((w) => w.workspaceId === cred.workspaceId); | ||
| if (existingIndex >= 0) { | ||
| data.workspaces[existingIndex] = cred; | ||
| } | ||
| else { | ||
| data.workspaces.push(cred); | ||
| } | ||
| await writeFile(data); | ||
| } | ||
| export async function removeWorkspaceCredential(workspaceId) { | ||
| const data = await readFile(); | ||
| const before = data.workspaces.length; | ||
| data.workspaces = data.workspaces.filter((w) => w.workspaceId !== workspaceId); | ||
| await writeFile(data); | ||
| return data.workspaces.length < before; | ||
| } | ||
| export { CREDENTIALS_PATH }; |
| const API_BASE_URL = process.env.DPF_API_BASE_URL || 'https://api.dpf-it.com'; | ||
| const OAUTH_TOKEN_URL = process.env.DPF_OAUTH_TOKEN_URL || `${API_BASE_URL}/iceberg/v1/oauth/tokens`; | ||
| // Cache JWTs per clientId for the lifetime of this server process. | ||
| const tokenCache = new Map(); | ||
| export class DpfApiError extends Error { | ||
| body; | ||
| constructor(message, body) { | ||
| super(message); | ||
| this.body = body; | ||
| } | ||
| } | ||
| async function postJson(url, body, headers = {}) { | ||
| const res = await fetch(url, { | ||
| method: 'POST', | ||
| headers: { 'Content-Type': 'application/json', ...headers }, | ||
| body: JSON.stringify(body), | ||
| }); | ||
| const data = await res.json().catch(() => ({})); | ||
| if (!res.ok || data?.success === false) { | ||
| throw new DpfApiError(data?.error?.message || `Request to ${url} failed (${res.status})`, data); | ||
| } | ||
| return data; | ||
| } | ||
| /** Register a new DPF account. No auth required. */ | ||
| export async function registerAccount(params) { | ||
| return postJson(`${API_BASE_URL}/auth/register`, params); | ||
| } | ||
| /** Verify a newly registered account's email with the 6-digit OTP sent to it. */ | ||
| export async function verifyOtp(email, otp) { | ||
| return postJson(`${API_BASE_URL}/auth/verify-otp`, { email, otp }); | ||
| } | ||
| /** Resend the registration verification code to an unverified account. */ | ||
| export async function resendOtp(email) { | ||
| return postJson(`${API_BASE_URL}/auth/resend-otp`, { email }); | ||
| } | ||
| /** Log in with email/password, returns a short-lived JWT (not persisted by this server). */ | ||
| export async function login(email, password) { | ||
| const data = await postJson(`${API_BASE_URL}/auth/login`, { email, password }); | ||
| return data.data.token; | ||
| } | ||
| /** List all workspaces the logged-in user can access, including their permission on each. */ | ||
| export async function getWorkspaces(token) { | ||
| return postJson(`${API_BASE_URL}/workspaces`, { action: 'get-workspaces' }, { Authorization: `Bearer ${token}` }); | ||
| } | ||
| /** Create a workspace using a freshly-obtained user JWT (e.g. right after login). */ | ||
| export async function createWorkspace(token, name, description) { | ||
| return postJson(`${API_BASE_URL}/workspaces`, { action: 'create', name, description }, { Authorization: `Bearer ${token}` }); | ||
| } | ||
| /** Create a client_credentials pair for a workspace, using a freshly-obtained user JWT. */ | ||
| export async function createCatalogCredential(token, workspaceId, label) { | ||
| const data = await postJson(`${API_BASE_URL}/workspaces`, { action: 'create-catalog-credential', workspaceId, label }, { Authorization: `Bearer ${token}` }); | ||
| return data.data; | ||
| } | ||
| /** | ||
| * Exchange a clientId/clientSecret for a short-lived bearer JWT via the existing | ||
| * OAuth2 client_credentials grant (same endpoint used for Iceberg REST federation). | ||
| */ | ||
| async function exchangeClientCredentials(clientId, clientSecret) { | ||
| const res = await fetch(OAUTH_TOKEN_URL, { | ||
| method: 'POST', | ||
| headers: { 'Content-Type': 'application/x-www-form-urlencoded' }, | ||
| body: new URLSearchParams({ | ||
| grant_type: 'client_credentials', | ||
| client_id: clientId, | ||
| client_secret: clientSecret, | ||
| }), | ||
| }); | ||
| const data = await res.json().catch(() => ({})); | ||
| if (!res.ok || !data.access_token) { | ||
| throw new DpfApiError(data?.error_description || 'OAuth token exchange failed', data); | ||
| } | ||
| return data.access_token; | ||
| } | ||
| /** Get a valid bearer token for a workspace credential, refreshing if expired/missing. */ | ||
| export async function getAccessToken(cred) { | ||
| const cached = tokenCache.get(cred.clientId); | ||
| if (cached && cached.expiresAt > Date.now() + 30_000) { | ||
| return cached.token; | ||
| } | ||
| const token = await exchangeClientCredentials(cred.clientId, cred.clientSecret); | ||
| // DPF JWTs are issued with a 24h expiry; refresh a little early to be safe. | ||
| tokenCache.set(cred.clientId, { token, expiresAt: Date.now() + 23 * 60 * 60 * 1000 }); | ||
| return token; | ||
| } | ||
| /** Generic authenticated call against the /data-specs action endpoint. */ | ||
| export async function dataSpecsAction(cred, action, params = {}) { | ||
| const token = await getAccessToken(cred); | ||
| return postJson(`${API_BASE_URL}/data-specs`, { action, workspaceId: cred.workspaceId, ...params }, { Authorization: `Bearer ${token}` }); | ||
| } | ||
| /** Generic authenticated call against the /connections action endpoint. */ | ||
| export async function connectionsAction(cred, action, params = {}) { | ||
| const token = await getAccessToken(cred); | ||
| return postJson(`${API_BASE_URL}/connections`, { action, workspaceId: cred.workspaceId, ...params }, { Authorization: `Bearer ${token}` }); | ||
| } | ||
| /** Generic authenticated call against the /job-triggers action endpoint. */ | ||
| export async function jobTriggersAction(cred, action, params = {}) { | ||
| const token = await getAccessToken(cred); | ||
| return postJson(`${API_BASE_URL}/job-triggers`, { action, workspaceId: cred.workspaceId, ...params }, { Authorization: `Bearer ${token}` }); | ||
| } | ||
| /** Generic authenticated call against the /workspaces action endpoint. */ | ||
| export async function workspacesAction(cred, action, params = {}) { | ||
| const token = await getAccessToken(cred); | ||
| return postJson(`${API_BASE_URL}/workspaces`, { action, workspaceId: cred.workspaceId, ...params }, { Authorization: `Bearer ${token}` }); | ||
| } | ||
| /** | ||
| * Generic authenticated call against ANY DPF multi-action endpoint, by path — the | ||
| * fallback used by call_dpf_api for endpoints that don't have a dedicated wrapper | ||
| * above. `workspaceId` is merged in like the dedicated *Action helpers; pass it in | ||
| * `params` to override (e.g. omitting it entirely) for endpoints that treat it | ||
| * specially. | ||
| */ | ||
| export async function callApi(cred, path, action, params = {}) { | ||
| const token = await getAccessToken(cred); | ||
| return postJson(`${API_BASE_URL}${path}`, { action, workspaceId: cred.workspaceId, ...params }, { Authorization: `Bearer ${token}` }); | ||
| } | ||
| /** | ||
| * Same as callApi, but authenticates with an explicit bearer token (e.g. a cached | ||
| * user-login session) instead of a workspace API credential. Needed for endpoints | ||
| * like /workspaces that reject workspace credentials outright and require a real | ||
| * user login. No workspaceId is injected automatically — pass it in params if the | ||
| * action needs one. | ||
| */ | ||
| export async function callApiWithToken(token, path, action, params = {}) { | ||
| return postJson(`${API_BASE_URL}${path}`, { action, ...params }, { Authorization: `Bearer ${token}` }); | ||
| } | ||
| /** | ||
| * Submit a SQL query against a workspace's queryable tables. | ||
| * `namespace` is always the last 12 characters of the workspaceId — it is not a chosen name, | ||
| * and a workspace API credential is only ever authorized for its own workspace's namespace. | ||
| */ | ||
| export async function submitQuery(cred, namespace, sql) { | ||
| const token = await getAccessToken(cred); | ||
| return postJson(`${API_BASE_URL}/query`, { namespace, query: sql }, { Authorization: `Bearer ${token}` }); | ||
| } | ||
| /** Upload a local file to a presigned S3 URL returned by create-spec / create-job. */ | ||
| export async function uploadToPresignedUrl(url, fileBuffer, contentType) { | ||
| const res = await fetch(url, { | ||
| method: 'PUT', | ||
| headers: { 'Content-Type': contentType }, | ||
| body: new Uint8Array(fileBuffer), | ||
| }); | ||
| if (!res.ok) { | ||
| throw new DpfApiError(`File upload failed (${res.status})`); | ||
| } | ||
| } | ||
| export { API_BASE_URL }; |
| #!/usr/bin/env node | ||
| import { readFileSync } from 'node:fs'; | ||
| import path from 'node:path'; | ||
| import { fileURLToPath } from 'node:url'; | ||
| import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; | ||
| import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js'; | ||
| import { registerBootstrapTools } from './tools/bootstrap.js'; | ||
| import { registerPrimitiveTools } from './tools/primitives.js'; | ||
| import { registerSkillTools } from './tools/skills.js'; | ||
| import { registerFallbackTools } from './tools/fallback.js'; | ||
| import { registerExampleResources } from './resources.js'; | ||
| // Runs from dist/, so package.json is one level up in both the repo and the published package. | ||
| const pkg = JSON.parse(readFileSync(path.join(path.dirname(fileURLToPath(import.meta.url)), '..', 'package.json'), 'utf-8')); | ||
| async function main() { | ||
| const server = new McpServer({ | ||
| name: 'dpf-mcp-server', | ||
| version: pkg.version, | ||
| }); | ||
| registerBootstrapTools(server); | ||
| registerPrimitiveTools(server); | ||
| registerSkillTools(server); | ||
| registerFallbackTools(server); | ||
| registerExampleResources(server); | ||
| const transport = new StdioServerTransport(); | ||
| await server.connect(transport); | ||
| } | ||
| main().catch((err) => { | ||
| console.error('Fatal error starting DPF MCP server:', err); | ||
| process.exit(1); | ||
| }); |
| import { readFileSync } from 'node:fs'; | ||
| import path from 'node:path'; | ||
| import { fileURLToPath } from 'node:url'; | ||
| // Runs from dist/, so package.json/examples are one level up in both the repo and the published package. | ||
| const packageRoot = path.join(path.dirname(fileURLToPath(import.meta.url)), '..'); | ||
| // html/dpf_openapi.yaml is the DPF repo's source of truth and is deployed to this exact | ||
| // URL by sync-html-to-s3.sh — fetched live (with a short cache) instead of bundled, so it | ||
| // can't drift out of date between publishes of this npm package. | ||
| const OPENAPI_SPEC_URL = 'https://dpf-it.com/dpf_openapi.yaml'; | ||
| const OPENAPI_SPEC_CACHE_TTL_MS = 60 * 60 * 1000; | ||
| let openApiSpecCache; | ||
| async function fetchOpenApiSpec() { | ||
| if (openApiSpecCache && Date.now() - openApiSpecCache.fetchedAt < OPENAPI_SPEC_CACHE_TTL_MS) { | ||
| return openApiSpecCache.text; | ||
| } | ||
| const res = await fetch(OPENAPI_SPEC_URL); | ||
| if (!res.ok) { | ||
| throw new Error(`Failed to fetch DPF OpenAPI spec from ${OPENAPI_SPEC_URL} (${res.status})`); | ||
| } | ||
| const text = await res.text(); | ||
| openApiSpecCache = { text, fetchedAt: Date.now() }; | ||
| return text; | ||
| } | ||
| export function registerExampleResources(server) { | ||
| server.registerResource('auth-and-query-example', 'dpf://examples/auth-and-query.html', { | ||
| title: 'DPF auth + query example (HTML)', | ||
| description: 'Self-contained, dependency-free reference page showing how to talk to the DPF REST API directly ' + | ||
| 'from a browser: log in, store the JWT in a cookie, restore/expire that session on reload, run a ' + | ||
| 'query, and log out. Deliberately unstyled — copy it as a starting point for a hand-rolled frontend.', | ||
| mimeType: 'text/html', | ||
| }, async (uri) => { | ||
| const html = readFileSync(path.join(packageRoot, 'examples', 'auth-and-query.html'), 'utf-8'); | ||
| return { | ||
| contents: [{ uri: uri.href, mimeType: 'text/html', text: html }], | ||
| }; | ||
| }); | ||
| server.registerResource('dpf-openapi-spec', 'dpf://openapi/spec.yaml', { | ||
| title: 'DPF OpenAPI spec', | ||
| description: 'The full DPF REST API spec (source of truth for every endpoint, action, and field — including ones ' + | ||
| 'without a dedicated MCP tool). Read this before calling call_dpf_api for an action you\'re not sure ' + | ||
| `about, instead of guessing field names. Fetched live from ${OPENAPI_SPEC_URL} (cached for up to an ` + | ||
| 'hour), so it always reflects the current API, not just what this server\'s tools were built against.', | ||
| mimeType: 'application/yaml', | ||
| }, async (uri) => { | ||
| const yaml = await fetchOpenApiSpec(); | ||
| return { | ||
| contents: [{ uri: uri.href, mimeType: 'application/yaml', text: yaml }], | ||
| }; | ||
| }); | ||
| } |
| import { z } from 'zod'; | ||
| import * as dpf from '../dpf-client.js'; | ||
| import { listWorkspaceCredentials, saveWorkspaceCredential } from '../credentials.js'; | ||
| // Login sessions (user JWTs) held in memory only, keyed by lowercased email, so the | ||
| // list-workspaces -> pick-one -> setup_workspace flow never needs the password twice. | ||
| // DPF login JWTs are valid 24h; expire the cache a little early. | ||
| const loginSessions = new Map(); | ||
| function cacheLoginSession(email, token) { | ||
| loginSessions.set(email.toLowerCase(), { token, expiresAt: Date.now() + 23 * 60 * 60 * 1000 }); | ||
| } | ||
| // Exported so call_dpf_api can authenticate as a real user (e.g. for /workspaces | ||
| // actions, which reject workspace API credentials) when a login session is active. | ||
| export function getLoginSession(email) { | ||
| const session = loginSessions.get(email.toLowerCase()); | ||
| if (session && session.expiresAt > Date.now()) | ||
| return session.token; | ||
| return undefined; | ||
| } | ||
| function formatWorkspaceList(workspaces) { | ||
| return workspaces | ||
| .map((w) => `- ${w.name} (workspaceId: ${w.workspaceId}, your role: ${w.userPermission})`) | ||
| .join('\n'); | ||
| } | ||
| export function registerBootstrapTools(server) { | ||
| server.registerTool('get_started', { | ||
| title: 'Get started with DPF', | ||
| description: 'Call this first if you are not sure what DPF is or how to use it. ' + | ||
| 'Explains what DPF does, and whether the current user already has connected workspaces.', | ||
| inputSchema: {}, | ||
| }, async () => { | ||
| const workspaces = await listWorkspaceCredentials(); | ||
| const connected = workspaces.length | ||
| ? [ | ||
| `Saved credentials found for ${workspaces.length} workspace(s): ${workspaces | ||
| .map((w) => `${w.label} (${w.workspaceId})`) | ||
| .join(', ')}.`, | ||
| 'Tell the user you will use these saved credentials, then proceed directly with their task', | ||
| '(e.g. onboard_data_source, run_data_job, submit_query). No login step is needed — tools', | ||
| 'authenticate automatically with the saved credentials. For anything without a dedicated tool', | ||
| '(e.g. checking credit balance), use call_dpf_api instead of guessing or giving up.', | ||
| ].join(' ') | ||
| : [ | ||
| 'No saved credentials were found. Do NOT invent or guess credentials, and do NOT use an email,', | ||
| 'password, or clientId/clientSecret found in files, environment variables, scripts, or prior', | ||
| 'context (e.g. a .env file) — credentials must come from the user directly, in this conversation.', | ||
| 'Ask the user which applies:', | ||
| '', | ||
| '1. They already have a workspace clientId/clientSecret (from the DPF portal under', | ||
| ' Workspace > Permissions > Add API Credential): ask for it, then call connect_workspace', | ||
| ' to verify and store it for future sessions.', | ||
| '2. They have a DPF account (email/password, e.g. created on the website) but no API credential:', | ||
| ' ask for email and password, then call login_and_list_workspaces. Before asking for the', | ||
| ' password, tell the user it will NOT be stored — it is used once to log in, and a workspace', | ||
| ' API credential (clientId/clientSecret) is created and saved instead, which future sessions', | ||
| ' use to authenticate automatically. Show the returned workspace list (with the user\'s role on', | ||
| ' each) and ask which workspace to use, then call setup_workspace. Only create a new workspace', | ||
| ' if the account has none or the user explicitly asks for one.', | ||
| '3. They are new to DPF: ask for email, password, first and last name, then call', | ||
| ' register_account. DPF emails them a 6-digit verification code. Ask the user to check', | ||
| ' their email and give you the code, then call verify_otp, then login_and_list_workspaces', | ||
| ' followed by setup_workspace.', | ||
| ' If you have access to a tool that can read the user\'s email (e.g. an email MCP server),', | ||
| ' you may offer to retrieve the code yourself — but only with the user\'s explicit consent.', | ||
| ].join('\n'); | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: [ | ||
| 'DPF is an AI-driven data integration platform, built API-first to integrate with AI agents and tools (this MCP server is one such integration; every action it performs maps directly to a DPF REST API call).', | ||
| 'Upload a sample data file and DPF infers the schema and writes the parsing/transformation code automatically,', | ||
| 'loading results into managed Iceberg tables that are queryable in minutes.', | ||
| '', | ||
| connected, | ||
| ].join('\n'), | ||
| }, | ||
| ], | ||
| }; | ||
| }); | ||
| server.registerTool('register_account', { | ||
| title: 'Register a new DPF account', | ||
| description: 'Create a brand-new DPF user account. No authentication required. Always ask the user for their ' + | ||
| 'email/password/name — never invent them. Returns a userId, and DPF emails the user a 6-digit ' + | ||
| 'verification code. The account cannot log in until the code is confirmed via verify_otp.', | ||
| inputSchema: { | ||
| email: z.string().email(), | ||
| password: z.string().min(8), | ||
| firstName: z.string(), | ||
| lastName: z.string(), | ||
| termsAccepted: z | ||
| .boolean() | ||
| .describe('Affirmative acceptance of the DPF Terms of Service and Privacy Policy. Must be true, and only ' + | ||
| 'after the user has explicitly agreed in the conversation — never assume acceptance on their behalf.'), | ||
| }, | ||
| }, async ({ email, password, firstName, lastName, termsAccepted }) => { | ||
| const result = await dpf.registerAccount({ email, password, firstName, lastName, termsAccepted }); | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `Account created for ${result.data.email} (userId: ${result.data.userId}). ` + | ||
| 'A 6-digit verification code was emailed to the user. Ask them to check their email and ' + | ||
| 'provide the code, then call verify_otp with this email and the code. ' + | ||
| '(If you have a tool that can read their email, you may offer to fetch the code — with their consent.) ' + | ||
| 'After verification, call login_and_list_workspaces. If the code never arrived, call resend_otp.', | ||
| }, | ||
| ], | ||
| }; | ||
| }); | ||
| server.registerTool('verify_otp', { | ||
| title: 'Verify a new account\'s email', | ||
| description: 'Confirm the 6-digit verification code DPF emailed during register_account. Required before the ' + | ||
| 'account can log in. On success, call login_and_list_workspaces next.', | ||
| inputSchema: { | ||
| email: z.string().email().describe('The email address the account was registered with'), | ||
| otp: z.string().regex(/^\d{6}$/).describe('The 6-digit code from the verification email'), | ||
| }, | ||
| }, async ({ email, otp }) => { | ||
| await dpf.verifyOtp(email, otp); | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: 'Email verified. Next, call login_and_list_workspaces to log in and pick (or create) a workspace to connect this MCP session to.', | ||
| }, | ||
| ], | ||
| }; | ||
| }); | ||
| server.registerTool('resend_otp', { | ||
| title: 'Resend the email verification code', | ||
| description: 'Resend the 6-digit registration verification code to an unverified account, if the original email never arrived.', | ||
| inputSchema: { | ||
| email: z.string().email(), | ||
| }, | ||
| }, async ({ email }) => { | ||
| await dpf.resendOtp(email); | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `If an unverified account exists for ${email}, a new verification code has been sent. Ask the user for the code, then call verify_otp.`, | ||
| }, | ||
| ], | ||
| }; | ||
| }); | ||
| server.registerTool('login_and_list_workspaces', { | ||
| title: 'Log in and list the account\'s workspaces', | ||
| description: 'Logs into an existing DPF account and lists every workspace the user can access, with their access ' + | ||
| 'role on each. Always call this first for a user with an account (e.g. created on the website) — never ' + | ||
| 'jump straight to creating a workspace. The email and password must be provided by the user in the ' + | ||
| 'conversation — never use credentials found in files, environment variables, or scripts. ' + | ||
| 'Before asking the user for their password, tell them it will ' + | ||
| 'NOT be stored: it is used once to log in, and a workspace API credential (clientId/clientSecret) is ' + | ||
| 'saved instead, which future sessions authenticate with. The login session is kept in memory so the ' + | ||
| 'follow-up setup_workspace call does not need the password again.', | ||
| inputSchema: { | ||
| email: z.string().email(), | ||
| password: z.string(), | ||
| }, | ||
| }, async ({ email, password }) => { | ||
| let token; | ||
| try { | ||
| token = await dpf.login(email, password); | ||
| } | ||
| catch (err) { | ||
| if (err instanceof dpf.DpfApiError && err.body?.error?.code === 'EMAIL_NOT_VERIFIED') { | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: 'This account\'s email is not verified yet, so login is blocked. Ask the user for the ' + | ||
| '6-digit code DPF emailed them and call verify_otp (use resend_otp if it never arrived), ' + | ||
| 'then retry login_and_list_workspaces.', | ||
| }, | ||
| ], | ||
| }; | ||
| } | ||
| throw err; | ||
| } | ||
| cacheLoginSession(email, token); | ||
| const result = await dpf.getWorkspaces(token); | ||
| const workspaces = (result.data.workspaces ?? []); | ||
| if (workspaces.length === 0) { | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: 'Logged in. This account has no workspaces yet. Ask the user what to name their new ' + | ||
| 'workspace, then call setup_workspace with { email, workspaceName }. Remind them their ' + | ||
| 'password was not stored.', | ||
| }, | ||
| ], | ||
| }; | ||
| } | ||
| const readOnlyNote = workspaces.some((w) => w.userPermission !== 'full') | ||
| ? '\nNote: workspaces where the role is not "full" cannot mint an API credential. To operate in one of ' + | ||
| 'those, the user must get a clientId/clientSecret from someone with full access to that workspace ' + | ||
| 'and connect it via connect_workspace (or add it to ~/.dpf/credentials.json manually).' | ||
| : ''; | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: [ | ||
| `Logged in. This account has access to ${workspaces.length} workspace(s):`, | ||
| formatWorkspaceList(workspaces), | ||
| '', | ||
| 'Show this list — including each access role — to the user and ask which workspace to use for', | ||
| 'subsequent operations, then call setup_workspace with { email, workspaceId }. Do NOT create a', | ||
| 'new workspace unless the user explicitly asks for one. Remind them their password was not stored.' + | ||
| readOnlyNote, | ||
| ].join('\n'), | ||
| }, | ||
| ], | ||
| }; | ||
| }); | ||
| server.registerTool('setup_workspace', { | ||
| title: 'Connect a chosen workspace (or create one if the account has none)', | ||
| description: 'Second step after login_and_list_workspaces: mints a clientId/clientSecret for the chosen workspace ' + | ||
| 'and persists it locally so future sessions are already authenticated. Pass workspaceId to connect an ' + | ||
| 'existing workspace (preferred), or workspaceName to create a new one — only when the account has no ' + | ||
| 'workspaces or the user explicitly asked for a new one. Uses the in-memory login session; the password ' + | ||
| 'is never stored.', | ||
| inputSchema: { | ||
| email: z | ||
| .string() | ||
| .email() | ||
| .describe('Same email used in login_and_list_workspaces (identifies the cached login session)'), | ||
| workspaceId: z | ||
| .string() | ||
| .optional() | ||
| .describe('Existing workspace to connect, chosen by the user from login_and_list_workspaces'), | ||
| workspaceName: z | ||
| .string() | ||
| .optional() | ||
| .describe('Name for a NEW workspace, e.g. "Acme Production" — only if none exist or the user asked for a new one'), | ||
| workspaceDescription: z.string().optional(), | ||
| }, | ||
| }, async ({ email, workspaceId, workspaceName, workspaceDescription }) => { | ||
| const token = getLoginSession(email); | ||
| if (!token) { | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `No active login session for ${email} (it may have expired). Call login_and_list_workspaces first, then retry.`, | ||
| }, | ||
| ], | ||
| }; | ||
| } | ||
| if ((workspaceId && workspaceName) || (!workspaceId && !workspaceName)) { | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: 'Pass exactly one of workspaceId (connect existing) or workspaceName (create new).', | ||
| }, | ||
| ], | ||
| }; | ||
| } | ||
| let connectedId; | ||
| let label; | ||
| let created = false; | ||
| if (workspaceId) { | ||
| const result = await dpf.getWorkspaces(token); | ||
| const workspaces = (result.data.workspaces ?? []); | ||
| const match = workspaces.find((w) => w.workspaceId === workspaceId); | ||
| if (!match) { | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: [ | ||
| `This account has no access to workspace ${workspaceId}. Available workspaces:`, | ||
| formatWorkspaceList(workspaces), | ||
| ].join('\n'), | ||
| }, | ||
| ], | ||
| }; | ||
| } | ||
| if (match.userPermission !== 'full') { | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `The user's access to workspace "${match.name}" (${workspaceId}) is "${match.userPermission}", ` + | ||
| 'but creating an API credential requires "full" access. Instruct the user to get a ' + | ||
| 'clientId/clientSecret from someone with full access to this workspace (created in the DPF ' + | ||
| 'portal under Workspace > Permissions > Add API Credential). Once they have it, call ' + | ||
| 'connect_workspace with those values — it validates them and writes them to ' + | ||
| '~/.dpf/credentials.json — or they can add them to ~/.dpf/credentials.json manually, then try again.', | ||
| }, | ||
| ], | ||
| }; | ||
| } | ||
| connectedId = match.workspaceId; | ||
| label = match.name; | ||
| } | ||
| else { | ||
| const workspaceResult = await dpf.createWorkspace(token, workspaceName, workspaceDescription); | ||
| connectedId = workspaceResult.data.workspaceId; | ||
| label = workspaceName; | ||
| created = true; | ||
| } | ||
| const { clientId, clientSecret } = await dpf.createCatalogCredential(token, connectedId, 'MCP Server'); | ||
| await saveWorkspaceCredential({ workspaceId: connectedId, label, clientId, clientSecret }); | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `Workspace "${label}" (${connectedId}) is ${created ? 'created and ' : ''}connected. ` + | ||
| 'Let the user know their password was not stored: a workspace API credential (clientId/clientSecret) ' + | ||
| 'was created and saved locally, and future sessions authenticate with that automatically. ' + | ||
| 'Try onboard_data_source next.', | ||
| }, | ||
| ], | ||
| }; | ||
| }); | ||
| server.registerTool('connect_workspace', { | ||
| title: 'Connect an existing workspace credential', | ||
| description: 'Use this if you already have a clientId/clientSecret for a DPF workspace (created from the DPF web portal ' + | ||
| 'under Workspace > Permissions > Add API Credential). Persists it locally so future sessions are pre-authenticated.', | ||
| inputSchema: { | ||
| workspaceId: z.string(), | ||
| label: z.string(), | ||
| clientId: z.string(), | ||
| clientSecret: z.string(), | ||
| }, | ||
| }, async ({ workspaceId, label, clientId, clientSecret }) => { | ||
| // Verify the credential works before saving it. | ||
| await dpf.getAccessToken({ workspaceId, label, clientId, clientSecret }); | ||
| await saveWorkspaceCredential({ workspaceId, label, clientId, clientSecret }); | ||
| return { | ||
| content: [{ type: 'text', text: `Connected workspace "${label}" (${workspaceId}).` }], | ||
| }; | ||
| }); | ||
| server.registerTool('list_workspaces', { | ||
| title: 'List connected workspaces', | ||
| description: 'Lists the DPF workspaces this MCP server is currently connected to.', | ||
| inputSchema: {}, | ||
| }, async () => { | ||
| const workspaces = await listWorkspaceCredentials(); | ||
| if (workspaces.length === 0) { | ||
| return { content: [{ type: 'text', text: 'No workspaces connected yet.' }] }; | ||
| } | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: workspaces.map((w) => `- ${w.label} (workspaceId: ${w.workspaceId})`).join('\n'), | ||
| }, | ||
| ], | ||
| }; | ||
| }); | ||
| } |
| import { z } from 'zod'; | ||
| import * as dpf from '../dpf-client.js'; | ||
| import { resolveWorkspace } from './workspace-resolver.js'; | ||
| import { getLoginSession } from './bootstrap.js'; | ||
| const workspaceIdSchema = z | ||
| .string() | ||
| .optional() | ||
| .describe('Workspace to act on. Defaults to the only/most-recently connected workspace if omitted.'); | ||
| function asText(data) { | ||
| return { content: [{ type: 'text', text: JSON.stringify(data, null, 2) }] }; | ||
| } | ||
| export function registerFallbackTools(server) { | ||
| server.registerTool('call_dpf_api', { | ||
| title: 'Call any DPF API action (fallback for requests with no dedicated tool)', | ||
| description: 'Escape hatch for DPF capabilities that don\'t have a dedicated tool yet. ALWAYS prefer a dedicated ' + | ||
| 'tool when one exists — get_status, list_data, submit_query, delete_data_spec, onboard_data_source, ' + | ||
| 'update_data_spec, run_data_job, manage_connection, manage_trigger, setup_scheduled_pull, or the ' + | ||
| 'workspace/auth tools in get_started\'s flow — and reach for this only when none of those fit (e.g. ' + | ||
| '"how many credits do I have?" -> path "/auth/billing", action "get-balance"; a brand-new action added ' + | ||
| 'to the API since this server\'s tools were last updated). ' + | ||
| 'Every DPF endpoint is POST <path> with a JSON body of { action, ...fields }. By default this ' + | ||
| 'authenticates with the connected workspace\'s API credential (workspaceId is filled in automatically) ' + | ||
| '— that covers most endpoints and any action scoped to that one workspace: data-specs, connections, ' + | ||
| 'job-triggers, "/auth/billing" get-balance, and under "/workspaces" specifically get-workspace (e.g. ' + | ||
| '"who has access to this workspace?" -> action "get-workspace" -> response\'s `permissions` array of ' + | ||
| '{ userId, email, firstName, lastName, permission }), list-queries, list-bytes-accessed, list-storage, ' + | ||
| 'list-processed-files, and list-trigger-runs (workspace-wide usage/audit logs — pass pageSize/cursor/ ' + | ||
| 'filter fields per the OpenAPI spec). Some actions are account-level or cross-workspace instead and ' + | ||
| 'reject a workspace credential outright, requiring a real user login: under "/workspaces", create, ' + | ||
| 'get-workspaces, grant-permission, revoke-permission, update/delete-workspace, and catalog-credential ' + | ||
| 'management; under "/auth/billing", purchase-credits, modify-subscription, and manage-payment. For ' + | ||
| 'those, pass `email` (the same email ' + | ||
| 'used with login_and_list_workspaces, which must have been called earlier in this session — if it ' + | ||
| 'hasn\'t, call it first, even if a workspace credential is already saved) to authenticate as that ' + | ||
| 'logged-in user instead; workspaceId is NOT auto-filled in this mode, so include it in `params` if the ' + | ||
| 'action needs one. If a call unexpectedly fails with "Invalid or expired token" or FORBIDDEN despite a ' + | ||
| 'connected workspace, retry with `email` instead of assuming the credential is broken. ' + | ||
| 'If unsure of an action\'s exact fields, read the "dpf-openapi-spec" resource (dpf://openapi/spec.yaml) ' + | ||
| 'rather than guessing — it\'s fetched live from https://dpf-it.com/dpf_openapi.yaml. If resource reads ' + | ||
| 'aren\'t available, fetch that URL directly, or view it rendered at https://dpf-it.com/api-viewer.html. ' + | ||
| 'Returns the raw response data.', | ||
| inputSchema: { | ||
| workspaceId: workspaceIdSchema, | ||
| path: z.string().describe('API path, e.g. "/auth/billing" (leading slash, no query string).'), | ||
| action: z.string().describe('The "action" field this endpoint routes on, e.g. "get-balance".'), | ||
| email: z | ||
| .string() | ||
| .email() | ||
| .optional() | ||
| .describe('Authenticate as this logged-in user instead of the workspace API credential, using the login ' + | ||
| 'session from an earlier login_and_list_workspaces call in this session. Required for endpoints ' + | ||
| 'that reject workspace credentials (e.g. "/workspaces"). When set, workspaceId is NOT auto-filled ' + | ||
| 'into the request — include it in params if needed.'), | ||
| params: z | ||
| .record(z.any()) | ||
| .optional() | ||
| .describe('Additional action-specific fields to merge into the request body alongside action/workspaceId.'), | ||
| }, | ||
| }, async ({ workspaceId, path, action, email, params }) => { | ||
| if (email) { | ||
| const token = getLoginSession(email); | ||
| if (!token) { | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `No active login session for ${email} (it may have expired, or login_and_list_workspaces ` + | ||
| 'was never called this session). Call login_and_list_workspaces first, then retry.', | ||
| }, | ||
| ], | ||
| }; | ||
| } | ||
| const result = await dpf.callApiWithToken(token, path, action, params ?? {}); | ||
| return asText(result.data); | ||
| } | ||
| const cred = await resolveWorkspace(workspaceId); | ||
| const result = await dpf.callApi(cred, path, action, params ?? {}); | ||
| return asText(result.data); | ||
| }); | ||
| } |
| import { z } from 'zod'; | ||
| import * as dpf from '../dpf-client.js'; | ||
| import { resolveWorkspace } from './workspace-resolver.js'; | ||
| const workspaceIdSchema = z | ||
| .string() | ||
| .optional() | ||
| .describe('Workspace to act on. Defaults to the only/most-recently connected workspace if omitted.'); | ||
| function asText(data) { | ||
| return { content: [{ type: 'text', text: JSON.stringify(data, null, 2) }] }; | ||
| } | ||
| export function registerPrimitiveTools(server) { | ||
| server.registerTool('list_data', { | ||
| title: 'List data specs or jobs', | ||
| description: 'List either the data specs (parsing + mapping rule sets, resource: "specs") or the data processing ' + | ||
| 'jobs (executions of a spec, resource: "jobs") defined in a workspace. Each spec includes its specId ' + | ||
| 'and current status — poll a specific one with get_status.', | ||
| inputSchema: { | ||
| workspaceId: workspaceIdSchema, | ||
| resource: z.enum(['specs', 'jobs']).describe('Which kind of resource to list'), | ||
| }, | ||
| }, async ({ workspaceId, resource }) => { | ||
| const cred = await resolveWorkspace(workspaceId); | ||
| const result = await dpf.dataSpecsAction(cred, resource === 'specs' ? 'list-specs' : 'list-jobs'); | ||
| return asText(result.data); | ||
| }); | ||
| server.registerTool('get_status', { | ||
| title: 'Get spec or job status', | ||
| description: 'Poll the status of either a data spec\'s own process (schema inference + code generation, run by ' + | ||
| 'start-analysis — pass specId, reaches "ready" or "failed") or a data-load job (pass jobId, reaches ' + | ||
| '"complete" or "failed"). Pass exactly one of specId or jobId. Right after create-spec/update-spec + ' + | ||
| 'start-analysis, poll by specId; once that reaches "ready", its response\'s lastJobId (if present) ' + | ||
| 'points at the data-load job — poll that separately by jobId for load progress.', | ||
| inputSchema: { | ||
| workspaceId: workspaceIdSchema, | ||
| specId: z.string().optional().describe('Poll a data spec\'s analysis status. Pass exactly one of specId or jobId.'), | ||
| jobId: z.string().optional().describe('Poll a data-load job\'s status. Pass exactly one of specId or jobId.'), | ||
| }, | ||
| }, async ({ workspaceId, specId, jobId }) => { | ||
| if ((specId && jobId) || (!specId && !jobId)) { | ||
| throw new Error('Pass exactly one of specId (spec analysis status) or jobId (data-load job status).'); | ||
| } | ||
| const cred = await resolveWorkspace(workspaceId); | ||
| const result = await dpf.dataSpecsAction(cred, 'get-status', specId ? { specId } : { jobId }); | ||
| return asText(result.data); | ||
| }); | ||
| server.registerTool('delete_data_spec', { | ||
| title: 'Delete a data spec', | ||
| description: 'Permanently delete a data spec and its associated configuration.', | ||
| inputSchema: { workspaceId: workspaceIdSchema, specName: z.string() }, | ||
| }, async ({ workspaceId, specName }) => { | ||
| const cred = await resolveWorkspace(workspaceId); | ||
| const result = await dpf.dataSpecsAction(cred, 'delete-spec', { specName }); | ||
| return asText(result.data); | ||
| }); | ||
| server.registerTool('submit_query', { | ||
| title: 'Query workspace data', | ||
| description: 'Run a SQL query against the Iceberg tables loaded into a workspace.', | ||
| inputSchema: { | ||
| workspaceId: workspaceIdSchema, | ||
| sql: z.string().describe('SQL query, e.g. SELECT * FROM customers LIMIT 10'), | ||
| }, | ||
| }, async ({ workspaceId, sql }) => { | ||
| const cred = await resolveWorkspace(workspaceId); | ||
| // The query API's namespace is not a human-chosen name — it's the last 12 chars of the | ||
| // workspaceId, and a workspace API credential (what this server always uses) is only ever | ||
| // authorized for its own workspace's namespace. Derive it instead of asking the caller. | ||
| const namespace = cred.workspaceId.slice(-12); | ||
| const result = await dpf.submitQuery(cred, namespace, sql); | ||
| const data = result.data ?? {}; | ||
| // The query API returns typed columns as columnTypes: [{ name, duckdbType, | ||
| // typeOid }]. Surface a name→DuckDB-type schema so the model knows each | ||
| // column's type; the pg typeOid is wire-protocol detail not useful here. | ||
| const schema = (data.columnTypes ?? []).map((c) => ({ name: c.name, type: c.duckdbType })); | ||
| return asText({ | ||
| schema, | ||
| rows: data.rows ?? [], | ||
| rowCount: data.rowCount, | ||
| executionTimeMs: data.executionTimeMs, | ||
| }); | ||
| }); | ||
| server.registerTool('manage_connection', { | ||
| title: 'Manage an external data-source connection (SFTP)', | ||
| description: 'Create, list, test, or delete a workspace connection to an external data source. Currently only ' + | ||
| '"sftp" is supported. create generates a keypair and returns the public key — it must be installed in ' + | ||
| 'the remote server\'s authorized_keys before test (or a trigger using this connection) will succeed. ' + | ||
| 'For a first-time "pull files from this server on a schedule" request, prefer setup_scheduled_pull, ' + | ||
| 'which chains create + test + create-trigger for you.', | ||
| inputSchema: { | ||
| workspaceId: workspaceIdSchema, | ||
| action: z.enum(['create', 'list', 'test', 'delete']), | ||
| type: z.enum(['sftp']).optional().describe('Connection type. Required for create; currently only "sftp".'), | ||
| hostname: z.string().optional().describe('Remote server hostname. Required for create.'), | ||
| username: z.string().optional().describe('Remote username. Optional for create; defaults to "sftpuser".'), | ||
| connectionId: z.string().optional().describe('Existing connection to test or delete. Required for test/delete.'), | ||
| }, | ||
| }, async ({ workspaceId, action, type, hostname, username, connectionId }) => { | ||
| const cred = await resolveWorkspace(workspaceId); | ||
| if (action === 'create') { | ||
| if (!hostname) | ||
| throw new Error('hostname is required to create a connection.'); | ||
| const result = await dpf.connectionsAction(cred, 'create-connection', { type: type ?? 'sftp', hostname, username }); | ||
| return asText(result.data); | ||
| } | ||
| if (action === 'list') { | ||
| const result = await dpf.connectionsAction(cred, 'get-connections'); | ||
| return asText(result.data); | ||
| } | ||
| if (!connectionId) | ||
| throw new Error(`connectionId is required for action "${action}".`); | ||
| if (action === 'test') { | ||
| const result = await dpf.connectionsAction(cred, 'test-connection', { connectionId }); | ||
| return asText(result.data); | ||
| } | ||
| const result = await dpf.connectionsAction(cred, 'delete-connection', { connectionId }); | ||
| return asText(result.data); | ||
| }); | ||
| server.registerTool('manage_trigger', { | ||
| title: 'Manage a scheduled ingestion trigger (SFTP)', | ||
| description: 'Create, list, update, delete, or fire a workspace job trigger that pulls files from a connection into ' + | ||
| 'an already-analyzed data spec on a schedule (hourly/daily/monthly, UTC). Natural-language preRules ' + | ||
| '(which files to pick up) and postRules (what to do after upload) are compiled into executable code ' + | ||
| 'server-side — never pass raw code. The referenced spec must already have been analyzed once (see ' + | ||
| 'onboard_data_source) and the referenced connection must already exist (see manage_connection). For a ' + | ||
| 'first-time "set up a daily/scheduled pull" request, prefer setup_scheduled_pull, which sets up the ' + | ||
| 'connection and trigger together. After firing a trigger (action "run-now") — or any one-off manual ' + | ||
| 'run — use action "run-history" to monitor its outcome: it starts as `status: running` and settles to ' + | ||
| '`success`, `failed`, or `no-files`, with `filesPulled` and a `message`.', | ||
| inputSchema: { | ||
| workspaceId: workspaceIdSchema, | ||
| action: z.enum(['create', 'list', 'update', 'delete', 'run-now', 'clear-processed-files', 'run-history']), | ||
| triggerId: z | ||
| .string() | ||
| .optional() | ||
| .describe('Existing trigger. Required for update/delete/run-now/clear-processed-files.'), | ||
| specName: z.string().optional().describe('Already-analyzed data spec to feed. Required for create.'), | ||
| connectionId: z | ||
| .string() | ||
| .optional() | ||
| .describe('Connection to pull from. Required for create. Also usable as a run-history filter.'), | ||
| type: z.enum(['sftp']).optional().describe('Trigger type. Optional for create; currently only "sftp".'), | ||
| frequency: z | ||
| .object({ | ||
| unit: z.enum(['hourly', 'daily', 'monthly']), | ||
| hourOfDay: z.number().int().min(0).max(23).optional().describe('Required for daily/monthly (UTC).'), | ||
| dayOfMonth: z.number().int().min(1).max(31).optional().describe('Required for monthly.'), | ||
| }) | ||
| .optional() | ||
| .describe('Required for create; optional for update to change the schedule.'), | ||
| preRules: z | ||
| .string() | ||
| .optional() | ||
| .describe('Natural language: which files to pick up (e.g. "only *.csv under /outbound").'), | ||
| postRules: z | ||
| .string() | ||
| .optional() | ||
| .describe('Natural language: what to do after a file loads (e.g. "rename with .done suffix").'), | ||
| dedupe: z | ||
| .boolean() | ||
| .optional() | ||
| .describe('Skip files already loaded into the spec by a previous run (matched by file name).'), | ||
| enabled: z.boolean().optional(), | ||
| specId: z.string().optional().describe('run-history: filter to runs of triggers feeding this spec.'), | ||
| pageSize: z.number().int().min(1).max(100).optional().describe('run-history: records per page (default 25).'), | ||
| cursor: z.string().optional().describe('run-history: opaque `nextCursor` from a prior page (omit for the first page).'), | ||
| startTime: z.string().optional().describe('run-history: ISO 8601 lower bound (inclusive) on when the run started.'), | ||
| endTime: z.string().optional().describe('run-history: ISO 8601 upper bound (inclusive) on when the run started.'), | ||
| }, | ||
| }, async ({ workspaceId, action, triggerId, specName, connectionId, type, frequency, preRules, postRules, dedupe, enabled, specId, pageSize, cursor, startTime, endTime, }) => { | ||
| const cred = await resolveWorkspace(workspaceId); | ||
| if (action === 'create') { | ||
| if (!specName || !connectionId || !frequency) { | ||
| throw new Error('create requires specName, connectionId, and frequency.'); | ||
| } | ||
| const result = await dpf.jobTriggersAction(cred, 'create-trigger', { | ||
| type: type ?? 'sftp', | ||
| specName, | ||
| connectionId, | ||
| frequency, | ||
| preRules, | ||
| postRules, | ||
| dedupe, | ||
| enabled, | ||
| }); | ||
| return asText(result.data); | ||
| } | ||
| if (action === 'list') { | ||
| const result = await dpf.jobTriggersAction(cred, 'get-triggers'); | ||
| return asText(result.data); | ||
| } | ||
| if (action === 'run-history') { | ||
| // list-trigger-runs has no server-side triggerId filter (only specId/connectionId/time | ||
| // range) — filter that one down to a single trigger client-side within the fetched page. | ||
| const result = await dpf.workspacesAction(cred, 'list-trigger-runs', { | ||
| specId, | ||
| connectionId, | ||
| pageSize, | ||
| cursor, | ||
| startTime, | ||
| endTime, | ||
| }); | ||
| const data = result.data; | ||
| if (triggerId && data?.runs) { | ||
| data.runs = data.runs.filter((run) => run.triggerId === triggerId); | ||
| } | ||
| return asText(data); | ||
| } | ||
| if (!triggerId) | ||
| throw new Error(`triggerId is required for action "${action}".`); | ||
| if (action === 'update') { | ||
| const result = await dpf.jobTriggersAction(cred, 'update-trigger', { | ||
| triggerId, | ||
| frequency, | ||
| preRules, | ||
| postRules, | ||
| dedupe, | ||
| enabled, | ||
| }); | ||
| return asText(result.data); | ||
| } | ||
| if (action === 'delete') { | ||
| const result = await dpf.jobTriggersAction(cred, 'delete-trigger', { triggerId }); | ||
| return asText(result.data); | ||
| } | ||
| if (action === 'run-now') { | ||
| const result = await dpf.jobTriggersAction(cred, 'run-trigger-now', { triggerId }); | ||
| return asText(result.data); | ||
| } | ||
| const result = await dpf.jobTriggersAction(cred, 'clear-processed-files', { triggerId }); | ||
| return asText(result.data); | ||
| }); | ||
| } |
| import { z } from 'zod'; | ||
| import { promises as fs } from 'node:fs'; | ||
| import path from 'node:path'; | ||
| import * as dpf from '../dpf-client.js'; | ||
| import { resolveWorkspace } from './workspace-resolver.js'; | ||
| const workspaceIdSchema = z | ||
| .string() | ||
| .optional() | ||
| .describe('Workspace to act on. Defaults to the only/most-recently connected workspace if omitted.'); | ||
| function contentTypeFor(filePath) { | ||
| const ext = path.extname(filePath).toLowerCase(); | ||
| switch (ext) { | ||
| case '.json': | ||
| return 'application/json'; | ||
| case '.xml': | ||
| return 'application/xml'; | ||
| case '.xls': | ||
| return 'application/vnd.ms-excel'; | ||
| case '.xlsx': | ||
| return 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'; | ||
| case '.csv': | ||
| case '.tsv': | ||
| case '.txt': | ||
| case '.dat': | ||
| case '.log': | ||
| return 'text/plain'; | ||
| default: | ||
| return 'application/octet-stream'; | ||
| } | ||
| } | ||
| async function pollJobUntilDone(cred, jobId, { maxPolls = 60, intervalMs = 5000 } = {}) { | ||
| for (let i = 0; i < maxPolls; i++) { | ||
| const result = await dpf.dataSpecsAction(cred, 'get-status', { jobId }); | ||
| const status = result.data.status; | ||
| if (status === 'complete' || status === 'failed') { | ||
| return result.data; | ||
| } | ||
| await new Promise((resolve) => setTimeout(resolve, intervalMs)); | ||
| } | ||
| throw new Error(`Job ${jobId} did not finish within ${(maxPolls * intervalMs) / 1000}s`); | ||
| } | ||
| // A spec's own process (schema inference + code generation, run by start-analysis) is tracked | ||
| // on the data_specs record itself and reaches a different terminal vocabulary than a job: | ||
| // 'ready' or 'failed', not 'complete'/'failed'. Poll by specId, not jobId, to watch it. | ||
| async function pollSpecUntilDone(cred, specId, { maxPolls = 60, intervalMs = 5000 } = {}) { | ||
| for (let i = 0; i < maxPolls; i++) { | ||
| const result = await dpf.dataSpecsAction(cred, 'get-status', { specId }); | ||
| const status = result.data.status; | ||
| if (status === 'ready' || status === 'failed') { | ||
| return result.data; | ||
| } | ||
| await new Promise((resolve) => setTimeout(resolve, intervalMs)); | ||
| } | ||
| throw new Error(`Spec ${specId} analysis did not finish within ${(maxPolls * intervalMs) / 1000}s`); | ||
| } | ||
| export function registerSkillTools(server) { | ||
| // Mirrors the sequence in test_data_specs.sh phases 1-4: | ||
| // create-spec -> upload sample (+optional format file) -> start-analysis -> poll to completion. | ||
| server.registerTool('onboard_data_source', { | ||
| title: 'Onboard a new data source (create + analyze a data spec)', | ||
| description: 'End-to-end workflow to set up a new data integration: creates a data spec, uploads the sample file ' + | ||
| '(and optional format spec file), kicks off AI analysis to infer the schema/transformation, and waits ' + | ||
| 'until the spec\'s own analysis process reaches "ready" or "failed" (polled by specId, not jobId — ' + | ||
| 'analysis and the data load are tracked separately). Use this instead of calling the individual ' + | ||
| 'data-spec tools yourself.', | ||
| inputSchema: { | ||
| workspaceId: workspaceIdSchema, | ||
| specName: z.string(), | ||
| description: z.string().optional(), | ||
| targetOption: z | ||
| .enum(['auto-infer', 'existing-tables', 'target-schema-file']) | ||
| .optional() | ||
| .describe('Where transformed data should land: "auto-infer" (default) lets the AI design new target tables ' + | ||
| 'from the sample, "existing-tables" loads into tables already in the workspace (requires ' + | ||
| 'targetTables), "target-schema-file" uses a provided schema file (requires targetSchemaPath).'), | ||
| targetTables: z | ||
| .array(z.string()) | ||
| .optional() | ||
| .describe('Names of existing workspace tables to load into. Required when targetOption is "existing-tables".'), | ||
| targetSchemaPath: z | ||
| .string() | ||
| .optional() | ||
| .describe('Local filesystem path to a target schema file. Required when targetOption is "target-schema-file".'), | ||
| samplePath: z.string().describe('Local filesystem path to a sample data file'), | ||
| formatPath: z.string().optional().describe('Local filesystem path to an optional format spec file'), | ||
| loadSampleData: z | ||
| .boolean() | ||
| .optional() | ||
| .describe('Whether to load the sample file and trigger the data-load job once analysis finishes ' + | ||
| '(default true). Set false to only infer the schema and generate code, without loading any data.'), | ||
| }, | ||
| }, async ({ workspaceId, specName, description, targetOption, targetTables, targetSchemaPath, samplePath, formatPath, loadSampleData }) => { | ||
| // Default: no explicit target means let the AI infer the target schema. | ||
| const effectiveTargetOption = targetOption ?? (targetSchemaPath ? 'target-schema-file' : targetTables?.length ? 'existing-tables' : 'auto-infer'); | ||
| if (effectiveTargetOption === 'existing-tables' && !targetTables?.length) { | ||
| throw new Error('targetOption "existing-tables" requires targetTables (names of existing workspace tables).'); | ||
| } | ||
| if (effectiveTargetOption === 'target-schema-file' && !targetSchemaPath) { | ||
| throw new Error('targetOption "target-schema-file" requires targetSchemaPath (local path to the schema file).'); | ||
| } | ||
| const cred = await resolveWorkspace(workspaceId); | ||
| const sampleFileName = path.basename(samplePath); | ||
| const formatFileName = formatPath ? path.basename(formatPath) : undefined; | ||
| const targetSchemaFileName = targetSchemaPath ? path.basename(targetSchemaPath) : undefined; | ||
| const createResult = await dpf.dataSpecsAction(cred, 'create-spec', { | ||
| specName, | ||
| description, | ||
| targetOption: effectiveTargetOption, | ||
| targetTables: effectiveTargetOption === 'existing-tables' ? targetTables : undefined, | ||
| targetSchemaFileName: effectiveTargetOption === 'target-schema-file' ? targetSchemaFileName : undefined, | ||
| sampleFileName, | ||
| formatFileName, | ||
| }); | ||
| const { signedUrls, specId } = createResult.data; | ||
| const sampleBuffer = await fs.readFile(samplePath); | ||
| await dpf.uploadToPresignedUrl(signedUrls[sampleFileName], sampleBuffer, contentTypeFor(samplePath)); | ||
| if (formatPath && formatFileName) { | ||
| const formatBuffer = await fs.readFile(formatPath); | ||
| await dpf.uploadToPresignedUrl(signedUrls[formatFileName], formatBuffer, contentTypeFor(formatPath)); | ||
| } | ||
| if (effectiveTargetOption === 'target-schema-file' && targetSchemaPath && targetSchemaFileName) { | ||
| const schemaBuffer = await fs.readFile(targetSchemaPath); | ||
| await dpf.uploadToPresignedUrl(signedUrls[targetSchemaFileName], schemaBuffer, contentTypeFor(targetSchemaPath)); | ||
| } | ||
| await dpf.dataSpecsAction(cred, 'start-analysis', { specName, loadSampleData }); | ||
| const finalStatus = await pollSpecUntilDone(cred, specId); | ||
| const jobNote = finalStatus.lastJobId | ||
| ? ` The data load job ${finalStatus.lastJobId} was triggered — use get_status with that jobId to poll its progress separately.` | ||
| : ''; | ||
| const scheduleNote = finalStatus.status === 'ready' | ||
| ? ' To keep this spec fed automatically from a remote server on a schedule, use setup_scheduled_pull.' | ||
| : ''; | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `Data spec "${specName}" analysis finished with status: ${finalStatus.status}.${jobNote}${scheduleNote}\n${JSON.stringify(finalStatus, null, 2)}`, | ||
| }, | ||
| ], | ||
| }; | ||
| }); | ||
| // Mirrors the update-spec action: change spec fields/files in place, then (by default) re-run | ||
| // start-analysis and poll to completion, same as onboard_data_source does for a new spec. | ||
| server.registerTool('update_data_spec', { | ||
| title: 'Update an existing data spec', | ||
| description: 'Change an existing data spec\'s configuration and/or replace its sample, format, or target-schema ' + | ||
| 'file, then (by default) re-run AI analysis and wait until the spec reaches "ready" or "failed". ' + | ||
| 'Only pass the fields you want to change — omitted fields keep their current value. Set runAnalysis ' + | ||
| 'to false to save changes without reprocessing (e.g. when batching several edits before one analysis run).', | ||
| inputSchema: { | ||
| workspaceId: workspaceIdSchema, | ||
| specName: z.string().describe('Name of the existing data spec to update'), | ||
| description: z.string().optional(), | ||
| targetOption: z | ||
| .enum(['auto-infer', 'existing-tables', 'target-schema-file']) | ||
| .optional() | ||
| .describe('Change where transformed data lands. Omit to keep the current setting.'), | ||
| targetTables: z | ||
| .array(z.string()) | ||
| .optional() | ||
| .describe('New list of existing workspace tables to load into. Required when setting targetOption to "existing-tables".'), | ||
| targetSchemaPath: z | ||
| .string() | ||
| .optional() | ||
| .describe('Local filesystem path to a replacement target schema file. Required when setting targetOption to "target-schema-file".'), | ||
| samplePath: z.string().optional().describe('Local filesystem path to a replacement sample data file'), | ||
| formatPath: z.string().optional().describe('Local filesystem path to a replacement format spec file'), | ||
| merge: z.boolean().optional().describe('Whether new data should merge/upsert into existing rows rather than append'), | ||
| computeSize: z.enum(['small', 'large']).optional(), | ||
| additionalPrompt: z.string().optional().describe('Extra natural-language guidance for the AI schema inference/mapping'), | ||
| loadSampleData: z | ||
| .boolean() | ||
| .optional() | ||
| .describe('Whether re-analysis should also trigger the data-load job (default true). Only used when runAnalysis is true.'), | ||
| runAnalysis: z | ||
| .boolean() | ||
| .optional() | ||
| .describe('Whether to call start-analysis and wait for it after saving the changes (default true).'), | ||
| }, | ||
| }, async ({ workspaceId, specName, description, targetOption, targetTables, targetSchemaPath, samplePath, formatPath, merge, computeSize, additionalPrompt, loadSampleData, runAnalysis, }) => { | ||
| if (targetOption === 'existing-tables' && !targetTables?.length) { | ||
| throw new Error('targetOption "existing-tables" requires targetTables (names of existing workspace tables).'); | ||
| } | ||
| if (targetOption === 'target-schema-file' && !targetSchemaPath) { | ||
| throw new Error('targetOption "target-schema-file" requires targetSchemaPath (local path to the schema file).'); | ||
| } | ||
| const cred = await resolveWorkspace(workspaceId); | ||
| const sampleFileName = samplePath ? path.basename(samplePath) : undefined; | ||
| const formatFileName = formatPath ? path.basename(formatPath) : undefined; | ||
| const targetSchemaFileName = targetSchemaPath ? path.basename(targetSchemaPath) : undefined; | ||
| const changedFiles = []; | ||
| if (sampleFileName) | ||
| changedFiles.push('sampleFile'); | ||
| if (formatFileName) | ||
| changedFiles.push('formatFile'); | ||
| if (targetSchemaFileName) | ||
| changedFiles.push('schemaFile'); | ||
| const updateResult = await dpf.dataSpecsAction(cred, 'update-spec', { | ||
| specName, | ||
| description, | ||
| targetOption, | ||
| targetTables: targetOption === 'existing-tables' ? targetTables : undefined, | ||
| targetSchemaFileName, | ||
| sampleFileName, | ||
| formatFileName, | ||
| merge, | ||
| computeSize, | ||
| additionalPrompt, | ||
| changedFiles, | ||
| }); | ||
| const { signedUrls, specId } = updateResult.data; | ||
| if (samplePath && sampleFileName) { | ||
| const sampleBuffer = await fs.readFile(samplePath); | ||
| await dpf.uploadToPresignedUrl(signedUrls[sampleFileName], sampleBuffer, contentTypeFor(samplePath)); | ||
| } | ||
| if (formatPath && formatFileName) { | ||
| const formatBuffer = await fs.readFile(formatPath); | ||
| await dpf.uploadToPresignedUrl(signedUrls[formatFileName], formatBuffer, contentTypeFor(formatPath)); | ||
| } | ||
| if (targetSchemaPath && targetSchemaFileName) { | ||
| const schemaBuffer = await fs.readFile(targetSchemaPath); | ||
| await dpf.uploadToPresignedUrl(signedUrls[targetSchemaFileName], schemaBuffer, contentTypeFor(targetSchemaPath)); | ||
| } | ||
| if (runAnalysis === false) { | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `Data spec "${specName}" updated (no analysis run). Call update_data_spec again or use the ` + | ||
| 'individual tools when ready, or re-invoke with runAnalysis: true (default) to process now.', | ||
| }, | ||
| ], | ||
| }; | ||
| } | ||
| await dpf.dataSpecsAction(cred, 'start-analysis', { specName, loadSampleData }); | ||
| const finalStatus = await pollSpecUntilDone(cred, specId); | ||
| const jobNote = finalStatus.lastJobId | ||
| ? ` The data load job ${finalStatus.lastJobId} was triggered — use get_status with that jobId to poll its progress separately.` | ||
| : ''; | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `Data spec "${specName}" updated and re-analyzed with status: ${finalStatus.status}.${jobNote}\n${JSON.stringify(finalStatus, null, 2)}`, | ||
| }, | ||
| ], | ||
| }; | ||
| }); | ||
| // Mirrors the sequence in test_data_specs.sh phases 5-7: | ||
| // create-job (with files[]) -> upload each file -> start-job -> poll to completion. | ||
| server.registerTool('run_data_job', { | ||
| title: 'Run a data processing job for an existing spec', | ||
| description: 'End-to-end workflow to process new data files through an already-configured data spec: creates a job, ' + | ||
| 'uploads each file, starts processing, and waits until the job completes or fails. Use this instead of ' + | ||
| 'calling the individual job tools yourself.', | ||
| inputSchema: { | ||
| workspaceId: workspaceIdSchema, | ||
| specName: z.string(), | ||
| filePaths: z.array(z.string()).min(1).describe('Local filesystem paths of the data files to process'), | ||
| }, | ||
| }, async ({ workspaceId, specName, filePaths }) => { | ||
| const cred = await resolveWorkspace(workspaceId); | ||
| const fileNames = filePaths.map((p) => path.basename(p)); | ||
| const createResult = await dpf.dataSpecsAction(cred, 'create-job', { specName, files: fileNames }); | ||
| const { signedUrls, jobId } = createResult.data; | ||
| for (const filePath of filePaths) { | ||
| const fileName = path.basename(filePath); | ||
| const buffer = await fs.readFile(filePath); | ||
| await dpf.uploadToPresignedUrl(signedUrls[fileName], buffer, contentTypeFor(filePath)); | ||
| } | ||
| await dpf.dataSpecsAction(cred, 'start-job', { specName, jobId }); | ||
| const finalStatus = await pollJobUntilDone(cred, jobId); | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `Job ${jobId} for spec "${specName}" finished with status: ${finalStatus.status}.\n${JSON.stringify(finalStatus, null, 2)}`, | ||
| }, | ||
| ], | ||
| }; | ||
| }); | ||
| // End-to-end workflow for "pull files from this server on a schedule" requests: reuse-or-create | ||
| // the connection, test it, then create the trigger. Mirrors onboard_data_source's role relative | ||
| // to the individual data-spec tools, but for manage_connection + manage_trigger. | ||
| server.registerTool('setup_scheduled_pull', { | ||
| title: 'Set up a scheduled SFTP pull into an existing data spec', | ||
| description: 'End-to-end workflow for "pull files from this SFTP server on a schedule" requests: reuses a matching ' + | ||
| 'connection (same hostname/username) if one already exists in the workspace, otherwise creates one; ' + | ||
| 'tests it; then creates a trigger that feeds an already-analyzed data spec (see onboard_data_source) on ' + | ||
| 'the given frequency. Use this instead of calling manage_connection + manage_trigger yourself for ' + | ||
| 'first-time setup. If the connection test fails (e.g. the public key is not yet installed on the ' + | ||
| 'remote server), no trigger is created — install the key and re-run this tool, which will reuse the ' + | ||
| 'same connection and pick up where it left off.', | ||
| inputSchema: { | ||
| workspaceId: workspaceIdSchema, | ||
| hostname: z.string().describe('SFTP server hostname to pull from'), | ||
| username: z.string().optional().describe('SFTP username. Defaults to "sftpuser".'), | ||
| specName: z.string().describe('Already-analyzed data spec to load files into (see onboard_data_source)'), | ||
| frequency: z.object({ | ||
| unit: z.enum(['hourly', 'daily', 'monthly']), | ||
| hourOfDay: z.number().int().min(0).max(23).optional().describe('Required for daily/monthly (UTC).'), | ||
| dayOfMonth: z.number().int().min(1).max(31).optional().describe('Required for monthly.'), | ||
| }), | ||
| preRules: z | ||
| .string() | ||
| .optional() | ||
| .describe('Natural language: which files to pick up (e.g. "only *.csv under /outbound")'), | ||
| postRules: z | ||
| .string() | ||
| .optional() | ||
| .describe('Natural language: what to do after a file loads (e.g. "rename with .done suffix")'), | ||
| dedupe: z | ||
| .boolean() | ||
| .optional() | ||
| .describe('Skip files already loaded into the spec by a previous run (default true)'), | ||
| }, | ||
| }, async ({ workspaceId, hostname, username, specName, frequency, preRules, postRules, dedupe }) => { | ||
| const cred = await resolveWorkspace(workspaceId); | ||
| const existing = await dpf.connectionsAction(cred, 'get-connections'); | ||
| const connections = (existing.data.connections ?? []); | ||
| let connection = connections.find((c) => c.hostname === hostname && (!username || c.username === username)); | ||
| let connectionNote; | ||
| if (connection) { | ||
| connectionNote = `Reusing existing connection ${connection.connectionId} for ${hostname}.`; | ||
| } | ||
| else { | ||
| const created = await dpf.connectionsAction(cred, 'create-connection', { type: 'sftp', hostname, username }); | ||
| connection = created.data; | ||
| connectionNote = | ||
| `Created connection ${connection.connectionId} for ${hostname}. Install this public key in the ` + | ||
| `server's authorized_keys before the trigger can pull data:\n${connection.publicKey}`; | ||
| } | ||
| const test = await dpf.connectionsAction(cred, 'test-connection', { connectionId: connection.connectionId }); | ||
| if (!test.data.success) { | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `${connectionNote}\n\nConnection test failed: ${test.data.message}\n` + | ||
| 'Ask the user to install the public key (if not already done) and confirm network access, then ' + | ||
| 're-run setup_scheduled_pull once ready — it will reuse this connection and proceed to create the trigger.', | ||
| }, | ||
| ], | ||
| }; | ||
| } | ||
| const trigger = await dpf.jobTriggersAction(cred, 'create-trigger', { | ||
| type: 'sftp', | ||
| specName, | ||
| connectionId: connection.connectionId, | ||
| frequency, | ||
| preRules, | ||
| postRules, | ||
| dedupe: dedupe ?? true, | ||
| }); | ||
| return { | ||
| content: [ | ||
| { | ||
| type: 'text', | ||
| text: `${connectionNote}\nConnection test succeeded (${test.data.message}).\n` + | ||
| `Trigger ${trigger.data.triggerId} created for spec "${specName}" on a ${frequency.unit} schedule. ` + | ||
| 'Use manage_trigger with action "run-now" to fire it immediately, "list" to check lastRunAt/lastRunStatus, ' + | ||
| 'or "run-history" for the detailed per-run log (status, filesPulled, message) after firing.\n' + | ||
| JSON.stringify(trigger.data, null, 2), | ||
| }, | ||
| ], | ||
| }; | ||
| }); | ||
| } |
| import { getDefaultWorkspaceCredential, getWorkspaceCredential } from '../credentials.js'; | ||
| /** Resolve which workspace credential to use: explicit workspaceId, or the only/first connected one. */ | ||
| export async function resolveWorkspace(workspaceId) { | ||
| if (workspaceId) { | ||
| const cred = await getWorkspaceCredential(workspaceId); | ||
| if (!cred) { | ||
| throw new Error(`No connected credentials for workspace ${workspaceId}. Use connect_workspace or login_and_list_workspaces first.`); | ||
| } | ||
| return cred; | ||
| } | ||
| const cred = await getDefaultWorkspaceCredential(); | ||
| if (!cred) { | ||
| throw new Error('No workspace connected yet. Use connect_workspace or login_and_list_workspaces first.'); | ||
| } | ||
| return cred; | ||
| } |
| <!-- | ||
| Serve this file over http(s) — do not open it directly from disk (a file:// URL). | ||
| Most browsers won't persist cookies for file:// pages, which breaks the login session. | ||
| From this directory: `npx serve` (or `python3 -m http.server`), then open the printed URL. | ||
| --> | ||
| <!DOCTYPE html> | ||
| <html lang="en"> | ||
| <head> | ||
| <meta charset="UTF-8" /> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> | ||
| <title>DPF — auth + query example</title> | ||
| <style> | ||
| /* Deliberately plain: this file demonstrates the auth/cookie/query mechanics, | ||
| not a design system. Swap in your own styling/charting when you adapt it. */ | ||
| body { font-family: system-ui, sans-serif; max-width: 900px; margin: 40px auto; padding: 0 16px; } | ||
| fieldset { border: 1px solid #ccc; border-radius: 6px; padding: 16px; margin-bottom: 16px; } | ||
| label { display: block; font-size: 13px; margin-bottom: 4px; } | ||
| input, textarea, select { font: inherit; width: 100%; box-sizing: border-box; padding: 6px 8px; margin-bottom: 10px; } | ||
| textarea { font-family: ui-monospace, monospace; font-size: 13px; min-height: 100px; } | ||
| button { font: inherit; padding: 6px 14px; cursor: pointer; } | ||
| .row { display: flex; gap: 10px; align-items: center; } | ||
| .hidden { display: none !important; } | ||
| .status { padding: 8px 10px; border-radius: 4px; font-size: 13px; margin-bottom: 12px; } | ||
| .status.err { background: #fdecea; color: #611a15; } | ||
| .status.ok { background: #eaf6ec; color: #1e4620; } | ||
| table { border-collapse: collapse; width: 100%; font-size: 13px; } | ||
| th, td { border: 1px solid #ddd; padding: 6px 8px; text-align: left; } | ||
| th { background: #f4f4f4; } | ||
| </style> | ||
| </head> | ||
| <body> | ||
| <h1>DPF query example</h1> | ||
| <p><b>Must be served over http(s)</b>, not opened as a <code>file://</code> URL — most browsers won't | ||
| keep the login cookie for local files. Run <code>npx serve</code> (or <code>python3 -m http.server</code>) | ||
| from this directory and open the printed URL.</p> | ||
| <div id="status" class="status hidden"></div> | ||
| <!-- Logged-out view --> | ||
| <fieldset id="loginBox"> | ||
| <legend>Log in</legend> | ||
| <label for="email">Email</label> | ||
| <input id="email" type="email" autocomplete="username" /> | ||
| <label for="password">Password</label> | ||
| <input id="password" type="password" autocomplete="current-password" /> | ||
| <button id="loginBtn">Log in</button> | ||
| </fieldset> | ||
| <!-- Logged-in view --> | ||
| <div id="appBox" class="hidden"> | ||
| <div class="row" style="justify-content: space-between; margin-bottom: 16px;"> | ||
| <span>Signed in as <b id="whoami"></b></span> | ||
| <button id="logoutBtn">Log out</button> | ||
| </div> | ||
| <fieldset> | ||
| <legend>Query</legend> | ||
| <label for="sql">SQL</label> | ||
| <textarea id="sql" spellcheck="false">SHOW TABLES</textarea> | ||
| <button id="runBtn">Run query</button> | ||
| </fieldset> | ||
| <div id="resultsWrap"></div> | ||
| </div> | ||
| <script> | ||
| "use strict"; | ||
| // ---- configuration ---- | ||
| const API_BASE = "https://api.dpf-it.com"; | ||
| const COOKIE_NAME = "dpf_jwt"; | ||
| // RULE: a DPF namespace is always the last 12 characters of the workspaceId — it is a fixed | ||
| // derivation, never a name you pick. Find yours via: | ||
| // curl -H "Authorization: Bearer <token>" https://api.dpf-it.com/iceberg/v1/namespaces | ||
| const NAMESPACE = "xxxxxxxxxxxx"; // <-- replace with your workspace's namespace (last 12 chars of its workspaceId) | ||
| // ============================================================ | ||
| // Cookie-backed token storage | ||
| // | ||
| // A cookie set from client-side JS can't be marked httpOnly, so it's | ||
| // readable by any script on the page — same exposure as localStorage. | ||
| // This is the standard tradeoff for a pure browser SPA with no backend | ||
| // of its own to set a real httpOnly session cookie; treat this file as | ||
| // a demo of the API/token mechanics, not a hardened auth pattern. | ||
| // ============================================================ | ||
| function setTokenCookie(token, maxAgeSeconds) { | ||
| const secure = location.protocol === "https:" ? "; Secure" : ""; | ||
| document.cookie = `${COOKIE_NAME}=${encodeURIComponent(token)}; path=/; max-age=${Math.max(0, maxAgeSeconds)}; SameSite=Lax${secure}`; | ||
| } | ||
| function getTokenCookie() { | ||
| const match = document.cookie.match(new RegExp(`(?:^|; )${COOKIE_NAME}=([^;]*)`)); | ||
| return match ? decodeURIComponent(match[1]) : null; | ||
| } | ||
| function clearTokenCookie() { | ||
| document.cookie = `${COOKIE_NAME}=; path=/; max-age=0`; | ||
| } | ||
| // Decode a JWT's payload without verifying the signature — fine here since we | ||
| // only read it to display who's logged in and to pre-empt an obviously-expired | ||
| // token; the API itself is the source of truth and re-validates on every call. | ||
| function decodeJwtPayload(token) { | ||
| try { | ||
| const payload = token.split(".")[1]; | ||
| const base64 = payload.replace(/-/g, "+").replace(/_/g, "/"); | ||
| return JSON.parse(atob(base64)); | ||
| } catch { | ||
| return null; | ||
| } | ||
| } | ||
| // Small buffer so we don't hand the API a token that expires mid-request. | ||
| function isTokenValid(token) { | ||
| if (!token) return false; | ||
| const claims = decodeJwtPayload(token); | ||
| return !!claims && !!claims.exp && claims.exp > Date.now() / 1000 + 5; | ||
| } | ||
| // ---- API calls ---- | ||
| async function apiFetch(path, opts = {}) { | ||
| const token = getTokenCookie(); | ||
| const headers = Object.assign({ "Content-Type": "application/json" }, opts.headers || {}); | ||
| if (token) headers["Authorization"] = "Bearer " + token; | ||
| const res = await fetch(API_BASE + path, Object.assign({}, opts, { headers })); | ||
| let body; | ||
| try { body = await res.json(); } catch { body = {}; } | ||
| if (res.status === 401) { | ||
| // Covers server-side revocation/expiry the client-side exp check couldn't see. | ||
| clearTokenCookie(); | ||
| showLoginView("Session expired — please log in again."); | ||
| throw new Error("Session expired"); | ||
| } | ||
| if (!res.ok) { | ||
| const message = (body && body.error && body.error.message) || res.statusText || `HTTP ${res.status}`; | ||
| throw new Error(message); | ||
| } | ||
| return body; | ||
| } | ||
| // ---- UI helpers ---- | ||
| const el = (id) => document.getElementById(id); | ||
| function setStatus(message, kind) { | ||
| const box = el("status"); | ||
| if (!message) { box.className = "status hidden"; return; } | ||
| box.className = "status " + (kind || ""); | ||
| box.textContent = message; | ||
| } | ||
| function showLoginView(message) { | ||
| el("loginBox").classList.remove("hidden"); | ||
| el("appBox").classList.add("hidden"); | ||
| if (message) setStatus(message, "err"); | ||
| } | ||
| function showAppView(claims) { | ||
| el("loginBox").classList.add("hidden"); | ||
| el("appBox").classList.remove("hidden"); | ||
| el("whoami").textContent = (claims && (claims.email || claims.userId)) || "user"; | ||
| setStatus(""); | ||
| } | ||
| function escapeHtml(value) { | ||
| return String(value).replace(/[&<>"']/g, (c) => ({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c])); | ||
| } | ||
| // ---- auth actions ---- | ||
| async function login() { | ||
| const email = el("email").value.trim(); | ||
| const password = el("password").value; | ||
| if (!email || !password) { setStatus("Enter email and password.", "err"); return; } | ||
| el("loginBtn").disabled = true; | ||
| setStatus("Signing in…"); | ||
| try { | ||
| const body = await apiFetch("/auth/login", { | ||
| method: "POST", | ||
| body: JSON.stringify({ email, password }), | ||
| }); | ||
| const token = body.data && body.data.token; | ||
| if (!token) throw new Error("Login succeeded but no token was returned."); | ||
| const claims = decodeJwtPayload(token); | ||
| const maxAge = claims && claims.exp ? claims.exp - Date.now() / 1000 : 3600; | ||
| setTokenCookie(token, maxAge); | ||
| // Most browsers silently refuse to persist cookies for pages opened directly from disk | ||
| // (a file:// URL) — the login call above still succeeds since it doesn't need the cookie, | ||
| // but every later request would silently go out with no Authorization header and 401. | ||
| // Catch that here instead of leaving it to surface as a confusing failure on the first query. | ||
| if (getTokenCookie() !== token) { | ||
| throw new Error( | ||
| "Logged in, but the browser wouldn't store the session cookie — this page is likely open as a " + | ||
| "file:// URL. Serve it over http instead, e.g. `npx serve` (or `python3 -m http.server`) from " + | ||
| "this examples/ directory, then reload." | ||
| ); | ||
| } | ||
| el("password").value = ""; | ||
| showAppView(claims); | ||
| } catch (e) { | ||
| setStatus("Login failed: " + e.message, "err"); | ||
| } finally { | ||
| el("loginBtn").disabled = false; | ||
| } | ||
| } | ||
| function logout() { | ||
| clearTokenCookie(); | ||
| el("resultsWrap").innerHTML = ""; | ||
| showLoginView(); | ||
| } | ||
| // ---- data actions ---- | ||
| async function runQuery() { | ||
| const sql = el("sql").value.trim(); | ||
| if (!sql) { setStatus("Enter a SQL query.", "err"); return; } | ||
| el("runBtn").disabled = true; | ||
| setStatus("Running query…"); | ||
| try { | ||
| const body = await apiFetch("/query", { | ||
| method: "POST", | ||
| body: JSON.stringify({ namespace: NAMESPACE, query: sql }), | ||
| }); | ||
| renderResults(body.data || {}); | ||
| setStatus(`${body.data.rowCount ?? 0} rows in ${body.data.executionTimeMs ?? "?"} ms`, "ok"); | ||
| } catch (e) { | ||
| el("resultsWrap").innerHTML = ""; | ||
| setStatus("Query failed: " + e.message, "err"); | ||
| } finally { | ||
| el("runBtn").disabled = false; | ||
| } | ||
| } | ||
| function renderResults(data) { | ||
| const columns = (data.columnTypes || []).map((c) => c.name); | ||
| const rows = data.rows || []; | ||
| if (columns.length === 0) { el("resultsWrap").innerHTML = "<p>No columns returned.</p>"; return; } | ||
| const head = columns.map((c) => `<th>${escapeHtml(c)}</th>`).join(""); | ||
| const body = rows | ||
| .map((row) => "<tr>" + columns.map((c) => `<td>${escapeHtml(row[c] ?? "")}</td>`).join("") + "</tr>") | ||
| .join(""); | ||
| el("resultsWrap").innerHTML = `<table><thead><tr>${head}</tr></thead><tbody>${body}</tbody></table>`; | ||
| } | ||
| // ---- wire up ---- | ||
| el("loginBtn").addEventListener("click", login); | ||
| el("password").addEventListener("keydown", (e) => { if (e.key === "Enter") login(); }); | ||
| el("logoutBtn").addEventListener("click", logout); | ||
| el("runBtn").addEventListener("click", runQuery); | ||
| // ---- init: restore session from cookie, or show the login screen ---- | ||
| (function init() { | ||
| const token = getTokenCookie(); | ||
| if (isTokenValid(token)) { | ||
| showAppView(decodeJwtPayload(token)); | ||
| } else { | ||
| if (token) clearTokenCookie(); // present but expired/unparseable | ||
| showLoginView(); | ||
| } | ||
| })(); | ||
| </script> | ||
| </body> | ||
| </html> |
Major refactor
Supply chain riskPackage has recently undergone a major refactor. It may be unstable or indicate significant internal changes. Use caution when updating to versions that include significant changes.
URL strings
Supply chain riskPackage contains fragments of external URLs or IP addresses, which the package may be accessing at runtime.
Network access
Supply chain riskThis module accesses the network.
Environment variable access
Supply chain riskPackage accesses environment variables, which may be a sign of credential stuffing or data theft.
Found 2 instances
Filesystem access
Supply chain riskAccesses the file system, and could potentially read sensitive data.
No v1
QualityPackage is not semver >=1. This means it is not stable and does not support ^ ranges.
URL strings
Supply chain riskPackage contains fragments of external URLs or IP addresses, which the package may be accessing at runtime.
0
-100%0
-100%1
-50%2
-75%1
-75%3913
-95.72%4
-69.23%14
-98.95%33
-67.96%- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed
- Removed