@icedq/mcp-server
Advanced tools
| ⛔ MANDATORY TOOL SEQUENCE — READ THIS FIRST, NO EXCEPTIONS: | ||
| fetch_api_sample_data (source) → [fetch_api_sample_data (target, API only)] → analyze_recon_mapping → update_rule | ||
| NEVER call create_api_recon_rule. It is an anti-pattern for this workflow. | ||
| --- | ||
| SAP ECC AS TARGET: If target connection is connectorId="sap-ecc", use customSql (table name only, no schema prefix) instead of schema+table navigation. Ask user which columns to map (max 8-12, 512-byte row limit). Example: `customSql: "SELECT MATNR, MTART FROM MARA"` | ||
| HUMAN-IN-THE-LOOP (RECOMMENDED DEFAULT): | ||
| - API Recon rules compare row-level data between a REST API source and any target (DB, File, or another API). This workflow must be approval-gated. | ||
| - ⛔ CRITICAL PRINCIPLE: NEVER auto-pick connections, databases, schemas, tables, join keys, or mapped columns when multiple options exist. | ||
| - ⛔ ALWAYS present complete lists and wait for explicit user selection at EACH step, even if there's only one option. | ||
| - ⛔ DO NOT assume user intent from names — always ask. | ||
| - This applies to BOTH source and target sides of the reconciliation. | ||
| RULE TYPE: api-recon | ||
| - Source side: ALWAYS an apidb connection (REST API endpoint) | ||
| - Target side: Any connection type — Database (rdbms/clouddb), File, or another API (apidb) | ||
| NOTE ON RULE TYPE PARAMETER: | ||
| - For API recon rules, ONLY "api-recon" is supported (lowercase) | ||
| - The system will validate and reject any other format | ||
| - Checksum rules are NOT supported for API connections | ||
| --- | ||
| APPROVAL GATES (do these in order and WAIT after each): | ||
| PHASE 1 — SOURCE SIDE (same as API validation rule): | ||
| 1) Workspace selection | ||
| - list_workspaces → present options → user selects workspaceId | ||
| 2) Source connection selection (API only) | ||
| - list_connections(workspaceId, connectorType="apidb") → show ACTIVE apidb connections only → user selects sourceConnectionId | ||
| 3) Folder selection | ||
| - list_folders(workspaceId) → user selects folderId | ||
| 4) Source API configuration | ||
| - endPoint: User provides API path (e.g., "/posts", "/api/v1/users") or null for root | ||
| - requestMethod: GET (default) | POST | ||
| - dataModel: "Document" (flat) | "FlattenedDocuments" (nested, recommended for nested JSON) | ||
| - tableName: Descriptive name for this API dataset (e.g., "post_list", "user_profiles") | ||
| - jsonPath: ONLY if dataModel="FlattenedDocuments" AND data is nested (e.g., "$.data.items") — leave empty for root level | ||
| 5) Source data registration | ||
| - fetch_api_sample_data(workspaceId, connectionId=sourceConnId, ruleId=null, folderId, | ||
| ruleType="api-recon", connectionType="source", | ||
| apiConfig={ endPoint, requestMethod, tableName, dataModel, jsonPath, ... }) | ||
| → returns: ruleId (draft api-recon rule), source columns, sample data rows | ||
| → STOP: show preview of source columns and first 5 rows → user confirms data looks correct | ||
| → KEEP this ruleId — it is used in ALL subsequent steps | ||
| PHASE 2 — TARGET SIDE (same as recon rule; if API target → follow source process): | ||
| 6) Target connection selection (any type) | ||
| - list_connections(workspaceId) → show ALL ACTIVE connections → user selects targetConnectionId | ||
| - Detect target connection type from response: | ||
| • rdbms / clouddb → DATABASE target | ||
| • apidb → API target | ||
| • flat-file / parquet / excel / json / xml / flat-file-sql → FILE target | ||
| 7) Target dataset selection and wiring: | ||
| - If target is DATABASE: | ||
| ⛔ CRITICAL: STOP at each level and present full lists. NEVER auto-select databases, schemas, or tables. | ||
| • get_database_metadata(connectionId) → check supportedHierarchy | ||
| • If includes "database": | ||
| - list_connection_metadata(entity="database") → STOP → present ALL databases → user selects databaseName | ||
| • list_connection_metadata(entity="schema", databaseName) → STOP → present ALL schemas → user selects schemaName | ||
| • list_connection_metadata(entity="table", databaseName, schemaName) → STOP → present ALL tables → user selects tableName | ||
| • list_connection_metadata(entity="column", databaseName, schemaName, tableName) → get target column list | ||
| ⛔ DO NOT proceed to next level until user explicitly selects from the current level | ||
| - If target is API: | ||
| • Gather target API config (endPoint, requestMethod, dataModel, tableName, jsonPath) from user | ||
| • fetch_api_sample_data(workspaceId, connectionId=targetConnId, ruleId=<from step 5>, | ||
| folderId=null, ruleType="api-recon", connectionType="target", | ||
| apiConfig={ endPoint, requestMethod, tableName, dataModel, jsonPath, ... }) | ||
| → returns: target columns, patches target dataset on the existing draft rule | ||
| - If target is FILE: | ||
| • list_files(workspaceId, targetConnectionId) → user picks fileName | ||
| • fetch_file_sample_data(workspaceId, connectionId=<file>, ruleId=<from step 5>, | ||
| folderId=null, ruleType="Recon", connectionType="target", fileName) | ||
| → patches target dataset on the existing draft rule | ||
| PHASE 3 — REMAINING (same as recon rule): | ||
| 8) Mapping analysis + approval (required) | ||
| - analyze_recon_mapping(sourceColumns, targetColumns) → present HIGH/MEDIUM/LOW matches | ||
| - User confirms join key columns and which column checks to include | ||
| - Ask user about custom Groovy checks if needed | ||
| 9) Result types + rule name approval | ||
| - Confirm which resultTypes: a-b (source orphans), b-a (target orphans), Xp (column diffs). Default: all three. | ||
| - Ask the user for ruleName. Convention: {APITableName}_vs_{TargetTable}_Recon | ||
| - Do NOT auto-generate | ||
| 10) Publish rule | ||
| - update_rule(workspaceId, ruleId=<from step 5>, ruleName, | ||
| targetConfig={ connectionId: targetConnId, databaseName, schemaName, tableName }, ← DB target only; omit for API or File (already wired in step 7) | ||
| joinKeys=[...confirmed...], checksToAdd=[...confirmed...]) | ||
| → wires DB target (if applicable), adds join keys + checks, and publishes | ||
| 11) Optional execution (separate approval) | ||
| - Only run execute_rules_or_workflows if user explicitly says to execute now | ||
| --- | ||
| DATA MODEL SELECTION (for API sides): | ||
| Document: | ||
| - Use for flat, simple JSON responses | ||
| - Nested objects/arrays remain as JSON text columns | ||
| - Example: { "id": 1, "name": "Product A", "price": 99.99 } | ||
| FlattenedDocuments: | ||
| - Use for nested JSON structures | ||
| - Automatically flattens nested objects into dot-notation columns (e.g., category.id, category.name) | ||
| - Arrays expand into separate rows (one row per array item) | ||
| - Requires jsonPath if data is nested under keys (e.g., $.data.items) | ||
| JSONPath examples: | ||
| - $.data → data nested under 'data' key | ||
| - $.results → results array at root | ||
| - $.data.items → items nested two levels deep | ||
| - (empty/null) → use root level data | ||
| --- | ||
| CHECK TYPES FOR API RECON: | ||
| - SimpleCompare (default): {sourceColumn: "id", targetColumn: "CustomerId"} — equality comparison | ||
| - Custom (Groovy): {name: "Chk_Status", expression: "(S.[status] == \"active\") == (T.[isActive] == 1)"} | ||
| CUSTOM CHECK PATTERN (TRUE = PASS): | ||
| - Recon custom checks use TRUE = PASS (same as validation) | ||
| - Write expressions that return TRUE when data is CORRECT | ||
| - Examples: | ||
| S.[id] == T.[CustomerId] → equality | ||
| S.[userId].toString() == T.[LegacyId].toString() → cross-type comparison | ||
| (S.[status] == "active") == (T.[isActive] == true) → value mapping | ||
| RESULT TYPES: | ||
| - a-b: Orphaned source rows (in API but not in target) | ||
| - b-a: Orphaned target rows (in target but not in API) | ||
| - Xp: Column mismatches (rows that matched on join key but have value differences) | ||
| NAMING CONVENTION: {APITableName}_vs_{TargetTable}_Recon | ||
| --- | ||
| ERROR RECOVERY: | ||
| - "No columns returned" from API → verify endPoint, try different jsonPath or dataModel | ||
| - "Source API dataset columns are empty" → re-run fetch_api_sample_data for source before update_rule | ||
| - "fileSchemaId missing" → re-call fetch_api_sample_data with the existing ruleId for that side | ||
| - "Join keys required" → ensure joinKeys array is populated from analyze_recon_mapping results | ||
| - "Dataset has no connectionId" → re-run fetch_api_sample_data with ruleId and correct connectionType | ||
| ANTI-PATTERNS: | ||
| - ⛔ Do NOT call create_api_recon_rule — EVER. Always use fetch_api_sample_data + update_rule | ||
| - ⛔ Do NOT call create_recon_rule for API source connections | ||
| - Do NOT pass targetConfig to update_rule when target is already API-wired (step 7 handles it) | ||
| - Do NOT skip analyze_recon_mapping — always analyze before wiring checks | ||
| - Do NOT auto-execute without approval |
| HUMAN-IN-THE-LOOP (RECOMMENDED DEFAULT): | ||
| - This workflow is intended to be approval-gated. If the user does NOT provide explicit IDs/names for each choice, you MUST stop and ask. | ||
| - Do NOT auto-pick defaults when multiple options exist. | ||
| API CONNECTION DETECTION: | ||
| - API connection type: apidb (REST API endpoints) | ||
| - API connections require special handling similar to file connections | ||
| - NEVER call create_validation_rule directly for API connections — use API CONNECTION FLOW below | ||
| APPROVAL GATES (do these in order and WAIT after each): | ||
| 1) Workspace selection | ||
| - list_workspaces → present options → user selects workspaceId | ||
| 2) Connection selection | ||
| - list_connections(workspaceId, connectorType="apidb") → user selects connectionId | ||
| - IMPORTANT: Always filter for connectorType="apidb" to show only API connections | ||
| 3) Folder selection | ||
| - list_folders(workspaceId, optional nameFilter) → user selects folderId | ||
| - If folder does not exist: call get_guidance('rule_organization'), propose folderName + parent folder, then ONLY create_folder after user approves | ||
| 4) API configuration | ||
| - endPoint: User provides API path (e.g., "/products", "/api/v1/users") or null for root | ||
| - requestMethod: GET (default) | POST (ONLY GET and POST are supported for API validation rules) | ||
| - dataModel: "Document" (flat) | "FlattenedDocuments" (nested, recommended) | ||
| - tableName: User provides descriptive name (e.g., "products_catalog", "user_profiles") | ||
| - jsonPath: ONLY if dataModel="FlattenedDocuments" AND data is nested (e.g., "$.data.items", "$.results") — leave empty for root level | ||
| - AUTO-INFER: for FlattenedDocuments + single-segment endpoints (e.g. "/products"), MCP auto-uses "$.products" | ||
| - baseUrl: auto-resolved from connection metadata when possible; pass explicitly if connection has no stored URL | ||
| 5) Sample data verification | ||
| - Show preview: columns and first 5 rows | ||
| - User confirms: "Does this look correct?" | ||
| 6) Checks approval | ||
| - Present suggested checks and WAIT for user approval (include/exclude/modify) | ||
| 7) Rule name approval | ||
| - Ask user for ruleName. Do NOT auto-generate | ||
| 8) Publish rule | ||
| - Apply approved checks and publish | ||
| 9) Optional execution (separate approval) | ||
| - Only run execute_rules_or_workflows if user explicitly says to execute now | ||
| --- | ||
| API CONNECTION FLOW: | ||
| NOTE ON RULE TYPE PARAMETER: | ||
| - For API validation rules, ONLY "api-validation" is supported (lowercase) | ||
| - The system will validate and reject any other format | ||
| Phase 1 — Register API schema and create draft rule: | ||
| Step 1 Resolve workspaceId, connectionId (apidb type), folderId | ||
| Step 2 Gather API configuration from user (approval gate #4): | ||
| - endPoint (required, can be null for root) | ||
| - requestMethod (default: GET) | ||
| - dataModel: Document | FlattenedDocuments | ||
| - tableName (required) | ||
| - jsonPath (conditional: only if FlattenedDocuments + nested data) | ||
| Step 3 Internal: fetchApiFileSampleData(workspaceId, connectionId, ruleId=null, folderId, | ||
| ruleType="api-validation", connectionType="source", | ||
| apiConfig={ baseUrl, endPoint, requestMethod, tableName, dataModel, jsonPath, ... }) | ||
| → returns: ruleId (draft), columns, sample data rows, fileSchemaId | ||
| → STOP: a draft Validation rule now exists linked to the API schema | ||
| Phase 2 — Profile, approve checks, and publish: | ||
| Step 4 Show sample data preview → user verifies columns and data structure | ||
| Step 5 profile_data(sampleData) → suggest_quality_checks(profileData) | ||
| → present suggested checks to user → wait for approval | ||
| Step 6 Ask user for ruleName | ||
| Step 7 update_rule(workspaceId, ruleId, ruleName, checksToAdd=[...approved checks...]) | ||
| → publishes the rule | ||
| --- | ||
| DATA MODEL SELECTION: | ||
| Document: | ||
| - Use for flat, simple JSON responses | ||
| - Nested objects/arrays remain as JSON text columns | ||
| - Example: { "id": 1, "name": "Product A", "price": 99.99 } | ||
| FlattenedDocuments: | ||
| - Use for nested JSON structures | ||
| - Automatically flattens nested objects into dot-notation columns (e.g., category.id, category.name) | ||
| - Arrays expand into separate rows (one row per array item) | ||
| - Requires jsonPath if data is nested under keys (e.g., $.data.items) | ||
| JSONPath examples: | ||
| - $.data → data nested under 'data' key | ||
| - $.results → results array at root | ||
| - $.data.items → items nested two levels deep | ||
| - (empty/null) → use root level data | ||
| --- | ||
| GROOVY EXPRESSION PATTERN (Custom checks): | ||
| - All custom checks use TRUE = PASS, FALSE = FAIL | ||
| - Write expressions that describe VALID data conditions | ||
| - Examples: | ||
| S.[id] != null > ID required | ||
| S.[price] > 0 > price must be positive | ||
| S.[email] != null && S.[email].contains("@") > email must contain @ | ||
| S.[status] in ["Active","Pending","Completed"] > status must be valid | ||
| S.[category.id] != null > nested field required (FlattenedDocuments) | ||
| S.[updatedAt] >= S.[createdAt] > dates in order | ||
| SUPPORTED CHECK TYPES: | ||
| - NotNull: {checkType: "NotNull", column: "col"} | ||
| - ValidValues: {checkType: "ValidValues", column: "col", expectedValues: ["A","B"]} | ||
| - Format: {checkType: "Format", column: "col", pattern: "Email|Phone|SSN|ZipCode|URL|IP"} | ||
| - Length: {checkType: "Length", column: "col", expectedLength: 10, operator: "equal to"} | ||
| - Date: {checkType: "Date", column: "col", dateFormat: "yyyy-MM-dd"} | ||
| - Custom: {checkType: "Custom", column: "col", expression: "S.[col] > 0"} | ||
| FAST PATH (when user provides all values in one message): | ||
| - If workspace, connection, folder, endpoint, method, tableName, ruleName, and check approval are all given upfront: | ||
| 1) fetch_api_sample_data (single call — registers schema, creates draft, returns sample) | ||
| 2) update_rule with checksToAdd + ruleName (publish) | ||
| - Skip redundant approval gates when user explicitly says "use defaults" or "approve all checks" | ||
| ERROR RECOVERY: | ||
| - "No columns returned" or "Empty data" → verify endpoint path, try different jsonPath ($.data, $.results, or empty), or switch dataModel | ||
| - "baseUrl is required" → pass apiConfig.baseUrl explicitly or fix connection URL in iceDQ UI | ||
| - HTTP 500 on new rule creation → ensure MCP server is up to date (rule payload must include RecordCheck with ResultType) | ||
| - "Schema not found" or "fileSchemaId missing" → re-call fetchApiFileSampleData with same parameters | ||
| - "Rule already exists" → call get_rule to find existing, then use update_rule with existing ruleId | ||
| - "Invalid dataModel" → must be "Document" or "FlattenedDocuments" (case-sensitive) | ||
| - Columns look wrong → adjust jsonPath or switch dataModel | ||
| ANTI-PATTERNS: | ||
| - Do NOT call create_validation_rule for API connections — use fetchApiFileSampleData + update_rule | ||
| - Do NOT skip API configuration approval — always confirm endpoint, dataModel, tableName with user | ||
| - Do NOT create separate rules per check — combine into ONE rule per endpoint | ||
| - Do NOT auto-execute without approval — always ask first | ||
| NOTES: | ||
| - API connections handle pagination via connection configuration (not in rule) | ||
| - Nested field columns use dot notation: S.[category.name], S.[address.zipCode] | ||
| - Use brackets in expressions for all column names: S.[columnName] | ||
| - Consider rate limits when scheduling API rule execution |
@@ -0,1 +1,7 @@ | ||
| SAP ECC CHECKSUM RULES: | ||
| - If source or target connectorId="sap-ecc" → use custom SQL for that side (table name only, no schema prefix) | ||
| - Row count: `SELECT COUNT(*) AS SOURCE_COUNT FROM MARA`. For SUM/AVG, ask user which column to aggregate first. | ||
| COLUMN SELECTION: For flat-file connections, ask user which column to aggregate before creating rule | ||
| HUMAN-IN-THE-LOOP (RECOMMENDED DEFAULT): | ||
@@ -196,5 +202,7 @@ - Checksum rules compare TWO sides (source vs target). This workflow must be approval-gated. | ||
| REQUIREMENTS: | ||
| - Each SQL must return exactly 1 row, 1 numeric column | ||
| - Each SQL MUST return exactly 1 row, 1 numeric column (VALIDATION ENFORCED) | ||
| - Multiple columns in SQL will cause a validation error at rule creation | ||
| - Column must have an alias (SOURCE_COUNT, TARGET_COUNT, or custom) | ||
| - Source and target aliases must be different | ||
| - For multiple metrics (e.g., COUNT + SUM), create separate checksum rules | ||
@@ -201,0 +209,0 @@ NAMING CONVENTION: {SourceTable}_vs_{TargetTable}_Checksum |
@@ -0,1 +1,6 @@ | ||
| SAP ECC DUPLICATE RULES: | ||
| - SAP connection: connectorId="sap-ecc" → ⚠️ ALWAYS use customSql (table name only, no schema prefix) | ||
| - Flow: list_connection_metadata(entity="table") → pick table → list columns → ask user "Which columns to check for duplicates?" (max 8-12 cols, 512-byte row limit) → fetch_db_sample_data(customSql=...) → create_duplicate_rule(customSql=..., duplicateColumns=[...]) | ||
| - Example: `customSql: "SELECT MATNR, WERKS FROM MARA"`, `duplicateColumns: ["MATNR","WERKS"]` | ||
| HUMAN-IN-THE-LOOP (RECOMMENDED DEFAULT): | ||
@@ -22,3 +27,3 @@ - This workflow is approval-gated. If the user does NOT provide explicit IDs/names for each choice, you MUST stop and ask. | ||
| 8) Optional execution (separate approval) | ||
| - Only run execute_rule if user explicitly says to execute now. | ||
| - Only run execute_rules_or_workflows if user explicitly says to execute now. | ||
@@ -33,3 +38,3 @@ WORKFLOW: | ||
| 4. Create: create_duplicate_rule | ||
| 5. Execute: execute_rule > check results | ||
| 5. Execute: execute_rules_or_workflows > check results | ||
| 6. Review exceptions: get_checks_exception_report shows duplicate records with DUPLICATE_COUNT | ||
@@ -36,0 +41,0 @@ |
@@ -0,1 +1,6 @@ | ||
| SAP ECC PUSHDOWN RULES: | ||
| - SAP connection: connectorId="sap-ecc" → SQL must use table name only (no schema.table prefix) | ||
| - Flow: list_connection_metadata(entity="table"/"column") → show columns → ask user "Which columns to include and what defines a failing row?" (max 8-12 cols, 512-byte row limit) | ||
| - Example: `SELECT MATNR FROM MARA WHERE MATNR IS NULL` (rows returned = failures) | ||
| HUMAN-IN-THE-LOOP (RECOMMENDED DEFAULT): | ||
@@ -2,0 +7,0 @@ - Pushdown rules are powerful and can be costly; treat them as approval-gated. |
@@ -0,1 +1,6 @@ | ||
| SAP ECC RECON RULES: | ||
| - If source or target connectorId="sap-ecc" → use customSql for that side (table name only, no schema prefix) | ||
| - Flow: list_connection_metadata(entity="column") for the SAP side → ask user "Which columns for join key and comparison?" (max 8-12 cols, 512-byte row limit) → analyze_recon_mapping → create_recon_rule | ||
| - Example: `customSql: "SELECT MATNR, MTART, MEINS FROM MARA"` | ||
| HUMAN-IN-THE-LOOP (RECOMMENDED DEFAULT): | ||
@@ -33,3 +38,3 @@ - Recon rules compare row-level data across TWO sides (source vs target). This workflow must be approval-gated. | ||
| 9) Optional execution (separate approval) | ||
| - Only run execute_rule if user explicitly says to execute now. | ||
| - Only run execute_rules_or_workflows if user explicitly says to execute now. | ||
@@ -50,3 +55,3 @@ --- | ||
| - checkSourceColumns: "first_name,salary", checkTargetColumns: "name,salary" | ||
| 6. Execute: execute_rule > get_workflow_run_status_or_result (action=status) > get_workflow_run_status_or_result (action=result) | ||
| 6. Execute: execute_rules_or_workflows > get_workflow_run_status_or_result (action=status) > get_workflow_run_status_or_result (action=result) | ||
| 7. Review exceptions: get_checks_exception_report — look at difftype column: | ||
@@ -53,0 +58,0 @@ - ANB = source orphan (exists in source, not in target) |
@@ -10,2 +10,15 @@ HUMAN-IN-THE-LOOP (RECOMMENDED DEFAULT): | ||
| SAP ECC VALIDATION RULES: | ||
| - SAP connection: connectorId="sap-ecc" | ||
| - ⚠️ **ALWAYS use customSql** for create_validation_rule (schemaName+tableName mode not supported) | ||
| - **Rule creation pattern:** | ||
| 1. list_connection_metadata(entity="table") → pick table | ||
| 2. list_connection_metadata(entity="column") → show columns to user | ||
| 3. Ask user: "Which columns to validate?" → user picks 8-12 columns max (512-byte limit) | ||
| 4. fetch_db_sample_data(customSql="SELECT col1,col2,... FROM TableName", limit=100) | ||
| 5. profile_data → suggest_quality_checks → present suggestions → WAIT for approval | ||
| 6. create_validation_rule(customSql="SELECT col1,col2,... FROM TableName", checksJson=...) | ||
| - Example: `customSql: "SELECT MANDT, MATNR, ERSDA, ERNAM FROM MARA"` | ||
| - **Crawling:** Required before first use. If metadata fails: "Run crawling in iceDQ UI first" | ||
| APPROVAL GATES (do these in order and WAIT after each): | ||
@@ -36,11 +49,12 @@ 1) Workspace selection | ||
| STANDARD WORKFLOW (DB connections): | ||
| 1. Identify target: Get connectionId (list_connections), then navigate database > schema > table (list_connection_metadata with entity="database" > "schema" > "table") | ||
| 2. Get column metadata: list_connection_metadata(entity="column") to understand datatypes, PKs, nullability | ||
| 3. Sample data: fetch_db_sample_data with databaseName="" for Azure SQL | ||
| 4. Profile: profile_data with the sample data array | ||
| 5. Get suggestions: suggest_quality_checks with the profile output | ||
| 6. Review: Present suggested checks to user before creating | ||
| 7. Create: Use create_validation_rule — combine ALL checks for the same table into ONE rule (avoid rule sprawl) | ||
| 8. Execute: execute_rules_or_workflows (objectIds: [ruleId]) > get_workflow_run_status_or_result (action=status) > get_workflow_run_status_or_result (action=result) | ||
| 9. Review exceptions: get_checks_exception_report for row-level failure details | ||
| 1. Navigate: list_connections → list_connection_metadata (database > schema > table) | ||
| 2. Show columns: list_connection_metadata(entity="column") → present columns to user | ||
| 3. Ask user: "Which columns do you want to validate?" → user selects columns | ||
| 4. Sample data: fetch_db_sample_data (use selected columns only, limit=100) | ||
| 5. Profile: profile_data with the sample data array | ||
| 6. Suggest: suggest_quality_checks with the profile output | ||
| 7. Approve: Present suggested checks → WAIT for user approval/modifications | ||
| 8. Create: create_validation_rule with approved checks (ONE rule for all checks) | ||
| 9. Execute (optional): execute_rules_or_workflows if user requests | ||
| 10. Review exceptions: get_checks_exception_report for failures | ||
@@ -47,0 +61,0 @@ --- |
@@ -8,8 +8,8 @@ STEP 0 — ASK THE USER FIRST (MANDATORY): | ||
| MULTIPLE RUN INSTANCES — ALWAYS ASK: | ||
| Before calling get_exception_report_url or get_checks_exception_report, fetch the run history via get_rule_workflow_run_history (for rules) or get_workflow_run_result (for workflows). If more than one completed instance exists, present the list with run date and status and ask the user which instance they want the exception report for. Do NOT auto-select the latest. | ||
| Before calling get_exception_report_url or get_checks_exception_report, fetch the run history via get_rule_workflow_run_history (for rules) or get_workflow_run_status_or_result (action=result, for workflows). If more than one completed instance exists, present the list with run date and status and ask the user which instance they want the exception report for. Do NOT auto-select the latest. | ||
| RETRIEVING EXCEPTION REPORTS: | ||
| 1. From rule name: list_rules (find ruleId) > get_rule_workflow_run_history (find objectInstanceId) > get_checks_exception_report | ||
| 2. From execution: execute_rule (get instanceId) > get_workflow_run_status_or_result (action=result, find activity instance.id) > get_checks_exception_report OR get_exception_report_url | ||
| 4. For workflow exception report: get_workflow (find workflowId and workflowName) > get_workflow_run_result (list instances, ask user to pick if >1) > get_exception_report_url (entityType="workflow") | ||
| 2. From execution: execute_rules_or_workflows (get instanceId) > get_workflow_run_status_or_result (action=result, find activity instance.id) > get_checks_exception_report OR get_exception_report_url | ||
| 4. For workflow exception report: list_workflows (find workflowId and workflowName) > get_workflow_run_status_or_result (action=result, list instances, ask user to pick if >1) > get_exception_report_url (entityType="workflow") | ||
@@ -16,0 +16,0 @@ |
@@ -69,3 +69,3 @@ FILE-BASED RULE CREATION — WORKFLOW GUIDE | ||
| → returns: ruleId (draft), source columns | ||
| Step 4 list_schemas / list_tables / list_columns on DB connection → user picks target schema + table | ||
| Step 4 list_connection_metadata(entity="schema"/"table"/"column") on DB connection → user picks target schema + table | ||
| Step 5 update_rule(workspaceId, ruleId, | ||
@@ -82,3 +82,3 @@ targetConfig={ connectionId:<db>, databaseName, schemaName, tableName }, | ||
| → returns: ruleId (draft), target columns | ||
| Step 4 list_schemas / list_tables / list_columns on DB connection → user picks source schema + table | ||
| Step 4 list_connection_metadata(entity="schema"/"table"/"column") on DB connection → user picks source schema + table | ||
| Step 5 update_rule(workspaceId, ruleId, | ||
@@ -85,0 +85,0 @@ sourceConfig={ connectionId:<db>, databaseName, schemaName, tableName }, |
+3
-3
| { | ||
| "name": "@icedq/mcp-server", | ||
| "mcpName": "io.github.icedq-tools/mcp-server", | ||
| "version": "1.0.5", | ||
| "description": "MCP server for iceDQ Data Reliability Platform — 48 tools for validation, reconciliation, duplicate detection, scheduling, and data exploration.", | ||
| "version": "2.0.0-beta", | ||
| "description": "MCP server for iceDQ Data Reliability Platform — 49 tools for validation, reconciliation, duplicate detection, scheduling, and data exploration.", | ||
| "repository": { | ||
@@ -50,3 +50,3 @@ "type": "git", | ||
| "engines": { | ||
| "node": ">=20.0.0" | ||
| "node": ">=18.0.0" | ||
| }, | ||
@@ -53,0 +53,0 @@ "files": [ |
+349
-239
| <p align="center"> | ||
| <img src="https://raw.githubusercontent.com/icedq-tools/mcp-server/master/icon.svg" alt="iceDQ Logo" width="80" /> | ||
| <img src="https://cdn-ildhhnd.nitrocdn.com/lLTTsRqXojmKENiGvwrypcTvmrbIWtKJ/assets/images/source/rev-bd4cb96/icedq.com/wp-content/uploads/2025/01/icedq-logo.svg" alt="iceDQ Logo" width="80" /> | ||
| </p> | ||
| <h1 align="center">iceDQ MCP Server</h1> | ||
| <h1 align="center">iceDQ MCP Server v2.0.0 Beta</h1> | ||
| <p align="center"> | ||
| <strong>Connect your AI assistant to the iceDQ Data Reliability Platform</strong> | ||
| <strong>Connect your AI assistant to iceDQ Data Quality Platform</strong> | ||
| </p> | ||
@@ -19,112 +19,75 @@ | ||
| <p align="center"> | ||
| <img src="https://img.shields.io/badge/version-1.0.5-blue.svg" alt="Version" /> | ||
| <img src="https://img.shields.io/badge/license-Apache--2.0-green.svg" alt="License" /> | ||
| <img src="https://img.shields.io/badge/platform-Windows%20%7C%20macOS-lightgrey.svg" alt="Platform" /> | ||
| <img src="https://img.shields.io/badge/version-2.0.0--beta-blue.svg" alt="Version" /> | ||
| <img src="https://img.shields.io/badge/status-beta-orange.svg" alt="Status" /> | ||
| <img src="https://img.shields.io/badge/license-Apache_2.0-green.svg" alt="License" /> | ||
| <img src="https://img.shields.io/badge/node-%3E%3D18.0.0-brightgreen.svg" alt="Node" /> | ||
| </p> | ||
| --- | ||
| > **Beta release.** This is a pre-release build of v2.0.0 for early testing. Everything below is functional and | ||
| > stable enough for real use, but interfaces may still change slightly before the general-availability release. | ||
| > Found an issue or have feedback? Reach us at [getsupport@icedq.com](mailto:getsupport@icedq.com). | ||
| ## What Is This? | ||
| The iceDQ MCP Server lets you manage your entire data quality lifecycle through conversation with an AI assistant. | ||
| Ask it to explore your data sources, create validation rules, run reconciliations, monitor executions, and analyze | ||
| results — no UI switching required. | ||
| **48 tools** covering the full data quality lifecycle, grouped by what they do (see | ||
| [`manifest.json`](./manifest.json) for the exact tool names and descriptions the assistant calls): | ||
| | Category | Tools | What you can do | | ||
| |----------------------------------|:-----:|---------------------------------------------------------------------------------------------------------------| | ||
| | **Data Exploration** | 8 | Browse workspaces, connections, databases, schemas, tables, and columns; verify a connection is reachable | | ||
| | **Data Profiling & AI Suggestions** | 3 | Pull real sample rows, get null/uniqueness/pattern stats per column, and get AI-suggested checks from that profile | | ||
| | **Rule Creation** | 6 | Create any of the five rule types — including AI-suggested join keys and column mappings before building a reconciliation rule | | ||
| | **Rule Management** | 3 | Search/filter existing rules, inspect full configuration, and update checks, source/target, or join keys | | ||
| | **Workflows** | 4 | Chain rules into a workflow and adjust membership later | | ||
| | **Schedules & Automation** | 6 | Set up one-time/daily/weekly schedules, add more jobs later, trigger on demand, review run history | | ||
| | **Execution & Monitoring** | 4 | Run a rule or workflow, poll it to completion, and pull per-activity results and history | | ||
| | **Results & Exception Reporting**| 2 | Get the specific failing rows and reasons, or a link to view the report in the iceDQ UI | | ||
| | **Organization** | 5 | Organize rules/workflows into folders and track the async move operations | | ||
| | **Reusable Parameters** | 3 | Define reusable thresholds/date ranges/reference values, including bulk-loading from CSV | | ||
| | **Data Warehouse Analytics** | 3 | Ask natural-language questions about DQ history via schema-validated structured queries (no raw SQL) | | ||
| *(The 48th tool, `get_guidance`, isn't listed above — the assistant calls it internally before complex multi-step | ||
| operations; it's not something you ask for directly.)* | ||
| --- | ||
| ## Compatibility | ||
| ## How It Works? | ||
| Per the [v1.0.0 release notes](https://docs.icedq.com/guides/mcp-server/releases/v1/v1.0.0): | ||
| The iceDQ MCP Server connects **Claude Desktop**, **VS Code**, and **Cursor** to your **iceDQ Data Quality Platform** | ||
| instance, letting you manage data quality using natural language. Ask your AI assistant to explore your data, profile | ||
| tables, and create **Validation**, **Duplicate**, **Checksum**, **Pushdown**, and **Reconciliation** rules — then | ||
| execute, monitor, and analyze results, all through conversation. | ||
| | Client | Support | Setup guide | | ||
| |----------------------------------|----------------|-------------------------------------------------------------------------------| | ||
| | **Claude Desktop** | ✅ MCP Bundle | [Step-by-step](#claude-desktop) | | ||
| | **VS Code + GitHub Copilot Chat** | ✅ MCP client | [Step-by-step](#vs-code--github-copilot-chat) | | ||
| | **VS Code + Claude Code** | ✅ MCP client | [Step-by-step](#vs-code--claude-code) | | ||
| | **Cursor** | ✅ MCP client | [Step-by-step](#cursor) | | ||
| | **Windows** | ✅ Tested | — | | ||
| | **macOS** | ✅ Tested | — | | ||
| | **Node.js** | 18.x or higher | Only needed if you launch via `npx` — Claude Desktop's `.mcpb` path doesn't | | ||
| **49 tools** covering the full data quality lifecycle: | ||
| > **Recommended AI model:** Claude Sonnet 4 or higher, for the most accurate rule creation and workflow understanding. | ||
| | Capability | What you can do | | ||
| |----------------------------|--------------------------------------------------------------------------------------------------| | ||
| | **Data Exploration** | Browse workspaces, connections, databases, schemas, tables, columns, and files | | ||
| | **Multi-Source Connectors**| Query databases, flat files (CSV, Excel, Parquet, JSON, XML, MongoDB), and REST APIs as rule sources | | ||
| | **Data Profiling** | Fetch real sample data and analyze quality metrics (nulls, patterns, types) | | ||
| | **AI Suggestions** | Get intelligent check recommendations based on your data profile | | ||
| | **Validation Rules** | Create row-level rules with NotNull, Format, ValidValues, Length, Date, and Custom Groovy checks | | ||
| | **Duplicate Detection** | Identify duplicates on business keys, composite keys, or conditional criteria | | ||
| | **Pushdown Rules** | SQL-driven aggregate validation (GROUP BY, JOINs, referential integrity) | | ||
| | **Checksum Rules** | Cross-source comparison (row counts, sums) between two different connections | | ||
| | **Reconciliation** | Row-level cross-source matching with AI-powered join key and column mapping | | ||
| | **Custom Functions** | Create and manage reusable Java/Groovy functions for use across validation checks | | ||
| | **Workflows** | Chain multiple rules into sequential execution workflows | | ||
| | **Schedules** | Automate rule execution with one-time, daily, or weekly schedules | | ||
| | **Execution & Monitoring** | Run rules on demand, track status, and view exception reports | | ||
| | **Organization** | Manage folders, move rules in batch, create reusable parameters | | ||
| --- | ||
| ## Before You Start: Get Your iceDQ Credentials | ||
| ## System Requirements | ||
| Every install method below needs the same values from your iceDQ instance. See the | ||
| [Credentials Guide](https://docs.icedq.com/guides/mcp-server/credentials) for exactly where to find each one in the | ||
| iceDQ UI, and the [Authentication Guide](https://docs.icedq.com/guides/mcp-server/icedq-mcp-authentication) for how | ||
| the two auth modes differ. | ||
| | Requirement | Details | | ||
| |----------------------|-------------------------------------------------------------------| | ||
| | **Operating System** | Windows 10+, macOS 10.15+, or Linux *(see note below)* | | ||
| | **AI Client** | One of: Claude Desktop, VS Code, or Cursor (latest version) | | ||
| | **Node.js** | 18.0.0+ *(required for npx-based setup; the Claude Desktop extension bundles its own runtime)* | | ||
| | **iceDQ** | v7.5.0+ with a valid user account | | ||
| | Value | Env var | Required for | Example | | ||
| |----------------------|------------------------|-----------------------------|----------------------------------------| | ||
| | **Base URL** | `ICEDQ_BASE_URL` | Both modes | *No default — always your own instance URL* | | ||
| | **Realm** | `ICEDQ_REALM` | Both modes | `icedq` or `iam.icedq` | | ||
| | **Client ID** | `ICEDQ_CLIENT_ID` | Both modes | — | | ||
| | **Client Secret** | `ICEDQ_CLIENT_SECRET` | `username_password` only | — | | ||
| | **Username** | `ICEDQ_USERNAME` | `username_password` only | — | | ||
| | **Password** | `ICEDQ_PASSWORD` | `username_password` only | — | | ||
| | **Tokens file path** | `TOKENS_PATH` | `access_token` only | — | | ||
| | **Organization ID** | `ICEDQ_ORG_ID` | Both modes | `org-icedq` or `org-iam.icedq` | | ||
| > **Linux users:** Install via the npx method (works in VS Code and Cursor). The packaged Claude Desktop extension (`.mcpb`) is currently macOS and Windows only because Claude Desktop itself does not ship a Linux build. | ||
| Optional: `VERIFY_SSL` (default `true`; set `"false"` only for self-signed certs), `REQUEST_TIMEOUT` (default `60` | ||
| seconds), `DEBUG` (default `false`). | ||
| > **Base URL has no default.** Every organization runs its own iceDQ instance — the `https://app.icedq.net` | ||
| > value used throughout this guide's examples is illustrative only, not a shared cloud endpoint. Always replace | ||
| > it with your own instance's URL. | ||
| > **If you're launching via `npx`** (every client below except Claude Desktop's packaged extension), also set | ||
| > `NODE_OPTIONS=--use-system-ca` in the `env` block — this is in every official config example and avoids TLS errors | ||
| > on machines with a corporate root CA installed. | ||
| --- | ||
| ## Installation | ||
| ## Quick Start | ||
| Jump to your client: | ||
| ### Step 1 — Get Your iceDQ Credentials | ||
| - [Claude Desktop](#claude-desktop) | ||
| - [VS Code + GitHub Copilot Chat](#vs-code--github-copilot-chat) | ||
| - [VS Code + Claude Code](#vs-code--claude-code) | ||
| - [Cursor](#cursor) | ||
| You need six values from your iceDQ instance before you can configure the MCP server: | ||
| Each section below is a condensed quick-start. For the full walkthrough with screenshots and troubleshooting, follow | ||
| the linked guide on docs.icedq.com. | ||
| - iceDQ Base URL | ||
| - Realm (default `iam.icedq`) | ||
| - Client ID and Client Secret (created in iceDQ → Administration → Security → Client Credentials) | ||
| - Your iceDQ username and password | ||
| - Organization ID (read from any rule's metadata) | ||
| ### Claude Desktop | ||
| For step-by-step instructions with screenshots, see the [Credentials Guide](docs/guides/mcp-server/CREDENTIALS.md). | ||
| 📖 **Full guide:** [Setup in Claude Desktop](https://docs.icedq.com/guides/mcp-server/setup-in-claude-desktop) | ||
| ### Step 2 — Install via npm | ||
| Two paths — **Path A (recommended)** installs a packaged `.mcpb` extension with a settings form and stores your | ||
| password in your OS keychain; **Path B** hand-edits a config file and needs Node.js 18+. | ||
| The iceDQ MCP Server is published on npm as **[`@icedq/mcp-server`](https://www.npmjs.com/package/@icedq/mcp-server)**. Most AI clients can launch it automatically with `npx` — no manual download or build step required. | ||
| **Path A:** Download the `.mcpb` from the [Releases page](https://github.com/icedq-tools/mcp-server/releases), then | ||
| in Claude Desktop go to **Settings → Extensions → Install Extension** and select the file. Fill in the credentials | ||
| form that appears and click **Save**. | ||
| Add the following to your AI client's MCP configuration: | ||
| **Path B:** Edit `claude_desktop_config.json` (**Windows:** `%APPDATA%\Claude\claude_desktop_config.json`, | ||
| **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`): | ||
| ```json | ||
@@ -138,9 +101,10 @@ { | ||
| "ICEDQ_BASE_URL": "https://app.icedq.net", | ||
| "ICEDQ_REALM": "icedq", | ||
| "ICEDQ_CLIENT_ID": "your-client-id", | ||
| "ICEDQ_CLIENT_SECRET": "your-client-secret", | ||
| "ICEDQ_REALM": "iam.icedq", | ||
| "ICEDQ_CLIENT_ID": "<your-client-id>", | ||
| "ICEDQ_CLIENT_SECRET": "<your-client-secret>", | ||
| "AUTH_TYPE": "username_password", | ||
| "ICEDQ_USERNAME": "your-username", | ||
| "ICEDQ_PASSWORD": "your-password", | ||
| "ICEDQ_ORG_ID": "your-org-id" | ||
| "ICEDQ_USERNAME": "<your-username>", | ||
| "ICEDQ_PASSWORD": "<your-password>", | ||
| "ICEDQ_ORG_ID": "<your-org-id>", | ||
| "NODE_OPTIONS": "--use-system-ca" | ||
| } | ||
@@ -152,18 +116,28 @@ } | ||
| Fully quit and reopen Claude Desktop (closing the window isn't enough), then verify with `List my iceDQ workspaces`. | ||
| #### Configuration Reference | ||
| --- | ||
| | Variable | Required | Description | | ||
| |-----------------------|----------------------------|----------------------------------------------------------------------| | ||
| | `ICEDQ_BASE_URL` | Yes | Base URL of your iceDQ instance (e.g. `https://app.icedq.net`) | | ||
| | `ICEDQ_REALM` | Yes | Authentication realm (default `iam.icedq`) | | ||
| | `ICEDQ_CLIENT_ID` | Yes | OAuth client ID for API authentication | | ||
| | `AUTH_TYPE` | Yes | `username_password`, `access_token`, or `device_flow` | | ||
| | `ICEDQ_ORG_ID` | Yes | Your iceDQ organization ID | | ||
| | `ICEDQ_CLIENT_SECRET` | For `username_password` | OAuth client secret | | ||
| | `ICEDQ_USERNAME` | For `username_password` | Your iceDQ username | | ||
| | `ICEDQ_PASSWORD` | For `username_password` | Your iceDQ password | | ||
| | `TOKENS_PATH` | For `access_token` | Path to a token JSON file with `accessToken` and `refreshToken` | | ||
| | `REMEMBER_ME` | Optional, for `device_flow`| Defaults to remembering the cached session. Set to `false` to wipe stored tokens and force a fresh browser login | | ||
| | `DEBUG` | Optional | Set to `true` for verbose logging | | ||
| | `NODE_OPTIONS` | Optional | Set to `--use-system-ca` so Node trusts your OS certificate store (needed if your iceDQ instance uses a corporate/self-signed CA) | | ||
| ### VS Code + GitHub Copilot Chat | ||
| > **Claude Desktop users** can install the packaged extension instead of editing JSON — follow the setup guide below. | ||
| 📖 **Full guide:** [Setup in VS Code & Cursor](https://docs.icedq.com/guides/mcp-server/setup-in-vs-code-and-cursor#configure-vs-code) | ||
| #### Alternative: Device Flow Authentication (no password required) | ||
| Requires Node.js 18+ and the GitHub Copilot Chat extension, installed and signed in. | ||
| Instead of supplying a username and password, you can authenticate via **Device Flow** ([RFC 8628](https://datatracker.ietf.org/doc/html/rfc8628)) — the server opens a browser login page for you, and tokens are cached securely in your OS keychain (Windows Credential Manager, macOS Keychain, or Linux libsecret) so you only log in once. This is the recommended option for SSO/MFA-enabled accounts or shared machines where you don't want credentials stored in the MCP config. | ||
| 1. Command Palette (`Ctrl+Shift+P` / `Cmd+Shift+P`) → **MCP: Open user configuration** → opens `mcp.json`. | ||
| 2. Add: | ||
| ```json | ||
| { | ||
| "servers": { | ||
| "mcpServers": { | ||
| "icedq": { | ||
@@ -174,9 +148,6 @@ "command": "npx", | ||
| "ICEDQ_BASE_URL": "https://app.icedq.net", | ||
| "ICEDQ_REALM": "icedq", | ||
| "ICEDQ_CLIENT_ID": "your-client-id", | ||
| "ICEDQ_CLIENT_SECRET": "your-client-secret", | ||
| "AUTH_TYPE": "username_password", | ||
| "ICEDQ_USERNAME": "your-username", | ||
| "ICEDQ_PASSWORD": "your-password", | ||
| "ICEDQ_ORG_ID": "your-org-id", | ||
| "ICEDQ_REALM": "iam.icedq", | ||
| "ICEDQ_CLIENT_ID": "<your-client-id>", | ||
| "AUTH_TYPE": "device_flow", | ||
| "ICEDQ_ORG_ID": "<your-org-id>", | ||
| "NODE_OPTIONS": "--use-system-ca" | ||
@@ -189,21 +160,8 @@ } | ||
| 3. Save (`Ctrl+S`/`Cmd+S`) — VS Code shows a **Start** option next to the `icedq` entry. Click it. | ||
| 4. Verify in Copilot Chat (`Ctrl+Alt+I`/`Cmd+Ctrl+I`): `List my iceDQ workspaces`. | ||
| On first run, the server prints a verification URL and code to the console and opens your browser automatically. Once you log in, tokens are cached (OS keychain, falling back to a token file) and silently refreshed on subsequent runs — no need to re-authenticate. Set `REMEMBER_ME=false` to skip the cache and force a fresh login every time. See the [Device Flow internals guide](docs/deployment/device-flow-auth.md) for details. | ||
| > ⚠️ This file stores your password in plain text. Don't commit `.vscode/mcp.json` to git if you're using | ||
| > workspace-scoped settings. | ||
| #### Alternative: Access Token Authentication (pre-issued tokens) | ||
| --- | ||
| If you already have an OAuth access/refresh token pair (e.g. issued by your own automation or a prior login), point the server at a token JSON file instead of supplying credentials directly: | ||
| ### VS Code + Claude Code | ||
| 📖 **Full guide:** [Setup with Claude Code](https://docs.icedq.com/guides/mcp-server/setup-with-claude-code) | ||
| Use this instead of the Copilot Chat guide if you have a Claude subscription (Pro/Max/Team/Enterprise) or an | ||
| Anthropic API key rather than a Copilot subscription. Requires Node.js 18+ and the **Claude Code for VS Code** | ||
| extension published by **Anthropic** (`anthropic.claude-code`) — skip third-party wrappers. | ||
| **Path A — edit `.claude.json`** (home directory: **Windows** `%USERPROFILE%\.claude.json`, **macOS/Linux** | ||
| `~/.claude.json`): | ||
| ```json | ||
@@ -217,9 +175,7 @@ { | ||
| "ICEDQ_BASE_URL": "https://app.icedq.net", | ||
| "ICEDQ_REALM": "icedq", | ||
| "ICEDQ_CLIENT_ID": "your-client-id", | ||
| "ICEDQ_CLIENT_SECRET": "your-client-secret", | ||
| "AUTH_TYPE": "username_password", | ||
| "ICEDQ_USERNAME": "your-username", | ||
| "ICEDQ_PASSWORD": "your-password", | ||
| "ICEDQ_ORG_ID": "your-org-id", | ||
| "ICEDQ_REALM": "iam.icedq", | ||
| "ICEDQ_CLIENT_ID": "<your-client-id>", | ||
| "AUTH_TYPE": "access_token", | ||
| "TOKENS_PATH": "/path/to/tokens.json", | ||
| "ICEDQ_ORG_ID": "<your-org-id>", | ||
| "NODE_OPTIONS": "--use-system-ca" | ||
@@ -232,109 +188,245 @@ } | ||
| **Path B — CLI** (`npm install -g @anthropic-ai/claude-code` first if you don't have it): | ||
| `TOKENS_PATH` must point to a JSON file shaped like: | ||
| ```bash | ||
| claude mcp add icedq \ | ||
| --scope user \ | ||
| --env ICEDQ_BASE_URL=https://app.icedq.net \ | ||
| --env ICEDQ_REALM=icedq \ | ||
| --env ICEDQ_CLIENT_ID=<your-client-id> \ | ||
| --env ICEDQ_CLIENT_SECRET=<your-client-secret> \ | ||
| --env AUTH_TYPE=username_password \ | ||
| --env ICEDQ_USERNAME=<your-username> \ | ||
| --env ICEDQ_PASSWORD=<your-password> \ | ||
| --env ICEDQ_ORG_ID=<your-org-id> \ | ||
| --env NODE_OPTIONS=--use-system-ca \ | ||
| npx --yes @icedq/mcp-server | ||
| ```json | ||
| { | ||
| "accessToken": "<JWT access token>", | ||
| "refreshToken": "<JWT refresh token>" | ||
| } | ||
| ``` | ||
| (Windows PowerShell: use `` ` `` for line continuation instead of `\`, or put it all on one line.) | ||
| The server reads this file on startup, uses the access token until it expires, and automatically refreshes it (rewriting the file) using the refresh token — no browser or password prompt involved. This is the recommended option for headless automation, CI, or server-to-server integrations where interactive login isn't possible. | ||
| Verify either path with `claude mcp list` (should show `icedq`), reload VS Code | ||
| (**Developer: Reload Window**), then ask in the Claude Code panel: `List my iceDQ workspaces`. | ||
| ### Setup Guides | ||
| Choose your AI client for a step-by-step walkthrough: | ||
| | Client | Guide | | ||
| |------------------------------|-------------------------------------------------------------------------------------------------------------| | ||
| | **Claude Desktop** | [Installation in Claude Desktop](https://docs.icedq.com/guides/mcp-server/icedq-mcp-installation-in-claude) | | ||
| | **VS Code + Copilot Chat** | [VS Code Setup](https://docs.icedq.com/guides/mcp-server/icedq-mcp-vs-cursor-setup#configure-vs-code) | | ||
| | **VS Code + Claude Code** | [Claude Code Setup](https://docs.icedq.com/guides/mcp-server/icedq-mcp-claude-code-setup) | | ||
| | **Cursor** | [Cursor Setup](https://docs.icedq.com/guides/mcp-server/icedq-mcp-vs-cursor-setup#configure-cursor) | | ||
| For authentication configuration, see | ||
| the [Authentication Guide](https://docs.icedq.com/guides/mcp-server/icedq-mcp-authentication). | ||
| --- | ||
| ### Cursor | ||
| ## Usage Examples | ||
| 📖 **Full guide:** [Setup in VS Code & Cursor](https://docs.icedq.com/guides/mcp-server/setup-in-vs-code-and-cursor#configure-cursor) | ||
| ### 1. Explore Your Data | ||
| Cursor has built-in MCP support — no extra extension needed. Requires Node.js 18+. | ||
| **Prompt:** List my iceDQ workspaces and show the connections | ||
| 1. Settings (gear icon, or `Cmd+,`/`Ctrl+,`) → search **Tools & MCP** → **Add Custom MCP** → opens `mcp.json` | ||
| (**macOS:** `~/.cursor/mcp.json`, **Windows:** `%USERPROFILE%\.cursor\mcp.json`). | ||
| 2. Add: | ||
| **Response:** | ||
| > Found 2 workspaces: | ||
| > - Production_DQ (Active) | ||
| > - Development_DQ (Active) | ||
| > | ||
| > Production_DQ has 3 connections: | ||
| > - SQL_Server_Production (SqlServer) - ACTIVE | ||
| > - Snowflake_Warehouse (Snowflake) - ACTIVE | ||
| > - Azure_DataLake (AzureBlob) - ACTIVE | ||
| ```json | ||
| { | ||
| "mcpServers": { | ||
| "icedq": { | ||
| "command": "npx", | ||
| "args": ["-y", "@icedq/mcp-server"], | ||
| "env": { | ||
| "ICEDQ_BASE_URL": "https://app.icedq.net", | ||
| "ICEDQ_REALM": "icedq", | ||
| "ICEDQ_CLIENT_ID": "your-client-id", | ||
| "ICEDQ_CLIENT_SECRET": "your-client-secret", | ||
| "AUTH_TYPE": "username_password", | ||
| "ICEDQ_USERNAME": "your-username", | ||
| "ICEDQ_PASSWORD": "your-password", | ||
| "ICEDQ_ORG_ID": "your-org-id", | ||
| "NODE_OPTIONS": "--use-system-ca" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| ### 2. Profile Data Quality | ||
| 3. Save, go back to **Settings → Tools & MCP**, and enable the toggle next to **icedq** — status should show | ||
| **Active**. | ||
| 4. Verify in Cursor chat (`Cmd+L`/`Ctrl+L`): `List my iceDQ workspaces`. | ||
| **Prompt:** Fetch sample data from the Customers table and suggest quality checks | ||
| --- | ||
| **Response:** | ||
| > Fetched 10 rows from dbo.Customers. Profiling results: | ||
| > | ||
| > Email: 33% null, valid email pattern in 60% of values | ||
| > FirstName: 1 empty value detected | ||
| > Status: 2 distinct values: Active, Inactive | ||
| > CreatedDate: Date pattern yyyy-MM-dd detected | ||
| > | ||
| > Suggested checks: | ||
| > 1. Email - NotNull (33% nulls detected) | ||
| > 2. Email - Format: Email pattern | ||
| > 3. FirstName - NotNull (empty values found) | ||
| > 4. Status - ValidValues: Active, Inactive | ||
| > 5. CreatedDate - Date format: yyyy-MM-dd | ||
| ### Access token mode (all clients) | ||
| ### 3. Create a Validation Rule | ||
| If you'd rather not store a password in a config file, every client above also supports **access token** mode — | ||
| swap the `username_password` fields for: | ||
| **Prompt:** Create a validation rule called Customer_Quality_Checks with those suggested checks | ||
| ```json | ||
| "env": { | ||
| "ICEDQ_BASE_URL": "https://app.icedq.net", | ||
| "ICEDQ_REALM": "icedq", | ||
| "ICEDQ_CLIENT_ID": "your-oauth-client-id", | ||
| "AUTH_TYPE": "access_token", | ||
| "TOKENS_PATH": "/full/path/to/icedq-tokens.json", | ||
| "ICEDQ_ORG_ID": "your-org-id" | ||
| } | ||
| ``` | ||
| **Response:** | ||
| > Rule created successfully! | ||
| > Rule ID: rule-12345678-abcd-ef01-2345-678901234567 | ||
| > Name: Customer_Quality_Checks | ||
| > Checks: 5 checks configured | ||
| > Status: Draft (ready to publish in iceDQ UI) | ||
| Generate the token file from iceDQ's **Profile → Token Generation**. See the | ||
| [Authentication Guide](https://docs.icedq.com/guides/mcp-server/icedq-mcp-authentication#access-token-mode) for the | ||
| full walkthrough — the connector refreshes the token automatically and writes the new pair back to the same file. | ||
| ### 4. Detect Duplicates | ||
| **Prompt:** Check for duplicate emails in the Customers table | ||
| **Response:** | ||
| > Duplicate detection rule created! | ||
| > Rule ID: rule-aabbccdd-1122-3344-5566-778899001122 | ||
| > Name: Customer_Email_Duplicates | ||
| > Columns: Email | ||
| > Status: Published (ready to execute) | ||
| ### 5. Compare Data Across Sources | ||
| **Prompt:** Compare row counts between SQL Server Orders table and Snowflake FACT_ORDERS | ||
| **Response:** | ||
| > Checksum rule created! | ||
| > Rule ID: rule-eeff0011-2233-4455-6677-889900aabbcc | ||
| > Name: Orders_vs_FACT_ORDERS_Checksum | ||
| > Source: SQL_Server_Production / dbo.Orders | ||
| > Target: Snowflake_Warehouse / DATAOPS_DW.FACT_ORDERS | ||
| > Check: SOURCE_COUNT - TARGET_COUNT == 0 | ||
| ### 6. Execute and Monitor | ||
| **Prompt:** Execute the Customer_Quality_Checks rule and show results | ||
| **Response:** | ||
| > Execution completed! | ||
| > Status: Success | ||
| > Source rows: 1,000 | ||
| > Exceptions: 5 rows failed validation | ||
| > Exit code: 5 | ||
| > | ||
| > Breakdown: | ||
| > - Email_NotNull: 3 failures | ||
| > - Email_Format: 1 failure | ||
| > - FirstName_NotNull: 1 failure | ||
| ### 7. Analyze Rule Execution History | ||
| **Prompt:** Analyze all rules executed last month and give me a summary of passed, failed, and errored rules | ||
| **Response:** | ||
| > Execution Summary — Last 30 Days | ||
| > | ||
| > Total rules executed: 24 | ||
| > | ||
| > Passed: 17 rules | ||
| > - Customer_Quality_Checks — 0 exceptions | ||
| > - Orders_NotNull_Check — 0 exceptions | ||
| > - Product_Format_Validation — 0 exceptions | ||
| > - ... 14 more | ||
| > | ||
| > Failed: 5 rules (exceptions found) | ||
| > - Email_Duplicate_Check — 312 exceptions | ||
| > - Orders_vs_FACT_ORDERS_Checksum — count mismatch detected | ||
| > - Address_Format_Check — 87 exceptions | ||
| > - ... 2 more | ||
| > | ||
| > Errored: 2 rules (execution did not complete) | ||
| > - Inventory_Recon_Rule — connection timeout | ||
| > - Sales_Pushdown_Check — query syntax error | ||
| --- | ||
| ## Usage Examples | ||
| ## Complete Tool Reference | ||
| **Explore your data:** | ||
| > "List my iceDQ workspaces and show the connections in Production_DQ" | ||
| ### Discovery & Exploration (14 tools) | ||
| **Profile a table:** | ||
| > "Fetch sample data from the Customers table and suggest quality checks" | ||
| | Tool | Description | | ||
| |-----------------------|-----------------------------------------------------------------------| | ||
| | List Workspaces | List all workspaces in your iceDQ instance | | ||
| | List Connections | List data source connections in a workspace | | ||
| | Test Connection | Test connectivity for a data source connection | | ||
| | List Folders | List folders for organizing rules | | ||
| | List Rules | Search and filter rules by name, state, or type | | ||
| | List Workflows | List all workflows in a workspace | | ||
| | List Schedules | List all schedules in a workspace | | ||
| | List Databases | List databases for a connection | | ||
| | List Schemas | List schemas in a database | | ||
| | List Tables | List tables in a schema | | ||
| | List Columns | List columns and metadata for a table | | ||
| | List Files | List files available in a flat-file connection (CSV, Excel, blob/S3) | | ||
| | Get Database Metadata | Get connection details and capabilities | | ||
| | Get Rule | Get full rule configuration and checks | | ||
| **Create a validation rule:** | ||
| > "Create a validation rule called Customer_Quality_Checks with those suggested checks" | ||
| ### Data Analysis (5 tools) | ||
| **Run and monitor:** | ||
| > "Execute Customer_Quality_Checks and show me the results" | ||
| | Tool | Description | | ||
| |-------------------------|---------------------------------------------------------------------------------------| | ||
| | Fetch DB Sample Data | Execute SQL and fetch real sample rows from a database table | | ||
| | Fetch File Sample Data | Preview rows from a flat-file, Parquet, Excel, JSON, XML, or MongoDB connection and register its schema | | ||
| | Fetch API Sample Data | Call a REST API endpoint and fetch sample rows/columns to drive API rule creation | | ||
| | Profile Data | Analyze sample data for nulls, patterns, types, uniqueness | | ||
| | Suggest Quality Checks | AI-powered check recommendations from profiled data | | ||
| **Cross-source comparison:** | ||
| > "Compare row counts between SQL Server Orders and Snowflake FACT_ORDERS" | ||
| ### Rule Creation (6 tools) | ||
| **Reconciliation:** | ||
| > "Reconcile the Customers table between Oracle and Snowflake using email as the join key" | ||
| | Tool | Description | | ||
| |------------------------|----------------------------------------------------| | ||
| | Create Validation Rule | Row-level validation with 6 check types | | ||
| | Create Duplicate Rule | Duplicate detection on single or composite columns | | ||
| | Create Pushdown Rule | SQL-driven aggregate and cross-table validation | | ||
| | Create Checksum Rule | Cross-source numeric comparison (COUNT, SUM, AVG) | | ||
| | Analyze Recon Mapping | AI-powered join key and column mapping suggestions | | ||
| | Create Recon Rule | Row-level cross-source reconciliation | | ||
| **Analytics:** | ||
| > "Show me the top 5 rules that failed most often last week" | ||
| ### Custom Functions (3 tools) | ||
| | Tool | Description | | ||
| |------------------------|-----------------------------------------------------------------| | ||
| | Manage Custom Function | Create or update a reusable Java/Groovy function for use in checks | | ||
| | List Custom Functions | List all custom functions available in a workspace | | ||
| | Get Custom Function | Retrieve the full definition of a custom function by ID or name | | ||
| ### Rule Management (3 tools) | ||
| | Tool | Description | | ||
| |----------------|-----------------------------------------------| | ||
| | Update Rule | Add/remove checks, change source table or SQL | | ||
| | Move Rules | Move rules between folders (batch supported) | | ||
| | Move Workflows | Move workflows between folders | | ||
| ### Execution & Monitoring (9 tools) | ||
| | Tool | Description | | ||
| |-------------------------------|----------------------------------------------------------------------------------------| | ||
| | Execute Rule | Execute a rule or workflow on demand | | ||
| | Execute Schedule | Trigger a schedule on demand | | ||
| | Check Task Status | Monitor async operations (moves, etc.) | | ||
| | Check Workflow Run Status | Track rule/workflow execution progress | | ||
| | Get Workflow Run Result | Get detailed results with per-check exit codes | | ||
| | Get Checks Exception Report | View row-level failure details | | ||
| | Get Exception Report URL | Get the iceDQ UI URL to view the full exception report for a rule or workflow instance | | ||
| | Get Rule Workflow Run History | View execution history for a rule or workflow | | ||
| | Get Scheduler Runs History | View execution history for a schedule | | ||
| ### Organization (8 tools) | ||
| | Tool | Description | | ||
| |-----------------------------------|----------------------------------------------------| | ||
| | Create Folder | Create folders to organize rules | | ||
| | Create Workflow | Chain rules into sequential workflows | | ||
| | Add Rules to Workflow | Add rules to an existing workflow | | ||
| | Remove Rules from Workflow | Remove rules from a workflow | | ||
| | Create Schedule | Schedule automated rule execution | | ||
| | Modify Schedule | Update schedule timing and configuration | | ||
| | Add Rules & Workflows to Schedule | Add rules/workflows to a schedule | | ||
| | Get Guidance | Get step-by-step workflow guidance for iceDQ tasks | | ||
| ### Parameters (5 tools) | ||
| | Tool | Description | | ||
| |------------------------------|--------------------------------------| | ||
| | List Parameters | List all parameters in a workspace | | ||
| | Get Parameter | Retrieve the full configuration of a parameter by ID | | ||
| | Create Parameter | Create reusable configuration values | | ||
| | Update Parameter | Update parameter key-value pairs | | ||
| | Parse CSV & Create Parameter | Import parameters from a CSV file | | ||
| ### Data Warehouse Queries (3 tools) | ||
| | Tool | Description | | ||
| |-------------------------------------|-----------------------------------------------| | ||
| | Data Warehouse Query Schema | Get data warehouse query schema | | ||
| | Data Warehouse Query Executor | Execute structured data warehouse queries | | ||
| | Validate & Explain Structured Query | Validate and preview a query before execution | | ||
| --- | ||
@@ -344,15 +436,19 @@ | ||
| | Issue | Solution | | ||
| |-----------------------------|-------------------------------------------------------------------------------------------------------| | ||
| | **Server/extension not appearing** | Claude Desktop: fully quit and reopen. VS Code/Cursor: reload the window. Claude Code: `claude mcp list` | | ||
| | **Authentication failed** | Verify Client ID, Client Secret, username, and password by logging into iceDQ in your browser with the same values | | ||
| | **No workspaces returned** | Check `ICEDQ_ORG_ID` and confirm your user has workspace access | | ||
| | **SSL certificate error** | Set `VERIFY_SSL` to `false` (self-signed certs only — not for production) | | ||
| | **Invalid JSON** | One missing comma/quote breaks the config — validate at [jsonlint.com](https://jsonlint.com/) | | ||
| | **npm download blocked by corporate proxy** | `npm install -g @icedq/mcp-server`, then set `"command": "icedq-mcp-server"` with empty `args` | | ||
| | Issue | Solution | | ||
| |-----------------------------------------|-----------------------------------------------------------------------------------| | ||
| | **Organization ID required** | Add your Organization ID in configuration (e.g. `org-icedq`) | | ||
| | **SSL certificate verification failed** | Uncheck "Verify SSL" in settings (for self-signed certificates only) | | ||
| | **No workspaces returned** | Verify credentials, check base URL, ensure user has workspace access | | ||
| | **Sample data not returning** | Check connection is ACTIVE, verify table name (case-sensitive), check permissions | | ||
| | **Authentication failures** | Verify client ID, client secret, username, and password are correct | | ||
| Each client's full guide (linked above) has an exhaustive troubleshooting section, including exact Debug Mode steps | ||
| and log locations for that client. | ||
| ### Enable Debug Mode | ||
| Claude Desktop log file locations: | ||
| For detailed troubleshooting, enable verbose logging: | ||
| - **Claude Desktop (extension):** Settings → Extensions → iceDQ → Configure → **Debug Mode: ON** | ||
| - **npx / manual configuration:** add `"DEBUG": "true"` to the `env` block of your MCP configuration | ||
| Claude Desktop extension log locations: | ||
| - **Windows:** `%APPDATA%\Claude\Logs\extensions\` | ||
@@ -365,13 +461,25 @@ - **macOS:** `~/Library/Logs/Claude/extensions/` | ||
| - Claude Desktop's packaged extension (Path A) stores your password in your OS keychain (Windows Credential | ||
| Manager / macOS Keychain). Every other setup path — Claude Desktop Path B, VS Code, Claude Code, and Cursor — | ||
| stores credentials in plain text in that client's config file. Don't commit those files to version control. | ||
| - All communication uses HTTPS with OAuth 2.0 | ||
| - Data flows directly between your AI client and your iceDQ instance — no third parties, no vendor-hosted relay | ||
| - No telemetry or tracking of any kind | ||
| - No data persistence beyond the active session | ||
| - Use a separate OAuth client per user, and prefer `access_token` mode on shared machines | ||
| ### How Your Data is Protected | ||
| Full details: [https://icedq.com/privacy-policy](https://icedq.com/privacy-policy) | ||
| - **Credentials** are provided through your AI client's configuration and sent only to your iceDQ instance — the Claude Desktop extension stores them in your operating system keychain | ||
| - **All communication** uses HTTPS with OAuth 2.0 authentication | ||
| - **Data flows directly** between your AI client and your iceDQ instance -- no third parties | ||
| - **No telemetry** or tracking of any kind | ||
| - **No data persistence** by the MCP server beyond the active session | ||
| - **SSL verification** is enabled by default | ||
| ### Privacy Policy | ||
| **Data collection:** None. The MCP server collects no usage data, telemetry, or analytics. | ||
| **Usage & storage:** All data flows directly between your AI client and your iceDQ instance. The MCP server holds credentials and API tokens in memory only for the duration of the active session. In `access_token` mode, tokens are persisted to the file path you supply (`TOKENS_PATH`) on your local machine — no data is written anywhere else. | ||
| **Third-party sharing:** None. No data is transmitted to Anthropic, iceDQ, or any third party beyond your own iceDQ instance. | ||
| **Data retention:** The MCP server retains nothing after the session ends. Token files (if used) remain on your local machine under your full control and can be deleted at any time. | ||
| **Contact:** [getsupport@icedq.com](mailto:getsupport@icedq.com) | ||
| For full details, see: [https://icedq.com/privacy-policy](https://icedq.com/privacy-policy) | ||
| --- | ||
@@ -381,14 +489,16 @@ | ||
| | Channel | Contact | | ||
| |-------------------|-----------------------------------------------------| | ||
| | **Email** | [getsupport@icedq.com](mailto:getsupport@icedq.com) | | ||
| | **Documentation** | [docs.icedq.com](https://docs.icedq.com) | | ||
| | **Website** | [icedq.com](https://icedq.com) | | ||
| Need help? We're here for you. | ||
| | Channel | Contact | | ||
| |-------------------|--------------------------------------------------| | ||
| | **Email** | [getsupport@icedq.com](mailto:getsupport@icedq.com) | | ||
| | **Documentation** | [docs.icedq.com](https://docs.icedq.com) | | ||
| | **Website** | [icedq.com](https://icedq.com) | | ||
| --- | ||
| <p align="center"> | ||
| <strong>iceDQ Data Reliability Platform</strong><br/> | ||
| <strong>iceDQ Data Quality Platform</strong><br/> | ||
| <em>End-to-end data reliability, powered by AI</em><br/><br/> | ||
| <a href="https://icedq.com">icedq.com</a> | ||
| </p> |
Sorry, the diff of this file is too big to display
No v1
QualityPackage is not semver >=1. This means it is not stable and does not support ^ ranges.
Major refactor
Supply chain riskPackage has recently undergone a major refactor. It may be unstable or indicate significant internal changes. Use caution when updating to versions that include significant changes.
2385931
55.96%20
11.11%60574
76.68%495
28.57%0
-100%3
50%