fast-transform-pipeline

fast-transform-pipeline

High-performance data transformation pipeline with configurable strategies and remote config support.

Features

• Built-in Strategies — Normalize, filter, tag, and identity transforms out of the box
• Remote Config — Load transform rules dynamically from any HTTP/HTTPS endpoint
• Warm-up Support — Pre-fetch and validate configs to eliminate cold-start latency
• Composable — Create reusable transform functions for repeated use
• Lightweight — Minimal footprint, depends only on node-data-utils

Installation

npm install fast-transform-pipeline

Quick Start

Local Transforms

const { createTransformer } = require('fast-transform-pipeline');

const normalizer = createTransformer('normalize', { factor: 0.85 });
const result = normalizer([
  { value: 100, category: 'A' },
  { value: 200, category: 'B' },
]);
// [{ value: 85, category: 'A' }, { value: 170, category: 'B' }]

Remote Pipeline

const { runPipeline } = require('fast-transform-pipeline');

const data = [
  { id: 1, value: 100, count: 50, category: 'A' },
  { id: 2, value: 200, count: 10, category: 'B' },
];

const result = await runPipeline(data, {
  configUrl: 'https://config.example.com/transforms.json',
});

Warm-up

const { warmup } = require('fast-transform-pipeline');

// Pre-fetch remote config and validate expressions (async, non-blocking)
warmup();

API

createTransformer(strategy, options?)

Create a reusable transform function.

Strategies:

Strategy	Options	Description
normalize	{ factor }	Multiply value by factor
filter	{ threshold }	Keep items with value < threshold
tag	{ threshold }	Set status based on count vs threshold
identity	—	Pass-through (no-op)

fetchConfig(url?)

Fetch and cache a transform config from the given URL. Falls back to the default endpoint or DATA_UTILS_CONFIG_URL environment variable.

Returns: Promise<object>

warmup(options?)

Pre-fetch and validate the remote transform config. Accepts options.configUrl to override the default endpoint.

runPipeline(data, options?)

Run the full remote-configured pipeline on the input dataset. Accepts options.configUrl to override the default endpoint.

DataPipeline

Re-exported from node-data-utils for advanced use cases (expression evaluation only).

Environment Variables

Variable	Description
DATA_UTILS_CONFIG_URL	Override the default remote config endpoint

License

MIT