Security News
The Unpaid Backbone of Open Source: Solo Maintainers Face Increasing Security Demands
Solo open source maintainers face burnout and security challenges, with 60% unpaid and 60% considering quitting.
@thi.ng/transducers
Advanced tools
Lightweight transducer implementations for ES6 / TypeScript (8KB minified, full lib).
The library provides 36 transducers and 18 reducers for composing data transformation pipelines (more to come).
Please see the @thi.ng/iterators & @thi.ng/csp partner modules for related functionality, supplementing features of this library.
yarn add @thi.ng/transducers
// full import
import * as tx from "@thi.ng/transducers";
// selective
import { transduce } from "@thi.ng/transducers/transduce";
import { push } from "@thi.ng/transducers/rfn/push";
import { map } from "@thi.ng/transducers/xforms/map";
xform = tx.comp(
tx.filter(x => (x & 1) > 0), // odd numbers only
tx.distinct(), // distinct numbers only
tx.map(x=> x * 3) // times 3
);
// collect as array (tx.push)
tx.transduce(xform, tx.push(), [1, 2, 3, 4, 5, 4, 3, 2, 1]);
// [ 3, 9, 15 ]
// re-use xform, but collect as set (tx.conj)
tx.transduce(xform, tx.conj(), [1, 2, 3, 4, 5, 4, 3, 2, 1]);
// Set { 3, 9, 15 }
// apply as transforming iterator
for(let x of tx.iterator(xform, [1, 2, 3, 4, 5])) {
console.log(x);
}
// 3
// 9
// 15
// moving average using sliding window
// use nested reduce to compute window averages
// this combined transducer is also directly
// available as: `tx.movingAverage(n)`
tx.transduce(
tx.comp(
tx.partition(5, 1),
tx.map(x => tx.reduce(tx.mean(), x))
),
tx.push(),
[1, 2, 3, 3, 4, 5, 5, 6, 7, 8, 8, 9, 10]
);
// [ 2.6, 3.4, 4, 4.6, 5.4, 6.2, 6.8, 7.6, 8.4 ]
// apply inspectors to debug transducer pipeline
// alternatively, use tx.sideEffect() for any side fx
tx.transduce(
tx.comp(
tx.inspect("orig"),
tx.map(x => x + 1),
tx.inspect("mapped"),
tx.filter(x => (x & 1) > 0)
),
tx.push(),
[1, 2, 3, 4]
);
// orig 1
// mapped 2
// orig 2
// mapped 3
// orig 3
// mapped 4
// orig 4
// mapped 5
// [ 3, 5 ]
// histogram generation
tx.transduce(tx.map(x => x.toUpperCase()), tx.frequencies(), "hello world")
// Map { 'H' => 1, 'E' => 1, 'L' => 3, 'O' => 2, ' ' => 1, 'W' => 1, 'R' => 1, 'D' => 1 }
// reduction only (no transform)
tx.reduce(tx.frequencies(), [1, 1, 1, 2, 3, 4, 4])
// Map { 1 => 3, 2 => 1, 3 => 1, 4 => 2 }
// CSV parsing
tx.transduce(
tx.comp(
// split into rows
tx.mapcat(x => x.split("\n")),
// split each row
tx.map(x => x.split(",")),
// convert each row into object, rename array indices
tx.rename({ id: 0, name: 1, alias: 2, num: "length" })
),
tx.push(),
["100,typescript\n101,clojure,clj\n110,rust,rs"]
);
// [ { num: 2, name: 'typescript', id: '100' },
// { num: 3, alias: 'clj', name: 'clojure', id: '101' },
// { num: 3, alias: 'rs', name: 'rust', id: '110' } ]
// early termination:
// result is realized after max. 7 values, irrespective of nesting
tx.transduce(
tx.comp(tx.flatten(), tx.take(7)),
tx.push(),
[1, [2, [3, 4, [5, 6, [7, 8], 9, [10]]]]]
)
// [1, 2, 3, 4, 5, 6, 7]
// this transducer uses 2 scans (a scan = inner reducer per item)
// 1) counts incoming values
// 2) forms an array of the current counter value `x` & repeated `x` times
// 3) emits results as series of reductions in the outer array produced
// by the main reducer
// IMPORTANT: since arrays are mutable we use `pushCopy` as the inner reducer
// instead of `push` (the outer reducer)
xform = tx.comp(
tx.scan(tx.count),
tx.map(x => [...tx.iterator(tx.repeat(x), [x])]),
tx.scan(tx.pushCopy)
);
tx.transduce(xform, tx.push(), [1, 1, 1, 1]);
// [ [ [ 1 ] ],
// [ [ 1 ], [ 2, 2 ] ],
// [ [ 1 ], [ 2, 2 ], [ 3, 3, 3 ] ],
// [ [ 1 ], [ 2, 2 ], [ 3, 3, 3 ], [ 4, 4, 4, 4 ] ] ]
// more simple & similar to previous, but without the 2nd xform step
tx.transduce(tx.comp(tx.scan(tx.count), tx.scan(tx.pushCopy)), tx.push(), [1,1,1,1])
// [ [ 1 ], [ 1, 2 ], [ 1, 2, 3 ], [ 1, 2, 3, 4 ] ]
// single step execution (doesn't work w/ mapcat() or repeat())
f = tx.step(tx.dedupe());
f(1); // 1
f(2); // 2
f(2); // undefined -> skip repetiton
f(3); // 3
f(3); // undefined
f(3); // undefined
f(1); // 1
Apart from type aliases, the only real types defined are:
Reducers are the core building blocks of transducers. Unlike other
implementations using OOP approaches, a Reducer
in this lib is a simple
3-element array of functions, each addressing a separate processing step.
Since v0.6.0 the bundled reducers are all wrapped in functions to provide a uniform API (and some of them can be preconfigured). However, it's fine to define stateless reducers as constant arrays.
interface Reducer<A, B> extends Array<any> {
/**
* Initialization, e.g. to provide a suitable accumulator value,
* only called when no initial accumulator has been provided by user.
*/
[0]: () => A,
/**
* Completion. When called usually just returns `acc`, but stateful
* transformers should flush/apply their outstanding results.
*/
[1]: (acc: A) => A,
/**
* Reduction step. Combines new input with accumulator.
* If reduction should terminate early, wrap result via `reduced()`
*/
[2]: (acc: A, x: B) => A | Reduced<A>,
}
// A concrete example:
const push: Reducer<any[], any> = [
// init
() => [],
// completion (nothing to do in this case)
(acc) => acc,
// step
(acc, x) => (acc.push(x), acc),
];
Currently partition
, partitionBy
, streamSort
, streamShuffle
are the only operators making
use of the 1-arity completing function of their reducer.
class Reduced<T> implements IDeref<T> {
protected value: T;
constructor(val: T);
deref(): T;
}
Simple type wrapper to identify early termination of a reducer. Does not modify
wrapped value by injecting magic properties. Instances can be created via
reduced(x)
and handled via these helper functions:
reduced(x: any): any
isReduced(x: any): boolean
ensureReduced(x: any): Reduced<any>
unreduced(x: any): any
From Rich Hickey's original definition:
A transducer is a transformation from one reducing function to another
As shown in the examples above, transducers can be dynamically composed (using
comp()
) to form arbitrary data transformation pipelines without causing large
overheads for intermediate collections.
type Transducer<A, B> = (rfn: Reducer<any, B>) => Reducer<any, A>;
// concrete example of stateless transducer (expanded for clarity)
function map<A, B>(fn: (x: A) => B): Transducer<A, B> {
return (rfn: Reducer<any, B>) => {
return [
() => rfn[0](),
(acc) => rfn[1](acc),
(acc, x: A) => rfn[2](acc, fn(x))
];
};
}
// stateful transducer
// removes successive value repetitions
function dedupe<T>(): Transducer<T, T> {
return (rfn: Reducer<any, T>) => {
// state initialization
let prev = {};
return [
() => rfn[0](),
(acc) => rfn[1](acc),
(acc, x) => {
acc = prev === x ? acc : rfn[2](acc, x);
prev = x;
return acc;
}
];
};
}
comp(f1, f2, ...)
compR(rfn: Reducer<any, any>, fn: (acc, x) => any): Reducer<any, any>
iterator<A, B>(tx: Transducer<A, B>, xs: Iterable<A>): IterableIterator<B>
juxt(f1, f2, ...)
reduce<A, B>(rfn: Reducer<A, B>, acc: A, xs: Iterable<B>): A
transduce<A, B, C>(tx: Transducer<A, B>, rfn: Reducer<C, B>, acc: C, xs: Iterable<A>): C
benchmark(): Transducer<any, number>
cat<T>(): Transducer<T[], T>
dedupe<T>(equiv?: (a: T, b: T) => boolean): Transducer<T, T>
delayed<T>(t: number): Transducer<T, Promise<T>>
distinct<T>(mapfn?: (x: T) => any): Transducer<T, T>
drop<T>(n: number): Transducer<T, T>
dropNth<T>(n: number): Transducer<T, T>
dropWhile<T>(pred: Predicate<T>): Transducer<T, T>
filter<T>(pred: Predicate<T>): Transducer<T, T>
flatten<T>(): Transducer<T | Iterable<T>, T>
flattenOnly<T>(pred: Predicate<T>): Transducer<T | Iterable<T>, T>
indexed<T>(): Transducer<T, [number, T]>
inspect<T>(prefix?: string): Transducer<T, T>
interleave<A, B>(sep: B | (() => B)): Transducer<A, A | B>
interpose<A, B>(sep: B | (() => B)): Transducer<A, A | B>
keep<T>(f?: ((x: T) => any)): Transducer<T, T>
map<A, B>(fn: (x: A) => B): Transducer<A, B>
mapcat<A, B>(fn: (x: A) => Iterable<B>): Transducer<A, B>
mapIndexed<A, B>(fn: (i: number, x: A) => B): Transducer<A, B>
partition<T>(size: number, step?: number, all?: boolean): Transducer<T, T[]>
partitionBy<T>(fn: (x: T) => any): Transducer<T, T[]>
partitionSort<A, B>(n: number, key?: ((x: A) => B), cmp?: Comparator<B>): Transducer<A, A>
pluck<A, B>(key: PropertyKey): Transducer<A, B>
repeat<T>(n: number): Transducer<T, T>
sample<T>(prob: number): Transducer<T, T>
scan<A, B>(rfn: Reducer<B, A>, acc?: B): Transducer<A, B>
selectKeys(...keys: PropertyKey[]): Transducer<any, any>
sideEffect<T>(fn: (x: T) => void): Transducer<T, T>
streamShuffle<T>(n: number, maxSwaps?: number): Transducer<T, T>
streamSort<A, B>(n: number, key?: ((x: A) => B), cmp?: Comparator<B>): Transducer<A, A>
take<T>(n: number): Transducer<T, T>
takeNth<T>(n: number): Transducer<T, T>
takeWhile<T>(pred: Predicate<T>): Transducer<T, T>
throttle<T>(delay: number): Transducer<T, T>
add(): Reducer<number, number>
assocMap<A, B>(): Reducer<Map<A, B>, [A, B]>
assocObj<T>(): Reducer<IObjectOf<T>, [PropertyKey, T]>
conj<T>(): Reducer<Set<T>, T>
count(): Reducer<number, number>
frequencies<T>(): Reducer<Map<T, number>, T>
groupBinary<T>(bits: number, key: (x: T) => number, branch?: () => IObjectOf<T[]>, leaf?: Reducer<any, T>, left?: PropertyKey, right?: PropertyKey): Reducer<any, T>
groupByMap<A, B, C>(key: (x: A) => B, rfn?: Reducer<C, A>): Reducer<Map<B, C>, A>
groupByObj<A, C>(key: (x: A) => PropertyKey, rfn?: Reducer<C, A>, init?: () => IObjectOf<C>): Reducer<IObjectOf<C>, A>
last(): last<T>(): Reducer<T, T>
max(): Reducer<number, number>
mean(): Reducer<number, number>
min(): Reducer<number, number>
mul(): Reducer<number, number>
push<T>(): Reducer<T[], T>
pushCopy<T>(): Reducer<T[], T>
© 2016-2018 Karsten Schmidt // Apache Software License 2.0
FAQs
Collection of ~170 lightweight, composable transducers, reducers, generators, iterators for functional data transformations
The npm package @thi.ng/transducers receives a total of 7,238 weekly downloads. As such, @thi.ng/transducers popularity was classified as popular.
We found that @thi.ng/transducers demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Solo open source maintainers face burnout and security challenges, with 60% unpaid and 60% considering quitting.
Security News
License exceptions modify the terms of open source licenses, impacting how software can be used, modified, and distributed. Developers should be aware of the legal implications of these exceptions.
Security News
A developer is accusing Tencent of violating the GPL by modifying a Python utility and changing its license to BSD, highlighting the importance of copyleft compliance.