@thi.ng/iterators
Overview
Collection of approx. 50 composable, iterator-centric data processing functions,
largely implemented as ES6 iterators / generators, inspired by
clojure.core API.
Written in TypeScript.
See changelog for recent updates.
Installation
yarn add @thi.ng/iterators
All functions are defined as sub-modules and exposed as default exports. This is an additional feature. The full library can still be imported as before. Function sub-modules use Kebab case whereas function names are in Camel case.
let ti = require("@thi.ng/iterators");
let map = require("@thi.ng/iterators/map").default;
import map from "@thi.ng/iterators/map";
API
In alphabetical order:
butLast<T>(input: Iterable<T>) => IterableIterator<T>
Yields iterator of all but the last value of input.
[...ti.butLast([])]
[...ti.butLast([1])]
[...ti.butLast([1,2])]
[...ti.butLast([1,2,3])]
[...ti.butLast("hello")]
[...ti.butLast(ti.range(10))]
cached<T>(input: Iterable<T>) => CachedIterableIterator<T>
Consumes and lazily caches values of a finite input and returns a
no-arg function, which when called return new iterator. These iterator
instances always start from the beginning of the cache and allows for separate
states and sharing of cached results among arbitrary number of iterators. The
original input is only consumed when attempting to read beyond current cache
boundary.
c = ti.cached(ti.range(10));
a = c();
b = c();
a.next();
b.next();
b.next();
a.next();
angles = ti.cached(ti.range(0, 360, 45));
[...angles()]
ti.zip(
angles(),
ti.map(
ti.juxt(Math.sin, Math.cos),
ti.map(
(x)=> x * 180 / Math.PI,
angles()
)
)
);
concat<T>(...inputs: Iterable<T>[]) => IterableIterator<T>
Produces iterator yielding lazy concatenation of given iterables. For
practical purposes none but the last input should be infinite. Any null
or
undefined
input arguments are skipped in the output.
[... ti.concat([1, 2, 3], [10, 20, 30], [100, 200, 300])]
[...ti.concat.apply(null, ["abc", null, [1, 2, 3]])]
constantly<T>(x: T) => (...args: any[]) => T
Helper function returning a new fn which takes any number of args and always
returns x
.
iter = ti.takeWhile(x => x < 0.98, ti.repeatedly(()=> Math.random()));
ti.reduce((a, b)=> a + b, 0, ti.map(ti.constantly(1), iter))
times10 = (flt, coll) => [...ti.map(x => x * 10, ti.filter(flt, coll))];
filterModes = {
odd: x => (x % 2) == 1,
even: x => (x % 2) == 0,
all: ti.constantly(true)
};
times10(filterModes[???], [1, 2, 3]);
consume(input: Iterator<any>, n?: number) => void
Helper function. Consumes & discards items from iterator (possibly for
side effects) and optionally only up to the given number of steps.
cycle<T>(input: Iterable<T>) => IterableIterator<T>
Produces iterator which lazily caches & infinitely repeats sequence of input.
Important: Input MUST be finite, use take
to truncate input or output if necessary.
[... ti.take(10, ti.cycle(ti.range(3)))]
dedupe<T>(input: Iterable<T>) => IterableIterator<T>
Produces iterator which discards successive duplicate values from input.
Important: Uses ===
for equality checks.
[... ti.dedupe([1, 2, 2, 3, 4, 4, 4, 3])]
dedupeWith<T>(equiv: (a:T, b: T) => boolean, input: Iterable<T>) => IterableIterator<T>
Like dedupe
, but uses given function equiv
to determine equivalence
of successive values.
var coll = [{ a: 1 }, { a: 1, b: 2 }, { a: 2, b: 2 }, { a: 2, b: 2 }, { a: 3 }];
var eq = (a, b) => a.a === b.a;
[...ti.dedupeWith(eq, coll)]
dense<T>(input: Iterable<T>) => IterableIterator<T>
Yields iterator of all non-null
and non-undefined
values of input (e.g. a
sparse array).
var a = []
a[10] = 1;
a[20] = 2;
[...ti.dense(a)]
drop<T>(n: number, input: Iterable<T>) => IterableIterator<T>
Consumes & discards up to n
items from input and returns remaining
(possibly exhausted) iterator.
[... ti.drop(5, ti.range(10))]
[... ti.drop(5, ti.range(3))]
[... ti.take(5, ti.drop(5, ti.range()))]
dropNth<T>(n: number, input: Iterable<T>) => IterableIterator<T>
Produces iterator which drops every n
th item from input.
[... ti.dropNth(2, ti.range(10))]
[... ti.dropNth(3, ti.range(10))]
dropWhile<T>(pred: (x: T) => boolean, input: Iterable<T>) => IterableIterator<T>
Consumes input and calls pred
for each item.
Discards all items whilst pred
returns true
, then returns remaining
(possibly exhausted) iterator.
[... ti.dropWhile((x) => x < 5, ti.range(10))]
every<T>(pred: (x: T) => boolean, input: Iterable<T>) => boolean
Consumes input and calls pred
for each item. When pred
returns not true
,
process stops and returns false
itself. When all items pass the predicate,
the function returns true
.
If input is empty/exhausted prior to execution, every
will return false
.
var nums = ti.iterator([2, 4, 6, 8, 10]);
ti.every((x) => (x % 2) === 0, nums);
nums.next()
nums = ti.iterator([2, 3, 4]);
ti.every((x) => (x % 2) === 0, nums);
nums.next()
ti.every((x) => true, [])
filter<T>(pred: (x: T) => boolean, input: Iterable<T>) => IterableIterator<T>
Consumes input and calls pred
for each item.
Yields iterator of items for which pred
returned true
.
var multOf3 = (x) => (x % 3) === 0;
[... ti.filter(multOf3, ti.range(10))];
flatten(input: Iterable<any>, flatObjects = true) => IterableIterator<any>
Produces iterator which recursively flattens input (using flattenWith
).
Important: Recursion only applies to iterable types (excluding strings) and
objects (enabled by default, using objectIterator
, see below).
[... ti.flatten([1, [2, 3], [4, [5, ["abc"]]]])]
[...ti.flatten([{ a: 1 }, { a: 23, b: 42, c: [1, 2, 3] }])]
[...ti.flatten([{ a: 1 }, [1, 2, 3], { a: 23, b: 42, c: [1, 2, 3] }], false)]
flattenWith(tx: (x: any) => any, input: Iterable<any>) => IterableIterator<any>
Produces iterator which selectively & recursively flattens input. The first arg
tx
is a transformation fn called for each non-null/undefined
value taken
from the input. It's used to check and possibly obtain an iteratable value for
further flattening. The transformer must return undefined
if the value can't
or shouldn't be flattened. If a value is returned it MUST be iterable.
The default transformer used by flatten
is:
let defaultTx = x =>
(typeof x !== "string" && (maybeIterator(x) || maybeObjectIterator(x))) ||
undefined;
let tx = x => typeof x == "string" ? ti.map(x => x.charCodeAt(0), x) : ti.maybeIterator(x);
[...ti.flattenWith(tx, ["ROOT", undefined, ["CHILD_1", null, ["CHILD_2"]]])]
fnil(fn: (...args: any[]) => any, ...ctors: (() => any)[]) => (...args: any[]) => any
Takes a function fn
and up to 3 ctor
functions. Produces a new function
that calls fn
, replacing a null
or undefined
arg with the value obtained
by calling its related positional ctor
fn (e.g. the first ctor is used to
supply first arg, etc.).
The function fn
can take any number of arguments, however only the first 3
are being potentially patched, how many depends on the number of ctor
fns
supplied.
hello = (greet, name) => `${greet}, ${name}!`;
helloEN = ti.fnil(hello, () => "Hi", () => "user");
helloDE = ti.fnil(hello, () => "Hallo", () => `Teilnehmer #${(Math.random()*100)|0}`);
helloEN();
helleEN(null,"toxi");
helloEN("Howdy","toxi");
helloDE();
inc = x => x + 1
adder = ti.fnil(inc, () => 0);
adder();
adder(null);
adder(10);
updateWith = f => (obj, id) => { return obj[id] = f(obj[id]), obj; }
ti.reduce(updateWith(adder), {}, "abracadabra");
fork<T>(input: Iterable<T>, cacheLimit = 16) => () => IterableIterator<T>
Similar to cached
, this function allows multiple consumers of a single input,
however is only using a sliding window of N cached values (cached
stores the
entire input).
fork
returns a no-arg function, which returns a new forked iterator when
called. There's no limit to the number of active forks.
Important: The use case for fork
is synchronized consumption at similar
speeds (up to cacheLimit
divergence). The cache is shared by all forks. If
one of the forks consumes the input faster than the given cacheLimit
, the
other forks will lose intermediate values. If in doubt, increase the cache
limit to a higher value (default 16). The cache uses
@thi.ng/dcons to avoid unnecessary copying during window
sliding.
src = ti.partition(5, 1, ti.repeatedly(()=> (Math.random() * 100) | 0, 10));
f = ti.fork(src, 1);
raw = f();
sma = ti.map((part)=> ti.reduce((a, b) => a + b, 0, part) / part.length, f());
min = ti.map((part)=> ti.reduce((a, b) => Math.min(a, b), 100, part), f());
max = ti.map((part)=> ti.reduce((a, b) => Math.max(a, b), -1, part), f());
for(let part of raw) {
console.log(`part: ${part}, avg: ${sma.next().value}, min: ${min.next().value}, ${max.next().value}`);
}
frequencies<T>(input: Iterable<T>, key?: (v: T) => any): IterableIterator<FrequencyPair<T>[]>
Consumes input, applies key
fn (if given) to each item and yields iterator of
2-element tuples, each [key, freq]
(where freq
is the number of times the
item occurred). Important: The input MUST be finite. Implementation uses
JSON.stringify
to determine key equality. If no key
fn is given, the
original values will be used as key.
[... ti.frequencies([[1,2], [2,3], [1,2], [2,4]])]
[... ti.frequencies([1, 2, 3, 4, 5, 9, 3], (x) => x & ~1)]
var isLetter = (x) => /[a-z]/i.test(x);
[... ti.frequencies(ti.filter(isLetter, "hello world!"))].sort((a, b) => b[1] - a[1])
groupBy<T>(key: (v) => any, input: Iterable<T>) => { [id: string]: T[] }
Consumes input, applies key
fn to each item and returns object of items
grouped by result of key
fn.
Important: The input MUST be finite.
Implementation uses JSON.stringify
to determine key equality.
groupBy((x) => x & ~1, [1, 2, 3, 4, 5, 9, 3])
groupBy((x) => x.toUpperCase(), "AbRaCadaBra")
identity<T>(x: T) => T
Helper function. Simply returns its argument.
ti.identity()
ti.identity(null)
ti.identity(42)
tests = [true, true, undefined, true]
ti.every(ti.identity, tests);
ti.every(ti.fnil(ti.identity, () => true), tests);
indexed<T>(input: Iterable<T>) => IterableIterator<[number, T]>
Yields iterator producing [index, value]
pairs of input.
[...ti.indexed("hello")]
interleave(...inputs: Iterable<any>[]) => IterableIterator<any>
Takes an arbitrary number of iterators and yields iterator of interleaved items from each input.
Terminates as soon as one of the inputs is exhausted.
[... ti.interleave(ti.range(), ti.range(100, 200), ti.range(200, 205))]
interpose(x: any, input: Iterable<any>) => IterableIterator<any>
Produces an iterator which injects x
inbetween each item from input iter
.
[... ti.interpose("/", ti.range(5))]
iterate<T>(fn: (x: T) => T, seed: T) => IterableIterator<T>
Produces an iterator of the infinite results of iteratively applying fn
to seed
.
Important: Use take
to truncate sequence.
[... ti.take(10, ti.iterate((x) => x * 2, 1))]
iterator<T>(x: Iterable<T>) => Iterator<T>
Syntax sugar. Returns x[Symbol.iterator]()
.
Most functions in this module call this automatically for each input.
juxt<T>(...fns: ((x: T) => any)[]) => (x: T) => any[]
Takes arbitrary number of functions and returns new function,
which takes single argument x
and produces array of result
values of applying each input function to x
(juxtoposition of the given transformation functions).
var kernel = ti.juxt(
(x) => x - 1,
(x) => x,
(x) => x + 1
);
kernel(1)
[... ti.map(kernel, ti.range(3))]
last<T>(input: Iterable<T>) => T
Consumes a finite iterator and returns last item.
Important: Never attempt to use with an infinite input!
ti.last(ti.range(10))
ti.last(ti.take(10, ti.range()))
map<T>(fn: (...args: any[]) => T, ...inputs: Iterable<any>[]) => IterableIterator<T>
Consumes an arbitrary number of iterators and applies fn
to each of their values.
Produces iterator of results. Iteration stops as soon as one of the inputs is exhausted.
The mapping fn
should accept as many arguments as there are inputs to map
.
Provides a fast path for single input mapping.
[... ti.map((x)=> x*10, ti.range(10))]
[... ti.map((x, y, z) => [x, y, z], ti.range(5), ti.range(0, 100, 10), ti.range(0, 1000, 100))]
mapcat<T>(fn: (...args: any[]) => Iterable<T>, ...inputs: Iterable<any>[]) => IterableIterator<T>
Like map
, but expects mapping fn to return an iterable result and
produces iterator which yields the flat concatenation of results
(only the first level of nesting is removed).
null
or undefined
values returned by the mapping fn are skipped
in the output.
[... ti.mapcat((x) => ti.repeat(x, 3), "hello")]
[...ti.mapcat((x) => x < 5 ? ti.repeat(x,x) : null, ti.range(10))]
mapIndexed<T>(fn: (i: number, ...args: any[]) => T[], ...inputs: Iterable<any>[]) => IterableIterator<T>
Like map
, but too passes a monotonically increasing index
as first
argument to mapping fn.
[... ti.mapIndexed((i, a, b) => [i, a, b], "hello", "there")]
maybeIterator(x: any) => any
Helper function, returning arg's iterator (if present) or else undefined
.
maybeObjectIterator(x: any) => any
Helper function, checks if x
is object-like (but no generator) and returns
objectIterator(x)
or else undefined
.
objectIterator(x: any) => IterableIterator<any[]>
Produces iterator of an object"s key/value pairs.
[... ti.objectIterator({a: 23, b: 42, c: [1, 2, 3]})]
partition<T>(n: number, step: number, input: Iterable<T>, all = false) => IterableIterator<T[]>
Produces iterator of fixed size partitions/chunks of input values.
Produces overlapping partitions, if step
< partition size n
.
Unless the optional all
flag is enabled, returns only completely filled
partitions. If all = true
, the last partition
produced may have less than n
items (though never empty).
[... ti.partition(3, 3, ti.range(10))]
[... ti.partition(3, 3, ti.range(10), true)]
[... ti.partition(3, 1, ti.range(10))]
[... ti.partition(3, 5, ti.range(10))]
partitionBy<T>(fn: (x: T) => any, input: Iterable<T>) => IterableIterator<T[]>
Produces iterator of partitions/chunks of input values. Applies fn
to each item
and starts new partition each time fn
returns new result.
[... ti.partitionBy((x) => x / 5 | 0, ti.range(11))]
randomSample<T>(prob: number, input: Iterable<T>) => IterableIterator<T>
Produces iterator which consumes input and yields values with given probability
(0 .. 1 range).
[... ti.randomSample(0.1, ti.range(100))]
range(from?: number, to?: number, step?: number) => IterableIterator<number>
Produces iterator of monotonically increasing or decreasing values with optional step
value.
- If called without arguments, produces values from 0 .. +∞.
- If called with 1 arg: 0 ... n (exclusive)
- If called with 2 arg:
from
... to
(exclusive)
If from
> to
and no step
is given, a step
of -1
is used.
[... ti.take(5, ti.range())]
[... ti.range(5)]
[... ti.range(100, 105)]
[... ti.range(5,0)]
[... ti.range(0, 50, 10)]
[... ti.range(50, -1, -10)]
reduce<A, B>(rfn: (acc: B, x: A) => B | ReducedValue<B>, acc: B, input: Iterable<A>) => B
Consumes and reduces input using reduction function rfn
,
then returns reduced result value. rfn
can abort reduction
process by returning a value wrapped using reduced(x)
.
If this is done, then this value is unwrapped and returned
as final result.
If input is empty, returns initial acc
umulator arg.
ti.reduce((acc, x) => acc + x, 0, ti.range(10))
ti.reduce((acc, x) => { return acc += x, acc >= 15 ? ti.reduced(acc) : acc }, 0, ti.range())
reductions<A, B>(rfn: (acc: B, x: A) => B | ReducedValue<B>, acc: B, input: Iterable<A>) => IterableIterator<B[]>
Like reduce
, but yields an iterator of all intermediate reduction results.
Always yields at least initial acc
umulator arg, even if input is empty.
Thus, the result is the equivalent of an exclusive scan operation
(with the exception of possible early bail out via reduced
).
[... ti.reductions((acc, x) => acc + x, 0, ti.range(10))]
[... ti.reductions((acc, x) => { return acc += x, acc >= 15 ? ti.reduced(acc) : acc }, 0, ti.range())]
reduced<T>(x: T) => ReducedValue<T>
For use inside reduction functions only.
Wraps result in marker type to cause early termination of reduction
(see example above).
repeat<T>(x: T, n?: number) => IterableIterator<T>
Produces an iterator of infinite (by default) repetitions of value x
.
If n
is given, produces only that many values.
[... ti.take(5, ti.repeat(42))]
[... ti.repeat(42, 5)]
repeatedly<T>(fn: () => T, n?: number) => IterableIterator<T>
Produces an iterator of infinite (by default) results of calling the no-arg fn
repeatedly.
If n
is given, produces only that many values.
[... ti.take(3, ti.repeatedly(() => Math.random()))]
[... ti.repeatedly(() => Math.random(), 3)]
reverse<T>(input: Iterable<T>) => IterableIterator<T>
Yields iterator lazily producing reverse result order of input (must be
finite). If input is not already array-like (strings are for this purpose),
the function first consumes & caches input as array.
[...ti.reverse([1, 2, 3])]
[...ti.reverse("hello")]
[...ti.reverse(ti.take(10, ti.iterate(x => x * 2, 1)))]
some<T>(pred: (x: T) => boolean, input: Iterable<T>) => T
Consumes iterator and calls pred
for each item.
When pred
returns true
, process stops and returns this first successful item.
When none of the items pass the predicate, the function returns undefined
.
var nums = ti.iterator([1, 2, 3]);
ti.some((x) => (x % 2) === 0, nums);
nums.next()
nums = ti.iterator([1, 2, 3]);
ti.some((x) => x > 3, nums);
nums.next()
take<T>(n: number, input: Iterable<T>) => IterableIterator<T>
Produces iterator of the first n
values of input
(or less than n
, if input is too short...)
[... ti.take(3, ti.range())]
takeNth<T>(n: number, input: Iterable<T>) => IterableIterator<T>
Produces an iterator only yielding every n
th item from input.
[... ti.takeNth(3, ti.range(10))]
takeWhile<T>(pred: (x: T) => boolean, input: Iterable<T>) => IterableIterator<T>
Produces iterator which calls pred
for each item of input and terminates as
soon as pred
returns false
.
Important: Due to lack of look-ahead in the ES6 iterator API, the
value failing the given pred
will be lost when working with the original
iterator after takeWhile
.
var input = ti.range(10);
[... ti.takeWhile((x)=> x < 5, input)]
[... input]
takeLast<T>(n: number, input: Iterable<T>) => IterableIterator<T>
Consumes input and produces iterator of the last n
values of input (or less
than n
, if input is too short...)
Important: Never attempt to use with infinite inputs!
[... ti.takeLast(5, ti.range(1000))]
[... ti.takeLast(5, ti.range(3))]
walk(fn: (x: any) => void, input: Iterable<any>, postOrder = false) => void
Recursively walks input and applies fn
to each element (for in-place editing
or side effects). Only iterable values and objects (but not strings) are
traversed further. Traversal is pre-order by default, but can be changed to
post-order via last arg.
let doc = {
tag: "svg",
content: [
{
tag: "g",
content: [
{ tag: "rect" },
{ tag: "circle" }
]
},
{ tag: "circle", attr: { fill: "red" } }
]
};
let circleTX = x => {
if (x.tag === "circle") {
x.attr = x.attr || {};
x.attr.x = Math.random()*100;
x.attr.y = Math.random()*100;
x.attr.r = 5;
}
};
ti.walk(circleTX, doc);
doc.content[0].content[1]
doc.content[1]
walkIterator(input: Iterable<any>, postOrder = false) => IterableIterator<any>
Yields an iterator performing either pre-order (default) or post-order
traversal of input.
Only iterable values and objects (but not strings) are traversed further.
[...ti.map(JSON.stringify, ti.walkIterator([[[1, [2]], [3, [4]]], [5]], false))]
[...ti.map(JSON.stringify, ti.walkIterator([[[1, [2]], [3, [4]]], [5]], true))]
zip(keys: Iterable<any>, vals: Iterable<any>, target?: any) => any
Takes an iterator of keys and iterator of values, pairwise combines items
from each input and associates them as key-value mappings in a given target
object (if target
is missing, returns new object).
Stops as soon as either input is exhausted.
ti.zip("abcdef", ti.range())
ti.zip(ti.range(5,10), ti.range(100,200), new Uint8Array(16))
var langs=[
{id: "js", name: "JavaScript"},
{id: "clj", name: "Clojure"},
{id: "ts", name: "TypeScript"}
];
ti.zip(ti.map((x)=> x.id, langs), langs)
Authors
License
© 2017 Karsten Schmidt // Apache Software License 2.0