async-iterable-limiter

async-iterable-limiter

Iterate over async iterable with concurrency limit.

Difference from many other similar modules is that it supports iterating over async iterables (like ReadableStream or database cursor).

Note: documentation and tests are work in progress, but it should already work :)

Usage

Simple:

const Limiter = require('async-iterable-limiter');

const cursor = db.collection().find(); // todo: better example

await Limiter.iterate(cursor, 5, async (item) => {
	await doWork(item);
});

Complex (internal implementation of iterate function):

const Limiter = require('async-iterable-limiter');

const limiter = new Limiter({ concurrency: 5 });
for await (const item of iterable) {
	await limiter.ready();
	limiter.add(taskReturningPromise(item));
}
await limiter.finished();

TODO:

Functionality:

• add stopOnError parameter (true by default in current implementation)
• maybe add Limiter.map function that would produce an array with sequential results
• maybe add Limiter.mapIterable function that would produce another async iterable
• make inProgress and concurrency readonly properties, to prevent mess

Tests:

• add test for error handling of async iterable
• add test for error handling of task
• add test for invalid input of params
• add test for expected behavior of multiple calls of Limiter.ready/finished class

Documentation:

• readme
• api surface documentation
• typescript definitions