What is @supercharge/promise-pool?
@supercharge/promise-pool is an npm package that allows you to process a large number of promises concurrently with a specified limit on the number of promises that can run at the same time. This helps in managing resources efficiently and avoiding overwhelming the system.
What are @supercharge/promise-pool's main functionalities?
Concurrent Processing
This feature allows you to process a list of items concurrently with a specified limit on the number of concurrent operations. In this example, the concurrency limit is set to 5.
const { PromisePool } = require('@supercharge/promise-pool');
async function processItems(items) {
const { results, errors } = await PromisePool
.for(items)
.withConcurrency(5)
.process(async item => {
// Process each item
return await processItem(item);
});
console.log('Results:', results);
console.log('Errors:', errors);
}
async function processItem(item) {
// Simulate async processing
return new Promise(resolve => setTimeout(() => resolve(item), 1000));
}
processItems([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
Error Handling
This feature demonstrates how to handle errors that occur during the processing of items. Errors are collected and can be reviewed after the processing is complete.
const { PromisePool } = require('@supercharge/promise-pool');
async function processItems(items) {
const { results, errors } = await PromisePool
.for(items)
.withConcurrency(5)
.process(async item => {
if (item % 2 === 0) {
throw new Error('Even number error');
}
return await processItem(item);
});
console.log('Results:', results);
console.log('Errors:', errors);
}
async function processItem(item) {
// Simulate async processing
return new Promise(resolve => setTimeout(() => resolve(item), 1000));
}
processItems([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
Dynamic Concurrency
This feature allows you to dynamically set the concurrency level based on certain conditions. In this example, the concurrency is set to 10 if the number of items is greater than 5, otherwise, it is set to 2.
const { PromisePool } = require('@supercharge/promise-pool');
async function processItems(items) {
const concurrency = items.length > 5 ? 10 : 2;
const { results, errors } = await PromisePool
.for(items)
.withConcurrency(concurrency)
.process(async item => {
return await processItem(item);
});
console.log('Results:', results);
console.log('Errors:', errors);
}
async function processItem(item) {
// Simulate async processing
return new Promise(resolve => setTimeout(() => resolve(item), 1000));
}
processItems([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
Other packages similar to @supercharge/promise-pool
p-limit
p-limit is a package that allows you to run multiple promise-returning & async functions with a concurrency limit. It is simpler and more lightweight compared to @supercharge/promise-pool, but it lacks some of the advanced features like error collection and dynamic concurrency.
async
async is a utility module which provides straight-forward, powerful functions for working with asynchronous JavaScript. It includes a variety of methods for managing concurrency, such as `async.eachLimit` and `async.queue`. It is more feature-rich and versatile compared to @supercharge/promise-pool, but it can be more complex to use.
bluebird
bluebird is a fully featured promise library with focus on innovative features and performance. It includes methods like `Promise.map` which can be used to limit concurrency. It is more comprehensive and offers more features beyond just concurrency control compared to @supercharge/promise-pool.
Installation
npm i @supercharge/promise-pool
Usage
Using the promise pool is pretty straightforward. The pacakge exposes a class and you can create a promise pool instance using the fluent interface. Here’s a working example:
const PromisePool = require('@supercharge/promise-pool')
const users = [
{ name: 'Marcus' },
{ name: 'Norman' },
{ name: 'Christian' }
]
const { results, errors } = await new PromisePool()
.for(users)
.withConcurrency(2)
.process(async data => {
const user = await User.createIfNotExisting(data)
return user
})
API
new PromisePool({ concurrency?, items? })
constructor
Creates a new promise pool. The constructor takes an optional options
object allowing you to passing in the concurrency
and items
.
const pool = new PromisePool({ concurrency: 2, items: [1, 2, 3] })
.withConcurrency(amount)
Set the maximum number of functions to process in parallel. Default concurrency: 10
. Returns the promise pool instance.
const pool = new PromisePool().withConcurrency(5)
.for(items)
Set the items to be processed in the promise pool. Returns the promise pool instance.
const users = [
{ name: 'Marcus' },
{ name: 'Norman' },
{ name: 'Christian' }
]
const pool = new PromisePool().withConcurrency(5).for(users)
.process(callback)
Starts processing the promise pool by iterating over the items and passing each item to the async mapper function. Returns an object containing the results and errors.
const users = [
{ name: 'Marcus' },
{ name: 'Norman' },
{ name: 'Christian' }
]
const pool = new PromisePool().withConcurrency(5).for(users)
const { results, errors } = await pool.process(async (user) => {
await User.createIfNotExisting(user)
})
Contributing
- Create a fork
- Create your feature branch:
git checkout -b my-feature
- Commit your changes:
git commit -am 'Add some feature'
- Push to the branch:
git push origin my-new-feature
- Submit a pull request 🚀
License
MIT © Supercharge
superchargejs.com ·
GitHub @superchargejs ·
Twitter @superchargejs