@supercharge/promise-pool
Advanced tools
@supercharge/promise-pool is an npm package that allows you to process a large number of promises concurrently with a specified limit on the number of promises that can run at the same time. This helps in managing resources efficiently and avoiding overwhelming the system.
Concurrent Processing
This feature allows you to process a list of items concurrently with a specified limit on the number of concurrent operations. In this example, the concurrency limit is set to 5.
const { PromisePool } = require('@supercharge/promise-pool');
async function processItems(items) {
const { results, errors } = await PromisePool
.for(items)
.withConcurrency(5)
.process(async item => {
// Process each item
return await processItem(item);
});
console.log('Results:', results);
console.log('Errors:', errors);
}
async function processItem(item) {
// Simulate async processing
return new Promise(resolve => setTimeout(() => resolve(item), 1000));
}
processItems([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);Error Handling
This feature demonstrates how to handle errors that occur during the processing of items. Errors are collected and can be reviewed after the processing is complete.
const { PromisePool } = require('@supercharge/promise-pool');
async function processItems(items) {
const { results, errors } = await PromisePool
.for(items)
.withConcurrency(5)
.process(async item => {
if (item % 2 === 0) {
throw new Error('Even number error');
}
return await processItem(item);
});
console.log('Results:', results);
console.log('Errors:', errors);
}
async function processItem(item) {
// Simulate async processing
return new Promise(resolve => setTimeout(() => resolve(item), 1000));
}
processItems([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);Dynamic Concurrency
This feature allows you to dynamically set the concurrency level based on certain conditions. In this example, the concurrency is set to 10 if the number of items is greater than 5, otherwise, it is set to 2.
const { PromisePool } = require('@supercharge/promise-pool');
async function processItems(items) {
const concurrency = items.length > 5 ? 10 : 2;
const { results, errors } = await PromisePool
.for(items)
.withConcurrency(concurrency)
.process(async item => {
return await processItem(item);
});
console.log('Results:', results);
console.log('Errors:', errors);
}
async function processItem(item) {
// Simulate async processing
return new Promise(resolve => setTimeout(() => resolve(item), 1000));
}
processItems([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);p-limit is a package that allows you to run multiple promise-returning & async functions with a concurrency limit. It is simpler and more lightweight compared to @supercharge/promise-pool, but it lacks some of the advanced features like error collection and dynamic concurrency.
async is a utility module which provides straight-forward, powerful functions for working with asynchronous JavaScript. It includes a variety of methods for managing concurrency, such as `async.eachLimit` and `async.queue`. It is more feature-rich and versatile compared to @supercharge/promise-pool, but it can be more complex to use.
bluebird is a fully featured promise library with focus on innovative features and performance. It includes methods like `Promise.map` which can be used to limit concurrency. It is more comprehensive and offers more features beyond just concurrency control compared to @supercharge/promise-pool.
Map-like, concurrent promise processing for Node.js.
Installation · Docs · Usage
Follow @marcuspoehls and @superchargejs for updates!
npm i @supercharge/promise-pool
Using the promise pool is pretty straightforward. The package exposes a class and you can create a promise pool instance using the fluent interface.
Here’s an example using a concurrency of 2:
import { PromisePool } from '@supercharge/promise-pool'
const users = [
{ name: 'Marcus' },
{ name: 'Norman' },
{ name: 'Christian' }
]
const { results, errors } = await PromisePool
.withConcurrency(2)
.for(users)
.process(async (userData, index, pool) => {
const user = await User.createIfNotExisting(userData)
return user
})
The promise pool uses a default concurrency of 10:
await PromisePool
.for(users)
.process(async data => {
// processes 10 items in parallel by default
})
You can stop the processing of a promise pool using the pool instance provided to the .process() and .handleError() methods. Here’s an example how you can stop an active promise pool from within the .process() method:
await PromisePool
.for(users)
.process(async (user, index, pool) => {
if (condition) {
return pool.stop()
}
// processes the `user` data
})
You may also stop the pool from within the .handleError() method in case you need to:
import { PromisePool } from '@supercharge/promise-pool'
await PromisePool
.for(users)
.handleError(async (error, user, pool) => {
if (error instanceof SomethingBadHappenedError) {
return pool.stop()
}
// handle the given `error`
})
.process(async (user, index, pool) => {
// processes the `user` data
})
The promise pool allows for custom error handling. You can take over the error handling by implementing an error handler using the .handleError(handler).
If you provide an error handler, the promise pool doesn’t collect any errors. You must then collect errors yourself.
Providing a custom error handler allows you to exit the promise pool early by throwing inside the error handler function. Throwing errors is in line with Node.js error handling using async/await.
import { PromisePool } from '@supercharge/promise-pool'
try {
const errors = []
const { results } = await PromisePool
.for(users)
.withConcurrency(4)
.handleError(async (error, user) => {
if (error instanceof ValidationError) {
errors.push(error) // you must collect errors yourself
return
}
if (error instanceof ThrottleError) { // Execute error handling on specific errors
await retryUser(user)
return
}
throw error // Uncaught errors will immediately stop PromisePool
})
.process(async data => {
// the harder you work for something,
// the greater you’ll feel when you achieve it
})
await handleCollected(errors) // this may throw
return { results }
} catch (error) {
await handleThrown(error)
}
You can use the onTaskStarted and onTaskFinished methods to hook into the processing of tasks. The provided callback for each method will be called when a task started/finished processing:
import { PromisePool } from '@supercharge/promise-pool'
await PromisePool
.for(users)
.onTaskStarted((item, pool) => {
console.log(`Progress: ${pool.processedPercentage()}%`)
console.log(`Active tasks: ${pool.processedItems().length}`)
console.log(`Active tasks: ${pool.activeTasksCount()}`)
console.log(`Finished tasks: ${pool.processedItems().length}`)
console.log(`Finished tasks: ${pool.processedCount()}`)
})
.onTaskFinished((item, pool) => {
// update a progress bar or something else :)
})
.process(async (user, index, pool) => {
// processes the `user` data
})
You can also chain multiple onTaskStarted and onTaskFinished handling (in case you want to separate some functionality):
import { PromisePool } from '@supercharge/promise-pool'
await PromisePool
.for(users)
.onTaskStarted(() => {})
.onTaskStarted(() => {})
.onTaskFinished(() => {})
.onTaskFinished(() => {})
.process(async (user, index, pool) => {
// processes the `user` data
})
Sometimes it’s useful to configure a timeout in which a task must finish processing. A task that times out is marked as failed. You may use the withTaskTimeout(<milliseconds>) method to configure a task’s timeout:
import { PromisePool } from '@supercharge/promise-pool'
await PromisePool
.for(users)
.withTaskTimeout(2000) // milliseconds
.process(async (user, index, pool) => {
// processes the `user` data
})
Notice: a configured timeout is configured for each task, not for the whole pool. The example configures a 2-second timeout for each task in the pool.
Sometimes you want the processed results to align with your source items. The resulting items should have the same position in the results array as their related source items. Use the useCorrespondingResults method to apply this behavior:
import { setTimeout } from 'node:timers/promises'
import { PromisePool } from '@supercharge/promise-pool'
const { results } = await PromisePool
.for([1, 2, 3])
.withConcurrency(5)
.useCorrespondingResults()
.process(async (number, index) => {
const value = number * 2
return await setTimeout(10 - index, value)
})
/**
* source array: [1, 2, 3]
* result array: [2, 4 ,6]
* --> result values match the position of their source items
*/
For example, you may have three items you want to process. Using corresponding results ensures that the processed result for the first item from the source array is located at the first position in the result array (=index 0). The result for the second item from the source array is placed at the second position in the result array, and so on …
The results array returned by the promise pool after processing has a mixed return type. Each returned item is one of this type:
Symbol('notRun'): for tasks that didn’t runSymbol('failed'): for tasks that failed processingThe PromisePool exposes both symbols and you may access them using
Symbol('notRun'): exposed as PromisePool.notRunSymbol('failed'): exposed as PromisePool.failedYou may repeat processing for all tasks that didn’t run or failed:
import { PromisePool } from '@supercharge/promise-pool'
const { results, errors } = await PromisePool
.for([1, 2, 3])
.withConcurrency(5)
.useCorrespondingResults()
.process(async (number) => {
// …
})
const itemsNotRun = results.filter(result => {
return result === PromisePool.notRun
})
const failedItems = results.filter(result => {
return result === PromisePool.failed
})
When using corresponding results, you need to go through the errors array yourself. The default error handling (collect errors) stays the same and you can follow the described error handling section above.
git checkout -b my-featuregit commit -am 'Add some feature'git push origin my-new-featureMIT © Supercharge
superchargejs.com · GitHub @supercharge · Twitter @superchargejs
FAQs
Map-like, concurrent promise processing for Node.js
The npm package @supercharge/promise-pool receives a total of 394,025 weekly downloads. As such, @supercharge/promise-pool popularity was classified as popular.
We found that @supercharge/promise-pool demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 3 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Malicious Go packages are impersonating popular libraries to install hidden loader malware on Linux and macOS, targeting developers with obfuscated payloads.
Security News
Bybit's $1.46B hack by North Korea's Lazarus Group pushes 2025 crypto losses to $1.6B in just two months, already surpassing all of 2024's $1.49B total.
Security News
OpenSSF has published OSPS Baseline, an initiative designed to establish a minimum set of security-related best practices for open source software projects.