@supercharge/promise-pool

What is @supercharge/promise-pool?

@supercharge/promise-pool is an npm package that allows you to process a large number of promises concurrently with a specified limit on the number of promises that can run at the same time. This helps in managing resources efficiently and avoiding overwhelming the system.

What are @supercharge/promise-pool's main functionalities?

Concurrent Processing

This feature allows you to process a list of items concurrently with a specified limit on the number of concurrent operations. In this example, the concurrency limit is set to 5.

const { PromisePool } = require('@supercharge/promise-pool');

async function processItems(items) {
  const { results, errors } = await PromisePool
    .for(items)
    .withConcurrency(5)
    .process(async item => {
      // Process each item
      return await processItem(item);
    });

  console.log('Results:', results);
  console.log('Errors:', errors);
}

async function processItem(item) {
  // Simulate async processing
  return new Promise(resolve => setTimeout(() => resolve(item), 1000));
}

processItems([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);

Error Handling

This feature demonstrates how to handle errors that occur during the processing of items. Errors are collected and can be reviewed after the processing is complete.

const { PromisePool } = require('@supercharge/promise-pool');

async function processItems(items) {
  const { results, errors } = await PromisePool
    .for(items)
    .withConcurrency(5)
    .process(async item => {
      if (item % 2 === 0) {
        throw new Error('Even number error');
      }
      return await processItem(item);
    });

  console.log('Results:', results);
  console.log('Errors:', errors);
}

async function processItem(item) {
  // Simulate async processing
  return new Promise(resolve => setTimeout(() => resolve(item), 1000));
}

processItems([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);

Dynamic Concurrency

This feature allows you to dynamically set the concurrency level based on certain conditions. In this example, the concurrency is set to 10 if the number of items is greater than 5, otherwise, it is set to 2.

const { PromisePool } = require('@supercharge/promise-pool');

async function processItems(items) {
  const concurrency = items.length > 5 ? 10 : 2;
  const { results, errors } = await PromisePool
    .for(items)
    .withConcurrency(concurrency)
    .process(async item => {
      return await processItem(item);
    });

  console.log('Results:', results);
  console.log('Errors:', errors);
}

async function processItem(item) {
  // Simulate async processing
  return new Promise(resolve => setTimeout(() => resolve(item), 1000));
}

processItems([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);

Other packages similar to @supercharge/promise-pool

p-limit

p-limit is a package that allows you to run multiple promise-returning & async functions with a concurrency limit. It is simpler and more lightweight compared to @supercharge/promise-pool, but it lacks some of the advanced features like error collection and dynamic concurrency.

async

async is a utility module which provides straight-forward, powerful functions for working with asynchronous JavaScript. It includes a variety of methods for managing concurrency, such as `async.eachLimit` and `async.queue`. It is more feature-rich and versatile compared to @supercharge/promise-pool, but it can be more complex to use.

bluebird

bluebird is a fully featured promise library with focus on innovative features and performance. It includes methods like `Promise.map` which can be used to limit concurrency. It is more comprehensive and offers more features beyond just concurrency control compared to @supercharge/promise-pool.

README

Promise Pool

Map-like, concurrent promise processing for Node.js.

Installation · Docs · Usage

Follow @marcuspoehls and @superchargejs for updates!

---

Installation

npm i @supercharge/promise-pool

Docs

Find all the details and available methods in the extensive Supercharge docs.

Usage

Using the promise pool is pretty straightforward. The package exposes a class and you can create a promise pool instance using the fluent interface.

Here’s an example using a concurrency of 2:

const PromisePool = require('@supercharge/promise-pool')

const users = [
  { name: 'Marcus' },
  { name: 'Norman' },
  { name: 'Christian' }
]

const { results, errors } = await PromisePool
  .withConcurrency(2)
  .for(users)
  .process(async data => {
    const user = await User.createIfNotExisting(data)

    return user
  })

The promise pool uses a default concurrency of 10:

await PromisePool
  .for(users)
  .process(async data => {
    // processes 10 items in parallel by default
  })

Contributing

1. Create a fork
2. Create your feature branch: git checkout -b my-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request 🚀

License

MIT © Supercharge

---

superchargejs.com  ·  GitHub @superchargejs  ·  Twitter @superchargejs

Changelog

1.4.0 - 2020-09-17

Added

• improved types supporting typed return values
• improved error handling when rejecting a promise without an error instance (thank you wzh)

Updated

• bump dependencies
• change main entrypoint in package.json to dist folder
• move test runner from @hapi/lab to jest
• move assertions from @hapi/code to jest