@shelf/dynamodb-parallel-scan

dynamodb-parallel-scan

Scan DynamoDB table concurrently (up to 1,000,000 segments), recursively read all items from every segment

Install

$ yarn add @shelf/dynamodb-parallel-scan

This library has 2 peer dependencies:

• @aws-sdk/client-dynamodb
• @aws-sdk/lib-dynamodb

Make sure to install them alongside this library.

Usage

Fetch everything at once

const {parallelScan} = require('@shelf/dynamodb-parallel-scan');

(async () => {
  const items = await parallelScan(
    {
      TableName: 'files',
      FilterExpression: 'attribute_exists(#fileSize)',
      ExpressionAttributeNames: {
        '#fileSize': 'fileSize',
      },
      ProjectionExpression: 'fileSize',
    },
    {concurrency: 1000}
  );

  console.log(items);
})();

Use as async generator (or streams)

Note: highWaterMark determines items count threshold, so Parallel Scan can fetch concurrency * 1MB more data even after highWaterMark was reached.

const {parallelScanAsStream} = require('@shelf/dynamodb-parallel-scan');

(async () => {
  const stream = await parallelScanAsStream(
    {
      TableName: 'files',
      FilterExpression: 'attribute_exists(#fileSize)',
      ExpressionAttributeNames: {
        '#fileSize': 'fileSize',
      },
      ProjectionExpression: 'fileSize',
    },
    {concurrency: 1000, chunkSize: 10000, highWaterMark: 10000}
  );

  for await (const items of stream) {
    console.log(items); // 10k items here
  }
})();

Read

• Taking Advantage of Parallel Scans
• Working with Scans

Publish

$ git checkout master
$ yarn version
$ yarn publish
$ git push origin master

License

MIT © Shelf