Security News
Cloudflare Adds Security.txt Setup Wizard
Cloudflare has launched a setup wizard allowing users to easily create and manage a security.txt file for vulnerability disclosure on their websites.
@shelf/dynamodb-parallel-scan
Advanced tools
Scan DynamoDB table concurrently (up to 1,000,000 segments), recursively read all items from every segment
A blog post going into details about this library.
$ yarn add @shelf/dynamodb-parallel-scan
This library has 2 peer dependencies:
@aws-sdk/client-dynamodb
@aws-sdk/lib-dynamodb
Make sure to install them alongside this library.
Easily parallelize scan requests to fetch all items from a table at once. This is useful when you need to scan a large table to find a small number of items that will fit the node.js memory.
Scan huge tables using async generator or stream. And yes, it supports streams backpressure! Useful when you need to process a large number of items while you scan them. It allows receiving chunks of scanned items, wait until you process them, and then resume scanning when you're ready.
const {parallelScan} = require('@shelf/dynamodb-parallel-scan');
(async () => {
const items = await parallelScan(
{
TableName: 'files',
FilterExpression: 'attribute_exists(#fileSize)',
ExpressionAttributeNames: {
'#fileSize': 'fileSize',
},
ProjectionExpression: 'fileSize',
},
{concurrency: 1000}
);
console.log(items);
})();
Note: highWaterMark
determines items count threshold, so Parallel Scan can fetch concurrency
* 1MB more data even after highWaterMark was reached.
const {parallelScanAsStream} = require('@shelf/dynamodb-parallel-scan');
(async () => {
const stream = await parallelScanAsStream(
{
TableName: 'files',
FilterExpression: 'attribute_exists(#fileSize)',
ExpressionAttributeNames: {
'#fileSize': 'fileSize',
},
ProjectionExpression: 'fileSize',
},
{concurrency: 1000, chunkSize: 10000, highWaterMark: 10000}
);
for await (const items of stream) {
console.log(items); // 10k items here
}
})();
$ git checkout master
$ yarn version
$ yarn publish
$ git push origin master --tags
MIT © Shelf
FAQs
Scan large DynamoDB tables faster with parallelism
The npm package @shelf/dynamodb-parallel-scan receives a total of 8,308 weekly downloads. As such, @shelf/dynamodb-parallel-scan popularity was classified as popular.
We found that @shelf/dynamodb-parallel-scan demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Cloudflare has launched a setup wizard allowing users to easily create and manage a security.txt file for vulnerability disclosure on their websites.
Security News
The Socket Research team breaks down a malicious npm package targeting the legitimate DOMPurify library. It uses obfuscated code to hide that it is exfiltrating browser and crypto wallet data.
Security News
ENISA’s 2024 report highlights the EU’s top cybersecurity threats, including rising DDoS attacks, ransomware, supply chain vulnerabilities, and weaponized AI.