What is dynamodb-wrapper?
- Enhanced AWS SDK: Public interface closely resembles the AWS SDK, making it easier to learn and use.
- Bulk I/O: Easily read, write or delete entire collections of items in DynamoDB with a single API call.
- Events: Add event hooks to be notified of important events, such as whenever read/write capacity is consumed, or requests are retried due to throttling.
- Table prefixes: DynamoDBWrapper can add a table name prefix in requests and remove it in responses. This is helpful if you have multiple environments within the same AWS Account and region.
Installing
npm install dynamodb-wrapper
Usage
Setup
Construct the DynamoDBWrapper class
var AWS = require('aws-sdk');
var DynamoDBWrapper = require('dynamodb-wrapper');
var dynamoDB = new AWS.DynamoDB({
maxRetries: 0
});
var dynamoDBWrapper = new DynamoDBWrapper(dynamoDB, {
maxRetries: 6,
retryDelayOptions: {
base: 100
}
});
(Optional) If you use DynamoDBWrapper retry logic instead of AWS retry logic, you gain the following benefits:
- Improved batch processing: DynamoDBWrapper will automatically retry any UnprocessedItems in your
batchWriteItem
requests. - You can add a
retry
event listener to be notified when requests are throttled. In your application, you can log these events, or even respond by increasing provisioned throughput on the affected table. - DynamoDBWrapper's
retryDelayOptions
actually work as documented (this functionality doesn't work in the AWS JavaScript SDK yet, but there's an open ticket for this feature request).
dynamoDBWrapper.events.on('retry', function (e) {
console.log(
'An API call to DynamoDB.' + e.method + '() acting on table ' +
e.tableName + ' was throttled. Retry attempt #' + e.retryCount +
' will occur after a delay of ' + e.retryDelayMs + 'ms.'
);
});
// An API call to DynamoDB.batchWriteItem() acting on table MyTable
// was throttled. Retry attempt #3 will occur after a delay of 800ms.
(Optional) If you use the ReturnConsumedCapacity
property in your AWS requests, the consumedCapacity
event listener can notify you whenever read/write capacity is consumed.
dynamoDBWrapper.events.on('consumedCapacity', function (e) {
console.log(
'An API call to DynamoDB.' + e.method + '() consumed ' +
e.capacityType, JSON.stringify(e.consumedCapacity, null, 2)
);
});
// An API call to DynamoDB.batchWriteItem() consumed WriteCapacityUnits
// [
// {
// "TableName": "MyTable",
// "CapacityUnits": 20
// }
// ]
(Optional) When using the DynamoDBWrapper.batchWriteItem()
API method, there is a batchGroupWritten
event that will notify you of how many items have been processed so far.
dynamoDBWrapper.events.on('batchGroupWritten', function (e) {
console.log(e.tableName, e.processedCount);
});
Example: Bulk Read
Read large collections of data from a DynamoDB table with a single API call. Multiple pages of data are aggregated and returned in a single response.
@see http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html
var sampleQueryParams = {
TableName: 'MyTable',
KeyConditionExpression: 'MyPartitionKey = :pk',
ExpressionAttributeValues: {
':pk': {
N: '1'
}
}
};
dynamoDBWrapper.query(sampleQueryParams)
.then(function (response) {
console.log(response.Items);
})
.catch(function (err) {
console.error(err);
});
Example: Bulk Write/Delete
Insert or delete large collections of items in one or more DynamoDB tables with a single API call. DynamoDBWrapper batches your requests and aggregates the results into a single response. Use configuration values to fine tune throughput consumption for your use case.
@see http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html
var sampleParams = {
RequestItems: {
MyTable: [
{
PutRequest: {
Item: {
MyPartitionKey: { N: '1' }
}
}
},
{
DeleteRequest: {
Key: {
MyPartitionKey: { N: '2' }
}
}
},
],
AnotherTable: [
{
PutRequest: {
Item: {
User: { S: 'Batman' }
}
}
},
{
DeleteRequest: {
Key: {
User: { S: 'Superman' }
}
}
},
]
}
};
dynamoDBWrapper.batchWriteItem(sampleParams, {
MyTable: {
partitionStrategy: 'EqualItemCount',
targetItemCount: 10,
groupDelayMs: 500
}
AnotherTable: {
partitionStrategy: 'EvenlyDistributedGroupWCU',
targetGroupWCU: 50,
groupDelayMs: 1000
}
})
.then(function (response) {
console.log(response);
})
.catch(function (err) {
console.error(err);
});
Example: Table Prefixes
You may wish to work with duplicate copies of the same set of tables. For example: "dev-MyTable" and "stg-MyTable" if you have dev and stage environments under the same AWS account. DynamoDBWrapper supports this use-case via a configuration-driven tableNamePrefix
option.
var environment = 'dev-';
var AWS = require('aws-sdk');
var DynamoDBWrapper = require('dynamodb-wrapper');
var dynamoDB = new AWS.DynamoDB();
var dynamoDBWrapper = new DynamoDBWrapper(dynamoDB, {
tableNamePrefix: environment
});
dynamoDBWrapper.createTable({
TableName: 'MyTable',
});
var promise = dynamoDBWrapper.getItem({
TableName: 'MyTable',
ReturnConsumedCapacity: 'TOTAL',
});
promise.then(function (response) {
console.log(response);
});
Configuration
This section is copied from the IDynamoDBWrapperOptions
interface in the index.d.ts
file.
The DynamoDBWrapper
constructor accepts an optional configuration object with the following properties:
tableNamePrefix
(string) - A prefix to add to all requests and remove from all responses.groupDelayMs
(number) - The delay (in millseconds) between individual requests made by query()
, scan()
, and batchWriteItem()
. Defaults to 100 ms.maxRetries
(number) - The maximum amount of retries to attempt with a request. Note: this property is identical to the one described in the AWS documentation.retryDelayOptions
(object) - A set of options to configure the retry delay on retryable errors. Note: this property is identical to the one described in the AWS documentation. Currently supported options are:
base
(number) - The base number of milliseconds to use in the exponential backoff for operation retries. Defaults to 100 ms.customBackoff
(Function) - A custom function that accepts a retry count and returns the amount of time to delay in milliseconds. The base
option will be ignored if this option is supplied.
API
The DynamoDBWrapper
class supports a Promise-based API with the following methods. These are wrappers around the AWS SDK method of the same name. Please refer to the AWS API documentation and JavaScript SDK documentation for more details:
The following methods are passed straight through to the AWS JavaScript SDK:
createTable(params)
updateTable(params)
describeTable(params)
deleteTable(params)
getItem(params)
updateItem(params)
putItem(params)
deleteItem(params)
batchGetItem(params)
Enhanced API methods
The following API methods have enhanced behavior to support bulk I/O:
query(params, options)
- Fetches all pages of data from a DynamoDB query, making multiple requests and aggregating responses when necessary.
options.groupDelayMs
(number) - the delay between individual requests. Overrides the configuration property of the same name in the constructor. Defaults to 100 ms.
scan(params, options)
- Fetches all pages of data from a DynamoDB scan, making multiple requests and aggregating responses when necessary.
options.groupDelayMs
(number) - the delay between individual requests. Overrides the configuration property of the same name in the constructor. Defaults to 100 ms.
batchWriteItem(params, options)
- Writes or deletes large collections of items in multiple DynamoDB tables, batching items and making multiple requests when necessary.
options
is a mapping of table names to option hashes. Each option hash may have the following properties:
groupDelayMs
(number) - the delay between individual requests. Overrides the configuration property of the same name in the constructor. Defaults to 100 ms.partitionStrategy
(string) - strategy to use when partitioning the write requests array. Possible values: EqualItemCount or EvenlyDistributedGroupWCU.targetItemCount
(number) - the number of items to put in each group when using the EqualItemCount partition strategy.targetGroupWCU
(number) - the size threshold (in WriteCapacityUnits) of each group when using the EvenlyDistributedGroupWCU partition strategy.
Roadmap
- Streams: Add method signatures that return Streams (instead of Promises), allowing for better integration ecosystems such as gulp