@cumulus/aws-client

@cumulus/aws-client

Utilities for working with AWS. These utilities can be used for interacting with live AWS services or Localstack. For ease of setup, testing, and credential management, code interacting with AWS services should use the helpers in this module.

⚠️ The documented API of this package will not change without a deprecation warning being provided in earlier releases. Code in this package that is not documented in this README may change without warning, and is not considered part of the package's public API.

Usage

npm install @cumulus/aws-client

Interacting with Localstack

To use these utilities with Localstack, make sure you have a running instance of Localstack and set this environment variable:

NODE_ENV=test

API

Modules

CloudFormation

CloudwatchEvents

DynamoDb

Kinesis

Lambda

S3

SNS

SQS

STS

SecretsManager

StepFunctions

Classes

DynamoDbSearchQueue

Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.

S3ListObjectsV2Queue

Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.

S3ObjectStore

Class to use when interacting with S3

Functions

createKey(params) ⇒ Promise.<Object>

Create a KMS key

See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/KMS.html#createKey-property for allowed params and return value.

encrypt(KeyId, Plaintext) ⇒ Promise.<string>

Encrypt a string using KMS

decryptBase64String(ciphertext) ⇒ string

Decrypt a KMS-encrypted string, Base 64 encoded

CloudFormation

• CloudFormation
  • ~describeCfStack(StackName) ⇒ Promise.<CloudFormation.Stack>
  • ~describeCfStackResources(StackName) ⇒ Promise.<CloudFormation.StackResources>
  • ~getCfStackParameterValues(stackName, parameterKeys) ⇒ Promise.<Object>

CloudFormation~describeCfStack(StackName) ⇒ Promise.<CloudFormation.Stack>

Describes a given CloudFormation stack

See CloudFormation.Stack

Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.Stack> - The resources belonging to the stack

Param	Type	Description
StackName	string	The name of the CloudFormation stack to query

CloudFormation~describeCfStackResources(StackName) ⇒ Promise.<CloudFormation.StackResources>

Describes the resources belonging to a given CloudFormation stack

See CloudFormation.StackResources

Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.StackResources> - The resources belonging to the stack

Param	Type	Description
StackName	string	The name of the CloudFormation stack to query

CloudFormation~getCfStackParameterValues(stackName, parameterKeys) ⇒ Promise.<Object>

Get parameter values for the given CloudFormation stack

Kind: inner method of CloudFormation
Returns: Promise.<Object> - Object keyed by parameter names

Param	Type	Description
stackName	string	The name of the CloudFormation stack to query
parameterKeys	Array.<string>	Key names for the stack parameters that you want to return

CloudwatchEvents

CloudwatchEvents~putEvent(name, schedule, state, [description], [role]) ⇒ Promise.<CloudWatchEvents.PutRuleResponse>

Create a CloudWatch Events rule

Kind: inner method of CloudwatchEvents

Param	Type	Description
name	string	the rule name
schedule	string	a ScheduleExpression
state	string	the state of the rule
[description]	string
[role]	string	a Role ARN

DynamoDb

• DynamoDb
  • static
    • .scan ⇒ Promise.<Object>
    • .createAndWaitForDynamoDbTable(params) ⇒ Promise.<Object>
    • .deleteAndWaitForDynamoDbTableNotExists(params) ⇒ Promise
  • inner
    • ~get(params) ⇒ Promise.<Object>
    • ~parallelScan(params) ⇒ Promise

DynamoDb.scan ⇒ Promise.<Object>

Call DynamoDb client scan

See DocumentClient.scan() for descriptions of params and the return data.

Kind: static property of DynamoDb

Param	Type
params	Object

DynamoDb.createAndWaitForDynamoDbTable(params) ⇒ Promise.<Object>

Create a DynamoDB table and then wait for the table to exist

Kind: static method of DynamoDb
Returns: Promise.<Object> - the output of the createTable call

Param	Type	Description
params	Object	the same params that you would pass to AWS.createTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/classes/dynamodb.html#createtable

DynamoDb.deleteAndWaitForDynamoDbTableNotExists(params) ⇒ Promise

Delete a DynamoDB table and then wait for the table to not exist

Kind: static method of DynamoDb

Param	Type	Description
params	Object	the same params that you would pass to AWS.deleteTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/classes/dynamodb.html#deletetable

DynamoDb~get(params) ⇒ Promise.<Object>

Call DynamoDb client get

See DocumentClient.get() for descriptions of params and the return data.

Kind: inner method of DynamoDb
Throws:

• RecordDoesNotExist if a record cannot be found

Param	Type	Description
params	Object
params.tableName	string	Table name to read
params.item	GetCommandInput.Key	Key identifying object to get
params.client	DynamoDBDocument	Instance of a DynamoDb DocumentClient
params.getParams	Object	Additional parameters for DocumentClient.get()

DynamoDb~parallelScan(params) ⇒ Promise

Do a parallel scan of DynamoDB table using a document client.

See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.ParallelScan. See DocumentClient.scan().

Kind: inner method of DynamoDb

Param	Type	Description
params	Object
params.totalSegments	number	Total number of segments to divide table into for parallel scanning
params.scanParams	ScanInput	Params for the DynamoDB client scan operation See https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html
params.processItemsFunc	function	Function used to process returned items by scan
[params.dynamoDbClient]	DynamoDBDocument	Instance of Dynamo DB document client
[params.retryOptions]	pRetry.Options	Retry options for scan operations

Kinesis

Kinesis~describeStream(params, retryOptions) ⇒ Promise.<Object>

Describe a Kinesis stream.

Kind: inner method of Kinesis
Returns: Promise.<Object> - The stream description response

Param	Type	Description
params	Object
params.StreamName	string	A Kinesis stream name
retryOptions	Object	Options passed to p-retry module

Lambda

• Lambda
  • ~isEventBridgeEvent()
  • ~invoke()

Lambda~isEventBridgeEvent()

Bare check for EventBridge shape

Kind: inner method of Lambda

Lambda~invoke()

Invoke a Lambda function

Kind: inner method of Lambda

S3

• S3
  • static
    • .getS3Object ⇒ Promise
    • .recursivelyDeleteS3Bucket ⇒
    • .listS3ObjectsV2(params) ⇒ Promise.<Array>
    • .listS3ObjectsV2Batch(params)
  • inner
    • ~s3Join(...args) ⇒ string
    • ~parseS3Uri(uri) ⇒ Object
    • ~buildS3Uri(bucket, key) ⇒ string
    • ~s3TagSetToQueryString(tagset) ⇒ string
    • ~deleteS3Object(bucket, key) ⇒ Promise
    • ~headObject(Bucket, Key, retryOptions) ⇒
    • ~s3ObjectExists(params) ⇒ Promise.<boolean>
    • ~waitForObjectToExist(params) ⇒ Promise.<undefined>
    • ~s3PutObject(params) ⇒ Promise
    • ~putFile(bucket, key, filename) ⇒ Promise
    • ~s3CopyObject(params) ⇒ Promise
    • ~promiseS3Upload(params) ⇒ Promise
    • ~streamS3Upload(uploadStream, uploadParams) ⇒ Promise
    • ~getObjectReadStream(params) ⇒ Promise.<Readable>
    • ~downloadS3File(s3Obj, filepath) ⇒ Promise.<string>
    • ~getObjectSize(params) ⇒ Promise.<(number|undefined)>
    • ~s3GetObjectTagging(bucket, key) ⇒ Promise.<GetObjectTaggingOutput>
    • ~s3PutObjectTagging(Bucket, Key, ObjectTagging) ⇒ Promise
    • ~getObject(s3Client, params) ⇒ Promise.<GetObjectOutput>
    • ~waitForObject(s3Client, params, [retryOptions]) ⇒ Promise.<GetObjectOutput>
    • ~getObjectStreamContents(objectReadStream) ⇒ Promise.<string>
    • ~getTextObject(bucket, key) ⇒ Promise.<string>
    • ~getJsonS3Object(bucket, key) ⇒ Promise.<*>
    • ~fileExists(bucket, key) ⇒ Promise
    • ~deleteS3Files(s3Objs) ⇒ Promise
    • ~uploadS3FileStream(fileStream, bucket, key, s3opts) ⇒ Promise
    • ~listS3Objects()
    • ~deleteS3Buckets(buckets) ⇒ Promise
    • ~calculateObjectHash(params)
    • ~validateS3ObjectChecksum(params) ⇒ Promise.<boolean>
    • ~getFileBucketAndKey(pathParams) ⇒ Array.<string>
    • ~createBucket(Bucket) ⇒ Promise
    • ~createS3Buckets(buckets) ⇒ Promise
    • ~multipartCopyObject(params) ⇒ Promise.<{etag: string}>
    • ~moveObject(params) ⇒ Promise.<undefined>

S3.getS3Object ⇒ Promise

Deprecated

Gets an object from S3.

Kind: static property of S3
Returns: Promise - returns response from S3.getObject as a promise

Param	Type	Description
Bucket	string	name of bucket
Key	string	key for object (filepath + filename)
retryOptions	Object	options to control retry behavior when an object does not exist. See https://github.com/tim-kos/node-retry#retryoperationoptions By default, retries will not be performed

S3.recursivelyDeleteS3Bucket ⇒

Delete a bucket and all of its objects from S3

Kind: static property of S3
Returns: the promised result of S3.deleteBucket

Param	Description
bucket	name of the bucket

S3.listS3ObjectsV2(params) ⇒ Promise.<Array>

Fetch complete list of S3 objects

listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.

The passed params must be compatible with the listObjectsV2 call.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property

Kind: static method of S3
Returns: Promise.<Array> - resolves to an array of objects corresponding to the Contents property of the listObjectsV2 response

Param	Type	Description
params	Object	params for the s3.listObjectsV2 call

S3.listS3ObjectsV2Batch(params)

Fetch lazy list of S3 objects

listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.

The passed params must be compatible with the listObjectsV2 call.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property

Kind: static method of S3

Param	Description
params	params for the s3.listObjectsV2 call

S3~s3Join(...args) ⇒ string

Join strings into an S3 key without a leading slash

Kind: inner method of S3
Returns: string - the full S3 key

Param	Type	Description
...args	string | Array.<string>	the strings to join

S3~parseS3Uri(uri) ⇒ Object

parse an s3 uri to get the bucket and key

Kind: inner method of S3
Returns: Object - Returns an object with Bucket and Key properties

Param	Type	Description
uri	string	must be a uri with the s3:// protocol

S3~buildS3Uri(bucket, key) ⇒ string

Given a bucket and key, return an S3 URI

Kind: inner method of S3
Returns: string - an S3 URI

Param	Type	Description
bucket	string	an S3 bucket name
key	string	an S3 key

S3~s3TagSetToQueryString(tagset) ⇒ string

Convert S3 TagSet Object to query string e.g. [{ Key: 'tag', Value: 'value }] to 'tag=value'

Kind: inner method of S3
Returns: string - tags query string

Param	Type	Description
tagset	Array.<Object>	S3 TagSet array

S3~deleteS3Object(bucket, key) ⇒ Promise

Delete an object from S3

Kind: inner method of S3
Returns: Promise - promise of the object being deleted

Param	Type	Description
bucket	string	bucket where the object exists
key	string	key of the object to be deleted

S3~headObject(Bucket, Key, retryOptions) ⇒

Get an object header from S3

Kind: inner method of S3
Returns: returns response from S3.headObject as a promise

Param	Description
Bucket	name of bucket
Key	key for object (filepath + filename)
retryOptions	options to control retry behavior when an object does not exist. See https://github.com/tim-kos/node-retry#retryoperationoptions By default, retries will not be performed

S3~s3ObjectExists(params) ⇒ Promise.<boolean>

Test if an object exists in S3

Kind: inner method of S3
Returns: Promise.<boolean> - a Promise that will resolve to a boolean indicating if the object exists

Param	Type	Description
params	Object	same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property

S3~waitForObjectToExist(params) ⇒ Promise.<undefined>

Wait for an object to exist in S3

Kind: inner method of S3

Param	Type	Default	Description
params	Object
params.bucket	string
params.key	string
[params.interval]	number	1000	interval before retries, in ms
[params.timeout]	number	30000	timeout, in ms

S3~s3PutObject(params) ⇒ Promise

Put an object on S3

Kind: inner method of S3

Param	Type	Description
params	Object	same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property promise of the object being put

S3~putFile(bucket, key, filename) ⇒ Promise

Upload a file to S3

Kind: inner method of S3

Param	Type	Description
bucket	string	the destination S3 bucket
key	string	the destination S3 key
filename	filename	the local file to be uploaded

S3~s3CopyObject(params) ⇒ Promise

Copy an object from one location on S3 to another

Kind: inner method of S3
Returns: Promise - promise of the object being copied

Param	Type	Description
params	Object	same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property

S3~promiseS3Upload(params) ⇒ Promise

Upload data to S3

see https://github.com/aws/aws-sdk-js-v3/tree/main/lib/lib-storage

Kind: inner method of S3

Param	Type
params	UploadOptions

S3~streamS3Upload(uploadStream, uploadParams) ⇒ Promise

Upload data to S3 using a stream

Kind: inner method of S3

Param	Type	Description
uploadStream	Readable	Stream of data to upload
uploadParams	Object

S3~getObjectReadStream(params) ⇒ Promise.<Readable>

Get a readable stream for an S3 object

Kind: inner method of S3

Param	Type	Description
params	Object
params.s3	S3	an S3 instance
params.bucket	string	the bucket of the requested object
params.key	string	the key of the requested object

S3~downloadS3File(s3Obj, filepath) ⇒ Promise.<string>

Downloads the given s3Obj to the given filename in a streaming manner

Kind: inner method of S3
Returns: Promise.<string> - returns filename if successful

Param	Type	Description
s3Obj	Object	The parameters to send to S3 getObject call
filepath	string	The filepath of the file that is downloaded

S3~getObjectSize(params) ⇒ Promise.<(number|undefined)>

Get the size of an S3 object

Kind: inner method of S3
Returns: Promise.<(number|undefined)> - object size, in bytes

Param	Type	Description
params	Object
params.bucket	string
params.key	string
params.s3	S3	an S3 client instance

S3~s3GetObjectTagging(bucket, key) ⇒ Promise.<GetObjectTaggingOutput>

Get object Tagging from S3

Kind: inner method of S3
Returns: Promise.<GetObjectTaggingOutput> - the promised response from S3.getObjectTagging

Param	Type	Description
bucket	string	name of bucket
key	string	key for object (filepath + filename)

S3~s3PutObjectTagging(Bucket, Key, ObjectTagging) ⇒ Promise

Puts object Tagging in S3 https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObjectTagging-property

Kind: inner method of S3
Returns: Promise - returns response from S3.getObjectTagging as a promise

Param	Type	Description
Bucket	string	name of bucket
Key	string	key for object (filepath + filename)
ObjectTagging	Object	tagging object

S3~getObject(s3Client, params) ⇒ Promise.<GetObjectOutput>

Gets an object from S3.

Kind: inner method of S3
Returns: Promise.<GetObjectOutput> - response from S3.getObject() as a Promise

Param	Type	Description
s3Client	S3	an S3 instance
params	GetObjectCommandInput	parameters object to pass through to S3.getObject()

Example

const obj = await getObject(s3(), { Bucket: 'b', Key: 'k' })

S3~waitForObject(s3Client, params, [retryOptions]) ⇒ Promise.<GetObjectOutput>

Get an object from S3, waiting for it to exist and, if specified, have the correct ETag.

Kind: inner method of S3

Param	Type	Default
s3Client	S3
params	GetObjectCommandInput
[retryOptions]	pRetry.Options	{}

S3~getObjectStreamContents(objectReadStream) ⇒ Promise.<string>

Transform streaming response from S3 object to text content

Kind: inner method of S3
Returns: Promise.<string> - the contents of the S3 object

Param	Type	Description
objectReadStream	Readable	Readable stream of S3 object

S3~getTextObject(bucket, key) ⇒ Promise.<string>

Fetch the contents of an S3 object

Kind: inner method of S3
Returns: Promise.<string> - the contents of the S3 object

Param	Type	Description
bucket	string	the S3 object's bucket
key	string	the S3 object's key

S3~getJsonS3Object(bucket, key) ⇒ Promise.<*>

Fetch JSON stored in an S3 object

Kind: inner method of S3
Returns: Promise.<*> - the contents of the S3 object, parsed as JSON

Param	Type	Description
bucket	string	the S3 object's bucket
key	string	the S3 object's key

S3~fileExists(bucket, key) ⇒ Promise

Check if a file exists in an S3 object

Kind: inner method of S3
Returns: Promise - returns the response from S3.headObject as a promise

Param	Type	Description
bucket	string	name of the S3 bucket
key	string	key of the file in the S3 bucket

S3~deleteS3Files(s3Objs) ⇒ Promise

Delete files from S3

Kind: inner method of S3
Returns: Promise - A promise that resolves to an Array of the data returned from the deletion operations

Param	Type	Description
s3Objs	Array	An array of objects containing keys 'Bucket' and 'Key'

S3~uploadS3FileStream(fileStream, bucket, key, s3opts) ⇒ Promise

Upload the file associated with the given stream to an S3 bucket

Kind: inner method of S3
Returns: Promise - A promise

Param	Type	Description
fileStream	ReadableStream	The stream for the file's contents
bucket	string	The S3 bucket to which the file is to be uploaded
key	string	The key to the file in the bucket
s3opts	Object	Options to pass to the AWS sdk call (defaults to {})

S3~listS3Objects()

List the objects in an S3 bucket

Kind: inner method of S3

S3~deleteS3Buckets(buckets) ⇒ Promise

Delete a list of buckets and all of their objects from S3

Kind: inner method of S3
Returns: Promise - the promised result of S3.deleteBucket

Param	Type	Description
buckets	Array	list of bucket names

S3~calculateObjectHash(params)

Calculate the cryptographic hash of an S3 object

Kind: inner method of S3

Param	Type	Description
params	Object
params.s3	S3	an S3 instance
params.algorithm	string	cksum, or an algorithm listed in openssl list -digest-algorithms
params.bucket	string
params.key	string

S3~validateS3ObjectChecksum(params) ⇒ Promise.<boolean>

Validate S3 object checksum against expected sum

Kind: inner method of S3
Returns: Promise.<boolean> - returns true for success
Throws:

• InvalidChecksum - Throws error if validation fails

Param	Type	Description
params	Object	params
params.algorithm	string	checksum algorithm
params.bucket	string	S3 bucket
params.key	string	S3 key
params.expectedSum	number | string	expected checksum
[params.options]	Object	crypto.createHash options

S3~getFileBucketAndKey(pathParams) ⇒ Array.<string>

Extract the S3 bucket and key from the URL path parameters

Kind: inner method of S3
Returns: Array.<string> - [Bucket, Key]

Param	Type	Description
pathParams	string	path parameters from the URL bucket/key in the form of

S3~createBucket(Bucket) ⇒ Promise

Create an S3 bucket

Kind: inner method of S3

Param	Type	Description
Bucket	string	the name of the S3 bucket to create

S3~createS3Buckets(buckets) ⇒ Promise

Create multiple S3 buckets

Kind: inner method of S3

Param	Type	Description
buckets	Array.<string>	the names of the S3 buckets to create

S3~multipartCopyObject(params) ⇒ Promise.<{etag: string}>

Copy an S3 object to another location in S3 using a multipart copy

Kind: inner method of S3
Returns: Promise.<{etag: string}> - object containing the ETag of the destination object

note: this method may error if used with zero byte files. see CUMULUS-2557 and https://github.com/nasa/cumulus/pull/2117.

Param	Type	Default	Description
params	Object
params.sourceBucket	string
params.sourceKey	string
params.destinationBucket	string
params.destinationKey	string
[params.sourceObject]	S3.HeadObjectOutput		Output from https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property
[params.ACL]	string		an S3 Canned ACL
[params.copyTags]	boolean	false
[params.chunkSize]	number		chunk size of the S3 multipart uploads

S3~moveObject(params) ⇒ Promise.<undefined>

Move an S3 object to another location in S3

Kind: inner method of S3

Param	Type	Default	Description
params	Object
params.sourceBucket	string
params.sourceKey	string
params.destinationBucket	string
params.destinationKey	string
[params.ACL]	string		an S3 Canned ACL
[params.copyTags]	boolean	false
[params.chunkSize]	number		chunk size of the S3 multipart uploads

SNS

• SNS
  • ~publishSnsMessageWithRetry(snsTopicArn, message, retryOptions) ⇒ Promise.<undefined>
  • ~createSnsTopic(snsTopicName) ⇒

SNS~publishSnsMessageWithRetry(snsTopicArn, message, retryOptions) ⇒ Promise.<undefined>

Publish a message to an SNS topic. Does not catch errors, to allow more specific handling by the caller.

Kind: inner method of SNS

Param	Description
snsTopicArn	SNS topic ARN
message	Message object
retryOptions	options to control retry behavior when publishing a message fails. See https://github.com/tim-kos/node-retry#retryoperationoptions

SNS~createSnsTopic(snsTopicName) ⇒

Create an SNS topic with a given name.

Kind: inner method of SNS
Returns: - ARN of the created SNS topic

Param	Description
snsTopicName	Name of the SNS topic

SQS

• SQS
  • ~createQueue()
  • ~sendSQSMessage()
  • ~receiveSQSMessages()
  • ~isSQSRecordLike()
  • ~parseSQSMessageBody()
  • ~deleteSQSMessage()
  • ~sqsQueueExists()

SQS~createQueue()

Create an SQS Queue. Properly handles localstack queue URLs

Kind: inner method of SQS

SQS~sendSQSMessage()

Send a message to AWS SQS

Kind: inner method of SQS

SQS~receiveSQSMessages()

Receives SQS messages from a given queue. The number of messages received can be set and the timeout is also adjustable.

Kind: inner method of SQS

SQS~isSQSRecordLike()

Bare check for SQS message Shape

Kind: inner method of SQS

SQS~parseSQSMessageBody()

Extract SQS message body

Kind: inner method of SQS

SQS~deleteSQSMessage()

Delete a given SQS message from a given queue.

Kind: inner method of SQS

SQS~sqsQueueExists()

Test if an SQS queue exists

Kind: inner method of SQS

STS

SecretsManager

StepFunctions

• StepFunctions
  • static
    • .describeExecution(params) ⇒ Promise.<DescribeExecutionOutput>
    • .describeStateMachine(params) ⇒ Promise.<DescribeStateMachineOutput>
    • .getExecutionHistory(params) ⇒ Promise.<GetExecutionHistoryOutput>
    • .listExecutions(params) ⇒ Promise.<Object>
  • inner
    • ~executionExists(executionArn) ⇒ Promise.<boolean>

StepFunctions.describeExecution(params) ⇒ Promise.<DescribeExecutionOutput>

Call StepFunctions DescribeExecution

See StepFunctions.describeExecution() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param	Type
params	DescribeExecutionInput

StepFunctions.describeStateMachine(params) ⇒ Promise.<DescribeStateMachineOutput>

Call StepFunctions DescribeStateMachine

See StepFunctions.describeStateMachine() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param	Type
params	DescribeStateMachineInput

StepFunctions.getExecutionHistory(params) ⇒ Promise.<GetExecutionHistoryOutput>

Call StepFunctions GetExecutionHistory

See StepFunctions.getExecutionHistory() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param	Type
params	GetExecutionHistoryInput

StepFunctions.listExecutions(params) ⇒ Promise.<Object>

Call StepFunctions ListExecutions

See StepFunctions.listExecutions() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param	Type
params	Object

StepFunctions~executionExists(executionArn) ⇒ Promise.<boolean>

Check if a Step Function Execution exists

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: inner method of StepFunctions

Param	Type	Description
executionArn	string	the ARN of the Step Function Execution to check for

DynamoDbSearchQueue

Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.

Kind: global class

• DynamoDbSearchQueue
  • .empty() ⇒ Promise.<Array>
  • .peek() ⇒ Promise.<Object>
  • .shift() ⇒ Promise.<Object>

dynamoDbSearchQueue.empty() ⇒ Promise.<Array>

Drain all values from the searchQueue, and return to the user. Warning: This can be very memory intensive.

Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Array> - array of search results.

dynamoDbSearchQueue.peek() ⇒ Promise.<Object>

View the next item in the queue

This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.

Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object> - an item from the DynamoDB table

dynamoDbSearchQueue.shift() ⇒ Promise.<Object>

Remove the next item from the queue

When there are no more items in the queue, returns 'null'.

Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object> - an item from the DynamoDB table

S3ListObjectsV2Queue

Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.

Kind: global class

• S3ListObjectsV2Queue
  • .peek() ⇒ Promise.<Object>
  • .shift() ⇒ Promise.<Object>

s3ListObjectsV2Queue.peek() ⇒ Promise.<Object>

View the next item in the queue

This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.

Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object> - an S3 object description

s3ListObjectsV2Queue.shift() ⇒ Promise.<Object>

Remove the next item from the queue

When there are no more items in the queue, returns 'null'.

Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object> - an S3 object description

S3ObjectStore

Class to use when interacting with S3

Kind: global class

• S3ObjectStore
  • .signGetObject(objectUrl, [options], [queryParams], presignOptions) ⇒ Promise.<string>
  • .signHeadObject(objectUrl, [options], [queryParams], presignOptions) ⇒ Promise.<string>

s3ObjectStore.signGetObject(objectUrl, [options], [queryParams], presignOptions) ⇒ Promise.<string>

Returns an HTTPS URL that can be used to perform a GET on the given object store URL

Kind: instance method of S3ObjectStore
Returns: Promise.<string> - a signed URL
Throws:

• TypeError - if the URL is not a recognized protocol or cannot be parsed

Param	Type	Description
objectUrl	string	the URL of the object to sign
[options]	string	options to pass to S3.getObject
[queryParams]	string	a mapping of parameter key/values to put in the URL
presignOptions	RequestPresigningArguments	presignOptions

s3ObjectStore.signHeadObject(objectUrl, [options], [queryParams], presignOptions) ⇒ Promise.<string>

Returns an HTTPS URL that can be used to perform a HEAD on the given object store URL

Kind: instance method of S3ObjectStore
Returns: Promise.<string> - a signed URL
Throws:

• TypeError - if the URL is not a recognized protocol or cannot be parsed

Param	Type	Description
objectUrl	string	the URL of the object to sign
[options]	string	options to pass to S3.getObject
[queryParams]	string	a mapping of parameter key/values to put in the URL
presignOptions	RequestPresigningArguments	presignOptions

createKey(params) ⇒ Promise.<Object>

Create a KMS key

See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/KMS.html#createKey-property for allowed params and return value.

Kind: global function

Param	Type
params	Object

encrypt(KeyId, Plaintext) ⇒ Promise.<string>

Encrypt a string using KMS

Kind: global function
Returns: Promise.<string> - the Base 64 encoding of the encrypted value

Param	Type	Description
KeyId	string	the KMS key to use for encryption
Plaintext	string	the string to be encrypted

decryptBase64String(ciphertext) ⇒ string

Decrypt a KMS-encrypted string, Base 64 encoded

Kind: global function
Returns: string - the plaintext

Param	Type	Description
ciphertext	string	a KMS-encrypted value, Base 64 encoded

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Contributing

To make a contribution, please see our contributing guidelines.

---

Generated automatically using npm run build-docs

Changelog

[v19.1.0] 2024-10-07

Migration Notes

This release contains changes listed here as well as changes listed in v19.0.0, despite v19.0.0 being deprecated. Please review Changelog entries and Migration Notes for each Cumulus version between your current version and v19.1.0 as normal.

Added

• CUMULUS-3020

  • Updated sfEventSqsToDbRecords to allow override of the default value (var.rds_connection_timing_configuration.acquireTimeoutMillis / 1000) + 60) via a key 'sfEventSqsToDbRecords' on var.lambda_timeouts on the main cumulus module/archive module

  Please note - updating this configuration is for adavanced users only. Value changes will modify the visibility timeout on sfEventSqsToDbRecordsDeadLetterQueue and sfEventSqsToDbRecordsInputQueue and may lead to system instability.

• CUMULUS-3756

  • Added excludeFileRegex configuration to UpdateGranulesCmrMetadataFileLinks
  • This is to allow files matching specified regex to be excluded when updating the Related URLs list
  • Defaults to the current behavior of excluding no files.

• CUMULUS-3773

  • Added sftpFastDownload configuration to SyncGranule task.
  • Updated @cumulus/sftp-client and @cumulus/ingest/SftpProviderClient to support both regular and fastDownload.
  • Added sftp support to FakeProvider
  • Added sftp integration test

Changed

• CUMULUS-3928
  • updated publish scripting to use [e-mail] for user email
  • updated publish scripting to use esm over common import of latest-version
  • updated bigint testing to remove intermitted failure source.
  • updated postgres dependency version
• CUMULUS-3838
  • Updated python dependencies to latest:
    • cumulus-process-py 1.4.0
    • cumulus-message-adapter-python 2.3.0
• CUMULUS-3906
  • Bumps example ORCA deployment to version v10.0.1.

Fixed

• CUMULUS-3904
  • Passed sqs_message_consumer_watcher_message_limit and sqs_message_consumer_watcher_time_limit through the cumulus terraform module to the ingest terraform module.
• CUMULUS-3902
  • Update error handling to use AWS SDK V3 error classes instead of properties on js objects