@cumulus/aws-client

@cumulus/aws-client

Utilities for working with AWS. These utilities can be used for interacting with live AWS services or Localstack. For ease of setup, testing, and credential management, code interacting with AWS services should use the helpers in this module.

⚠️ The documented API of this package will not change without a deprecation warning being provided in earlier releases. Code in this package that is not documented in this README may change without warning, and is not considered part of the package's public API.

Usage

npm install @cumulus/aws-client

Interacting with Localstack

To use these utilities with Localstack, make sure you have a running instance of Localstack and set this environment variable:

NODE_ENV=test

API

Modules

CloudFormation

CloudwatchEvents

DynamoDb

Kinesis

Lambda

S3

SecretsManager

SNS

SQS

StepFunctions

STS

Classes

DynamoDbSearchQueue

Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.

S3ListObjectsV2Queue

Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.

S3ObjectStore

Class to use when interacting with S3

Functions

createKey(params) ⇒ Promise.<Object>

Create a KMS key

See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/KMS.html#createKey-property for allowed params and return value.

encrypt(KeyId, Plaintext) ⇒ Promise.<string>

Encrypt a string using KMS

decryptBase64String(ciphertext) ⇒ string

Decrypt a KMS-encrypted string, Base 64 encoded

CloudFormation

• CloudFormation
  • ~describeCfStack(StackName) ⇒ Promise.<CloudFormation.Stack>
  • ~describeCfStackResources(StackName) ⇒ Promise.<CloudFormation.StackResources>
  • ~getCfStackParameterValues(stackName, parameterKeys) ⇒ Promise.<Object>

CloudFormation~describeCfStack(StackName) ⇒ Promise.<CloudFormation.Stack>

Describes a given CloudFormation stack

See CloudFormation.Stack

Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.Stack> - The resources belonging to the stack

Param	Type	Description
StackName	string	The name of the CloudFormation stack to query

CloudFormation~describeCfStackResources(StackName) ⇒ Promise.<CloudFormation.StackResources>

Describes the resources belonging to a given CloudFormation stack

See CloudFormation.StackResources

Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.StackResources> - The resources belonging to the stack

Param	Type	Description
StackName	string	The name of the CloudFormation stack to query

CloudFormation~getCfStackParameterValues(stackName, parameterKeys) ⇒ Promise.<Object>

Get parameter values for the given CloudFormation stack

Kind: inner method of CloudFormation
Returns: Promise.<Object> - Object keyed by parameter names

Param	Type	Description
stackName	string	The name of the CloudFormation stack to query
parameterKeys	Array.<string>	Key names for the stack parameters that you want to return

CloudwatchEvents

CloudwatchEvents~putEvent(name, schedule, state, [description], [role]) ⇒ Promise.<CloudWatchEvents.PutRuleResponse>

Create a CloudWatch Events rule

Kind: inner method of CloudwatchEvents

Param	Type	Description
name	string	the rule name
schedule	string	a ScheduleExpression
state	string	the state of the rule
[description]	string
[role]	string	a Role ARN

DynamoDb

• DynamoDb
  • static
    • .scan ⇒ Promise.<Object>
    • .createAndWaitForDynamoDbTable(params) ⇒ Promise.<Object>
    • .deleteAndWaitForDynamoDbTableNotExists(params) ⇒ Promise
  • inner
    • ~get(params) ⇒ Promise.<Object>
    • ~parallelScan(params) ⇒ Promise

DynamoDb.scan ⇒ Promise.<Object>

Call DynamoDb client scan

See DocumentClient.scan() for descriptions of params and the return data.

Kind: static property of DynamoDb

Param	Type
params	Object

DynamoDb.createAndWaitForDynamoDbTable(params) ⇒ Promise.<Object>

Create a DynamoDB table and then wait for the table to exist

Kind: static method of DynamoDb
Returns: Promise.<Object> - the output of the createTable call

Param	Type	Description
params	Object	the same params that you would pass to AWS.createTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/classes/dynamodb.html#createtable

DynamoDb.deleteAndWaitForDynamoDbTableNotExists(params) ⇒ Promise

Delete a DynamoDB table and then wait for the table to not exist

Kind: static method of DynamoDb

Param	Type	Description
params	Object	the same params that you would pass to AWS.deleteTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/classes/dynamodb.html#deletetable

DynamoDb~get(params) ⇒ Promise.<Object>

Call DynamoDb client get

See DocumentClient.get() for descriptions of params and the return data.

Kind: inner method of DynamoDb
Throws:

• RecordDoesNotExist if a record cannot be found

Param	Type	Description
params	Object
params.tableName	string	Table name to read
params.item	GetCommandInput.Key	Key identifying object to get
params.client	DynamoDBDocument	Instance of a DynamoDb DocumentClient
params.getParams	Object	Additional parameters for DocumentClient.get()

DynamoDb~parallelScan(params) ⇒ Promise

Do a parallel scan of DynamoDB table using a document client.

See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.ParallelScan. See DocumentClient.scan().

Kind: inner method of DynamoDb

Param	Type	Description
params	Object
params.totalSegments	number	Total number of segments to divide table into for parallel scanning
params.scanParams	ScanInput	Params for the DynamoDB client scan operation See https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html
params.processItemsFunc	function	Function used to process returned items by scan
[params.dynamoDbClient]	DynamoDBDocument	Instance of Dynamo DB document client
[params.retryOptions]	pRetry.Options	Retry options for scan operations

Kinesis

Kinesis~describeStream(params, retryOptions) ⇒ Promise.<Object>

Describe a Kinesis stream.

Kind: inner method of Kinesis
Returns: Promise.<Object> - The stream description response

Param	Type	Description
params	Object
params.StreamName	string	A Kinesis stream name
retryOptions	Object	Options passed to p-retry module

Lambda

• Lambda
  • ~isEventBridgeEvent()
  • ~invoke()

Lambda~isEventBridgeEvent()

Bare check for EventBridge shape

Kind: inner method of Lambda

Lambda~invoke()

Invoke a Lambda function

Kind: inner method of Lambda

S3

• S3
  • static
    • .getS3Object
    • .recursivelyDeleteS3Bucket ⇒
    • .listS3ObjectsV2(params) ⇒ Promise.<Array>
    • .listS3ObjectsV2Batch(params)
  • inner
    • ~s3Join(...args) ⇒ string
    • ~parseS3Uri(uri) ⇒ Object
    • ~buildS3Uri(bucket, key) ⇒ string
    • ~s3TagSetToQueryString(tagset) ⇒ string
    • ~deleteS3Object(bucket, key) ⇒ Promise
    • ~headObject(Bucket, Key, retryOptions) ⇒
    • ~s3ObjectExists(params) ⇒ Promise.<boolean>
    • ~waitForObjectToExist(params) ⇒
    • ~s3PutObject(params)
    • ~putFile()
    • ~s3CopyObject()
    • ~promiseS3Upload()
    • ~streamS3Upload()
    • ~getObjectReadStream()
    • ~downloadS3File()
    • ~getObjectSize()
    • ~s3GetObjectTagging()
    • ~s3PutObjectTagging()
    • ~getObject()
    • ~waitForObject()
    • ~getObjectStreamContents()
    • ~getTextObject()
    • ~getJsonS3Object()
    • ~fileExists()
    • ~deleteS3Files(s3Objs)
    • ~uploadS3FileStream(fileStream, bucket, key, s3opts) ⇒ Promise
    • ~listS3Objects()
    • ~deleteS3Buckets(buckets) ⇒ Promise
    • ~calculateObjectHash(params)
    • ~validateS3ObjectChecksum(params) ⇒ Promise.<boolean>
    • ~getFileBucketAndKey(pathParams) ⇒ Array.<string>
    • ~createBucket(Bucket) ⇒ Promise
    • ~createS3Buckets(buckets) ⇒ Promise
    • ~multipartCopyObject(params) ⇒ Promise.<{etag: string}>
    • ~copyObject()
    • ~moveObject() ⇒ Promise.<undefined>

S3.getS3Object

Deprecated

Gets an object from S3.

Kind: static property of S3

S3.recursivelyDeleteS3Bucket ⇒

Delete a bucket and all of its objects from S3

Kind: static property of S3
Returns: the promised result of S3.deleteBucket

Param	Description
bucket	name of the bucket

S3.listS3ObjectsV2(params) ⇒ Promise.<Array>

Fetch complete list of S3 objects

listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.

The passed params must be compatible with the listObjectsV2 call.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property

Kind: static method of S3
Returns: Promise.<Array> - resolves to an array of objects corresponding to the Contents property of the listObjectsV2 response

Param	Type	Description
params	Object	params for the s3.listObjectsV2 call

S3.listS3ObjectsV2Batch(params)

Fetch lazy list of S3 objects

listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.

The passed params must be compatible with the listObjectsV2 call.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property

Kind: static method of S3

Param	Description
params	params for the s3.listObjectsV2 call

S3~s3Join(...args) ⇒ string

Join strings into an S3 key without a leading slash

Kind: inner method of S3
Returns: string - the full S3 key

Param	Type	Description
...args	string | Array.<string>	the strings to join

S3~parseS3Uri(uri) ⇒ Object

parse an s3 uri to get the bucket and key

Kind: inner method of S3
Returns: Object - Returns an object with Bucket and Key properties

Param	Type	Description
uri	string	must be a uri with the s3:// protocol

S3~buildS3Uri(bucket, key) ⇒ string

Given a bucket and key, return an S3 URI

Kind: inner method of S3
Returns: string - an S3 URI

Param	Type	Description
bucket	string	an S3 bucket name
key	string	an S3 key

S3~s3TagSetToQueryString(tagset) ⇒ string

Convert S3 TagSet Object to query string e.g. [{ Key: 'tag', Value: 'value }] to 'tag=value'

Kind: inner method of S3
Returns: string - tags query string

Param	Type	Description
tagset	Array.<Object>	S3 TagSet array

S3~deleteS3Object(bucket, key) ⇒ Promise

Delete an object from S3

Kind: inner method of S3
Returns: Promise - promise of the object being deleted

Param	Type	Description
bucket	string	bucket where the object exists
key	string	key of the object to be deleted

S3~headObject(Bucket, Key, retryOptions) ⇒

Get an object header from S3

Kind: inner method of S3
Returns: returns response from S3.headObject as a promise

Param	Description
Bucket	name of bucket
Key	key for object (filepath + filename)
retryOptions	options to control retry behavior when an object does not exist. See https://github.com/tim-kos/node-retry#retryoperationoptions By default, retries will not be performed

S3~s3ObjectExists(params) ⇒ Promise.<boolean>

Test if an object exists in S3

Kind: inner method of S3
Returns: Promise.<boolean> - a Promise that will resolve to a boolean indicating if the object exists

Param	Description
params	same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property

S3~waitForObjectToExist(params) ⇒

Asynchronously waits for an S3 object to exist at a specified location.

This function uses p-wait-for to repeatedly check for the object's existence until it's found or a timeout is reached. It provides configurable interval between checks and a total timeout duration.

Kind: inner method of S3
Returns: A Promise that resolves when the S3 object is found.
Throws:

• TimeoutError If the object does not exist within the specified timeout period.
• Error If an unexpected error occurs during the S3 existence check.

Param	Description
params	The parameters for waiting for the object.
params.bucket	The name of the S3 bucket where the object is expected.
params.key	The key (path) of the S3 object within the bucket.
params.interval	The time in milliseconds to wait between checks. Defaults to 1000ms (1 second).
params.timeout	The maximum time in milliseconds to wait before giving up and throwing a TimeoutError. Defaults to 30000ms (30 seconds).

S3~s3PutObject(params)

Put an object on S3

Kind: inner method of S3

Param	Description
params	same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property

S3~putFile()

Upload a file to S3

Kind: inner method of S3

S3~s3CopyObject()

Copy an object from one location on S3 to another

Kind: inner method of S3

S3~promiseS3Upload()

Upload data to S3

see https://github.com/aws/aws-sdk-js-v3/tree/main/lib/lib-storage

Kind: inner method of S3

S3~streamS3Upload()

Upload data to S3 using a stream

Kind: inner method of S3

S3~getObjectReadStream()

Get a readable stream for an S3 object

Kind: inner method of S3

S3~downloadS3File()

Downloads the given s3Obj to the given filename in a streaming manner

Kind: inner method of S3

S3~getObjectSize()

Get the size of an S3 object

Kind: inner method of S3

S3~s3GetObjectTagging()

Get object Tagging from S3

Kind: inner method of S3

S3~s3PutObjectTagging()

Puts object Tagging in S3 https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObjectTagging-property

Kind: inner method of S3

S3~getObject()

Gets an object from S3.

Kind: inner method of S3

S3~waitForObject()

Get an object from S3, waiting for it to exist and, if specified, have the correct ETag.

Kind: inner method of S3

S3~getObjectStreamContents()

Transform streaming response from S3 object to text content

Kind: inner method of S3

S3~getTextObject()

Fetch the contents of an S3 object

Kind: inner method of S3

S3~getJsonS3Object()

Fetch JSON stored in an S3 object

Kind: inner method of S3

S3~fileExists()

Check if a file exists in an S3 object

Kind: inner method of S3

S3~deleteS3Files(s3Objs)

Delete files from S3

Kind: inner method of S3

Param	Description
s3Objs	An array of objects containing keys 'Bucket' and 'Key'

S3~uploadS3FileStream(fileStream, bucket, key, s3opts) ⇒ Promise

Upload the file associated with the given stream to an S3 bucket

Kind: inner method of S3
Returns: Promise - A promise

Param	Type	Description
fileStream	ReadableStream	The stream for the file's contents
bucket	string	The S3 bucket to which the file is to be uploaded
key	string	The key to the file in the bucket
s3opts	Object	Options to pass to the AWS sdk call (defaults to {})

S3~listS3Objects()

List the objects in an S3 bucket

Kind: inner method of S3

S3~deleteS3Buckets(buckets) ⇒ Promise

Delete a list of buckets and all of their objects from S3

Kind: inner method of S3
Returns: Promise - the promised result of S3.deleteBucket

Param	Type	Description
buckets	Array	list of bucket names

S3~calculateObjectHash(params)

Calculate the cryptographic hash of an S3 object

Kind: inner method of S3

Param	Type	Description
params	Object
params.s3	S3	an S3 instance
params.algorithm	string	cksum, or an algorithm listed in openssl list -digest-algorithms
params.bucket	string
params.key	string

S3~validateS3ObjectChecksum(params) ⇒ Promise.<boolean>

Validate S3 object checksum against expected sum

Kind: inner method of S3
Returns: Promise.<boolean> - returns true for success
Throws:

• InvalidChecksum - Throws error if validation fails

Param	Type	Description
params	Object	params
params.algorithm	string	checksum algorithm
params.bucket	string	S3 bucket
params.key	string	S3 key
params.expectedSum	number | string	expected checksum
[params.options]	Object	crypto.createHash options

S3~getFileBucketAndKey(pathParams) ⇒ Array.<string>

Extract the S3 bucket and key from the URL path parameters

Kind: inner method of S3
Returns: Array.<string> - [Bucket, Key]

Param	Type	Description
pathParams	string	path parameters from the URL bucket/key in the form of

S3~createBucket(Bucket) ⇒ Promise

Create an S3 bucket

Kind: inner method of S3

Param	Type	Description
Bucket	string	the name of the S3 bucket to create

S3~createS3Buckets(buckets) ⇒ Promise

Create multiple S3 buckets

Kind: inner method of S3

Param	Type	Description
buckets	Array.<string>	the names of the S3 buckets to create

S3~multipartCopyObject(params) ⇒ Promise.<{etag: string}>

Copy an S3 object to another location in S3 using a multipart copy

Kind: inner method of S3
Returns: Promise.<{etag: string}> - object containing the ETag of the destination object

note: this method may error if used with zero byte files. see CUMULUS-2557 and https://github.com/nasa/cumulus/pull/2117.

Param	Type	Default	Description
params	Object
params.sourceBucket	string
params.sourceKey	string
params.destinationBucket	string
params.destinationKey	string
[params.sourceObject]	S3.HeadObjectOutput		Output from https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property
[params.ACL]	string		an S3 Canned ACL
[params.copyTags]	boolean	false
[params.chunkSize]	number		chunk size of the S3 multipart uploads

S3~copyObject()

Copy an S3 object to another location in S3

Kind: inner method of S3

Param	Default	Description
params.ACL		an S3 Canned ACL
params.copyTags	false
params.chunkSize		chunk size of the S3 multipart uploads

S3~moveObject() ⇒ Promise.<undefined>

Move an S3 object to another location in S3

Kind: inner method of S3

Param	Type	Default	Description
[params.ACL]	string		an S3 Canned ACL
[params.copyTags]	boolean	false
[params.chunkSize]	number		chunk size of the S3 multipart uploads

SecretsManager

SNS

• SNS
  • ~publishSnsMessageWithRetry(snsTopicArn, message, retryOptions) ⇒ Promise.<undefined>
  • ~createSnsTopic(snsTopicName) ⇒

SNS~publishSnsMessageWithRetry(snsTopicArn, message, retryOptions) ⇒ Promise.<undefined>

Publish a message to an SNS topic. Does not catch errors, to allow more specific handling by the caller.

Kind: inner method of SNS

Param	Description
snsTopicArn	SNS topic ARN
message	Message object
retryOptions	options to control retry behavior when publishing a message fails. See https://github.com/tim-kos/node-retry#retryoperationoptions

SNS~createSnsTopic(snsTopicName) ⇒

Create an SNS topic with a given name.

Kind: inner method of SNS
Returns: - ARN of the created SNS topic

Param	Description
snsTopicName	Name of the SNS topic

SQS

• SQS
  • ~createQueue()
  • ~sendSQSMessage()
  • ~receiveSQSMessages()
  • ~isSQSRecordLike()
  • ~parseSQSMessageBody()
  • ~deleteSQSMessage()
  • ~sqsQueueExists()

SQS~createQueue()

Create an SQS Queue. Properly handles localstack queue URLs

Kind: inner method of SQS

SQS~sendSQSMessage()

Send a message to AWS SQS

Kind: inner method of SQS

SQS~receiveSQSMessages()

Receives SQS messages from a given queue. The number of messages received can be set and the timeout is also adjustable.

Kind: inner method of SQS

SQS~isSQSRecordLike()

Bare check for SQS message Shape

Kind: inner method of SQS

SQS~parseSQSMessageBody()

Extract SQS message body

Kind: inner method of SQS

SQS~deleteSQSMessage()

Delete a given SQS message from a given queue.

Kind: inner method of SQS

SQS~sqsQueueExists()

Test if an SQS queue exists

Kind: inner method of SQS

StepFunctions

• StepFunctions
  • static
    • .describeExecution(params) ⇒ Promise.<DescribeExecutionOutput>
    • .describeStateMachine(params) ⇒ Promise.<DescribeStateMachineOutput>
    • .getExecutionHistory(params) ⇒ Promise.<GetExecutionHistoryOutput>
    • .listExecutions(params) ⇒ Promise.<Object>
  • inner
    • ~executionExists(executionArn) ⇒ Promise.<boolean>

StepFunctions.describeExecution(params) ⇒ Promise.<DescribeExecutionOutput>

Call StepFunctions DescribeExecution

See StepFunctions.describeExecution() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param	Type
params	DescribeExecutionInput

StepFunctions.describeStateMachine(params) ⇒ Promise.<DescribeStateMachineOutput>

Call StepFunctions DescribeStateMachine

See StepFunctions.describeStateMachine() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param	Type
params	DescribeStateMachineInput

StepFunctions.getExecutionHistory(params) ⇒ Promise.<GetExecutionHistoryOutput>

Call StepFunctions GetExecutionHistory

See StepFunctions.getExecutionHistory() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param	Type
params	GetExecutionHistoryInput

StepFunctions.listExecutions(params) ⇒ Promise.<Object>

Call StepFunctions ListExecutions

See StepFunctions.listExecutions() for descriptions of params and the return data.

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: static method of StepFunctions

Param	Type
params	Object

StepFunctions~executionExists(executionArn) ⇒ Promise.<boolean>

Check if a Step Function Execution exists

If a ThrottlingException is received, this function will retry using an exponential backoff.

Kind: inner method of StepFunctions

Param	Type	Description
executionArn	string	the ARN of the Step Function Execution to check for

STS

DynamoDbSearchQueue

Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.

Kind: global class

• DynamoDbSearchQueue
  • .empty() ⇒ Promise.<Array>
  • .peek() ⇒ Promise.<Object>
  • .shift() ⇒ Promise.<Object>

dynamoDbSearchQueue.empty() ⇒ Promise.<Array>

Drain all values from the searchQueue, and return to the user. Warning: This can be very memory intensive.

Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Array> - array of search results.

dynamoDbSearchQueue.peek() ⇒ Promise.<Object>

View the next item in the queue

This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.

Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object> - an item from the DynamoDB table

dynamoDbSearchQueue.shift() ⇒ Promise.<Object>

Remove the next item from the queue

When there are no more items in the queue, returns 'null'.

Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object> - an item from the DynamoDB table

S3ListObjectsV2Queue

Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.

Kind: global class

• S3ListObjectsV2Queue
  • .peek() ⇒ Promise.<Object>
  • .shift() ⇒ Promise.<Object>

s3ListObjectsV2Queue.peek() ⇒ Promise.<Object>

View the next item in the queue

This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.

Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object> - an S3 object description

s3ListObjectsV2Queue.shift() ⇒ Promise.<Object>

Remove the next item from the queue

When there are no more items in the queue, returns 'null'.

Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object> - an S3 object description

S3ObjectStore

Class to use when interacting with S3

Kind: global class

• S3ObjectStore
  • .signGetObject(objectUrl, [options], [queryParams], presignOptions) ⇒ Promise.<string>
  • .signHeadObject(objectUrl, [options], [queryParams], presignOptions) ⇒ Promise.<string>

s3ObjectStore.signGetObject(objectUrl, [options], [queryParams], presignOptions) ⇒ Promise.<string>

Returns an HTTPS URL that can be used to perform a GET on the given object store URL

Kind: instance method of S3ObjectStore
Returns: Promise.<string> - a signed URL
Throws:

• TypeError - if the URL is not a recognized protocol or cannot be parsed

Param	Type	Description
objectUrl	string	the URL of the object to sign
[options]	string	options to pass to S3.getObject
[queryParams]	string	a mapping of parameter key/values to put in the URL
presignOptions	RequestPresigningArguments	presignOptions

s3ObjectStore.signHeadObject(objectUrl, [options], [queryParams], presignOptions) ⇒ Promise.<string>

Returns an HTTPS URL that can be used to perform a HEAD on the given object store URL

Kind: instance method of S3ObjectStore
Returns: Promise.<string> - a signed URL
Throws:

• TypeError - if the URL is not a recognized protocol or cannot be parsed

Param	Type	Description
objectUrl	string	the URL of the object to sign
[options]	string	options to pass to S3.getObject
[queryParams]	string	a mapping of parameter key/values to put in the URL
presignOptions	RequestPresigningArguments	presignOptions

createKey(params) ⇒ Promise.<Object>

Create a KMS key

See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/KMS.html#createKey-property for allowed params and return value.

Kind: global function

Param	Type
params	Object

encrypt(KeyId, Plaintext) ⇒ Promise.<string>

Encrypt a string using KMS

Kind: global function
Returns: Promise.<string> - the Base 64 encoding of the encrypted value

Param	Type	Description
KeyId	string	the KMS key to use for encryption
Plaintext	string	the string to be encrypted

decryptBase64String(ciphertext) ⇒ string

Decrypt a KMS-encrypted string, Base 64 encoded

Kind: global function
Returns: string - the plaintext

Param	Type	Description
ciphertext	string	a KMS-encrypted value, Base 64 encoded

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Contributing

To make a contribution, please see our contributing guidelines.

Generated automatically using npm run build-docs

Changelog

[v21.0.0] 2025-09-09

CUMULUS-4058 Epic: Handle Granules with Identical producerGranuleId in Different Collections

Migration Notes

CUMULUS-4069 Update granules table to include producer_granule_id column

Please follow the instructions before upgrading Cumulus

• The updates in CUMULUS-4069 require a manual update to the PostgreSQL database in the production environment. Please follow the instructions in Update granules to include producer_granule_id

Breaking Changes

• CUMULUS-4078
  • Move Granules task will now check on file collision if the existing file is registered in Core's database to another collection. If it is, the granule (and the task execution) will fail, regardless of the duplicate behavior configuration. If this behavior is undesirable for performance or logic reasons, the checkCrossCollectionCollisions may be set to false to disable the behavior on a per-workflow, per-collection or other config driven criteria.
• CUMULUS-4072
  • Updated the parse-pdr task component to throw an error if multiple granules within the same PDR have the same granuleId after applying the granuleIdFilter, unless the uniquifyGranuleId configuration parameter is explicitly set to true.
• CUMULUS-4074
  • Updates updateGranulesCmrMetadataFileLinks to always ensure producerGranuleId identifier is set in updated CMR metadata
• CUMULUS-4121
  • Updates example deployment cnm_response_task to use newest version v3.2.0, which supports producerGranuleId.
  • Users must ensure that cumulus-tf includes cnm_response_version = "3.2.0" or greater.

Added

• CUMULUS-4059
  • Added new non-null column producer_granule_id to Postgres granules table.
  • Added producerGranuleId property to granule record schema.
  • Updated @cumulus api/db/message packages to handle producer_granule_id and producerGranuleId.
  • Updated @cumulus/api/lib/writeGranulesFromMessage to set producerGranuleId = granuleId if not set.
  • Updated queue-granules task to set producerGranuleId = granuleId if not set.
• CUMULUS-4061
  • Added GenerateUniqueGranuleId to @cumulus/ingest for use in generating a hashed/'uniquified' granuleID
• CUMULUS-4062
  • Added producerGranuleId to LzardsBackup task component and lambda input/output schema
  • Updated LzardsBackup task component to submit producerGranuleId for storage in the lzards record as a key in the metadata object.
• CUMULUS-4069
  • Added migration script and instructions to add the producer_granule_id column to the granules table and populate it in the production environment.
• CUMULUS-4072
  • Updated parse-pdr task component to have the following behaviors:
    • Always populate producerGranuleId from the incoming parsed granuleId
    • If uniquifyGranuleId configuration value is set to true, parse-PDR will update the granuleId for all found granules to have a unique granule hash appended to the existing ID
    • Updated parse-pdr such that if the uniquifyGranuleId configuration parameter is not set to true , and a duplicate granuleId is created as part of the output after passing the granuleIdFilter, the task will throw with an error.
  • Added ingestFromPdrWithUniqueGranuleIdsSpec.js to the spec tests to demonstrate the ingest workflow works as expected with unique granuleIds and producerGranuleIds set.
• CUMULUS-4073
  • Adds AddUniqueGranuleId task to ingest terraform module for deployment with Core. This task will update a payload of existing granules to have 'uniquified' IDs and preserve the original identifier in the producerGranuleId field
• CUMULUS-4074
  • Updated IngestGranuleSuccessSpec/IngestUMMGSuccessSpec to validate producerGranuleId is populated in CMR post ingest
  • Updated IngestGranuleSuccessSpec to include a producerGranuleId in the default test case
  • Added ticket-relevant typing doc/ts-check updates to @cumulus/cmrjs/cmr-utils
  • Updated updateCMRMetadata to take updateGranuleIdentifiers configuration flag/producerGranuleId such that that routine now will modify the CMR metadata object with the correct GranuleUR/ProducerGranuleId values in the CMR metadata.
  • Added unit test/refactored mocks to use direct injection for cmr-utils
  • Added getCmrMetadata helper to @cumulus/integration-tests to allow access to the full CMR metadata object for verification of record metadata fields
  • Added ApiFileGranuleIdOptional to @cumulus/types/api for cases where an ApiFile is being generated and refactored existing code to use this type instead of custom relaxed typing
  • Updates update-granules-cmr-metadata-file-links to use the updated cmrjs logic to set producerGranuleId identifiers in the CMR metadata, either equal to granuleId or the producerGranuleID set on the granule.
  • Updates @cumulus/tasks/sync-granule/GranuleFetcher to allow and pass through an incoming granule.producerGranuleId
• CUMULUS-4077
  • Updated @cumulus/api/lib/ingest.reingestGranule to only update the original granule to 'queued' if the original payload contains the granule. This avoids a situation where the original granule is updated to 'queued', but the reingest workflow creates a new granule, leaving the original granule stuck in 'queued'.
• CUMULUS-4078
  • Added getGranuleIdAndCollectionIdFromFile query method to @cumulus/db to retrieve granule and collection metadata from a file's S3 location.
  • Added new API route GET /granules/files/get_collection_and_granule_id/:bucket/:key in @cumulus/api to return the granule ID and collection ID associated with a file.
  • Added getFileGranuleAndCollectionByBucketAndKey method to @cumulus/api-client/granules to allow use of new endpoint.
  • Added integration and unit tests for the new DB query, API endpoint, and client method.
  • Updated move-granules task to validate cross-collection file collisions using the new lookup logic when checkCrossCollectionCollisions is enabled.
  • Update @cumulus/db to add getGranuleIdAndCollectionIdFromFile query method
• CUMULUS-4079
  • Added duplicate granule handling and related feature documentation, and updated related documentation to match
  • Added update-granules-cmr-metadata-file-links task README
• CUMULUS-4080
  • Add documentation for duplicate granule handling and, specifically, Collection configuration for duplicates.
  • Update urlPathTemplate to allow falling back from one null/undefined interpolated value to a second argument
• CUMULUS-4082
  • Updated example deployment to deploy cnmResponse lambda version 3.1.0-alpha.2-SNAPSHOT which utilizes producerGranuleId.
  • Updated example deployment to deploy cnmToGranule lambda version 2.1.0.
  • Added FakeProcessing task configuration matchFilesWithProducerGranuleId to determine if the generated cmr file names should match granuleId or producerGranuleId
  • Updated AddUniqueGranuleId task configuration hashLength to accept additional types and removed the use of hashDepth.
  • Updated FilesToGranules task configuration matchFilesWithProducerGranuleId to accept additional types.
  • Updated ParsePdr task configuration hashLength to accept additional types.
  • Fixed tf-modules/cumulus AddUniqueGranuleId task output.
  • Updated example deployment workflow CNMExampleWorkflow to uniquify granuleIds based on collection configuration
  • Added KinesisTestTriggerWithUniqueGranuleIdsSpec.js to the spec test to demonstrate that the CNM ingest workflow ingests granules with unique granuleIds and producerGranuleIds set, and that CnmResponse sends responses using producerGranuleIds
• CUMULUS-4085
  • Added config option for files-to-granules task to use producerGranuleId when mapping files to their granules.
• CUMULUS-4089
  • Add integration testing for duplicate granule workflows. This includes new specs and workflows in the ingestGranule, discoverGranules, lzardsBackup, cnmWorkflow, and orca specs.
• CUMULUS-4110
  • Added the workflow_configurations variable to the tf-modules/ingest and tf-modules/cumulus modules. The property sf_event_sqs_to_db_records_types has been added to workflow_template.json under the cumulus_meta field to control which record types should be written to the database during different workflow execution statuses. Currently, both "execution" and "pdr" must be written to the database, so the record type list must include both.
  • Updated the SfSqsReport task to set meta.reportMessageSource in the Cumulus message.
  • Updated the @cumulus/api/sfEventSqsToDbRecords lambda to determine which record types ("execution", "granule", "pdr") should be written to the database based on the cumulus_meta.sf_event_sqs_to_db_records_types and meta.reportMessageSource fields. By default, all record types will be written to the database.
  • Added @cumulus/api/lib.writeRecords.writeGranuleExecutionAssociationsFromMessage to write granule-execution associations from message.
  • Updated the @cumulus/integration-tests cmr.generateAndStoreCmrXml to apply matchFilesWithProducerGranuleId when generaing OnlineAccessURL.
• CUMULUS-4119
  • Added assertions in KinesisTestTriggerWithUniqueGranuleIdsSpec to cover "duplicate" Granules in separate Collections.
• CUMULUS-4162
  • Added an optional includeTimestampHashKey parameter to the generateUniqueGranuleId function in the @cumulus/ingest/granule, with a default value of false.
  • Added an optional includeTimestampHashKey configuration to the add-unique-granuleId and parse-pdr tasks, also with a default value of false.
  • Added a documentation page titled "Generate Unique GranuleId" to explain the algorithm for generating unique granuleIds.
• CUMULUS-4028
  • Update AddUniqueGranuleId task to output the input payload in addition to the modified granules.
  • Added 'unique' version of ingest_and_publish granule workflow for 'uniquiy' feature ingest tests
• CUMULUS-4209
  • Updated the producer_granule_id migration script to disable autovacuum before the migration and re-enable it afterward to improve performance.

Changed

• CUMULUS-4165
  • Update Async Operation container to new version 54, cumuluss/async-operation:54. Users should update their references to async-operation with the new version.
• CUMULUS-4205
  • Add S3 Replicator lambda ARN to s3-replicator outputs