@cumulus/aws-client
Advanced tools
Utilities for working with AWS. These utilities can be used for interacting with live AWS services or Localstack. For ease of setup, testing, and credential management, code interacting with AWS services should use the helpers in this module.
⚠️ The documented API of this package will not change without a deprecation warning being provided in earlier releases. Code in this package that is not documented in this README may change without warning, and is not considered part of the package's public API.
npm install @cumulus/aws-client
To use these utilities with Localstack, make sure you have a running instance of Localstack and set this environment variable:
NODE_ENV=test
Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.
Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.
Class to use when interacting with S3
Promise.<CloudFormation.Stack>Promise.<CloudFormation.StackResources>Promise.<Object>Promise.<CloudFormation.Stack>Describes a given CloudFormation stack
Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.Stack> - The resources belonging to the stack
| Param | Type | Description |
|---|---|---|
| StackName | string | The name of the CloudFormation stack to query |
Promise.<CloudFormation.StackResources>Describes the resources belonging to a given CloudFormation stack
See CloudFormation.StackResources
Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.StackResources> - The resources belonging to the stack
| Param | Type | Description |
|---|---|---|
| StackName | string | The name of the CloudFormation stack to query |
Promise.<Object>Get parameter values for the given CloudFormation stack
Kind: inner method of CloudFormation
Returns: Promise.<Object> - Object keyed by parameter names
| Param | Type | Description |
|---|---|---|
| stackName | string | The name of the CloudFormation stack to query |
| parameterKeys | Array.<string> | Key names for the stack parameters that you want to return |
Promise.<CloudWatchEvents.PutRuleResponse>Create a CloudWatch Events rule
Kind: inner method of CloudwatchEvents
| Param | Type | Description |
|---|---|---|
| name | string | the rule name |
| schedule | string | a ScheduleExpression |
| state | string | the state of the rule |
| [description] | string | |
| [role] | string | a Role ARN |
Promise.<Object>Promise.<Object>Promise.<Object>PromisePromisePromise.<Object>Call DynamoDb client get
See DocumentClient.get()
for descriptions of params and the return data.
Kind: static property of DynamoDb
Throws:
RecordDoesNotExist if a record cannot be found| Param | Type | Description |
|---|---|---|
| params | Object | |
| params.tableName | string | Table name to read |
| params.item | AWS.DynamoDB.DocumentClient.Key | Key identifying object to get |
| params.client | AWS.DynamoDB.DocumentClient | Instance of a DynamoDb DocumentClient |
| params.getParams | Object | Additional parameters for DocumentClient.get() |
Promise.<Object>Call DynamoDb client scan
See DocumentClient.scan()
for descriptions of params and the return data.
Kind: static property of DynamoDb
| Param | Type |
|---|---|
| params | Object |
Promise.<Object>Create a DynamoDB table and then wait for the table to exist
Kind: static method of DynamoDb
Returns: Promise.<Object> - the output of the createTable call
| Param | Type | Description |
|---|---|---|
| params | Object | the same params that you would pass to AWS.createTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#createTable-property |
PromiseDelete a DynamoDB table and then wait for the table to not exist
Kind: static method of DynamoDb
| Param | Type | Description |
|---|---|---|
| params | Object | the same params that you would pass to AWS.deleteTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#deleteTable-property |
PromiseDo a parallel scan of DynamoDB table using a document client.
See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.ParallelScan. See DocumentClient.scan().
Kind: inner method of DynamoDb
| Param | Type | Description |
|---|---|---|
| params | Object | |
| params.totalSegments | number | Total number of segments to divide table into for parallel scanning |
| params.scanParams | DocumentClient.ScanInput | Params for the DynamoDB client scan operation See https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html |
| params.processItemsFunc | function | Function used to process returned items by scan |
| [params.dynamoDbClient] | DocumentClient | Instance of Dynamo DB document client |
| [params.retryOptions] | pRetry.Options | Retry options for scan operations |
Promise.<Object>Promise.<string>stringPromise.<Object>Create a KMS key
See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/KMS.html#createKey-property for allowed params and return value.
Kind: inner method of KMS
| Param | Type |
|---|---|
| params | Object |
Promise.<string>Encrypt a string using KMS
Kind: inner method of KMS
Returns: Promise.<string> - the Base 64 encoding of the encrypted value
| Param | Type | Description |
|---|---|---|
| KeyId | string | the KMS key to use for encryption |
| Plaintext | string | the string to be encrypted |
stringDecrypt a KMS-encrypted string, Base 64 encoded
Kind: inner method of KMS
Returns: string - the plaintext
| Param | Type | Description |
|---|---|---|
| ciphertext | string | a KMS-encrypted value, Base 64 encoded |
Promise.<Object>Describe a Kinesis stream.
Kind: inner method of Kinesis
Returns: Promise.<Object> - The stream description response
| Param | Type | Description |
|---|---|---|
| params | Object | |
| params.StreamName | string | A Kinesis stream name |
| retryOptions | Object | Options passed to p-retry module |
Promise.<AWS.Lambda.InvocationResponse>Invoke a Lambda function
Kind: static method of Lambda
| Param | Type | Default | Description |
|---|---|---|---|
| name | string | Lambda function name | |
| payload | any | the payload to the Lambda function | |
| type | string | "Event" | the invocation type |
PromisePromisePromisePromise.<AWS.S3.GetObjectTaggingOutput>PromisePromisePromisePromise.<Array>stringObjectstringstringPromise.<boolean>Promise.<undefined>PromisePromisePromise.<string>Promise.<(number|undefined)>Promise.<AWS.S3.GetObjectOutput>Promise.<AWS.S3.GetObjectOutput>Promise.<string>Promise.<*>ReadablePromisePromisePromisePromisePromisePromise.<boolean>Array.<string>PromisePromisePromise.<{etag: string}>Promise.<undefined>Delete an object from S3
Kind: static property of S3
| Param | Type | Description |
|---|---|---|
| bucket | string | bucket where the object exists |
| key | string | key of the object to be deleted promise of the object being deleted |
PromiseGet an object header from S3
Kind: static property of S3
Returns: Promise - returns response from S3.headObject as a promise
| Param | Type | Description |
|---|---|---|
| Bucket | string | name of bucket |
| Key | string | key for object (filepath + filename) |
| retryOptions | Object | options to control retry behavior when an object does not exist. See https://github.com/tim-kos/node-retry#retryoperationoptions By default, retries will not be performed |
Put an object on S3
Kind: static property of S3
| Param | Type | Description |
|---|---|---|
| params | Object | same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property promise of the object being put |
PromiseCopy an object from one location on S3 to another
Kind: static property of S3
Returns: Promise - promise of the object being copied
| Param | Type | Description |
|---|---|---|
| params | Object | same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property |
PromiseUpload data to S3
Note: This is equivalent to calling aws.s3().upload(params).promise()
Kind: static property of S3
Returns: Promise - see S3.upload()
| Param | Type | Description |
|---|---|---|
| params | Object | see S3.upload() |
Promise.<AWS.S3.GetObjectTaggingOutput>Get object Tagging from S3
Kind: static property of S3
Returns: Promise.<AWS.S3.GetObjectTaggingOutput> - the promised response from S3.getObjectTagging
| Param | Type | Description |
|---|---|---|
| bucket | string | name of bucket |
| key | string | key for object (filepath + filename) |
PromisePuts object Tagging in S3 https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObjectTagging-property
Kind: static property of S3
Returns: Promise - returns response from S3.getObjectTagging as a promise
| Param | Type | Description |
|---|---|---|
| Bucket | string | name of bucket |
| Key | string | key for object (filepath + filename) |
| Tagging | Object | tagging object |
PromiseDeprecated
Gets an object from S3.
Kind: static property of S3
Returns: Promise - returns response from S3.getObject as a promise
| Param | Type | Description |
|---|---|---|
| Bucket | string | name of bucket |
| Key | string | key for object (filepath + filename) |
| retryOptions | Object | options to control retry behavior when an object does not exist. See https://github.com/tim-kos/node-retry#retryoperationoptions By default, retries will not be performed |
PromiseDelete a bucket and all of its objects from S3
Kind: static property of S3
Returns: Promise - the promised result of S3.deleteBucket
| Param | Type | Description |
|---|---|---|
| bucket | string | name of the bucket |
Promise.<Array>Fetch complete list of S3 objects
listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.
The passed params must be compatible with the listObjectsV2 call.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property
Kind: static method of S3
Returns: Promise.<Array> - resolves to an array of objects corresponding to
the Contents property of the listObjectsV2 response
| Param | Type | Description |
|---|---|---|
| params | Object | params for the s3.listObjectsV2 call |
stringJoin strings into an S3 key without a leading slash
Kind: inner method of S3
Returns: string - the full S3 key
| Param | Type | Description |
|---|---|---|
| ...args | string | Array.<string> | the strings to join |
Objectparse an s3 uri to get the bucket and key
Kind: inner method of S3
Returns: Object - Returns an object with Bucket and Key properties
| Param | Type | Description |
|---|---|---|
| uri | string | must be a uri with the s3:// protocol |
stringGiven a bucket and key, return an S3 URI
Kind: inner method of S3
Returns: string - an S3 URI
| Param | Type | Description |
|---|---|---|
| bucket | string | an S3 bucket name |
| key | string | an S3 key |
stringConvert S3 TagSet Object to query string e.g. [{ Key: 'tag', Value: 'value }] to 'tag=value'
Kind: inner method of S3
Returns: string - tags query string
| Param | Type | Description |
|---|---|---|
| tagset | Array.<Object> | S3 TagSet array |
Promise.<boolean>Test if an object exists in S3
Kind: inner method of S3
Returns: Promise.<boolean> - a Promise that will resolve to a boolean indicating
if the object exists
| Param | Type | Description |
|---|---|---|
| params | Object | same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property |
Promise.<undefined>Wait for an object to exist in S3
Kind: inner method of S3
| Param | Type | Default | Description |
|---|---|---|---|
| params | Object | ||
| params.bucket | string | ||
| params.key | string | ||
| [params.interval] | number | 1000 | interval before retries, in ms |
| [params.timeout] | number | 30000 | timeout, in ms |
PromiseUpload a file to S3
Kind: inner method of S3
| Param | Type | Description |
|---|---|---|
| bucket | string | the destination S3 bucket |
| key | string | the destination S3 key |
| filename | filename | the local file to be uploaded |
PromiseUpload data to S3 using a stream
We are not using s3.upload().promise() due to errors observed in testing
with uncaught exceptions. By creating our own promise, we can ensure any
errors from the streams or upload cause this promise to reject.
Kind: inner method of S3
| Param | Type | Description |
|---|---|---|
| uploadStream | Readable | Stream of data to upload |
| uploadParams | Object | see S3.upload() |
Promise.<string>Downloads the given s3Obj to the given filename in a streaming manner
Kind: inner method of S3
Returns: Promise.<string> - returns filename if successful
| Param | Type | Description |
|---|---|---|
| s3Obj | Object | The parameters to send to S3 getObject call |
| filepath | string | The filepath of the file that is downloaded |
Promise.<(number|undefined)>Get the size of an S3 object
Kind: inner method of S3
Returns: Promise.<(number|undefined)> - object size, in bytes
| Param | Type | Description |
|---|---|---|
| params | Object | |
| params.bucket | string | |
| params.key | string | |
| params.s3 | AWS.S3 | an S3 client instance |
Promise.<AWS.S3.GetObjectOutput>Gets an object from S3.
Kind: inner method of S3
Returns: Promise.<AWS.S3.GetObjectOutput> - response from AWS.S3.getObject()
as a Promise
| Param | Type | Description |
|---|---|---|
| s3Client | AWS.S3 | an AWS.S3 instance |
| params | AWS.S3.GetObjectRequest | parameters object to pass through to AWS.S3.getObject() |
Example
const obj = await getObject(s3(), { Bucket: 'b', Key: 'k' })
Promise.<AWS.S3.GetObjectOutput>Get an object from S3, waiting for it to exist and, if specified, have the correct ETag.
Kind: inner method of S3
| Param | Type | Default |
|---|---|---|
| s3Client | AWS.S3 | |
| params | AWS.S3.GetObjectRequest | |
| [retryOptions] | pRetry.Options | {} |
Promise.<string>Fetch the contents of an S3 object
Kind: inner method of S3
Returns: Promise.<string> - the contents of the S3 object
| Param | Type | Description |
|---|---|---|
| bucket | string | the S3 object's bucket |
| key | string | the S3 object's key |
Promise.<*>Fetch JSON stored in an S3 object
Kind: inner method of S3
Returns: Promise.<*> - the contents of the S3 object, parsed as JSON
| Param | Type | Description |
|---|---|---|
| bucket | string | the S3 object's bucket |
| key | string | the S3 object's key |
ReadableGet a readable stream for an S3 object
Kind: inner method of S3
| Param | Type | Description |
|---|---|---|
| params | Object | |
| params.s3 | AWS.S3 | an AWS.S3 instance |
| params.bucket | string | the bucket of the requested object |
| params.key | string | the key of the requested object |
PromiseCheck if a file exists in an S3 object
Kind: inner method of S3
Returns: Promise - returns the response from S3.headObject as a promise
| Param | Type | Description |
|---|---|---|
| bucket | string | name of the S3 bucket |
| key | string | key of the file in the S3 bucket |
PromiseDelete files from S3
Kind: inner method of S3
Returns: Promise - A promise that resolves to an Array of the data returned
from the deletion operations
| Param | Type | Description |
|---|---|---|
| s3Objs | Array | An array of objects containing keys 'Bucket' and 'Key' |
PromiseDelete a list of buckets and all of their objects from S3
Kind: inner method of S3
Returns: Promise - the promised result of S3.deleteBucket
| Param | Type | Description |
|---|---|---|
| buckets | Array | list of bucket names |
PromiseUpload the file associated with the given stream to an S3 bucket
Kind: inner method of S3
Returns: Promise - A promise
| Param | Type | Description |
|---|---|---|
| fileStream | ReadableStream | The stream for the file's contents |
| bucket | string | The S3 bucket to which the file is to be uploaded |
| key | string | The key to the file in the bucket |
| s3opts | Object | Options to pass to the AWS sdk call (defaults to {}) |
PromiseList the objects in an S3 bucket
Kind: inner method of S3
Returns: Promise - A promise that resolves to the list of objects. Each S3
object is represented as a JS object with the following attributes: Key,
ETag, LastModified, Owner, Size, StorageClass.
| Param | Type | Default | Description |
|---|---|---|---|
| bucket | string | The name of the bucket | |
| prefix | string | Only objects with keys starting with this prefix will be included (useful for searching folders in buckets, e.g., '/PDR') | |
| skipFolders | boolean | true | If true don't return objects that are folders (defaults to true) |
Calculate the cryptographic hash of an S3 object
Kind: inner method of S3
| Param | Type | Description |
|---|---|---|
| params | Object | |
| params.s3 | AWS.S3 | an AWS.S3 instance |
| params.algorithm | string | cksum, or an algorithm listed in openssl list -digest-algorithms |
| params.bucket | string | |
| params.key | string |
Promise.<boolean>Validate S3 object checksum against expected sum
Kind: inner method of S3
Returns: Promise.<boolean> - returns true for success
Throws:
InvalidChecksum - Throws error if validation fails| Param | Type | Description |
|---|---|---|
| params | Object | params |
| params.algorithm | string | checksum algorithm |
| params.bucket | string | S3 bucket |
| params.key | string | S3 key |
| params.expectedSum | number | string | expected checksum |
| [params.options] | Object | crypto.createHash options |
Array.<string>Extract the S3 bucket and key from the URL path parameters
Kind: inner method of S3
Returns: Array.<string> - [Bucket, Key]
| Param | Type | Description |
|---|---|---|
| pathParams | string | path parameters from the URL bucket/key in the form of |
PromiseCreate an S3 bucket
Kind: inner method of S3
| Param | Type | Description |
|---|---|---|
| Bucket | string | the name of the S3 bucket to create |
PromiseCreate multiple S3 buckets
Kind: inner method of S3
| Param | Type | Description |
|---|---|---|
| buckets | Array.<string> | the names of the S3 buckets to create |
Promise.<{etag: string}>Copy an S3 object to another location in S3 using a multipart copy
Kind: inner method of S3
Returns: Promise.<{etag: string}> - object containing the ETag of the
destination object
| Param | Type | Default | Description |
|---|---|---|---|
| params | Object | ||
| params.sourceBucket | string | ||
| params.sourceKey | string | ||
| params.destinationBucket | string | ||
| params.destinationKey | string | ||
| [params.sourceObject] | AWS.S3.HeadObjectOutput | Output from https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property | |
| [params.ACL] | string | an S3 Canned ACL | |
| [params.copyTags] | boolean | false | |
| [params.chunkSize] | number | chunk size of the S3 multipart uploads |
Promise.<undefined>Move an S3 object to another location in S3
Kind: inner method of S3
| Param | Type | Default | Description |
|---|---|---|---|
| params | Object | ||
| params.sourceBucket | string | ||
| params.sourceKey | string | ||
| params.destinationBucket | string | ||
| params.destinationKey | string | ||
| [params.ACL] | string | an S3 Canned ACL | |
| [params.copyTags] | boolean | false | |
| [params.chunkSize] | number | chunk size of the S3 multipart uploads |
Promise.<undefined>Publish a message to an SNS topic. Does not catch errors, to allow more specific handling by the caller.
Kind: inner method of SNS
| Param | Type | Description |
|---|---|---|
| snsTopicArn | string | SNS topic ARN |
| message | Object | Message object |
| retryOptions | Object | options to control retry behavior when publishing a message fails. See https://github.com/tim-kos/node-retry#retryoperationoptions |
PromisePromisePromise.<string>Promise.<Array>Promise.<boolean>PromiseSend a message to AWS SQS
Kind: static property of SQS
Returns: Promise - resolves when the messsage has been sent
| Param | Type | Description |
|---|---|---|
| queueUrl | string | url of the SQS queue |
| message | string | Object | either string or object message. If an object it will be serialized into a JSON string. |
| [logOverride] | Logger | optional Logger passed in for testing |
PromiseDelete a given SQS message from a given queue.
Kind: static property of SQS
Returns: Promise - an AWS SQS response
| Param | Type | Description |
|---|---|---|
| queueUrl | string | url of the SQS queue |
| receiptHandle | integer | the unique identifier of the sQS message |
Promise.<string>Create an SQS Queue. Properly handles localstack queue URLs
Kind: static method of SQS
Returns: Promise.<string> - the Queue URL
| Param | Type | Description |
|---|---|---|
| QueueName | string | queue name |
Promise.<Array>Receives SQS messages from a given queue. The number of messages received can be set and the timeout is also adjustable.
Kind: inner method of SQS
Returns: Promise.<Array> - an array of messages
| Param | Type | Default | Description |
|---|---|---|---|
| queueUrl | string | url of the SQS queue | |
| options | Object | options object | |
| [options.numOfMessages] | integer | 1 | number of messages to read from the queue |
| [options.visibilityTimeout] | integer | 30 | number of seconds a message is invisible after read |
| [options.waitTimeSeconds] | integer | 0 | number of seconds to poll SQS queue (long polling) |
Promise.<boolean>Test if an SQS queue exists
Kind: inner method of SQS
Returns: Promise.<boolean> - - a Promise that will resolve to a boolean indicating
if the queue exists
| Param | Type | Description |
|---|---|---|
| queueUrl | Object | queue url |
Promise.<Object>Promise.<Object>Promise.<Object>Promise.<Object>Promise.<boolean>Promise.<Object>Call StepFunctions DescribeExecution
See StepFunctions.describeExecution()
for descriptions of params and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
| Param | Type |
|---|---|
| params | Object |
Promise.<Object>Call StepFunctions DescribeStateMachine
See StepFunctions.describeStateMachine()
for descriptions of params and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
| Param | Type |
|---|---|
| params | Object |
Promise.<Object>Call StepFunctions GetExecutionHistory
See StepFunctions.getExecutionHistory()
for descriptions of params and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
| Param | Type |
|---|---|
| params | Object |
Promise.<Object>Call StepFunctions ListExecutions
See StepFunctions.listExecutions()
for descriptions of params and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
| Param | Type |
|---|---|
| params | Object |
Promise.<boolean>Check if a Step Function Execution exists
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: inner method of StepFunctions
| Param | Type | Description |
|---|---|---|
| executionArn | string | the ARN of the Step Function Execution to check for |
Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.
Kind: global class
Promise.<Array>Drain all values from the searchQueue, and return to the user. Warning: This can be very memory intensive.
Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Array> - array of search results.
Promise.<Object>View the next item in the queue
This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.
Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object> - an item from the DynamoDB table
Promise.<Object>Remove the next item from the queue
When there are no more items in the queue, returns 'null'.
Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object> - an item from the DynamoDB table
Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.
Kind: global class
Promise.<Object>View the next item in the queue
This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.
Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object> - an S3 object description
Promise.<Object>Remove the next item from the queue
When there are no more items in the queue, returns 'null'.
Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object> - an S3 object description
Class to use when interacting with S3
Kind: global class
Promise.<string>Promise.<string>Promise.<string>Returns an HTTPS URL that can be used to perform a GET on the given object store URL
Kind: instance method of S3ObjectStore
Returns: Promise.<string> - a signed URL
Throws:
| Param | Type | Description |
|---|---|---|
| objectUrl | string | the URL of the object to sign |
| [options] | string | options to pass to S3.getObject |
| [queryParams] | string | a mapping of parameter key/values to put in the URL |
Promise.<string>Returns an HTTPS URL that can be used to perform a HEAD on the given object store URL
Kind: instance method of S3ObjectStore
Returns: Promise.<string> - a signed URL
Throws:
| Param | Type | Description |
|---|---|---|
| objectUrl | string | the URL of the object to sign |
| [options] | string | options to pass to S3.getObject |
| [queryParams] | string | a mapping of parameter key/values to put in the URL |
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
To make a contribution, please see our contributing guidelines.
Generated automatically using npm run build-docs
[v11.0.0] 2022-03-24 [STABLE]
Release v11.0 is a maintenance release series, replacing v9.9. If you are upgrading to or past v11 from v9.9.x to this release, please pay attention to the following migration notes from prior releases:
data-persistence module, but before deploying the main cumulus module/rules/<name> endpoint, the rule records in PostgreSQL may be
out of sync with records in DynamoDB. In order to bring the records into sync, re-deploy and re-run the
data-migration1 Lambda with a payload of
{"forceRulesMigration": true}:aws lambda invoke --function-name $PREFIX-data-migration1 \
--payload $(echo '{"forceRulesMigration": true}' | base64) $OUTFILE
cumulus deploymenttask-config for all workflows that use the sync-granule task to include workflowStartTime set to
{$.cumulus_meta.workflow_start_time}. See here for an example.cumulus deploymentAs part of the work on the RDS Phase 2 feature, it was decided to re-add the
granule file type property on the file table (detailed reasoning
https://wiki.earthdata.nasa.gov/pages/viewpage.action?pageId=219186829). This
change was implemented as part of CUMULUS-2672/CUMULUS-2673, however granule
records ingested prior to v11 will not have the file.type property stored in the
PostGreSQL database, and on installation of v11 API calls to get granule.files
will not return this value. We anticipate most users are impacted by this issue.
Users that are impacted by these changes should re-run the granule migration lambda to only migrate granule file records:
PAYLOAD=$(echo '{"migrationsList": ["granules"], "granuleMigrationParams": {"migrateOnlyFiles": "true"}}' | base64)
aws lambda invoke --function-name $PREFIX-postgres-migration-async-operation \
--payload $PAYLOAD $OUTFILE
You should note that this will only move files for granule records in PostgreSQL. If you have not completed the phase 1 data migration or have granule records in dynamo that are not in PostgreSQL, the migration will report failure for both the DynamoDB granule and all the associated files and the file records will not be updated.
If you prefer to do a full granule and file migration, you may instead
opt to run the migration with the migrateAndOverwrite option instead, this will re-run a
full granule/files migration and overwrite all values in the PostgreSQL database from
what is in DynamoDB for both granules and associated files:
PAYLOAD=$(echo '{"migrationsList": ["granules"], "granuleMigrationParams": {"migrateAndOverwrite": "true"}}' | base64)
aws lambda invoke --function-name $PREFIX-postgres-migration-async-operation \
--payload $PAYLOAD $OUTFILE
Please note: Since this data migration is copying all of your granule data
from DynamoDB to PostgreSQL, it can take multiple hours (or even days) to run,
depending on how much data you have and how much parallelism you configure the
migration to use. In general, the more parallelism you configure the migration
to use, the faster it will go, but the higher load it will put on your
PostgreSQL database. Excessive database load can cause database outages and
result in data loss/recovery scenarios. Thus, the parallelism settings for the
migration are intentionally set by default to conservative values but are
configurable. If this impacts only some of your data products you may want
to consider using other granuleMigrationParams.
Please see the second data migration docs for more on this tool if you are unfamiliar with the various options.
ORCA Backup is now a supported reportType for the POST /reconciliationReports endpoint@cumulus/message/utils.parseException to parse exception objects@cumulus/message/Granules:
getGranuleProductVolumegetGranuleTimeToPreprocessgetGranuleTimeToArchivegenerateGranuleApiRecord@cumulus/message/PDRs/generatePdrApiRecordFromMessage to generate PDR from Cumulus workflow message@cumulus/es-client/indexer:
deleteAsyncOperation to delete async operation records from ElasticsearchupdateAsyncOperation to update an async operation record in ElasticsearchPUT endpoint to Cumulus API for updating a granule.
Requests to this endpoint should be submitted without an action
attribute in the request body.@cumulus/api-client/granules.updateGranule to update granule via the API@cumulus/db/translate/providers@cumulus/db/translate/collectionssearchWithUpdatedAtRange method to
@cumulus/db/models/collectionsdeletePdr to @cumulus/api-client/pdrsmove action to update granules in the index
and utilize postgres as the authoritative datastore@cumulus/db/base.deleteExcluding method to allow for deletion of a
record set with an exclusion list of cumulus_ids@cumulus/db/getFilesAndGranuleInfoQuery() to build a query for searching file
records in PostgreSQL and return specified granule information for each file@cumulus/db/QuerySearchClient library to handle sequentially fetching and paging
through results for an arbitrary PostgreSQL queryinsert method to all @cumulus/db models to handle inserting multiple records into
the database at once@cumulus/db/translatePostgresGranuleResultToApiGranule helper to
translate custom PostgreSQL granule result to API granuletype text column to Postgres database files table@cumulus/es-client/indexer.upsertExecution to upsert an execution@cumulus/es-client/indexer.upsertPdr to upsert a PDR@cumulus/es-client/indexer.upsertGranule to upsert a granuleexecution_sns_topic_arn environment variable to
sf_event_sqs_to_db_records lambda TF definition.sf_event_sqs_to_db_records_lambda IAM policy to include
permissions for SNS publish for report_executions_topiccollection_sns_topic_arn environment variable to
PrivateApiLambda and ApiEndpoints lambdas.updateCollection to @cumulus/api-client.ecs_cluster IAM policy to include permissions for SNS publish
for report_executions_sns_topic_arn, report_pdrs_sns_topic_arn,
report_granules_sns_topic_arnprocess_dead_letter_archive.tfbulk_operation.tfpdr_sns_topic_arn environment variable to
sf_event_sqs_to_db_records lambda TF definition.publishSnsMessageByDataType in @cumulus/api to
publish SNS messages to the report topics to PDRs, Collections, and
Executions.publishSnsMessageUtils to handle
publishing SNS messages for specific data and event types:
publishCollectionUpdateSnsMessagepublishCollectionCreateSnsMessagepublishCollectionDeleteSnsMessagepublishGranuleUpdateSnsMessagepublishGranuleDeleteSnsMessagepublishGranuleCreateSnsMessagepublishExecutionSnsMessagepublishPdrSnsMessagepublishGranuleSnsMessageByEventTypeecs_cluster IAM policy to include permissions for SNS publish
for report_executions_topic and report_pdrs_topic.paginateByCumulusId to @cumulus/db BasePgModel to allow for paginated
full-table select queries in support of elasticsearch indexing.getMaxCumulusId to @cumulus/db BasePgModel to allow all
derived table classes to support querying the current max cumulus_id.ES_HOST environment variable to postgres-migration-async-operation
Lambda using value of elasticsearch_hostname Terraform variable.elasticsearch_security_group_id to security groups for
postgres-migration-async-operation lambda.DynamoDb:DeleteItem to
postgres-migration-async-operation lambda.async_operation_image in
tf-modules/cumulus/variables.tf to cumuluss/async-operation:41ES_HOST environment variable to async operation ECS task
definition to ensure that async operation tasks write to the correct
Elasticsearch domain@cumulus/api/lambdas/reports/orca-backup-reconciliation-report to create
ORCA Backup reconciliation reportdbIndexer Lambda for DynamoDB tables:
<prefix>-AsyncOperationsTable<prefix>-CollectionsTable<prefix>-ExecutionsTable<prefix>-GranulesTable<prefix>-PdrsTable<prefix>-ProvidersTable<prefix>-RulesTable@ingest/granule.moveGranuleFileswaitForModelStatus from example/spec/helpers/apiUtils integration test helpersstream_enabled and stream_view_type from executions_table TF
definition.aws_lambda_event_source_mapping TF definition on executions
DynamoDB table.stream_enabled and stream_view_type from collections_table
TF definition.aws_lambda_event_source_mapping TF definition on collections
DynamoDB table.publish_collections TF resource.aws_lambda_event_source_mapping TF definition on granulesstream_enabled and stream_view_type from pdrs_table TF
definition.aws_lambda_event_source_mapping TF definition on PDRs
DynamoDB table.@cumulus/api/models/granules.storeGranulesFromCumulusMessage() methodaddToLocalES in POST /granules endpoint since it is
redundant.addToLocalES in POST and PUT /executions endpoints
since it is redundant.addToLocalES from es-client package since it is no
longer used._updateGranuleStatus to update granule to "running" from @cumulus/api/lib/ingest.reingestGranule
and @cumulus/api/lib/ingest.applyWorkflow@cumulus/common to address CVE-2022-2477CUMULUS_2641
@cumulus/message package to set productVolume as string
(calculated with file.size as a BigInt) to match API schema@cumulus/db granule translation to translate granule objects to
match the updated API schemaCUMULUS-2714
CUMULUS-2672
data-migration2 lambda to migrate Dynamo granule.files[].type
instead of dropping it.@cumlus/db translateApiFiletoPostgresFile to retain type@cumulus/db translatePostgresFileToApiFile to retain type@cumulus/types.api.file to add type to the typing.CUMULUS-2315
index-from-database lambda/ECS task and elasticsearch endpoint to read
from PostgreSQL databaseindex-from-database endpoint to add the following configuration
tuning parameters:
CUMULUS-2308
/granules/<granule_id> GET endpoint to return PostgreSQL Granules instead of DynamoDB Granules/granules/<granule_id> PUT endpoint to use PostgreSQL Granule as source rather than DynamoDB GranuleunpublishGranule (used in /granules PUT) to use PostgreSQL Granule as source rather than DynamoDB GranulewaitForApiStatus instead of waitForModelStatusCUMULUS-2302
reingest endpoint to read collection from PostgreSQL
database instead of DynamoDB@cumulus/api unit helper from ./lib to
.test/helpersCUMULUS-2208
@cumulus/api/es/* code to new @cumulus/es-client packagesfEventSqsToDbRecords Lambda now writes following data directly to
Elasticsearch in parallel with writes to DynamoDB/PostgreSQL:
packages/api/lib/granules.getGranuleProductVolume ->
@cumulus/message/Granules.getGranuleProductVolumepackages/api/lib/granules.getGranuleTimeToPreprocess
-> @cumulus/message/Granules.getGranuleTimeToPreprocesspackages/api/lib/granules.getGranuleTimeToArchive ->
@cumulus/message/Granules.getGranuleTimeToArchivepackages/api/models/Granule.generateGranuleRecord
-> @cumulus/message/Granules.generateGranuleApiRecordCUMULUS-2306
api/bin/serve.js) setup code to add cleanup/executions
related recordswaitForModelStatusCUMULUS-2303
CUMULUS-2301
getAsyncOperation to read from PostgreSQL database instead of
DynamoDB.translatePostgresAsyncOperationToApiAsyncOperation function in
@cumulus/db/translate/async-operation.translateApiAsyncOperationToPostgresAsyncOperation function to
ensure that output is properly translated to an object for the
PostgreSQL record for the following cases of output on the incoming API
record:
record.output is a JSON stringified objectrecord.output is a JSON stringified arrayrecord.output is a JSON stringified stringrecord.output is a stringCUMULUS-2317
CUMULUS-2304
CUMULUS-2634
sfEventSqsToDbRecords Lambda to use new upsert helpers for executions, granules, and PDRs
to ensure out-of-order writes are handled correctly when writing to ElasticsearchCUMULUS-2510
@cumulus/api/lib/writeRecords/write-execution to publish SNS
messages after a successful write to Postgres, DynamoDB, and ES.create and upsert in the db model for Executions
to return an array of objects containing all columns of the created or
updated records.@cumulus/api/endpoints/collections to publish an SNS message
after a successful collection delete, update (PUT), create (POST).create and upsert in the db model for Collections
to return an array of objects containing all columns for the created or
updated records.create and upsert in the db model for Granules
to return an array of objects containing all columns for the created or
updated records.@cumulus/api/lib/writeRecords/write-granules to publish SNS
messages after a successful write to Postgres, DynamoDB, and ES.@cumulus/api/lib/writeRecords/write-pdr to publish SNS
messages after a successful write to Postgres, DynamoDB, and ES.CUMULUS-2733
_writeGranuleFiles function creates an aggregate error which
contains the workflow error, if any, as well as any error that may occur
from writing granule files.CUMULUS-2674
DELETE endpoints for the following data types to check that record exists in
PostgreSQL or Elasticsearch before proceeding with deletion:
providerasync operationscollectionsgranulesexecutionsPDRsrulesCUMULUS-2294
CUMULUS-2642
collectionIds, granuleIds, and providers to allow
targeting/filtering of the results.CUMULUS-2694
sfEventSqsToDbRccords to log message if Cumulus
workflow message is from pre-RDS deployment but still attempt parallel writing to DynamoDB
and PostgreSQLsfEventSqsToDbRccords to throw error if requirements to write execution to PostgreSQL cannot be metCUMULUS-2660
/executions endpoint to publish SNS message of created record to executions SNS topicCUMULUS-2661
/executions/<arn> endpoint to publish SNS message of updated record to executions SNS topicCUMULUS-2765
updateGranuleStatusToQueued in write-granules to write to
Elasticsearch and publish SNS message to granules topic.CUMULUS-2774
constructGranuleSnsMessage and constructCollectionSnsMessage
to throw error if eventType is invalid or undefined.CUMULUS-2776
getTableIndexDetails in db-indexer to use correct
deleteFnName for reconciliation reports.CUMULUS-2780
CUMULUS-2778
async_operation_image in tf-modules/cumulus/variables.tf to cumuluss/async-operation:38CUMULUS-2854
createRuleTrigger from create.rulesModel.createRuleTrigger directly to create rule trigger.rulesModel.createRuleTrigger if update fails and reversion needs to occur.GET /executions/status responseFAQs
Utilities for working with AWS
The npm package @cumulus/aws-client receives a total of 605 weekly downloads. As such, @cumulus/aws-client popularity was classified as not popular.
We found that @cumulus/aws-client demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
The Socket Research Team has discovered six new malicious npm packages linked to North Korea’s Lazarus Group, designed to steal credentials and deploy backdoors.
Security News
Socket CEO Feross Aboukhadijeh discusses the open web, open source security, and how Socket tackles software supply chain attacks on The Pair Program podcast.
Security News
Opengrep continues building momentum with the alpha release of its Playground tool, demonstrating the project's rapid evolution just two months after its initial launch.