@cumulus/aws-client
Advanced tools
Utilities for working with AWS. These utilities can be used for interacting with live AWS services or Localstack. For ease of setup, testing, and credential management, code interacting with AWS services should use the helpers in this module.
⚠️ The documented API of this package will not change without a deprecation warning being provided in earlier releases. Code in this package that is not documented in this README may change without warning, and is not considered part of the package's public API.
npm install @cumulus/aws-client
To use these utilities with Localstack, make sure you have a running instance of Localstack and set this environment variable:
NODE_ENV=test
Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.
Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.
Promise.<CloudFormation.Stack>Promise.<CloudFormation.StackResources>Promise.<Object>Promise.<CloudFormation.Stack>Describes a given CloudFormation stack
Kind: static method of CloudFormation
Returns: Promise.<CloudFormation.Stack> - The resources belonging to the stack
| Param | Type | Description |
|---|---|---|
| StackName | string | The name of the CloudFormation stack to query |
Promise.<CloudFormation.StackResources>Describes the resources belonging to a given CloudFormation stack
See CloudFormation.StackResources
Kind: static method of CloudFormation
Returns: Promise.<CloudFormation.StackResources> - The resources belonging to the stack
| Param | Type | Description |
|---|---|---|
| StackName | string | The name of the CloudFormation stack to query |
Promise.<Object>Get parameter values for the given CloudFormation stack
Kind: static method of CloudFormation
Returns: Promise.<Object> - Object keyed by parameter names
| Param | Type | Description |
|---|---|---|
| stackName | string | The name of the CloudFormation stack to query |
| parameterKeys | Array.<string> | Key names for the stack parameters that you want to return |
Promise.<CloudWatchEvents.PutRuleResponse>Create a CloudWatch Events rule
Kind: static method of CloudwatchEvents
| Param | Type | Description |
|---|---|---|
| name | string | the rule name |
| schedule | string | a ScheduleExpression |
| state | string | the state of the rule |
| [description] | string | |
| [role] | string | a Role ARN |
Promise.<Object>Promise.<Object>Promise.<Object>PromisePromise.<Object>Call DynamoDb client get
See DocumentClient.get()
for descriptions of params and the return data.
Kind: static property of DynamoDb
Throws:
RecordDoesNotExist if a record cannot be found| Param | Type | Description |
|---|---|---|
| params | Object | |
| params.tableName | string | Table name to read |
| params.item | AWS.DynamoDB.DocumentClient.Key | Key identifying object to get |
| params.client | AWS.DynamoDB.DocumentClient | Instance of a DynamoDb DocumentClient |
| params.getParams | Object | Additional parameters for DocumentClient.get() |
Promise.<Object>Call DynamoDb client scan
See DocumentClient.scan()
for descriptions of params and the return data.
Kind: static property of DynamoDb
| Param | Type |
|---|---|
| params | Object |
Promise.<Object>Create a DynamoDB table and then wait for the table to exist
Kind: static method of DynamoDb
Returns: Promise.<Object> - the output of the createTable call
| Param | Type | Description |
|---|---|---|
| params | Object | the same params that you would pass to AWS.createTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#createTable-property |
PromiseDelete a DynamoDB table and then wait for the table to not exist
Kind: static method of DynamoDb
| Param | Type | Description |
|---|---|---|
| params | Object | the same params that you would pass to AWS.deleteTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#deleteTable-property |
Promise.<Object>Promise.<string>stringPromise.<Object>Create a KMS key
See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/KMS.html#createKey-property for allowed params and return value.
Kind: static method of KMS
| Param | Type |
|---|---|
| params | Object |
Promise.<string>Encrypt a string using KMS
Kind: static method of KMS
Returns: Promise.<string> - the Base 64 encoding of the encrypted value
| Param | Type | Description |
|---|---|---|
| KeyId | string | the KMS key to use for encryption |
| Plaintext | string | the string to be encrypted |
stringDecrypt a KMS-encrypted string, Base 64 encoded
Kind: static method of KMS
Returns: string - the plaintext
| Param | Type | Description |
|---|---|---|
| ciphertext | string | a KMS-encrypted value, Base 64 encoded |
Promise.<Object>Describe a Kinesis stream.
Kind: static method of Kinesis
Returns: Promise.<Object> - The stream description response
| Param | Type | Description |
|---|---|---|
| params | Object | |
| params.StreamName | string | A Kinesis stream name |
| retryOptions | Object | Options passed to p-retry module |
Promise.<AWS.Lambda.InvocationResponse>Invoke a Lambda function
Kind: static method of Lambda
| Param | Type | Description |
|---|---|---|
| name | string | Lambda function name |
| payload | any | the payload to the Lambda function |
| type | string | the invocation type |
PromisePromisePromisePromise.<AWS.S3.GetObjectTaggingOutput>PromisePromiseReadableStreamPromisePromise.<(number|string)>stringObjectstringstringPromise.<boolean>Promise.<undefined>PromisePromise.<string>Promise.<(number|undefined)>Promise.<string>Promise.<*>ReadableReadableStreamPromisePromisePromisePromisePromise.<Array>Promise.<boolean>Array.<string>PromisePromise.<{etag: string}>Promise.<undefined>Delete an object from S3
Kind: static property of S3
| Param | Type | Description |
|---|---|---|
| bucket | string | bucket where the object exists |
| key | string | key of the object to be deleted promise of the object being deleted |
PromiseGet an object header from S3
Kind: static property of S3
Returns: Promise - returns response from S3.headObject as a promise
| Param | Type | Description |
|---|---|---|
| Bucket | string | name of bucket |
| Key | string | key for object (filepath + filename) |
| retryOptions | Object | options to control retry behavior when an object does not exist. See https://github.com/tim-kos/node-retry#retryoperationoptions By default, retries will not be performed |
Put an object on S3
Kind: static property of S3
| Param | Type | Description |
|---|---|---|
| params | Object | same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property promise of the object being put |
PromiseCopy an object from one location on S3 to another
Kind: static property of S3
Returns: Promise - promise of the object being copied
| Param | Type | Description |
|---|---|---|
| params | Object | same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property |
PromiseUpload data to S3
Note: This is equivalent to calling aws.s3().upload(params).promise()
Kind: static property of S3
Returns: Promise - see S3.upload()
| Param | Type | Description |
|---|---|---|
| params | Object | see S3.upload() |
Promise.<AWS.S3.GetObjectTaggingOutput>Get object Tagging from S3
Kind: static property of S3
Returns: Promise.<AWS.S3.GetObjectTaggingOutput> - the promised response from S3.getObjectTagging
| Param | Type | Description |
|---|---|---|
| bucket | string | name of bucket |
| key | string | key for object (filepath + filename) |
PromisePuts object Tagging in S3 https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObjectTagging-property
Kind: static property of S3
Returns: Promise - returns response from S3.getObjectTagging as a promise
| Param | Type | Description |
|---|---|---|
| Bucket | string | name of bucket |
| Key | string | key for object (filepath + filename) |
| Tagging | Object | tagging object |
PromiseGet an object from S3
Kind: static property of S3
Returns: Promise - returns response from S3.getObject as a promise
| Param | Type | Description |
|---|---|---|
| Bucket | string | name of bucket |
| Key | string | key for object (filepath + filename) |
| retryOptions | Object | options to control retry behavior when an object does not exist. See https://github.com/tim-kos/node-retry#retryoperationoptions By default, retries will not be performed |
ReadableStreamDeprecated
Get a readable stream for an S3 object.
Kind: static property of S3
Throws:
Error if S3 object cannot be found| Param | Type | Description |
|---|---|---|
| bucket | string | the S3 object's bucket |
| key | string | the S3 object's key |
PromiseDelete a bucket and all of its objects from S3
Kind: static property of S3
Returns: Promise - the promised result of S3.deleteBucket
| Param | Type | Description |
|---|---|---|
| bucket | string | name of the bucket |
Promise.<(number|string)>Deprecated
Calculate checksum for S3 Object
Kind: static property of S3
Returns: Promise.<(number|string)> - calculated checksum
| Param | Type | Description |
|---|---|---|
| params | Object | params |
| params.algorithm | string | checksum algorithm |
| params.bucket | string | S3 bucket |
| params.key | string | S3 key |
| [params.options] | Object | crypto.createHash options |
stringJoin strings into an S3 key without a leading slash
Kind: static method of S3
Returns: string - the full S3 key
| Param | Type | Description |
|---|---|---|
| args | string | Array.<string> | the strings to join |
Objectparse an s3 uri to get the bucket and key
Kind: static method of S3
Returns: Object - Returns an object with Bucket and Key properties
| Param | Type | Description |
|---|---|---|
| uri | string | must be a uri with the s3:// protocol |
stringGiven a bucket and key, return an S3 URI
Kind: static method of S3
Returns: string - an S3 URI
| Param | Type | Description |
|---|---|---|
| bucket | string | an S3 bucket name |
| key | string | an S3 key |
stringConvert S3 TagSet Object to query string e.g. [{ Key: 'tag', Value: 'value }] to 'tag=value'
Kind: static method of S3
Returns: string - tags query string
| Param | Type | Description |
|---|---|---|
| tagset | Array.<Object> | S3 TagSet array |
Promise.<boolean>Test if an object exists in S3
Kind: static method of S3
Returns: Promise.<boolean> - a Promise that will resolve to a boolean indicating
if the object exists
| Param | Type | Description |
|---|---|---|
| params | Object | same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property |
Promise.<undefined>Wait for an object to exist in S3
Kind: static method of S3
| Param | Type | Default | Description |
|---|---|---|---|
| params | Object | ||
| params.bucket | string | ||
| params.key | string | ||
| [params.interval] | number | 1000 | interval before retries, in ms |
| [params.timeout] | number | 30000 | timeout, in ms |
PromiseUpload a file to S3
Kind: static method of S3
| Param | Type | Description |
|---|---|---|
| bucket | string | the destination S3 bucket |
| key | string | the destination S3 key |
| filename | filename | the local file to be uploaded |
Promise.<string>Downloads the given s3Obj to the given filename in a streaming manner
Kind: static method of S3
Returns: Promise.<string> - returns filename if successful
| Param | Type | Description |
|---|---|---|
| s3Obj | Object | The parameters to send to S3 getObject call |
| filepath | string | The filepath of the file that is downloaded |
Promise.<(number|undefined)>Get the size of an S3 object
Kind: static method of S3
Returns: Promise.<(number|undefined)> - object size, in bytes
| Param | Type | Description |
|---|---|---|
| params | Object | |
| params.bucket | string | |
| params.key | string | |
| params.s3 | AWS.S3 | an S3 client instance |
Promise.<string>Fetch the contents of an S3 object
Kind: static method of S3
Returns: Promise.<string> - the contents of the S3 object
| Param | Type | Description |
|---|---|---|
| bucket | string | the S3 object's bucket |
| key | string | the S3 object's key |
Promise.<*>Fetch JSON stored in an S3 object
Kind: static method of S3
Returns: Promise.<*> - the contents of the S3 object, parsed as JSON
| Param | Type | Description |
|---|---|---|
| bucket | string | the S3 object's bucket |
| key | string | the S3 object's key |
ReadableGet a readable stream for an S3 object
Kind: static method of S3
| Param | Type | Description |
|---|---|---|
| params | Object | |
| params.s3 | AWS.S3 | an AWS.S3 instance |
| params.bucket | string | the bucket of the requested object |
| params.key | string | the key of the requested object |
ReadableStreamGet a readable stream for an S3 object.
Use getS3Object() before fetching stream to deal
with eventual consistency issues by checking for object
with retries.
Kind: static method of S3
Throws:
Error if S3 object cannot be found| Param | Type | Description |
|---|---|---|
| bucket | string | the S3 object's bucket |
| key | string | the S3 object's key |
PromiseCheck if a file exists in an S3 object
Kind: static method of S3
Returns: Promise - returns the response from S3.headObject as a promise
| Param | Type | Description |
|---|---|---|
| bucket | string | name of the S3 bucket |
| key | string | key of the file in the S3 bucket |
PromiseDelete files from S3
Kind: static method of S3
Returns: Promise - A promise that resolves to an Array of the data returned
from the deletion operations
| Param | Type | Description |
|---|---|---|
| s3Objs | Array | An array of objects containing keys 'Bucket' and 'Key' |
PromiseUpload the file associated with the given stream to an S3 bucket
Kind: static method of S3
Returns: Promise - A promise
| Param | Type | Description |
|---|---|---|
| fileStream | ReadableStream | The stream for the file's contents |
| bucket | string | The S3 bucket to which the file is to be uploaded |
| key | string | The key to the file in the bucket |
| s3opts | Object | Options to pass to the AWS sdk call (defaults to {}) |
PromiseList the objects in an S3 bucket
Kind: static method of S3
Returns: Promise - A promise that resolves to the list of objects. Each S3
object is represented as a JS object with the following attributes: Key,
ETag, LastModified, Owner, Size, StorageClass.
| Param | Type | Description |
|---|---|---|
| bucket | string | The name of the bucket |
| prefix | string | Only objects with keys starting with this prefix will be included (useful for searching folders in buckets, e.g., '/PDR') |
| skipFolders | boolean | If true don't return objects that are folders (defaults to true) |
Promise.<Array>Fetch complete list of S3 objects
listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.
The passed params must be compatible with the listObjectsV2 call.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property
Kind: static method of S3
Returns: Promise.<Array> - resolves to an array of objects corresponding to
the Contents property of the listObjectsV2 response
| Param | Type | Description |
|---|---|---|
| params | Object | params for the s3.listObjectsV2 call |
Calculate the cryptographic hash of an S3 object
Kind: static method of S3
| Param | Type | Description |
|---|---|---|
| params | Object | |
| params.s3 | AWS.S3 | an AWS.S3 instance |
| params.algorithm | string | cksum, or an algorithm listed in openssl list -digest-algorithms |
| params.bucket | string | |
| params.key | string |
Promise.<boolean>Validate S3 object checksum against expected sum
Kind: static method of S3
Returns: Promise.<boolean> - returns true for success
Throws:
InvalidChecksum - Throws error if validation fails| Param | Type | Description |
|---|---|---|
| params | Object | params |
| params.algorithm | string | checksum algorithm |
| params.bucket | string | S3 bucket |
| params.key | string | S3 key |
| params.expectedSum | number | string | expected checksum |
| [params.options] | Object | crypto.createHash options |
Array.<string>Extract the S3 bucket and key from the URL path parameters
Kind: static method of S3
Returns: Array.<string> - [Bucket, Key]
| Param | Type | Description |
|---|---|---|
| pathParams | string | path parameters from the URL bucket/key in the form of |
PromiseCreate an S3 bucket
Kind: static method of S3
| Param | Type | Description |
|---|---|---|
| Bucket | string | the name of the S3 bucket to create |
Promise.<{etag: string}>Copy an S3 object to another location in S3 using a multipart copy
Kind: static method of S3
Returns: Promise.<{etag: string}> - object containing the ETag of the
destination object
| Param | Type | Default | Description |
|---|---|---|---|
| params | Object | ||
| params.sourceBucket | string | ||
| params.sourceKey | string | ||
| params.destinationBucket | string | ||
| params.destinationKey | string | ||
| [params.ACL] | string | an S3 Canned ACL | |
| [params.copyTags] | boolean | false |
Promise.<undefined>Move an S3 object to another location in S3
Kind: static method of S3
| Param | Type | Default | Description |
|---|---|---|---|
| params | Object | ||
| params.sourceBucket | string | ||
| params.sourceKey | string | ||
| params.destinationBucket | string | ||
| params.destinationKey | string | ||
| [params.ACL] | string | an S3 Canned ACL | |
| [params.copyTags] | boolean | false |
Promise.<undefined>Publish a message to an SNS topic. Does not catch errors, to allow more specific handling by the caller.
Kind: static method of SNS
| Param | Type | Description |
|---|---|---|
| snsTopicArn | string | SNS topic ARN |
| message | Object | Message object |
| retryOptions | Object | options to control retry behavior when publishing a message fails. See https://github.com/tim-kos/node-retry#retryoperationoptions |
PromisePromise.<string>PromisePromise.<Array>Promise.<boolean>PromiseDelete a given SQS message from a given queue.
Kind: static property of SQS
Returns: Promise - an AWS SQS response
| Param | Type | Description |
|---|---|---|
| queueUrl | string | url of the SQS queue |
| receiptHandle | integer | the unique identifier of the sQS message |
Promise.<string>Create an SQS Queue. Properly handles localstack queue URLs
Kind: static method of SQS
Returns: Promise.<string> - the Queue URL
| Param | Type | Description |
|---|---|---|
| QueueName | string | queue name |
PromiseSend a message to AWS SQS
Kind: static method of SQS
Returns: Promise - resolves when the messsage has been sent
| Param | Type | Description |
|---|---|---|
| queueUrl | string | url of the SQS queue |
| message | string | Object | either string or object message. If an object it will be serialized into a JSON string. |
Promise.<Array>Receives SQS messages from a given queue. The number of messages received can be set and the timeout is also adjustable.
Kind: static method of SQS
Returns: Promise.<Array> - an array of messages
| Param | Type | Default | Description |
|---|---|---|---|
| queueUrl | string | url of the SQS queue | |
| options | Object | options object | |
| [options.numOfMessages] | integer | 1 | number of messages to read from the queue |
| [options.visibilityTimeout] | integer | 30 | number of seconds a message is invisible after read |
| [options.waitTimeSeconds] | integer | 0 | number of seconds to poll SQS queue (long polling) |
Promise.<boolean>Test if an SQS queue exists
Kind: static method of SQS
Returns: Promise.<boolean> - a Promise that will resolve to a boolean indicating
if the queue exists
| Param | Type | Description |
|---|---|---|
| queue | Object | queue name or url |
Promise.<Object>Promise.<Object>Promise.<boolean>Promise.<Object>Promise.<Object>Promise.<Object>Call StepFunctions DescribeExecution
See StepFunctions.describeExecution()
for descriptions of params and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
| Param | Type |
|---|---|
| params | Object |
Promise.<Object>Call StepFunctions DescribeStateMachine
See StepFunctions.describeStateMachine()
for descriptions of params and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
| Param | Type |
|---|---|
| params | Object |
Promise.<boolean>Check if a Step Function Execution exists
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
| Param | Type | Description |
|---|---|---|
| executionArn | string | the ARN of the Step Function Execution to check for |
Promise.<Object>Call StepFunctions GetExecutionHistory
See StepFunctions.getExecutionHistory()
for descriptions of params and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
| Param | Type |
|---|---|
| params | Object |
Promise.<Object>Call StepFunctions ListExecutions
See StepFunctions.listExecutions()
for descriptions of params and the return data.
If a ThrottlingException is received, this function will retry using an exponential backoff.
Kind: static method of StepFunctions
| Param | Type |
|---|---|
| params | Object |
Class to efficiently search all of the items in a DynamoDB table, without loading them all into memory at once. Handles paging.
Kind: global class
Promise.<Object>View the next item in the queue
This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.
Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object> - an item from the DynamoDB table
Promise.<Object>Remove the next item from the queue
When there are no more items in the queue, returns 'null'.
Kind: instance method of DynamoDbSearchQueue
Returns: Promise.<Object> - an item from the DynamoDB table
Class to efficiently list all of the objects in an S3 bucket, without loading them all into memory at once. Handles paging of listS3ObjectsV2 requests.
Kind: global class
Promise.<Object>View the next item in the queue
This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.
Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object> - an S3 object description
Promise.<Object>Remove the next item from the queue
When there are no more items in the queue, returns 'null'.
Kind: instance method of S3ListObjectsV2Queue
Returns: Promise.<Object> - an S3 object description
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
To make a contribution, please see our contributing guidelines.
Generated automatically using npm run build-docs
[v2.0.0] 2020-07-23
@cumulus/api-client package
CumulusApiClientError class must now be imported using
const { CumulusApiClientError } = require('@cumulus/api-client/CumulusApiClientError')@cumulus/sftp-client/SftpClient class must now be imported using
const { SftpClient } = require('@cumulus/sftp-client');@cumulus/ingest/SftpProviderClient no longer implicitly connect
when download, list, or sync are called. You must call connect on the
provider client before issuing one of those calls. Failure to do so will
result in a "Client not connected" exception being thrown.@cumulus/ingest/SftpProviderClient no longer implicitly
disconnect from the SFTP server when list is called.@cumulus/sftp-client/SftpClient must now be explicitly closed
by calling .end()@cumulus/sftp-client/SftpClient no longer implicitly connect to
the server when download, unlink, syncToS3, syncFromS3, and list are
called. You must explicitly call connect before calling one of those
methods.@cumulus/common package
cloudwatch-event.getSfEventMessageObject() now returns undefined if the
message could not be found or could not be parsed. It previously returned
null.S3KeyPairProvider.decrypt() now throws an exception if the bucket
containing the key cannot be determined.S3KeyPairProvider.decrypt() now throws an exception if the stack cannot be
determined.S3KeyPairProvider.encrypt() now throws an exception if the bucket
containing the key cannot be determined.S3KeyPairProvider.encrypt() now throws an exception if the stack cannot be
determined.sns-event.getSnsEventMessageObject() now returns undefined if it could
not be parsed. It previously returned null.aws module has been removed.BucketsConfig.buckets property is now read-only and privatetest-utils.validateConfig() function now resolves to undefined
rather than true.test-utils.validateInput() function now resolves to undefined rather
than true.test-utils.validateOutput() function now resolves to undefined
rather than true.S3KeyPairProvider.retrieveKey() function has been removed.@cumulus/cmrjs package
@cumulus/cmrjs.constructOnlineAccessUrl() and
@cumulus/cmrjs/cmr-utils.constructOnlineAccessUrl() previously took a
buckets parameter, which was an instance of
@cumulus/common/BucketsConfig. They now take a bucketTypes parameter,
which is a simple object mapping bucket names to bucket types. Example:
{ 'private-1': 'private', 'public-1': 'public' }@cumulus/cmrjs.reconcileCMRMetadata() and
@cumulus/cmrjs/cmr-utils.reconcileCMRMetadata() now take a required
bucketTypes parameter, which is a simple object mapping bucket names to
bucket types. Example: { 'private-1': 'private', 'public-1': 'public' }@cumulus/cmrjs.updateCMRMetadata() and
@cumulus/cmrjs/cmr-utils.updateCMRMetadata() previously took an optional
inBuckets parameter, which was an instance of
@cumulus/common/BucketsConfig. They now take a required bucketTypes
parameter, which is a simple object mapping bucket names to bucket types.
Example: { 'private-1': 'private', 'public-1': 'public' }cumuluss/cumulus-ecs-task Docker image must be updated to
cumuluss/cumulus-ecs-task:1.7.0. This can be done by updating the image
property of any tasks defined using the cumulus_ecs_service Terraform
module.@cumulus/aws-client/S3
getObjectSize function has changed. It now takes a
params object with three properties:
getObjectSize function will no longer retry if the object does not
exist@cumulus/message/Collections.getCollectionIdFromMessage now throws a
CumulusMessageError if collectionName and collectionVersion are missing
from meta.collection. Previously this method would return
'undefined___undefined' instead@cumulus/integration-tests/addCollections now returns an array of collections that
were added rather than the count of added collections@cumulus/common/util.uuid() function has been removed@cumulus/aws-client/S3.multipartCopyObject now returns an object with the
AWS etag of the destination object@cumulus/ingest/S3ProviderClient.list now sets a file object's path
property to undefined instead of null when the file is at the top level
of its bucketsync methods of the following classes in the @cumulus/ingest package
now return an object with the AWS s3uri and etag of the destination file
(they previously returned only a string representing the S3 URI)
FtpProviderClientHttpProviderClientS3ProviderClientSftpProviderClient@cumulus/cmr-js/cmr-utils were made
async, and added distributionBucketMap as a parameter:
DiscoverPdrs task now expects provider_path to be provided at
event.config.provider_path, not event.config.collection.provider_pathevent.config.provider_path is now a required parameter of the
DiscoverPdrs taskevent.config.collection is no longer a parameter to the DiscoverPdrs
taskprovider_path property. The tasks that
relied on that property are now referencing config.meta.provider_path.
Workflows should be updated accordingly./bulkDelete to
/granules/bulkDeletefile.hdf is the filename of the given resource) as the resource description instead of "File to download"Due to an issue with the AWS API Gateway and how the Thin Egress App Cloudformation template applies updates, you may need to redeploy your
thin-egress-app-EgressGateway manually as a one time migration step. If your deployment fails with an
error similar to:
Error: Lambda function (<stack>-tf-TeaCache) returned error: ({"errorType":"HTTPError","errorMessage":"Response code 404 (Not Found)"})
Then follow the AWS
instructions
to Redeploy a REST API to a stage for your egress API and re-run terraform apply.
CUMULUS-2081
CUMULUS-1902
CUMULUS-2058
lambda_processing_role_name as an output from the cumulus module
to provide the processing role nameCUMULUS-1417
checksumFor property to collection files config. Set this
property on a checksum file's definition matching the regex of the target
file. More details in the 'Data Cookbooks
Setup'
documentation.checksumFor validation to collections model.CUMULUS-1956
@cumulus/earthata-login-client package/s3credentials endpoint that is deployed as part of distribution now
supports authentication using tokens created by a different application. If
a request contains the EDL-ClientId and EDL-Token headers,
authentication will be handled using that token rather than attempting to
use OAuth.@cumulus/earthata-login-client.getTokenUsername() now accepts an
xRequestId argument, which will be included as the X-Request-Id header
when calling Earthdata Login.s3Credentials endpoint is invoked with an EDL token and an
X-Request-Id header, that X-Request-Id header will be forwarded to
Earthata Login.CUMULUS-1957
EDL-Client-Name header
is set, @the-client-name will be appended to the end of the Earthdata
Login username that is used as the RoleSessionName of the temporary IAM
credentials. This value will show up in the AWS S3 server access logs.CUMULUS-1958
bucket_map_key to the cumulus
terraform module as an override for the default .yaml values that are passed
to TEA by Core. Using this option requires that each configured
Cumulus 'distribution' bucket (e.g. public/protected buckets) have a single
TEA mapping. Multiple maps per bucket are not supported.distribution_bucket_map.CUMULUS-1961
CUMULUS-1970
add-missing-file-checksums workflow task@cumulus/aws-client/S3.calculateObjectHash() function@cumulus/aws-client/S3.getObjectReadStream() functionCUMULUS-1887
CUMULUS-2019
infix search to es query builder @cumulus/api/es/es/queries to
support partial matching of the keywordsCUMULUS-2032
httpListTimeout to set the default timeout for discovery HTTP/HTTPS
requests, and updates the default for the provider to 5 minutes (300 seconds).CUMULUS-176
CUMULUS-1861
sfEventSqsToDbRecords and
sfEventSqsToDbRecordsInputQueue. Previously failure to write a database
record would result in lambda success, and an error log in the CloudWatch
logs. The lambda has been updated to manually add a record to
the sfEventSqsToDbRecordsDeadLetterQueue if the granule, execution, or
pdr record fails to write, in addition to the previous error logging.CUMULUS-1956
/s3credentials endpoint that is deployed as part of distribution now
supports authentication using tokens created by a different application. If
a request contains the EDL-ClientId and EDL-Token headers,
authentication will be handled using that token rather than attempting to
use OAuth.CUMULUS-1977
/granules/bulk now returns a 202 status on a successful
response instead of a 200 response/granules/<granule-id> now returns a 404 status if the
granule record was already deleted@cumulus/api/models/Granule.update() now returns the updated granule
record/granules/bulkDelete API endpoint to support deleting
granules specified by ID or returned by the provided query in the request
body. If the request is successful, the endpoint returns the async operation
ID that has been started to remove the granules.
@cumulus/api/models/Granule.getRecord() method to return raw record
from DynamoDB@cumulus/api/models/Granule.delete() method which handles deleting
the granule record from DynamoDB and the granule files from S3CUMULUS-1982
globalConnectionLimit property of providers is now optional and
defaults to "unlimited"CUMULUS-1997
launchpad configuration to @cumulus/hyrax-metadata-updates task config schema.CUMULUS-1991
@cumulus/cmrjs/src/cmr-utils/constructOnlineAccessUrls() now throws an error if cmrGranuleUrlType = "distribution" and no distribution endpoint argument is providedCUMULUS-2011
CUMULUS-2016
CUMULUS-1991
DISTRIBUTION_ENDPOINT environment variable for API lambdas. This environment variable is required for API requests to move granules.CUMULUS-1961
@cumulus/api-client@cumulus/aws-client/S3.calculateS3ObjectChecksum()
@cumulus/aws-client/S3.getS3ObjectReadStream()
@cumulus/common/log.convertLogLevel()
@cumulus/collection-config-store
@cumulus/common/util.sleep()
CUMULUS-1930
@cumulus/common/log.convertLogLevel()@cumulus/common/util.isNull()@cumulus/common/util.isUndefined()@cumulus/common/util.negate()@cumulus/common/util.noop()@cumulus/common/util.isNil()@cumulus/common/util.renameProperty()@cumulus/common/util.lookupMimeType()@cumulus/common/util.thread()@cumulus/common/util.mkdtempSync()@cumulus/common.bucketsConfigJsonObject function has been
removed@cumulus/common.CollectionConfigStore class has been removed@cumulus/common.concurrency module has been removed@cumulus/common.constructCollectionId function has been
removed@cumulus/common.launchpad module has been removed@cumulus/common.LaunchpadToken class has been removed@cumulus/common.Semaphore class has been removed@cumulus/common.stringUtils module has been removed@cumulus/common/aws.cloudwatchlogs function has been removed@cumulus/common/aws.deleteS3Files function has been removed@cumulus/common/aws.deleteS3Object function has been removed@cumulus/common/aws.dynamodb function has been removed@cumulus/common/aws.dynamodbDocClient function has been
removed@cumulus/common/aws.getExecutionArn function has been removed@cumulus/common/aws.headObject function has been removed@cumulus/common/aws.listS3ObjectsV2 function has been removed@cumulus/common/aws.parseS3Uri function has been removed@cumulus/common/aws.promiseS3Upload function has been removed@cumulus/common/aws.recursivelyDeleteS3Bucket function has
been removed@cumulus/common/aws.s3CopyObject function has been removed@cumulus/common/aws.s3ObjectExists function has been removed@cumulus/common/aws.s3PutObject function has been removed@cumulus/common/bucketsConfigJsonObject function has been
removed@cumulus/common/CloudWatchLogger class has been removed@cumulus/common/collection-config-store.CollectionConfigStore
class has been removed@cumulus/common/collection-config-store.constructCollectionId
function has been removed@cumulus/common/concurrency.limit function has been removed@cumulus/common/concurrency.mapTolerant function has been
removed@cumulus/common/concurrency.promiseUrl function has been
removed@cumulus/common/concurrency.toPromise function has been
removed@cumulus/common/concurrency.unless function has been removed@cumulus/common/config.parseConfig function has been removed@cumulus/common/config.resolveResource function has been
removed@cumulus/common/DynamoDb.get function has been removed@cumulus/common/DynamoDb.scan function has been removed@cumulus/common/FieldPattern class has been removed@cumulus/common/launchpad.getLaunchpadToken function has been
removed@cumulus/common/launchpad.validateLaunchpadToken function has
been removed@cumulus/common/LaunchpadToken class has been removed@cumulus/common/message.buildCumulusMeta function has been
removed@cumulus/common/message.buildQueueMessageFromTemplate
function has been removed@cumulus/common/message.getCollectionIdFromMessage function
has been removed@cumulus/common/message.getMaximumExecutions function has
been removed@cumulus/common/message.getMessageExecutionArn function has
been removed@cumulus/common/message.getMessageExecutionName function has
been removed@cumulus/common/message.getMessageFromTemplate function has
been removed@cumulus/common/message.getMessageGranules function has been
removed@cumulus/common/message.getMessageStateMachineArn function
has been removed@cumulus/common/message.getQueueName function has been
removed@cumulus/common/message.getQueueNameByUrl function has been
removed@cumulus/common/message.hasQueueAndExecutionLimit function
has been removed@cumulus/common/Semaphore class has been removed@cumulus/common/string.globalReplace function has been removed@cumulus/common/string.isNonEmptyString function has been
removed@cumulus/common/string.isValidHostname function has been
removed@cumulus/common/string.match function has been removed@cumulus/common/string.matches function has been removed@cumulus/common/string.replace function has been removed@cumulus/common/string.toLower function has been removed@cumulus/common/string.toUpper function has been removed@cumulus/common/testUtils.getLocalstackEndpoint function has been removed@cumulus/common/util.setErrorStack function has been removed@cumulus/common/util.uuid function has been removed@cumulus/common/workflows.getWorkflowArn function has been
removed@cumulus/common/workflows.getWorkflowFile function has been
removed@cumulus/common/workflows.getWorkflowList function has been
removed@cumulus/common/workflows.getWorkflowTemplate function has
been removed@cumulus/aws-client/StepFunctions.toSfnExecutionName()@cumulus/aws-client/StepFunctions.fromSfnExecutionName()@cumulus/aws-client/StepFunctions.getExecutionArn()@cumulus/aws-client/StepFunctions.getExecutionUrl()@cumulus/aws-client/StepFunctions.getStateMachineArn()@cumulus/aws-client/StepFunctions.pullStepFunctionEvent()@cumulus/common/test-utils/throttleOnce()@cumulus/integration-tests/api/distribution.invokeApiDistributionLambda()@cumulus/integration-tests/api/distribution.getDistributionApiRedirect()@cumulus/integration-tests/api/distribution.getDistributionApiFileStream()FAQs
Utilities for working with AWS
The npm package @cumulus/aws-client receives a total of 81 weekly downloads. As such, @cumulus/aws-client popularity was classified as not popular.
We found that @cumulus/aws-client demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.