@cumulus/aws-client

@cumulus/aws-client

Utilities for working with AWS. These utilities can be used for interacting with live AWS services or Localstack. For ease of setup, testing, and credential management, code interacting with AWS services should use the helpers in this module.

Usage

npm install @cumulus/aws-client

Interacting with Localstack

To use these utilities with Localstack, make sure you have a running instance of Localstack and set this environment variable:

NODE_ENV=test

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Contributing

To make a contribution, please see our contributing guidelines.

Changelog

[v1.21.0] 2020-03-30

PLEASE NOTE

• CUMULUS-1762: the messageConsumer for sns and kinesis-type rules now fetches the collection information from the message. You should ensure that your rule's collection name and version match what is in the message for these ingest messages to be processed. If no matching rule is found, an error will be thrown and logged in the messageConsumer Lambda function's log group.

Added

• CUMULUS-1629`

  • Updates discover-granules task to respect/utilize duplicateHandling configuration such that
    • skip: Duplicates will be filtered from the granule list
    • error: Duplicates encountered will result in step failure
    • replace, version: Duplicates will be ignored and handled as normal.
  • Adds a new copy of the API lambda PrivateApiLambda() which is configured to not require authentication. This Lambda is not connected to an API gateway
  • Adds @cumulus/api-client with functions for use by workflow lambdas to call the API when needed

• CUMULUS-1732

  • Added Python task/activity workflow and integration test (PythonReferenceSpec) to test cumulus-message-adapter-pythonand cumulus-process-py integration.

• CUMULUS-1795

  • Added an IAM policy on the Cumulus EC2 creation to enable SSM when the deploy_to_ngap flag is true

Changed

• CUMULUS-1762
  • the messageConsumer for sns and kinesis-type rules now fetches the collection information from the message.

Deprecated

• CUMULUS-1629
  • Deprecate granulesApi, rulesApi, emsApi, executionsAPI from @cumulus/integration-test/api in favor of code moved to @cumulus/api-client

Removed

• CUMULUS-1799: Deprecated code removals
  • Removed deprecated method @cumulus/api/models/Granule.createGranulesFromSns()
  • Removed deprecated method @cumulus/api/models/Granule.removeGranuleFromCmr()
  • Removed from @cumulus/common/aws:
    • apigateway()
    • buildS3Uri()
    • calculateS3ObjectChecksum()
    • cf()
    • cloudwatch()
    • cloudwatchevents()
    • cloudwatchlogs()
    • createAndWaitForDynamoDbTable()
    • createQueue()
    • deleteSQSMessage()
    • describeCfStackResources()
    • downloadS3File()
    • downloadS3Files()
    • DynamoDbSearchQueue class
    • dynamodbstreams()
    • ec2()
    • ecs()
    • fileExists()
    • findResourceArn()
    • fromSfnExecutionName()
    • getFileBucketAndKey()
    • getJsonS3Object()
    • getQueueUrl()
    • getObjectSize()
    • getS3ObjectReadStream()
    • getSecretString()
    • getStateMachineArn()
    • headObject()
    • isThrottlingException()
    • kinesis()
    • lambda()
    • listS3Objects()
    • promiseS3Upload()
    • publishSnsMessage()
    • putJsonS3Object()
    • receiveSQSMessages()
    • s3CopyObject()
    • s3GetObjectTagging()
    • s3Join()
    • S3ListObjectsV2Queue class
    • s3TagSetToQueryString()
    • s3PutObjectTagging()
    • secretsManager()
    • sendSQSMessage()
    • sfn()
    • sns()
    • sqs()
    • sqsQueueExists()
    • toSfnExecutionName()
    • uploadS3FileStream()
    • uploadS3Files()
    • validateS3ObjectChecksum()
  • Removed @cumulus/common/CloudFormationGateway class
  • Removed @cumulus/common/concurrency/Mutex class
  • Removed @cumulus/common/errors
  • Removed @cumulus/common/sftp
  • Removed @cumulus/common/string.unicodeEscape
  • Removed @cumulus/cmrjs/cmr-utils.getGranuleId()
  • Removed @cumulus/cmrjs/cmr-utils.getCmrFiles()
  • Removed @cumulus/cmrjs/cmr/CMR class
  • Removed @cumulus/cmrjs/cmr/CMRSearchConceptQueue class
  • Removed @cumulus/cmrjs/utils.getHost()
  • Removed @cumulus/cmrjs/utils.getIp()
  • Removed @cumulus/cmrjs/utils.hostId()
  • Removed @cumulus/cmrjs/utils/ummVersion()
  • Removed @cumulus/cmrjs/utils.updateToken()
  • Removed @cumulus/cmrjs/utils.validateUMMG()
  • Removed @cumulus/ingest/aws.getEndpoint()
  • Removed @cumulus/ingest/aws.getExecutionUrl()
  • Removed @cumulus/ingest/aws/invoke()
  • Removed @cumulus/ingest/aws/CloudWatch class
  • Removed @cumulus/ingest/aws/ECS class
  • Removed @cumulus/ingest/aws/Events class
  • Removed @cumulus/ingest/aws/SQS class
  • Removed @cumulus/ingest/aws/StepFunction class
  • Removed @cumulus/ingest/util.normalizeProviderPath()
  • Removed @cumulus/integration-tests/index.listCollections()
  • Removed @cumulus/integration-tests/index.listProviders()
  • Removed @cumulus/integration-tests/index.rulesList()
  • Removed @cumulus/integration-tests/api/api.addCollectionApi()