@cumulus/common

@cumulus/common

Common libraries used in Cumulus.

Usage

  npm install @cumulus/common

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

General Utilities

• @cumulus/common/aws - Utilities for working with AWS. For ease of setup, testing, and credential management, code should obtain AWS client objects from helpers in this module.
• @cumulus/common/concurrency - Implementations of distributed concurrency primitives (mutex, semaphore) using DynamoDB
• @cumulus/common/errors - Classes for thrown errors
• @cumulus/common/local-helpers - Provides methods for setting up message payloads for use in development / local testing
• @cumulus/common/log - muting or potentially shipping logs to alternative locations
• @cumulus/common/string - Utilities for manipulating strings
• @cumulus/common/test-utils - Utilities for writing tests
• @cumulus/common/URLUtils - a collection of utilities for working with URLs
• @cumulus/common/util - Other misc general utilities

Contributing

To make a contribution, please see our contributing guidelines.

Changelog

[v1.21.0] 2020-03-30

PLEASE NOTE

• CUMULUS-1762: the messageConsumer for sns and kinesis-type rules now fetches the collection information from the message. You should ensure that your rule's collection name and version match what is in the message for these ingest messages to be processed. If no matching rule is found, an error will be thrown and logged in the messageConsumer Lambda function's log group.

Added

• CUMULUS-1629`

  • Updates discover-granules task to respect/utilize duplicateHandling configuration such that
    • skip: Duplicates will be filtered from the granule list
    • error: Duplicates encountered will result in step failure
    • replace, version: Duplicates will be ignored and handled as normal.
  • Adds a new copy of the API lambda PrivateApiLambda() which is configured to not require authentication. This Lambda is not connected to an API gateway
  • Adds @cumulus/api-client with functions for use by workflow lambdas to call the API when needed

• CUMULUS-1732

  • Added Python task/activity workflow and integration test (PythonReferenceSpec) to test cumulus-message-adapter-pythonand cumulus-process-py integration.

• CUMULUS-1795

  • Added an IAM policy on the Cumulus EC2 creation to enable SSM when the deploy_to_ngap flag is true

Changed

• CUMULUS-1762
  • the messageConsumer for sns and kinesis-type rules now fetches the collection information from the message.

Deprecated

• CUMULUS-1629
  • Deprecate granulesApi, rulesApi, emsApi, executionsAPI from @cumulus/integration-test/api in favor of code moved to @cumulus/api-client

Removed

• CUMULUS-1799: Deprecated code removals
  • Removed deprecated method @cumulus/api/models/Granule.createGranulesFromSns()
  • Removed deprecated method @cumulus/api/models/Granule.removeGranuleFromCmr()
  • Removed from @cumulus/common/aws:
    • apigateway()
    • buildS3Uri()
    • calculateS3ObjectChecksum()
    • cf()
    • cloudwatch()
    • cloudwatchevents()
    • cloudwatchlogs()
    • createAndWaitForDynamoDbTable()
    • createQueue()
    • deleteSQSMessage()
    • describeCfStackResources()
    • downloadS3File()
    • downloadS3Files()
    • DynamoDbSearchQueue class
    • dynamodbstreams()
    • ec2()
    • ecs()
    • fileExists()
    • findResourceArn()
    • fromSfnExecutionName()
    • getFileBucketAndKey()
    • getJsonS3Object()
    • getQueueUrl()
    • getObjectSize()
    • getS3ObjectReadStream()
    • getSecretString()
    • getStateMachineArn()
    • headObject()
    • isThrottlingException()
    • kinesis()
    • lambda()
    • listS3Objects()
    • promiseS3Upload()
    • publishSnsMessage()
    • putJsonS3Object()
    • receiveSQSMessages()
    • s3CopyObject()
    • s3GetObjectTagging()
    • s3Join()
    • S3ListObjectsV2Queue class
    • s3TagSetToQueryString()
    • s3PutObjectTagging()
    • secretsManager()
    • sendSQSMessage()
    • sfn()
    • sns()
    • sqs()
    • sqsQueueExists()
    • toSfnExecutionName()
    • uploadS3FileStream()
    • uploadS3Files()
    • validateS3ObjectChecksum()
  • Removed @cumulus/common/CloudFormationGateway class
  • Removed @cumulus/common/concurrency/Mutex class
  • Removed @cumulus/common/errors
  • Removed @cumulus/common/sftp
  • Removed @cumulus/common/string.unicodeEscape
  • Removed @cumulus/cmrjs/cmr-utils.getGranuleId()
  • Removed @cumulus/cmrjs/cmr-utils.getCmrFiles()
  • Removed @cumulus/cmrjs/cmr/CMR class
  • Removed @cumulus/cmrjs/cmr/CMRSearchConceptQueue class
  • Removed @cumulus/cmrjs/utils.getHost()
  • Removed @cumulus/cmrjs/utils.getIp()
  • Removed @cumulus/cmrjs/utils.hostId()
  • Removed @cumulus/cmrjs/utils/ummVersion()
  • Removed @cumulus/cmrjs/utils.updateToken()
  • Removed @cumulus/cmrjs/utils.validateUMMG()
  • Removed @cumulus/ingest/aws.getEndpoint()
  • Removed @cumulus/ingest/aws.getExecutionUrl()
  • Removed @cumulus/ingest/aws/invoke()
  • Removed @cumulus/ingest/aws/CloudWatch class
  • Removed @cumulus/ingest/aws/ECS class
  • Removed @cumulus/ingest/aws/Events class
  • Removed @cumulus/ingest/aws/SQS class
  • Removed @cumulus/ingest/aws/StepFunction class
  • Removed @cumulus/ingest/util.normalizeProviderPath()
  • Removed @cumulus/integration-tests/index.listCollections()
  • Removed @cumulus/integration-tests/index.listProviders()
  • Removed @cumulus/integration-tests/index.rulesList()
  • Removed @cumulus/integration-tests/api/api.addCollectionApi()