@cumulus/common/aws - Utilities for working with AWS. For ease of setup, testing, and credential management, code should obtain AWS client objects from helpers in this module.
@cumulus/common/concurrency - Implementations of distributed concurrency primitives (mutex, semaphore) using DynamoDB
@cumulus/common/errors - Classes for thrown errors
@cumulus/common/local-helpers - Provides methods for setting up message payloads for use in development / local testing
@cumulus/common/log - muting or potentially shipping logs to alternative locations
@cumulus/common/string - Utilities for manipulating strings
@cumulus/common/test-utils - Utilities for writing tests
@cumulus/common/URLUtils - a collection of utilities for working with URLs
@cumulus/common/util - Other misc general utilities

Contributing

See Cumulus README

[v1.14.0] - 2019-08-22

PLEASE NOTE

We have encountered transient lambda service errors in our integration testing. Please handle transient service errors following these guidelines. The workflows in the example/workflows folder have been updated with retries configured for these errors.
CUMULUS-799 added additional IAM permissions to support reading CloudWatch and API Gateway, so you will have to redeploy your IAM stack.
CUMULUS-800 Several items:
- Delete existing API Gateway stages: To allow enabling of API Gateway logging, Cumulus now creates and manages a Stage resource during deployment. Before upgrading Cumulus, it is necessary to delete the API Gateway stages on both the Backend API and the Distribution API. Instructions are included in the documentation under Delete API Gateway Stages.
- Set up account permissions for API Gateway to write to CloudWatch: In a one time operation for your AWS account, to enable CloudWatch Logs for API Gateway, you must first grant the API Gateway permission to read and write logs to CloudWatch for your account. The AmazonAPIGatewayPushToCloudWatchLogs managed policy (with an ARN of arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs) has all the required permissions. You can find a simple how to in the documentation under Enable API Gateway Logging.
- Configure API Gateway to write logs to CloudWatch To enable execution logging for the distribution API set config.yaml apiConfigs.distribution.logApigatewayToCloudwatch value to true. More information Enable API Gateway Logs
- Configure CloudWatch log delivery: It is possible to deliver CloudWatch API execution and access logs to a cross-account shared AWS::Logs::Destination. An operator does this by adding the key logToSharedDestination to the config.yml at the default level with a value of a writable log destination. More information in the documentation under Configure CloudWatch Logs Delivery.
- Additional Lambda Logging: It is now possible to configure any lambda to deliver logs to a shared subscriptions by setting logToSharedDestination to the ARN of a writable location (either an AWS::Logs::Destination or a Kinesis Stream) on any lambda config. Documentation for Lambda Log Subscriptions
- Configure S3 Server Access Logs: If you are running Cumulus in an NGAP environment you may configure S3 Server Access Logs to be delivered to a shared bucket where the Metrics Team will ingest the logs into their ELK stack. Contact the Metrics team for permission and location.
CUMULUS-1368 The Cumulus distribution API has been deprecated and is being replaced by ASF's Thin Egress App. By default, the distribution API will not deploy. Please follow the instructions for deploying and configuring Thin Egress.

To instead continue to deploy and use the legacy Cumulus distribution app, add the following to your config.yml:

deployDistributionApi: true

If you deploy with no distribution app your deployment will succeed but you may encounter errors in your workflows, particularly in the MoveGranule task.

CUMULUS-1418 Users who are packaging the CMA in their Lambdas outside of Cumulus may need to update their Lambda configuration. Please see BREAKING CHANGES below for details.

Added

CUMULUS-642
- Adds Launchpad as an authentication option for the Cumulus API.
- Updated deployment documentation and added instructions to setup Cumulus API Launchpad authentication
CUMULUS-1418
- Adds usage docs/testing of lambda layers (introduced in PR1125), updates Core example tasks to use the updated cumulus-ecs-task and a CMA layer instead of kes CMA injection.
- Added Terraform module to publish CMA as layer to user account.
PR1125 - Adds layers config option to support deploying Lambdas with layers
PR1128 - Added useXRay config option to enable AWS X-Ray for Lambdas.
CUMULUS-1345
- Adds new variables to the app deployment under cmr.
- cmrEnvironment values are SIT, UAT, or OPS with UAT as the default.
- cmrLimit and cmrPageSize have been added as configurable options.
CUMULUS-1273
- Added lambda function EmsProductMetadataReport to generate EMS Product Metadata report
CUMULUS-1226
- Added API endpoint elasticsearch/index-from-database to index to an Elasticsearch index from the database for recovery purposes and elasticsearch/indices-status to check the status of Elasticsearch indices via the API.
CUMULUS-824
- Added new Collection parameter reportToEms to configure whether the collection is reported to EMS
CUMULUS-1357
- Added new BackendApi endpoint ems that generates EMS reports.
CUMULUS-1241
- Added information about queues with maximum execution limits defined to default workflow templates (meta.queueExecutionLimits)
CUMULUS-1311
- Added @cumulus/common/message with various message parsing/preparation helpers
CUMULUS-812
- Added support for limiting the number of concurrent executions started from a queue. See the data cookbook for more information.
CUMULUS-1337
- Adds cumulus.stackName value to the instanceMetadata endpoint.
CUMULUS-1368
- Added cmrGranuleUrlType to the @cumulus/move-granules task. This determines what kind of links go in the CMR files. The options are distribution, s3, or none, with the default being distribution. If there is no distribution API being used with Cumulus, you must set the value to s3 or none.
Added packages/s3-replicator Terraform module to allow same-region s3 replication to metrics bucket.
CUMULUS-1392
- Added tf-modules/report-granules Terraform module which processes granule ingest notifications received via SNS and stores granule data to a database. The module includes:
  - SNS topic for publishing granule ingest notifications
  - Lambda to process granule notifications and store data
  - IAM permissions for the Lambda
  - Subscription for the Lambda to the SNS topic
CUMULUS-1393
- Added tf-modules/report-pdrs Terraform module which processes PDR ingest notifications received via SNS and stores PDR data to a database. The module includes:
  - SNS topic for publishing PDR ingest notifications
  - Lambda to process PDR notifications and store data
  - IAM permissions for the Lambda
  - Subscription for the Lambda to the SNS topic
- Added unit tests for @cumulus/api/models/pdrs.createPdrFromSns()
CUMULUS-1400
- Added tf-modules/report-executions Terraform module which processes workflow execution information received via SNS and stores it to a database. The module includes:
  - SNS topic for publishing execution data
  - Lambda to process and store execution data
  - IAM permissions for the Lambda
  - Subscription for the Lambda to the SNS topic
- Added @cumulus/common/sns-event which contains helpers for SNS events:
  - isSnsEvent() returns true if event is from SNS
  - getSnsEventMessage() extracts and parses the message from an SNS event
  - getSnsEventMessageObject() extracts and parses message object from an SNS event
- Added @cumulus/common/cloudwatch-event which contains helpers for Cloudwatch events:
  - isSfExecutionEvent() returns true if event is from Step Functions
  - isTerminalSfStatus() determines if a Step Function status from a Cloudwatch event is a terminal status
  - getSfEventStatus() gets the Step Function status from a Cloudwatch event
  - getSfEventDetailValue() extracts a Step Function event detail field from a Cloudwatch event
  - getSfEventMessageObject() extracts and parses Step Function detail object from a Cloudwatch event
CUMULUS-1429
- Added tf-modules/data-persistence Terraform module which includes resources for data persistence in Cumulus:
  - DynamoDB tables
  - Elasticsearch with optional support for VPC
  - Cloudwatch alarm for number of Elasticsearch nodes
CUMULUS-1379 CMR Launchpad Authentication
- Added launchpad configuration to @cumulus/deployment/app/config.yml, and cloudformation templates, workflow message, lambda configuration, api endpoint configuration
- Added @cumulus/common/LaunchpadToken and @cumulus/common/launchpad to provide methods to get token and validate token
- Updated lambdas to use Launchpad token for CMR actions (ingest and delete granules)
- Updated deployment documentation and added instructions to setup CMR client for Launchpad authentication

Changed

CUMULUS-1232
- Added retries to update @cumulus/cmr-client updateToken()
CUMULUS-1245 CUMULUS-795
- Added additional ems configuration parameters for sending the ingest reports to EMS
- Added functionality to send daily ingest reports to EMS
CUMULUS-1241
- Removed the concept of "priority levels" and added ability to define a number of maximum concurrent executions per SQS queue
- Changed mapping of Cumulus message properties for the sqs2sfThrottle lambda:
  - Queue name is read from cumulus_meta.queueName
  - Maximum executions for the queue is read from meta.queueExecutionLimits[queueName], where queueName is cumulus_meta.queueName
- Changed sfSemaphoreDown lambda to only attempt decrementing semaphores when:
  - the message is for a completed/failed/aborted/timed out workflow AND
  - cumulus_meta.queueName exists on the Cumulus message AND
  - An entry for the queue name (cumulus_meta.queueName) exists in the the object meta.queueExecutionLimits on the Cumulus message
CUMULUS-1338
- Updated sfSemaphoreDown lambda to be triggered via AWS Step Function Cloudwatch events instead of subscription to sfTracker SNS topic
CUMULUS-1311
- Updated @cumulus/queue-granules to set cumulus_meta.queueName for queued execution messages
- Updated @cumulus/queue-pdrs to set cumulus_meta.queueName for queued execution messages
- Updated sqs2sfThrottle lambda to immediately decrement queue semaphore value if dispatching Step Function execution throws an error
CUMULUS-1362
- Granule processingStartTime and processingEndTime will be set to the execution start time and end time respectively when there is no sync granule or post to cmr task present in the workflow
CUMULUS-1400
- Deprecated @cumulus/ingest/aws/getExecutionArn. Use @cumulus/common/aws/getExecutionArn instead.

Fixed

CUMULUS-1439
- Fix bug with rule.logEventArn deletion on Kinesis rule update and fix unit test to verify
CUMULUS-796
- Added production information (collection ShortName and Version, granuleId) to EMS distribution report
- Added functionality to send daily distribution reports to EMS
CUMULUS-1319
- Fixed a bug where granule ingest times were not being stored to the database
CUMULUS-1356
- The Collection model's delete method now removes the specified item from the collection config store that was inserted by the create method. Previously, this behavior was missing.
CUMULUS-1374
- Addressed audit concerns (https://www.npmjs.com/advisories/782) in api package

BREAKING CHANGES

Changed

CUMULUS-1418
- Adding a default cmaDir key to configuration will cause CUMULUS_MESSAGE_ADAPTER_DIR to be set by default to /opt for any Lambda not setting useCma to true, or explicitly setting the CMA environment variable. In lambdas that package the CMA independently of the Cumulus packaging. Lambdas manually packaging the CMA should have their Lambda configuration updated to set the CMA path, or alternately if not using the CMA as a Lambda layer in this deployment set cmaDir to ./cumulus-message-adapter.

Removed

CUMULUS-1337
- Removes the S3 Access Metrics package added in CUMULUS-799
PR1130
- Removed code deprecated since v1.11.1:
  - Removed @cumulus/common/step-functions. Use @cumulus/common/StepFunctions instead.
  - Removed @cumulus/api/lib/testUtils.fakeFilesFactory. Use @cumulus/api/lib/testUtils.fakeFileFactory instead.
  - Removed @cumulus/cmrjs/cmr functions: searchConcept, ingestConcept, deleteConcept. Use the functions in @cumulus/cmr-client instead.
  - Removed @cumulus/ingest/aws.getExecutionHistory. Use @cumulus/common/StepFunctions.getExecutionHistory instead.

Keywords

FAQs

What is @cumulus/common?

Is @cumulus/common popular?

Is @cumulus/common well maintained?

Package last updated on 22 Aug 2019

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install