@cumulus/common/aws - Utilities for working with AWS. For ease of setup, testing, and credential management, code should obtain AWS client objects from helpers in this module.
@cumulus/common/concurrency - Implementations of distributed concurrency primitives (mutex, semaphore) using DynamoDB
@cumulus/common/errors - Classes for thrown errors
@cumulus/common/local-helpers - Provides methods for setting up message payloads for use in development / local testing
@cumulus/common/log - muting or potentially shipping logs to alternative locations
@cumulus/common/string - Utilities for manipulating strings
@cumulus/common/test-utils - Utilities for writing tests
@cumulus/common/URLUtils - a collection of utilities for working with URLs
@cumulus/common/util - Other misc general utilities

Contributing

To make a contribution, please see our contributing guidelines.

[v1.19.0] 2020-02-28

BREAKING CHANGES

CUMULUS-1736
- The @cumulus/discover-granules task now sets the dataType of discovered granules based on the name of the configured collection, not the dataType.
- The config schema of the @cumulus/discover-granules task now requires that collections contain a version.
- The @cumulus/sync-granule task will set the dataType and version of a granule based on the configured collection if those fields are not already set on the granule. Previously it was using the dataType field of the configured collection, then falling back to the name field of the collection. This update will just use the name field of the collection to set the dataType field of the granule.
CUMULUS-1446
- Update the @cumulus/integration-tests/api/executions.getExecution() function to parse the response and return the execution, rather than return the full API response.
CUMULUS-1672
- The cumulus Terraform module in previous releases set a Deployment = var.prefix tag on all resources that it managed. In this release, a tags input variable has been added to the cumulus Terraform module to allow resource tagging to be customized. No default tags will be applied to Cumulus-managed resources. To replicate the previous behavior, set tags = { Deployment: var.prefix } as an input variable for the cumulus Terraform module.
CUMULUS-1684 Migration Instructions
- In previous releases, a provider's username and password were encrypted using a custom encryption library. That has now been updated to use KMS. This release includes a Lambda function named <prefix>-ProviderSecretsMigration, which will re-encrypt existing provider credentials to use KMS. After this release has been deployed, you will need to manually invoke that Lambda function using either the AWS CLI or AWS Console. It should only need to be successfully run once.
- Future releases of Cumulus will invoke a <prefix>-VerifyProviderSecretsMigration Lambda function as part of the deployment, which will cause the deployment to fail if the migration Lambda has not been run.
CUMULUS-1718
- The @cumulus/sf-sns-report task for reporting mid-workflow updates has been retired. This task was used as the PdrStatusReport task in our ParsePdr example workflow. If you have a ParsePdr or other workflow using this task, use @cumulus/sf-sqs-report instead. Trying to deploy the old task will result in an error as the cumulus module no longer exports sf_sns_report_task.
- Migration instruction: In your workflow definition, for each step using the old task change: "Resource": "${module.cumulus.sf_sns_report_task.task_arn}" to "Resource": "${module.cumulus.sf_sqs_report_task.task_arn}"
CUMULUS-1755
- The thin_egress_jwt_secret_name variable for the tf-modules/cumulus Terraform module is now required. This variable is passed on to the Thin Egress App in tf-modules/distribution/main.tf, which uses the keys stored in the secret to sign JWTs. See the Thin Egress App documentation on how to create a value for this secret.

Added

CUMULUS-1446
- Add @cumulus/common/FileUtils.readJsonFile() function
- Add @cumulus/common/FileUtils.readTextFile() function
- Add @cumulus/integration-tests/api/collections.createCollection() function
- Add @cumulus/integration-tests/api/collections.deleteCollection() function
- Add @cumulus/integration-tests/api/collections.getCollection() function
- Add @cumulus/integration-tests/api/providers.getProvider() function
- Add @cumulus/integration-tests/index.getExecutionOutput() function
- Add @cumulus/integration-tests/index.loadCollection() function
- Add @cumulus/integration-tests/index.loadProvider() function
- Add @cumulus/integration-tests/index.readJsonFilesFromDir() function
CUMULUS-1672
- Add a tags input variable to the archive Terraform module
- Add a tags input variable to the cumulus Terraform module
- Add a tags input variable to the cumulus_ecs_service Terraform module
- Add a tags input variable to the data-persistence Terraform module
- Add a tags input variable to the distribution Terraform module
- Add a tags input variable to the ingest Terraform module
- Add a tags input variable to the s3-replicator Terraform module
CUMULUS-1707
- Enable logrotate on ECS cluster
CUMULUS-1684
- Add a @cumulus/aws-client/KMS library of KMS-related functions
- Add @cumulus/aws-client/S3.getTextObject()
- Add @cumulus/sftp-client package
- Create ProviderSecretsMigration Lambda function
- Create VerifyProviderSecretsMigration Lambda function
CUMULUS-1548
- Add ability to put default Cumulus logs in Metrics' ELK stack
- Add ability to add custom logs to Metrics' ELK Stack
CUMULUS-1702
- When logs are sent to Metrics' ELK stack, the logs endpoints will return results from there
CUMULUS-1459
- Async Operations are indexed in Elasticsearch
- To index any existing async operations you'll need to perform an index from database function.
CUMULUS-1717
- Add @cumulus/aws-client/deleteAndWaitForDynamoDbTableNotExists, which deletes a DynamoDB table and waits to ensure the table no longer exists
- Added publishGranules Lambda to handle publishing granule messages to SNS when granule records are written to DynamoDB
- Added @cumulus/api/models/Granule.storeGranulesFromCumulusMessage to store granules from a Cumulus message to DynamoDB
CUMULUS-1718
- Added @cumulus/sf-sqs-report task to allow mid-workflow reporting updates.
- Added stepfunction_event_reporter_queue_url and sf_sqs_report_task outputs to the cumulus module.
- Added publishPdrs Lambda to handle publishing PDR messages to SNS when PDR records are written to DynamoDB.
- Added @cumulus/api/models/Pdr.storePdrFromCumulusMessage to store PDRs from a Cumulus message to DynamoDB.
- Added @cumulus/aws-client/parseSQSMessageBody to parse an SQS message body string into an object.
Ability to set custom backend API url in the archive module
- Add api_url definition in tf-modules/cumulus/archive.tf
- Add archive_api_url variable in tf-modules/cumulus/variables.tf
CUMULUS-1741
- Added an optional elasticsearch_security_group_ids variable to the data-persistence Terraform module to allow additional security groups to be assigned to the Elasticsearch Domain.
CUMULUS-1752
- Added @cumulus/integration-tests/api/distribution.invokeTEADistributionLambda to simulate a request to the Thin Egress App by invoking the Lambda and getting a response payload.
- Added @cumulus/integration-tests/api/distribution.getTEARequestHeaders to generate necessary request headers for a request to the Thin Egress App
- Added @cumulus/integration-tests/api/distribution.getTEADistributionApiFileStream to get a response stream for a file served by Thin Egress App
- Added @cumulus/integration-tests/api/distribution.getTEADistributionApiRedirect to get a redirect response from the Thin Egress App
CUMULUS-1755
- Added @cumulus/aws-client/CloudFormation.describeCfStack() to describe a Cloudformation stack
- Added @cumulus/aws-client/CloudFormation.getCfStackParameterValues() to get multiple parameter values for a Cloudformation stack

Changed

CUMULUS-1725
- Moved the logic that updates the granule files cache Dynamo table into its own Lambda function called granuleFilesCacheUpdater.
CUMULUS-1736
- The collections model in the API package now determines the name of a collection based on the name property, rather than using dataType and then falling back to name.
- The @cumulus/integration-tests.loadCollection() function no longer appends the postfix to the end of the collection's dataType.
- The @cumulus/integration-tests.addCollections() function no longer appends the postfix to the end of the collection's dataType.
CUMULUS-1672
- Add a retryOptions parameter to the @cumulus/aws-client/S3.headObject function, which will retry if the object being queried does not exist.
CUMULUS-1446
- Mark the @cumulus/integration-tests/api.addCollectionApi() function as deprecated
- Mark the @cumulus/integration-tests/index.listCollections() function as deprecated
- Mark the @cumulus/integration-tests/index.listProviders() function as deprecated
- Mark the @cumulus/integration-tests/index.rulesList() function as deprecated
CUMULUS-1672
- Previously, the cumulus module defaulted to setting a Deployment = var.prefix tag on all resources that it managed. In this release, the cumulus module will now accept a tags input variable that defines the tags to be assigned to all resources that it manages.
- Previously, the data-persistence module defaulted to setting a Deployment = var.prefix tag on all resources that it managed. In this release, the data-persistence module will now accept a tags input variable that defines the tags to be assigned to all resources that it manages.
- Previously, the distribution module defaulted to setting a Deployment = var.prefix tag on all resources that it managed. In this release, the distribution module will now accept a tags input variable that defines the tags to be assigned to all resources that it manages.
- Previously, the ingest module defaulted to setting a Deployment = var.prefix tag on all resources that it managed. In this release, the ingest module will now accept a tags input variable that defines the tags to be assigned to all resources that it manages.
- Previously, the s3-replicator module defaulted to setting a Deployment = var.prefix tag on all resources that it managed. In this release, the s3-replicator module will now accept a tags input variable that defines the tags to be assigned to all resources that it manages.
CUMULUS-1684
- Update the API package to encrypt provider credentials using KMS instead of using RSA keys stored in S3
CUMULUS-1717
- Changed name of cwSfExecutionEventToDb Lambda to cwSfEventToDbRecords
- Updated cwSfEventToDbRecords to write granule records to DynamoDB from the incoming Cumulus message
CUMULUS-1718
- Renamed cwSfEventToDbRecords to sfEventSqsToDbRecords due to architecture change to being a consumer of an SQS queue of Step Function Cloudwatch events.
- Updated sfEventSqsToDbRecords to write PDR records to DynamoDB from the incoming Cumulus message
- Moved data-cookbooks/sns.md to data-cookbooks/ingest-notifications.md and updated it to reflect recent changes.
CUMULUS-1748
- (S)FTP discovery tasks now use the provider-path as-is instead of forcing it to a relative path.
- Improved error handling to catch permission denied FTP errors better and log them properly. Workflows will still fail encountering this error and we intend to consider that approach in a future ticket.
CUMULUS-1752
- Moved class for parsing distribution events to its own file: @cumulus/api/lib/DistributionEvent.js
  - Updated DistributionEvent to properly parse S3 access logs generated by requests from the Thin Egress App
CUMULUS-1753 - Changes to @cumulus/ingest/HttpProviderClient.js:
- Removed regex filter in HttpProviderClient.list() that was used to return only files with an extension between 1 and 4 characters long. HttpProviderClient.list() will now return all files linked from the HTTP provider host.
CUMULUS-1755
- Updated the Thin Egress App module used in tf-modules/distribution/main.tf to build 61. See the release notes.
CUMULUS-1757
- Update @cumulus/cmr-client CMRSearchConceptQueue to take optional cmrEnvironment parameter

Deprecated

CUMULUS-1684
- Deprecate @cumulus/common/key-pair-provider/S3KeyPairProvider
- Deprecate @cumulus/common/key-pair-provider/S3KeyPairProvider.encrypt()
- Deprecate @cumulus/common/key-pair-provider/S3KeyPairProvider.decrypt()
- Deprecate @cumulus/common/kms/KMS
- Deprecate @cumulus/common/kms/KMS.encrypt()
- Deprecate @cumulus/common/kms/KMS.decrypt()
- Deprecate @cumulus/common/sftp.Sftp
CUMULUS-1717
- Deprecate @cumulus/api/models/Granule.createGranulesFromSns
CUMULUS-1718
- Deprecate @cumulus/sf-sns-report.
  - This task has been updated to always throw an error directing the user to use @cumulus/sf-sqs-report instead. This was done because there is no longer an SNS topic to which to publish, and no consumers to listen to it.
CUMULUS-1748
- Deprecate @cumulus/ingest/util.normalizeProviderPath
CUMULUS-1752
- Deprecate @cumulus/integration-tests/api/distribution.getDistributionApiFileStream
- Deprecate @cumulus/integration-tests/api/distribution.getDistributionApiRedirect
- Deprecate @cumulus/integration-tests/api/distribution.invokeApiDistributionLambda

Removed

CUMULUS-1684
- Remove the deployment script that creates encryption keys and stores them to S3
CUMULUS-1768
- Removed API endpoints stats/histogram and stats/average. All advanced stats needs should be acquired from Cloud Metrics or similarly configured ELK stack.

Fixed

Fix default values for urs_url in variables.tf files
- Remove trailing / from default urs_url values.
CUMULUS-1610 - Add the Elasticsearch security group to the EC2 security groups
CUMULUS-1740 - cumulus_meta.workflow_start_time is now set in Cumulus messages
CUMULUS-1753 - Fixed @cumulus/ingest/HttpProviderClient.js to properly handle HTTP providers with:
- Multiple link tags (e.g. <a>) per line of source code
- Link tags in uppercase or lowercase (e.g. <A>)
- Links with filepaths in the link target (e.g. <a href="/path/to/file.txt">). These files will be returned from HTTP file discovery as the file name only (e.g. file.txt).
CUMULUS-1768
- Fix an issue in the stats endpoints in @cumulus/api to send back stats for the correct type

Keywords

FAQs

What is @cumulus/common?

Is @cumulus/common popular?

Is @cumulus/common well maintained?

Package last updated on 02 Mar 2020

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install