@cumulus/logger - npm Package Versions

[v1.21.0] 2020-03-30

PLEASE NOTE

• CUMULUS-1762: the messageConsumer for sns and kinesis-type rules now fetches the collection information from the message. You should ensure that your rule's collection name and version match what is in the message for these ingest messages to be processed. If no matching rule is found, an error will be thrown and logged in the messageConsumer Lambda function's log group.

Added

• CUMULUS-1629`

  • Updates discover-granules task to respect/utilize duplicateHandling configuration such that
    • skip: Duplicates will be filtered from the granule list
    • error: Duplicates encountered will result in step failure
    • replace, version: Duplicates will be ignored and handled as normal.
  • Adds a new copy of the API lambda PrivateApiLambda() which is configured to not require authentication. This Lambda is not connected to an API gateway
  • Adds @cumulus/api-client with functions for use by workflow lambdas to call the API when needed

• CUMULUS-1732

  • Added Python task/activity workflow and integration test (PythonReferenceSpec) to test cumulus-message-adapter-pythonand cumulus-process-py integration.

• CUMULUS-1795

  • Added an IAM policy on the Cumulus EC2 creation to enable SSM when the deploy_to_ngap flag is true

Changed

• CUMULUS-1762
  • the messageConsumer for sns and kinesis-type rules now fetches the collection information from the message.

Deprecated

• CUMULUS-1629
  • Deprecate granulesApi, rulesApi, emsApi, executionsAPI from @cumulus/integration-test/api in favor of code moved to @cumulus/api-client

Removed

• CUMULUS-1799: Deprecated code removals
  • Removed deprecated method @cumulus/api/models/Granule.createGranulesFromSns()
  • Removed deprecated method @cumulus/api/models/Granule.removeGranuleFromCmr()
  • Removed from @cumulus/common/aws:
    • apigateway()
    • buildS3Uri()
    • calculateS3ObjectChecksum()
    • cf()
    • cloudwatch()
    • cloudwatchevents()
    • cloudwatchlogs()
    • createAndWaitForDynamoDbTable()
    • createQueue()
    • deleteSQSMessage()
    • describeCfStackResources()
    • downloadS3File()
    • downloadS3Files()
    • DynamoDbSearchQueue class
    • dynamodbstreams()
    • ec2()
    • ecs()
    • fileExists()
    • findResourceArn()
    • fromSfnExecutionName()
    • getFileBucketAndKey()
    • getJsonS3Object()
    • getQueueUrl()
    • getObjectSize()
    • getS3ObjectReadStream()
    • getSecretString()
    • getStateMachineArn()
    • headObject()
    • isThrottlingException()
    • kinesis()
    • lambda()
    • listS3Objects()
    • promiseS3Upload()
    • publishSnsMessage()
    • putJsonS3Object()
    • receiveSQSMessages()
    • s3CopyObject()
    • s3GetObjectTagging()
    • s3Join()
    • S3ListObjectsV2Queue class
    • s3TagSetToQueryString()
    • s3PutObjectTagging()
    • secretsManager()
    • sendSQSMessage()
    • sfn()
    • sns()
    • sqs()
    • sqsQueueExists()
    • toSfnExecutionName()
    • uploadS3FileStream()
    • uploadS3Files()
    • validateS3ObjectChecksum()
  • Removed @cumulus/common/CloudFormationGateway class
  • Removed @cumulus/common/concurrency/Mutex class
  • Removed @cumulus/common/errors
  • Removed @cumulus/common/sftp
  • Removed @cumulus/common/string.unicodeEscape
  • Removed @cumulus/cmrjs/cmr-utils.getGranuleId()
  • Removed @cumulus/cmrjs/cmr-utils.getCmrFiles()
  • Removed @cumulus/cmrjs/cmr/CMR class
  • Removed @cumulus/cmrjs/cmr/CMRSearchConceptQueue class
  • Removed @cumulus/cmrjs/utils.getHost()
  • Removed @cumulus/cmrjs/utils.getIp()
  • Removed @cumulus/cmrjs/utils.hostId()
  • Removed @cumulus/cmrjs/utils/ummVersion()
  • Removed @cumulus/cmrjs/utils.updateToken()
  • Removed @cumulus/cmrjs/utils.validateUMMG()
  • Removed @cumulus/ingest/aws.getEndpoint()
  • Removed @cumulus/ingest/aws.getExecutionUrl()
  • Removed @cumulus/ingest/aws/invoke()
  • Removed @cumulus/ingest/aws/CloudWatch class
  • Removed @cumulus/ingest/aws/ECS class
  • Removed @cumulus/ingest/aws/Events class
  • Removed @cumulus/ingest/aws/SQS class
  • Removed @cumulus/ingest/aws/StepFunction class
  • Removed @cumulus/ingest/util.normalizeProviderPath()
  • Removed @cumulus/integration-tests/index.listCollections()
  • Removed @cumulus/integration-tests/index.listProviders()
  • Removed @cumulus/integration-tests/index.rulesList()
  • Removed @cumulus/integration-tests/api/api.addCollectionApi()

[v1.20.0] 2020-03-12

BREAKING CHANGES

• CUMULUS-1714
  • Changed the format of the message sent to the granule SNS Topic. Message includes the granule record under record and the type of event under event. Messages with deleted events will have the record that was deleted with a deletedAt timestamp. Options for event are Create | Update | Delete
• CUMULUS-1769 - deploy_to_ngap is now a required variable for the tf-modules/cumulus module. For those deploying to NGAP environments, this variable should always be set to true.

Notable changes

• CUMULUS-1739 - You can now exclude Elasticsearch from your tf-modules/data-persistence deployment (via include_elasticsearch = false) and your tf-modules/cumulus module will still deploy successfully.

• CUMULUS-1769 - If you set deploy_to_ngap = true for the tf-modules/archive Terraform module, you can only deploy your archive API gateway as PRIVATE, not EDGE.

Added

• Added @cumulus/aws-client/S3.getS3ObjectReadStreamAsync() to deal with S3 eventual consistency issues by checking for the existence an S3 object with retries before getting a readable stream for that object.
• CUMULUS-1769
  • Added deploy_to_ngap boolean variable for the tf-modules/cumulus and tf-modules/archive Terraform modules. This variable is required. For those deploying to NGAP environments, this variable should always be set to true.
• HYRAX-70
  • Add the hyrax-metadata-update task

Changed

• AccessToken.get() now enforces strongly consistent reads from DynamoDB
• CUMULUS-1739
  • Updated tf-modules/data-persistence to make Elasticsearch alarm resources and outputs conditional on the include_elasticsearch variable
  • Updated @cumulus/aws-client/S3.getObjectSize to include automatic retries for any failures from S3.headObject
• CUMULUS-1784
  • Updated @cumulus/api/lib/DistributionEvent.remoteIP() to parse the IP address in an S3 access log from the A-sourceip query parameter if present, otherwise fallback to the original parsing behavior.
• CUMULUS-1768
  • The stats/summary endpoint reports the distinct collections for the number of granules reported

Fixed

• CUMULUS-1739 - Fixed the tf-modules/cumulus and tf-modules/archive modules to make these Elasticsearch variables truly optional:

  • elasticsearch_domain_arn
  • elasticsearch_hostname
  • elasticsearch_security_group_id

• CUMULUS-1768

  • Fixed the stats/ endpoint so that data is correctly filtered by timestamp and processingTime is calculated correctly.

• CUMULUS-1769

  • In the tf-modules/archive Terraform module, the lifecycle block ignoring changes to the policy of the archive API gateway is now only enforced if deploy_to_ngap = true. This fixes a bug where users deploying outside of NGAP could not update their API gateway's resource policy when going from PRIVATE to EDGE, preventing their API from being accessed publicly.

• CUMULUS-1775

  • Fix/update api endpoint to use updated google auth endpoints such that it will work with new accounts

Removed

• CUMULUS-1768
  • Removed API endpoints stats/histogram and stats/average. All advanced stats needs should be acquired from Cloud Metrics or similarly configured ELK stack.

[v1.19.0] 2020-02-28

BREAKING CHANGES

• CUMULUS-1736

  • The @cumulus/discover-granules task now sets the dataType of discovered granules based on the name of the configured collection, not the dataType.
  • The config schema of the @cumulus/discover-granules task now requires that collections contain a version.
  • The @cumulus/sync-granule task will set the dataType and version of a granule based on the configured collection if those fields are not already set on the granule. Previously it was using the dataType field of the configured collection, then falling back to the name field of the collection. This update will just use the name field of the collection to set the dataType field of the granule.

• CUMULUS-1446

  • Update the @cumulus/integration-tests/api/executions.getExecution() function to parse the response and return the execution, rather than return the full API response.

• CUMULUS-1672

  • The cumulus Terraform module in previous releases set a Deployment = var.prefix tag on all resources that it managed. In this release, a tags input variable has been added to the cumulus Terraform module to allow resource tagging to be customized. No default tags will be applied to Cumulus-managed resources. To replicate the previous behavior, set tags = { Deployment: var.prefix } as an input variable for the cumulus Terraform module.

• CUMULUS-1684 Migration Instructions

  • In previous releases, a provider's username and password were encrypted using a custom encryption library. That has now been updated to use KMS. This release includes a Lambda function named <prefix>-ProviderSecretsMigration, which will re-encrypt existing provider credentials to use KMS. After this release has been deployed, you will need to manually invoke that Lambda function using either the AWS CLI or AWS Console. It should only need to be successfully run once.
  • Future releases of Cumulus will invoke a <prefix>-VerifyProviderSecretsMigration Lambda function as part of the deployment, which will cause the deployment to fail if the migration Lambda has not been run.

• CUMULUS-1718

  • The @cumulus/sf-sns-report task for reporting mid-workflow updates has been retired. This task was used as the PdrStatusReport task in our ParsePdr example workflow. If you have a ParsePdr or other workflow using this task, use @cumulus/sf-sqs-report instead. Trying to deploy the old task will result in an error as the cumulus module no longer exports sf_sns_report_task.
  • Migration instruction: In your workflow definition, for each step using the old task change: "Resource": "${module.cumulus.sf_sns_report_task.task_arn}" to "Resource": "${module.cumulus.sf_sqs_report_task.task_arn}"

• CUMULUS-1755

  • The thin_egress_jwt_secret_name variable for the tf-modules/cumulus Terraform module is now required. This variable is passed on to the Thin Egress App in tf-modules/distribution/main.tf, which uses the keys stored in the secret to sign JWTs. See the Thin Egress App documentation on how to create a value for this secret.

Added

• CUMULUS-1446

  • Add @cumulus/common/FileUtils.readJsonFile() function
  • Add @cumulus/common/FileUtils.readTextFile() function
  • Add @cumulus/integration-tests/api/collections.createCollection() function
  • Add @cumulus/integration-tests/api/collections.deleteCollection() function
  • Add @cumulus/integration-tests/api/collections.getCollection() function
  • Add @cumulus/integration-tests/api/providers.getProvider() function
  • Add @cumulus/integration-tests/index.getExecutionOutput() function
  • Add @cumulus/integration-tests/index.loadCollection() function
  • Add @cumulus/integration-tests/index.loadProvider() function
  • Add @cumulus/integration-tests/index.readJsonFilesFromDir() function

• CUMULUS-1672

  • Add a tags input variable to the archive Terraform module
  • Add a tags input variable to the cumulus Terraform module
  • Add a tags input variable to the cumulus_ecs_service Terraform module
  • Add a tags input variable to the data-persistence Terraform module
  • Add a tags input variable to the distribution Terraform module
  • Add a tags input variable to the ingest Terraform module
  • Add a tags input variable to the s3-replicator Terraform module

• CUMULUS-1707

  • Enable logrotate on ECS cluster

• CUMULUS-1684

  • Add a @cumulus/aws-client/KMS library of KMS-related functions
  • Add @cumulus/aws-client/S3.getTextObject()
  • Add @cumulus/sftp-client package
  • Create ProviderSecretsMigration Lambda function
  • Create VerifyProviderSecretsMigration Lambda function

• CUMULUS-1548

  • Add ability to put default Cumulus logs in Metrics' ELK stack
  • Add ability to add custom logs to Metrics' ELK Stack

• CUMULUS-1702

  • When logs are sent to Metrics' ELK stack, the logs endpoints will return results from there

• CUMULUS-1459

  • Async Operations are indexed in Elasticsearch
  • To index any existing async operations you'll need to perform an index from database function.

• CUMULUS-1717

  • Add @cumulus/aws-client/deleteAndWaitForDynamoDbTableNotExists, which deletes a DynamoDB table and waits to ensure the table no longer exists
  • Added publishGranules Lambda to handle publishing granule messages to SNS when granule records are written to DynamoDB
  • Added @cumulus/api/models/Granule.storeGranulesFromCumulusMessage to store granules from a Cumulus message to DynamoDB

• CUMULUS-1718

  • Added @cumulus/sf-sqs-report task to allow mid-workflow reporting updates.
  • Added stepfunction_event_reporter_queue_url and sf_sqs_report_task outputs to the cumulus module.
  • Added publishPdrs Lambda to handle publishing PDR messages to SNS when PDR records are written to DynamoDB.
  • Added @cumulus/api/models/Pdr.storePdrFromCumulusMessage to store PDRs from a Cumulus message to DynamoDB.
  • Added @cumulus/aws-client/parseSQSMessageBody to parse an SQS message body string into an object.

• Ability to set custom backend API url in the archive module

  • Add api_url definition in tf-modules/cumulus/archive.tf
  • Add archive_api_url variable in tf-modules/cumulus/variables.tf

• CUMULUS-1741

  • Added an optional elasticsearch_security_group_ids variable to the data-persistence Terraform module to allow additional security groups to be assigned to the Elasticsearch Domain.

• CUMULUS-1752

  • Added @cumulus/integration-tests/api/distribution.invokeTEADistributionLambda to simulate a request to the Thin Egress App by invoking the Lambda and getting a response payload.
  • Added @cumulus/integration-tests/api/distribution.getTEARequestHeaders to generate necessary request headers for a request to the Thin Egress App
  • Added @cumulus/integration-tests/api/distribution.getTEADistributionApiFileStream to get a response stream for a file served by Thin Egress App
  • Added @cumulus/integration-tests/api/distribution.getTEADistributionApiRedirect to get a redirect response from the Thin Egress App

• CUMULUS-1755

  • Added @cumulus/aws-client/CloudFormation.describeCfStack() to describe a Cloudformation stack
  • Added @cumulus/aws-client/CloudFormation.getCfStackParameterValues() to get multiple parameter values for a Cloudformation stack

Changed

• CUMULUS-1725

  • Moved the logic that updates the granule files cache Dynamo table into its own Lambda function called granuleFilesCacheUpdater.

• CUMULUS-1736

  • The collections model in the API package now determines the name of a collection based on the name property, rather than using dataType and then falling back to name.
  • The @cumulus/integration-tests.loadCollection() function no longer appends the postfix to the end of the collection's dataType.
  • The @cumulus/integration-tests.addCollections() function no longer appends the postfix to the end of the collection's dataType.

• CUMULUS-1672

  • Add a retryOptions parameter to the @cumulus/aws-client/S3.headObject function, which will retry if the object being queried does not exist.

• CUMULUS-1446

  • Mark the @cumulus/integration-tests/api.addCollectionApi() function as deprecated
  • Mark the @cumulus/integration-tests/index.listCollections() function as deprecated
  • Mark the @cumulus/integration-tests/index.listProviders() function as deprecated
  • Mark the @cumulus/integration-tests/index.rulesList() function as deprecated

• CUMULUS-1672

  • Previously, the cumulus module defaulted to setting a Deployment = var.prefix tag on all resources that it managed. In this release, the cumulus module will now accept a tags input variable that defines the tags to be assigned to all resources that it manages.
  • Previously, the data-persistence module defaulted to setting a Deployment = var.prefix tag on all resources that it managed. In this release, the data-persistence module will now accept a tags input variable that defines the tags to be assigned to all resources that it manages.
  • Previously, the distribution module defaulted to setting a Deployment = var.prefix tag on all resources that it managed. In this release, the distribution module will now accept a tags input variable that defines the tags to be assigned to all resources that it manages.
  • Previously, the ingest module defaulted to setting a Deployment = var.prefix tag on all resources that it managed. In this release, the ingest module will now accept a tags input variable that defines the tags to be assigned to all resources that it manages.
  • Previously, the s3-replicator module defaulted to setting a Deployment = var.prefix tag on all resources that it managed. In this release, the s3-replicator module will now accept a tags input variable that defines the tags to be assigned to all resources that it manages.

• CUMULUS-1684

  • Update the API package to encrypt provider credentials using KMS instead of using RSA keys stored in S3

• CUMULUS-1717

  • Changed name of cwSfExecutionEventToDb Lambda to cwSfEventToDbRecords
  • Updated cwSfEventToDbRecords to write granule records to DynamoDB from the incoming Cumulus message

• CUMULUS-1718

  • Renamed cwSfEventToDbRecords to sfEventSqsToDbRecords due to architecture change to being a consumer of an SQS queue of Step Function Cloudwatch events.
  • Updated sfEventSqsToDbRecords to write PDR records to DynamoDB from the incoming Cumulus message
  • Moved data-cookbooks/sns.md to data-cookbooks/ingest-notifications.md and updated it to reflect recent changes.

• CUMULUS-1748

  • (S)FTP discovery tasks now use the provider-path as-is instead of forcing it to a relative path.
  • Improved error handling to catch permission denied FTP errors better and log them properly. Workflows will still fail encountering this error and we intend to consider that approach in a future ticket.

• CUMULUS-1752

  • Moved class for parsing distribution events to its own file: @cumulus/api/lib/DistributionEvent.js
    • Updated DistributionEvent to properly parse S3 access logs generated by requests from the Thin Egress App

• CUMULUS-1753 - Changes to @cumulus/ingest/HttpProviderClient.js:

  • Removed regex filter in HttpProviderClient.list() that was used to return only files with an extension between 1 and 4 characters long. HttpProviderClient.list() will now return all files linked from the HTTP provider host.

• CUMULUS-1755

  • Updated the Thin Egress App module used in tf-modules/distribution/main.tf to build 61. See the release notes.

• CUMULUS-1757

  • Update @cumulus/cmr-client CMRSearchConceptQueue to take optional cmrEnvironment parameter

Deprecated

• CUMULUS-1684

  • Deprecate @cumulus/common/key-pair-provider/S3KeyPairProvider
  • Deprecate @cumulus/common/key-pair-provider/S3KeyPairProvider.encrypt()
  • Deprecate @cumulus/common/key-pair-provider/S3KeyPairProvider.decrypt()
  • Deprecate @cumulus/common/kms/KMS
  • Deprecate @cumulus/common/kms/KMS.encrypt()
  • Deprecate @cumulus/common/kms/KMS.decrypt()
  • Deprecate @cumulus/common/sftp.Sftp

• CUMULUS-1717

  • Deprecate @cumulus/api/models/Granule.createGranulesFromSns

• CUMULUS-1718

  • Deprecate @cumulus/sf-sns-report.
    • This task has been updated to always throw an error directing the user to use @cumulus/sf-sqs-report instead. This was done because there is no longer an SNS topic to which to publish, and no consumers to listen to it.

• CUMULUS-1748

  • Deprecate @cumulus/ingest/util.normalizeProviderPath

• CUMULUS-1752

  • Deprecate @cumulus/integration-tests/api/distribution.getDistributionApiFileStream
  • Deprecate @cumulus/integration-tests/api/distribution.getDistributionApiRedirect
  • Deprecate @cumulus/integration-tests/api/distribution.invokeApiDistributionLambda

Removed

• CUMULUS-1684

  • Remove the deployment script that creates encryption keys and stores them to S3

• CUMULUS-1768

  • Removed API endpoints stats/histogram and stats/average. All advanced stats needs should be acquired from Cloud Metrics or similarly configured ELK stack.

Fixed

• Fix default values for urs_url in variables.tf files

  • Remove trailing / from default urs_url values.

• CUMULUS-1610 - Add the Elasticsearch security group to the EC2 security groups

• CUMULUS-1740 - cumulus_meta.workflow_start_time is now set in Cumulus messages

• CUMULUS-1753 - Fixed @cumulus/ingest/HttpProviderClient.js to properly handle HTTP providers with:

  • Multiple link tags (e.g. <a>) per line of source code
  • Link tags in uppercase or lowercase (e.g. <A>)
  • Links with filepaths in the link target (e.g. <a href="/path/to/file.txt">). These files will be returned from HTTP file discovery as the file name only (e.g. file.txt).

• CUMULUS-1768

  • Fix an issue in the stats endpoints in @cumulus/api to send back stats for the correct type

[v1.18.0] 2020-02-03

BREAKING CHANGES

• CUMULUS-1686

  • ecs_cluster_instance_image_id is now a required variable of the cumulus module, instead of optional.

• CUMULUS-1698

  • Change variable saml_launchpad_metadata_path to saml_launchpad_metadata_url in the tf-modules/cumulus Terraform module.

• CUMULUS-1703

  • Remove the unused forceDownload option from the sync-granule tasks's config
  • Remove the @cumulus/ingest/granule.Discover class
  • Remove the @cumulus/ingest/granule.Granule class
  • Remove the @cumulus/ingest/pdr.Discover class
  • Remove the @cumulus/ingest/pdr.Granule class
  • Remove the @cumulus/ingest/parse-pdr.parsePdr function

Added

• CUMULUS-1040

  • Added @cumulus/aws-client package to provide utilities for working with AWS services and the Node.js AWS SDK
  • Added @cumulus/errors package which exports error classes for use in Cumulus workflow code
  • Added @cumulus/integration-tests/sfnStep to provide utilities for parsing step function execution histories

• CUMULUS-1102

  • Adds functionality to the @cumulus/api package for better local testing.
    • Adds data seeding for @cumulus/api's localAPI.
      • seed functions allow adding collections, executions, granules, pdrs, providers, and rules to a Localstack Elasticsearch and DynamoDB via addCollections, addExecutions, addGranules, addPdrs, addProviders, and addRules.
    • Adds eraseDataStack function to local API server code allowing resetting of local datastack for testing (ES and DynamoDB).
    • Adds optional parameters to the @cumulus/api bin serve to allow for launching the api without destroying the current data.

• CUMULUS-1697

  • Added the @cumulus/tf-inventory package that provides command line utilities for managing Terraform resources in your AWS account

• CUMULUS-1703

  • Add @cumulus/aws-client/S3.createBucket function
  • Add @cumulus/aws-client/S3.putFile function
  • Add @cumulus/common/string.isNonEmptyString function
  • Add @cumulus/ingest/FtpProviderClient class
  • Add @cumulus/ingest/HttpProviderClient class
  • Add @cumulus/ingest/S3ProviderClient class
  • Add @cumulus/ingest/SftpProviderClient class
  • Add @cumulus/ingest/providerClientUtils.buildProviderClient function
  • Add @cumulus/ingest/providerClientUtils.fetchTextFile function

• CUMULUS-1731

  • Add new optional input variables to the Cumulus Terraform module to support TEA upgrade:
    • thin_egress_cookie_domain - Valid domain for Thin Egress App cookie
    • thin_egress_domain_cert_arn - Certificate Manager SSL Cert ARN for Thin Egress App if deployed outside NGAP/CloudFront
    • thin_egress_download_role_in_region_arn - ARN for reading of Thin Egress App data buckets for in-region requests
    • thin_egress_jwt_algo - Algorithm with which to encode the Thin Egress App JWT cookie
    • thin_egress_jwt_secret_name - Name of AWS secret where keys for the Thin Egress App JWT encode/decode are stored
    • thin_egress_lambda_code_dependency_archive_key - Thin Egress App - S3 Key of packaged python modules for lambda dependency layer

• CUMULUS-1733

  • Add discovery-filtering operator doc to document previously undocumented functionality.

• CUMULUS-1737

  • Added the cumulus-test-cleanup module to run a nightly cleanup on resources left over from the integration tests run from the example/spec directory.

Changed

• CUMULUS-1102

  • Updates @cumulus/api/auth/testAuth to use JWT instead of random tokens.
  • Updates the default AMI for the ecs_cluster_instance_image_id.

• CUMULUS-1622

  • Mutex class has been deprecated in @cumulus/common/concurrency and will be removed in a future release.

• CUMULUS-1686

  • Changed ecs_cluster_instance_image_id to be a required variable of the cumulus module and removed the default value. The default was not available across accounts and regions, nor outside of NGAP and therefore not particularly useful.

• CUMULUS-1688

  • Updated @cumulus/aws.receiveSQSMessages not to replace message.Body with a parsed object. This behavior was undocumented and confusing as received messages appeared to contradict AWS docs that state message.Body is always a string.
  • Replaced sf_watcher CloudWatch rule from cloudwatch-events.tf with an EventSourceMapping on sqs2sf mapped to the start_sf SQS queue (in event-sources.tf).
  • Updated sqs2sf with an EventSourceMapping handler and unit test.

• CUMULUS-1698

  • Change variable saml_launchpad_metadata_path to saml_launchpad_metadata_url in the tf-modules/cumulus Terraform module.
  • Updated @cumulus/api/launchpadSaml to download launchpad IDP metadata from configured location when the metadata in s3 is not valid, and to work with updated IDP metadata and SAML response.

• CUMULUS-1731

  • Upgrade the version of the Thin Egress App deployed by Cumulus to v48
    • Note: New variables available, see the 'Added' section of this changelog.

Fixed

• CUMULUS-1664

  • Updated dbIndexer Lambda to remove hardcoded references to DynamoDB table names.

• CUMULUS-1733

  • Fixed granule discovery recursion algorithm used in S/FTP protocols.

Removed

• CUMULUS-1481
  • removed process config and output from PostToCmr as it was not required by the task nor downstream steps, and should still be in the output message's meta regardless.

Deprecated

• CUMULUS-1040
  • Deprecated the following code. For cases where the code was moved into another package, the new code location is noted:
    • @cumulus/common/CloudFormationGateway -> @cumulus/aws-client/CloudFormationGateway
    • @cumulus/common/DynamoDb -> @cumulus/aws-client/DynamoDb
    • @cumulus/common/errors -> @cumulus/errors
    • @cumulus/common/StepFunctions -> @cumulus/aws-client/StepFunctions
    • All of the exported functions in @cumulus/commmon/aws (moved into @cumulus/aws-client), except:
      • @cumulus/common/aws/isThrottlingException -> @cumulus/errors/isThrottlingException
      • @cumulus/common/aws/improveStackTrace (not deprecated)
      • @cumulus/common/aws/retryOnThrottlingException (not deprecated)
    • @cumulus/common/sfnStep/SfnStep.parseStepMessage -> @cumulus/integration-tests/sfnStep/SfnStep.parseStepMessage
    • @cumulus/common/sfnStep/ActivityStep -> @cumulus/integration-tests/sfnStep/ActivityStep
    • @cumulus/common/sfnStep/LambdaStep -> @cumulus/integration-tests/sfnStep/LambdaStep
    • @cumulus/common/string/unicodeEscape -> @cumulus/aws-client/StepFunctions.unicodeEscape
    • @cumulus/common/util/setErrorStack -> @cumulus/aws-client/util/setErrorStack
    • @cumulus/ingest/aws/invoke -> @cumulus/aws-client/Lambda/invoke
    • @cumulus/ingest/aws/CloudWatch.bucketSize
    • @cumulus/ingest/aws/CloudWatch.cw
    • @cumulus/ingest/aws/ECS.ecs
    • @cumulus/ingest/aws/ECS
    • @cumulus/ingest/aws/Events.putEvent -> @cumulus/aws-client/CloudwatchEvents.putEvent
    • @cumulus/ingest/aws/Events.deleteEvent -> @cumulus/aws-client/CloudwatchEvents.deleteEvent
    • @cumulus/ingest/aws/Events.deleteTarget -> @cumulus/aws-client/CloudwatchEvents.deleteTarget
    • @cumulus/ingest/aws/Events.putTarget -> @cumulus/aws-client/CloudwatchEvents.putTarget
    • @cumulus/ingest/aws/SQS.attributes -> @cumulus/aws-client/SQS.getQueueAttributes
    • @cumulus/ingest/aws/SQS.deleteMessage -> @cumulus/aws-client/SQS.deleteSQSMessage
    • @cumulus/ingest/aws/SQS.deleteQueue -> @cumulus/aws-client/SQS.deleteQueue
    • @cumulus/ingest/aws/SQS.getUrl -> @cumulus/aws-client/SQS.getQueueUrlByName
    • @cumulus/ingest/aws/SQS.receiveMessage -> @cumulus/aws-client/SQS.receiveSQSMessages
    • @cumulus/ingest/aws/SQS.sendMessage -> @cumulus/aws-client/SQS.sendSQSMessage
    • @cumulus/ingest/aws/StepFunction.getExecutionStatus -> @cumulus/aws-client/StepFunction.getExecutionStatus
    • @cumulus/ingest/aws/StepFunction.getExecutionUrl -> @cumulus/aws-client/StepFunction.getExecutionUrl

[v1.17.0] - 2019-12-31

BREAKING CHANGES

• CUMULUS-1498
  • The @cumulus/cmrjs.publish2CMR function expects that the value of its creds.password parameter is a plaintext password.
  • Rather than using an encrypted password from the cmr_password environment variable, the @cumulus/cmrjs.updateCMRMetadata function now looks for an environment variable called cmr_password_secret_name and fetches the CMR password from that secret in AWS Secrets Manager.
  • The @cumulus/post-to-cmr task now expects a config.cmr.passwordSecretName value, rather than config.cmr.password. The CMR password will be fetched from that secret in AWS Secrets Manager.

Added

• CUMULUS-630

  • Added support for replaying Kinesis records on a stream into the Cumulus Kinesis workflow triggering mechanism: either all the records, or some time slice delimited by start and end timestamps.
  • Added /replays endpoint to the operator API for triggering replays.
  • Added Replay Kinesis Messages documentation to Operator Docs.
  • Added manualConsumer lambda function to consume a Kinesis stream. Used by the replay AsyncOperation.

• CUMULUS-1687

  • Added new API endpoint for listing async operations at /asyncOperations
  • All asyncOperations now include the fields description and operationType. operationType can be one of the following. [Bulk Delete, Bulk Granules, ES Index, Kinesis Replay]

Changed

• CUMULUS-1626

  • Updates Cumulus to use node10/CMA 1.1.2 for all of its internal lambdas in prep for AWS node 8 EOL

• CUMULUS-1498

  • Remove the DynamoDB Users table. The list of OAuth users who are allowed to use the API is now stored in S3.
  • The CMR password and Launchpad passphrase are now stored in Secrets Manager

[v1.14.5] - 2019-12-30 - [BACKPORT]

Updated

• CUMULUS-1626
  • Updates Cumulus to use node10/CMA 1.1.2 for all of its internal lambdas in prep for AWS node 8 EOL

[v1.16.1] - 2019-12-6

Please note:

• The region argument to the cumulus Terraform module has been removed. You may see a warning or error if you have that variable populated.
• Your workflow tasks should use the following versions of the CMA libraries to utilize new granule, parentArn, asyncOperationId, and stackName fields on the logs:
  • cumulus-message-adapter-js version 1.0.10+
  • cumulus-message-adapter-python version 1.1.1+
  • cumulus-message-adapter-java version 1.2.11+
• The data-persistence module no longer manages the creation of an Elasticsearch service-linked role for deploying Elasticsearch to a VPC. Follow the deployment instructions on preparing your VPC for guidance on how to create the Elasticsearch service-linked role manually.
• There is now a distribution_api_gateway_stage variable for the tf-modules/cumulus Terraform module that will be used as the API gateway stage name used for the distribution API (Thin Egress App)
• Default value for the urs_url variable is now https://uat.urs.earthdata.nasa.gov/ in the tf-modules/cumulus and tf-modules/archive Terraform modules. So deploying the cumulus module without a urs_url variable set will integrate your Cumulus deployment with the UAT URS environment.

Added

• CUMULUS-1563

  • Added custom_domain_name variable to tf-modules/data-persistence module

• CUMULUS-1654

  • Added new helpers to @cumulus/common/execution-history:
    • getStepExitedEvent() returns the TaskStateExited event in a workflow execution history after the given step completion/failure event
    • getTaskExitedEventOutput() returns the output message for a TaskStateExited event in a workflow execution history

Changed

• CUMULUS-1578

  • Updates SAML launchpad configuration to authorize via configured userGroup. See the NASA specific documentation (protected)

• CUMULUS-1579

  • Elasticsearch list queries use match instead of term. term had been analyzing the terms and not supporting - in the field values.

• CUMULUS-1619

  • Adds 4 new keys to @cumulus/logger to display granules, parentArn, asyncOperationId, and stackName.
  • Depends on cumulus-message-adapter-js version 1.0.10+. Cumulus tasks updated to use this version.

• CUMULUS-1654

  • Changed @cumulus/common/SfnStep.parseStepMessage() to a static class method

• CUMULUS-1641

  • Added meta.retries and meta.visibilityTimeout properties to sqs-type rule. To create sqs-type rule, you're required to configure a dead-letter queue on your queue.
  • Added sqsMessageRemover lambda which removes the message from SQS queue upon successful workflow execution.
  • Updated sqsMessageConsumer lambda to not delete message from SQS queue, and to retry the SQS message for configured number of times.

Removed

• Removed create_service_linked_role variable from tf-modules/data-persistence module.

• CUMULUS-1321

  • The region argument to the cumulus Terraform module has been removed

Fixed

• CUMULUS-1668 - Fixed a race condition where executions may not have been added to the database correctly
• CUMULUS-1654 - Fixed issue with publishReports Lambda not including workflow execution error information for failed workflows with a single step
• Fixed tf-modules/cumulus module so that the urs_url variable is passed on to its invocation of the tf-modules/archive module

[v1.16.0] - 2019-11-15

Added

• CUMULUS-1321

  • A deploy_distribution_s3_credentials_endpoint variable has been added to the cumulus Terraform module. If true, the NGAP-backed S3 credentials endpoint will be added to the Thin Egress App's API. Default: true

• CUMULUS-1544

  • Updated the /granules/bulk endpoint to correctly query Elasticsearch when granule ids are not provided.

• CUMULUS-1580

  • Added /granules/bulk endpoint to @cumulus/api to perform bulk actions on granules given either a list of granule ids or an Elasticsearch query and the workflow to perform.

Changed

• CUMULUS-1561

  • Fix the way that we are handling Terraform provider version requirements
  • Pass provider configs into child modules using the method that the Terraform documentation suggests
  • Remove the region input variable from the s3_access_test Terraform module
  • Remove the aws_profile and aws_region input variables from the s3-replicator Terraform module

• CUMULUS-1639

  • Because of S3's Data Consistency Model, there may be situations where a GET operation for an object can temporarily return a NoSuchKey response even if that object has been created. The @cumulus/common/aws.getS3Object() function has been updated to support retries if a NoSuchKey response is returned by S3. This behavior can be enabled by passing a retryOptions object to that function. Supported values for that object can be found here: https://github.com/tim-kos/node-retry#retryoperationoptions

Removed

• CUMULUS-1559
  • logToSharedDestination has been migrated to the Terraform deployment as log_api_gateway_to_cloudwatch and will ONLY apply to egress lambdas. Due to the differences in the Terraform deployment model, we cannot support a global log subscription toggle for a configurable subset of lambdas. However, setting up your own log forwarding for a Lambda with Terraform is fairly simple, as you will only need to add SubscriptionFilters to your Terraform configuration, one per log group. See the Terraform documentation for details on how to do this. An empty FilterPattern ("") will capture all logs in a group.

[v1.15.0] - 2019-11-04

BREAKING CHANGES

• CUMULUS-1644 - When a workflow execution begins or ends, the workflow payload is parsed and any new or updated PDRs or granules referenced in that workflow are stored to the Cumulus archive. The defined interface says that a PDR in payload.pdr will be added to the archive, and any granules in payload.granules will also be added to the archive. In previous releases, PDRs found in meta.pdr and granules found in meta.input_granules were also added to the archive. This caused unexpected behavior and has been removed. Only PDRs from payload.pdr and granules from payload.granules will now be added to the Cumulus archive.

• CUMULUS-1449 - Cumulus now uses a universal workflow template when starting a workflow that contains general information specific to the deployment, but not specific to the workflow. Workflow task configs must be defined using AWS step function parameters. As part of this change, CumulusConfig has been retired and task configs must now be defined under the cma.task_config key in the Parameters section of a step function definition.

  Migration instructions:

  NOTE: These instructions require the use of Cumulus Message Adapter v1.1.x+. Please ensure you are using a compatible version before attempting to migrate workflow configurations. When defining workflow steps, remove any CumulusConfig section, as shown below:

  ParsePdr:
    CumulusConfig:
      provider: "{$.meta.provider}"
      bucket: "{$.meta.buckets.internal.name}"
      stack: "{$.meta.stack}"
  

  Instead, use AWS Parameters to pass task_config for the task directly into the Cumulus Message Adapter:

  ParsePdr:
    Parameters:
      cma:
        event.$: "$"
        task_config:
          provider: "{$.meta.provider}"
          bucket: "{$.meta.buckets.internal.name}"
          stack: "{$.meta.stack}"
  

  In this example, the cma key is used to pass parameters to the message adapter. Using task_config in combination with event.$: '$' allows the message adapter to process task_config as the config passed to the Cumulus task. See example/workflows/sips.yml in the core repository for further examples of how to set the Parameters.

  Additionally, workflow configurations for the QueueGranules and QueuePdrs tasks need to be updated:

  • queue-pdrs config changes:
    • parsePdrMessageTemplateUri replaced with parsePdrWorkflow, which is the workflow name (i.e. top-level name in config.yml, e.g. 'ParsePdr').
    • internalBucket and stackName configs now required to look up configuration from the deployment. Brings the task config in line with that of queue-granules.
  • queue-granules config change: ingestGranuleMessageTemplateUri replaced with ingestGranuleWorkflow, which is the workflow name (e.g. 'IngestGranule').

• CUMULUS-1396 - Workflow steps at the beginning and end of a workflow using the SfSnsReport Lambda have now been deprecated (e.g. StartStatus, StopStatus) and should be removed from your workflow definitions. These steps were used for publishing ingest notifications and have been replaced by an implementation using Cloudwatch events for Step Functions to trigger a Lambda that publishes ingest notifications. For further detail on how ingest notifications are published, see the notes below on CUMULUS-1394. For examples of how to update your workflow definitions, see our example workflow definitions.

• CUMULUS-1470

  • Remove Cumulus-defined ECS service autoscaling, allowing integrators to better customize autoscaling to meet their needs. In order to use autoscaling with ECS services, appropriate AWS::ApplicationAutoScaling::ScalableTarget, AWS::ApplicationAutoScaling::ScalingPolicy, and AWS::CloudWatch::Alarm resources should be defined in a kes overrides file. See this example for an example.
  • The following config parameters are no longer used:
    • ecs.services.<NAME>.minTasks
    • ecs.services.<NAME>.maxTasks
    • ecs.services.<NAME>.scaleInActivityScheduleTime
    • ecs.services.<NAME>.scaleInAdjustmentPercent
    • ecs.services.<NAME>.scaleOutActivityScheduleTime
    • ecs.services.<NAME>.scaleOutAdjustmentPercent
    • ecs.services.<NAME>.activityName

Added

• CUMULUS-1100

  • Added 30-day retention properties to all log groups that were missing those policies.

• CUMULUS-1396

  • Added @cumulus/common/sfnStep:
    • LambdaStep - A class for retrieving and parsing input and output to Lambda steps in AWS Step Functions
    • ActivityStep - A class for retrieving and parsing input and output to ECS activity steps in AWS Step Functions

• CUMULUS-1574

  • Added GET /token endpoint for SAML authorization when cumulus is protected by Launchpad. This lets a user retrieve a token by hand that can be presented to the API.

• CUMULUS-1625

  • Added sf_start_rate variable to the ingest Terraform module, equivalent to sqs_consumer_rate in the old model, but will not be automatically applied to custom queues as that was.

• CUMULUS-1513

  • Added sqs-type rule support in the Cumulus API @cumulus/api
  • Added sqsMessageConsumer lambda which processes messages from the SQS queues configured in the sqs rules.

Changed

• CUMULUS-1639

  • Because of S3's Data Consistency Model, there may be situations where a GET operation for an object can temporarily return a NoSuchKey response even if that object has been created. The @cumulus/common/aws.getS3Object() function will now retry up to 10 times if a NoSuchKey response is returned by S3. This can behavior can be overridden by passing { retries: 0 } as the retryOptions argument.

• CUMULUS-1449

  • queue-pdrs & queue-granules config changes. Details in breaking changes section.
  • Cumulus now uses a universal workflow template when starting workflow that contains general information specific to the deployment, but not specific to the workflow.
  • Changed the way workflow configs are defined, from CumulusConfig to a task_config AWS Parameter.

• CUMULUS-1452

  • Changed the default ECS docker storage drive to devicemapper

• CUMULUS-1453

  • Removed config schema for @cumulus/sf-sns-report task
  • Updated @cumulus/sf-sns-report to always assume that it is running as an intermediate step in a workflow, not as the first or last step

Removed

• CUMULUS-1449
  • Retired CumulusConfig as part of step function definitions, as this is an artifact of the way Kes parses workflow definitions that was not possible to migrate to Terraform. Use AWS Parameters and the task_config key instead. See change note above.
  • Removed individual workflow templates.

Fixed

• CUMULUS-1620 - Fixed bug where message_adapter_version does not correctly inject the CMA

• CUMULUS-1396 - Updated @cumulus/common/StepFunctions.getExecutionHistory() to recursively fetch execution history when nextToken is returned in response

• CUMULUS-1571 - Updated @cumulus/common/DynamoDb.get() to throw any errors encountered when trying to get a record and the record does exist

• CUMULUS-1452

  • Updated the EC2 initialization scripts to use full volume size for docker storage
  • Changed the default ECS docker storage drive to devicemapper