@cumulus/common - npm Package Versions

[v1.18.0] 2020-02-03

BREAKING CHANGES

• CUMULUS-1686

  • ecs_cluster_instance_image_id is now a required variable of the cumulus module, instead of optional.

• CUMULUS-1698

  • Change variable saml_launchpad_metadata_path to saml_launchpad_metadata_url in the tf-modules/cumulus Terraform module.

• CUMULUS-1703

  • Remove the unused forceDownload option from the sync-granule tasks's config
  • Remove the @cumulus/ingest/granule.Discover class
  • Remove the @cumulus/ingest/granule.Granule class
  • Remove the @cumulus/ingest/pdr.Discover class
  • Remove the @cumulus/ingest/pdr.Granule class
  • Remove the @cumulus/ingest/parse-pdr.parsePdr function

Added

• CUMULUS-1040

  • Added @cumulus/aws-client package to provide utilities for working with AWS services and the Node.js AWS SDK
  • Added @cumulus/errors package which exports error classes for use in Cumulus workflow code
  • Added @cumulus/integration-tests/sfnStep to provide utilities for parsing step function execution histories

• CUMULUS-1102

  • Adds functionality to the @cumulus/api package for better local testing.
    • Adds data seeding for @cumulus/api's localAPI.
      • seed functions allow adding collections, executions, granules, pdrs, providers, and rules to a Localstack Elasticsearch and DynamoDB via addCollections, addExecutions, addGranules, addPdrs, addProviders, and addRules.
    • Adds eraseDataStack function to local API server code allowing resetting of local datastack for testing (ES and DynamoDB).
    • Adds optional parameters to the @cumulus/api bin serve to allow for launching the api without destroying the current data.

• CUMULUS-1697

  • Added the @cumulus/tf-inventory package that provides command line utilities for managing Terraform resources in your AWS account

• CUMULUS-1703

  • Add @cumulus/aws-client/S3.createBucket function
  • Add @cumulus/aws-client/S3.putFile function
  • Add @cumulus/common/string.isNonEmptyString function
  • Add @cumulus/ingest/FtpProviderClient class
  • Add @cumulus/ingest/HttpProviderClient class
  • Add @cumulus/ingest/S3ProviderClient class
  • Add @cumulus/ingest/SftpProviderClient class
  • Add @cumulus/ingest/providerClientUtils.buildProviderClient function
  • Add @cumulus/ingest/providerClientUtils.fetchTextFile function

• CUMULUS-1731

  • Add new optional input variables to the Cumulus Terraform module to support TEA upgrade:
    • thin_egress_cookie_domain - Valid domain for Thin Egress App cookie
    • thin_egress_domain_cert_arn - Certificate Manager SSL Cert ARN for Thin Egress App if deployed outside NGAP/CloudFront
    • thin_egress_download_role_in_region_arn - ARN for reading of Thin Egress App data buckets for in-region requests
    • thin_egress_jwt_algo - Algorithm with which to encode the Thin Egress App JWT cookie
    • thin_egress_jwt_secret_name - Name of AWS secret where keys for the Thin Egress App JWT encode/decode are stored
    • thin_egress_lambda_code_dependency_archive_key - Thin Egress App - S3 Key of packaged python modules for lambda dependency layer

• CUMULUS-1733

  • Add discovery-filtering operator doc to document previously undocumented functionality.

• CUMULUS-1737

  • Added the cumulus-test-cleanup module to run a nightly cleanup on resources left over from the integration tests run from the example/spec directory.

Changed

• CUMULUS-1102

  • Updates @cumulus/api/auth/testAuth to use JWT instead of random tokens.
  • Updates the default AMI for the ecs_cluster_instance_image_id.

• CUMULUS-1622

  • Mutex class has been deprecated in @cumulus/common/concurrency and will be removed in a future release.

• CUMULUS-1686

  • Changed ecs_cluster_instance_image_id to be a required variable of the cumulus module and removed the default value. The default was not available across accounts and regions, nor outside of NGAP and therefore not particularly useful.

• CUMULUS-1688

  • Updated @cumulus/aws.receiveSQSMessages not to replace message.Body with a parsed object. This behavior was undocumented and confusing as received messages appeared to contradict AWS docs that state message.Body is always a string.
  • Replaced sf_watcher CloudWatch rule from cloudwatch-events.tf with an EventSourceMapping on sqs2sf mapped to the start_sf SQS queue (in event-sources.tf).
  • Updated sqs2sf with an EventSourceMapping handler and unit test.

• CUMULUS-1698

  • Change variable saml_launchpad_metadata_path to saml_launchpad_metadata_url in the tf-modules/cumulus Terraform module.
  • Updated @cumulus/api/launchpadSaml to download launchpad IDP metadata from configured location when the metadata in s3 is not valid, and to work with updated IDP metadata and SAML response.

• CUMULUS-1731

  • Upgrade the version of the Thin Egress App deployed by Cumulus to v48
    • Note: New variables available, see the 'Added' section of this changelog.

Fixed

• CUMULUS-1664

  • Updated dbIndexer Lambda to remove hardcoded references to DynamoDB table names.

• CUMULUS-1733

  • Fixed granule discovery recursion algorithm used in S/FTP protocols.

Removed

• CUMULUS-1481
  • removed process config and output from PostToCmr as it was not required by the task nor downstream steps, and should still be in the output message's meta regardless.

Deprecated

• CUMULUS-1040
  • Deprecated the following code. For cases where the code was moved into another package, the new code location is noted:
    • @cumulus/common/CloudFormationGateway -> @cumulus/aws-client/CloudFormationGateway
    • @cumulus/common/DynamoDb -> @cumulus/aws-client/DynamoDb
    • @cumulus/common/errors -> @cumulus/errors
    • @cumulus/common/StepFunctions -> @cumulus/aws-client/StepFunctions
    • All of the exported functions in @cumulus/commmon/aws (moved into @cumulus/aws-client), except:
      • @cumulus/common/aws/isThrottlingException -> @cumulus/errors/isThrottlingException
      • @cumulus/common/aws/improveStackTrace (not deprecated)
      • @cumulus/common/aws/retryOnThrottlingException (not deprecated)
    • @cumulus/common/sfnStep/SfnStep.parseStepMessage -> @cumulus/integration-tests/sfnStep/SfnStep.parseStepMessage
    • @cumulus/common/sfnStep/ActivityStep -> @cumulus/integration-tests/sfnStep/ActivityStep
    • @cumulus/common/sfnStep/LambdaStep -> @cumulus/integration-tests/sfnStep/LambdaStep
    • @cumulus/common/string/unicodeEscape -> @cumulus/aws-client/StepFunctions.unicodeEscape
    • @cumulus/common/util/setErrorStack -> @cumulus/aws-client/util/setErrorStack
    • @cumulus/ingest/aws/invoke -> @cumulus/aws-client/Lambda/invoke
    • @cumulus/ingest/aws/CloudWatch.bucketSize
    • @cumulus/ingest/aws/CloudWatch.cw
    • @cumulus/ingest/aws/ECS.ecs
    • @cumulus/ingest/aws/ECS
    • @cumulus/ingest/aws/Events.putEvent -> @cumulus/aws-client/CloudwatchEvents.putEvent
    • @cumulus/ingest/aws/Events.deleteEvent -> @cumulus/aws-client/CloudwatchEvents.deleteEvent
    • @cumulus/ingest/aws/Events.deleteTarget -> @cumulus/aws-client/CloudwatchEvents.deleteTarget
    • @cumulus/ingest/aws/Events.putTarget -> @cumulus/aws-client/CloudwatchEvents.putTarget
    • @cumulus/ingest/aws/SQS.attributes -> @cumulus/aws-client/SQS.getQueueAttributes
    • @cumulus/ingest/aws/SQS.deleteMessage -> @cumulus/aws-client/SQS.deleteSQSMessage
    • @cumulus/ingest/aws/SQS.deleteQueue -> @cumulus/aws-client/SQS.deleteQueue
    • @cumulus/ingest/aws/SQS.getUrl -> @cumulus/aws-client/SQS.getQueueUrlByName
    • @cumulus/ingest/aws/SQS.receiveMessage -> @cumulus/aws-client/SQS.receiveSQSMessages
    • @cumulus/ingest/aws/SQS.sendMessage -> @cumulus/aws-client/SQS.sendSQSMessage
    • @cumulus/ingest/aws/StepFunction.getExecutionStatus -> @cumulus/aws-client/StepFunction.getExecutionStatus
    • @cumulus/ingest/aws/StepFunction.getExecutionUrl -> @cumulus/aws-client/StepFunction.getExecutionUrl

[v1.17.0] - 2019-12-31

BREAKING CHANGES

• CUMULUS-1498
  • The @cumulus/cmrjs.publish2CMR function expects that the value of its creds.password parameter is a plaintext password.
  • Rather than using an encrypted password from the cmr_password environment variable, the @cumulus/cmrjs.updateCMRMetadata function now looks for an environment variable called cmr_password_secret_name and fetches the CMR password from that secret in AWS Secrets Manager.
  • The @cumulus/post-to-cmr task now expects a config.cmr.passwordSecretName value, rather than config.cmr.password. The CMR password will be fetched from that secret in AWS Secrets Manager.

Added

• CUMULUS-630

  • Added support for replaying Kinesis records on a stream into the Cumulus Kinesis workflow triggering mechanism: either all the records, or some time slice delimited by start and end timestamps.
  • Added /replays endpoint to the operator API for triggering replays.
  • Added Replay Kinesis Messages documentation to Operator Docs.
  • Added manualConsumer lambda function to consume a Kinesis stream. Used by the replay AsyncOperation.

• CUMULUS-1687

  • Added new API endpoint for listing async operations at /asyncOperations
  • All asyncOperations now include the fields description and operationType. operationType can be one of the following. [Bulk Delete, Bulk Granules, ES Index, Kinesis Replay]

Changed

• CUMULUS-1626

  • Updates Cumulus to use node10/CMA 1.1.2 for all of its internal lambdas in prep for AWS node 8 EOL

• CUMULUS-1498

  • Remove the DynamoDB Users table. The list of OAuth users who are allowed to use the API is now stored in S3.
  • The CMR password and Launchpad passphrase are now stored in Secrets Manager

[v1.14.5] - 2019-12-30 - [BACKPORT]

Updated

• CUMULUS-1626
  • Updates Cumulus to use node10/CMA 1.1.2 for all of its internal lambdas in prep for AWS node 8 EOL

[v1.16.1] - 2019-12-6

Please note:

• The region argument to the cumulus Terraform module has been removed. You may see a warning or error if you have that variable populated.
• Your workflow tasks should use the following versions of the CMA libraries to utilize new granule, parentArn, asyncOperationId, and stackName fields on the logs:
  • cumulus-message-adapter-js version 1.0.10+
  • cumulus-message-adapter-python version 1.1.1+
  • cumulus-message-adapter-java version 1.2.11+
• The data-persistence module no longer manages the creation of an Elasticsearch service-linked role for deploying Elasticsearch to a VPC. Follow the deployment instructions on preparing your VPC for guidance on how to create the Elasticsearch service-linked role manually.
• There is now a distribution_api_gateway_stage variable for the tf-modules/cumulus Terraform module that will be used as the API gateway stage name used for the distribution API (Thin Egress App)
• Default value for the urs_url variable is now https://uat.urs.earthdata.nasa.gov/ in the tf-modules/cumulus and tf-modules/archive Terraform modules. So deploying the cumulus module without a urs_url variable set will integrate your Cumulus deployment with the UAT URS environment.

Added

• CUMULUS-1563

  • Added custom_domain_name variable to tf-modules/data-persistence module

• CUMULUS-1654

  • Added new helpers to @cumulus/common/execution-history:
    • getStepExitedEvent() returns the TaskStateExited event in a workflow execution history after the given step completion/failure event
    • getTaskExitedEventOutput() returns the output message for a TaskStateExited event in a workflow execution history

Changed

• CUMULUS-1578

  • Updates SAML launchpad configuration to authorize via configured userGroup. See the NASA specific documentation (protected)

• CUMULUS-1579

  • Elasticsearch list queries use match instead of term. term had been analyzing the terms and not supporting - in the field values.

• CUMULUS-1619

  • Adds 4 new keys to @cumulus/logger to display granules, parentArn, asyncOperationId, and stackName.
  • Depends on cumulus-message-adapter-js version 1.0.10+. Cumulus tasks updated to use this version.

• CUMULUS-1654

  • Changed @cumulus/common/SfnStep.parseStepMessage() to a static class method

• CUMULUS-1641

  • Added meta.retries and meta.visibilityTimeout properties to sqs-type rule. To create sqs-type rule, you're required to configure a dead-letter queue on your queue.
  • Added sqsMessageRemover lambda which removes the message from SQS queue upon successful workflow execution.
  • Updated sqsMessageConsumer lambda to not delete message from SQS queue, and to retry the SQS message for configured number of times.

Removed

• Removed create_service_linked_role variable from tf-modules/data-persistence module.

• CUMULUS-1321

  • The region argument to the cumulus Terraform module has been removed

Fixed

• CUMULUS-1668 - Fixed a race condition where executions may not have been added to the database correctly
• CUMULUS-1654 - Fixed issue with publishReports Lambda not including workflow execution error information for failed workflows with a single step
• Fixed tf-modules/cumulus module so that the urs_url variable is passed on to its invocation of the tf-modules/archive module

[v1.16.0] - 2019-11-15

Added

• CUMULUS-1321

  • A deploy_distribution_s3_credentials_endpoint variable has been added to the cumulus Terraform module. If true, the NGAP-backed S3 credentials endpoint will be added to the Thin Egress App's API. Default: true

• CUMULUS-1544

  • Updated the /granules/bulk endpoint to correctly query Elasticsearch when granule ids are not provided.

• CUMULUS-1580

  • Added /granules/bulk endpoint to @cumulus/api to perform bulk actions on granules given either a list of granule ids or an Elasticsearch query and the workflow to perform.

Changed

• CUMULUS-1561

  • Fix the way that we are handling Terraform provider version requirements
  • Pass provider configs into child modules using the method that the Terraform documentation suggests
  • Remove the region input variable from the s3_access_test Terraform module
  • Remove the aws_profile and aws_region input variables from the s3-replicator Terraform module

• CUMULUS-1639

  • Because of S3's Data Consistency Model, there may be situations where a GET operation for an object can temporarily return a NoSuchKey response even if that object has been created. The @cumulus/common/aws.getS3Object() function has been updated to support retries if a NoSuchKey response is returned by S3. This behavior can be enabled by passing a retryOptions object to that function. Supported values for that object can be found here: https://github.com/tim-kos/node-retry#retryoperationoptions

Removed

• CUMULUS-1559
  • logToSharedDestination has been migrated to the Terraform deployment as log_api_gateway_to_cloudwatch and will ONLY apply to egress lambdas. Due to the differences in the Terraform deployment model, we cannot support a global log subscription toggle for a configurable subset of lambdas. However, setting up your own log forwarding for a Lambda with Terraform is fairly simple, as you will only need to add SubscriptionFilters to your Terraform configuration, one per log group. See the Terraform documentation for details on how to do this. An empty FilterPattern ("") will capture all logs in a group.

[v1.15.0] - 2019-11-04

BREAKING CHANGES

• CUMULUS-1644 - When a workflow execution begins or ends, the workflow payload is parsed and any new or updated PDRs or granules referenced in that workflow are stored to the Cumulus archive. The defined interface says that a PDR in payload.pdr will be added to the archive, and any granules in payload.granules will also be added to the archive. In previous releases, PDRs found in meta.pdr and granules found in meta.input_granules were also added to the archive. This caused unexpected behavior and has been removed. Only PDRs from payload.pdr and granules from payload.granules will now be added to the Cumulus archive.

• CUMULUS-1449 - Cumulus now uses a universal workflow template when starting a workflow that contains general information specific to the deployment, but not specific to the workflow. Workflow task configs must be defined using AWS step function parameters. As part of this change, CumulusConfig has been retired and task configs must now be defined under the cma.task_config key in the Parameters section of a step function definition.

  Migration instructions:

  NOTE: These instructions require the use of Cumulus Message Adapter v1.1.x+. Please ensure you are using a compatible version before attempting to migrate workflow configurations. When defining workflow steps, remove any CumulusConfig section, as shown below:

  ParsePdr:
    CumulusConfig:
      provider: "{$.meta.provider}"
      bucket: "{$.meta.buckets.internal.name}"
      stack: "{$.meta.stack}"
  

  Instead, use AWS Parameters to pass task_config for the task directly into the Cumulus Message Adapter:

  ParsePdr:
    Parameters:
      cma:
        event.$: "$"
        task_config:
          provider: "{$.meta.provider}"
          bucket: "{$.meta.buckets.internal.name}"
          stack: "{$.meta.stack}"
  

  In this example, the cma key is used to pass parameters to the message adapter. Using task_config in combination with event.$: '$' allows the message adapter to process task_config as the config passed to the Cumulus task. See example/workflows/sips.yml in the core repository for further examples of how to set the Parameters.

  Additionally, workflow configurations for the QueueGranules and QueuePdrs tasks need to be updated:

  • queue-pdrs config changes:
    • parsePdrMessageTemplateUri replaced with parsePdrWorkflow, which is the workflow name (i.e. top-level name in config.yml, e.g. 'ParsePdr').
    • internalBucket and stackName configs now required to look up configuration from the deployment. Brings the task config in line with that of queue-granules.
  • queue-granules config change: ingestGranuleMessageTemplateUri replaced with ingestGranuleWorkflow, which is the workflow name (e.g. 'IngestGranule').

• CUMULUS-1396 - Workflow steps at the beginning and end of a workflow using the SfSnsReport Lambda have now been deprecated (e.g. StartStatus, StopStatus) and should be removed from your workflow definitions. These steps were used for publishing ingest notifications and have been replaced by an implementation using Cloudwatch events for Step Functions to trigger a Lambda that publishes ingest notifications. For further detail on how ingest notifications are published, see the notes below on CUMULUS-1394. For examples of how to update your workflow definitions, see our example workflow definitions.

• CUMULUS-1470

  • Remove Cumulus-defined ECS service autoscaling, allowing integrators to better customize autoscaling to meet their needs. In order to use autoscaling with ECS services, appropriate AWS::ApplicationAutoScaling::ScalableTarget, AWS::ApplicationAutoScaling::ScalingPolicy, and AWS::CloudWatch::Alarm resources should be defined in a kes overrides file. See this example for an example.
  • The following config parameters are no longer used:
    • ecs.services.<NAME>.minTasks
    • ecs.services.<NAME>.maxTasks
    • ecs.services.<NAME>.scaleInActivityScheduleTime
    • ecs.services.<NAME>.scaleInAdjustmentPercent
    • ecs.services.<NAME>.scaleOutActivityScheduleTime
    • ecs.services.<NAME>.scaleOutAdjustmentPercent
    • ecs.services.<NAME>.activityName

Added

• CUMULUS-1100

  • Added 30-day retention properties to all log groups that were missing those policies.

• CUMULUS-1396

  • Added @cumulus/common/sfnStep:
    • LambdaStep - A class for retrieving and parsing input and output to Lambda steps in AWS Step Functions
    • ActivityStep - A class for retrieving and parsing input and output to ECS activity steps in AWS Step Functions

• CUMULUS-1574

  • Added GET /token endpoint for SAML authorization when cumulus is protected by Launchpad. This lets a user retrieve a token by hand that can be presented to the API.

• CUMULUS-1625

  • Added sf_start_rate variable to the ingest Terraform module, equivalent to sqs_consumer_rate in the old model, but will not be automatically applied to custom queues as that was.

• CUMULUS-1513

  • Added sqs-type rule support in the Cumulus API @cumulus/api
  • Added sqsMessageConsumer lambda which processes messages from the SQS queues configured in the sqs rules.

Changed

• CUMULUS-1639

  • Because of S3's Data Consistency Model, there may be situations where a GET operation for an object can temporarily return a NoSuchKey response even if that object has been created. The @cumulus/common/aws.getS3Object() function will now retry up to 10 times if a NoSuchKey response is returned by S3. This can behavior can be overridden by passing { retries: 0 } as the retryOptions argument.

• CUMULUS-1449

  • queue-pdrs & queue-granules config changes. Details in breaking changes section.
  • Cumulus now uses a universal workflow template when starting workflow that contains general information specific to the deployment, but not specific to the workflow.
  • Changed the way workflow configs are defined, from CumulusConfig to a task_config AWS Parameter.

• CUMULUS-1452

  • Changed the default ECS docker storage drive to devicemapper

• CUMULUS-1453

  • Removed config schema for @cumulus/sf-sns-report task
  • Updated @cumulus/sf-sns-report to always assume that it is running as an intermediate step in a workflow, not as the first or last step

Removed

• CUMULUS-1449
  • Retired CumulusConfig as part of step function definitions, as this is an artifact of the way Kes parses workflow definitions that was not possible to migrate to Terraform. Use AWS Parameters and the task_config key instead. See change note above.
  • Removed individual workflow templates.

Fixed

• CUMULUS-1620 - Fixed bug where message_adapter_version does not correctly inject the CMA

• CUMULUS-1396 - Updated @cumulus/common/StepFunctions.getExecutionHistory() to recursively fetch execution history when nextToken is returned in response

• CUMULUS-1571 - Updated @cumulus/common/DynamoDb.get() to throw any errors encountered when trying to get a record and the record does exist

• CUMULUS-1452

  • Updated the EC2 initialization scripts to use full volume size for docker storage
  • Changed the default ECS docker storage drive to devicemapper

[v1.14.4] - 2019-10-28

Fixed

• CUMULUS-1632 - Pinned aws-elasticsearch-connector package in @cumulus/api to version 8.1.3, since 8.2.0 includes breaking changes

[v1.14.3] - 2019-10-18

Fixed

• CUMULUS-1620 - Fixed bug where message_adapter_version does not correctly inject the CMA

• CUMULUS-1572 - A granule is now included in discovery results even when none of its files has a matching file type in the associated collection configuration. Previously, if all files for a granule were unmatched by a file type configuration, the granule was excluded from the discovery results. Further, added support for a boolean property ignoreFilesConfigForDiscovery, which controls how a granule's files are filtered at discovery time.

[v1.14.2] - 2019-10-08

BREAKING CHANGES

Your Cumulus Message Adapter version should be pinned to v1.0.13 or lower in your app/config.yml using message_adapter_version: v1.0.13 OR you should use the workflow migration steps below to work with CMA v1.1.1+.

• CUMULUS-1394 - The implementation of the SfSnsReport Lambda requires additional environment variables for integration with the new ingest notification SNS topics. Therefore, you must update the definition of SfSnsReport in your lambdas.yml like so:

SfSnsReport:
  handler: index.handler
  timeout: 300
  source: node_modules/@cumulus/sf-sns-report/dist
  tables:
    - ExecutionsTable
  envs:
    execution_sns_topic_arn:
      function: Ref
      value: reportExecutionsSns
    granule_sns_topic_arn:
      function: Ref
      value: reportGranulesSns
    pdr_sns_topic_arn:
      function: Ref
      value: reportPdrsSns

• CUMULUS-1447 - The newest release of the Cumulus Message Adapter (v1.1.1) requires that parameterized configuration be used for remote message functionality. Once released, Kes will automatically bring in CMA v1.1.1 without additional configuration.

  Migration instructions Oversized messages are no longer written to S3 automatically. In order to utilize remote messaging functionality, configure a ReplaceConfig AWS Step Function parameter on your CMA task:

  ParsePdr:
    Parameters:
      cma:
        event.$: "$"
        ReplaceConfig:
          FullMessage: true
  

  Accepted fields in ReplaceConfig include MaxSize, FullMessage, Path and TargetPath. See https://github.com/nasa/cumulus-message-adapter/blob/master/CONTRACT.md#remote-message-configuration for full details.

  As this change is backward compatible in Cumulus Core, users wishing to utilize the previous version of the CMA may opt to transition to using a CMA lambda layer, or set message_adapter_version in their configuration to a version prior to v1.1.0.

PLEASE NOTE

• CUMULUS-1394 - Ingest notifications are now provided via 3 separate SNS topics for executions, granules, and PDRs, instead of a single sftracker SNS topic. Whereas the sftracker SNS topic received a full Cumulus execution message, the new topics all receive generated records for the given object. The new topics are only published to if the given object exists for the current execution. For a given execution/granule/PDR, two messages will be received by each topic: one message indicating that ingest is running and another message indicating that ingest has completed or failed. The new SNS topics are:

  • reportExecutions - Receives 1 message per execution
  • reportGranules - Receives 1 message per granule in an execution
  • reportPdrs - Receives 1 message per PDR

Added

• CUMULUS-639

  • Adds SAML JWT and launchpad token authentication to Cumulus API (configurable)
    • NOTE to authenticate with Launchpad ensure your launchpad user_id is in the <prefix>-UsersTable
    • when Cumulus configured to protect API via Launchpad:
      • New endpoints
        • GET /saml/login - starting point for SAML SSO creates the login request url and redirects to the SAML Identity Provider Service (IDP)
        • POST /saml/auth - SAML Assertion Consumer Service. POST receiver from SAML IDP. Validates response, logs the user in, and returns a SAML-based JWT.
    • Disabled endpoints
      • POST /refresh
      • Changes authorization worklow:
      • ensureAuthorized now presumes the bearer token is a JWT and tries to validate. If the token is malformed, it attempts to validate the token against Launchpad. This allows users to bring their own token as described here https://wiki.earthdata.nasa.gov/display/CUMULUS/Cumulus+API+with+Launchpad+Authentication. But it also allows dashboard users to manually authenticate via Launchpad SAML to receive a Launchpad-based JWT.

• CUMULUS-1394

  • Added Granule.generateGranuleRecord() method to granules model to generate a granule database record from a Cumulus execution message
  • Added Pdr.generatePdrRecord() method to PDRs model to generate a granule database record from a Cumulus execution message
  • Added helpers to @cumulus/common/message:
    • getMessageExecutionName() - Get the execution name from a Cumulus execution message
    • getMessageStateMachineArn() - Get the state machine ARN from a Cumulus execution message
    • getMessageExecutionArn() - Get the execution ARN for a Cumulus execution message
    • getMessageGranules() - Get the granules from a Cumulus execution message, if any.
  • Added @cumulus/common/cloudwatch-event/isFailedSfStatus() to determine if a Step Function status from a Cloudwatch event is a failed status

Changed

• CUMULUS-1308

  • HTTP PUT of a Collection, Provider, or Rule via the Cumulus API now performs full replacement of the existing object with the object supplied in the request payload. Previous behavior was to perform a modification (partial update) by merging the existing object with the (possibly partial) object in the payload, but this did not conform to the HTTP standard, which specifies PATCH as the means for modifications rather than replacements.

• CUMULUS-1375

  • Migrate Cumulus from deprecated Elasticsearch JS client to new, supported one in @cumulus/api

• CUMULUS-1485 Update @cumulus/cmr-client to return error message from CMR for validation failures.

• CUMULUS-1394

  • Renamed Execution.generateDocFromPayload() to Execution.generateRecord() on executions model. The method generates an execution database record from a Cumulus execution message.

• CUMULUS-1432

  • logs endpoint takes the level parameter as a string and not a number
  • Elasticsearch term query generation no longer converts numbers to boolean

• CUMULUS-1447

  • Consolidated all remote message handling code into @common/aws
  • Update remote message code to handle updated CMA remote message flags
  • Update example SIPS workflows to utilize Parameterized CMA configuration

• CUMULUS-1448 Refactor workflows that are mutating cumulus_meta to utilize meta field

• CUMULUS-1451

  • Elasticsearch cluster setting auto_create_index will be set to false. This had been causing issues in the bootstrap lambda on deploy.

• CUMULUS-1456

  • @cumulus/api endpoints default error handler uses boom package to format errors, which is consistent with other API endpoint errors.

Fixed

• CUMULUS-1432 logs endpoint filter correctly filters logs by level
• CUMULUS-1484 useMessageAdapter now does not set CUMULUS_MESSAGE_ADAPTER_DIR when true

Removed

• CUMULUS-1394
  • Removed sfTracker SNS topic. Replaced by three new SNS topics for granule, execution, and PDR ingest notifications.
  • Removed unused functions from @cumulus/common/aws:
    • getGranuleS3Params()
    • setGranuleStatus()