@cumulus/deployment

@cumulus/deployment

@cumulus/deployment includes cloudformation templates needed for a successful deployment of a Cumulus Instance. The templates can be used with kes, a node CLI helper for AWS CloudFormation.

What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Usage

See the Cumulus deployment documentation for a detailed overview of how to deploy Cumulus.

Contributing

See Cumulus README

Changelog

[v1.15.0] - 2019-11-04

BREAKING CHANGES

• CUMULUS-1644 - When a workflow execution begins or ends, the workflow payload is parsed and any new or updated PDRs or granules referenced in that workflow are stored to the Cumulus archive. The defined interface says that a PDR in payload.pdr will be added to the archive, and any granules in payload.granules will also be added to the archive. In previous releases, PDRs found in meta.pdr and granules found in meta.input_granules were also added to the archive. This caused unexpected behavior and has been removed. Only PDRs from payload.pdr and granules from payload.granules will now be added to the Cumulus archive.

• CUMULUS-1449 - Cumulus now uses a universal workflow template when starting a workflow that contains general information specific to the deployment, but not specific to the workflow. Workflow task configs must be defined using AWS step function parameters. As part of this change, CumulusConfig has been retired and task configs must now be defined under the cma.task_config key in the Parameters section of a step function definition.

  Migration instructions:

  NOTE: These instructions require the use of Cumulus Message Adapter v1.1.x+. Please ensure you are using a compatible version before attempting to migrate workflow configurations. When defining workflow steps, remove any CumulusConfig section, as shown below:

  ParsePdr:
    CumulusConfig:
      provider: "{$.meta.provider}"
      bucket: "{$.meta.buckets.internal.name}"
      stack: "{$.meta.stack}"
  

  Instead, use AWS Parameters to pass task_config for the task directly into the Cumulus Message Adapter:

  ParsePdr:
    Parameters:
      cma:
        event.$: "$"
        task_config:
          provider: "{$.meta.provider}"
          bucket: "{$.meta.buckets.internal.name}"
          stack: "{$.meta.stack}"
  

  In this example, the cma key is used to pass parameters to the message adapter. Using task_config in combination with event.$: '$' allows the message adapter to process task_config as the config passed to the Cumulus task. See example/workflows/sips.yml in the core repository for further examples of how to set the Parameters.

  Additionally, workflow configurations for the QueueGranules and QueuePdrs tasks need to be updated:

  • queue-pdrs config changes:
    • parsePdrMessageTemplateUri replaced with parsePdrWorkflow, which is the workflow name (i.e. top-level name in config.yml, e.g. 'ParsePdr').
    • internalBucket and stackName configs now required to look up configuration from the deployment. Brings the task config in line with that of queue-granules.
  • queue-granules config change: ingestGranuleMessageTemplateUri replaced with ingestGranuleWorkflow, which is the workflow name (e.g. 'IngestGranule').

• CUMULUS-1396 - Workflow steps at the beginning and end of a workflow using the SfSnsReport Lambda have now been deprecated (e.g. StartStatus, StopStatus) and should be removed from your workflow definitions. These steps were used for publishing ingest notifications and have been replaced by an implementation using Cloudwatch events for Step Functions to trigger a Lambda that publishes ingest notifications. For further detail on how ingest notifications are published, see the notes below on CUMULUS-1394. For examples of how to update your workflow definitions, see our example workflow definitions.

• CUMULUS-1470

  • Remove Cumulus-defined ECS service autoscaling, allowing integrators to better customize autoscaling to meet their needs. In order to use autoscaling with ECS services, appropriate AWS::ApplicationAutoScaling::ScalableTarget, AWS::ApplicationAutoScaling::ScalingPolicy, and AWS::CloudWatch::Alarm resources should be defined in a kes overrides file. See this example for an example.
  • The following config parameters are no longer used:
    • ecs.services.<NAME>.minTasks
    • ecs.services.<NAME>.maxTasks
    • ecs.services.<NAME>.scaleInActivityScheduleTime
    • ecs.services.<NAME>.scaleInAdjustmentPercent
    • ecs.services.<NAME>.scaleOutActivityScheduleTime
    • ecs.services.<NAME>.scaleOutAdjustmentPercent
    • ecs.services.<NAME>.activityName

Added

• CUMULUS-1100

  • Added 30-day retention properties to all log groups that were missing those policies.

• CUMULUS-1396

  • Added @cumulus/common/sfnStep:
    • LambdaStep - A class for retrieving and parsing input and output to Lambda steps in AWS Step Functions
    • ActivityStep - A class for retrieving and parsing input and output to ECS activity steps in AWS Step Functions

• CUMULUS-1574

  • Added GET /token endpoint for SAML authorization when cumulus is protected by Launchpad. This lets a user retrieve a token by hand that can be presented to the API.

• CUMULUS-1625

  • Added sf_start_rate variable to the ingest Terraform module, equivalent to sqs_consumer_rate in the old model, but will not be automatically applied to custom queues as that was.

• CUMULUS-1513

  • Added sqs-type rule support in the Cumulus API @cumulus/api
  • Added sqsMessageConsumer lambda which processes messages from the SQS queues configured in the sqs rules.

Changed

• CUMULUS-1639

  • Because of S3's Data Consistency Model, there may be situations where a GET operation for an object can temporarily return a NoSuchKey response even if that object has been created. The @cumulus/common/aws.getS3Object() function will now retry up to 10 times if a NoSuchKey response is returned by S3. This can behavior can be overridden by passing { retries: 0 } as the retryOptions argument.

• CUMULUS-1449

  • queue-pdrs & queue-granules config changes. Details in breaking changes section.
  • Cumulus now uses a universal workflow template when starting workflow that contains general information specific to the deployment, but not specific to the workflow.
  • Changed the way workflow configs are defined, from CumulusConfig to a task_config AWS Parameter.

• CUMULUS-1452

  • Changed the default ECS docker storage drive to devicemapper

• CUMULUS-1453

  • Removed config schema for @cumulus/sf-sns-report task
  • Updated @cumulus/sf-sns-report to always assume that it is running as an intermediate step in a workflow, not as the first or last step

Removed

• CUMULUS-1449
  • Retired CumulusConfig as part of step function definitions, as this is an artifact of the way Kes parses workflow definitions that was not possible to migrate to Terraform. Use AWS Parameters and the task_config key instead. See change note above.
  • Removed individual workflow templates.

Fixed

• CUMULUS-1620 - Fixed bug where message_adapter_version does not correctly inject the CMA

• CUMULUS-1396 - Updated @cumulus/common/StepFunctions.getExecutionHistory() to recursively fetch execution history when nextToken is returned in response

• CUMULUS-1571 - Updated @cumulus/common/DynamoDb.get() to throw any errors encountered when trying to get a record and the record does exist

• CUMULUS-1452

  • Updated the EC2 initialization scripts to use full volume size for docker storage
  • Changed the default ECS docker storage drive to devicemapper