@cumulus/common - npm Package Versions

[v1.10.1] - 2018-09-4

Fixed

• Fixed cloudformation template errors in @cumulus/deployment/
  • Replaced references to Fn::Ref: with Ref:
  • Moved long form template references to a newline

[v1.10.0] - 2018-08-31

Removed

• Removed unused and broken code from @cumulus/common
  • Removed @cumulus/common/test-helpers
  • Removed @cumulus/common/task
  • Removed @cumulus/common/message-source
  • Removed the getPossiblyRemote function from @cumulus/common/aws
  • Removed the startPromisedSfnExecution function from @cumulus/common/aws
  • Removed the getCurrentSfnTask function from @cumulus/common/aws

Changed

• CUMULUS-839 - In @cumulus/sync-granule, 'collection' is now an optional config parameter

Fixed

• CUMULUS-859 Moved duplicate code in @cumulus/move-granules and @cumulus/post-to-cmr to @cumulus/ingest. Fixed imports making assumptions about directory structure.
• @cumulus/ingest/consumer correctly limits the number of messages being received and processed from SQS. Details:
  • Background: @cumulus/api includes a lambda <stack-name>-sqs2sf which processes messages from the <stack-name>-startSF SQS queue every minute. The sqs2sf lambda uses @cumulus/ingest/consumer to receive and process messages from SQS.
  • Bug: More than messageLimit number of messages were being consumed and processed from the <stack-name>-startSF SQS queue. Many step functions were being triggered simultaneously by the lambda <stack-name>-sqs2sf (which consumes every minute from the startSF queue) and resulting in step function failure with the error: An error occurred (ThrottlingException) when calling the GetExecutionHistory.
  • Fix: @cumulus/ingest/consumer#processMessages now processes messages until timeLimit has passed OR once it receives up to messageLimit messages. sqs2sf is deployed with a default messageLimit of 10.
  • IMPORTANT NOTE: consumer will actually process up to messageLimit * 2 - 1 messages. This is because sometimes receiveSQSMessages will return less than messageLimit messages and thus the consumer will continue to make calls to receiveSQSMessages. For example, given a messageLimit of 10 and subsequent calls to receiveSQSMessages returns up to 9 messages, the loop will continue and a final call could return up to 10 messages.

[v1.9.1] - 2018-08-22

Please Note To take advantage of the added granule tracking API functionality, updates are required for the message adapter and its libraries. You should be on the following versions:

• cumulus-message-adapter 1.0.9+
• cumulus-message-adapter-js 1.0.4+
• cumulus-message-adapter-java 1.2.7+
• cumulus-message-adapter-python 1.0.5+

Added

• CUMULUS-687 Added logs endpoint to search for logs from a specific workflow execution in @cumulus/api. Added integration test.
• CUMULUS-836 - @cumulus/deployment supports a configurable docker storage driver for ECS. ECS can be configured with either devicemapper (the default storage driver for AWS ECS-optimized AMIs) or overlay2 (the storage driver used by the NGAP 2.0 AMI). The storage driver can be configured in app/config.yml with ecs.docker.storageDriver: overlay2 | devicemapper. The default is overlay2.
  • To support this configuration, a Handlebars helper ifEquals was added to packages/deployment/lib/kes.js.
• CUMULUS-836 - @cumulus/api added IAM roles required by the NGAP 2.0 AMI. The NGAP 2.0 AMI runs a script register_instances_with_ssm.py which requires the ECS IAM role to include ec2:DescribeInstances and ssm:GetParameter permissions.

Fixed

• CUMULUS-836 - @cumulus/deployment uses overlay2 driver by default and does not attempt to write --storage-opt dm.basesize to fix this error.
• CUMULUS-413 Kinesis processing now captures all errors.
  • Added kinesis fallback mechanism when errors occur during record processing.
  • Adds FallbackTopicArn to @cumulus/api/lambdas.yml
  • Adds fallbackConsumer lambda to @cumulus/api
  • Adds fallbackqueue option to lambda definitions capture lambda failures after three retries.
  • Adds kinesisFallback SNS topic to signal incoming errors from kinesis stream.
  • Adds kinesisFailureSQS to capture fully failed events from all retries.
• CUMULUS-855 Adds integration test for kinesis' error path.
• CUMULUS-686 Added workflow task name and version tracking via @cumulus/api executions endpoint under new tasks property, and under workflow_tasks in step input/output.
  • Depends on cumulus-message-adapter 1.0.9+, cumulus-message-adapter-js 1.0.4+, cumulus-message-adapter-java 1.2.7+ and cumulus-message-adapter-python 1.0.5+
• CUMULUS-771
  • Updated sync-granule to stream the remote file to s3
  • Added integration test for ingesting granules from ftp provider
  • Updated http/https integration tests for ingesting granules from http/https providers
• CUMULUS-862 Updated @cumulus/integration-tests to handle remote lambda output
• CUMULUS-856 Set the rule state to have default value ENABLED

Changed

• In @cumulus/deployment, changed the example app config.yml to have additional IAM roles

[v1.9.0] - 2018-08-06

Please note additional information and upgrade instructions here

Added

• CUMULUS-712 - Added integration tests verifying expected behavior in workflows
• GITC-776-2 - Add support for versioned collections

Fixed

• CUMULUS-832
  • Fixed indentation in example config.yml in @cumulus/deployment
  • Fixed issue with new deployment using the default distribution endpoint in @cumulus/deployment and @cumulus/api

[v1.8.1] - 2018-08-01

Note IAM roles should be re-deployed with this release.

• Cumulus-726
  • Added function to @cumulus/integration-tests: sfnStep includes getStepInput which returns the input to the schedule event of a given step function step.
  • Added IAM policy @cumulus/deployment: Lambda processing IAM role includes kinesis::PutRecord so step function lambdas can write to kinesis streams.
• Cumulus Community Edition
  • Added Google OAuth authentication token logic to @cumulus/api. Refactored token endpoint to use environment variable flag OAUTH_PROVIDER when determining with authentication method to use.
  • Added API Lambda memory configuration variable api_lambda_memory to @cumulus/api and @cumulus/deployment.

Changed

• Cumulus-726
  • Changed function in @cumulus/api: models/rules.js#addKinesisEventSource was modified to call to deleteKinesisEventSource with all required parameters (rule's name, arn and type).
  • Changed function in @cumulus/integration-tests: getStepOutput can now be used to return output of failed steps. If users of this function want the output of a failed event, they can pass a third parameter eventType as 'failure'. This function will work as always for steps which completed successfully.

Removed

• Cumulus-726

  • Configuration change to @cumulus/deployment: Removed default auto scaling configuration for Granules and Files DynamoDB tables.

• CUMULUS-688

  • Add integration test for ExecutionStatus
  • Function addition to @cumulus/integration-tests: api includes getExecutionStatus which returns the execution status from the Cumulus API

[v1.7.1] - 2018-07-27 - [BACKPORT]

Fixed

• CUMULUS-766 - Backport from 1.8.0 - Populate database fileSize field from S3 if value not present in Ingest payload

[v1.8.0] - 2018-07-23

Added

• CUMULUS-718 Adds integration test for Kinesis triggering a workflow.

• GITC-776-3 Added more flexibility for rules. You can now edit all fields on the rule's record We may need to update the api documentation to reflect this.

• CUMULUS-681 - Add ingest-in-place action to granules endpoint

  • new applyWorkflow action at PUT /granules/{granuleid} Applying a workflow starts an execution of the provided workflow and passes the granule record as payload. Parameter(s):
    • workflow - the workflow name

• CUMULUS-685 - Add parent exeuction arn to the execution which is triggered from a parent step function

Changed

• CUMULUS-768 - Integration tests get S3 provider data from shared data folder

Fixed

• CUMULUS-746 - Move granule API correctly updates record in dynamo DB and cmr xml file
• CUMULUS-766 - Populate database fileSize field from S3 if value not present in Ingest payload

[v1.7.0] - 2018-07-02

[v1.6.0] - 2018-06-06

[v1.5.5] - 2018-05-30

Added

• CUMULUS-530 - PDR tracking through Queue-granules
  • Add optional pdr property to the sync-granule task's input config and output payload.
• CUMULUS-548 - Create a Lambda task that generates EMS distribution reports
  • In order to supply EMS Distribution Reports, you must enable S3 Server Access Logging on any S3 buckets used for distribution. See How Do I Enable Server Access Logging for an S3 Bucket? The "Target bucket" setting should point at the Cumulus internal bucket. The "Target prefix" should be "<STACK_NAME>/ems-distribution/s3-server-access-logs/", where "STACK_NAME" is replaced with the name of your Cumulus stack.

Fixed

• CUMULUS-546 - Kinesis Consumer should catch and log invalid JSON
  • Kinesis Consumer lambda catches and logs errors so that consumer doesn't get stuck in a loop re-processing bad json records.
• EMS report filenames are now based on their start time instead of the time instead of the time that the report was generated
• CUMULUS-552 - Cumulus API returns different results for the same collection depending on query
  • The collection, provider and rule records in elasticsearch are now replaced with records from dynamo db when the dynamo db records are updated.

Added

• @cumulus/deployment's default cloudformation template now configures storage for Docker to match the configured ECS Volume. The template defines Docker's devicemapper basesize (dm.basesize) using ecs.volumeSize. This addresses ECS default of limiting Docker containers to 10GB of storage (Read more).