@cumulus/common - npm Package Versions

[v1.5.0] - 2018-04-23

Changed

• Removed babel from all tasks and packages and increased minimum node requirements to version 8.10
• Lambda functions created by @cumulus/deployment will use node8.10 by default
• Moved cumulus-integration-tests to the example folder CUMULUS-512
• Streamlined all packages dependencies (e.g. remove redundant dependencies and make sure versions are the same across packages)
• CUMULUS-352: Update Cumulus Elasticsearch indices to use index aliases.
• CUMULUS-519: ECS tasks are no longer restarted after each CF deployment unless ecs.restartTasksOnDeploy is set to true
• CUMULUS-298: Updated log filterPattern to include all CloudWatch logs in ElasticSearch
• CUMULUS-518: Updates to the SyncGranule config schema
  • granuleIdExtraction is no longer a property
  • process is now an optional property
  • provider_path is no longer a property

Fixed

• CUMULUS-455 "Kes deployments using only an updated message adapter do not get automatically deployed"
  • prepended the hash value of cumulus-message-adapter.zip file to the zip file name of lambda which uses message adapter.
  • the lambda function will be redeployed when message adapter or lambda function are updated
• Fixed a bug in the bootstrap lambda function where it stuck during update process
• Fixed a bug where the sf-sns-report task did not return the payload of the incoming message as the output of the task [CUMULUS-441]

Added

• CUMULUS-352: Add reindex CLI to the API package.
• CUMULUS-465: Added mock http/ftp/sftp servers to the integration tests
• Added a delete method to the @common/CollectionConfigStore class
• CUMULUS-467 "@cumulus/integration-tests or cumulus-integration-tests should seed provider and collection in deployed DynamoDB"
  • example integration-tests populates providers and collections to database
  • example workflow messages are populated from workflow templates in s3, provider and collection information in database, and input payloads. Input templates are removed.
  • added https protocol to provider schema

[v1.4.1] - 2018-04-11

Fixed

• Sync-granule install

[v1.4.0] - 2018-04-09

Fixed

• CUMULUS-392 "queue-granules not returning the sfn-execution-arns queued"
  • updated queue-granules to return the sfn-execution-arns queued and pdr if exists.
  • added pdr to ingest message meta.pdr instead of payload, so the pdr information doesn't get lost in the ingest workflow, and ingested granule in elasticsearch has pdr name.
  • fixed sf-sns-report schema, remove the invalid part
  • fixed pdr-status-check schema, the failed execution contains arn and reason
• CUMULUS-206 make sure homepage and repository urls exist in package.json files of tasks and packages

Added

• Example folder with a cumulus deployment example

Changed

• CUMULUS-450 - Updated the config schema of the queue-granules task
  • The config no longer takes a "collection" property
  • The config now takes an "internalBucket" property
  • The config now takes a "stackName" property
• CUMULUS-450 - Updated the config schema of the parse-pdr task
  • The config no longer takes a "collection" property
  • The "stack", "provider", and "bucket" config properties are now required
• CUMULUS-469 Added a lambda to the API package to prototype creating an S3 bucket policy for direct, in-region S3 access for the prototype bucket

Removed

• Removed the findTmpTestDataDirectory() function from @cumulus/common/test-utils

Fixed

• CUMULUS-450
  • The queue-granules task now enqueues a sync-granule task with the correct collection config for that granule based on the granule's data-type. It had previously been using the collection config from the config of the queue-granules task, which was a problem if the granules being queued belonged to different data-types.
  • The parse-pdr task now handles the case where a PDR contains granules with different data types, and uses the correct granuleIdExtraction for each granule.

Added

• CUMULUS-448 Add code coverage checking using nyc.

[v1.3.0] - 2018-03-29

Deprecated

• discover-s3-granules is deprecated. The functionality is provided by the discover-granules task

Fixed

• CUMULUS-331: Fix aws.downloadS3File to handle non-existent key
• Using test ftp provider for discover-granules testing [CUMULUS-427]
• CUMULUS-304: "Add AWS API throttling to pdr-status-check task" Added concurrency limit on SFN API calls. The default concurrency is 10 and is configurable through Lambda environment variable CONCURRENCY.
• CUMULUS-414: "Schema validation not being performed on many tasks" revised npm build scripts of tasks that use cumulus-message-adapter to place schema directories into dist directories.
• CUMULUS-301: Update all tests to use test-data package for testing data.
• CUMULUS-271: "Empty response body from rules PUT endpoint" Added the updated rule to response body.
• Increased memory allotment for CustomBootstrap lambda function. Resolves failed deployments where CustomBootstrap lambda function was failing with error Process exited before completing request. This was causing deployments to stall, fail to update and fail to rollback. This error is thrown when the lambda function tries to use more memory than it is allotted.
• Cumulus repository folders structure updated:
  • removed the cumulus folder altogether
  • moved cumulus/tasks to tasks folder at the root level
  • moved the tasks that are not converted to use CMA to tasks/.not_CMA_compliant
  • updated paths where necessary

Added

• @cumulus/integration-tests - Added support for testing the output of an ECS activity as well as a Lambda function.

[v1.2.0] - 2018-03-20

Fixed

• Update vulnerable npm packages [CUMULUS-425]
• @cumulus/api: kinesis-consumer.js uses sf-scheduler.js#schedule instead of placing a message directly on the startSF SQS queue. This is a fix for CUMULUS-359 because sf-scheduler.js#schedule looks up the provider and collection data in DynamoDB and adds it to the meta object of the enqueued message payload.
• @cumulus/api: kinesis-consumer.js catches and logs errors instead of doing an error callback. Before this change, kinesis-consumer was failing to process new records when an existing record caused an error because it would call back with an error and stop processing additional records. It keeps trying to process the record causing the error because it's "position" in the stream is unchanged. Catching and logging the errors is part 1 of the fix. Proposed part 2 is to enqueue the error and the message on a "dead-letter" queue so it can be processed later (CUMULUS-413).
• CUMULUS-260: "PDR page on dashboard only shows zeros." The PDR stats in LPDAAC are all 0s, even if the dashboard has been fixed to retrieve the correct fields. The current version of pdr-status-check has a few issues.
  • pdr is not included in the input/output schema. It's available from the input event. So the pdr status and stats are not updated when the ParsePdr workflow is complete. Adding the pdr to the input/output of the task will fix this.
  • pdr-status-check doesn't update pdr stats which prevent the real time pdr progress from showing up in the dashboard. To solve this, added lambda function sf-sns-report which is copied from @cumulus/api/lambdas/sf-sns-broadcast with modification, sf-sns-report can be used to report step function status anywhere inside a step function. So add step sf-sns-report after each pdr-status-check, we will get the PDR status progress at real time.
  • It's possible an execution is still in the queue and doesn't exist in sfn yet. Added code to handle 'ExecutionDoesNotExist' error when checking the execution status.
• Fixed aws.cloudwatchevents() typo in packages/ingest/aws.js. This typo was the root cause of the error: Error: Could not process scheduled_ingest, Error: : aws.cloudwatchevents is not a constructor seen when trying to update a rule.

Removed

• @cumulus/ingest/aws: Remove queueWorkflowMessage which is no longer being used by @cumulus/api's kinesis-consumer.js.

[v1.1.4] - 2018-03-15

Added

• added flag useList to parse-pdr [CUMULUS-404]

Fixed

• Pass encrypted password to the ApiGranule Lambda function [CUMULUS-424]

[v1.1.3] - 2018-03-14

Fixed

• Changed @cumulus/deployment package install behavior. The build process will happen after installation

[v1.1.2] - 2018-03-14

Added

• added tools to @cumulus/integration-tests for local integration testing
• added end to end testing for discovering and parsing of PDRs
• yarn e2e command is available for end to end testing

Fixed

• CUMULUS-326: "Occasionally encounter "Too Many Requests" on deployment" The api gateway calls will handle throttling errors
• CUMULUS-175: "Dashboard providers not in sync with AWS providers." The root cause of this bug - DynamoDB operations not showing up in Elasticsearch - was shared by collections and rules. The fix was to update providers', collections' and rules; POST, PUT and DELETE endpoints to operate on DynamoDB and using DynamoDB streams to update Elasticsearch. The following packages were made:
  • @cumulus/deployment deploys DynamoDB streams for the Collections, Providers and Rules tables as well as a new lambda function called dbIndexer. The dbIndexer lambda has an event source mapping which listens to each of the DynamoDB streams. The dbIndexer lambda receives events referencing operations on the DynamoDB table and updates the elasticsearch cluster accordingly.
  • The @cumulus/api endpoints for collections, providers and rules only query DynamoDB, with the exception of LIST endpoints and the collections' GET endpoint.

Updated

• Broke up kes.override.js of @cumulus/deployment to multiple modules and moved to a new location
• Expanded @cumulus/deployment test coverage
• all tasks were updated to use cumulus-message-adapter-js 1.0.1
• added build process to integration-tests package to babelify it before publication
• Update @cumulus/integration-tests lambda.js getLambdaOutput to return the entire lambda output. Previously getLambdaOutput returned only the payload.

[v1.1.1] - 2018-03-08

Removed

• Unused queue lambda in api/lambdas [CUMULUS-359]

Fixed

• Kinesis message content is passed to the triggered workflow [CUMULUS-359]
• Kinesis message queues a workflow message and does not write to rules table [CUMULUS-359]

[v1.1.0] - 2018-03-05

Added

• Added a jlog function to common/test-utils to aid in test debugging
• Integration test package with command line tool [CUMULUS-200] by @laurenfrederick
• Test for FTP useList flag [CUMULUS-334] by @kkelly51

Updated

• The queue-pdrs task now uses the cumulus-message-adapter-js library
• Updated the queue-pdrs JSON schemas
• The test-utils schema validation functions now throw an error if validation fails
• The queue-granules task now uses the cumulus-message-adapter-js library
• Updated the queue-granules JSON schemas

Removed

• Removed the getSfnExecutionByName function from common/aws
• Removed the getGranuleStatus function from common/aws