@cumulus/deployment - npm Package Versions

[v1.6.0] - 2018-06-06

[v1.5.5] - 2018-05-30

Added

• CUMULUS-530 - PDR tracking through Queue-granules
  • Add optional pdr property to the sync-granule task's input config and output payload.
• CUMULUS-548 - Create a Lambda task that generates EMS distribution reports
  • In order to supply EMS Distribution Reports, you must enable S3 Server Access Logging on any S3 buckets used for distribution. See How Do I Enable Server Access Logging for an S3 Bucket? The "Target bucket" setting should point at the Cumulus internal bucket. The "Target prefix" should be "<STACK_NAME>/ems-distribution/s3-server-access-logs/", where "STACK_NAME" is replaced with the name of your Cumulus stack.

Fixed

• CUMULUS-546 - Kinesis Consumer should catch and log invalid JSON
  • Kinesis Consumer lambda catches and logs errors so that consumer doesn't get stuck in a loop re-processing bad json records.
• EMS report filenames are now based on their start time instead of the time instead of the time that the report was generated
• CUMULUS-552 - Cumulus API returns different results for the same collection depending on query
  • The collection, provider and rule records in elasticsearch are now replaced with records from dynamo db when the dynamo db records are updated.

Added

• @cumulus/deployment's default cloudformation template now configures storage for Docker to match the configured ECS Volume. The template defines Docker's devicemapper basesize (dm.basesize) using ecs.volumeSize. This addresses ECS default of limiting Docker containers to 10GB of storage (Read more).

[v1.5.4] - 2018-05-21

Added

• CUMULUS-535 - EMS Ingest, Archive, Archive Delete reports
  • Add lambda EmsReport to create daily EMS Ingest, Archive, Archive Delete reports
  • ems.provider property added to @cumulus/deployment/app/config.yml. To change the provider name, please add ems: provider property to app/config.yml.
• CUMULUS-480 Use DynamoDB to store granules, pdrs and execution records
  • Activate PointInTime feature on DynamoDB tables
  • Increase test coverage on api package
  • Add ability to restore metadata records from json files to DynamoDB
• CUMULUS-459 provide API endpoint for moving granules from one location on s3 to another

[v1.5.3] - 2018-05-18

Fixed

• CUMULUS-557 - "Add dataType to DiscoverGranules output"
  • Granules discovered by the DiscoverGranules task now include dataType
  • dataType is now a required property for granules used as input to the QueueGranules task
• CUMULUS-550 Update deployment app/config.yml to force elasticsearch updates for deleted granules

[v1.5.2] - 2018-05-15

Fixed

• CUMULUS-514 - "Unable to Delete the Granules"
  • updated cmrjs.deleteConcept to return success if the record is not found in CMR.

Added

• CUMULUS-547 - The distribution API now includes an "earthdataLoginUsername" query parameter when it returns a signed S3 URL
• CUMULUS-527 - "parse-pdr queues up all granules and ignores regex"
  • Add an optional config property to the ParsePdr task called "granuleIdFilter". This property is a regular expression that is applied against the filename of the first file of each granule contained in the PDR. If the regular expression matches, then the granule is included in the output. Defaults to '.', which will match all granules in the PDR.
• File checksums in PDRs now support MD5
• Deployment support to subscribe to an SNS topic that already exists
• CUMULUS-470, CUMULUS-471 In-region S3 Policy lambda added to API to update bucket policy for in-region access.
• CUMULUS-533 Added fields to granule indexer to support EMS ingest and archive record creation
• CUMULUS-534 Track deleted granules
  • added deletedgranule type to cumulus index.
  • Important Note: Force custom bootstrap to re-run by adding this to app/config.yml es: elasticSearchMapping: 7
• You can now deploy cumulus without ElasticSearch. Just add es: null to your app/config.yml file. This is only useful for debugging purposes. Cumulus still requires ElasticSearch to properly operate.
• @cumulus/integration-tests includes and exports the addRules function, which seeds rules into the DynamoDB table.
• Added capability to support EFS in cloud formation template. Also added optional capability to ssh to your instance and privileged lambda functions.
• Added support to force discovery of PDRs that have already been processed and filtering of selected data types
• @cumulus/cmrjs uses an environment variable USER_IP_ADDRESS or fallback IP address of 10.0.0.0 when a public IP address is not available. This supports lambda functions deployed into a VPC's private subnet, where no public IP address is available.

Changed

• CUMULUS-550 Custom bootstrap automatically adds new types to index on deployment

[v1.5.1] - 2018-04-23

Fixed

• add the missing dist folder to the hello-world task
• disable uglifyjs on the built version of the pdr-status-check (read: https://github.com/webpack-contrib/uglifyjs-webpack-plugin/issues/264)

[v1.5.0] - 2018-04-23

Changed

• Removed babel from all tasks and packages and increased minimum node requirements to version 8.10
• Lambda functions created by @cumulus/deployment will use node8.10 by default
• Moved cumulus-integration-tests to the example folder CUMULUS-512
• Streamlined all packages dependencies (e.g. remove redundant dependencies and make sure versions are the same across packages)
• CUMULUS-352: Update Cumulus Elasticsearch indices to use index aliases.
• CUMULUS-519: ECS tasks are no longer restarted after each CF deployment unless ecs.restartTasksOnDeploy is set to true
• CUMULUS-298: Updated log filterPattern to include all CloudWatch logs in ElasticSearch
• CUMULUS-518: Updates to the SyncGranule config schema
  • granuleIdExtraction is no longer a property
  • process is now an optional property
  • provider_path is no longer a property

Fixed

• CUMULUS-455 "Kes deployments using only an updated message adapter do not get automatically deployed"
  • prepended the hash value of cumulus-message-adapter.zip file to the zip file name of lambda which uses message adapter.
  • the lambda function will be redeployed when message adapter or lambda function are updated
• Fixed a bug in the bootstrap lambda function where it stuck during update process
• Fixed a bug where the sf-sns-report task did not return the payload of the incoming message as the output of the task [CUMULUS-441]

Added

• CUMULUS-352: Add reindex CLI to the API package.
• CUMULUS-465: Added mock http/ftp/sftp servers to the integration tests
• Added a delete method to the @common/CollectionConfigStore class
• CUMULUS-467 "@cumulus/integration-tests or cumulus-integration-tests should seed provider and collection in deployed DynamoDB"
  • example integration-tests populates providers and collections to database
  • example workflow messages are populated from workflow templates in s3, provider and collection information in database, and input payloads. Input templates are removed.
  • added https protocol to provider schema

[v1.4.1] - 2018-04-11

Fixed

• Sync-granule install

[v1.4.0] - 2018-04-09

Fixed

• CUMULUS-392 "queue-granules not returning the sfn-execution-arns queued"
  • updated queue-granules to return the sfn-execution-arns queued and pdr if exists.
  • added pdr to ingest message meta.pdr instead of payload, so the pdr information doesn't get lost in the ingest workflow, and ingested granule in elasticsearch has pdr name.
  • fixed sf-sns-report schema, remove the invalid part
  • fixed pdr-status-check schema, the failed execution contains arn and reason
• CUMULUS-206 make sure homepage and repository urls exist in package.json files of tasks and packages

Added

• Example folder with a cumulus deployment example

Changed

• CUMULUS-450 - Updated the config schema of the queue-granules task
  • The config no longer takes a "collection" property
  • The config now takes an "internalBucket" property
  • The config now takes a "stackName" property
• CUMULUS-450 - Updated the config schema of the parse-pdr task
  • The config no longer takes a "collection" property
  • The "stack", "provider", and "bucket" config properties are now required
• CUMULUS-469 Added a lambda to the API package to prototype creating an S3 bucket policy for direct, in-region S3 access for the prototype bucket

Removed

• Removed the findTmpTestDataDirectory() function from @cumulus/common/test-utils

Fixed

• CUMULUS-450
  • The queue-granules task now enqueues a sync-granule task with the correct collection config for that granule based on the granule's data-type. It had previously been using the collection config from the config of the queue-granules task, which was a problem if the granules being queued belonged to different data-types.
  • The parse-pdr task now handles the case where a PDR contains granules with different data types, and uses the correct granuleIdExtraction for each granule.

Added

• CUMULUS-448 Add code coverage checking using nyc.

[v1.3.0] - 2018-03-29

Deprecated

• discover-s3-granules is deprecated. The functionality is provided by the discover-granules task

Fixed

• CUMULUS-331: Fix aws.downloadS3File to handle non-existent key
• Using test ftp provider for discover-granules testing [CUMULUS-427]
• CUMULUS-304: "Add AWS API throttling to pdr-status-check task" Added concurrency limit on SFN API calls. The default concurrency is 10 and is configurable through Lambda environment variable CONCURRENCY.
• CUMULUS-414: "Schema validation not being performed on many tasks" revised npm build scripts of tasks that use cumulus-message-adapter to place schema directories into dist directories.
• CUMULUS-301: Update all tests to use test-data package for testing data.
• CUMULUS-271: "Empty response body from rules PUT endpoint" Added the updated rule to response body.
• Increased memory allotment for CustomBootstrap lambda function. Resolves failed deployments where CustomBootstrap lambda function was failing with error Process exited before completing request. This was causing deployments to stall, fail to update and fail to rollback. This error is thrown when the lambda function tries to use more memory than it is allotted.
• Cumulus repository folders structure updated:
  • removed the cumulus folder altogether
  • moved cumulus/tasks to tasks folder at the root level
  • moved the tasks that are not converted to use CMA to tasks/.not_CMA_compliant
  • updated paths where necessary

Added

• @cumulus/integration-tests - Added support for testing the output of an ECS activity as well as a Lambda function.