@cumulus/common - npm Package Versions

published 1.16.0 • 5 years ago

Changelog

[v1.16.0] - 2019-11-15

Added

CUMULUS-1321
- A deploy_distribution_s3_credentials_endpoint variable has been added to the cumulus Terraform module. If true, the NGAP-backed S3 credentials endpoint will be added to the Thin Egress App's API. Default: true
CUMULUS-1544
- Updated the /granules/bulk endpoint to correctly query Elasticsearch when granule ids are not provided.
CUMULUS-1580
- Added /granules/bulk endpoint to @cumulus/api to perform bulk actions on granules given either a list of granule ids or an Elasticsearch query and the workflow to perform.

Changed

CUMULUS-1561
- Fix the way that we are handling Terraform provider version requirements
- Pass provider configs into child modules using the method that the Terraform documentation suggests
- Remove the region input variable from the s3_access_test Terraform module
- Remove the aws_profile and aws_region input variables from the s3-replicator Terraform module
CUMULUS-1639
- Because of S3's Data Consistency Model, there may be situations where a GET operation for an object can temporarily return a NoSuchKey response even if that object has been created. The @cumulus/common/aws.getS3Object() function has been updated to support retries if a NoSuchKey response is returned by S3. This behavior can be enabled by passing a retryOptions object to that function. Supported values for that object can be found here: https://github.com/tim-kos/node-retry#retryoperationoptions

Removed

CUMULUS-1559
- logToSharedDestination has been migrated to the Terraform deployment as log_api_gateway_to_cloudwatch and will ONLY apply to egress lambdas. Due to the differences in the Terraform deployment model, we cannot support a global log subscription toggle for a configurable subset of lambdas. However, setting up your own log forwarding for a Lambda with Terraform is fairly simple, as you will only need to add SubscriptionFilters to your Terraform configuration, one per log group. See the Terraform documentation for details on how to do this. An empty FilterPattern ("") will capture all logs in a group.

1.15.0

published 1.15.0 • 5 years ago

Changelog

[v1.15.0] - 2019-11-04

BREAKING CHANGES

CUMULUS-1644 - When a workflow execution begins or ends, the workflow payload is parsed and any new or updated PDRs or granules referenced in that workflow are stored to the Cumulus archive. The defined interface says that a PDR in payload.pdr will be added to the archive, and any granules in payload.granules will also be added to the archive. In previous releases, PDRs found in meta.pdr and granules found in meta.input_granules were also added to the archive. This caused unexpected behavior and has been removed. Only PDRs from payload.pdr and granules from payload.granules will now be added to the Cumulus archive.
CUMULUS-1449 - Cumulus now uses a universal workflow template when starting a workflow that contains general information specific to the deployment, but not specific to the workflow. Workflow task configs must be defined using AWS step function parameters. As part of this change, CumulusConfig has been retired and task configs must now be defined under the cma.task_config key in the Parameters section of a step function definition.

Migration instructions:

NOTE: These instructions require the use of Cumulus Message Adapter v1.1.x+. Please ensure you are using a compatible version before attempting to migrate workflow configurations. When defining workflow steps, remove any CumulusConfig section, as shown below:
```
ParsePdr:
  CumulusConfig:
    provider: "{$.meta.provider}"
    bucket: "{$.meta.buckets.internal.name}"
    stack: "{$.meta.stack}"
```
Instead, use AWS Parameters to pass task_config for the task directly into the Cumulus Message Adapter:
```
ParsePdr:
  Parameters:
    cma:
      event.$: "$"
      task_config:
        provider: "{$.meta.provider}"
        bucket: "{$.meta.buckets.internal.name}"
        stack: "{$.meta.stack}"
```
In this example, the cma key is used to pass parameters to the message adapter. Using task_config in combination with event.$: '$' allows the message adapter to process task_config as the config passed to the Cumulus task. See example/workflows/sips.yml in the core repository for further examples of how to set the Parameters.

Additionally, workflow configurations for the QueueGranules and QueuePdrs tasks need to be updated:
- queue-pdrs config changes:
  - parsePdrMessageTemplateUri replaced with parsePdrWorkflow, which is the workflow name (i.e. top-level name in config.yml, e.g. 'ParsePdr').
  - internalBucket and stackName configs now required to look up configuration from the deployment. Brings the task config in line with that of queue-granules.
- queue-granules config change: ingestGranuleMessageTemplateUri replaced with ingestGranuleWorkflow, which is the workflow name (e.g. 'IngestGranule').
CUMULUS-1396 - Workflow steps at the beginning and end of a workflow using the SfSnsReport Lambda have now been deprecated (e.g. StartStatus, StopStatus) and should be removed from your workflow definitions. These steps were used for publishing ingest notifications and have been replaced by an implementation using Cloudwatch events for Step Functions to trigger a Lambda that publishes ingest notifications. For further detail on how ingest notifications are published, see the notes below on CUMULUS-1394. For examples of how to update your workflow definitions, see our example workflow definitions.
CUMULUS-1470
- Remove Cumulus-defined ECS service autoscaling, allowing integrators to better customize autoscaling to meet their needs. In order to use autoscaling with ECS services, appropriate AWS::ApplicationAutoScaling::ScalableTarget, AWS::ApplicationAutoScaling::ScalingPolicy, and AWS::CloudWatch::Alarm resources should be defined in a kes overrides file. See this example for an example.
- The following config parameters are no longer used:
  - ecs.services.<NAME>.minTasks
  - ecs.services.<NAME>.maxTasks
  - ecs.services.<NAME>.scaleInActivityScheduleTime
  - ecs.services.<NAME>.scaleInAdjustmentPercent
  - ecs.services.<NAME>.scaleOutActivityScheduleTime
  - ecs.services.<NAME>.scaleOutAdjustmentPercent
  - ecs.services.<NAME>.activityName

Added

CUMULUS-1100
- Added 30-day retention properties to all log groups that were missing those policies.
CUMULUS-1396
- Added @cumulus/common/sfnStep:
  - LambdaStep - A class for retrieving and parsing input and output to Lambda steps in AWS Step Functions
  - ActivityStep - A class for retrieving and parsing input and output to ECS activity steps in AWS Step Functions
CUMULUS-1574
- Added GET /token endpoint for SAML authorization when cumulus is protected by Launchpad. This lets a user retrieve a token by hand that can be presented to the API.
CUMULUS-1625
- Added sf_start_rate variable to the ingest Terraform module, equivalent to sqs_consumer_rate in the old model, but will not be automatically applied to custom queues as that was.
CUMULUS-1513
- Added sqs-type rule support in the Cumulus API @cumulus/api
- Added sqsMessageConsumer lambda which processes messages from the SQS queues configured in the sqs rules.

Changed

CUMULUS-1639
- Because of S3's Data Consistency Model, there may be situations where a GET operation for an object can temporarily return a NoSuchKey response even if that object has been created. The @cumulus/common/aws.getS3Object() function will now retry up to 10 times if a NoSuchKey response is returned by S3. This can behavior can be overridden by passing { retries: 0 } as the retryOptions argument.
CUMULUS-1449
- queue-pdrs & queue-granules config changes. Details in breaking changes section.
- Cumulus now uses a universal workflow template when starting workflow that contains general information specific to the deployment, but not specific to the workflow.
- Changed the way workflow configs are defined, from CumulusConfig to a task_config AWS Parameter.
CUMULUS-1452
- Changed the default ECS docker storage drive to devicemapper
CUMULUS-1453
- Removed config schema for @cumulus/sf-sns-report task
- Updated @cumulus/sf-sns-report to always assume that it is running as an intermediate step in a workflow, not as the first or last step

Removed

CUMULUS-1449
- Retired CumulusConfig as part of step function definitions, as this is an artifact of the way Kes parses workflow definitions that was not possible to migrate to Terraform. Use AWS Parameters and the task_config key instead. See change note above.
- Removed individual workflow templates.

Fixed

CUMULUS-1620 - Fixed bug where message_adapter_version does not correctly inject the CMA
CUMULUS-1396 - Updated @cumulus/common/StepFunctions.getExecutionHistory() to recursively fetch execution history when nextToken is returned in response
CUMULUS-1571 - Updated @cumulus/common/DynamoDb.get() to throw any errors encountered when trying to get a record and the record does exist
CUMULUS-1452
- Updated the EC2 initialization scripts to use full volume size for docker storage
- Changed the default ECS docker storage drive to devicemapper

1.13.6

published 1.13.6 • 5 years ago

1.14.4

published 1.14.4 • 5 years ago

Changelog

[v1.14.4] - 2019-10-28

Fixed

CUMULUS-1632 - Pinned aws-elasticsearch-connector package in @cumulus/api to version 8.1.3, since 8.2.0 includes breaking changes

1.14.3

published 1.14.3 • 5 years ago

Changelog

[v1.14.3] - 2019-10-18

Fixed

CUMULUS-1620 - Fixed bug where message_adapter_version does not correctly inject the CMA
CUMULUS-1572 - A granule is now included in discovery results even when none of its files has a matching file type in the associated collection configuration. Previously, if all files for a granule were unmatched by a file type configuration, the granule was excluded from the discovery results. Further, added support for a boolean property ignoreFilesConfigForDiscovery, which controls how a granule's files are filtered at discovery time.

1.14.2

published 1.14.2 • 5 years ago

Changelog

[v1.14.2] - 2019-10-08

BREAKING CHANGES

Your Cumulus Message Adapter version should be pinned to v1.0.13 or lower in your app/config.yml using message_adapter_version: v1.0.13 OR you should use the workflow migration steps below to work with CMA v1.1.1+.

CUMULUS-1394 - The implementation of the SfSnsReport Lambda requires additional environment variables for integration with the new ingest notification SNS topics. Therefore, you must update the definition of SfSnsReport in your lambdas.yml like so:

SfSnsReport:
  handler: index.handler
  timeout: 300
  source: node_modules/@cumulus/sf-sns-report/dist
  tables:
    - ExecutionsTable
  envs:
    execution_sns_topic_arn:
      function: Ref
      value: reportExecutionsSns
    granule_sns_topic_arn:
      function: Ref
      value: reportGranulesSns
    pdr_sns_topic_arn:
      function: Ref
      value: reportPdrsSns

CUMULUS-1447 - The newest release of the Cumulus Message Adapter (v1.1.1) requires that parameterized configuration be used for remote message functionality. Once released, Kes will automatically bring in CMA v1.1.1 without additional configuration.

Migration instructions Oversized messages are no longer written to S3 automatically. In order to utilize remote messaging functionality, configure a ReplaceConfig AWS Step Function parameter on your CMA task:
```
ParsePdr:
  Parameters:
    cma:
      event.$: "$"
      ReplaceConfig:
        FullMessage: true
```
Accepted fields in ReplaceConfig include MaxSize, FullMessage, Path and TargetPath. See https://github.com/nasa/cumulus-message-adapter/blob/master/CONTRACT.md#remote-message-configuration for full details.

As this change is backward compatible in Cumulus Core, users wishing to utilize the previous version of the CMA may opt to transition to using a CMA lambda layer, or set message_adapter_version in their configuration to a version prior to v1.1.0.

PLEASE NOTE

CUMULUS-1394 - Ingest notifications are now provided via 3 separate SNS topics for executions, granules, and PDRs, instead of a single sftracker SNS topic. Whereas the sftracker SNS topic received a full Cumulus execution message, the new topics all receive generated records for the given object. The new topics are only published to if the given object exists for the current execution. For a given execution/granule/PDR, two messages will be received by each topic: one message indicating that ingest is running and another message indicating that ingest has completed or failed. The new SNS topics are:
- reportExecutions - Receives 1 message per execution
- reportGranules - Receives 1 message per granule in an execution
- reportPdrs - Receives 1 message per PDR

Added

CUMULUS-639
- Adds SAML JWT and launchpad token authentication to Cumulus API (configurable)
  - NOTE to authenticate with Launchpad ensure your launchpad user_id is in the <prefix>-UsersTable
  - when Cumulus configured to protect API via Launchpad:
    - New endpoints
      - GET /saml/login - starting point for SAML SSO creates the login request url and redirects to the SAML Identity Provider Service (IDP)
      - POST /saml/auth - SAML Assertion Consumer Service. POST receiver from SAML IDP. Validates response, logs the user in, and returns a SAML-based JWT.
  - Disabled endpoints
    - POST /refresh
    - Changes authorization worklow:
    - ensureAuthorized now presumes the bearer token is a JWT and tries to validate. If the token is malformed, it attempts to validate the token against Launchpad. This allows users to bring their own token as described here https://wiki.earthdata.nasa.gov/display/CUMULUS/Cumulus+API+with+Launchpad+Authentication. But it also allows dashboard users to manually authenticate via Launchpad SAML to receive a Launchpad-based JWT.
CUMULUS-1394
- Added Granule.generateGranuleRecord() method to granules model to generate a granule database record from a Cumulus execution message
- Added Pdr.generatePdrRecord() method to PDRs model to generate a granule database record from a Cumulus execution message
- Added helpers to @cumulus/common/message:
  - getMessageExecutionName() - Get the execution name from a Cumulus execution message
  - getMessageStateMachineArn() - Get the state machine ARN from a Cumulus execution message
  - getMessageExecutionArn() - Get the execution ARN for a Cumulus execution message
  - getMessageGranules() - Get the granules from a Cumulus execution message, if any.
- Added @cumulus/common/cloudwatch-event/isFailedSfStatus() to determine if a Step Function status from a Cloudwatch event is a failed status

Changed

CUMULUS-1308
- HTTP PUT of a Collection, Provider, or Rule via the Cumulus API now performs full replacement of the existing object with the object supplied in the request payload. Previous behavior was to perform a modification (partial update) by merging the existing object with the (possibly partial) object in the payload, but this did not conform to the HTTP standard, which specifies PATCH as the means for modifications rather than replacements.
CUMULUS-1375
- Migrate Cumulus from deprecated Elasticsearch JS client to new, supported one in @cumulus/api
CUMULUS-1485 Update @cumulus/cmr-client to return error message from CMR for validation failures.
CUMULUS-1394
- Renamed Execution.generateDocFromPayload() to Execution.generateRecord() on executions model. The method generates an execution database record from a Cumulus execution message.
CUMULUS-1432
- logs endpoint takes the level parameter as a string and not a number
- Elasticsearch term query generation no longer converts numbers to boolean
CUMULUS-1447
- Consolidated all remote message handling code into @common/aws
- Update remote message code to handle updated CMA remote message flags
- Update example SIPS workflows to utilize Parameterized CMA configuration
CUMULUS-1448 Refactor workflows that are mutating cumulus_meta to utilize meta field
CUMULUS-1451
- Elasticsearch cluster setting auto_create_index will be set to false. This had been causing issues in the bootstrap lambda on deploy.
CUMULUS-1456
- @cumulus/api endpoints default error handler uses boom package to format errors, which is consistent with other API endpoint errors.

Fixed

CUMULUS-1432 logs endpoint filter correctly filters logs by level
CUMULUS-1484 useMessageAdapter now does not set CUMULUS_MESSAGE_ADAPTER_DIR when true

Removed

CUMULUS-1394
- Removed sfTracker SNS topic. Replaced by three new SNS topics for granule, execution, and PDR ingest notifications.
- Removed unused functions from @cumulus/common/aws:
  - getGranuleS3Params()
  - setGranuleStatus()

1.14.1

published 1.14.1 • 5 years ago

Changelog

[v1.14.1] - 2019-08-29

Fixed

CUMULUS-1455
- CMR token links updated to point to CMR legacy services rather than echo
CUMULUS-1211
- Errors thrown during granule discovery are no longer swallowed and ignored. Rather, errors are propagated to allow for proper error-handling and meaningful messaging.

1.13.5

published 1.13.5 • 5 years ago

Changelog