@cumulus/logger - npm Package Versions

published 1.14.2 • 5 years ago

Changelog

[v1.14.2] - 2019-10-08

BREAKING CHANGES

Your Cumulus Message Adapter version should be pinned to v1.0.13 or lower in your app/config.yml using message_adapter_version: v1.0.13 OR you should use the workflow migration steps below to work with CMA v1.1.1+.

CUMULUS-1394 - The implementation of the SfSnsReport Lambda requires additional environment variables for integration with the new ingest notification SNS topics. Therefore, you must update the definition of SfSnsReport in your lambdas.yml like so:

SfSnsReport:
  handler: index.handler
  timeout: 300
  source: node_modules/@cumulus/sf-sns-report/dist
  tables:
    - ExecutionsTable
  envs:
    execution_sns_topic_arn:
      function: Ref
      value: reportExecutionsSns
    granule_sns_topic_arn:
      function: Ref
      value: reportGranulesSns
    pdr_sns_topic_arn:
      function: Ref
      value: reportPdrsSns

CUMULUS-1447 - The newest release of the Cumulus Message Adapter (v1.1.1) requires that parameterized configuration be used for remote message functionality. Once released, Kes will automatically bring in CMA v1.1.1 without additional configuration.

Migration instructions Oversized messages are no longer written to S3 automatically. In order to utilize remote messaging functionality, configure a ReplaceConfig AWS Step Function parameter on your CMA task:
```
ParsePdr:
  Parameters:
    cma:
      event.$: "$"
      ReplaceConfig:
        FullMessage: true
```
Accepted fields in ReplaceConfig include MaxSize, FullMessage, Path and TargetPath. See https://github.com/nasa/cumulus-message-adapter/blob/master/CONTRACT.md#remote-message-configuration for full details.

As this change is backward compatible in Cumulus Core, users wishing to utilize the previous version of the CMA may opt to transition to using a CMA lambda layer, or set message_adapter_version in their configuration to a version prior to v1.1.0.

PLEASE NOTE

CUMULUS-1394 - Ingest notifications are now provided via 3 separate SNS topics for executions, granules, and PDRs, instead of a single sftracker SNS topic. Whereas the sftracker SNS topic received a full Cumulus execution message, the new topics all receive generated records for the given object. The new topics are only published to if the given object exists for the current execution. For a given execution/granule/PDR, two messages will be received by each topic: one message indicating that ingest is running and another message indicating that ingest has completed or failed. The new SNS topics are:
- reportExecutions - Receives 1 message per execution
- reportGranules - Receives 1 message per granule in an execution
- reportPdrs - Receives 1 message per PDR

Added

CUMULUS-639
- Adds SAML JWT and launchpad token authentication to Cumulus API (configurable)
  - NOTE to authenticate with Launchpad ensure your launchpad user_id is in the <prefix>-UsersTable
  - when Cumulus configured to protect API via Launchpad:
    - New endpoints
      - GET /saml/login - starting point for SAML SSO creates the login request url and redirects to the SAML Identity Provider Service (IDP)
      - POST /saml/auth - SAML Assertion Consumer Service. POST receiver from SAML IDP. Validates response, logs the user in, and returns a SAML-based JWT.
  - Disabled endpoints
    - POST /refresh
    - Changes authorization worklow:
    - ensureAuthorized now presumes the bearer token is a JWT and tries to validate. If the token is malformed, it attempts to validate the token against Launchpad. This allows users to bring their own token as described here https://wiki.earthdata.nasa.gov/display/CUMULUS/Cumulus+API+with+Launchpad+Authentication. But it also allows dashboard users to manually authenticate via Launchpad SAML to receive a Launchpad-based JWT.
CUMULUS-1394
- Added Granule.generateGranuleRecord() method to granules model to generate a granule database record from a Cumulus execution message
- Added Pdr.generatePdrRecord() method to PDRs model to generate a granule database record from a Cumulus execution message
- Added helpers to @cumulus/common/message:
  - getMessageExecutionName() - Get the execution name from a Cumulus execution message
  - getMessageStateMachineArn() - Get the state machine ARN from a Cumulus execution message
  - getMessageExecutionArn() - Get the execution ARN for a Cumulus execution message
  - getMessageGranules() - Get the granules from a Cumulus execution message, if any.
- Added @cumulus/common/cloudwatch-event/isFailedSfStatus() to determine if a Step Function status from a Cloudwatch event is a failed status

Changed

CUMULUS-1308
- HTTP PUT of a Collection, Provider, or Rule via the Cumulus API now performs full replacement of the existing object with the object supplied in the request payload. Previous behavior was to perform a modification (partial update) by merging the existing object with the (possibly partial) object in the payload, but this did not conform to the HTTP standard, which specifies PATCH as the means for modifications rather than replacements.
CUMULUS-1375
- Migrate Cumulus from deprecated Elasticsearch JS client to new, supported one in @cumulus/api
CUMULUS-1485 Update @cumulus/cmr-client to return error message from CMR for validation failures.
CUMULUS-1394
- Renamed Execution.generateDocFromPayload() to Execution.generateRecord() on executions model. The method generates an execution database record from a Cumulus execution message.
CUMULUS-1432
- logs endpoint takes the level parameter as a string and not a number
- Elasticsearch term query generation no longer converts numbers to boolean
CUMULUS-1447
- Consolidated all remote message handling code into @common/aws
- Update remote message code to handle updated CMA remote message flags
- Update example SIPS workflows to utilize Parameterized CMA configuration
CUMULUS-1448 Refactor workflows that are mutating cumulus_meta to utilize meta field
CUMULUS-1451
- Elasticsearch cluster setting auto_create_index will be set to false. This had been causing issues in the bootstrap lambda on deploy.
CUMULUS-1456
- @cumulus/api endpoints default error handler uses boom package to format errors, which is consistent with other API endpoint errors.

Fixed

CUMULUS-1432 logs endpoint filter correctly filters logs by level
CUMULUS-1484 useMessageAdapter now does not set CUMULUS_MESSAGE_ADAPTER_DIR when true

Removed

CUMULUS-1394
- Removed sfTracker SNS topic. Replaced by three new SNS topics for granule, execution, and PDR ingest notifications.
- Removed unused functions from @cumulus/common/aws:
  - getGranuleS3Params()
  - setGranuleStatus()

1.14.1

published 1.14.1 • 5 years ago

Changelog

[v1.14.1] - 2019-08-29

Fixed

CUMULUS-1455
- CMR token links updated to point to CMR legacy services rather than echo
CUMULUS-1211
- Errors thrown during granule discovery are no longer swallowed and ignored. Rather, errors are propagated to allow for proper error-handling and meaningful messaging.

1.13.5

published 1.13.5 • 5 years ago

Changelog

[v1.13.5] - 2019-08-29 - [BACKPORT]

Fixed

CUMULUS-1455 - CMR token links updated to point to CMR legacy services rather than echo

1.14.0

published 1.14.0 • 5 years ago

Changelog

[v1.14.0] - 2019-08-22

PLEASE NOTE

We have encountered transient lambda service errors in our integration testing. Please handle transient service errors following these guidelines. The workflows in the example/workflows folder have been updated with retries configured for these errors.
CUMULUS-799 added additional IAM permissions to support reading CloudWatch and API Gateway, so you will have to redeploy your IAM stack.
CUMULUS-800 Several items:
- Delete existing API Gateway stages: To allow enabling of API Gateway logging, Cumulus now creates and manages a Stage resource during deployment. Before upgrading Cumulus, it is necessary to delete the API Gateway stages on both the Backend API and the Distribution API. Instructions are included in the documentation under Delete API Gateway Stages.
- Set up account permissions for API Gateway to write to CloudWatch: In a one time operation for your AWS account, to enable CloudWatch Logs for API Gateway, you must first grant the API Gateway permission to read and write logs to CloudWatch for your account. The AmazonAPIGatewayPushToCloudWatchLogs managed policy (with an ARN of arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs) has all the required permissions. You can find a simple how to in the documentation under Enable API Gateway Logging.
- Configure API Gateway to write logs to CloudWatch To enable execution logging for the distribution API set config.yaml apiConfigs.distribution.logApigatewayToCloudwatch value to true. More information Enable API Gateway Logs
- Configure CloudWatch log delivery: It is possible to deliver CloudWatch API execution and access logs to a cross-account shared AWS::Logs::Destination. An operator does this by adding the key logToSharedDestination to the config.yml at the default level with a value of a writable log destination. More information in the documentation under Configure CloudWatch Logs Delivery.
- Additional Lambda Logging: It is now possible to configure any lambda to deliver logs to a shared subscriptions by setting logToSharedDestination to the ARN of a writable location (either an AWS::Logs::Destination or a Kinesis Stream) on any lambda config. Documentation for Lambda Log Subscriptions
- Configure S3 Server Access Logs: If you are running Cumulus in an NGAP environment you may configure S3 Server Access Logs to be delivered to a shared bucket where the Metrics Team will ingest the logs into their ELK stack. Contact the Metrics team for permission and location.
CUMULUS-1368 The Cumulus distribution API has been deprecated and is being replaced by ASF's Thin Egress App. By default, the distribution API will not deploy. Please follow the instructions for deploying and configuring Thin Egress.

To instead continue to deploy and use the legacy Cumulus distribution app, add the following to your config.yml:

deployDistributionApi: true

If you deploy with no distribution app your deployment will succeed but you may encounter errors in your workflows, particularly in the MoveGranule task.

CUMULUS-1418 Users who are packaging the CMA in their Lambdas outside of Cumulus may need to update their Lambda configuration. Please see BREAKING CHANGES below for details.

Added

CUMULUS-642
- Adds Launchpad as an authentication option for the Cumulus API.
- Updated deployment documentation and added instructions to setup Cumulus API Launchpad authentication
CUMULUS-1418
- Adds usage docs/testing of lambda layers (introduced in PR1125), updates Core example tasks to use the updated cumulus-ecs-task and a CMA layer instead of kes CMA injection.
- Added Terraform module to publish CMA as layer to user account.
PR1125 - Adds layers config option to support deploying Lambdas with layers
PR1128 - Added useXRay config option to enable AWS X-Ray for Lambdas.
CUMULUS-1345
- Adds new variables to the app deployment under cmr.
- cmrEnvironment values are SIT, UAT, or OPS with UAT as the default.
- cmrLimit and cmrPageSize have been added as configurable options.
CUMULUS-1273
- Added lambda function EmsProductMetadataReport to generate EMS Product Metadata report
CUMULUS-1226
- Added API endpoint elasticsearch/index-from-database to index to an Elasticsearch index from the database for recovery purposes and elasticsearch/indices-status to check the status of Elasticsearch indices via the API.
CUMULUS-824
- Added new Collection parameter reportToEms to configure whether the collection is reported to EMS
CUMULUS-1357
- Added new BackendApi endpoint ems that generates EMS reports.
CUMULUS-1241
- Added information about queues with maximum execution limits defined to default workflow templates (meta.queueExecutionLimits)
CUMULUS-1311
- Added @cumulus/common/message with various message parsing/preparation helpers
CUMULUS-812
- Added support for limiting the number of concurrent executions started from a queue. See the data cookbook for more information.
CUMULUS-1337
- Adds cumulus.stackName value to the instanceMetadata endpoint.
CUMULUS-1368
- Added cmrGranuleUrlType to the @cumulus/move-granules task. This determines what kind of links go in the CMR files. The options are distribution, s3, or none, with the default being distribution. If there is no distribution API being used with Cumulus, you must set the value to s3 or none.
Added packages/s3-replicator Terraform module to allow same-region s3 replication to metrics bucket.
CUMULUS-1392
- Added tf-modules/report-granules Terraform module which processes granule ingest notifications received via SNS and stores granule data to a database. The module includes:
  - SNS topic for publishing granule ingest notifications
  - Lambda to process granule notifications and store data
  - IAM permissions for the Lambda
  - Subscription for the Lambda to the SNS topic
CUMULUS-1393
- Added tf-modules/report-pdrs Terraform module which processes PDR ingest notifications received via SNS and stores PDR data to a database. The module includes:
  - SNS topic for publishing PDR ingest notifications
  - Lambda to process PDR notifications and store data
  - IAM permissions for the Lambda
  - Subscription for the Lambda to the SNS topic
- Added unit tests for @cumulus/api/models/pdrs.createPdrFromSns()
CUMULUS-1400
- Added tf-modules/report-executions Terraform module which processes workflow execution information received via SNS and stores it to a database. The module includes:
  - SNS topic for publishing execution data
  - Lambda to process and store execution data
  - IAM permissions for the Lambda
  - Subscription for the Lambda to the SNS topic
- Added @cumulus/common/sns-event which contains helpers for SNS events:
  - isSnsEvent() returns true if event is from SNS
  - getSnsEventMessage() extracts and parses the message from an SNS event
  - getSnsEventMessageObject() extracts and parses message object from an SNS event
- Added @cumulus/common/cloudwatch-event which contains helpers for Cloudwatch events:
  - isSfExecutionEvent() returns true if event is from Step Functions
  - isTerminalSfStatus() determines if a Step Function status from a Cloudwatch event is a terminal status
  - getSfEventStatus() gets the Step Function status from a Cloudwatch event
  - getSfEventDetailValue() extracts a Step Function event detail field from a Cloudwatch event
  - getSfEventMessageObject() extracts and parses Step Function detail object from a Cloudwatch event
CUMULUS-1429
- Added tf-modules/data-persistence Terraform module which includes resources for data persistence in Cumulus:
  - DynamoDB tables
  - Elasticsearch with optional support for VPC
  - Cloudwatch alarm for number of Elasticsearch nodes
CUMULUS-1379 CMR Launchpad Authentication
- Added launchpad configuration to @cumulus/deployment/app/config.yml, and cloudformation templates, workflow message, lambda configuration, api endpoint configuration
- Added @cumulus/common/LaunchpadToken and @cumulus/common/launchpad to provide methods to get token and validate token
- Updated lambdas to use Launchpad token for CMR actions (ingest and delete granules)
- Updated deployment documentation and added instructions to setup CMR client for Launchpad authentication

Changed

CUMULUS-1232
- Added retries to update @cumulus/cmr-client updateToken()
CUMULUS-1245 CUMULUS-795
- Added additional ems configuration parameters for sending the ingest reports to EMS
- Added functionality to send daily ingest reports to EMS
CUMULUS-1241
- Removed the concept of "priority levels" and added ability to define a number of maximum concurrent executions per SQS queue
- Changed mapping of Cumulus message properties for the sqs2sfThrottle lambda:
  - Queue name is read from cumulus_meta.queueName
  - Maximum executions for the queue is read from meta.queueExecutionLimits[queueName], where queueName is cumulus_meta.queueName
- Changed sfSemaphoreDown lambda to only attempt decrementing semaphores when:
  - the message is for a completed/failed/aborted/timed out workflow AND
  - cumulus_meta.queueName exists on the Cumulus message AND
  - An entry for the queue name (cumulus_meta.queueName) exists in the the object meta.queueExecutionLimits on the Cumulus message
CUMULUS-1338
- Updated sfSemaphoreDown lambda to be triggered via AWS Step Function Cloudwatch events instead of subscription to sfTracker SNS topic
CUMULUS-1311
- Updated @cumulus/queue-granules to set cumulus_meta.queueName for queued execution messages
- Updated @cumulus/queue-pdrs to set cumulus_meta.queueName for queued execution messages
- Updated sqs2sfThrottle lambda to immediately decrement queue semaphore value if dispatching Step Function execution throws an error
CUMULUS-1362
- Granule processingStartTime and processingEndTime will be set to the execution start time and end time respectively when there is no sync granule or post to cmr task present in the workflow
CUMULUS-1400
- Deprecated @cumulus/ingest/aws/getExecutionArn. Use @cumulus/common/aws/getExecutionArn instead.

Fixed

CUMULUS-1439
- Fix bug with rule.logEventArn deletion on Kinesis rule update and fix unit test to verify
CUMULUS-796
- Added production information (collection ShortName and Version, granuleId) to EMS distribution report
- Added functionality to send daily distribution reports to EMS
CUMULUS-1319
- Fixed a bug where granule ingest times were not being stored to the database
CUMULUS-1356
- The Collection model's delete method now removes the specified item from the collection config store that was inserted by the create method. Previously, this behavior was missing.
CUMULUS-1374
- Addressed audit concerns (https://www.npmjs.com/advisories/782) in api package

BREAKING CHANGES

Changed

CUMULUS-1418
- Adding a default cmaDir key to configuration will cause CUMULUS_MESSAGE_ADAPTER_DIR to be set by default to /opt for any Lambda not setting useCma to true, or explicitly setting the CMA environment variable. In lambdas that package the CMA independently of the Cumulus packaging. Lambdas manually packaging the CMA should have their Lambda configuration updated to set the CMA path, or alternately if not using the CMA as a Lambda layer in this deployment set cmaDir to ./cumulus-message-adapter.

Removed

CUMULUS-1337
- Removes the S3 Access Metrics package added in CUMULUS-799
PR1130
- Removed code deprecated since v1.11.1:
  - Removed @cumulus/common/step-functions. Use @cumulus/common/StepFunctions instead.
  - Removed @cumulus/api/lib/testUtils.fakeFilesFactory. Use @cumulus/api/lib/testUtils.fakeFileFactory instead.
  - Removed @cumulus/cmrjs/cmr functions: searchConcept, ingestConcept, deleteConcept. Use the functions in @cumulus/cmr-client instead.
  - Removed @cumulus/ingest/aws.getExecutionHistory. Use @cumulus/common/StepFunctions.getExecutionHistory instead.

1.13.4

published 1.13.4 • 5 years ago

Changelog

[v1.13.4] - 2019-07-29

CUMULUS-1411 - Fix deployment issue when using a template override

1.13.3

published 1.13.3 • 5 years ago

Changelog

[v1.13.3] - 2019-07-26

CUMULUS-1345 Full backport of CUMULUS-1345 features - Adds new variables to the app deployment under cmr.
- cmrEnvironment values are SIT, UAT, or OPS with UAT as the default.
- cmrLimit and cmrPageSize have been added as configurable options.

1.13.2

published 1.13.2 • 5 years ago

Changelog