@cumulus/logger - npm Package Versions

[v1.11.1] - 2018-12-18

Please Note

• Ensure your app/config.yml has a clientId specified in the cmr section. This will allow CMR to identify your requests for better support and metrics.
  • For an example, please see the example config.

Added

• Added a /tokenDelete endpoint in @cumulus/api to delete access token records

Changed

• CUMULUS-678 @cumulus/ingest/crypto moved and renamed to @cumulus/common/key-pair-provider @cumulus/ingest/aws function: KMSDecryptionFailed and class: KMS extracted and moved to @cumulus/common and KMS is exported as KMSProvider from @cumulus/common/key-pair-provider @cumulus/ingest/granule functions: publish, getGranuleId, getXMLMetadataAsString, getMetadataBodyAndTags, parseXmlString, getCmrXMLFiles, postS3Object, contructOnlineAccessUrls, updateMetadata, extracted and moved to @cumulus/cmrjs getGranuleId, getCmrXMLFiles, publish, updateMetadata removed from @cumulus/ingest/granule and added to @cumulus/cmrjs; updateMetadata renamed updateCMRMetadata. @cumulus/ingest test files renamed.
• CUMULUS-1070
  • Add 'Client-Id' header to all @cumulus/cmrjs requests (made via searchConcept, ingestConcept, and deleteConcept).
  • Updated cumulus/example/app/config.yml entry for cmr.clientId to use stackName for easier CMR-side identification.

[v1.11.0] - 2018-11-30

Please Note

• Redeploy IAM roles:
  • CUMULUS-817 includes a migration that requires reconfiguration/redeployment of IAM roles. Please see the upgrade instructions for more information.
  • CUMULUS-977 includes a few new SNS-related permissions added to the IAM roles that will require redeployment of IAM roles.
• cumulus-message-adapter v1.0.13+ is required for @cumulus/api granule reingest API to work properly. The latest version should be downloaded automatically by kes.
• A TOKEN_SECRET value (preferably 256-bit for security) must be added to .env to securely sign JWTs used for authorization in @cumulus/api

Changed

• CUUMULUS-1000 - Distribution endpoint now persists logins, instead of redirecting to Earthdata Login on every request
• CUMULUS-783 CUMULUS-790 - Updated @cumulus/sync-granule and @cumulus/move-granules tasks to always overwrite existing files for manually-triggered reingest.
• CUMULUS-906 - Updated @cumulus/api granule reingest API to
  • add reingestGranule: true and forceDuplicateOverwrite: true to Cumulus message cumulus_meta.cumulus_context field to indicate that the workflow is a manually triggered re-ingest.
  • return warning message to operator when duplicateHandling is not replace
  • cumulus-message-adapter v1.0.13+ is required.
• CUMULUS-793 - Updated the granule move PUT request in @cumulus/api to reject the move with a 409 status code if one or more of the files already exist at the destination location
• Updated @cumulus/helloworld to use S3 to store state for pass on retry tests
• Updated @cumulus/ingest:
  • [Required for MAAP] http.js#list will now find links with a trailing whitespace
  • Removed code from granule.js which looked for files in S3 using { Bucket: discoveredFile.bucket, Key: discoveredFile.name }. This is obsolete since @cumulus/ingest uses a file-staging and constructCollectionId() directory prefixes by default.
• CUMULUS-989
  • Updated @cumulus/api to use JWT (JSON Web Token) as the transport format for API authorization tokens and to use JWT verification in the request authorization
  • Updated /token endpoint in @cumulus/api to return tokens as JWTs
  • Added a /refresh endpoint in @cumulus/api to request new access tokens from the OAuth provider using the refresh token
  • Added refreshAccessToken to @cumulus/api/lib/EarthdataLogin to manage refresh token requests with the Earthdata OAuth provider

Added

• CUMULUS-1050
  • Separated configuration flags for originalPayload/finalPayload cleanup such that they can be set to different retention times
• CUMULUS-798
  • Added daily Executions cleanup CloudWatch event that triggers cleanExecutions lambda
  • Added cleanExecutions lambda that removes finalPayload/originalPayload field entries for records older than configured timeout value (execution_payload_retention_period), with a default of 30 days
• CUMULUS-815/816
  • Added 'originalPayload' and 'finalPayload' fields to Executions table
  • Updated Execution model to populate originalPayload with the execution payload on record creation
  • Updated Execution model code to populate finalPayload field with the execution payload on execution completion
  • Execution API now exposes the above fields
• CUMULUS-977
  • Rename kinesisConsumer to messageConsumer as it handles both Kinesis streams and SNS topics as of this version.
  • Add sns-type rule support. These rules create a subscription between an SNS topic and the messageConsumer. When a message is received, messageConsumer is triggered and passes the SNS message (JSON format expected) in its entirety to the workflow in the payload field of the Cumulus message. For more information on sns-type rules, see the documentation.
• CUMULUS-975
  • Add KinesisInboundEventLogger and KinesisOutboundEventLogger API lambdas. These lambdas are utilized to dump incoming and outgoing ingest workflow kinesis streams to cloudwatch for analytics in case of AWS/stream failure.
  • Update rules model to allow tracking of log_event ARNs related to Rule event logging. Kinesis rule types will now automatically log incoming events via a Kinesis event triggered lambda. CUMULUS-975-migration-4
  • Update migration code to require explicit migration names per run
  • Added migration_4 to migrate/update existing Kinesis rules to have a log event mapping
  • Added new IAM policy for migration lambda
• CUMULUS-775
  • Adds a instance metadata endpoint to the @cumulus/api package.
  • Adds a new convenience function hostId to the @cumulus/cmrjs to help build environment specific cmr urls.
  • Fixed @cumulus/cmrjs.searchConcept to search and return CMR results.
  • Modified @cumulus/cmrjs.CMR.searchGranule and @cumulus/cmrjs.CMR.searchCollection to include CMR's provider as a default parameter to searches.
• CUMULUS-965
  • Add @cumulus/test-data.loadJSONTestData(), @cumulus/test-data.loadTestData(), and @cumulus/test-data.streamTestData() to safely load test data. These functions should be used instead of using require() to load test data, which could lead to tests interfering with each other.
  • Add a @cumulus/common/util/deprecate() function to mark a piece of code as deprecated
• CUMULUS-986
  • Added waitForTestExecutionStart to @cumulus/integration-tests
• CUMULUS-919
  • In @cumulus/deployment, added support for NGAP permissions boundaries for IAM roles with useNgapPermissionBoundary flag in iam/config.yml. Defaults to false.

Fixed

• Fixed a bug where FTP sockets were not closed after an error, keeping the Lambda function active until it timed out [CUMULUS-972]
• CUMULUS-656
  • The API will no longer allow the deletion of a provider if that provider is referenced by a rule
  • The API will no longer allow the deletion of a collection if that collection is referenced by a rule
• Fixed a bug where @cumulus/sf-sns-report was not pulling large messages from S3 correctly.

Deprecated

• @cumulus/ingest/aws/StepFunction.pullEvent(). Use @cumulus/common/aws.pullStepFunctionEvent().
• @cumulus/ingest/consumer.Consume due to unpredictable implementation. Use @cumulus/ingest/consumer.Consumer. Call Consumer.consume() instead of Consume.read().

[v1.10.4] - 2018-11-28

Added

• CUMULUS-1008
  • New config.yml parameter for SQS consumers: sqs_consumer_rate: (default 500), which is the maximum number of messages the consumer will attempt to process per execution. Currently this is only used by the sf-starter consumer, which runs every minute by default, making this a messages-per-minute upper bound. SQS does not guarantee the number of messages returned per call, so this is not a fixed rate of consumption, only attempted number of messages received.

Deprecated

• @cumulus/ingest/consumer.Consume due to unpredictable implementation. Use @cumulus/ingest/consumer.Consumer.

Changed

• Backported update of packages/api dependency @mapbox/dyno to 1.4.2 to mitigate event-stream vulnerability.

[v1.10.3] - 2018-10-31

Added

• CUMULUS-817
  • Added AWS Dead Letter Queues for lambdas that are scheduled asynchronously/such that failures show up only in cloudwatch logs.
• CUMULUS-956
  • Migrated developer documentation and data-cookbooks to Docusaurus
    • supports versioning of documentation
  • Added docs/docs-how-to.md to outline how to do things like add new docs or locally install for testing.
  • Deployment/CI scripts have been updated to work with the new format
• CUMULUS-811
  • Added new S3 functions to @cumulus/common/aws:
    • aws.s3TagSetToQueryString: converts S3 TagSet array to querystring (for use with upload()).
    • aws.s3PutObject: Returns promise of S3 putObject, which puts an object on S3
    • aws.s3CopyObject: Returns promise of S3 copyObject, which copies an object in S3 to a new S3 location
    • aws.s3GetObjectTagging: Returns promise of S3 getObjectTagging, which returns an object containing an S3 TagSet.
  • @/cumulus/common/aws.s3PutObject defaults to an explicit ACL of 'private' if not overridden.
  • @/cumulus/common/aws.s3CopyObject defaults to an explicit TaggingDirective of 'COPY' if not overridden.

Deprecated

• CUMULUS-811
  • Deprecated @cumulus/ingest/aws.S3. Member functions of this class will now log warnings pointing to similar functionality in @cumulus/common/aws.