@cumulus/test-data

@cumulus/test-data

@cumulus/test-data provides a collection of example data for use in testing Cumulus modules.

What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Installation

npm install @cumulus/test-data

Using data from this package

Using require or import:

const payload = require('@cumulus/test-data/payloads/new-message-schema/ingest.json');
import payload from '@cumulus/test-data/payloads/new-message-schema/ingest.json';

Contributing

See Cumulus README

Changelog

[v1.11.0] - 2018-11-30

Please Note

• Redeploy IAM roles:
  • CUMULUS-817 includes a migration that requires reconfiguration/redeployment of IAM roles. Please see the upgrade instructions for more information.
  • CUMULUS-977 includes a few new SNS-related permissions added to the IAM roles that will require redeployment of IAM roles.
• cumulus-message-adapter v1.0.13+ is required for @cumulus/api granule reingest API to work properly. The latest version should be downloaded automatically by kes.
• A TOKEN_SECRET value (preferably 256-bit for security) must be added to .env to securely sign JWTs used for authorization in @cumulus/api

Changed

• CUUMULUS-1000 - Distribution endpoint now persists logins, instead of redirecting to Earthdata Login on every request
• CUMULUS-783 CUMULUS-790 - Updated @cumulus/sync-granule and @cumulus/move-granules tasks to always overwrite existing files for manually-triggered reingest.
• CUMULUS-906 - Updated @cumulus/api granule reingest API to
  • add reingestGranule: true and forceDuplicateOverwrite: true to Cumulus message cumulus_meta.cumulus_context field to indicate that the workflow is a manually triggered re-ingest.
  • return warning message to operator when duplicateHandling is not replace
  • cumulus-message-adapter v1.0.13+ is required.
• CUMULUS-793 - Updated the granule move PUT request in @cumulus/api to reject the move with a 409 status code if one or more of the files already exist at the destination location
• Updated @cumulus/helloworld to use S3 to store state for pass on retry tests
• Updated @cumulus/ingest:
  • [Required for MAAP] http.js#list will now find links with a trailing whitespace
  • Removed code from granule.js which looked for files in S3 using { Bucket: discoveredFile.bucket, Key: discoveredFile.name }. This is obsolete since @cumulus/ingest uses a file-staging and constructCollectionId() directory prefixes by default.
• CUMULUS-989
  • Updated @cumulus/api to use JWT (JSON Web Token) as the transport format for API authorization tokens and to use JWT verification in the request authorization
  • Updated /token endpoint in @cumulus/api to return tokens as JWTs
  • Added a /refresh endpoint in @cumulus/api to request new access tokens from the OAuth provider using the refresh token
  • Added refreshAccessToken to @cumulus/api/lib/EarthdataLogin to manage refresh token requests with the Earthdata OAuth provider

Added

• CUMULUS-1050
  • Separated configuration flags for originalPayload/finalPayload cleanup such that they can be set to different retention times
• CUMULUS-798
  • Added daily Executions cleanup CloudWatch event that triggers cleanExecutions lambda
  • Added cleanExecutions lambda that removes finalPayload/originalPayload field entries for records older than configured timeout value (execution_payload_retention_period), with a default of 30 days
• CUMULUS-815/816
  • Added 'originalPayload' and 'finalPayload' fields to Executions table
  • Updated Execution model to populate originalPayload with the execution payload on record creation
  • Updated Execution model code to populate finalPayload field with the execution payload on execution completion
  • Execution API now exposes the above fields
• CUMULUS-977
  • Rename kinesisConsumer to messageConsumer as it handles both Kinesis streams and SNS topics as of this version.
  • Add sns-type rule support. These rules create a subscription between an SNS topic and the messageConsumer. When a message is received, messageConsumer is triggered and passes the SNS message (JSON format expected) in its entirety to the workflow in the payload field of the Cumulus message. For more information on sns-type rules, see the documentation.
• CUMULUS-975
  • Add KinesisInboundEventLogger and KinesisOutboundEventLogger API lambdas. These lambdas are utilized to dump incoming and outgoing ingest workflow kinesis streams to cloudwatch for analytics in case of AWS/stream failure.
  • Update rules model to allow tracking of log_event ARNs related to Rule event logging. Kinesis rule types will now automatically log incoming events via a Kinesis event triggered lambda. CUMULUS-975-migration-4
  • Update migration code to require explicit migration names per run
  • Added migration_4 to migrate/update existing Kinesis rules to have a log event mapping
  • Added new IAM policy for migration lambda
• CUMULUS-775
  • Adds a instance metadata endpoint to the @cumulus/api package.
  • Adds a new convenience function hostId to the @cumulus/cmrjs to help build environment specific cmr urls.
  • Fixed @cumulus/cmrjs.searchConcept to search and return CMR results.
  • Modified @cumulus/cmrjs.CMR.searchGranule and @cumulus/cmrjs.CMR.searchCollection to include CMR's provider as a default parameter to searches.
• CUMULUS-965
  • Add @cumulus/test-data.loadJSONTestData(), @cumulus/test-data.loadTestData(), and @cumulus/test-data.streamTestData() to safely load test data. These functions should be used instead of using require() to load test data, which could lead to tests interfering with each other.
  • Add a @cumulus/common/util/deprecate() function to mark a piece of code as deprecated
• CUMULUS-986
  • Added waitForTestExecutionStart to @cumulus/integration-tests
• CUMULUS-919
  • In @cumulus/deployment, added support for NGAP permissions boundaries for IAM roles with useNgapPermissionBoundary flag in iam/config.yml. Defaults to false.

Fixed

• Fixed a bug where FTP sockets were not closed after an error, keeping the Lambda function active until it timed out [CUMULUS-972]
• CUMULUS-656
  • The API will no longer allow the deletion of a provider if that provider is referenced by a rule
  • The API will no longer allow the deletion of a collection if that collection is referenced by a rule
• Fixed a bug where @cumulus/sf-sns-report was not pulling large messages from S3 correctly.

Deprecated

• @cumulus/ingest/aws/StepFunction.pullEvent(). Use @cumulus/common/aws.pullStepFunctionEvent().
• @cumulus/ingest/consumer.Consume due to unpredictable implementation. Use @cumulus/ingest/consumer.Consumer. Call Consumer.consume() instead of Consume.read().