@cumulus/test-data

@cumulus/test-data

@cumulus/test-data provides a collection of example data for use in testing Cumulus modules.

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Installation

npm install @cumulus/test-data

Using data from this package

Using require or import:

const payload = require('@cumulus/test-data/payloads/new-message-schema/ingest.json');
import payload from '@cumulus/test-data/payloads/new-message-schema/ingest.json';

Contributing

To make a contribution, please see our contributing guidelines.

Changelog

[v1.20.0] 2020-03-12

BREAKING CHANGES

• CUMULUS-1714
  • Changed the format of the message sent to the granule SNS Topic. Message includes the granule record under record and the type of event under event. Messages with deleted events will have the record that was deleted with a deletedAt timestamp. Options for event are Create | Update | Delete
• CUMULUS-1769 - deploy_to_ngap is now a required variable for the tf-modules/cumulus module. For those deploying to NGAP environments, this variable should always be set to true.

Notable changes

• CUMULUS-1739 - You can now exclude Elasticsearch from your tf-modules/data-persistence deployment (via include_elasticsearch = false) and your tf-modules/cumulus module will still deploy successfully.

• CUMULUS-1769 - If you set deploy_to_ngap = true for the tf-modules/archive Terraform module, you can only deploy your archive API gateway as PRIVATE, not EDGE.

Added

• Added @cumulus/aws-client/S3.getS3ObjectReadStreamAsync() to deal with S3 eventual consistency issues by checking for the existence an S3 object with retries before getting a readable stream for that object.
• CUMULUS-1769
  • Added deploy_to_ngap boolean variable for the tf-modules/cumulus and tf-modules/archive Terraform modules. This variable is required. For those deploying to NGAP environments, this variable should always be set to true.
• HYRAX-70
  • Add the hyrax-metadata-update task

Changed

• AccessToken.get() now enforces strongly consistent reads from DynamoDB
• CUMULUS-1739
  • Updated tf-modules/data-persistence to make Elasticsearch alarm resources and outputs conditional on the include_elasticsearch variable
  • Updated @cumulus/aws-client/S3.getObjectSize to include automatic retries for any failures from S3.headObject
• CUMULUS-1784
  • Updated @cumulus/api/lib/DistributionEvent.remoteIP() to parse the IP address in an S3 access log from the A-sourceip query parameter if present, otherwise fallback to the original parsing behavior.
• CUMULUS-1768
  • The stats/summary endpoint reports the distinct collections for the number of granules reported

Fixed

• CUMULUS-1739 - Fixed the tf-modules/cumulus and tf-modules/archive modules to make these Elasticsearch variables truly optional:

  • elasticsearch_domain_arn
  • elasticsearch_hostname
  • elasticsearch_security_group_id

• CUMULUS-1768

  • Fixed the stats/ endpoint so that data is correctly filtered by timestamp and processingTime is calculated correctly.

• CUMULUS-1769

  • In the tf-modules/archive Terraform module, the lifecycle block ignoring changes to the policy of the archive API gateway is now only enforced if deploy_to_ngap = true. This fixes a bug where users deploying outside of NGAP could not update their API gateway's resource policy when going from PRIVATE to EDGE, preventing their API from being accessed publicly.

• CUMULUS-1775

  • Fix/update api endpoint to use updated google auth endpoints such that it will work with new accounts

Removed

• CUMULUS-1768
  • Removed API endpoints stats/histogram and stats/average. All advanced stats needs should be acquired from Cloud Metrics or similarly configured ELK stack.