@cumulus/test-data

@cumulus/test-data

@cumulus/test-data provides a collection of example data for use in testing Cumulus modules.

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Installation

npm install @cumulus/test-data

Using data from this package

Using require or import:

const payload = require('@cumulus/test-data/payloads/new-message-schema/ingest.json');
import payload from '@cumulus/test-data/payloads/new-message-schema/ingest.json';

Contributing

To make a contribution, please see our contributing guidelines.

Changelog

[v9.3.0] 2021-07-26

BREAKING CHANGES

• All API requests made by @cumulus/api-client will now throw an error if the status code does not match the expected response (200 for most requests and 202 for a few requests that trigger async operations). Previously the helpers in this package would return the response regardless of the status code, so you may need to update any code using helpers from this package to catch or to otherwise handle errors that you may encounter.
• The Cumulus API Lambda function has now been configured with reserved concurrency to ensure availability in a high-concurrency environment. However, this also caps max concurrency which may result in throttling errors if trying to reach the Cumulus API multiple times in a short period. Reserved concurrency can be configured with the archive_api_reserved_concurrency terraform variable on the Cumulus module and increased if you are seeing throttling errors. The default reserved concurrency value is 8.

Notable changes

• cmr_custom_host variable for cumulus module can now be used to configure Cumulus to integrate with a custom CMR host name and protocol (e.g. http://custom-cmr-host.com). Note that you must include a protocol (http:// or `https://) if specifying a value for this variable.
• The cumulus module configuration valuerds_connetion_heartbeat and it's behavior has been replaced by a more robust database connection 'retry' solution. Users can remove this value from their configuration, regardless of value. See the Changed section notes on CUMULUS-2528 for more details.

Added

• Added user doc describing new features related to the Cumulus dead letter archive.

• CUMULUS-2327

  • Added reserved concurrency setting to the Cumulus API lambda function.
  • Added relevant tfvars to the archive and cumulus terraform modules.

• CUMULUS-2460

  • Adds POST /executions/search-by-granules for retrieving executions from a list of granules or granule query
  • Adds searchExecutionsByGranules to @cumulus/api-client/executions

• CUMULUS-2475

  • Adds GET endpoint to distribution API

• CUMULUS-2463

  • PUT /granules reingest action allows a user to override the default execution to use by providing an optional workflowName or executionArn parameter on the request body.
  • PUT /granules/bulkReingest action allows a user to override the default execution/workflow combination to reingest with by providing an optional workflowName on the request body.

• Adds workflowName and executionArn params to @cumulus/api-client/reingestGranules

• CUMULUS-2476

  • Adds handler for authenticated HEAD Distribution requests replicating current behavior of TEA

• CUMULUS-2478

  • Implemented bucket map.
  • Implemented /locate endpoint
  • Cumulus distribution API checks the file request against bucket map:
    • retrieves the bucket and key from file path
    • determines if the file request is public based on the bucket map rather than the bucket type
    • (EDL only) restricts download from PRIVATE_BUCKETS to users who belong to certain EDL User Groups
    • bucket prefix and object prefix are supported
  • Add 'Bearer token' support as an authorization method

• CUMULUS-2486

  • Implemented support for custom headers
  • Added 'Bearer token' support as an authorization method

• CUMULUS-2487

  • Added integration test for cumulus distribution API

• CUMULUS-2569

  • Created bucket map cache for cumulus distribution API

• CUMULUS-2568

  • Add deletePdr/PDR deletion functionality to @cumulus/api-client/pdrs
  • Add removeCollectionAndAllDependencies to integration test helpers
  • Added example/spec/apiUtils.waitForApiStatus to wait for a record to be returned by the API with a specific value for status
  • Added example/spec/discoverUtils.uploadS3GranuleDataForDiscovery to upload granule data fixtures to S3 with a randomized granule ID for discover-granules based integration tests
  • Added example/spec/Collections.removeCollectionAndAllDependencies to remove a collection and all dependent objects (e.g. PDRs, granules, executions) from the database via the API
  • Added helpers to @cumulus/api-client:
    • pdrs.deletePdr - Delete a PDR via the API
    • replays.postKinesisReplays - Submit a POST request to the /replays endpoint for replaying Kinesis messages

• @cumulus/api-client/granules.getGranuleResponse to return the raw endpoint response from the GET /granules/<granuleId> endpoint

Changed

• Moved functions from @cumulus/integration-tests to example/spec/helpers/workflowUtils:
  • startWorkflowExecution
  • startWorkflow
  • executeWorkflow
  • buildWorkflow
  • testWorkflow
  • buildAndExecuteWorkflow
  • buildAndStartWorkflow
• example/spec/helpers/workflowUtils.executeWorkflow now uses waitForApiStatus to ensure that the execution is completed or failed before resolving
• example/spec/helpers/testUtils.updateAndUploadTestFileToBucket now accepts an object of parameters rather than positional arguments
• Removed PDR from the payload in the input payload test fixture for reconciliation report integration tests
• The following integration tests for PDR-based workflows were updated to use randomized granule IDs:
  • example/spec/parallel/ingest/ingestFromPdrSpec.js
  • example/spec/parallel/ingest/ingestFromPdrWithChildWorkflowMetaSpec.js
  • example/spec/parallel/ingest/ingestFromPdrWithExecutionNamePrefixSpec.js
  • example/spec/parallel/ingest/ingestPdrWithNodeNameSpec.js
• Updated the @cumulus/api-client/CumulusApiClientError error class to include new properties that can be accessed directly on the error object:
  • statusCode - The HTTP status code of the API response
  • apiMessage - The message from the API response
• Added params.pRetryOptions parameter to @cumulus/api-client/granules.deleteGranule to control the retry behavior
• Updated cmr_custom_host variable to accept a full protocol and host name (e.g. http://cmr-custom-host.com), whereas it previously only accepted a host name
• CUMULUS-2482
  • Switches the default distribution app in the example/cumulus-tf deployment to the new Cumulus Distribution
  • TEA is still available by following instructions in example/README.md
• CUMULUS-2463
  • Increases the duration of allowed backoff times for a successful test from 0.5 sec to 1 sec.
• CUMULUS-2528
  • Removed rds_connection_heartbeat as a configuration option from all Cumulus terraform modules
  • Removed dbHeartBeat as an environmental switch from @cumulus/db.getKnexClient in favor of more comprehensive general db connect retry solution
  • Added new rds_connection_timing_configuration string map to allow for configuration and tuning of Core's internal database retry/connection timeout behaviors. These values map to connection pool configuration values for tarn (https://github.com/vincit/tarn.js/) which Core's database module / knex(https://www.npmjs.com/package/knex) use for this purpose:
    • acquireTimeoutMillis
    • createRetryIntervalMillis
    • createTimeoutMillis
    • idleTimeoutMillis
    • reapIntervalMillis Connection errors will result in a log line prepended with 'knex failed on attempted connection error' and sent from '@cumulus/db/connection'
  • Updated @cumulus/db and all terraform mdules to set default retry configuration values for the database module to cover existing database heartbeat connection failures as well as all other knex/tarn connection creation failures.

Fixed

• Fixed bug where cmr_custom_host variable was not properly forwarded into archive, ingest, and sqs-message-remover modules from cumulus module
• Fixed bug where parse-pdr set a granule's provider to the entire provider record when a NODE_NAME is present. Expected behavior consistent with other tasks is to set the provider name in that field.
• CUMULUS-2568
  • Update reconciliation report integration test to have better cleanup/failure behavior
  • Fixed @cumulus/api-client/pdrs.getPdr to request correct endpoint for returning a PDR from the API
• CUMULUS-2620
  • Fixed a bug where a granule could be removed from CMR but still be set as published: true and with a CMR link in the Dynamo/PostgreSQL databases. Now, the CMR deletion and the Dynamo/PostgreSQL record updates will all succeed or fail together, preventing the database records from being out of sync with CMR.
  • Fixed @cumulus/api-client/pdrs.getPdr to request correct endpoint for returning a PDR from the API