@cumulus/test-data
@cumulus/test-data provides a collection of example data for use in testing Cumulus modules.
About Cumulus
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
Cumulus Documentation
Installation
npm install @cumulus/test-data
Using data from this package
Using require
or import
:
const payload = require('@cumulus/test-data/payloads/new-message-schema/ingest.json');
import payload from '@cumulus/test-data/payloads/new-message-schema/ingest.json';
Contributing
To make a contribution, please see our contributing guidelines.
[v16.0.0] 2023-05-09
Notable Changes
- The async_operation_image property of cumulus module should be updated to pull
the ECR image for cumuluss/async-operation:46
MIGRATION notes
PI release version
When updating directly to v16 from prior releases older that V15, please make sure to
read through all prior release notes.
Notable migration concerns since the last PI release version (11.1.x):
- [v14.1.0] - Postgres compatibility update to Aurora PostgreSQL 11.13.
- [v13.1.0] - Postgres update to add
files_granules_cumulus_id_index
to the
files
table may require manual steps depending on load.
RDS Phase 3 migration notes
This release includes updates that remove existing DynamoDB tables as part of
release deployment process. This release cannot be properly rolled back in
production as redeploying a prior version of Cumulus will not recover the
associated Dynamo tables.
Please read the full change log for RDS Phase 3 and consult the RDS Phase 3 update
documentation
API Endpoint Versioning
As part of the work on CUMULUS-3072, we have added a required header for the
granule PUT/PATCH endpoints -- to ensure that older clients/utilities do not
unexpectedly make destructive use of those endpoints, a validation check of a
header value against supported versions has been implemented.
Moving forward, if a breaking change is made to an existing endpoint that
requires user updates, as part of that update we will set the current version of
the core API and require a header that confirms the client is compatible with
the version required or greater.
In this instance, the granule PUT/PATCH
endpoints will require a Cumulus-API-Version
value of at least 2
.
curl --request PUT https://example.com/granules/granuleId.A19990103.006.1000\
--header 'Cumulus-API-Version: 2'\
--header 'Content-Type: application/json'\
--header 'Authorization: Bearer ReplaceWithToken'\
--data ...
Users/clients that do not make use of these endpoints will not be impacted.
RDS Phase 3
Breaking Changes
- CUMULUS-2688
- Updated bulk operation logic to use collectionId in addition to granuleId to fetch granules.
- Tasks using the
bulk-operation
Lambda should provide collectionId and granuleId e.g. { granuleId: xxx, collectionId: xxx }
- CUMULUS-2856
- Update execution PUT endpoint to no longer respect message write constraints and update all values passed in
Changed
- CUMULUS-3282
- Updated internal granule endpoint parameters from :granuleName to :granuleId
for maintenance/consistency reasons
- CUMULUS-2312 - RDS Migration Epic Phase 3
- CUMULUS-2645
- Removed unused index functionality for all tables other than
ReconciliationReportsTable
from dbIndexer
lambda
- CUMULUS-2398
- Remove all dynamoDB updates for
@cumulus/api/ecs/async-operation/*
- Updates all api endpoints with updated signature for
asyncOperationsStart
calls - Remove all dynamoDB models calls from async-operations api endpoints
- CUMULUS-2801
- Move
getFilesExistingAtLocation
from api granules model to api/lib, update granules put
endpoint to remove model references
- CUMULUS-2804
- Updates api/lib/granule-delete.deleteGranuleAndFiles:
- Updates dynamoGranule -> apiGranule in the signature and throughout the dependent code
- Updates logic to make apiGranule optional, but pgGranule required, and
all lookups use postgres instead of ES/implied apiGranule values
- Updates logic to make pgGranule optional - in this case the logic removes the entry from ES only
- Removes all dynamo model logic from api/endpoints/granules
- Removes dynamo write logic from api/lib/writeRecords.*
- Removes dynamo write logic from api/lib/ingest.*
- Removes all granule model calls from api/lambdas/bulk-operations and any dependencies
- Removes dynamo model calls from api/lib/granule-remove-from-cmr.unpublishGranule
- Removes Post Deployment execution check from sf-event-sqs-to-db-records
- Moves describeGranuleExecution from api granule model to api/lib/executions.js
- CUMULUS-2806
- Remove DynamoDB logic from executions
POST
endpoint - Remove DynamoDB logic from sf-event-sqs-to-db-records lambda execution writes.
- Remove DynamoDB logic from executions
PUT
endpoint
- CUMULUS-2808
- Remove DynamoDB logic from executions
DELETE
endpoint
- CUMULUS-2809
- Remove DynamoDB logic from providers
PUT
endpoint - Updates DB models asyncOperation, provider and rule to return all fields on upsert.
- CUMULUS-2810
- Removes addition of DynamoDB record from API endpoint POST /provider/<name>
- CUMULUS-2811
- Removes deletion of DynamoDB record from API endpoint DELETE /provider/<name>
- CUMULUS-2817
- Removes deletion of DynamoDB record from API endpoint DELETE /collection/<name>/<version>
- CUMULUS-2814
- Move event resources deletion logic from
rulesModel
to rulesHelper
- CUMULUS-2815
- Move File Config and Core Config validation logic for Postgres Collections from
api/models/collections.js
to api/lib/utils.js
- CUMULUS-2813
- Removes creation and deletion of DynamoDB record from API endpoint POST /rules/
- CUMULUS-2816
- Removes addition of DynamoDB record from API endpoint POST /collections
- CUMULUS-2797
- Move rule helper functions to separate rulesHelpers file
- CUMULUS-2821
- Remove DynamoDB logic from
sfEventSqsToDbRecords
lambda
- CUMULUS-2856
- Update API/Message write logic to handle nulls as deletion in execution PUT/message write logic
Added
- CUMULUS-2312 - RDS Migration Epic Phase 3
- CUMULUS-2813
- Added function
create
in the db
model for Rules
to return an array of objects containing all columns of the created record.
- CUMULUS-2812
- Move event resources logic from
rulesModel
to rulesHelper
- CUMULUS-2820
- Remove deletion of DynamoDB record from API endpoint DELETE /pdr/<pdrName>
- CUMULUS-2688
- Add new endpoint to fetch granules by collectionId as well as granuleId: GET /collectionId/granuleId
- Add new endpoints to update and delete granules by collectionId as well as
granuleId
Removed
- CUMULUS-2994
- Delete code/lambdas that publish DynamoDB stream events to SNS
- CUMULUS-3226
- Removed Dynamo Async Operations table
- CUMULUS-3199
- Removed DbIndexer lambda and all associated terraform resources
- CUMULUS-3009
- Removed Dynamo PDRs table
- CUMULUS-3008
- Removed DynamoDB Collections table
- CUMULUS-2815
- Remove update of DynamoDB record from API endpoint PUT /collections/<name>/<version>
- CUMULUS-2814
- Remove DynamoDB logic from rules
DELETE
endpoint
- CUMULUS-2812
- Remove DynamoDB logic from rules
PUT
endpoint
- CUMULUS-2798
- Removed AsyncOperations model
- CUMULUS-2797
- CUMULUS-2795
- Removed API executions model
- CUMULUS-2796
- Remove API pdrs model and all related test code
- Remove API Rules model and all related test code
- CUMULUS-2794
- Remove API Collections model and all related test code
- Remove lambdas/postgres-migration-count-tool, api/endpoints/migrationCounts and api-client/migrationCounts
- Remove lambdas/data-migration1 tool
- Remove lambdas/data-migration2 and
lambdas/postgres-migration-async-operation
- CUMULUS-2793
- Removed Provider Dynamo model and related test code
- CUMULUS-2792
- Remove API Granule model and all related test code
- Remove granule-csv endpoint
- CUMULUS-2645
- Removed dynamo structural migrations and related code from
@cumulus/api
- Removed
executeMigrations
lambda - Removed
granuleFilesCacheUpdater
lambda - Removed dynamo files table from
data-persistence
module. This table and
all of its data will be removed on deployment.
Added
- CUMULUS-3072
- Added
replaceGranule
to @cumulus/api-client/granules
to add usage of the
updated RESTful PUT logic
- CUMULUS-3121
- Added a map of variables for the cloud_watch_log retention_in_days for the various cloudwatch_log_groups, as opposed to keeping them hardcoded at 30 days. Can be configured by adding the <module>_<cloudwatch_log_group_name>_log_retention value in days to the cloudwatch_log_retention_groups map variable
- CUMULUS-3201
- Added support for sha512 as checksumType for LZARDs backup task.
Changed
- CUMULUS-3315
- Updated
@cumulus/api-client/granules.bulkOperation
to remove ids
parameter in favor of granules
parameter, in the form of a
@cumulus/types/ApiGranule
that requires the following keys: [granuleId, collectionId]
- CUMULUS-3307
- Pinned cumulus dependency on
pg
to v8.10.x
- CUMULUS-3279
- Updated core dependencies on
xml2js
to v0.5.0
- Forcibly updated downstream dependency for
xml2js
in saml2-js
to
v0.5.0
- Added audit-ci CVE override until July 1 to allow for Core package releases
- CUMULUS-3106
- Updated localstack version to 1.4.0 and removed 'skip' from all skipped tests
- CUMULUS-3115
- Fixed DiscoverGranules' workflow's duplicateHandling when set to
skip
or error
to stop retrying
after receiving a 404 Not Found Response Error from the cumulus-api
.
- CUMULUS-3165
- Update example/cumulus-tf/orca.tf to use orca v6.0.3
Fixed
- CUMULUS-3315
- Update CI scripts to use shell logic/GNU timeout to bound test timeouts
instead of NPM
parallel
package, as timeouts were not resulting in
integration test failure
- CUMULUS-3223
- Update
@cumulus/cmrjs/cmr-utils.getGranuleTemporalInfo
to handle the error when the cmr file s3url is not available - Update
sfEventSqsToDbRecords
lambda to return partial batch failure,
and only reprocess messages when cumulus message can't be retrieved from the execution events. - Update
@cumulus/cumulus-message-adapter-js
to 2.0.5
for all cumulus tasks