@cumulus/cmrjs

@cumulus/cmrjs

A Node.js client for Cumulus-specific functionality for reading from, writing to, and deleting from NASA's Common Metadata Repository (CMR) API. General CMR functionality should go in @cumulus/cmr-client.

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Test

Test with npm run test.

CMR Docs

Documentation on the CMR REST API endpoints is here:

• https://cmr.earthdata.nasa.gov/search/site/search_api_docs.html
• https://cmr.earthdata.nasa.gov/ingest/site/ingest_api_docs.html

Contributing

To make a contribution, please see our contributing guidelines.

Changelog

[v21.1.0]

Migration Notes

• This release updates all core integration deployments to target cumulus-message-adapter v1.5.0. It is suggested that users update their deployment to utilize the updated CMA. Updates are not required for compatibility in custom lambdas.

Notable Changes

• CUMULUS-4124 When these changes are deployed, if no action is taken to reconfigure the cron, it will run once per day in the early morning, archiving

  • 100k granules
  • 100k executions
  • that are more than 1 year old.

  Being archived changes nothing about the record except to set a boolean flag (archived=true). this behavior can be reconfigured or turned off entirely. see features/record_archival.md for more details.

Added

• CUMULUS-4124
  • Add api endpoint granules/archive to archive granules
  • Add api endpoint executions/archive to archive executions
  • Task lambda to call above api endpoints with configuration
  • Add cron scheduler to call above endpoints and archive old records
• CUMULUS-4032
  • Added S3 jitter functionality to prevent AWS S3 SlowDown errors during high-concurrency operations
  • Added sync_granule_s3_jitter_max_ms Terraform variable to configure random jitter delay (0-59000ms) for SyncGranule task
  • S3 operations in @cumulus/aws-client now support optional jitter via S3_JITTER_MAX_MS environment variable
  • Jitter is applied to: headObject, putObject, copyObject, getObject, downloadS3File, promiseS3Upload, and multipartCopyObject

Changed

• CUMULUS-4244
  • Improve logging for Ingest Granules
    • Upgrade log level from debug to error for 403/401 errors
    • Add detailed error context (status code, error type, bucket, key)
    • Add actionable remediation suggestions for permission issues
    • Add try-catch in write-granules.js for better error context
• CUMULUS-4155
  • Update Cumulus integration tests to utilize:
    • Cumulus Message Adapter: v2.0.5
    • Cumulus Message Adapter-py: v2.4.0
    • Cumulus Process: 1.6.0
  • Update all Python dependencies to use boto >=1.40.29
  • Update all Core integration lambdas to use Python 3.12
  • Update external CNM lambdas to run on Java 21 in integration
• CUMULUS-4191
  • Updated messageConsumer and sqsMessageConsumer Lambdas to apply rule filtering based on the provider from the record message.
  • Updated messageConsumer lambda handler to async/await style
• CUMULUS-4200
  • updated metrics_es_host terraform variable description and validation
  • Users should ensure that the metrics_es_host does not include https://
• CUMULUS-4242
  • Skipped lzards api response assertions from lzards integration tests due to lzards api changes
• CUMULUS-4252
  • Fixed @aws-client/S3 unit test failures caused by stricter validation introduced in @aws-sdk/lib-storage@3.896.0
• CUMULUS-4242
  • Updated @cumulus/lizards-api-client to include configured provider via lzards_provider env var in all queries
  • Updated LZARDS integration tests to work with updated API client query requirements for API version 1.5.25
• CUMULUS-4232
  • Update MoveGranules CUMULUS-4078 behavior such that it no longer defaults to throwing on an orphan (S3 file record not in database) situation when checking cross-collection file collisions.
  • Added configuration crossCollectionThrowOnObjectNotFound to allow setting MoveGranules to fail in a collision/orphan situation
  • Added collectionCheckRetryCount to allow configuration of the retry count for the MoveGranules crossCollection lookup
• CUMULUS-4254
  • Moved @cumulus/api/lib/utils.errorify function to @cumulus/errors and updated it to remove circular reference
  • Used errorify instead of JSON.stringify for AWS errors
  • Added required collection field to lzards api request in LzardsBackupSpec integration test to fix the bug in CUMULUS-4242