@cumulus/discover-granules

@cumulus/discover-granules

Discover Granules in FTP/HTTP/HTTPS/SFTP/S3 endpoints

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Contributing

To make a contribution, please see our contributing guidelines.

Changelog

[v19.1.0] 2024-10-07

Migration Notes

This release contains changes listed here as well as changes listed in v19.0.0, despite v19.0.0 being deprecated. Please review Changelog entries and Migration Notes for each Cumulus version between your current version and v19.1.0 as normal.

Added

• CUMULUS-3020

  • Updated sfEventSqsToDbRecords to allow override of the default value (var.rds_connection_timing_configuration.acquireTimeoutMillis / 1000) + 60) via a key 'sfEventSqsToDbRecords' on var.lambda_timeouts on the main cumulus module/archive module

  Please note - updating this configuration is for adavanced users only. Value changes will modify the visibility timeout on sfEventSqsToDbRecordsDeadLetterQueue and sfEventSqsToDbRecordsInputQueue and may lead to system instability.

• CUMULUS-3756

  • Added excludeFileRegex configuration to UpdateGranulesCmrMetadataFileLinks
  • This is to allow files matching specified regex to be excluded when updating the Related URLs list
  • Defaults to the current behavior of excluding no files.

• CUMULUS-3773

  • Added sftpFastDownload configuration to SyncGranule task.
  • Updated @cumulus/sftp-client and @cumulus/ingest/SftpProviderClient to support both regular and fastDownload.
  • Added sftp support to FakeProvider
  • Added sftp integration test

Changed

• CUMULUS-3928
  • updated publish scripting to use [e-mail] for user email
  • updated publish scripting to use esm over common import of latest-version
  • updated bigint testing to remove intermitted failure source.
  • updated postgres dependency version
• CUMULUS-3838
  • Updated python dependencies to latest:
    • cumulus-process-py 1.4.0
    • cumulus-message-adapter-python 2.3.0
• CUMULUS-3906
  • Bumps example ORCA deployment to version v10.0.1.

Fixed

• CUMULUS-3904
  • Passed sqs_message_consumer_watcher_message_limit and sqs_message_consumer_watcher_time_limit through the cumulus terraform module to the ingest terraform module.
• CUMULUS-3902
  • Update error handling to use AWS SDK V3 error classes instead of properties on js objects