@cumulus/discover-granules

@cumulus/discover-granules

Discover Granules in FTP/HTTP/HTTPS/SFTP/S3 endpoints

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Contributing

To make a contribution, please see our contributing guidelines.

Changelog

[v13.0.0] 2022-06-13

MIGRATION NOTES

• The changes introduced in CUMULUS-2955 should result in removal of files_granule_cumulus_id_index from the files table (added in the v11.1.1 release). The success of this operation is dependent on system ingest load.

  In rare cases where data-persistence deployment fails because the postgres-db-migration times out, it may be required to manually remove the index and then redeploy:

  DROP INDEX IF EXISTS files_granule_cumulus_id_index;
  

Breaking Changes

• CUMULUS-2931

  • Updates CustomBootstrap lambda to default to failing if attempting to remove a pre-existing cumulus-alias index that would collide with the required cumulus-alias alias. A configuration parameter elasticsearch_remove_index_alias_conflict on the cumulus and archive modules has been added to enable the original behavior that would remove the invalid index (and all it's data).
  • Updates @cumulus/es-client.bootstrapElasticSearch signature to be parameterized and accommodate a new parameter removeAliasConflict which allows/disallows the deletion of a conflicting cumulus-alias index

Notable changes

• CUMULUS-2929
  • Updated move-granule task to check the optional collection configuration parameter meta.granuleMetadataFileExtension to determine the granule metadata file. If none is specified, the granule CMR metadata or ISO metadata file is used.

Added

• CUMULUS-2929

  • Added optional collection configuration meta.granuleMetadataFileExtension to specify CMR metadata file extension for tasks that utilize metadata file lookups

• CUMULUS-2939

  • Added @cumulus/api/lambdas/start-async-operation to start an async operation

• CUMULUS-2953

  • Added skipMetadataCheck flag to config for Hyrax metadata updates task.
  • If this config flag is set to true, and a granule has no CMR file, the task will simply return the input values.

• CUMULUS-2966

  • Added extractPath operation and support of nested string replacement to url_path in the collection configuration

Changed

• CUMULUS-2965

  • Update cumulus-rds-tf module to ignore engine_version lifecycle changes

• CUMULUS-2967

  • Added fix example/spec/helpers/Provider that doesn't fail deletion 404 in case of deletion race conditions

• CUMULUS-2955

  • Updates 20220126172008_files_granule_id_index to not create an index on granule_cumulus_id on the files table.
  • Adds 20220609024044_remove_files_granule_id_index migration to revert changes from 20220126172008_files_granule_id_index on any deployed stacks that might have the index to ensure consistency in deployed stacks

• CUMULUS-2923

  • Changed public key setup for SFTP local testing.

• CUMULUS-2939

  • Updated @cumulus/api granules/bulk*, elasticsearch/index-from-database and POST reconciliationReports endpoints to invoke StartAsyncOperation lambda

Fixed

• CUMULUS-2863

  • Fixed @cumulus/api validateAndUpdateSqsRule method to allow 0 retries and 0 visibilityTimeout in rule's meta.

• CUMULUS-2961

  • Fixed data-migration2 granule migration logic to allow for DynamoDb granules that have a null/empty string value for execution. The migration will now migrate them without a linked execution.
  • Fixed @cumulus/api validateAndUpdateSqsRule method to allow 0 retries and 0 visibilityTimeout in rule's meta.

• CUMULUS-2959

  • Fixed @cumulus/api granules module to convert numeric productVolume to string when an old granule record is retrieved from DynamoDB.