@cumulus/discover-granules

@cumulus/discover-granules

Discover Granules in FTP/HTTP/HTTPS/SFTP/S3 endpoints

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Contributing

To make a contribution, please see our contributing guidelines.

Changelog

[v14.0.0] 2022-12-08

Breaking Changes

• CUMULUS-2915
  • API endpoint GET /executions/status/${executionArn} returns presignedS3Url and data
  • The user (dashboard) must read the s3SignedURL and data from the return
• CUMULUS-3070/3074
  • Updated granule PUT/POST endpoints to no longer respect message write constraints. Functionally this means that:
    • Granules with older createdAt values will replace newer ones, instead of ignoring the write request
    • Granules that attempt to set a non-complete state (e.g. 'queued' and 'running') will now ignore execution state/state change and always write
    • Granules being set to non-complete state will update all values passed in, instead of being restricted to ['createdAt', 'updatedAt', 'timestamp', 'status', 'execution']

Added

• CUMULUS-3070
  • Remove granules dynamoDb model logic that sets default publish value on record validation
  • Update API granule write logic to not set default publish value on record updates to avoid overwrite (PATCH behavior)
  • Update API granule write logic to publish to false on record creation if not specified
  • Update message granule write logic to set default publish value on record creation update.
  • Update granule write logic to set published to default value of false if null is explicitly set with intention to delete the value.
  • Removed dataType/version from api granule schema
  • Added @cumulus/api/endpoints/granules unit to cover duration overwrite logic for PUT/PATCH endpoint.
• CUMULUS-3098
  • Added task configuration setting named failTaskWhenFileBackupFail to the lzards-backup task. This setting is false by default, but when set to true, task will fail if one of the file backup request fails.

Changed

• Updated CI deploy process to utilize the distribution module in the published zip file which will be run against for the integration tests
• CUMULUS-2915
  • Updated API endpoint GET /executions/status/${executionArn} to return the presigned s3 URL in addition to execution status data
• CUMULUS-3045
  • Update GitHub FAQs:
    • Add new and refreshed content for previous sections
    • Add new dedicated Workflows section
• CUMULUS-3070
  • Updated API granule write logic to no longer require createdAt value in dynamo/API granule validation. Write-time createdAt defaults will be set in the case of new API granule writes without the value set, and createdAt will be overwritten if it already exists.
  • Refactored granule write logic to allow PATCH behavior on API granule update such that existing createdAt values will be retained in case of overwrite across all API granule writes.
  • Updated granule write code to validate written createdAt is synced between datastores in cases where granule.createdAt is not provided for a new granule.
  • Updated @cumulus/db/translate/granules.translateApiGranuleToPostgresGranuleWithoutNilsRemoved to validate incoming values to ensure values that can't be set to null are not
  • Updated @cumulus/db/translate/granules.translateApiGranuleToPostgresGranuleWithoutNilsRemoved to handle null values in incoming ApiGranule
  • Updated @cumulus/db/types/granules.PostgresGranule typings to allow for null values
  • Added ApiGranuleRecord to @cumulus/api/granule type to represent a written/retrieved from datastore API granule record.
  • Update API/Message write logic to handle nulls as deletion in granule PUT/message write logic
• CUMULUS-3075
  • Changed the API endpoint return value for a granule with no files. When a granule has no files, the return value beforehand for the translatePostgresGranuletoApiGranule, the function which does the translation of a Postgres granule to an API granule, was undefined, now changed to an empty array.
  • Existing behavior which relied on the pre-disposed undefined value was changed to instead accept the empty array.
  • Standardized tests in order to expect an empty array for a granule with no files files' object instead of undefined.
• CUMULUS-3077
  • Updated lambdas/data-migration2 granule and files migration to have a removeExcessFiles function like in write-granules that will remove file records no longer associated with a granule being migrated
• CUMULUS-3080
  • Changed the retention period in days from 14 to 30 for cloudwatch logs for NIST-5 compliance
• CUMULUS-3100
  • Updated POST granules endpoint to check if granuleId exists across all collections rather than a single collection.
  • Updated PUT granules endpoint to check if granuleId exists across a different collection and throw conflict error if so.
  • Updated logic for writing granules from a message to check if granuleId exists across a different collection and throw conflict error if so.

Fixed

• CUMULUS-3070
  • Fixed inaccurate typings for PostgresGranule in @cumulus/db/types/granule
  • Fixed inaccurate typings for @cumulus/api/granules.ApiGranule and updated to allow null
• CUMULUS-3104
  • Fixed TS compilation error on aws-client package caused by @aws-sdk/client-s3 3.202.0 upgrade
• CUMULUS-3116
  • Reverted the default ElasticSearch sorting behavior to the pre-13.3.0 configuration
  • Results from ElasticSearch are sorted by default by the timestamp field. This means that the order is not guaranteed if two or more records have identical timestamps as there is no secondary sort/tie-breaker.