@cumulus/discover-granules

@cumulus/discover-granules

Discover Granules in FTP/HTTP/HTTPS/SFTP/S3 endpoints

What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Contributing

See Cumulus README

Changelog

[v1.10.0] - 2018-08-31

Removed

• Removed unused and broken code from @cumulus/common
  • Removed @cumulus/common/test-helpers
  • Removed @cumulus/common/task
  • Removed @cumulus/common/message-source
  • Removed the getPossiblyRemote function from @cumulus/common/aws
  • Removed the startPromisedSfnExecution function from @cumulus/common/aws
  • Removed the getCurrentSfnTask function from @cumulus/common/aws

Changed

• CUMULUS-839 - In @cumulus/sync-granule, 'collection' is now an optional config parameter

Fixed

• CUMULUS-859 Moved duplicate code in @cumulus/move-granules and @cumulus/post-to-cmr to @cumulus/ingest. Fixed imports making assumptions about directory structure.
• @cumulus/ingest/consumer correctly limits the number of messages being received and processed from SQS. Details:
  • Background: @cumulus/api includes a lambda <stack-name>-sqs2sf which processes messages from the <stack-name>-startSF SQS queue every minute. The sqs2sf lambda uses @cumulus/ingest/consumer to receive and process messages from SQS.
  • Bug: More than messageLimit number of messages were being consumed and processed from the <stack-name>-startSF SQS queue. Many step functions were being triggered simultaneously by the lambda <stack-name>-sqs2sf (which consumes every minute from the startSF queue) and resulting in step function failure with the error: An error occurred (ThrottlingException) when calling the GetExecutionHistory.
  • Fix: @cumulus/ingest/consumer#processMessages now processes messages until timeLimit has passed OR once it receives up to messageLimit messages. sqs2sf is deployed with a default messageLimit of 10.
  • IMPORTANT NOTE: consumer will actually process up to messageLimit * 2 - 1 messages. This is because sometimes receiveSQSMessages will return less than messageLimit messages and thus the consumer will continue to make calls to receiveSQSMessages. For example, given a messageLimit of 10 and subsequent calls to receiveSQSMessages returns up to 9 messages, the loop will continue and a final call could return up to 10 messages.