@cumulus/discover-granules

@cumulus/discover-granules

Discover Granules in FTP/HTTP/SFTP endpoints

What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Contributing

See Cumulus README

Changelog

[v1.2.0] - 2018-03-20

Fixed

• Update vulnerable npm packages [CUMULUS-425]
• @cumulus/api: kinesis-consumer.js uses sf-scheduler.js#schedule instead of placing a message directly on the startSF SQS queue. This is a fix for CUMULUS-359 because sf-scheduler.js#schedule looks up the provider and collection data in DynamoDB and adds it to the meta object of the enqueued message payload.
• @cumulus/api: kinesis-consumer.js catches and logs errors instead of doing an error callback. Before this change, kinesis-consumer was failing to process new records when an existing record caused an error because it would call back with an error and stop processing additional records. It keeps trying to process the record causing the error because it's "position" in the stream is unchanged. Catching and logging the errors is part 1 of the fix. Proposed part 2 is to enqueue the error and the message on a "dead-letter" queue so it can be processed later (CUMULUS-413).
• CUMULUS-260: "PDR page on dashboard only shows zeros." The PDR stats in LPDAAC are all 0s, even if the dashboard has been fixed to retrieve the correct fields. The current version of pdr-status-check has a few issues.
  • pdr is not included in the input/output schema. It's available from the input event. So the pdr status and stats are not updated when the ParsePdr workflow is complete. Adding the pdr to the input/output of the task will fix this.
  • pdr-status-check doesn't update pdr stats which prevent the real time pdr progress from showing up in the dashboard. To solve this, added lambda function sf-sns-report which is copied from @cumulus/api/lambdas/sf-sns-broadcast with modification, sf-sns-report can be used to report step function status anywhere inside a step function. So add step sf-sns-report after each pdr-status-check, we will get the PDR status progress at real time.
  • It's possible an execution is still in the queue and doesn't exist in sfn yet. Added code to handle 'ExecutionDoesNotExist' error when checking the execution status.
• Fixed aws.cloudwatchevents() typo in packages/ingest/aws.js. This typo was the root cause of the error: Error: Could not process scheduled_ingest, Error: : aws.cloudwatchevents is not a constructor seen when trying to update a rule.

Removed

• @cumulus/ingest/aws: Remove queueWorkflowMessage which is no longer being used by @cumulus/api's kinesis-consumer.js.