@cumulus/cmrjs

@cumulus/cmrjs

A Node.js client for Cumulus-specific functionality for reading from, writing to, and deleting from NASA's Common Metadata Repository (CMR) API. General CMR functionality should go in @cumulus/cmr-client.

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Test

Test with npm run test.

CMR Docs

Documentation on the CMR REST API endpoints is here:

• https://cmr.earthdata.nasa.gov/search/site/search_api_docs.html
• https://cmr.earthdata.nasa.gov/ingest/site/ingest_api_docs.html

Contributing

To make a contribution, please see our contributing guidelines.

Changelog

[v9.7.0] 2021-10-01

Notable Changes

• CUMULUS-2583
  • The queue-granules task now updates granule status to queued when a granule is queued. In order to prevent issues with the private API endpoint and Lambda API request and concurrency limits, this functionality runs with limited concurrency, which may increase the task's overall runtime when large numbers of granules are being queued. If you are facing Lambda timeout errors with this task, we recommend converting your queue-granules task to an ECS activity. This concurrency is configurable via the task config's concurrency value.
• CUMULUS-2676
  • The discover-granules task has been updated to limit concurrency on checks to identify and skip already ingested granules in order to prevent issues with the private API endpoint and Lambda API request and concurrency limits. This may increase the task's overall runtime when large numbers of granules are discovered. If you are facing Lambda timeout errors with this task, we recommend converting your discover-granules task to an ECS activity. This concurrency is configurable via the task config's concurrency value.
• Updated memory of <prefix>-sfEventSqsToDbRecords Lambda to 1024MB

Added

• CUMULUS-2000
  • Updated @cumulus/queue-granules to respect a new config parameter: preferredQueueBatchSize. Queue-granules will respect this batchsize as best as it can to batch granules into workflow payloads. As workflows generally rely on information such as collection and provider expected to be shared across all granules in a workflow, queue-granules will break batches up by collection, as well as provider if there is a provider field on the granule. This may result in batches that are smaller than the preferred size, but never larger ones. The default value is 1, which preserves current behavior of queueing 1 granule per workflow.
• CUMULUS-2630
  • Adds a new workflow DiscoverGranulesToThrottledQueue that discovers and writes granules to a throttled background queue. This allows discovery and ingest of larger numbers of granules without running into limits with lambda concurrency.

Changed

• CUMULUS-2720
  • Updated Core CI scripts to validate CHANGELOG diffs as part of the lint process
• CUMULUS-2695
  • Updates the example/cumulus-tf deployment to change archive_api_reserved_concurrency from 8 to 5 to use fewer reserved lambda functions. If you see throttling errors on the <stack>-apiEndpoints you should increase this value.
  • Updates cumulus-tf/cumulus/variables.tf to change archive_api_reserved_concurrency from 8 to 15 to prevent throttling on the dashboard for default deployments.
• CUMULUS-2584
  • Updates api/endpoints/execution-status.js get method to include associated granules, as an array, for the provided execution.
  • Added getExecutionArnsByGranuleCumulusId returning a list of executionArns sorted by most recent first, for an input Granule Cumulus ID in support of the move of translatePostgresGranuleToApiGranule from RDS-Phase2 feature branch
  • Added getApiExecutionCumulusIds returning cumulus IDs for a given list of executions
• CUMULUS-NONE
  • Downgrades elasticsearch version in testing container to 5.3 to match AWS version.
  • Update serve.js -> eraseDynamoTables(). Changed the call Promise.all() to Promise.allSettled() to ensure all dynamo records (provider records in particular) are deleted prior to reseeding.

Fixed

• CUMULUS-2583
  • Fixed a race condition where granules set as “queued” were not able to be set as “running” or “completed”