@cumulus/cmrjs

@cumulus/cmrjs

A Node.js client to read from, write to, and delete from NASA's Common Metadata Repository (CMR) API.

What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Test

Test with npm run test.

CMR Docs

Documentation on the CMR REST API endpoints is here:

• https://cmr.earthdata.nasa.gov/search/site/search_api_docs.html
• https://cmr.earthdata.nasa.gov/ingest/site/ingest_api_docs.html

Contributing

See Cumulus README

Changelog

[v1.11.2] - 2019-2-15

Added

• CUMULUS-1169
  • Added a @cumulus/common/StepFunctions module. It contains functions for querying the AWS StepFunctions API. These functions have the ability to retry when a ThrottlingException occurs.
  • Added @cumulus/common/aws.retryOnThrottlingException(), which will wrap a function in code to retry on ThrottlingExceptions.
  • Added @cumulus/common/test-utils.throttleOnce(), which will cause a function to return a ThrottlingException the first time it is called, then return its normal result after that.
• CUMULUS-1103 Compare the collection holdings in CMR with Cumulus' internal data store
• CUMULUS-1099 Add support for UMMG JSON metadata versions > 1.4.
  • If a version is found in the metadata object, that version is used for processing and publishing to CMR otherwise, version 1.4 is assumed.
• CUMULUS-678
  • Added support for UMMG json v1.4 metadata files. reconcileCMRMetadata added to @cumulus/cmrjs to update metadata record with new file locations. @cumulus/common/errors adds two new error types CMRMetaFileNotFound and InvalidArgument. @cumulus/common/test-utils adds new function randomId to create a random string with id to help in debugging. @cumulus/common/BucketsConfig adds a new helper class BucketsConfig for working with bucket stack configuration and bucket names. @cumulus/common/aws adds new function s3PutObjectTagging as a convenience for the aws s3().putObjectTagging function. @cumulus/cmrjs Adds: - isCMRFile - Identify an echo10(xml) or UMMG(json) metadata file. - metadataObjectFromCMRFile Read and parse CMR XML file from s3. - updateCMRMetadata Modify a cmr metadata (xml/json) file with updated information. - publish2CMR Posts XML or UMMG CMR data to CMR service. - reconcileCMRMetadata Reconciles cmr metadata file after a file moves.
• Adds some ECS and other permissions to StepRole to enable running ECS tasks from a workflow
• Added Apache logs to cumulus api and distribution lambdas
• CUMULUS-1119 - Added @cumulus/integration-tests/api/EarthdataLogin.getEarthdataLoginRedirectResponse helper for integration tests to handle login with Earthdata and to return response from redirect to Cumulus API
• CUMULUS-673 Added @cumulus/common/file/getFileChecksumFromStream to get file checksum from a readable stream

Fixed

• CUMULUS-1123
  • Cloudformation template overrides now work as expected

Changed

• CUMULUS-1169

  • Deprecated the @cumulus/common/step-functions module.
  • Updated code that queries the StepFunctions API to use the retry-enabled functions from @cumulus/common/StepFunctions

• CUMULUS-1121

  • Schema validation is now strongly enforced when writing to the database. Additional properties are not allowed and will result in a validation error.

• CUMULUS-678 tasks/move-granules simplified and refactored to use functionality from cmrjs. ingest/granules.moveGranuleFiles now just moves granule files and returns a list of the updated files. Updating metadata now handled by @cumulus/cmrjs/reconcileCMRMetadata. move-granules.updateGranuleMetadata refactored and bugs fixed in the case of a file matching multiple collection.files.regexps. getCmrXmlFiles simplified and now only returns an object with the cmrfilename and the granuleId. @cumulus/test-processing - test processing task updated to generate UMM-G metadata

• CUMULUS-1043

  • @cumulus/api now uses express as the API engine.
  • All @cumulus/api endpoints on ApiGateway are consolidated to a single endpoint the uses {proxy+} definition.
  • All files under packages/api/endpoints along with associated tests are updated to support express's request and response objects.
  • Replaced environment variables internal, bucket and systemBucket with system_bucket.
  • Update @cumulus/integration-tests to work with updated cumulus-api express endpoints

• @cumulus/integration-tests - buildAndExecuteWorkflow and buildWorkflow updated to take a meta param to allow for additional fields to be added to the workflow meta

• CUMULUS-1049 Updated Retrieve Execution Status API in @cumulus/api: If the execution doesn't exist in Step Function API, Cumulus API returns the execution status information from the database.

• CUMULUS-1119

  • Renamed DISTRIBUTION_URL environment variable to DISTRIBUTION_ENDPOINT
  • Renamed DEPLOYMENT_ENDPOINT environment variable to DISTRIBUTION_REDIRECT_ENDPOINT
  • Renamed API_ENDPOINT environment variable to TOKEN_REDIRECT_ENDPOINT

Removed

• Functions deprecated before 1.11.0:
  • @cumulus/api/models/base: static Manager.createTable() and static Manager.deleteTable()
  • @cumulus/ingest/aws/S3
  • @cumulus/ingest/aws/StepFunction.getExecution()
  • @cumulus/ingest/aws/StepFunction.pullEvent()
  • @cumulus/ingest/consumer.Consume
  • @cumulus/ingest/granule/Ingest.getBucket()

Deprecated

@cmrjs/ingestConcept, instead use the CMR object methods. @cmrjs/CMR.ingestGranule or @cmrjs/CMR.ingestCollection @cmrjs/searchConcept, instead use the CMR object methods. @cmrjs/CMR.searchGranules or @cmrjs/CMR.searchCollections @cmrjs/deleteConcept, instead use the CMR object methods. @cmrjs/CMR.deleteGranule or @cmrjs/CMR.deleteCollection