@cumulus/cmr-client

@cumulus/cmr-client

A Node.js client to read from, write to, and delete from NASA's Common Metadata Repository (CMR) API.

API

Classes

CMR

A class to simplify requests to the CMR

CMRSearchConceptQueue

A class to efficiently list all of the concepts (collections/granules) from CMR search, without loading them all into memory at once. Handles paging.

CMR

A class to simplify requests to the CMR

Kind: global class

• CMR
  • new CMR(params)
  • .getToken() ⇒ Promise.<string>
  • .getHeaders(params) ⇒ Object
  • .ingestCollection(xml) ⇒ Promise.<Object>
  • .ingestGranule(xml) ⇒ Promise.<Object>
  • .ingestUMMGranule(ummgMetadata) ⇒ Promise.<Object>
  • .deleteCollection(datasetID) ⇒ Promise.<Object>
  • .deleteGranule(granuleUR) ⇒ Promise.<Object>
  • .searchCollections(params, [format]) ⇒ Promise.<Object>
  • .searchGranules(params, [format]) ⇒ Promise.<Object>

new CMR(params)

The constructor for the CMR class

Param	Type	Description
params	Object
params.provider	string	the CMR provider id
params.clientId	string	the CMR clientId
params.username	string	CMR username
params.password	string	CMR password

Example

const { CMR } = require('@cumulus/cmr-client');

const cmrClient = new CMR({
 provider: 'my-provider',
 clientId: 'my-clientId',
 username: 'my-username',
 password: 'my-password'
});

cmrClient.getToken() ⇒ Promise.<string>

The method for getting the token

Kind: instance method of CMR
Returns: Promise.<string> - the token

cmrClient.getHeaders(params) ⇒ Object

Return object containing CMR request headers

Kind: instance method of CMR
Returns: Object - CMR headers object

Param	Type	Description
params	Object
[params.token]	string	CMR request token
[params.ummgVersion]	string	UMMG metadata version string or null if echo10 metadata

cmrClient.ingestCollection(xml) ⇒ Promise.<Object>

Adds a collection record to the CMR

Kind: instance method of CMR
Returns: Promise.<Object> - the CMR response

Param	Type	Description
xml	string	the collection XML document

cmrClient.ingestGranule(xml) ⇒ Promise.<Object>

Adds a granule record to the CMR

Kind: instance method of CMR
Returns: Promise.<Object> - the CMR response

Param	Type	Description
xml	string	the granule XML document

cmrClient.ingestUMMGranule(ummgMetadata) ⇒ Promise.<Object>

Adds/Updates UMMG json metadata in the CMR

Kind: instance method of CMR
Returns: Promise.<Object> - to the CMR response object.

Param	Type	Description
ummgMetadata	Object	UMMG metadata object

cmrClient.deleteCollection(datasetID) ⇒ Promise.<Object>

Deletes a collection record from the CMR

Kind: instance method of CMR
Returns: Promise.<Object> - the CMR response

Param	Type	Description
datasetID	string	the collection unique id

cmrClient.deleteGranule(granuleUR) ⇒ Promise.<Object>

Deletes a granule record from the CMR

Kind: instance method of CMR
Returns: Promise.<Object> - the CMR response

Param	Type	Description
granuleUR	string	the granule unique id

cmrClient.searchCollections(params, [format]) ⇒ Promise.<Object>

Search in collections

Kind: instance method of CMR
Returns: Promise.<Object> - the CMR response

Param	Type	Default	Description
params	string		the search parameters
[format]	string	"json"	format of the response

cmrClient.searchGranules(params, [format]) ⇒ Promise.<Object>

Search in granules

Kind: instance method of CMR
Returns: Promise.<Object> - the CMR response

Param	Type	Default	Description
params	string		the search parameters
[format]	string	"'json'"	format of the response

CMRSearchConceptQueue

A class to efficiently list all of the concepts (collections/granules) from CMR search, without loading them all into memory at once. Handles paging.

Kind: global class

• CMRSearchConceptQueue
  • new CMRSearchConceptQueue(params)
  • .peek() ⇒ Promise.<Object>
  • .shift() ⇒ Promise.<Object>

new CMRSearchConceptQueue(params)

The constructor for the CMRSearchConceptQueue class

Param	Type	Default	Description
params	Object
params.provider	string		the CMR provider id
params.clientId	string		the CMR clientId
params.type	string		the type of search 'granule' or 'collection'
[params.searchParams]	string	"{}"	the search parameters
params.format	string		the result format

Example

const { CMRSearchConceptQueue } = require('@cumulus/cmr-client');

const cmrSearchConceptQueue = new CMRSearchConceptQueue({
  provider: 'my-provider',
  clientId: 'my-clientId',
  type: 'granule',
  searchParams: {},
  format: 'json'
});

cmrSearchConceptQueue.peek() ⇒ Promise.<Object>

View the next item in the queue

This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.

Kind: instance method of CMRSearchConceptQueue
Returns: Promise.<Object> - an item from the CMR search

cmrSearchConceptQueue.shift() ⇒ Promise.<Object>

Remove the next item from the queue

When there are no more items in the queue, returns null.

Kind: instance method of CMRSearchConceptQueue
Returns: Promise.<Object> - an item from the CMR search

CMR Docs

CMR REST API endpoint documentation is here:

• https://cmr.earthdata.nasa.gov/search/site/search_api_docs.html
• https://cmr.earthdata.nasa.gov/ingest/site/ingest_api_docs.html

What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Test

Test with npm run test.

Contributing

See Cumulus README

---

Generated automatically using npm run build-docs

Changelog

[v1.12.0] - 2019-4-4

Note: There was an issue publishing 1.12.0. Upgrade to 1.12.1.

BREAKING CHANGES

• CUMULUS-1139

  • granule.applyWorkflow uses the new-style granule record as input to workflows.

• CUMULUS-1171

  • Fixed provider handling in the API to make it consistent between protocols. NOTE: This is a breaking change. When applying this upgrade, users will need to:
    1. Disable all workflow rules
    2. Update any http or https providers so that the host field only contains a valid hostname or IP address, and the port field contains the provider port.
    3. Perform the deployment
    4. Re-enable workflow rules

• CUMULUS-1176:

  • @cumulus/move-granules input expectations have changed. @cumulus/files-to-granules is a new intermediate task to perform input translation in the old style. See the Added and Changed sections of this release changelog for more information.

• CUMULUS-670

  • The behavior of ParsePDR and related code has changed in this release. PDRs with FILE_TYPEs that do not conform to the PDR ICD (+ TGZ) (https://cdn.earthdata.nasa.gov/conduit/upload/6376/ESDS-RFC-030v1.0.pdf) will fail to parse.

• CUMULUS-1208

  • The granule object input to @cumulus/queue-granules will now be added to ingest workflow messages as is. In practice, this means that if you are using @cumulus/queue-granules to trigger ingest workflows and your granule objects input have invalid properties, then your ingest workflows will fail due to schema validation errors.

Added

• CUMULUS-777

  • Added new cookbook entry on configuring Cumulus to track ancillary files.

• CUMULUS-1183

  • Kes overrides will now abort with a warning if a workflow step is configured without a corresponding lambda configuration

• CUMULUS-1223

  • Adds convenience function @cumulus/common/bucketsConfigJsonObject for fetching stack's bucket configuration as an object.

• CUMULUS-853

  • Updated FakeProcessing example lambda to include option to generate fake browse
  • Added feature documentation for ancillary metadata export, a new cookbook entry describing a workflow with ancillary metadata generation(browse), and related task definition documentation

• CUMULUS-805

  • Added a CloudWatch alarm to check running ElasticSearch instances, and a CloudWatch dashboard to view the health of ElasticSearch
  • Specify AWS_REGION in .env to be used by deployment script

• CUMULUS-803

  • Added CloudWatch alarms to check running tasks of each ECS service, and add the alarms to CloudWatch dashboard

• CUMULUS-670

  • Added Ancillary Metadata Export feature (see https://nasa.github.io/cumulus/docs/features/ancillary_metadata for more information)
  • Added new Collection file parameter "fileType" that allows configuration of workflow granule file fileType

• CUMULUS-1184 - Added kes logging output to ensure we always see the state machine reference before failures due to configuration

• CUMULUS-1105 - Added a dashboard endpoint to serve the dashboard from an S3 bucket

• CUMULUS-1199 - Moves s3credentials endpoint from the backend to the distribution API.

• CUMULUS-666

  • Added @api/endpoints/s3credentials to allow EarthData Login authorized users to retrieve temporary security credentials for same-region direct S3 access.

• CUMULUS-671

  • Added @packages/integration-tests/api/distribution/getDistributionApiS3SignedUrl() to return the S3 signed URL for a file protected by the distribution API

• CUMULUS-672

  • Added cmrMetadataFormat and cmrConceptId to output for individual granules from @cumulus/post-to-cmr. cmrMetadataFormat will be read from the cmrMetadataFormat generated for each granule in @cumulus/cmrjs/publish2CMR()
  • Added helpers to @packages/integration-tests/api/distribution:
    • getDistributionApiFileStream() returns a stream to download files protected by the distribution API
    • getDistributionFileUrl() constructs URLs for requesting files from the distribution API

• CUMULUS-1185 @cumulus/api/models/Granule.removeGranuleFromCmrByGranule to replace @cumulus/api/models/Granule.removeGranuleFromCmr and use the Granule UR from the CMR metadata to remove the granule from CMR

• CUMULUS-1101

  • Added new @cumulus/checksum package. This package provides functions to calculate and validate checksums.
  • Added new checksumming functions to @cumulus/common/aws: calculateS3ObjectChecksum and validateS3ObjectChecksum, which depend on the checksum package.

• CUMULUS-1171

  • Added @cumulus/common API documentation to packages/common/docs/API.md
  • Added an npm run build-docs task to @cumulus/common
  • Added @cumulus/common/string#isValidHostname()
  • Added @cumulus/common/string#match()
  • Added @cumulus/common/string#matches()
  • Added @cumulus/common/string#toLower()
  • Added @cumulus/common/string#toUpper()
  • Added @cumulus/common/URLUtils#buildURL()
  • Added @cumulus/common/util#isNil()
  • Added @cumulus/common/util#isNull()
  • Added @cumulus/common/util#isUndefined()
  • Added @cumulus/common/util#negate()

• CUMULUS-1176

  • Added new @cumulus/files-to-granules task to handle converting file array output from cumulus-process tasks into granule objects. Allows simplification of @cumulus/move-granules and @cumulus/post-to-cmr, see Changed section for more details.

• CUMULUS-1151 Compare the granule holdings in CMR with Cumulus' internal data store

• CUMULUS-1152 Compare the granule file holdings in CMR with Cumulus' internal data store

Changed

• CUMULUS-1216 - Updated @cumulus/ingest/granule/ingestFile to download files to expected staging location.

• CUMULUS-1208 - Updated @cumulus/ingest/queue/enqueueGranuleIngestMessage() to not transform granule object passed to it when building an ingest message

• CUMULUS-1198 - @cumulus/ingest no longer enforces any expectations about whether provider_path contains a leading slash or not.

• CUMULUS-1170

  • Update scripts and docs to use npm instead of yarn
  • Use package-lock.json files to ensure matching versions of npm packages
  • Update CI builds to use npm ci instead of npm install

• CUMULUS-670

  • Updated ParsePDR task to read standard PDR types+ (+ tgz as an external customer requirement) and add a fileType to granule-files on Granule discovery
  • Updated ParsePDR to fail if unrecognized type is used
  • Updated all relevant task schemas to include granule->files->filetype as a string value
  • Updated tests/test fixtures to include the fileType in the step function/task inputs and output validations as needed
  • Updated MoveGranules task to handle incoming configuration with new "fileType" values and to add them as appropriate to the lambda output.
  • Updated DiscoverGranules step/related workflows to read new Collection file parameter fileType that will map a discovered file to a workflow fileType
  • Updated CNM parser to add the fileType to the defined granule file fileType on ingest and updated integration tests to verify/validate that behavior
  • Updated generateEcho10XMLString in cmr-utils.js to use a map/related library to ensure order as CMR requires ordering for their online resources.
  • Updated post-to-cmr task to appropriately export CNM filetypes to CMR in echo10/UMM exports

• CUMULUS-1139 - Granules stored in the API contain a files property. That schema has been greatly simplified and now better matches the CNM format.

  • The name property has been renamed to fileName.
  • The filepath property has been renamed to key.
  • The checksumValue property has been renamed to checksum.
  • The path property has been removed.
  • The url_path property has been removed.
  • The filename property (which contained an s3:// URL) has been removed, and the bucket and key properties should be used instead. Any requests sent to the API containing a granule.files[].filename property will be rejected, and any responses coming back from the API will not contain that filename property.
  • A source property has been added, which is a URL indicating the original source of the file.
  • @cumulus/ingest/granule.moveGranuleFiles() no longer includes a filename field in its output. The bucket and key fields should be used instead.

• CUMULUS-672

  • Changed @cumulus/integration-tests/api/EarthdataLogin.getEarthdataLoginRedirectResponse to @cumulus/integration-tests/api/EarthdataLogin.getEarthdataAccessToken. The new function returns an access response from Earthdata login, if successful.
  • @cumulus/integration-tests/cmr/getOnlineResources now accepts an object of options, including cmrMetadataFormat. Based on the cmrMetadataFormat, the function will correctly retrieve the online resources for each metadata format (ECHO10, UMM-G)

• CUMULUS-1101

  • Moved @cumulus/common/file/getFileChecksumFromStream into @cumulus/checksum, and renamed it to generateChecksumFromStream. This is a breaking change for users relying on @cumulus/common/file/getFileChecksumFromStream.
  • Refactored @cumulus/ingest/Granule to depend on new common/aws checksum functions and remove significantly present checksumming code.
    • Deprecated @cumulus/ingest/granule.validateChecksum. Replaced with @cumulus/ingest/granule.verifyFile.
    • Renamed granule.getChecksumFromFile to granule.retrieveSuppliedFileChecksumInformation to be more accurate.
  • Deprecated @cumulus/common/aws.checksumS3Objects. Use @cumulus/common/aws.calculateS3ObjectChecksum instead.

• CUMULUS-1171

  • Fixed provider handling in the API to make it consistent between protocols. Before this change, FTP providers were configured using the host and port properties. HTTP providers ignored port and protocol, and stored an entire URL in the host property. Updated the API to only accept valid hostnames or IP addresses in the provider.host field. Updated ingest code to properly build HTTP and HTTPS URLs from provider.protocol, provider.host, and provider.port.
  • The default provider port was being set to 21, no matter what protocol was being used. Removed that default.

• CUMULUS-1176

  • @cumulus/move-granules breaking change: Input to move-granules is now expected to be in the form of a granules object (i.e. { granules: [ { ... }, { ... } ] }); For backwards compatibility with array-of-files outputs from processing steps, use the new @cumulus/files-to-granules task as an intermediate step. This task will perform the input translation. This change allows move-granules to be simpler and behave more predictably. config.granuleIdExtraction and config.input_granules are no longer needed/used by move-granules.
  • @cumulus/post-to-cmr: config.granuleIdExtraction is no longer needed/used by post-to-cmr.

• CUMULUS-1174

  • Better error message and stacktrace for S3KeyPairProvider error reporting.

Fixed

• CUMULUS-1218 Reconciliation report will now scan only completed granules.
• @cumulus/api files and granules were not getting indexed correctly because files indexing was failing in db-indexer
• @cumulus/deployment A bug in the Cloudformation template was preventing the API from being able to be launched in a VPC, updated the IAM template to give the permissions to be able to run the API in a VPC

Deprecated

• @cumulus/api/models/Granule.removeGranuleFromCmr, instead use @cumulus/api/models/Granule.removeGranuleFromCmrByGranule
• @cumulus/ingest/granule.validateChecksum, instead use @cumulus/ingest/granule.verifyFile
• @cumulus/common/aws.checksumS3Objects, instead use @cumulus/common/aws.calculateS3ObjectChecksum
• @cumulus/cmrjs: getGranuleId and getCmrFiles are deprecated due to changes in input handling.