@cumulus/cmr-client

@cumulus/cmr-client

A Node.js client to read from, write to, and delete from NASA's Common Metadata Repository (CMR) API.

API

Classes

CMR

A class to simplify requests to the CMR

CMRSearchConceptQueue

A class to efficiently list all of the concepts (collections/granules) from CMR search, without loading them all into memory at once. Handles paging.

CMR

A class to simplify requests to the CMR

Kind: global class

• CMR
  • new CMR(params)
  • .getToken() ⇒ Promise.<string>
  • .getHeaders(params) ⇒ Object
  • .ingestCollection(xml) ⇒ Promise.<Object>
  • .ingestGranule(xml) ⇒ Promise.<Object>
  • .ingestUMMGranule(ummgMetadata) ⇒ Promise.<Object>
  • .deleteCollection(datasetID) ⇒ Promise.<Object>
  • .deleteGranule(granuleUR) ⇒ Promise.<Object>
  • .searchCollections(params, [format]) ⇒ Promise.<Object>
  • .searchGranules(params, [format]) ⇒ Promise.<Object>

new CMR(params)

The constructor for the CMR class

Param	Type	Description
params	Object
params.provider	string	the CMR provider id
params.clientId	string	the CMR clientId
params.username	string	CMR username
params.password	string	CMR password

Example

const { CMR } = require('@cumulus/cmr-client');

const cmrClient = new CMR({
 provider: 'my-provider',
 clientId: 'my-clientId',
 username: 'my-username',
 password: 'my-password'
});

cmrClient.getToken() ⇒ Promise.<string>

The method for getting the token

Kind: instance method of CMR Returns: Promise.<string> - the token

cmrClient.getHeaders(params) ⇒ Object

Return object containing CMR request headers

Kind: instance method of CMR Returns: Object - CMR headers object

Param	Type	Description
params	Object
[params.token]	string	CMR request token
[params.ummgVersion]	string	UMMG metadata version string or null if echo10 metadata

cmrClient.ingestCollection(xml) ⇒ Promise.<Object>

Adds a collection record to the CMR

Kind: instance method of CMR Returns: Promise.<Object> - the CMR response

Param	Type	Description
xml	string	the collection XML document

cmrClient.ingestGranule(xml) ⇒ Promise.<Object>

Adds a granule record to the CMR

Kind: instance method of CMR Returns: Promise.<Object> - the CMR response

Param	Type	Description
xml	string	the granule XML document

cmrClient.ingestUMMGranule(ummgMetadata) ⇒ Promise.<Object>

Adds/Updates UMMG json metadata in the CMR

Kind: instance method of CMR Returns: Promise.<Object> - to the CMR response object.

Param	Type	Description
ummgMetadata	Object	UMMG metadata object

cmrClient.deleteCollection(datasetID) ⇒ Promise.<Object>

Deletes a collection record from the CMR

Kind: instance method of CMR Returns: Promise.<Object> - the CMR response

Param	Type	Description
datasetID	string	the collection unique id

cmrClient.deleteGranule(granuleUR) ⇒ Promise.<Object>

Deletes a granule record from the CMR

Kind: instance method of CMR Returns: Promise.<Object> - the CMR response

Param	Type	Description
granuleUR	string	the granule unique id

cmrClient.searchCollections(params, [format]) ⇒ Promise.<Object>

Search in collections

Kind: instance method of CMR Returns: Promise.<Object> - the CMR response

Param	Type	Default	Description
params	string		the search parameters
[format]	string	"json"	format of the response

cmrClient.searchGranules(params, [format]) ⇒ Promise.<Object>

Search in granules

Kind: instance method of CMR Returns: Promise.<Object> - the CMR response

Param	Type	Default	Description
params	string		the search parameters
[format]	string	"'json'"	format of the response

CMRSearchConceptQueue

A class to efficiently list all of the concepts (collections/granules) from CMR search, without loading them all into memory at once. Handles paging.

Kind: global class

• CMRSearchConceptQueue
  • new CMRSearchConceptQueue(params)
  • .peek() ⇒ Promise.<Object>
  • .shift() ⇒ Promise.<Object>

new CMRSearchConceptQueue(params)

The constructor for the CMRSearchConceptQueue class

Param	Type	Default	Description
params	Object
params.provider	string		the CMR provider id
params.clientId	string		the CMR clientId
params.type	string		the type of search 'granule' or 'collection'
[params.searchParams]	string	"{}"	the search parameters
params.format	string		the result format

Example

const { CMRSearchConceptQueue } = require('@cumulus/cmr-client');

const cmrSearchConceptQueue = new CMRSearchConceptQueue({
  provider: 'my-provider',
  clientId: 'my-clientId',
  type: 'granule',
  searchParams: {},
  format: 'json'
});

cmrSearchConceptQueue.peek() ⇒ Promise.<Object>

View the next item in the queue

This does not remove the object from the queue. When there are no more items in the queue, returns 'null'.

Kind: instance method of CMRSearchConceptQueue Returns: Promise.<Object> - an item from the CMR search

cmrSearchConceptQueue.shift() ⇒ Promise.<Object>

Remove the next item from the queue

When there are no more items in the queue, returns null.

Kind: instance method of CMRSearchConceptQueue Returns: Promise.<Object> - an item from the CMR search

CMR Docs

CMR REST API endpoint documentation is here:

• https://cmr.earthdata.nasa.gov/search/site/search_api_docs.html
• https://cmr.earthdata.nasa.gov/ingest/site/ingest_api_docs.html

What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Test

Test with npm run test.

Contributing

See Cumulus README

---

Generated automatically using npm run build-docs

Changelog

[v1.14.2] - 2019-10-08

BREAKING CHANGES

Your Cumulus Message Adapter version should be pinned to v1.0.13 or lower in your app/config.yml using message_adapter_version: v1.0.13 OR you should use the workflow migration steps below to work with CMA v1.1.1+.

• CUMULUS-1394 - The implementation of the SfSnsReport Lambda requires additional environment variables for integration with the new ingest notification SNS topics. Therefore, you must update the definition of SfSnsReport in your lambdas.yml like so:

SfSnsReport:
  handler: index.handler
  timeout: 300
  source: node_modules/@cumulus/sf-sns-report/dist
  tables:
    - ExecutionsTable
  envs:
    execution_sns_topic_arn:
      function: Ref
      value: reportExecutionsSns
    granule_sns_topic_arn:
      function: Ref
      value: reportGranulesSns
    pdr_sns_topic_arn:
      function: Ref
      value: reportPdrsSns

• CUMULUS-1447 - The newest release of the Cumulus Message Adapter (v1.1.1) requires that parameterized configuration be used for remote message functionality. Once released, Kes will automatically bring in CMA v1.1.1 without additional configuration.

  Migration instructions Oversized messages are no longer written to S3 automatically. In order to utilize remote messaging functionality, configure a ReplaceConfig AWS Step Function parameter on your CMA task:

  ParsePdr:
    Parameters:
      cma:
        event.$: "$"
        ReplaceConfig:
          FullMessage: true
  

  Accepted fields in ReplaceConfig include MaxSize, FullMessage, Path and TargetPath. See https://github.com/nasa/cumulus-message-adapter/blob/master/CONTRACT.md#remote-message-configuration for full details.

  As this change is backward compatible in Cumulus Core, users wishing to utilize the previous version of the CMA may opt to transition to using a CMA lambda layer, or set message_adapter_version in their configuration to a version prior to v1.1.0.

PLEASE NOTE

• CUMULUS-1394 - Ingest notifications are now provided via 3 separate SNS topics for executions, granules, and PDRs, instead of a single sftracker SNS topic. Whereas the sftracker SNS topic received a full Cumulus execution message, the new topics all receive generated records for the given object. The new topics are only published to if the given object exists for the current execution. For a given execution/granule/PDR, two messages will be received by each topic: one message indicating that ingest is running and another message indicating that ingest has completed or failed. The new SNS topics are:

  • reportExecutions - Receives 1 message per execution
  • reportGranules - Receives 1 message per granule in an execution
  • reportPdrs - Receives 1 message per PDR

Added

• CUMULUS-639

  • Adds SAML JWT and launchpad token authentication to Cumulus API (configurable)
    • NOTE to authenticate with Launchpad ensure your launchpad user_id is in the <prefix>-UsersTable
    • when Cumulus configured to protect API via Launchpad:
      • New endpoints
        • GET /saml/login - starting point for SAML SSO creates the login request url and redirects to the SAML Identity Provider Service (IDP)
        • POST /saml/auth - SAML Assertion Consumer Service. POST receiver from SAML IDP. Validates response, logs the user in, and returns a SAML-based JWT.
    • Disabled endpoints
      • POST /refresh
      • Changes authorization worklow:
      • ensureAuthorized now presumes the bearer token is a JWT and tries to validate. If the token is malformed, it attempts to validate the token against Launchpad. This allows users to bring their own token as described here https://wiki.earthdata.nasa.gov/display/CUMULUS/Cumulus+API+with+Launchpad+Authentication. But it also allows dashboard users to manually authenticate via Launchpad SAML to receive a Launchpad-based JWT.

• CUMULUS-1394

  • Added Granule.generateGranuleRecord() method to granules model to generate a granule database record from a Cumulus execution message
  • Added Pdr.generatePdrRecord() method to PDRs model to generate a granule database record from a Cumulus execution message
  • Added helpers to @cumulus/common/message:
    • getMessageExecutionName() - Get the execution name from a Cumulus execution message
    • getMessageStateMachineArn() - Get the state machine ARN from a Cumulus execution message
    • getMessageExecutionArn() - Get the execution ARN for a Cumulus execution message
    • getMessageGranules() - Get the granules from a Cumulus execution message, if any.
  • Added @cumulus/common/cloudwatch-event/isFailedSfStatus() to determine if a Step Function status from a Cloudwatch event is a failed status

Changed

• CUMULUS-1308

  • HTTP PUT of a Collection, Provider, or Rule via the Cumulus API now performs full replacement of the existing object with the object supplied in the request payload. Previous behavior was to perform a modification (partial update) by merging the existing object with the (possibly partial) object in the payload, but this did not conform to the HTTP standard, which specifies PATCH as the means for modifications rather than replacements.

• CUMULUS-1375

  • Migrate Cumulus from deprecated Elasticsearch JS client to new, supported one in @cumulus/api

• CUMULUS-1485 Update @cumulus/cmr-client to return error message from CMR for validation failures.

• CUMULUS-1394

  • Renamed Execution.generateDocFromPayload() to Execution.generateRecord() on executions model. The method generates an execution database record from a Cumulus execution message.

• CUMULUS-1432

  • logs endpoint takes the level parameter as a string and not a number
  • Elasticsearch term query generation no longer converts numbers to boolean

• CUMULUS-1447

  • Consolidated all remote message handling code into @common/aws
  • Update remote message code to handle updated CMA remote message flags
  • Update example SIPS workflows to utilize Parameterized CMA configuration

• CUMULUS-1448 Refactor workflows that are mutating cumulus_meta to utilize meta field

• CUMULUS-1451

  • Elasticsearch cluster setting auto_create_index will be set to false. This had been causing issues in the bootstrap lambda on deploy.

• CUMULUS-1456

  • @cumulus/api endpoints default error handler uses boom package to format errors, which is consistent with other API endpoint errors.

Fixed

• CUMULUS-1432 logs endpoint filter correctly filters logs by level
• CUMULUS-1484 useMessageAdapter now does not set CUMULUS_MESSAGE_ADAPTER_DIR when true

Removed

• CUMULUS-1394
  • Removed sfTracker SNS topic. Replaced by three new SNS topics for granule, execution, and PDR ingest notifications.
  • Removed unused functions from @cumulus/common/aws:
    • getGranuleS3Params()
    • setGranuleStatus()