@cumulus/ingest

@cumulus/ingest

@cumulus/ingest is a collection of modules for discovering and ingesting data.

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

Installation

npm install @cumulus/ingest

Testing

Running tests locally requires localstack.

With localstack running, you can run tests using:

LOCALSTACK_HOST=localhost npm test

Modules

All modules are accessible using require: require('@cumulus/ingest/<MODULE_NAME>') or import: import <MODULE_NAME> from '@cumulus/ingest/<MODULE_NAME>'.

• consumer - comsumer for SQS messages
• crypto - provides encryption and decryption methods with a consistent API but differing mechanisms for dealing with encryption keys
• ftp - for accessing FTP servers
• granule - discovers and ingests granules
• http - for accessing data via HTTP and HTTPS
• lock - creates locks for S3 data
• log - stringifies JS object logs for ElasticSearch indexing
• parse-pdr - tools for validating PDRs and generating PDRD and PAN messages
• pdr - discovers and ingests pdrs
• queue - creates queues for ingesting data
• recursion - handles recursion of a FTP/SFTP list operation
• sftp - for accessing SFTP servers

Contributing

To make a contribution, please see our contributing guidelines.

Changelog

[v1.24.0] 2020-06-03

BREAKING CHANGES

• CUMULUS-1969

  • The DiscoverPdrs task now expects provider_path to be provided at event.config.provider_path, not event.config.collection.provider_path
  • event.config.provider_path is now a required parameter of the DiscoverPdrs task
  • event.config.collection is no longer a parameter to the DiscoverPdrs task
  • Collections no longer support the provider_path property. The tasks that relied on that property are now referencing config.meta.provider_path. Workflows should be updated accordingly.

• CUMULUS-1997

  • @cumulus/cmr-client/CMRSearchConceptQueue parameters have been changed to take a cmrSettings object containing clientId, provider, and auth information. This can be generated using @cumulus/cmrjs/cmr-utils/getCmrSettings. The cmrEnvironment variable has been removed.

Added

• CUMULUS-1800
  • Added task configuration setting named syncChecksumFiles to the SyncGranule task. This setting is false by default, but when set to true, all checksum files associated with data files that are downloaded will be downloaded as well.
• CUMULUS-1952
  • Updated HTTP(S) provider client to accept username/password for Basic authorization. This change adds support for Basic Authorization such as Earthdata login redirects to ingest (i.e. as implemented in SyncGranule), but not to discovery (i.e. as implemented in DiscoverGranules). Discovery still expects the provider's file system to be publicly accessible, but not the individual files and their contents.
  • NOTE: Using this in combination with the HTTP protocol may expose usernames and passwords to intermediary network entities. HTTPS is highly recommended.
• CUMULUS-1997
  • Added optional launchpad configuration to @cumulus/hyrax-metadata-updates task config schema.

Fixed

• CUMULUS-1997
  • Updated all CMR operations to use configured authentication scheme
• CUMULUS-2010
  • Updated @cumulus/api/launchpadSaml to support multiple userGroup attributes from the SAML response