@cumulus/ingest
Advanced tools
@cumulus/ingest is a collection of modules for discovering and ingesting data.
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
npm install @cumulus/ingest
Running tests locally requires localstack.
With localstack running, you can run tests using:
LOCALSTACK_HOST=localhost npm test
All modules are accessible using require: require('@cumulus/ingest/<MODULE_NAME>') or import: import <MODULE_NAME> from '@cumulus/ingest/<MODULE_NAME>'.
consumer - consumer for SQS messagescrypto - provides encryption and decryption methods with a consistent API but differing mechanisms for dealing with encryption keysftp - for accessing FTP serversgranule - discovers and ingests granuleshttp - for accessing data via HTTP and HTTPSlock - creates locks for S3 datalog - stringifies JS object logs for ElasticSearch indexingparse-pdr - tools for validating PDRs and generating PDRD and PAN messagespdr - discovers and ingests pdrsqueue - creates queues for ingesting datarecursion - handles recursion of a FTP/SFTP list operationsftp - for accessing SFTP serversTo make a contribution, please see our contributing guidelines.
[v19.1.0] 2024-10-07
This release contains changes listed here as well as changes listed in v19.0.0, despite v19.0.0 being deprecated. Please review Changelog entries and Migration Notes for each Cumulus version between your current version and v19.1.0 as normal.
CUMULUS-3020
var.lambda_timeouts on the main cumulus module/archive modulePlease note - updating this configuration is for adavanced users only. Value changes will modify the visibility
timeout on sfEventSqsToDbRecordsDeadLetterQueue and sfEventSqsToDbRecordsInputQueue and may lead to system
instability.
CUMULUS-3756
CUMULUS-3773
@cumulus/sftp-client and @cumulus/ingest/SftpProviderClient to support both regular and fastDownload.FAQs
Ingest utilities
The npm package @cumulus/ingest receives a total of 339 weekly downloads. As such, @cumulus/ingest popularity was classified as not popular.
We found that @cumulus/ingest demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Critics call the Node.js EOL CVE a misuse of the system, sparking debate over CVE standards and the growing noise in vulnerability databases.
Security News
cURL and Go security teams are publicly rejecting CVSS as flawed for assessing vulnerabilities and are calling for more accurate, context-aware approaches.
Security News
Bun 1.2 enhances its JavaScript runtime with 90% Node.js compatibility, built-in S3 and Postgres support, HTML Imports, and faster, cloud-first performance.