@cumulus/checksum

@cumulus/checksum

Checksum

The @cumulus/checksum library provides checksum functionality used by Cumulus packages and tasks. Currently the supported input includes file streams, and supported checksum algorithms include cksum and the algorithms available to the crypto package, as documented here.

Usage

const fs = require('fs');
const { generateChecksumFromStream } = require('@cumulus/checksum');

const stream = fs.createReadStream('myDataFile.hdf');
const myCksum = generateChecksumFromStream('cksum', stream);

API

checksum

• checksum
  • .generateChecksumFromStream(algorithm, stream, [options]) ⇒ Promise.<(number|string)>
  • .validateChecksumFromStream(algorithm, stream, expectedSum, [options]) ⇒ Promise.<boolean>

checksum.generateChecksumFromStream(algorithm, stream, [options]) ⇒ Promise.<(number|string)>

Create file checksum from readable stream

Kind: static method of checksum
Returns: Promise.<(number|string)> - the file checksum

Param	Type	Description
algorithm	string	Checksum algorithm type
stream	stream.Readable	A readable file stream
[options]	Object	Checksum options, see crypto.createHash()

checksum.validateChecksumFromStream(algorithm, stream, expectedSum, [options]) ⇒ Promise.<boolean>

Validate expected checksum against calculated checksum

Kind: static method of checksum
Returns: Promise.<boolean> - whether expectedSum === calculatedSum

Param	Type	Description
algorithm	string	Checksum algorithm
stream	stream.Readable	A readable file stream
expectedSum	number | string	expected checksum
[options]	Object	Checksum options

About Cumulus

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

---

Generated automatically using npm run build-docs

Changelog

[v18.4.0] 2024-08-16

Migration Notes

CUMULUS-3320 Update executions table

The work for CUMULUS-3320 required index updates as well as a modification of a table constraint. To install the update containing these changes you should:

• Pre-generate the indexes on the execution table. This can be done via manual procedure prior to upgrading without downtime, or done more quickly before or during upgrade with downtime.
• Update the executions_parent_cumulus_id_foreign constraint. This will require downtime as updating the constraint requires a table write lock, and the update may take some time.

Deployments with low volume databases and low activity and/or test/development environments should be able to install these updates via the normal automatic Cumulus deployment process.

Please carefully review the migration process documentation. Failure to make these updates properly will likely result in deployment failure and/or degraded execution table operations.

CUMULUS-3449 Please follow the instructions before upgrading Cumulus

• The updates in CUMULUS-3449 requires manual update to postgres database in production environment. Please follow Update Cumulus_id Type and Indexes

Breaking Changes

Added

• CUMULUS-3320
  • Added endpoint /executions/bulkDeleteExecutionsByCollection to allow bulk deletion of executions from elasticsearch by collectionId
  • Added Bulk Execution Delete migration type to async operations types
• CUMULUS-3608
  • Exposes variables for sqs_message_consumer_watcher messageLimit and timeLimit configurations. Descriptions of the variables here include notes on usage and what users should consider if configuring something other than the default values.
• CUMULUS-3449
  • Updated the following database columns to BIGINT: executions.cumulus_id, executions.parent_cumulus_id, files.granule_cumulus_id, granules_executions.granule_cumulus_id, granules_executions.execution_cumulus_id and pdrs.execution_cumulus_id
  • Changed granules table unique constraint to granules_collection_cumulus_id_granule_id_unique
  • Added indexes granules_granule_id_index and granules_provider_collection_cumulus_id_granule_id_index to granules table

Changed

• CUMULUS-3320
  • Updated executions table (please see Migration section and Upgrade Instructions for more information) to:
    • Add index on collection_cumulus_id
    • Add index on parent_cumulus_id
    • Update executions_parent_cumulus_id_foreign constraint to add ON DELETE SET NULL. This change will cause deletions in the execution table to allow deletion of parent executions, when this occurs the child will have it's parent reference set to NULL as part of the deletion operations.
• CUMULUS-3449
  • Updated @cumulus/db package and configure knex hook postProcessResponse to convert the return string from columns ending with "cumulus_id" to number.
• CUMULUS-3841
  • Increased fetchRules page size to default to 100 instead of 10. This improves overall query time when fetching all rules such as in sqsMessageConsumer.

Fixed

• CUMULUS-3817
  • updated applicable @aws-sdk dependencies to 3.621.0 to remove inherited vulnerability from fast-xml-parser
• CUMULUS-3320
  • Execution database deletions by cumulus_id should have greatly improved performance as a table scan will no longer be required for each record deletion to validate parent-child relationships
• CUMULUS-3818
  • Fixes default value (updated to tag 52) for async-operation-image in tf-modules/cumulus.
• CUMULUS-3840
  • Fixed @cumulus/api/bin/serve to correctly use EsClient.