@cumulus/checksum
Advanced tools
The @cumulus/checksum library provides checksum functionality used by Cumulus
packages and tasks. Currently the supported input includes file streams, and
supported checksum algorithms include cksum and the algorithms available to
the crypto package, as documented
here.
const fs = require('fs');
const { generateChecksumFromStream } = require('@cumulus/checksum');
const stream = fs.createReadStream('myDataFile.hdf');
const myCksum = generateChecksumFromStream('cksum', stream);
Promise.<(number|string)>Promise.<boolean>Promise.<(number|string)>Create file checksum from readable stream
Kind: static method of checksum
Returns: Promise.<(number|string)> - the file checksum
| Param | Type | Description |
|---|---|---|
| algorithm | string | Checksum algorithm type |
| stream | stream.Readable | A readable file stream |
| [options] | Object | Checksum options, see crypto.createHash() |
Promise.<boolean>Validate expected checksum against calculated checksum
Kind: static method of checksum
Returns: Promise.<boolean> - whether expectedSum === calculatedSum
| Param | Type | Description |
|---|---|---|
| algorithm | string | Checksum algorithm |
| stream | stream.Readable | A readable file stream |
| expectedSum | number | string | expected checksum |
| [options] | Object | Checksum options |
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
Generated automatically using npm run build-docs
[v20.0.0] 2025-02-04
ecs_cluster_instance_allow_ssh resource.ecs_cluster_instance_allow_ssh was implemented before SSM hosts were deployed
to NGAP accounts and allowed for SSHing into an instance from an SSH bastion, which no longer exists.sync-granule task to add useGranIdPath as a configuration flag.
This modifies the task behavior to stage granules to
<staging_path>/<collection_id>/<md5_granuleIdHash> to allow for better S3
partitioning/performance for large collections.
Because of this benefit
the default has been set to true, however as sync-granules relies on
object name collision, this configuration changes the duplicate collision
behavior of sync-granules to be per-granule-id instead of per-collection
when active.
If the prior behavior is desired, please add "useGranIdPath": false to your
task config in your workflow definitions that use sync-granule.To invoke the Lambda and start the ReconciliationReport migration, you can use the AWS Console or CLI:
aws lambda invoke --function-name $PREFIX-ReconciliationReportMigration $OUTFILE
PREFIX is your Cumulus deployment prefix.OUTFILE (optional) is the filepath where the Lambda output will be saved.External tooling making use of searchContext in the GET /granules/ endpoint will need to update to make use of standard pagination via limit and page scrolling, as searchContext is no longer supported/is an ES specific feature.
searchContext from API granules GET /granules endpoint.pdrs LIST endpoint to query postgresDEL and GET to not update Elasticsearchproviders list api endpoint and added ProviderSearch class to query postgresproviders endpointsasyncOperations api endpoint to query postgresPOST and DEL to not update
Elasticsearch@cumlus/api/ecs/async-operation to not update Elasticsearch index when
reporting status of async operation@cumulus/db/search to allow for ordered collation as a
dbQueryParameter@cumulus/db/search to allow dbQueryParameters.limit to be set to
null to allow for optional unlimited page sizes in search results@cumulus/api reconciliation report code@cumulus/cmr-client@cumulus/db@cumulus/client/CMR.getToken to throw if a non-launchpad token is requested without a usernameInventory and Granule Not Found reports to query postgreSQL
database instead of elasticsearch@cumulus/db/lib/granule.getGranulesByApiPropertiesQuery to
allow order by collation to be optionally specified@cumulus/db/lib/granule.getGranulesByApiPropertiesQuery to
be parameterized and include a modifier on temporalBoundByCreatedAtgetGranulesByApiPropertiesQueryreconciliation_reports table in RDS, including indexesreconciliationReports in @cumulus/dbreconciliation_reports in @cumulus/types/api@cumulus/api/endpoints/reconciliation-reports getReport and deleteReport to work with the new RDS table instead of Dynamoreconciliation_reports list api endpoint and added ReconciliationReportSearch class to query postgresreconciliationReports type to stats endpoint, so aggregate query will work for reconciliation reports@cumulus/api/bin/serveUtils to no longer add records to ElasticSearch@cumulus/api/lambdas/cleanExecutions leaving a dummy handler, as the code worked with ES. lambda will be rewritten with CUMULUS-3982@cumulus/api/endpoints/elasticsearch, @cumulus/api/lambdas/bootstrap, and @cumulus/api/lambdas/index-from-databasetf-modules/granules endpoint PATCH/bulkPatchGranuleCollection which updates a batch of granule records collectionId to a new collectionId. This endpoint takes a list of granules and a collectionId, updating the granules' to the collectionId passed with the payload in postgres./granules endpoint PATCH/bulkPatch which applies PATCH to a list of granules. For its payload, this endpoint takes a list of granules (the updates to be made to the granule, similar to the pre-existing PATCH), a dbConcurrency and dbMaxPool variables for configuring concurrency and database thoroughput for postgres to tailor to performance and database needs.disableSSL and rejectUnauthorized to tf-modules/cumulus-rds-tf module.iops and throughput options to elasticsearch_config variable
in tf-modules/data-persistence; These two options are necessary for gp3 EBS volume type.cumulus and archive module configuration to allow configuration of the dead_letter_recovery_operation task definition to better allow configuration of the tool's operating environment.batchSize, concurrency and dbMaxPool options to /endpoints/recoverCumulusMessage (note these values are correct at time of this release only):
batchSize - specifies how many DLA objects to read from S3 and hold in memory. Defaults to 1000.concurrency - specifies how many messages to process at the same time. Defaults to 30.dbMaxPool - specifies how many database connections to allow the process to utilize. Defaults to 30. Process should at minimum the value set for concurrency.@cumulus/async-operations.startAsyncOperation to take containerName` as a parameter name, allowing it to specify a container other than the default 'AsyncOperations' containertf-modules/cumulus/ecs_cluster ECS Autoscaling group from launch configurations to launch templatesVACUUM statements from db migrations. In cases where the PG database is very large, these queries
can take a long time and exceed the Lambda timeout, causing failures on deployment.force_new_deployment to cumulus_ecs_service to allow users to force
new task deployment on terraform redeploy. See docs for more details:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ecs_service#force_new_deployment"SendPan task to generate short pan with FAILED disposition.tf-modules/cumulus/ecs_cluster_instance_autoscaling_cf_template.yml.tmpl
user-data for compatibility with Amazon Linux 2023 AMItf-modules/cumulus scripts to use Instance Metadata Service V2fake-provider-cf.yml to work for Amazon Linux 2023 AMIPostToCmr task to be able to republish granulestf-modules/cumulus/ecs_cluster and fake-provider-cf.yml launch templates to require IMDSv2s3-replicator lambda cross region write failuretarget_region variable to tf-modules/s3-replicator module@octokit/graphql from 2.1.1 to ^2.3.0 to address [CVE-2024-21538]
(https://github.com/advisories/GHSA-3xgq-45jj-v275)FAQs
Cumulus checksum utilities
The npm package @cumulus/checksum receives a total of 262 weekly downloads. As such, @cumulus/checksum popularity was classified as not popular.
We found that @cumulus/checksum demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
OpenSSF has published OSPS Baseline, an initiative designed to establish a minimum set of security-related best practices for open source software projects.
Security News
Michigan TypeScript founder Dimitri Mitropoulos implements WebAssembly runtime in TypeScript types, enabling Doom to run after processing 177 terabytes of type definitions.
Security News
vlt's new "reproduce" tool verifies npm packages against their source code, outperforming traditional provenance adoption in the JavaScript ecosystem.