@cumulus/common
Advanced tools
Common libraries used in Cumulus.
npm install @cumulus/common
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
To make a contribution, please see our contributing guidelines.
[v9.0.0] 2021-05-03
@cumulus/api/models.asyncOperation class in
favor of the added @cumulus/async-operations module:
startstartAsyncOperationsasync-operations endpoint will now omit output instead of
returning none when the operation did not return output.@cumulus/api/models/granule.unpublishAndDeleteGranule in favor
of @cumulus/api/lib/granule-remove-from-cmr.unpublishGranule and
@cumulus/api/lib/granule-delete.deleteGranuleAndFiles.sf-event-sqs-to-db-records to write a granule's files to
PostgreSQL only after the workflow has exited the Running status.
Please note that any workflow that uses sf_sqs_report_task for
mid-workflow updates will be impacted.file schema and TypeScript type definition to require
bucket and key fields.move granule endpoint now moves granule files on a per-file basismove granule endpoint on granule file move failure will retain the
file at it's original location, but continue to move any other granule
files.move method from the @cumulus/api/models.granule class.
logic is now handled in @cumulus/api/endpoints/granules and is
accessible via the Core API.data-migration2 Lambda to be run after data-migration1data-migration2 Lambda for migrating execution records
from DynamoDB to PostgreSQL@cumulus/async-operations to core packages, exposing
startAsyncOperation which will handle starting an async operation and
adding an entry to both PostgreSQL and DynamoDbcollections tabledata-migration1 Lambda for migrating collection records
from Dynamo to PostgreSQLproviders tabledata-migration1 Lambda for migrating provider records
from Dynamo to PostgreSQLdata-migration1 Lambda for migrating async operation
records from Dynamo to PostgreSQLdata-migration1 Lambda for migrating rule records from
DynamoDB to PostgreSQLrules tableasyncOperations tableexecutions tabletranslateApiAsyncOperationToPostgresAsyncOperation function to @cumulus/dbdata-migration2 Lambda for migrating PDR records from
DynamoDB to PostgreSQLdata-migration2 Lambda for migrating Granule records from
DynamoDB to PostgreSQL and parse Granule records to store File records in
RDS.granules_executions table to PostgreSQL schema to allow for a
many-to-many relationship between granules and executions
granules_executions table referring to that record will also be
deleted.upsertGranuleWithExecutionJoinRecord helper to @cumulus/db to
allow for upserting a granule record and its corresponding
granules_execution record@cumulus/db/translate/file/translateApiFiletoPostgresFile@cumulus/db/translate/file/translateApiGranuletoPostgresGranule@cumulus/message/Providers/getMessageProvider@cumulus/message/Executions/getMessageExecutionOriginalPayload@cumulus/message/Executions/getMessageExecutionFinalPayload@cumulus/message/workflows/getMessageWorkflowTasks@cumulus/message/workflows/getMessageWorkflowStartTime@cumulus/message/workflows/getMessageWorkflowStopTime@cumulus/message/workflows/getMessageWorkflowName@cumulus/message/PDRs/getMessagePdrRunningExecutions@cumulus/message/PDRs/getMessagePdrCompletedExecutions@cumulus/message/PDRs/getMessagePdrFailedExecutions@cumulus/message/PDRs/getMessagePdrStats@cumulus/message/PDRs/getPdrPercentCompletion@cumulus/message/workflows/getWorkflowDurationtranslateApiRuleToPostgresRule to @cumulus/db to translate API
Rule to conform to Postgres Rule definition.sfEventSqsToDbRecords Lambda for granule and
file writes to the core PostgreSQL databasecreate in Rules Model to take in optional createdAt parameter
which sets the value of createdAt if not specified during function call.sfEventSqsToDbRecords Lambda to write execution, PDR,
and granule records to the core PostgreSQL database in parallel with
writes to DynamoDBsfEventSqsToDbRecords Lambda for PDR writes
to the core PostgreSQL databasesfEventSqsToDbRecords Lambda for execution
writes to the core PostgreSQL databaseasync-operations endpoint will now omit output instead of
returning none when the operation did not return output.files to remove filename and
name and only support file_name.files to remove size to only
support file_size.PostgresFile to remove duplicate fields filename and name and
rename size to file_size.sf-event-sqs-to-db-records behavior to discard and not throw an
error on an out-of-order/delayed message so as not to have it be sent to
the DLQ.DELETE /pdrs/{pdrname} API behavior to also delete record from
PostgreSQL database.DELETE /granules/{granuleName} API behavior to also delete
record from PostgreSQL database.Bulk operation BULK_GRANULE_DELETE API behavior to also delete
records from PostgreSQL database.granule_cumulus_id foreign key to granule in PostgreSQL files
table to use a CASCADE delete, so records in the files table are
automatically deleted by the database when the corresponding granule is
deleted.write-db-dlq-records-to-s3 lambda.process-dead-letter-archive lambda to pick up and process dead letters in the S3 system bucket dead letter archive./deadLetterArchive/recoverCumulusMessages endpoint to trigger an async operation to leverage this capability on demand.translateApiAsyncOperationToPostgresAsyncOperation to call
JSON.stringify and then JSON.parse on output.postgres-migration-async-operation lambda to start an ECS task to
run a the data-migration2 lambda.async_operations table to include Data Migration 2 as a new
operation_type.cumulus-tf/variables.tf to include optional_dynamo_tables that
will be merged with dynamo_tables.packages/db/src/types/summary.ts with
MigrationSummary and DataMigration1 and DataMigration2 types.data-migration1 and data-migration2 lambdas to return
MigrationSummary objects.RecordAlreadyMigrated logs in data-migration1 and
data-migration2granuleSearchParams.granuleId or granuleSearchParams.collectionId
properties in the payload for the
<prefix>-postgres-migration-async-operation LambdamigrationsList property in the payload for the
<prefix>-postgres-migration-async-operation LambdastoreErrors function which stores errors in system bucket.executions and granulesAndFiles data migrations to call storeErrors to store migration errors.system_bucket variable to data-migration2.executions and granules migrations to improve performance. The behavior of the parallel scanning and writes can be controlled via the following properties on the event input to the <prefix>-postgres-migration-async-operation Lambda:
granuleMigrationParams.parallelScanSegments: How many segments to divide your granules DynamoDB table into for parallel scanninggranuleMigrationParams.parallelScanLimit: The maximum number of granule records to evaluate for each parallel scanning segment of the DynamoDB tablegranuleMigrationParams.writeConcurrency: The maximum number of concurrent granule/file writes to perform to the PostgreSQL database across all DynamoDB segmentsexecutionMigrationParams.parallelScanSegments: How many segments to divide your executions DynamoDB table into for parallel scanningexecutionMigrationParams.parallelScanLimit: The maximum number of execution records to evaluate for each parallel scanning segment of the DynamoDB tableexecutionMigrationParams.writeConcurrency: The maximum number of concurrent execution writes to perform to the PostgreSQL database across all DynamoDB segments@cumulus/aws-client/DynamoDb.parallelScan helper to perform parallel scanning on DynamoDb tablesfailed in both Postgres and DynamoDB if any/all of its files fail to write to the database.@cumulus/ingest/moveGranuleFilesFAQs
Common utilities used across tasks
The npm package @cumulus/common receives a total of 438 weekly downloads. As such, @cumulus/common popularity was classified as not popular.
We found that @cumulus/common demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
A supply chain attack on Rspack's npm packages injected cryptomining malware, potentially impacting thousands of developers.
Research
Security News
Socket researchers discovered a malware campaign on npm delivering the Skuld infostealer via typosquatted packages, exposing sensitive data.
Security News
Sonar’s acquisition of Tidelift highlights a growing industry shift toward sustainable open source funding, addressing maintainer burnout and critical software dependencies.