@cumulus/common
Advanced tools
Common libraries used in Cumulus.
npm install @cumulus/common
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
To make a contribution, please see our contributing guidelines.
[v11.0.0] 2022-03-24 [STABLE]
Release v11.0 is a maintenance release series, replacing v9.9. If you are upgrading to or past v11 from v9.9.x to this release, please pay attention to the following migration notes from prior releases:
data-persistence module, but before deploying the main cumulus module/rules/<name> endpoint, the rule records in PostgreSQL may be
out of sync with records in DynamoDB. In order to bring the records into sync, re-deploy and re-run the
data-migration1 Lambda with a payload of
{"forceRulesMigration": true}:aws lambda invoke --function-name $PREFIX-data-migration1 \
--payload $(echo '{"forceRulesMigration": true}' | base64) $OUTFILE
cumulus deploymenttask-config for all workflows that use the sync-granule task to include workflowStartTime set to
{$.cumulus_meta.workflow_start_time}. See here for an example.cumulus deploymentAs part of the work on the RDS Phase 2 feature, it was decided to re-add the
granule file type property on the file table (detailed reasoning
https://wiki.earthdata.nasa.gov/pages/viewpage.action?pageId=219186829). This
change was implemented as part of CUMULUS-2672/CUMULUS-2673, however granule
records ingested prior to v11 will not have the file.type property stored in the
PostGreSQL database, and on installation of v11 API calls to get granule.files
will not return this value. We anticipate most users are impacted by this issue.
Users that are impacted by these changes should re-run the granule migration lambda to only migrate granule file records:
PAYLOAD=$(echo '{"migrationsList": ["granules"], "granuleMigrationParams": {"migrateOnlyFiles": "true"}}' | base64)
aws lambda invoke --function-name $PREFIX-postgres-migration-async-operation \
--payload $PAYLOAD $OUTFILE
You should note that this will only move files for granule records in PostgreSQL. If you have not completed the phase 1 data migration or have granule records in dynamo that are not in PostgreSQL, the migration will report failure for both the DynamoDB granule and all the associated files and the file records will not be updated.
If you prefer to do a full granule and file migration, you may instead
opt to run the migration with the migrateAndOverwrite option instead, this will re-run a
full granule/files migration and overwrite all values in the PostgreSQL database from
what is in DynamoDB for both granules and associated files:
PAYLOAD=$(echo '{"migrationsList": ["granules"], "granuleMigrationParams": {"migrateAndOverwrite": "true"}}' | base64)
aws lambda invoke --function-name $PREFIX-postgres-migration-async-operation \
--payload $PAYLOAD $OUTFILE
Please note: Since this data migration is copying all of your granule data
from DynamoDB to PostgreSQL, it can take multiple hours (or even days) to run,
depending on how much data you have and how much parallelism you configure the
migration to use. In general, the more parallelism you configure the migration
to use, the faster it will go, but the higher load it will put on your
PostgreSQL database. Excessive database load can cause database outages and
result in data loss/recovery scenarios. Thus, the parallelism settings for the
migration are intentionally set by default to conservative values but are
configurable. If this impacts only some of your data products you may want
to consider using other granuleMigrationParams.
Please see the second data migration docs for more on this tool if you are unfamiliar with the various options.
ORCA Backup is now a supported reportType for the POST /reconciliationReports endpoint@cumulus/message/utils.parseException to parse exception objects@cumulus/message/Granules:
getGranuleProductVolumegetGranuleTimeToPreprocessgetGranuleTimeToArchivegenerateGranuleApiRecord@cumulus/message/PDRs/generatePdrApiRecordFromMessage to generate PDR from Cumulus workflow message@cumulus/es-client/indexer:
deleteAsyncOperation to delete async operation records from ElasticsearchupdateAsyncOperation to update an async operation record in ElasticsearchPUT endpoint to Cumulus API for updating a granule.
Requests to this endpoint should be submitted without an action
attribute in the request body.@cumulus/api-client/granules.updateGranule to update granule via the API@cumulus/db/translate/providers@cumulus/db/translate/collectionssearchWithUpdatedAtRange method to
@cumulus/db/models/collectionsdeletePdr to @cumulus/api-client/pdrsmove action to update granules in the index
and utilize postgres as the authoritative datastore@cumulus/db/base.deleteExcluding method to allow for deletion of a
record set with an exclusion list of cumulus_ids@cumulus/db/getFilesAndGranuleInfoQuery() to build a query for searching file
records in PostgreSQL and return specified granule information for each file@cumulus/db/QuerySearchClient library to handle sequentially fetching and paging
through results for an arbitrary PostgreSQL queryinsert method to all @cumulus/db models to handle inserting multiple records into
the database at once@cumulus/db/translatePostgresGranuleResultToApiGranule helper to
translate custom PostgreSQL granule result to API granuletype text column to Postgres database files table@cumulus/es-client/indexer.upsertExecution to upsert an execution@cumulus/es-client/indexer.upsertPdr to upsert a PDR@cumulus/es-client/indexer.upsertGranule to upsert a granuleexecution_sns_topic_arn environment variable to
sf_event_sqs_to_db_records lambda TF definition.sf_event_sqs_to_db_records_lambda IAM policy to include
permissions for SNS publish for report_executions_topiccollection_sns_topic_arn environment variable to
PrivateApiLambda and ApiEndpoints lambdas.updateCollection to @cumulus/api-client.ecs_cluster IAM policy to include permissions for SNS publish
for report_executions_sns_topic_arn, report_pdrs_sns_topic_arn,
report_granules_sns_topic_arnprocess_dead_letter_archive.tfbulk_operation.tfpdr_sns_topic_arn environment variable to
sf_event_sqs_to_db_records lambda TF definition.publishSnsMessageByDataType in @cumulus/api to
publish SNS messages to the report topics to PDRs, Collections, and
Executions.publishSnsMessageUtils to handle
publishing SNS messages for specific data and event types:
publishCollectionUpdateSnsMessagepublishCollectionCreateSnsMessagepublishCollectionDeleteSnsMessagepublishGranuleUpdateSnsMessagepublishGranuleDeleteSnsMessagepublishGranuleCreateSnsMessagepublishExecutionSnsMessagepublishPdrSnsMessagepublishGranuleSnsMessageByEventTypeecs_cluster IAM policy to include permissions for SNS publish
for report_executions_topic and report_pdrs_topic.paginateByCumulusId to @cumulus/db BasePgModel to allow for paginated
full-table select queries in support of elasticsearch indexing.getMaxCumulusId to @cumulus/db BasePgModel to allow all
derived table classes to support querying the current max cumulus_id.ES_HOST environment variable to postgres-migration-async-operation
Lambda using value of elasticsearch_hostname Terraform variable.elasticsearch_security_group_id to security groups for
postgres-migration-async-operation lambda.DynamoDb:DeleteItem to
postgres-migration-async-operation lambda.async_operation_image in
tf-modules/cumulus/variables.tf to cumuluss/async-operation:41ES_HOST environment variable to async operation ECS task
definition to ensure that async operation tasks write to the correct
Elasticsearch domain@cumulus/api/lambdas/reports/orca-backup-reconciliation-report to create
ORCA Backup reconciliation reportdbIndexer Lambda for DynamoDB tables:
<prefix>-AsyncOperationsTable<prefix>-CollectionsTable<prefix>-ExecutionsTable<prefix>-GranulesTable<prefix>-PdrsTable<prefix>-ProvidersTable<prefix>-RulesTable@ingest/granule.moveGranuleFileswaitForModelStatus from example/spec/helpers/apiUtils integration test helpersstream_enabled and stream_view_type from executions_table TF
definition.aws_lambda_event_source_mapping TF definition on executions
DynamoDB table.stream_enabled and stream_view_type from collections_table
TF definition.aws_lambda_event_source_mapping TF definition on collections
DynamoDB table.publish_collections TF resource.aws_lambda_event_source_mapping TF definition on granulesstream_enabled and stream_view_type from pdrs_table TF
definition.aws_lambda_event_source_mapping TF definition on PDRs
DynamoDB table.@cumulus/api/models/granules.storeGranulesFromCumulusMessage() methodaddToLocalES in POST /granules endpoint since it is
redundant.addToLocalES in POST and PUT /executions endpoints
since it is redundant.addToLocalES from es-client package since it is no
longer used._updateGranuleStatus to update granule to "running" from @cumulus/api/lib/ingest.reingestGranule
and @cumulus/api/lib/ingest.applyWorkflow@cumulus/common to address CVE-2022-2477CUMULUS_2641
@cumulus/message package to set productVolume as string
(calculated with file.size as a BigInt) to match API schema@cumulus/db granule translation to translate granule objects to
match the updated API schemaCUMULUS-2714
CUMULUS-2672
data-migration2 lambda to migrate Dynamo granule.files[].type
instead of dropping it.@cumlus/db translateApiFiletoPostgresFile to retain type@cumulus/db translatePostgresFileToApiFile to retain type@cumulus/types.api.file to add type to the typing.CUMULUS-2315
index-from-database lambda/ECS task and elasticsearch endpoint to read
from PostgreSQL databaseindex-from-database endpoint to add the following configuration
tuning parameters:
CUMULUS-2308
/granules/<granule_id> GET endpoint to return PostgreSQL Granules instead of DynamoDB Granules/granules/<granule_id> PUT endpoint to use PostgreSQL Granule as source rather than DynamoDB GranuleunpublishGranule (used in /granules PUT) to use PostgreSQL Granule as source rather than DynamoDB GranulewaitForApiStatus instead of waitForModelStatusCUMULUS-2302
reingest endpoint to read collection from PostgreSQL
database instead of DynamoDB@cumulus/api unit helper from ./lib to
.test/helpersCUMULUS-2208
@cumulus/api/es/* code to new @cumulus/es-client packagesfEventSqsToDbRecords Lambda now writes following data directly to
Elasticsearch in parallel with writes to DynamoDB/PostgreSQL:
packages/api/lib/granules.getGranuleProductVolume ->
@cumulus/message/Granules.getGranuleProductVolumepackages/api/lib/granules.getGranuleTimeToPreprocess
-> @cumulus/message/Granules.getGranuleTimeToPreprocesspackages/api/lib/granules.getGranuleTimeToArchive ->
@cumulus/message/Granules.getGranuleTimeToArchivepackages/api/models/Granule.generateGranuleRecord
-> @cumulus/message/Granules.generateGranuleApiRecordCUMULUS-2306
api/bin/serve.js) setup code to add cleanup/executions
related recordswaitForModelStatusCUMULUS-2303
CUMULUS-2301
getAsyncOperation to read from PostgreSQL database instead of
DynamoDB.translatePostgresAsyncOperationToApiAsyncOperation function in
@cumulus/db/translate/async-operation.translateApiAsyncOperationToPostgresAsyncOperation function to
ensure that output is properly translated to an object for the
PostgreSQL record for the following cases of output on the incoming API
record:
record.output is a JSON stringified objectrecord.output is a JSON stringified arrayrecord.output is a JSON stringified stringrecord.output is a stringCUMULUS-2317
CUMULUS-2304
CUMULUS-2634
sfEventSqsToDbRecords Lambda to use new upsert helpers for executions, granules, and PDRs
to ensure out-of-order writes are handled correctly when writing to ElasticsearchCUMULUS-2510
@cumulus/api/lib/writeRecords/write-execution to publish SNS
messages after a successful write to Postgres, DynamoDB, and ES.create and upsert in the db model for Executions
to return an array of objects containing all columns of the created or
updated records.@cumulus/api/endpoints/collections to publish an SNS message
after a successful collection delete, update (PUT), create (POST).create and upsert in the db model for Collections
to return an array of objects containing all columns for the created or
updated records.create and upsert in the db model for Granules
to return an array of objects containing all columns for the created or
updated records.@cumulus/api/lib/writeRecords/write-granules to publish SNS
messages after a successful write to Postgres, DynamoDB, and ES.@cumulus/api/lib/writeRecords/write-pdr to publish SNS
messages after a successful write to Postgres, DynamoDB, and ES.CUMULUS-2733
_writeGranuleFiles function creates an aggregate error which
contains the workflow error, if any, as well as any error that may occur
from writing granule files.CUMULUS-2674
DELETE endpoints for the following data types to check that record exists in
PostgreSQL or Elasticsearch before proceeding with deletion:
providerasync operationscollectionsgranulesexecutionsPDRsrulesCUMULUS-2294
CUMULUS-2642
collectionIds, granuleIds, and providers to allow
targeting/filtering of the results.CUMULUS-2694
sfEventSqsToDbRccords to log message if Cumulus
workflow message is from pre-RDS deployment but still attempt parallel writing to DynamoDB
and PostgreSQLsfEventSqsToDbRccords to throw error if requirements to write execution to PostgreSQL cannot be metCUMULUS-2660
/executions endpoint to publish SNS message of created record to executions SNS topicCUMULUS-2661
/executions/<arn> endpoint to publish SNS message of updated record to executions SNS topicCUMULUS-2765
updateGranuleStatusToQueued in write-granules to write to
Elasticsearch and publish SNS message to granules topic.CUMULUS-2774
constructGranuleSnsMessage and constructCollectionSnsMessage
to throw error if eventType is invalid or undefined.CUMULUS-2776
getTableIndexDetails in db-indexer to use correct
deleteFnName for reconciliation reports.CUMULUS-2780
CUMULUS-2778
async_operation_image in tf-modules/cumulus/variables.tf to cumuluss/async-operation:38CUMULUS-2854
createRuleTrigger from create.rulesModel.createRuleTrigger directly to create rule trigger.rulesModel.createRuleTrigger if update fails and reversion needs to occur.GET /executions/status responseFAQs
Common utilities used across tasks
The npm package @cumulus/common receives a total of 438 weekly downloads. As such, @cumulus/common popularity was classified as not popular.
We found that @cumulus/common demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Ransomware negotiators share how modern cybercriminals operate like corporations, using specialized teams, negotiation tactics, and reputation management.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.