@cumulus/common
Advanced tools
Common libraries used in Cumulus.
npm install @cumulus/common
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
To make a contribution, please see our contributing guidelines.
[v3.0.0] 2020-10-7
CUMULUS-2099
meta.queues in workflow configuration must be replaced with references to queue URLs from Terraform resources. See the updated data cookbooks or example Discover Granules workflow configuration.CUMULUS-2111
Before you re-deploy your cumulus-tf module, note that the [thin-egress-app][thin-egress-app] is no longer deployed by default as part of the cumulus module, so you must add the TEA module to your deployment and manually modify your Terraform state to avoid losing your API gateway and impacting any Cloudfront endpoints pointing to those gateways. If you don't care about losing your API gateway and impacting Cloudfront endpoints, you can ignore the instructions for manually modifying state.
Add the [thin-egress-app][thin-egress-app] module to your cumulus-tf deployment as shown in the Cumulus example deployment.
tea_stack_name variable to the cumulus module and the stack_name variable to the thin_egress_app module must matchstage_name variable to the thin_egress_app module, the value of the tea_api_gateway_stage variable to the cumulus module must match itIf you want to preserve your existing thin-egress-app API gateway and avoid having to update your Cloudfront endpoint for distribution, then you must follow these instructions: https://nasa.github.io/cumulus/docs/upgrade-notes/migrate_tea_standalone. Otherwise, you can re-deploy as usual.
If you provide your own custom bucket map to TEA as a standalone module, you must ensure that your custom bucket map includes mappings for the protected and public buckets specified in your cumulus-tf/terraform.tfvars, otherwise Cumulus may not be able to determine the correct distribution URL for ingested files and you may encounter errors
CUMULUS-2197
ems_deploy is set to false by default, which will delete your EMS resources.ems_deploy variable set to true in your cumulus-tf/terraform.tfvarsGranule Inventory's
/reconciliationReport returns. The user (dashboard) must read the value
of url from the return to get the s3SignedURL and then download the report.meta.queues has been removed from Cumulus core workflow messages.@cumulus/sf-sqs-report workflow task no longer reads the reporting queue URL from input.meta.queues.reporting on the incoming event. Instead, it requires that the queue URL be set as the reporting_queue_url environment variable on the deployed Lambda.thin-egress-app module has be removed from tf-modules/distribution, which is a part of the tf-modules/cumulus module. Thus, the thin-egress-app module is no longer deployed for you by default. See the migration steps for details about how to add deployment for the thin-egress-app.parse-pdr task has been updated to respect the NODE_NAME property in
a PDR's FILE_GROUP. If a NODE_NAME is present, the task will query the
Cumulus API for a provider with that host. If a provider is found, the
output granule from the task will contain a provider property containing
that provider. If NODE_NAME is set but a provider with that host cannot be
found in the API, or if multiple providers are found with that same host,
the task will fail.queue-granules task has been updated to expect an optional
granule.provider property on each granule. If present, the granule will be
enqueued using that provider. If not present, the task's config.provider
will be used instead.@cumulus/api-client.providers.getProviders function now takes a
queryStringParameters parameter which can be used to filter the providers
which are returned@cumulus/aws-client/S3.getS3ObjectReadStreamAsync function has been
removed. It read the entire S3 object into memory before returning a read
stream, which could cause Lambdas to run out of memory. Use
@cumulus/aws-client/S3.getObjectReadStream instead.@cumulus/ingest/util.lookupMimeType function now returns undefined
rather than null if the mime type could not be found.@cumulus/ingest/lock.removeLock function now returns undefined@cumulus/ingest/granule.generateMoveFileParams function now returns
source: undefined and target :undefined on the response object if either could not be
determined. Previously, null had been returned.@cumulus/ingest/recursion.recursion function must now be imported using
const { recursion } = require('@cumulus/ingest/recursion');@cumulus/ingest/granule.getRenamedS3File function has been renamed to
listVersionedObjects@cumulus/common.http has been removed@cumulus/common/http.download has been removed@cumulus/aws-client/S3.getObject to get an AWS S3 object@cumulus/aws-client/S3.waitForObject to get an AWS S3 object,
retrying, if necessarystartTimestamp and endTimestamp parameters to endpoint
reconcilationReports. Setting these values will filter the returned
report to cumulus data that falls within the timestamps. It also causes the
report to be one directional, meaning cumulus is only reconciled with CMR,
but not the other direction. The Granules will be filtered by their
updatedAt values. Collections are filtered by the updatedAt time of their
granules, i.e. Collections with granules that are updatedAt a time between
the time parameters will be returned in the reconciliation reports.startTimestamp and endTimestamp parameters to create-reconciliation-reports
lambda function. If either of these params is passed in with a value that can be
converted to a date object, the inter-platform comparison between Cumulus and CMR will
be one way. That is, collections, granules, and files will be filtered by time for
those found in Cumulus and only those compared to the CMR holdings. For the moment
there is not enough information to change the internal consistency check, and S3 vs
Cumulus comparisons are unchanged by the timestamps.location as parameter to /reconciliationReports endpoint. Options are S3
resulting in a S3 vs. Cumulus database search or CMR resulting in CMR vs. Cumulus database search.granuleId as input parameter to /reconcilationReports
endpoint. Limits inputs parameters to either collectionId or granuleId
and will fail to create the report if both are provided. Adding granuleId
will find collections in Cumulus by granuleId and compare those one way
with those in CMR./reconciliationReports now validates any input json before starting the
async operation and the lambda handler no longer validates input
parameters.collectionId parameter to the /reconcilationReports
endpoint. Setting this value will limit the scope of the reconcilation
report to only the input collectionId when comparing Cumulus and
CMR. collectionId is provided an array of strings e.g. [shortname___version, shortname2___version2]update-cmr-access-constraints, that will set access constraints in CMR Metadata.
Currently supports UMMG-JSON and Echo10XML, where it will configure AccessConstraints and
RestrictionFlag/RestrictionComment, respectively.cumulus module:
tea_api_egress_log_grouptea_external_api_endpointtea_internal_api_endpointtea_rest_api_idtea_rest_api_root_resource_idtea_stack_namedistribution module:
tea_api_egress_log_grouptea_external_api_endpointtea_internal_api_endpointtea_rest_api_idtea_rest_api_root_resource_idtea_stack_name@cumulus/api/lambdas/internal-reconciliation-report, so create-reconciliation-report
lambda can create Internal reconciliation report@cumulus/api/models/granule.unpublishAndDeleteGranule which
unpublishes a granule from CMR and deletes it from Cumulus, but does not
update the record to published: false before deletioncumulus-rds-tf DB cluster module to tf-modules that adds a
serverless RDS Aurora/PostgreSQL database cluster to meet the PostgreSQL
requirements for future releases.rds_security_group for database accessdata-migration1 Terraform module and Lambda to migrate data from Dynamo to RDS
rds_connection_heartbeat to cumulus and data-migration tf
modules. If set to true, this diagnostic variable instructs Core's database
code to fire off a connection 'heartbeat' query and log the timing/results
for diagnostic purposes, and retry certain connection timeouts once.
This option is disabled by defaultInternal reconciliation reportdata-migration1 Lambda for migrating providers data from Dynamo to RDS
executionNamePrefix property. If set, any executions
triggered as a result of that rule will use that prefix in the name of the
execution.QueueGranules task now supports an executionNamePrefix property. Any
executions queued by that task will use that prefix in the name of the
execution. See the
example workflow
for usage.QueuePdrs task now supports an executionNamePrefix config property.
Any executions queued by that task will use that prefix in the name of the
execution. See the
example workflow
for usage./reconciliationReport endpoint. The new report
is Granule Inventory. This report is a CSV file of all the granules in
the Cumulus DB. This report will eventually replace the existing
granules-csv endpoint which has been deprecated.ems_deploy variable to the cumulus module. This is set to false by default, except
for our example deployment, where it is needed for integration tests.applyWorkflow functionality on the granules endpoint to take a meta property to pass into the workflow message.BULK_GRANULE functionality on the granules endpoint to support the above applyWorkflow change.distribution_api_gateway_stage variable for cumulus module to tea_api_gateway_stageapi_gateway_stage variable for distribution module to tea_api_gateway_stage/reconciliationReport's file reconciliation to include "EXTENDED METADATA" as a valid CMR relatedUrls Type.cumulus elasticsearch index results in the collection granule stats
queries failing for the collections list api endpointactiveCollections query only returning 10 resultsQueueGranules task. That task now has a
config.concurrency option that defaults to 3.public-read ACL from the move-granules taskcumulus index is recreated and attached to an alias if it has been previously deleted/token not working when using a Cloudfront endpoint to access the Cumulus API with Launchpad authentication enabled. The redirect should now work properly whether you are using a plain API gateway URL or a Cloudfront endpoint pointing at an API gateway URL.@cumulus/aws-client/S3.getS3Object()@cumulus/message/Queue.getQueueNameByUrl()@cumulus/message/Queue.getQueueName()@cumulus/api/endpoints/granules-csv/list()distribution_url and distribution_redirect_uri outputs from the cumulus modulecumulus module:
distribution_urllog_api_gateway_to_cloudwatchthin_egress_cookie_domainthin_egress_domain_cert_arnthin_egress_download_role_in_region_arnthin_egress_jwt_algothin_egress_jwt_secret_namethin_egress_lambda_code_dependency_archive_keythin_egress_stack_namedistribution module:
distribution_urlinternal_tea_apirest_api_idthin_egress_app_redirect_uridistribution module:
bucket_map_keydistribution_urllog_api_gateway_to_cloudwatchthin_egress_cookie_domainthin_egress_domain_cert_arnthin_egress_download_role_in_region_arnthin_egress_jwt_algothin_egress_jwt_secret_namethin_egress_lambda_code_dependency_archive_keyproviderSecretsMigration and verifyProviderSecretsMigration lambdas@cumulus/sf-sns-report task@cumulus/aws-client/S3.calculateS3ObjectChecksum@cumulus/aws-client/S3.getS3ObjectReadStream@cumulus/cmrjs.getFullMetadata@cumulus/cmrjs.getMetadata@cumulus/common/util.isNil@cumulus/common/util.isNull@cumulus/common/util.isUndefined@cumulus/common/util.lookupMimeType@cumulus/common/util.mkdtempSync@cumulus/common/util.negate@cumulus/common/util.noop@cumulus/common/util.omit@cumulus/common/util.renameProperty@cumulus/common/util.sleep@cumulus/common/util.thread@cumulus/ingest/granule.copyGranuleFile@cumulus/ingest/granule.moveGranuleFile@cumulus/integration-tests/api/rules.deleteRule@cumulus/integration-tests/api/rules.getRule@cumulus/integration-tests/api/rules.listRules@cumulus/integration-tests/api/rules.postRule@cumulus/integration-tests/api/rules.rerunRule@cumulus/integration-tests/api/rules.updateRule@cumulus/integration-tests/sfnStep.parseStepMessage@cumulus/message/Queue.getQueueName@cumulus/message/Queue.getQueueNameByUrlFAQs
Common utilities used across tasks
The npm package @cumulus/common receives a total of 210 weekly downloads. As such, @cumulus/common popularity was classified as not popular.
We found that @cumulus/common demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket researchers uncover a malicious npm package posing as a tool for detecting vulnerabilities in Etherium smart contracts.
Security News
Research
A supply chain attack on Rspack's npm packages injected cryptomining malware, potentially impacting thousands of developers.
Research
Security News
Socket researchers discovered a malware campaign on npm delivering the Skuld infostealer via typosquatted packages, exposing sensitive data.