@cumulus/common
Advanced tools
Common libraries used in Cumulus.
$ npm install @cumulus/common
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
See Cumulus README
[v1.15.0] - 2019-11-04
CUMULUS-1644 - When a workflow execution begins or ends, the workflow
payload is parsed and any new or updated PDRs or granules referenced in that
workflow are stored to the Cumulus archive. The defined interface says that a
PDR in payload.pdr will be added to the archive, and any granules in
payload.granules will also be added to the archive. In previous releases,
PDRs found in meta.pdr and granules found in meta.input_granules were also
added to the archive. This caused unexpected behavior and has been removed.
Only PDRs from payload.pdr and granules from payload.granules will now be
added to the Cumulus archive.
CUMULUS-1449 - Cumulus now uses a universal workflow template when
starting a workflow that contains general information specific to the
deployment, but not specific to the workflow. Workflow task configs must be
defined using AWS step function parameters. As part of this change,
CumulusConfig has been retired and task configs must now be defined under
the cma.task_config key in the Parameters section of a step function
definition.
Migration instructions:
NOTE: These instructions require the use of Cumulus Message Adapter v1.1.x+.
Please ensure you are using a compatible version before attempting to migrate
workflow configurations. When defining workflow steps, remove any
CumulusConfig section, as shown below:
ParsePdr:
CumulusConfig:
provider: "{$.meta.provider}"
bucket: "{$.meta.buckets.internal.name}"
stack: "{$.meta.stack}"
Instead, use AWS Parameters to pass task_config for the task directly into
the Cumulus Message Adapter:
ParsePdr:
Parameters:
cma:
event.$: "$"
task_config:
provider: "{$.meta.provider}"
bucket: "{$.meta.buckets.internal.name}"
stack: "{$.meta.stack}"
In this example, the cma key is used to pass parameters to the message
adapter. Using task_config in combination with event.$: '$' allows the
message adapter to process task_config as the config passed to the Cumulus
task. See example/workflows/sips.yml in the core repository for further
examples of how to set the Parameters.
Additionally, workflow configurations for the QueueGranules and QueuePdrs
tasks need to be updated:
queue-pdrs config changes:
parsePdrMessageTemplateUri replaced with parsePdrWorkflow, which is
the workflow name (i.e. top-level name in config.yml, e.g. 'ParsePdr').internalBucket and stackName configs now required to look up
configuration from the deployment. Brings the task config in line with
that of queue-granules.queue-granules config change: ingestGranuleMessageTemplateUri replaced
with ingestGranuleWorkflow, which is the workflow name (e.g.
'IngestGranule').CUMULUS-1396 - Workflow steps at the beginning and end of a workflow
using the SfSnsReport Lambda have now been deprecated (e.g. StartStatus,
StopStatus) and should be removed from your workflow definitions. These
steps were used for publishing ingest notifications and have been replaced by
an implementation using Cloudwatch events for Step Functions to trigger a
Lambda that publishes ingest notifications. For further detail on how ingest
notifications are published, see the notes below on CUMULUS-1394. For
examples of how to update your workflow definitions, see our
example workflow definitions.
CUMULUS-1470
AWS::ApplicationAutoScaling::ScalableTarget,
AWS::ApplicationAutoScaling::ScalingPolicy, and AWS::CloudWatch::Alarm
resources should be defined in a kes overrides file. See
this example
for an example.CUMULUS-1100
CUMULUS-1396
@cumulus/common/sfnStep:
LambdaStep - A class for retrieving and parsing input and output to Lambda steps in AWS Step FunctionsActivityStep - A class for retrieving and parsing input and output to ECS activity steps in AWS Step FunctionsCUMULUS-1574
GET /token endpoint for SAML authorization when cumulus is protected by Launchpad.
This lets a user retrieve a token by hand that can be presented to the API.CUMULUS-1625
sf_start_rate variable to the ingest Terraform module, equivalent to sqs_consumer_rate in the old model, but will not be automatically applied to custom queues as that was.CUMULUS-1513
sqs-type rule support in the Cumulus API @cumulus/apisqsMessageConsumer lambda which processes messages from the SQS queues configured in the sqs rules.CUMULUS-1639
NoSuchKey response even if that object has been created. The
@cumulus/common/aws.getS3Object() function will now retry up to 10 times
if a NoSuchKey response is returned by S3. This can behavior can be
overridden by passing { retries: 0 } as the retryOptions argument.CUMULUS-1449
queue-pdrs & queue-granules config changes. Details in breaking changes section.CumulusConfig to a task_config AWS Parameter.CUMULUS-1452
devicemapperCUMULUS-1453
@cumulus/sf-sns-report task@cumulus/sf-sns-report to always assume that it is running as an intermediate step in a workflow, not as the first or last stepCumulusConfig as part of step function definitions, as this is an artifact of the way Kes parses workflow definitions that was not possible to migrate to Terraform. Use AWS Parameters and the task_config key instead. See change note above.CUMULUS-1620 - Fixed bug where message_adapter_version does not correctly inject the CMA
CUMULUS-1396 - Updated @cumulus/common/StepFunctions.getExecutionHistory() to recursively fetch execution history when nextToken is returned in response
CUMULUS-1571 - Updated @cumulus/common/DynamoDb.get() to throw any errors encountered when trying to get a record and the record does exist
CUMULUS-1452
devicemapperFAQs
Common utilities used across tasks
The npm package @cumulus/common receives a total of 438 weekly downloads. As such, @cumulus/common popularity was classified as not popular.
We found that @cumulus/common demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket researchers uncover a malicious npm package posing as a tool for detecting vulnerabilities in Etherium smart contracts.
Security News
Research
A supply chain attack on Rspack's npm packages injected cryptomining malware, potentially impacting thousands of developers.
Research
Security News
Socket researchers discovered a malware campaign on npm delivering the Skuld infostealer via typosquatted packages, exposing sensitive data.