@cumulus/test-data
Advanced tools
@cumulus/test-data provides a collection of example data for use in testing Cumulus modules.
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
npm install @cumulus/test-data
Using require or import:
const payload = require('@cumulus/test-data/payloads/new-message-schema/ingest.json');
import payload from '@cumulus/test-data/payloads/new-message-schema/ingest.json';
See Cumulus README
[v1.14.0] - 2019-08-22
We have encountered transient lambda service errors in our integration testing. Please handle transient service errors following these guidelines. The workflows in the example/workflows folder have been updated with retries configured for these errors.
CUMULUS-799 added additional IAM permissions to support reading CloudWatch and API Gateway, so you will have to redeploy your IAM stack.
CUMULUS-800 Several items:
Delete existing API Gateway stages: To allow enabling of API Gateway logging, Cumulus now creates and manages a Stage resource during deployment. Before upgrading Cumulus, it is necessary to delete the API Gateway stages on both the Backend API and the Distribution API. Instructions are included in the documentation under Delete API Gateway Stages.
Set up account permissions for API Gateway to write to CloudWatch: In a one time operation for your AWS account, to enable CloudWatch Logs for API Gateway, you must first grant the API Gateway permission to read and write logs to CloudWatch for your account. The AmazonAPIGatewayPushToCloudWatchLogs managed policy (with an ARN of arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs) has all the required permissions. You can find a simple how to in the documentation under Enable API Gateway Logging.
Configure API Gateway to write logs to CloudWatch To enable execution logging for the distribution API set config.yaml apiConfigs.distribution.logApigatewayToCloudwatch value to true. More information Enable API Gateway Logs
Configure CloudWatch log delivery: It is possible to deliver CloudWatch API execution and access logs to a cross-account shared AWS::Logs::Destination. An operator does this by adding the key logToSharedDestination to the config.yml at the default level with a value of a writable log destination. More information in the documentation under Configure CloudWatch Logs Delivery.
Additional Lambda Logging: It is now possible to configure any lambda to deliver logs to a shared subscriptions by setting logToSharedDestination to the ARN of a writable location (either an AWS::Logs::Destination or a Kinesis Stream) on any lambda config. Documentation for Lambda Log Subscriptions
Configure S3 Server Access Logs: If you are running Cumulus in an NGAP environment you may configure S3 Server Access Logs to be delivered to a shared bucket where the Metrics Team will ingest the logs into their ELK stack. Contact the Metrics team for permission and location.
CUMULUS-1368 The Cumulus distribution API has been deprecated and is being replaced by ASF's Thin Egress App. By default, the distribution API will not deploy. Please follow the instructions for deploying and configuring Thin Egress.
To instead continue to deploy and use the legacy Cumulus distribution app, add the following to your config.yml:
deployDistributionApi: true
If you deploy with no distribution app your deployment will succeed but you may encounter errors in your workflows, particularly in the MoveGranule task.
BREAKING CHANGES below for details.CUMULUS-642
CUMULUS-1418
cumulus-ecs-task and a CMA layer instead of kes CMA injection.PR1125 - Adds layers config option to support deploying Lambdas with layers
PR1128 - Added useXRay config option to enable AWS X-Ray for Lambdas.
CUMULUS-1345
cmr.cmrEnvironment values are SIT, UAT, or OPS with UAT as the default.cmrLimit and cmrPageSize have been added as configurable options.CUMULUS-1273
CUMULUS-1226
elasticsearch/index-from-database to index to an Elasticsearch index from the database for recovery purposes and elasticsearch/indices-status to check the status of Elasticsearch indices via the API.CUMULUS-824
reportToEms to configure whether the collection is reported to EMSCUMULUS-1357
ems that generates EMS reports.CUMULUS-1241
meta.queueExecutionLimits)CUMULUS-1311
@cumulus/common/message with various message parsing/preparation helpersCUMULUS-812
CUMULUS-1337
cumulus.stackName value to the instanceMetadata endpoint.CUMULUS-1368
cmrGranuleUrlType to the @cumulus/move-granules task. This determines what kind of links go in the CMR files. The options are distribution, s3, or none, with the default being distribution. If there is no distribution API being used with Cumulus, you must set the value to s3 or none.Added packages/s3-replicator Terraform module to allow same-region s3 replication to metrics bucket.
CUMULUS-1392
tf-modules/report-granules Terraform module which processes granule ingest notifications received via SNS and stores granule data to a database. The module includes:
CUMULUS-1393
tf-modules/report-pdrs Terraform module which processes PDR ingest notifications received via SNS and stores PDR data to a database. The module includes:
@cumulus/api/models/pdrs.createPdrFromSns()CUMULUS-1400
tf-modules/report-executions Terraform module which processes workflow execution information received via SNS and stores it to a database. The module includes:
@cumulus/common/sns-event which contains helpers for SNS events:
isSnsEvent() returns true if event is from SNSgetSnsEventMessage() extracts and parses the message from an SNS eventgetSnsEventMessageObject() extracts and parses message object from an SNS event@cumulus/common/cloudwatch-event which contains helpers for Cloudwatch events:
isSfExecutionEvent() returns true if event is from Step FunctionsisTerminalSfStatus() determines if a Step Function status from a Cloudwatch event is a terminal statusgetSfEventStatus() gets the Step Function status from a Cloudwatch eventgetSfEventDetailValue() extracts a Step Function event detail field from a Cloudwatch eventgetSfEventMessageObject() extracts and parses Step Function detail object from a Cloudwatch eventCUMULUS-1429
tf-modules/data-persistence Terraform module which includes resources for data persistence in Cumulus:
CUMULUS-1379 CMR Launchpad Authentication
launchpad configuration to @cumulus/deployment/app/config.yml, and cloudformation templates, workflow message, lambda configuration, api endpoint configuration@cumulus/common/LaunchpadToken and @cumulus/common/launchpad to provide methods to get token and validate tokenCUMULUS-1232
@cumulus/cmr-client updateToken()CUMULUS-1245 CUMULUS-795
ems configuration parameters for sending the ingest reports to EMSCUMULUS-1241
sqs2sfThrottle lambda:
cumulus_meta.queueNamemeta.queueExecutionLimits[queueName], where queueName is cumulus_meta.queueNamesfSemaphoreDown lambda to only attempt decrementing semaphores when:
cumulus_meta.queueName exists on the Cumulus message ANDcumulus_meta.queueName) exists in the the object meta.queueExecutionLimits on the Cumulus messageCUMULUS-1338
sfSemaphoreDown lambda to be triggered via AWS Step Function Cloudwatch events instead of subscription to sfTracker SNS topicCUMULUS-1311
@cumulus/queue-granules to set cumulus_meta.queueName for queued execution messages@cumulus/queue-pdrs to set cumulus_meta.queueName for queued execution messagessqs2sfThrottle lambda to immediately decrement queue semaphore value if dispatching Step Function execution throws an errorCUMULUS-1362
processingStartTime and processingEndTime will be set to the execution start time and end time respectively when there is no sync granule or post to cmr task present in the workflowCUMULUS-1400
@cumulus/ingest/aws/getExecutionArn. Use @cumulus/common/aws/getExecutionArn instead.CUMULUS-1439
CUMULUS-796
CUMULUS-1319
CUMULUS-1356
Collection model's delete method now removes the specified item
from the collection config store that was inserted by the create method.
Previously, this behavior was missing.CUMULUS-1374
cmaDir key to configuration will cause CUMULUS_MESSAGE_ADAPTER_DIR to be set by default to /opt for any Lambda not setting useCma to true, or explicitly setting the CMA environment variable. In lambdas that package the CMA independently of the Cumulus packaging. Lambdas manually packaging the CMA should have their Lambda configuration updated to set the CMA path, or alternately if not using the CMA as a Lambda layer in this deployment set cmaDir to ./cumulus-message-adapter.CUMULUS-1337
PR1130
@cumulus/common/step-functions. Use @cumulus/common/StepFunctions instead.@cumulus/api/lib/testUtils.fakeFilesFactory. Use @cumulus/api/lib/testUtils.fakeFileFactory instead.@cumulus/cmrjs/cmr functions: searchConcept, ingestConcept, deleteConcept. Use the functions in @cumulus/cmr-client instead.@cumulus/ingest/aws.getExecutionHistory. Use @cumulus/common/StepFunctions.getExecutionHistory instead.FAQs
Includes the test data for various packages
The npm package @cumulus/test-data receives a total of 219 weekly downloads. As such, @cumulus/test-data popularity was classified as not popular.
We found that @cumulus/test-data demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket researchers uncover a malicious npm package posing as a tool for detecting vulnerabilities in Etherium smart contracts.
Security News
Research
A supply chain attack on Rspack's npm packages injected cryptomining malware, potentially impacting thousands of developers.
Research
Security News
Socket researchers discovered a malware campaign on npm delivering the Skuld infostealer via typosquatted packages, exposing sensitive data.