
Security News
Vite+ Joins the Push to Consolidate JavaScript Tooling
Evan You announces Vite+, a commercial, Rust-powered toolchain built on the Vite ecosystem to unify JavaScript development and fund open source.
@cumulus/integration-tests
Advanced tools
This package provides a CLI and functions for testing Cumulus.
⚠️ The documented API of this package will not change without a deprecation warning being provided in earlier releases. Code in this package that is not documented in this README may change without warning, and is not considered part of the package's public API.
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
$ npm install @cumulus/integration-tests
Example
const Collections = require('@cumulus/integration-test/Collections');
Promise.<Object>
⏏
Array.<Object>
Object
boolean
Promise.<undefined>
Promise.<Array.<Object>>
Promise.<Object>
⏏Create a randomized collection using the Cumulus API.
The default collection is very simple. It expects that, for any discovered file, the granule ID
is everything in the filename before the extension. For example, a file named gran-1.txt
would
have a granuleId of gran-1
. Filenames can only contain a single .
character.
Collection defaults:
collection-name-
collection-version-
false
'^[^.]+$'
'^([^.]+)\..+$'
'asdf.jpg'
[
{
bucket: 'protected',
regex: '^[^.]+\..+$',
sampleFileName: 'asdf.jpg'
}
]
Kind: Exported function
Returns: Promise.<Object>
- the generated collection
Param | Type | Description |
---|---|---|
prefix | string | the Cumulus stack name |
[overrides] | Object | properties to set on the collection, overriding the defaults |
Array.<Object>
Given a Cumulus collection configuration, return a list of the filetype
configs with their url_path
s updated.
Kind: inner method of createCollection
Returns: Array.<Object>
- a list of collection filetype configs
Param | Type | Description |
---|---|---|
collection | Object | a Cumulus collection |
customFilePath | string | path to be added to the end of the url_path |
Object
Update a collection with a custom file path, duplicate handling, and name updated with the postfix.
Kind: inner method of createCollection
Returns: Object
- an updated collection
Param | Type | Description |
---|---|---|
params | Object | |
params.collection | Object | a collection configuration |
params.customFilePath | string | path to be added to the end of the url_path |
params.duplicateHandling | string | duplicate handling setting |
params.postfix | string | a string to be appended to the end of the name |
boolean
Returns true if collection exists. False otherwise.
Kind: inner method of createCollection
Param | Type | Description |
---|---|---|
stackName | string | the prefix of the Cumulus stack |
collection | Object | a Cumulus collection |
Promise.<undefined>
Add a new collection to Cumulus
Kind: inner method of createCollection
Param | Type | Description |
---|---|---|
stackName | string | the prefix of the Cumulus stack |
collection | Object | a Cumulus collection |
Promise.<Array.<Object>>
Add collections to database
Kind: inner method of createCollection
Returns: Promise.<Array.<Object>>
- - collections that were added
Param | Type | Description |
---|---|---|
stackName | string | Cloud formation stack name |
bucketName | string | S3 internal bucket name |
dataDirectory | string | the directory of collection json files |
[postfix] | string | string to append to collection name |
[customFilePath] | string | |
[duplicateHandling] | string |
Example
const Executions = require('@cumulus/integration-test/Executions');
Promise.<string>
⏏Promise.<Object>
⏏Promise.<string>
⏏Find the execution ARN matching the matcher
function
Kind: Exported function
Returns: Promise.<string>
- the ARN of the matching execution
Param | Type | Default | Description |
---|---|---|---|
prefix | string | the name of the Cumulus stack | |
matcher | function | a predicate function that takes an execution and determines if this is the execution that is being searched for | |
[queryParameters] | Object | Optional query parameters | |
[options] | Object | ||
[options.timeout] | integer | 0 | the number of seconds to wait for a matching execution to be found |
Promise.<Object>
⏏Wait for an execution to have an expected status and return the execution
Kind: Exported function
Returns: Promise.<Object>
- the execution as returned by the GET /executions/<execution-arn>
endpoint
Param | Type | Default | Description |
---|---|---|---|
params | Object | ||
params.prefix | string | the name of the Cumulus stack | |
params.arn | string | the execution ARN to fetch | |
params.status | string | the status to wait for | |
[params.callback] | function | cumulusApiClient.invokeApifunction | an async function to invoke the API Lambda that takes a prefix / user payload |
[params.timeout] | integer | 30 | the number of seconds to wait for the execution to reach a terminal state |
Example
const Granules = require('@cumulus/integration-test/Granules');
Promise.<Object>
⏏
Promise.<Object>
Promise.<Object>
⏏Wait for a granule to have an expected status and return the granule
Kind: Exported function
Returns: Promise.<Object>
- the granule as returned by the GET /granules/<granule-id>
endpoint
Param | Type | Default | Description |
---|---|---|---|
params | Object | ||
params.prefix | string | the name of the Cumulus stack | |
params.granuleId | string | the granuleId of the granule | |
params.collectionId | string | the collectionId of the granule | |
params.status | string | the status to wait for | |
params.updatedAt | string | minimum updatedAt time the granule must have to return | |
[params.callback] | function | cumulusApiClient.invokeApifunction | an async function to invoke the API Lambda that takes a prefix / user payload |
[params.timeout] | integer | 30 | the number of seconds to wait for the execution to reach a terminal state |
Promise.<Object>
Wait for listGranules to return at least a single value before returning an empty result
Kind: inner method of getGranuleWithStatus
Returns: Promise.<Object>
- - results of a successful listGranules
Param | Type | Description |
---|---|---|
params | Object | parameters to listGranules function |
Example
const Providers = require('@cumulus/integration-test/Providers');
Promise.<Object>
⏏Create a provider using the Cumulus API
Provider defaults:
provider-
s3
10
Kind: Exported function
Returns: Promise.<Object>
- the generated provider
Param | Type | Description |
---|---|---|
prefix | string | the Cumulus stack name |
[overrides] | Object | properties to set on the provider, overriding the defaults |
Example
const Rules = require('@cumulus/integration-test/Rules');
Promise.<Object>
⏏Create a onetime
rule using the Cumulus API
Rule defaults:
rule_
{ type: 'onetime' }
Kind: Exported function
Returns: Promise.<Object>
- the generated rule
Param | Type | Description |
---|---|---|
prefix | string | the name of the Cumulus stack |
[overrides] | Object | properties to set on the rule, overriding the defaults |
Usage: cumulus-test TYPE COMMAND [options]
Options:
-V, --version output the version number
-s, --stack-name <stackName> AWS Cloud Formation stack name (default: null)
-b, --bucket-name <bucketName> AWS S3 internal bucket name (default: null)
-w, --workflow <workflow> Workflow name (default: null)
-i, --input-file <inputFile> Workflow input JSON file (default: null)
-h, --help output usage information
Commands:
workflow Execute a workflow and determine if the workflow completes successfully
For example, to test the HelloWorld workflow:
cumulus-test workflow --stack-name helloworld-cumulus --bucket-name cumulus-bucket-internal --workflow HelloWorldWorkflow --input-file ./helloWorldInput.json
To make a contribution, please see our contributing guidelines.
Generated automatically using npm run build-docs
[v21.0.0] 2025-09-09
Please follow the instructions before upgrading Cumulus
checkCrossCollectionCollisions
may be set to false
to
disable the behavior on a per-workflow, per-collection or other config
driven criteria.parse-pdr
task component to throw an error if multiple
granules within the same PDR have the same granuleId after applying the
granuleIdFilter, unless the uniquifyGranuleId
configuration parameter is
explicitly set to true
.updateGranulesCmrMetadataFileLinks
to always ensure
producerGranuleId
identifier is set in updated CMR metadatacnm_response_task
to use newest version v3.2.0
, which supports
producerGranuleId
.cumulus-tf
includes cnm_response_version = "3.2.0"
or greater.producer_granule_id
to Postgres granules
table.producerGranuleId
property to granule
record schema.@cumulus
api/db/message packages to handle producer_granule_id
and producerGranuleId
.@cumulus/api/lib/writeGranulesFromMessage
to set producerGranuleId
= granuleId if not set.queue-granules
task to set producerGranuleId = granuleId if not
set.producerGranuleId
to LzardsBackup
task component and lambda input/output schemaLzardsBackup
task component to submit producerGranuleId
for storage in the lzards record as a key in the metadata
object.parse-pdr
task component to have the following behaviors:
uniquifyGranuleId
configuration value is set to true, parse-PDR will
update the granuleId for all found granules to have a unique granule hash
appended to the existing IDparse-pdr
such that if the uniquifyGranuleId
configuration
parameter is not set to true
, and a duplicate granuleId is created as
part of the output after passing the granuleIdFilter
, the task will
throw with an error.ingestFromPdrWithUniqueGranuleIdsSpec.js
to the spec tests to
demonstrate the ingest workflow works as expected with unique granuleIds and
producerGranuleIds set.ingest
terraform module for deployment
with Core. This task will update a payload of existing granules to have
'uniquified' IDs and preserve the original identifier in the
producerGranuleId
fieldIngestGranuleSuccessSpec
/IngestUMMGSuccessSpec
to validate
producerGranuleId is populated in CMR post ingestproducerGranuleId
in the default test case@cumulus/cmrjs/cmr-utils
updateCMRMetadata
to take updateGranuleIdentifiers
configuration
flag/producerGranuleId
such that that routine now will modify the CMR
metadata object with the correct GranuleUR
/ProducerGranuleId
values in
the CMR metadata.cmr-utils
getCmrMetadata
helper to @cumulus/integration-tests
to allow
access to the full CMR metadata object for verification of record metadata
fieldsApiFileGranuleIdOptional
to @cumulus/types/api
for cases where an
ApiFile is being generated and refactored existing code to use this type
instead of custom relaxed typingupdate-granules-cmr-metadata-file-links
to use the updated cmrjs
logic to set producerGranuleId identifiers in the CMR metadata, either equal
to granuleId or the producerGranuleID
set on the granule.@cumulus/tasks/sync-granule/GranuleFetcher
to allow and pass through an
incoming granule.producerGranuleId
@cumulus/api/lib/ingest.reingestGranule
to only update the original granule
to 'queued' if the original payload contains the granule. This avoids a situation
where the original granule is updated to 'queued', but the reingest workflow
creates a new granule, leaving the original granule stuck in 'queued'.getGranuleIdAndCollectionIdFromFile
query method to @cumulus/db
to
retrieve granule and collection metadata from a file's S3 location.GET /granules/files/get_collection_and_granule_id/:bucket/:key
in @cumulus/api
to
return the granule ID and collection ID associated with a file.getFileGranuleAndCollectionByBucketAndKey
method to
@cumulus/api-client/granules
to allow use of new endpoint.move-granules
task to validate cross-collection file collisions
using the new lookup logic when checkCrossCollectionCollisions
is enabled.@cumulus/db
to add getGranuleIdAndCollectionIdFromFile query methodupdate-granules-cmr-metadata-file-links
task READMEurlPathTemplate
to allow falling back from one null/undefined interpolated value to a second argumentcnmResponse
lambda version
3.1.0-alpha.2-SNAPSHOT which utilizes producerGranuleId
.cnmToGranule
lambda version 2.1.0.FakeProcessing
task configuration matchFilesWithProducerGranuleId
to determine if the generated cmr file names should match
granuleId
or producerGranuleId
AddUniqueGranuleId
task configuration hashLength
to accept
additional types and removed the use of hashDepth
.FilesToGranules
task configuration
matchFilesWithProducerGranuleId
to accept additional types.ParsePdr
task configuration hashLength
to accept additional
types.tf-modules/cumulus
AddUniqueGranuleId
task output.CNMExampleWorkflow
to uniquify
granuleIds based on collection configurationKinesisTestTriggerWithUniqueGranuleIdsSpec.js
to the spec test to
demonstrate that the CNM ingest workflow ingests granules with unique
granuleIds and producerGranuleIds set, and that CnmResponse sends responses
using producerGranuleIdsproducerGranuleId
when mapping files to their granules.ingestGranule
, discoverGranules
,
lzardsBackup
, cnmWorkflow
, and orca
specs.workflow_configurations
variable to the tf-modules/ingest
and
tf-modules/cumulus
modules.
The property sf_event_sqs_to_db_records_types
has been added to
workflow_template.json
under the cumulus_meta
field to control which record
types should be written to the database during different workflow execution statuses.
Currently, both "execution" and "pdr" must be written to the database, so the
record type list must include both.SfSqsReport
task to set meta.reportMessageSource
in the Cumulus message.@cumulus/api/sfEventSqsToDbRecords
lambda to determine which
record types ("execution", "granule", "pdr") should be written to the database based on the
cumulus_meta.sf_event_sqs_to_db_records_types
and meta.reportMessageSource
fields.
By default, all record types will be written to the database.@cumulus/api/lib.writeRecords.writeGranuleExecutionAssociationsFromMessage
to write granule-execution associations from message.@cumulus/integration-tests
cmr.generateAndStoreCmrXml
to
apply matchFilesWithProducerGranuleId
when generaing OnlineAccessURL
.KinesisTestTriggerWithUniqueGranuleIdsSpec
to cover "duplicate"
Granules in separate Collections.includeTimestampHashKey
parameter to the generateUniqueGranuleId
function in the @cumulus/ingest/granule
, with a default value of false
.includeTimestampHashKey
configuration to the add-unique-granuleId
and parse-pdr tasks
, also with a default value of false
."Generate Unique GranuleId"
to explain the algorithm for generating unique granuleIds
.producer_granule_id
migration script to disable autovacuum before the
migration and re-enable it afterward to improve performance.cumuluss/async-operation:54
. Users should update their references to async-operation
with the new version.FAQs
Integration tests
The npm package @cumulus/integration-tests receives a total of 656 weekly downloads. As such, @cumulus/integration-tests popularity was classified as not popular.
We found that @cumulus/integration-tests demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 6 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Evan You announces Vite+, a commercial, Rust-powered toolchain built on the Vite ecosystem to unify JavaScript development and fund open source.
Security News
Ruby Central’s incident report on the RubyGems.org access dispute sparks backlash from former maintainers and renewed debate over project governance.
Research
/Security News
Socket researchers uncover how threat actors weaponize Discord across the npm, PyPI, and RubyGems ecosystems to exfiltrate sensitive data.