@cumulus/checksum
Advanced tools
Changelog
[v6.0.0] 2021-02-16
CUMULUS-2255 - Cumulus has upgraded its supported version of Terraform from 0.12.12 to 0.13.6. Please see the instructions to upgrade your deployments.
CUMULUS-2350
/s3credentialsREADME
, does not appear to be working after
deployment, manual redeployment
of the API-gateway stage may be necessary to finish the deployment.childWorkflowMeta
to queue-granules
config. Object passed to this
value will be merged into a child workflow message's meta
object. For an
example of how this can be used, see
example/cumulus-tf/discover_granules_workflow.asl.json
./s3credentialsREADME
, to the
s3-credentials-endpoint that displays information on how to use the
/s3credentials
endpointecs_custom_sg_ids
to Cumulus module to allow for
custom security group mappingssftp()
for use in @cumulus/sftp-client
CUMULUS-2323
source_bucket
key in granule.files
objects. If incoming payloads using
this task have a source_bucket
value for a file using the s3 provider, the
task will attempt to sync from the bucket defined in the file's
source_bucket
key instead of the provider
.
S3ProviderClient.sync
to allow for an optional bucket parameter
in support of the changed behavior.addBucketToFile
and related code from sync-granules taskCUMULUS-2255
CUMULUS-2321
/reconciliationReports/{name}
to return the
presigned s3 URL in addition to report dataUpdated hyrax-metadata-updates
task so the opendap url has Type 'USE SERVICE API'
CUMULUS-2310
CUMULUS-2351
Changelog
[v5.0.1] 2021-01-27
CUMULUS-2344
CUMULUS-2351
execution_cumulus_id
column from granules RDS schema and data typeChangelog
[v5.0.0] 2021-01-12
egress_lambda_log_group
and
egress_lambda_log_subscription_filter
resource were removed from the
cumulus
module, new definitions for these resources must be added to
cumulus-tf/main.tf
. For reference on how to define these resources, see
example/cumulus-tf/thin_egress_app.tf
.tea_stack_name
variable being passed into the cumulus
module should be removedasync_operation_image
as cumulus
module variable to allow for override of the async_operation container image. Users can optionally specify a non-default docker image for use with Core async operations.lzards-backup
Core task to facilitate making LZARDS backup requests in Cumulus ingest workflows@cumulus/hyrax-metadata-updates
Add component URI encoding for entry title id and granule ur to allow for values with special characters in them. For example, EntryTitleId 'Sentinel-6A MF/Jason-CS L2 Advanced Microwave Radiometer (AMR-C) NRT Geophysical Parameters' Now, URLs generated from such values will be encoded correctly and parsable by HyraxInTheCloudlzards-backup
Core task to facilitate making LZARDS backup requests in Cumulus ingest workflowsqueryFields
to granule schema, and this allows workflow tasks to add queryable data to granule record. For reference on how to add data to queryFields
field, see example/cumulus-tf/kinesis_trigger_test_workflow.tf
.async_operation_image
as cumulus
module variable to allow for override of the async_operation container image. Users can optionally specify a non-default docker image for use with Core async operations./search/granules.json
endpoint, now it is a direct link to /search/concepts/conceptid.format
parse-pdr
by updating @cumulus/pvl
to parse fields in a manner more consistent with the PDR ICD, with respect to numbers and dates. Anything not matching the ICD expectations, or incompatible with Javascript parsing, will be parsed as a string instead.tea_stack_name
variable from tf-modules/distribution/variables.tf
and tf-modules/cumulus/variables.tf
egress_lambda_log_group
and egress_lambda_log_subscription_filter
resources from tf-modules/distribution/main.tf
Changelog
[v4.0.0] 2020-11-20
cumulus_message_adapter_lambda_layer_arn
variable for the cumulus
module to cumulus_message_adapter_lambda_layer_version_arn
. The value of the variable should remain the same (a layer version ARN of a Lambda layer for the cumulus-message-adapter
.MoveGranules
step to add UpdateGranulesCmrMetadataFileLinksStep
that runs after it. See the example IngestAndPublishWorkflow
for reference.cumulus
module, a new resource definition for egress_api_gateway_log_subscription_filter
must be added to cumulus-tf/main.tf
. For reference on how to define this resource, see example/cumulus-tf/main.tf
.deploy_to_ngap
is false./collections[&getMMT=true]
and /collections/active[&getMMT=true]
endpoints. When a user provides a value of true
for getMMT
in the query parameters, the endpoint will search CMR and update each collection's results with new key MMTLink
containing a link to the MMT (Metadata Management Tool) if a CMR collection id is found.granules/bulkReingest
endpoint to @cumulus/api
log_api_gateway_to_cloudwatch
variable to example/cumulus-tf/variables.tf
.log_api_gateway_to_cloudwatch
variable to thin_egress_app
module definition./collection
and /collection/active
endpoints now return collections without granule aggregate statistics by default. The original behavior is preserved and can be found by including a query param of includeStats=true
on the request to the endpoint.es/collections
Collection class takes a new parameter includeStats. It no longer appends granule aggregate statistics to the returned results by default. One must set the new parameter to any non-false value.dbIndexer
lambda to process requests in serialtf-modules/distribution/main.tf
to example/cumulus-tf/main.tf
thin_egress_app
module output for a resource count.tea_api_egress_log_group
variable from tf-modules/distribution/variables.tf
and tf-modules/cumulus/variables.tf
.move-granules
task into a
new update-granules-cmr-metadata-file-links
task./collection
and /collection/active
endpoints now return collections without granule aggregate statistics by default. The original behavior is preserved and can be found by including a query param of includeStats=true
on the request to the endpoint. This is likely to affect the dashboard only but included here for the change of behavior.cumulus_message_adapter_lambda_layer_arn
output from the cumulus-message-adapter
module to cumulus_message_adapter_lambda_layer_version_arn
. The output value has changed from being the ARN of the Lambda layer without a version to the ARN of the Lambda layer with a version.cumulus
and ingest
modules from cumulus_message_adapter_lambda_layer_arn
to cumulus_message_adapter_lambda_layer_version_arn
Changelog
[v3.0.1] 2020-10-21
CUMULUS-2203
CUMULUS-2232
ajv
, lodash
, googleapis
, archiver
, and
@cumulus/aws-client
to remediate vulnerabilities found in SNYK scan.Changelog
[v3.0.0] 2020-10-7
CUMULUS-2099
meta.queues
in workflow configuration must be replaced with references to queue URLs from Terraform resources. See the updated data cookbooks or example Discover Granules workflow configuration.CUMULUS-2111
Before you re-deploy your cumulus-tf
module, note that the [thin-egress-app
][thin-egress-app] is no longer deployed by default as part of the cumulus
module, so you must add the TEA module to your deployment and manually modify your Terraform state to avoid losing your API gateway and impacting any Cloudfront endpoints pointing to those gateways. If you don't care about losing your API gateway and impacting Cloudfront endpoints, you can ignore the instructions for manually modifying state.
Add the [thin-egress-app
][thin-egress-app] module to your cumulus-tf
deployment as shown in the Cumulus example deployment.
tea_stack_name
variable to the cumulus
module and the stack_name
variable to the thin_egress_app
module must matchstage_name
variable to the thin_egress_app
module, the value of the tea_api_gateway_stage
variable to the cumulus
module must match itIf you want to preserve your existing thin-egress-app
API gateway and avoid having to update your Cloudfront endpoint for distribution, then you must follow these instructions: https://nasa.github.io/cumulus/docs/upgrade-notes/migrate_tea_standalone. Otherwise, you can re-deploy as usual.
If you provide your own custom bucket map to TEA as a standalone module, you must ensure that your custom bucket map includes mappings for the protected
and public
buckets specified in your cumulus-tf/terraform.tfvars
, otherwise Cumulus may not be able to determine the correct distribution URL for ingested files and you may encounter errors
CUMULUS-2197
ems_deploy
is set to false
by default, which will delete your EMS resources.ems_deploy
variable set to true
in your cumulus-tf/terraform.tfvars
Granule Inventory
's
/reconciliationReport
returns. The user (dashboard) must read the value
of url
from the return to get the s3SignedURL and then download the report.meta.queues
has been removed from Cumulus core workflow messages.@cumulus/sf-sqs-report
workflow task no longer reads the reporting queue URL from input.meta.queues.reporting
on the incoming event. Instead, it requires that the queue URL be set as the reporting_queue_url
environment variable on the deployed Lambda.thin-egress-app
module has be removed from tf-modules/distribution
, which is a part of the tf-modules/cumulus
module. Thus, the thin-egress-app
module is no longer deployed for you by default. See the migration steps for details about how to add deployment for the thin-egress-app
.parse-pdr
task has been updated to respect the NODE_NAME
property in
a PDR's FILE_GROUP
. If a NODE_NAME
is present, the task will query the
Cumulus API for a provider with that host. If a provider is found, the
output granule from the task will contain a provider
property containing
that provider. If NODE_NAME
is set but a provider with that host cannot be
found in the API, or if multiple providers are found with that same host,
the task will fail.queue-granules
task has been updated to expect an optional
granule.provider
property on each granule. If present, the granule will be
enqueued using that provider. If not present, the task's config.provider
will be used instead.@cumulus/api-client.providers.getProviders
function now takes a
queryStringParameters
parameter which can be used to filter the providers
which are returned@cumulus/aws-client/S3.getS3ObjectReadStreamAsync
function has been
removed. It read the entire S3 object into memory before returning a read
stream, which could cause Lambdas to run out of memory. Use
@cumulus/aws-client/S3.getObjectReadStream
instead.@cumulus/ingest/util.lookupMimeType
function now returns undefined
rather than null
if the mime type could not be found.@cumulus/ingest/lock.removeLock
function now returns undefined
@cumulus/ingest/granule.generateMoveFileParams
function now returns
source: undefined
and target :undefined
on the response object if either could not be
determined. Previously, null
had been returned.@cumulus/ingest/recursion.recursion
function must now be imported using
const { recursion } = require('@cumulus/ingest/recursion');
@cumulus/ingest/granule.getRenamedS3File
function has been renamed to
listVersionedObjects
@cumulus/common.http
has been removed@cumulus/common/http.download
has been removed@cumulus/aws-client/S3.getObject
to get an AWS S3 object@cumulus/aws-client/S3.waitForObject
to get an AWS S3 object,
retrying, if necessarystartTimestamp
and endTimestamp
parameters to endpoint
reconcilationReports
. Setting these values will filter the returned
report to cumulus data that falls within the timestamps. It also causes the
report to be one directional, meaning cumulus is only reconciled with CMR,
but not the other direction. The Granules will be filtered by their
updatedAt
values. Collections are filtered by the updatedAt time of their
granules, i.e. Collections with granules that are updatedAt a time between
the time parameters will be returned in the reconciliation reports.startTimestamp
and endTimestamp
parameters to create-reconciliation-reports
lambda function. If either of these params is passed in with a value that can be
converted to a date object, the inter-platform comparison between Cumulus and CMR will
be one way. That is, collections, granules, and files will be filtered by time for
those found in Cumulus and only those compared to the CMR holdings. For the moment
there is not enough information to change the internal consistency check, and S3 vs
Cumulus comparisons are unchanged by the timestamps.location
as parameter to /reconciliationReports
endpoint. Options are S3
resulting in a S3 vs. Cumulus database search or CMR
resulting in CMR vs. Cumulus database search.granuleId
as input parameter to /reconcilationReports
endpoint. Limits inputs parameters to either collectionId
or granuleId
and will fail to create the report if both are provided. Adding granuleId
will find collections in Cumulus by granuleId and compare those one way
with those in CMR./reconciliationReports
now validates any input json before starting the
async operation and the lambda handler no longer validates input
parameters.collectionId
parameter to the /reconcilationReports
endpoint. Setting this value will limit the scope of the reconcilation
report to only the input collectionId when comparing Cumulus and
CMR. collectionId
is provided an array of strings e.g. [shortname___version, shortname2___version2]
update-cmr-access-constraints
, that will set access constraints in CMR Metadata.
Currently supports UMMG-JSON and Echo10XML, where it will configure AccessConstraints
and
RestrictionFlag/RestrictionComment
, respectively.cumulus
module:
tea_api_egress_log_group
tea_external_api_endpoint
tea_internal_api_endpoint
tea_rest_api_id
tea_rest_api_root_resource_id
tea_stack_name
distribution
module:
tea_api_egress_log_group
tea_external_api_endpoint
tea_internal_api_endpoint
tea_rest_api_id
tea_rest_api_root_resource_id
tea_stack_name
@cumulus/api/lambdas/internal-reconciliation-report
, so create-reconciliation-report
lambda can create Internal
reconciliation report@cumulus/api/models/granule.unpublishAndDeleteGranule
which
unpublishes a granule from CMR and deletes it from Cumulus, but does not
update the record to published: false
before deletioncumulus-rds-tf
DB cluster module to tf-modules
that adds a
serverless RDS Aurora/PostgreSQL database cluster to meet the PostgreSQL
requirements for future releases.rds_security_group
for database accessdata-migration1
Terraform module and Lambda to migrate data from Dynamo to RDS
rds_connection_heartbeat
to cumulus
and data-migration
tf
modules. If set to true, this diagnostic variable instructs Core's database
code to fire off a connection 'heartbeat' query and log the timing/results
for diagnostic purposes, and retry certain connection timeouts once.
This option is disabled by defaultInternal
reconciliation reportdata-migration1
Lambda for migrating providers data from Dynamo to RDS
executionNamePrefix
property. If set, any executions
triggered as a result of that rule will use that prefix in the name of the
execution.QueueGranules
task now supports an executionNamePrefix
property. Any
executions queued by that task will use that prefix in the name of the
execution. See the
example workflow
for usage.QueuePdrs
task now supports an executionNamePrefix
config property.
Any executions queued by that task will use that prefix in the name of the
execution. See the
example workflow
for usage./reconciliationReport
endpoint. The new report
is Granule Inventory
. This report is a CSV file of all the granules in
the Cumulus DB. This report will eventually replace the existing
granules-csv
endpoint which has been deprecated.ems_deploy
variable to the cumulus
module. This is set to false by default, except
for our example deployment, where it is needed for integration tests.applyWorkflow
functionality on the granules endpoint to take a meta
property to pass into the workflow message.BULK_GRANULE
functionality on the granules endpoint to support the above applyWorkflow
change.distribution_api_gateway_stage
variable for cumulus
module to tea_api_gateway_stage
api_gateway_stage
variable for distribution
module to tea_api_gateway_stage
/reconciliationReport
's file reconciliation to include "EXTENDED METADATA"
as a valid CMR relatedUrls Type.cumulus
elasticsearch index results in the collection granule stats
queries failing for the collections list api endpointactiveCollections
query only returning 10 resultsQueueGranules
task. That task now has a
config.concurrency
option that defaults to 3
.public-read
ACL from the move-granules
taskcumulus
index is recreated and attached to an alias if it has been previously deleted/token
not working when using a Cloudfront endpoint to access the Cumulus API with Launchpad authentication enabled. The redirect should now work properly whether you are using a plain API gateway URL or a Cloudfront endpoint pointing at an API gateway URL.@cumulus/aws-client/S3.getS3Object()
@cumulus/message/Queue.getQueueNameByUrl()
@cumulus/message/Queue.getQueueName()
@cumulus/api/endpoints/granules-csv/list()
distribution_url
and distribution_redirect_uri
outputs from the cumulus
modulecumulus
module:
distribution_url
log_api_gateway_to_cloudwatch
thin_egress_cookie_domain
thin_egress_domain_cert_arn
thin_egress_download_role_in_region_arn
thin_egress_jwt_algo
thin_egress_jwt_secret_name
thin_egress_lambda_code_dependency_archive_key
thin_egress_stack_name
distribution
module:
distribution_url
internal_tea_api
rest_api_id
thin_egress_app_redirect_uri
distribution
module:
bucket_map_key
distribution_url
log_api_gateway_to_cloudwatch
thin_egress_cookie_domain
thin_egress_domain_cert_arn
thin_egress_download_role_in_region_arn
thin_egress_jwt_algo
thin_egress_jwt_secret_name
thin_egress_lambda_code_dependency_archive_key
providerSecretsMigration
and verifyProviderSecretsMigration
lambdas@cumulus/sf-sns-report
task@cumulus/aws-client/S3.calculateS3ObjectChecksum
@cumulus/aws-client/S3.getS3ObjectReadStream
@cumulus/cmrjs.getFullMetadata
@cumulus/cmrjs.getMetadata
@cumulus/common/util.isNil
@cumulus/common/util.isNull
@cumulus/common/util.isUndefined
@cumulus/common/util.lookupMimeType
@cumulus/common/util.mkdtempSync
@cumulus/common/util.negate
@cumulus/common/util.noop
@cumulus/common/util.omit
@cumulus/common/util.renameProperty
@cumulus/common/util.sleep
@cumulus/common/util.thread
@cumulus/ingest/granule.copyGranuleFile
@cumulus/ingest/granule.moveGranuleFile
@cumulus/integration-tests/api/rules.deleteRule
@cumulus/integration-tests/api/rules.getRule
@cumulus/integration-tests/api/rules.listRules
@cumulus/integration-tests/api/rules.postRule
@cumulus/integration-tests/api/rules.rerunRule
@cumulus/integration-tests/api/rules.updateRule
@cumulus/integration-tests/sfnStep.parseStepMessage
@cumulus/message/Queue.getQueueName
@cumulus/message/Queue.getQueueNameByUrl
Changelog
[v2.0.7] 2020-10-1 - [BACKPORT]
node-forge
dependency to 0.10.0 to address CVE findingChangelog
[v2.0.6] 2020-09-25 - [BACKPORT]
cumulus
elasticsearch index results in the collection granule stats
queries failing for the collections list api endpointChangelog
[v2.0.5] 2020-09-15 - [BACKPORT]
thin_egress_stack_name
variable to cumulus
and distribution
Terraform modules to allow overriding the default Cloudformation stack name used for the thin-egress-app
. Please note that if you change/set this value for an existing deployment, it will destroy and re-create your API gateway for the thin-egress-app
.