@cumulus/common - npm Package Versions

[v7.2.0] 2021-03-23

Added

• CUMULUS-2346
  • Added orca API endpoint to @cumulus/api to get recovery status
  • Add CopyToGlacier step to example IngestAndPublishGranuleWithOrca workflow

Changed

• HYRAX-357
  • Format of NGAP OPeNDAP URL changed and by default now is referring to concept id and optionally can include short name and version of collection.
  • addShortnameAndVersionIdToConceptId field has been added to the config inputs of the hyrax-metadata-updates task

[v7.1.0] 2021-03-12

Notable changes

• sync-granule task will now properly handle syncing 0 byte files to S3
• SQS/Kinesis rules now support scheduling workflows to a custom queue via the rule.queueUrl property. If provided, this value should be the full URL for an SQS queue.

Added

• tf-modules/cumulus module now supports a cmr_custom_host variable that can be used to set to an arbitrary host for making CMR requests (e.g. https://custom-cmr-host.com).
• Added buckets variable to tf-modules/archive
• CUMULUS-2345
  • Deploy ORCA with Cumulus, see example/cumulus-tf/orca.tf and example/cumulus-tf/terraform.tfvars.example
  • Add CopyToGlacier step to example IngestAndPublishGranule workflow
• CUMULUS-2424
  • Added childWorkflowMeta to queue-pdrs config. An object passed to this config value will be merged into a child workflow message's meta object. For an example of how this can be used, see example/cumulus-tf/discover_and_queue_pdrs_with_child_workflow_meta_workflow.asl.json.
• CUMULUS-2427
  • Added support for using a custom queue with SQS and Kinesis rules. Whatever queue URL is set on the rule.queueUrl property will be used to schedule workflows for that rule. This change allows SQS/Kinesis rules to use any throttled queues defined for a deployment.

Fixed

• CUMULUS-2394
  • Updated PDR and Granule writes to check the step function workflow_start_time against the createdAt field for each record to ensure old records do not overwrite newer ones

Changed

• <prefix>-lambda-api-gateway IAM role used by API Gateway Lambda now supports accessing all buckets defined in your buckets variable except "internal" buckets
• Updated the default scroll duration used in ESScrollSearch and part of the reconciliation report functions as a result of testing and seeing timeouts at its current value of 2min.
• CUMULUS-2355
  • Added logic to disable /s3Credentials endpoint based upon value for environment variable DISABLE_S3_CREDENTIALS. If set to "true", the endpoint will not dispense S3 credentials and instead return a message indicating that the endpoint has been disabled.
• CUMULUS-2397
  • Updated /elasticsearch endpoint's reindex function to prevent reindexing when source and destination indices are the same.
• CUMULUS-2420
  • Updated test function waitForAsyncOperationStatus to take a retryObject and use exponential backoff. Increased the total test duration for both AsycOperation specs and the ReconciliationReports tests.
  • Updated the default scroll duration used in ESScrollSearch and part of the reconciliation report functions as a result of testing and seeing timeouts at its current value of 2min.
• CUMULUS-2427
  • Removed queueUrl from the parameters object for @cumulus/message/Build.buildQueueMessageFromTemplate
  • Removed queueUrl from the parameters object for @cumulus/message/Build.buildCumulusMeta

Fixed

• Fixed issue in @cumulus/ingest/S3ProviderClient.sync() preventing 0 byte files from being synced to S3.

Removed

• Removed variables from tf-modules/archive:
  • private_buckets
  • protected_buckets
  • public_buckets

[v7.0.0] 2021-02-22

BREAKING CHANGES

• CUMULUS-2362 - Endpoints for the logs (/logs) will now throw an error unless Metrics is set up

Added

• CUMULUS-2345
  • Deploy ORCA with Cumulus, see example/cumulus-tf/orca.tf and example/cumulus-tf/terraform.tfvars.example
  • Add CopyToGlacier step to example IngestAndPublishGranule workflow
• CUMULUS-2376
  • Added cmrRevisionId as an optional parameter to post-to-cmr that will be used when publishing metadata to CMR.
• CUMULUS-2412
  • Adds function getCollectionsByShortNameAndVersion to @cumulus/cmrjs that performs a compound query to CMR to retrieve collection information on a list of collections. This replaces a series of calls to the CMR for each collection with a single call on the /collections endpoint and should improve performance when CMR return times are increased.

Changed

• CUMULUS-2362
  • Logs endpoints only work with Metrics set up
• CUMULUS-2376
  • Updated publishUMMGJSON2CMR to take in an optional revisionId parameter.
  • Updated publishUMMGJSON2CMR to throw an error if optional revisionId does not match resulting revision ID.
  • Updated publishECHO10XML2CMR to take in an optional revisionId parameter.
  • Updated publishECHO10XML2CMR to throw an error if optional revisionId does not match resulting revision ID.
  • Updated publish2CMR to take in optional cmrRevisionId.
  • Updated getWriteHeaders to take in an optional CMR Revision ID.
  • Updated ingestGranule to take in an optional CMR Revision ID to pass to getWriteHeaders.
  • Updated ingestUMMGranule to take in an optional CMR Revision ID to pass to getWriteHeaders.
• CUMULUS-2350
  • Updates the examples on the /s3credentialsREADME, to include Python and JavaScript code demonstrating how to refrsh the s3credential for programatic access.
• CUMULUS-2383
  • PostToCMR task will return CMRInternalError when a 500 status is returned from CMR

[v6.0.0] 2021-02-16

MIGRATION NOTES

• CUMULUS-2255 - Cumulus has upgraded its supported version of Terraform from 0.12.12 to 0.13.6. Please see the instructions to upgrade your deployments.

• CUMULUS-2350

  • If the /s3credentialsREADME, does not appear to be working after deployment, manual redeployment of the API-gateway stage may be necessary to finish the deployment.

BREAKING CHANGES

• CUMULUS-2255 - Cumulus has upgraded its supported version of Terraform from 0.12.12 to 0.13.6.

Added

• CUMULUS-2291
  • Add provider filter to Granule Inventory Report
• CUMULUS-2300
  • Added childWorkflowMeta to queue-granules config. Object passed to this value will be merged into a child workflow message's meta object. For an example of how this can be used, see example/cumulus-tf/discover_granules_workflow.asl.json.
• CUMULUS-2350
  • Adds an unprotected endpoint, /s3credentialsREADME, to the s3-credentials-endpoint that displays information on how to use the /s3credentials endpoint
• CUMULUS-2368
  • Add QueueWorkflow task
• CUMULUS-2391
  • Add reportToEms to collections.files file schema
• CUMULUS-2395
  • Add Core module parameter ecs_custom_sg_ids to Cumulus module to allow for custom security group mappings
• CUMULUS-2402
  • Officially expose sftp() for use in @cumulus/sftp-client

Changed

• CUMULUS-2323

  • The sync granules task when used with the s3 provider now uses the source_bucket key in granule.files objects. If incoming payloads using this task have a source_bucket value for a file using the s3 provider, the task will attempt to sync from the bucket defined in the file's source_bucket key instead of the provider.
    • Updated S3ProviderClient.sync to allow for an optional bucket parameter in support of the changed behavior.
  • Removed addBucketToFile and related code from sync-granules task

• CUMULUS-2255

  • Updated Terraform deployment code syntax for compatibility with version 0.13.6

• CUMULUS-2321

  • Updated API endpoint GET /reconciliationReports/{name} to return the presigned s3 URL in addition to report data

Fixed

• Updated hyrax-metadata-updates task so the opendap url has Type 'USE SERVICE API'

• CUMULUS-2310

  • Use valid filename for reconciliation report

• CUMULUS-2351

  • Inventory report no longer includes the File/Granule relation object in the okCountByGranules key of a report. The information is only included when a 'Granule Not Found' report is run.

Removed

• CUMULUS-2364
  • Remove the internal Cumulus logging lambda (log2elasticsearch)

[v5.0.1] 2021-01-27

Changed

• CUMULUS-2344

  • Elasticsearch API now allows you to reindex to an index that already exists
  • If using the Change Index operation and the new index doesn't exist, it will be created
  • Regarding instructions for CUMULUS-2020, you can now do a change index operation before a reindex operation. This will ensure that new data will end up in the new index while Elasticsearch is reindexing.

• CUMULUS-2351

  • Inventory report no longer includes the File/Granule relation object in the okCountByGranules key of a report. The information is only included when a 'Granule Not Found' report is run.

Removed

• CUMULUS-2367
  • Removed execution_cumulus_id column from granules RDS schema and data type

[v5.0.0] 2021-01-12

BREAKING CHANGES

• CUMULUS-2020
  • Elasticsearch data mappings have been updated to improve search and the API has been update to reflect those changes. See Migration notes on how to update the Elasticsearch mappings.

Migration notes

• CUMULUS-2020
  • Elasticsearch data mappings have been updated to improve search. For example, case insensitive searching will now work (e.g. 'MOD' and 'mod' will return the same granule results). To use the improved Elasticsearch queries, reindex to create a new index with the correct types. Then perform a change index operation to use the new index.
• CUMULUS-2258
  • Because the egress_lambda_log_group and egress_lambda_log_subscription_filter resource were removed from the cumulus module, new definitions for these resources must be added to cumulus-tf/main.tf. For reference on how to define these resources, see example/cumulus-tf/thin_egress_app.tf.
  • The tea_stack_name variable being passed into the cumulus module should be removed
• CUMULUS-2344
  • Regarding instructions for CUMULUS-2020, you can now do a change index operation before a reindex operation. This will ensure that new data will end up in the new index while Elasticsearch is reindexing.

BREAKING CHANGES

• CUMULUS-2020
  • Elasticsearch data mappings have been updated to improve search and the API has been updated to reflect those changes. See Migration notes on how to update the Elasticsearch mappings.

Added

• CUMULUS-2318
  • Addedasync_operation_image as cumulus module variable to allow for override of the async_operation container image. Users can optionally specify a non-default docker image for use with Core async operations.
• CUMULUS-2219
  • Added lzards-backup Core task to facilitate making LZARDS backup requests in Cumulus ingest workflows
• CUMULUS-2092
  • Add documentation for Granule Not Found Reports
• HYRAX-320
  • @cumulus/hyrax-metadata-updatesAdd component URI encoding for entry title id and granule ur to allow for values with special characters in them. For example, EntryTitleId 'Sentinel-6A MF/Jason-CS L2 Advanced Microwave Radiometer (AMR-C) NRT Geophysical Parameters' Now, URLs generated from such values will be encoded correctly and parsable by HyraxInTheCloud
• CUMULUS-1370
  • Add documentation for Getting Started section including FAQs
• CUMULUS-2092
  • Add documentation for Granule Not Found Reports
• CUMULUS-2219
  • Added lzards-backup Core task to facilitate making LZARDS backup requests in Cumulus ingest workflows
• CUMULUS-2280
  • In local api, retry to create tables if they fail to ensure localstack has had time to start fully.
• CUMULUS-2290
  • Add queryFields to granule schema, and this allows workflow tasks to add queryable data to granule record. For reference on how to add data to queryFields field, see example/cumulus-tf/kinesis_trigger_test_workflow.tf.
• CUMULUS-2318
  • Addedasync_operation_image as cumulus module variable to allow for override of the async_operation container image. Users can optionally specify a non-default docker image for use with Core async operations.

Changed

• CUMULUS-2020
  • Updated Elasticsearch mappings to support case-insensitive search
• CUMULUS-2124
  • cumulus-rds-tf terraform module now takes engine_version as an input variable.
• CUMULUS-2279
  • Changed the formatting of granule CMR links: instead of a link to the /search/granules.json endpoint, now it is a direct link to /search/concepts/conceptid.format
• CUMULUS-2296
  • Improved PDR spec compliance of parse-pdr by updating @cumulus/pvl to parse fields in a manner more consistent with the PDR ICD, with respect to numbers and dates. Anything not matching the ICD expectations, or incompatible with Javascript parsing, will be parsed as a string instead.
• CUMULUS-2344
  • Elasticsearch API now allows you to reindex to an index that already exists
  • If using the Change Index operation and the new index doesn't exist, it will be created

Removed

• CUMULUS-2258
  • Removed tea_stack_name variable from tf-modules/distribution/variables.tf and tf-modules/cumulus/variables.tf
  • Removed egress_lambda_log_group and egress_lambda_log_subscription_filter resources from tf-modules/distribution/main.tf

[v4.0.0] 2020-11-20

Migration notes

• Update the name of your cumulus_message_adapter_lambda_layer_arn variable for the cumulus module to cumulus_message_adapter_lambda_layer_version_arn. The value of the variable should remain the same (a layer version ARN of a Lambda layer for the cumulus-message-adapter.
• CUMULUS-2138 - Update all workflows using the MoveGranules step to add UpdateGranulesCmrMetadataFileLinksStepthat runs after it. See the example IngestAndPublishWorkflow for reference.
• CUMULUS-2251
  • Because it has been removed from the cumulus module, a new resource definition for egress_api_gateway_log_subscription_filter must be added to cumulus-tf/main.tf. For reference on how to define this resource, see example/cumulus-tf/main.tf.

Added

• CUMULUS-2248
  • Updates Integration Tests README to point to new fake provider template.
• CUMULUS-2239
  • Add resource declaration to create a VPC endpoint in tea-map-cache module if deploy_to_ngap is false.
• CUMULUS-2063
  • Adds a new, optional query parameter to the /collections[&getMMT=true] and /collections/active[&getMMT=true] endpoints. When a user provides a value of true for getMMT in the query parameters, the endpoint will search CMR and update each collection's results with new key MMTLink containing a link to the MMT (Metadata Management Tool) if a CMR collection id is found.
• CUMULUS-2170
  • Adds ability to filter granule inventory reports
• CUMULUS-2211
  • Adds granules/bulkReingest endpoint to @cumulus/api
• CUMULUS-2251
  • Adds log_api_gateway_to_cloudwatch variable to example/cumulus-tf/variables.tf.
  • Adds log_api_gateway_to_cloudwatch variable to thin_egress_app module definition.

Changed

• CUMULUS-2216
  • /collection and /collection/active endpoints now return collections without granule aggregate statistics by default. The original behavior is preserved and can be found by including a query param of includeStats=true on the request to the endpoint.
  • The es/collections Collection class takes a new parameter includeStats. It no longer appends granule aggregate statistics to the returned results by default. One must set the new parameter to any non-false value.
• CUMULUS-2201
  • Update dbIndexer lambda to process requests in serial
  • Fixes ingestPdrWithNodeNameSpec parsePdr provider error
• CUMULUS-2251
  • Moves Egress Api Gateway Log Group Filter from tf-modules/distribution/main.tf to example/cumulus-tf/main.tf

Fixed

• CUMULUS-2251
  • This fixes a deployment error caused by depending on the thin_egress_app module output for a resource count.

Removed

• CUMULUS-2251
  • Removes tea_api_egress_log_group variable from tf-modules/distribution/variables.tf and tf-modules/cumulus/variables.tf.

BREAKING CHANGES

• CUMULUS-2138 - CMR metadata update behavior has been removed from the move-granules task into a new update-granules-cmr-metadata-file-links task.
• CUMULUS-2216
  • /collection and /collection/active endpoints now return collections without granule aggregate statistics by default. The original behavior is preserved and can be found by including a query param of includeStats=true on the request to the endpoint. This is likely to affect the dashboard only but included here for the change of behavior.
• 1956
  • Update the name of the cumulus_message_adapter_lambda_layer_arn output from the cumulus-message-adapter module to cumulus_message_adapter_lambda_layer_version_arn. The output value has changed from being the ARN of the Lambda layer without a version to the ARN of the Lambda layer with a version.
  • Update the variable name in the cumulus and ingest modules from cumulus_message_adapter_lambda_layer_arn to cumulus_message_adapter_lambda_layer_version_arn

[v3.0.1] 2020-10-21

• CUMULUS-2203

  • Update Core tasks to use cumulus-message-adapter-js v2.0.0 to resolve memory leak/lambda ENOMEM constant failure issue. This issue caused lambdas to slowly use all memory in the run environment and prevented AWS from halting/restarting warmed instances when task code was throwing consistent errors under load.

• CUMULUS-2232

  • Updated versions for ajv, lodash, googleapis, archiver, and @cumulus/aws-client to remediate vulnerabilities found in SNYK scan.

Fixed

• CUMULUS-2233
  • Fixes /s3credentials bug where the expiration time on the cookie was set to a time that is always expired, so authentication was never being recognized as complete by the API. Consequently, the user would end up in a redirect loop and requests to /s3credentials would never complete successfully. The bug was caused by the fact that the code setting the expiration time for the cookie was expecting a time value in milliseconds, but was receiving the expirationTime from the EarthdataLoginClient in seconds. This bug has been fixed by converting seconds into milliseconds. Unit tests were added to test that the expiration time has been converted to milliseconds and checking that the cookie's expiration time is greater than the current time.

[v3.0.0] 2020-10-7

MIGRATION STEPS

• CUMULUS-2099

  • All references to meta.queues in workflow configuration must be replaced with references to queue URLs from Terraform resources. See the updated data cookbooks or example Discover Granules workflow configuration.
  • The steps for configuring queued execution throttling have changed. See the updated documentation.
  • In addition to the configuration for execution throttling, the internal mechanism for tracking executions by queue has changed. As a result, you should disable any rules or workflows scheduling executions via a throttled queue before upgrading. Otherwise, you may be at risk of having twice as many executions as are configured for the queue while the updated tracking is deployed. You can re-enable these rules/workflows once the upgrade is complete.

• CUMULUS-2111

  • Before you re-deploy your cumulus-tf module, note that the [thin-egress-app][thin-egress-app] is no longer deployed by default as part of the cumulus module, so you must add the TEA module to your deployment and manually modify your Terraform state to avoid losing your API gateway and impacting any Cloudfront endpoints pointing to those gateways. If you don't care about losing your API gateway and impacting Cloudfront endpoints, you can ignore the instructions for manually modifying state.

    1. Add the [thin-egress-app][thin-egress-app] module to your cumulus-tf deployment as shown in the Cumulus example deployment.

       • Note that the values for tea_stack_name variable to the cumulus module and the stack_name variable to the thin_egress_app module must match
       • Also, if you are specifying the stage_name variable to the thin_egress_app module, the value of the tea_api_gateway_stage variable to the cumulus module must match it

    2. If you want to preserve your existing thin-egress-app API gateway and avoid having to update your Cloudfront endpoint for distribution, then you must follow these instructions: https://nasa.github.io/cumulus/docs/upgrade-notes/migrate_tea_standalone. Otherwise, you can re-deploy as usual.

  • If you provide your own custom bucket map to TEA as a standalone module, you must ensure that your custom bucket map includes mappings for the protected and public buckets specified in your cumulus-tf/terraform.tfvars, otherwise Cumulus may not be able to determine the correct distribution URL for ingested files and you may encounter errors

• CUMULUS-2197

  • EMS resources are now optional, and ems_deploy is set to false by default, which will delete your EMS resources.
  • If you would like to keep any deployed EMS resources, add the ems_deploy variable set to true in your cumulus-tf/terraform.tfvars

BREAKING CHANGES

• CUMULUS-2200
  • Changes return from 303 redirect to 200 success for Granule Inventory's /reconciliationReport returns. The user (dashboard) must read the value of url from the return to get the s3SignedURL and then download the report.
• CUMULUS-2099
  • meta.queues has been removed from Cumulus core workflow messages.
  • @cumulus/sf-sqs-report workflow task no longer reads the reporting queue URL from input.meta.queues.reporting on the incoming event. Instead, it requires that the queue URL be set as the reporting_queue_url environment variable on the deployed Lambda.
• CUMULUS-2111
  • The deployment of the thin-egress-app module has be removed from tf-modules/distribution, which is a part of the tf-modules/cumulus module. Thus, the thin-egress-app module is no longer deployed for you by default. See the migration steps for details about how to add deployment for the thin-egress-app.
• CUMULUS-2141
  • The parse-pdr task has been updated to respect the NODE_NAME property in a PDR's FILE_GROUP. If a NODE_NAME is present, the task will query the Cumulus API for a provider with that host. If a provider is found, the output granule from the task will contain a provider property containing that provider. If NODE_NAME is set but a provider with that host cannot be found in the API, or if multiple providers are found with that same host, the task will fail.
  • The queue-granules task has been updated to expect an optional granule.provider property on each granule. If present, the granule will be enqueued using that provider. If not present, the task's config.provider will be used instead.
• CUMULUS-2197
  • EMS resources are now optional and will not be deployed by default. See migration steps for information about how to deploy EMS resources.

CODE CHANGES

• The @cumulus/api-client.providers.getProviders function now takes a queryStringParameters parameter which can be used to filter the providers which are returned
• The @cumulus/aws-client/S3.getS3ObjectReadStreamAsync function has been removed. It read the entire S3 object into memory before returning a read stream, which could cause Lambdas to run out of memory. Use @cumulus/aws-client/S3.getObjectReadStream instead.
• The @cumulus/ingest/util.lookupMimeType function now returns undefined rather than null if the mime type could not be found.
• The @cumulus/ingest/lock.removeLock function now returns undefined
• The @cumulus/ingest/granule.generateMoveFileParams function now returns source: undefined and target :undefined on the response object if either could not be determined. Previously, null had been returned.
• The @cumulus/ingest/recursion.recursion function must now be imported using const { recursion } = require('@cumulus/ingest/recursion');
• The @cumulus/ingest/granule.getRenamedS3File function has been renamed to listVersionedObjects
• @cumulus/common.http has been removed
• @cumulus/common/http.download has been removed

Added

• CUMULUS-1855
  • Fixed SyncGranule task to return an empty granules list when given an empty (or absent) granules list on input, rather than throwing an exception
• CUMULUS-1955
  • Added @cumulus/aws-client/S3.getObject to get an AWS S3 object
  • Added @cumulus/aws-client/S3.waitForObject to get an AWS S3 object, retrying, if necessary
• CUMULUS-1961
  • Adds startTimestamp and endTimestamp parameters to endpoint reconcilationReports. Setting these values will filter the returned report to cumulus data that falls within the timestamps. It also causes the report to be one directional, meaning cumulus is only reconciled with CMR, but not the other direction. The Granules will be filtered by their updatedAt values. Collections are filtered by the updatedAt time of their granules, i.e. Collections with granules that are updatedAt a time between the time parameters will be returned in the reconciliation reports.
  • Adds startTimestamp and endTimestamp parameters to create-reconciliation-reports lambda function. If either of these params is passed in with a value that can be converted to a date object, the inter-platform comparison between Cumulus and CMR will be one way. That is, collections, granules, and files will be filtered by time for those found in Cumulus and only those compared to the CMR holdings. For the moment there is not enough information to change the internal consistency check, and S3 vs Cumulus comparisons are unchanged by the timestamps.
• CUMULUS-1962
  • Adds location as parameter to /reconciliationReports endpoint. Options are S3 resulting in a S3 vs. Cumulus database search or CMR resulting in CMR vs. Cumulus database search.
• CUMULUS-1963
  • Adds granuleId as input parameter to /reconcilationReports endpoint. Limits inputs parameters to either collectionId or granuleId and will fail to create the report if both are provided. Adding granuleId will find collections in Cumulus by granuleId and compare those one way with those in CMR.
  • /reconciliationReports now validates any input json before starting the async operation and the lambda handler no longer validates input parameters.
• CUMULUS-1964
  • Reports can now be filtered on provider
• CUMULUS-1965
  • Adds collectionId parameter to the /reconcilationReports endpoint. Setting this value will limit the scope of the reconcilation report to only the input collectionId when comparing Cumulus and CMR. collectionId is provided an array of strings e.g. [shortname___version, shortname2___version2]
• CUMULUS-2107
  • Added a new task, update-cmr-access-constraints, that will set access constraints in CMR Metadata. Currently supports UMMG-JSON and Echo10XML, where it will configure AccessConstraints and RestrictionFlag/RestrictionComment, respectively.
  • Added an operator doc on how to configure and run the access constraint update workflow, which will update the metadata using the new task, and then publish the updated metadata to CMR.
  • Added an operator doc on bulk operations.
• CUMULUS-2111
  • Added variables to cumulus module:
    • tea_api_egress_log_group
    • tea_external_api_endpoint
    • tea_internal_api_endpoint
    • tea_rest_api_id
    • tea_rest_api_root_resource_id
    • tea_stack_name
  • Added variables to distribution module:
    • tea_api_egress_log_group
    • tea_external_api_endpoint
    • tea_internal_api_endpoint
    • tea_rest_api_id
    • tea_rest_api_root_resource_id
    • tea_stack_name
• CUMULUS-2112
  • Added @cumulus/api/lambdas/internal-reconciliation-report, so create-reconciliation-report lambda can create Internal reconciliation report
• CUMULUS-2116
  • Added @cumulus/api/models/granule.unpublishAndDeleteGranule which unpublishes a granule from CMR and deletes it from Cumulus, but does not update the record to published: false before deletion
• CUMULUS-2113
  • Added Granule not found report to reports endpoint
  • Update reports to return breakdown by Granule of files both in DynamoDB and S3
• CUMULUS-2123
  • Added cumulus-rds-tf DB cluster module to tf-modules that adds a serverless RDS Aurora/PostgreSQL database cluster to meet the PostgreSQL requirements for future releases.
  • Updated the default Cumulus module to take the following new required variables:
    • rds_user_access_secret_arn: AWS Secrets Manager secret ARN containing a JSON string of DB credentials (containing at least host, password, port as keys)
    • rds_security_group: RDS Security Group that provides connection access to the RDS cluster
  • Updated API lambdas and default ECS cluster to add them to the rds_security_group for database access
• CUMULUS-2126
  • The collections endpoint now writes to the RDS database
• CUMULUS-2127
  • Added migration to create collections relation for RDS database
• CUMULUS-2129
  • Added data-migration1 Terraform module and Lambda to migrate data from Dynamo to RDS
    • Added support to Lambda for migrating collections data from Dynamo to RDS
• CUMULUS-2155
  • Added rds_connection_heartbeat to cumulus and data-migration tf modules. If set to true, this diagnostic variable instructs Core's database code to fire off a connection 'heartbeat' query and log the timing/results for diagnostic purposes, and retry certain connection timeouts once. This option is disabled by default
• CUMULUS-2156
  • Support array inputs parameters for Internal reconciliation report
• CUMULUS-2157
  • Added support to data-migration1 Lambda for migrating providers data from Dynamo to RDS
    • The migration process for providers will convert any credentials that are stored unencrypted or encrypted with an S3 keypair provider to be encrypted with a KMS key instead
• CUMULUS-2161
  • Rules now support an executionNamePrefix property. If set, any executions triggered as a result of that rule will use that prefix in the name of the execution.
  • The QueueGranules task now supports an executionNamePrefix property. Any executions queued by that task will use that prefix in the name of the execution. See the example workflow for usage.
  • The QueuePdrs task now supports an executionNamePrefix config property. Any executions queued by that task will use that prefix in the name of the execution. See the example workflow for usage.
• CUMULUS-2162
  • Adds new report type to /reconciliationReport endpoint. The new report is Granule Inventory. This report is a CSV file of all the granules in the Cumulus DB. This report will eventually replace the existing granules-csv endpoint which has been deprecated.
• CUMULUS-2197
  • Added ems_deploy variable to the cumulus module. This is set to false by default, except for our example deployment, where it is needed for integration tests.

Changed

• Upgraded version of TEA deployed with Cumulus to build 88.
• CUMULUS-2107
  • Updated the applyWorkflow functionality on the granules endpoint to take a meta property to pass into the workflow message.
  • Updated the BULK_GRANULE functionality on the granules endpoint to support the above applyWorkflow change.
• CUMULUS-2111
  • Changed distribution_api_gateway_stage variable for cumulus module to tea_api_gateway_stage
  • Changed api_gateway_stage variable for distribution module to tea_api_gateway_stage
• CUMULUS-2224
  • Updated /reconciliationReport's file reconciliation to include "EXTENDED METADATA" as a valid CMR relatedUrls Type.

Fixed

• CUMULUS-2168
  • Fixed issue where large number of documents (generally logs) in the cumulus elasticsearch index results in the collection granule stats queries failing for the collections list api endpoint
• CUMULUS-1955
  • Due to AWS's eventual consistency model, it was possible for PostToCMR to publish an earlier version of a CMR metadata file, rather than the latest version created in a workflow. This fix guarantees that the latest version is published, as expected.
• CUMULUS-1961
  • Fixed activeCollections query only returning 10 results
• CUMULUS-2201
  • Fix Reconciliation Report integration test failures by waiting for collections appear in es list and ingesting a fake granule xml file to CMR
• CUMULUS-2015
  • Reduced concurrency of QueueGranules task. That task now has a config.concurrency option that defaults to 3.
• CUMULUS-2116
  • Fixed a race condition with bulk granule delete causing deleted granules to still appear in Elasticsearch. Granules removed via bulk delete should now be removed from Elasticsearch.
• CUMULUS-2163
  • Remove the public-read ACL from the move-granules task
• CUMULUS-2164
  • Fix issue where cumulus index is recreated and attached to an alias if it has been previously deleted
• CUMULUS-2195
  • Fixed issue with redirect from /token not working when using a Cloudfront endpoint to access the Cumulus API with Launchpad authentication enabled. The redirect should now work properly whether you are using a plain API gateway URL or a Cloudfront endpoint pointing at an API gateway URL.
• CUMULUS-2200
  • Fixed issue where __in and __not queries were stripping spaces from values

Deprecated

• CUMULUS-1955
  • @cumulus/aws-client/S3.getS3Object()
  • @cumulus/message/Queue.getQueueNameByUrl()
  • @cumulus/message/Queue.getQueueName()
• CUMULUS-2162
  • @cumulus/api/endpoints/granules-csv/list()

Removed

• CUMULUS-2111
  • Removed distribution_url and distribution_redirect_uri outputs from the cumulus module
  • Removed variables from the cumulus module:
    • distribution_url
    • log_api_gateway_to_cloudwatch
    • thin_egress_cookie_domain
    • thin_egress_domain_cert_arn
    • thin_egress_download_role_in_region_arn
    • thin_egress_jwt_algo
    • thin_egress_jwt_secret_name
    • thin_egress_lambda_code_dependency_archive_key
    • thin_egress_stack_name
  • Removed outputs from the distribution module:
    • distribution_url
    • internal_tea_api
    • rest_api_id
    • thin_egress_app_redirect_uri
  • Removed variables from the distribution module:
    • bucket_map_key
    • distribution_url
    • log_api_gateway_to_cloudwatch
    • thin_egress_cookie_domain
    • thin_egress_domain_cert_arn
    • thin_egress_download_role_in_region_arn
    • thin_egress_jwt_algo
    • thin_egress_jwt_secret_name
    • thin_egress_lambda_code_dependency_archive_key
• CUMULUS-2157
  • Removed providerSecretsMigration and verifyProviderSecretsMigration lambdas
• Removed deprecated @cumulus/sf-sns-report task
• Removed code:
  • @cumulus/aws-client/S3.calculateS3ObjectChecksum
  • @cumulus/aws-client/S3.getS3ObjectReadStream
  • @cumulus/cmrjs.getFullMetadata
  • @cumulus/cmrjs.getMetadata
  • @cumulus/common/util.isNil
  • @cumulus/common/util.isNull
  • @cumulus/common/util.isUndefined
  • @cumulus/common/util.lookupMimeType
  • @cumulus/common/util.mkdtempSync
  • @cumulus/common/util.negate
  • @cumulus/common/util.noop
  • @cumulus/common/util.omit
  • @cumulus/common/util.renameProperty
  • @cumulus/common/util.sleep
  • @cumulus/common/util.thread
  • @cumulus/ingest/granule.copyGranuleFile
  • @cumulus/ingest/granule.moveGranuleFile
  • @cumulus/integration-tests/api/rules.deleteRule
  • @cumulus/integration-tests/api/rules.getRule
  • @cumulus/integration-tests/api/rules.listRules
  • @cumulus/integration-tests/api/rules.postRule
  • @cumulus/integration-tests/api/rules.rerunRule
  • @cumulus/integration-tests/api/rules.updateRule
  • @cumulus/integration-tests/sfnStep.parseStepMessage
  • @cumulus/message/Queue.getQueueName
  • @cumulus/message/Queue.getQueueNameByUrl