@cumulus/cmrjs - npm Package Versions

[v18.4.0] 2024-08-16

Migration Notes

CUMULUS-3320 Update executions table

The work for CUMULUS-3320 required index updates as well as a modification of a table constraint. To install the update containing these changes you should:

• Pre-generate the indexes on the execution table. This can be done via manual procedure prior to upgrading without downtime, or done more quickly before or during upgrade with downtime.
• Update the executions_parent_cumulus_id_foreign constraint. This will require downtime as updating the constraint requires a table write lock, and the update may take some time.

Deployments with low volume databases and low activity and/or test/development environments should be able to install these updates via the normal automatic Cumulus deployment process.

Please carefully review the migration process documentation. Failure to make these updates properly will likely result in deployment failure and/or degraded execution table operations.

CUMULUS-3449 Please follow the instructions before upgrading Cumulus

• The updates in CUMULUS-3449 requires manual update to postgres database in production environment. Please follow Update Cumulus_id Type and Indexes

Breaking Changes

Added

• CUMULUS-3320
  • Added endpoint /executions/bulkDeleteExecutionsByCollection to allow bulk deletion of executions from elasticsearch by collectionId
  • Added Bulk Execution Delete migration type to async operations types
• CUMULUS-3608
  • Exposes variables for sqs_message_consumer_watcher messageLimit and timeLimit configurations. Descriptions of the variables here include notes on usage and what users should consider if configuring something other than the default values.
• CUMULUS-3449
  • Updated the following database columns to BIGINT: executions.cumulus_id, executions.parent_cumulus_id, files.granule_cumulus_id, granules_executions.granule_cumulus_id, granules_executions.execution_cumulus_id and pdrs.execution_cumulus_id
  • Changed granules table unique constraint to granules_collection_cumulus_id_granule_id_unique
  • Added indexes granules_granule_id_index and granules_provider_collection_cumulus_id_granule_id_index to granules table

Changed

• CUMULUS-3320
  • Updated executions table (please see Migration section and Upgrade Instructions for more information) to:
    • Add index on collection_cumulus_id
    • Add index on parent_cumulus_id
    • Update executions_parent_cumulus_id_foreign constraint to add ON DELETE SET NULL. This change will cause deletions in the execution table to allow deletion of parent executions, when this occurs the child will have it's parent reference set to NULL as part of the deletion operations.
• CUMULUS-3449
  • Updated @cumulus/db package and configure knex hook postProcessResponse to convert the return string from columns ending with "cumulus_id" to number.
• CUMULUS-3841
  • Increased fetchRules page size to default to 100 instead of 10. This improves overall query time when fetching all rules such as in sqsMessageConsumer.

Fixed

• CUMULUS-3817
  • updated applicable @aws-sdk dependencies to 3.621.0 to remove inherited vulnerability from fast-xml-parser
• CUMULUS-3320
  • Execution database deletions by cumulus_id should have greatly improved performance as a table scan will no longer be required for each record deletion to validate parent-child relationships
• CUMULUS-3818
  • Fixes default value (updated to tag 52) for async-operation-image in tf-modules/cumulus.
• CUMULUS-3840
  • Fixed @cumulus/api/bin/serve to correctly use EsClient.

[v18.3.3] 2024-08-09

Please note changes in v18.3.3 may not yet be released in future versions, as this is a backport/patch release on the v18.3.x series of releases. Updates that are included in the future will have a corresponding CHANGELOG entry in future releases.

Fixed

• CUMULUS-3824
  • Changed the ECS docker storage driver to overlay2, since devicemapper is removed in Docker Engine v25.0.
  • Removed ecs_docker_storage_driver property from cumulus module.
• CUMULUS-3836
  • Terraform configuration for cleanExecutions now correctly configures ES_HOST and lambda security group

[v18.3.2] 2024-07-24

Added

• CUMULUS-3700
  • Added volume_type option to elasticsearch_config in the data-persistance module to allow configuration of the EBS volume type for Elasticsarch; default remains gp2.
• CUMULUS-3424
  • Exposed auto_pause and seconds_until_auto_pause variables in cumulus-rds-tf module to modify aws_rds_cluster scaling_configuration
• CUMULUS-3760
  • Added guidance for handling large backlog of es executions
• CUMULUS-3742
  • Script for dumping data into postgres database for testing and replicating issues
• CUMULUS-3385
  • Added generate_db_executions to dump large scale postgres executions

Changed

• CUMULUS-3385
  • updated cleanExecutions lambda to clean up postgres execution payloads
  • updated cleanExecutions lambda with configurable limit to control for large size
• NDCUM-1051
  • Modified addHyraxUrlToUmmG to test whether the provide Hyrax URL is already included in the metadata, and if so return the metadata unaltered.
  • Modified addHyraxUrlToEcho10 to test whether the provide Hyrax URL is already included in the metadata, and if so return the metadata unaltered.

Fixed

• CUMULUS-3807
  • Pinned @aws-sdk/client-s3 to 3.614 to address timeout/bug in s3().listObjectsV2
• CUMULUS-3787
  • Fixed developer-side bug causing some ts errors to be swallowed in CI
• CUMULUS-3785
  • Fixed SftpProviderClient not awaiting decryptBase64String with AWS KMS
  • Fixed method typo in @cumulus/api/endpoints/dashboard.js
• CUMULUS-3385
  • fixed cleanExecutions lambda to clean up elasticsearch execution payloads
• CUMULUS-3326
  • Updated update-granules-cmr-metadata-file-links task to update the file size of the update metadata file and remove the invalidated checksum associated with this file.

[v18.3.1] 2024-07-08

Migration Notes

[v18.2.2] 2024-06-4

Migration Notes

CUMULUS-3591 - SNS topics set to use encrypted storage

As part of the requirements for this ticket Cumulus Core created SNS topics are being updated to use server-side encryption with an AWS managed key. No user action is required, this note is being added to increase visibility re: this modification.

Changed

• CUMULUS-3591
  • Enable server-side encryption for all SNS topcis deployed by Cumulus Core
  • Update all integration/unit tests to use encrypted SNS topics

Fixed

• CUMULUS-3547
  • Updated ECS Cluster /dev/xvdcz EBS volumes so they're encrypted.
• CUMULUS-3527
  • Added suppport for additional kex algorithms in the sftp-client.
• CUMULUS-3587
  • Ported https://github.com/scottcorgan/express-boom into API/lib to allow updates of sub-dependencies and maintain without refactoring errors in API/etc wholesale
  • Addresses CVE-2020-36604
• CUMULUS-3673
  • Fixes Granules API so that paths containing a granule and/or collection ID properly URI encode the ID.
• Audit Issues
  • Addressed CVE-2023-45133 by updating babel packages and .babelrc

[v18.2.1] 2024-05-08

Please note changes in 18.2.1 may not yet be released in future versions, as this is a backport/patch release on the 18.2.x series of releases. Updates that are included in the future will have a corresponding CHANGELOG entry in future releases.

Fixed

• CUMULUS-3721
  • Update lambda:GetFunctionConfiguration policy statement to fix error related to resource naming
• CUMULUS-3701
  • Updated @cumulus/api to no longer improperly pass PATCH/PUT null values to Eventbridge rules

[v18.2.0] 2024-02-02

Migration Notes

From this release forward, Cumulus Core will be tested against PostgreSQL v13. Users should migrate their datastores to Aurora PostgreSQL 13.9+ compatible data stores as soon as possible after upgrading to this release.

Database Upgrade

Users utilizing the cumulus-rds-tf module should reference cumulus-rds-tf upgrade instructions.

Breaking Changes

• CUMULUS-2889
  • Removed unused CloudWatch Logs AWS SDK client. This change removes the CloudWatch Logs client from the @cumulus/aws-client package.
• CUMULUS-2890
  • Removed unused CloudWatch AWS SDK client. This change removes the CloudWatch client from the @cumulus/aws-client package.

Changed

• CUMULUS-3492
  • add teclark to select-stack.js
• CUMULUS-3444
  • Update cumulus-rds-tf module to take additional parameters in support of migration from Aurora PostgreSQl v11 to v13. See Migration Notes for more details
• CUMULUS-3564
  • Update webpack configuration to explicitly disable chunking
• CUMULUS-2895
  • Updated KMS code to aws sdk v3
• CUMULUS-2888
  • Update CloudWatch Events code to AWS SDK v3
• CUMULUS-2893
  • Updated Kinesis code to AWS SDK v3
• CUMULUS-3555
  • Revert 3540, un-stubbing cmr facing tests
  • Raise memory_size of ftpPopulateTestLambda to 512MB
• CUMULUS-2887
  • Updated CloudFormation code to aws sdk v3
• CUMULUS-2899
  • Updated SNS code to aws sdk v3
• CUMULUS_3499
  • Update AWS-SDK dependency pin to "2.1490" to prevent SQS issue. Dependency pin expected to be changed with the resolution to CUMULUS-2900
• CUMULUS-2894
  • Update Lambda code to AWS SDK v3
• CUMULUS-3432
  • Update cumulus-rds-tf engine_version to 13.9
  • Update cumulus-rds-tf parameter_group_family to aurora-postgresql13
  • Update development/local stack postgres image version to postgres:13.9-alpine
• CUMULUS-2900
  • Update SQS code to AWS SDK v3
• CUMULUS-3352
  • Update example project to use CMA v2.0.3 for integration testing
  • Update example deployment to deploy cnmResponse lambda version 2.1.1-aplha.2-SNAPSHOT
  • Update example deployment to deploy cnmToGranule lambda version 1.7.0-alpha.2-SNAPSHOT
• CUMULUS-3501
  • Updated CreateReconciliationReport lambda to save report record to Elasticsearch.
  • Created docker image cumuluss/async-operation:48 from v16.1.2, and used it as default async_operation_image.
• CUMULUS-3502
  • Upgraded localstack to v3.0.0 to support recent aws-sdk releases and update unit tests.
• CUMULUS-3540
  • stubbed cmr interfaces in integration tests allow integration tests to pass
  • needed while cmr is failing to continue needed releases and progress
  • this change should be reverted ASAP when cmr is working as needed again

Fixed

• CUMULUS-3177
  • changed _removeGranuleFromCmr function for granule bulkDelete to not throw an error and instead catch the error when the granule is not found in CMR
• CUMULUS-3293
  • Process Dead Letter Archive is fixed to properly copy objects from /sqs/ to /failed-sqs/ location
• CUMULUS-3467
  • Added childWorkflowMeta to QueueWorkflow task configuration
• CUMULUS-3474
  • Fixed overridden changes to rules.buildPayload' to restore changes from ticket CUMULUS-2969which limited the definition object tonameandarn` to account for AWS character limits.
• CUMULUS-3479
  • Fixed typo in s3-replicator resource declaration where var.lambda_memory_size is supposed to be var.lambda_memory_sizes
• CUMULUS-3510
  • Fixed @cumulus/api validateAndUpdateSqsRule method to allow 0 retries and 0 visibilityTimeout in rule's meta. This fix from CUMULUS-2863 was not in release 16 and later.
• CUMULUS-3562
  • updated crypto-js to 4.2.0
  • updated aws-sdk/client-api-gateway to 3.499 to avoid older crypto-js dependency