@cumulus/common - npm Package Versions

[v18.2.0] 2024-02-02

Migration Notes

From this release forward, Cumulus Core will be tested against PostgreSQL v13. Users should migrate their datastores to Aurora PostgreSQL 13.9+ compatible data stores as soon as possible after upgrading to this release.

Database Upgrade

Users utilizing the cumulus-rds-tf module should reference cumulus-rds-tf upgrade instructions.

Breaking Changes

• CUMULUS-2889
  • Removed unused CloudWatch Logs AWS SDK client. This change removes the CloudWatch Logs client from the @cumulus/aws-client package.
• CUMULUS-2890
  • Removed unused CloudWatch AWS SDK client. This change removes the CloudWatch client from the @cumulus/aws-client package.

Changed

• CUMULUS-3492
  • add teclark to select-stack.js
• CUMULUS-3444
  • Update cumulus-rds-tf module to take additional parameters in support of migration from Aurora PostgreSQl v11 to v13. See Migration Notes for more details
• CUMULUS-3564
  • Update webpack configuration to explicitly disable chunking
• CUMULUS-2895
  • Updated KMS code to aws sdk v3
• CUMULUS-2888
  • Update CloudWatch Events code to AWS SDK v3
• CUMULUS-2893
  • Updated Kinesis code to AWS SDK v3
• CUMULUS-3555
  • Revert 3540, un-stubbing cmr facing tests
  • Raise memory_size of ftpPopulateTestLambda to 512MB
• CUMULUS-2887
  • Updated CloudFormation code to aws sdk v3
• CUMULUS-2899
  • Updated SNS code to aws sdk v3
• CUMULUS_3499
  • Update AWS-SDK dependency pin to "2.1490" to prevent SQS issue. Dependency pin expected to be changed with the resolution to CUMULUS-2900
• CUMULUS-2894
  • Update Lambda code to AWS SDK v3
• CUMULUS-3432
  • Update cumulus-rds-tf engine_version to 13.9
  • Update cumulus-rds-tf parameter_group_family to aurora-postgresql13
  • Update development/local stack postgres image version to postgres:13.9-alpine
• CUMULUS-2900
  • Update SQS code to AWS SDK v3
• CUMULUS-3352
  • Update example project to use CMA v2.0.3 for integration testing
  • Update example deployment to deploy cnmResponse lambda version 2.1.1-aplha.2-SNAPSHOT
  • Update example deployment to deploy cnmToGranule lambda version 1.7.0-alpha.2-SNAPSHOT
• CUMULUS-3501
  • Updated CreateReconciliationReport lambda to save report record to Elasticsearch.
  • Created docker image cumuluss/async-operation:48 from v16.1.2, and used it as default async_operation_image.
• CUMULUS-3502
  • Upgraded localstack to v3.0.0 to support recent aws-sdk releases and update unit tests.
• CUMULUS-3540
  • stubbed cmr interfaces in integration tests allow integration tests to pass
  • needed while cmr is failing to continue needed releases and progress
  • this change should be reverted ASAP when cmr is working as needed again

Fixed

• CUMULUS-3177
  • changed _removeGranuleFromCmr function for granule bulkDelete to not throw an error and instead catch the error when the granule is not found in CMR
• CUMULUS-3293
  • Process Dead Letter Archive is fixed to properly copy objects from /sqs/ to /failed-sqs/ location
• CUMULUS-3467
  • Added childWorkflowMeta to QueueWorkflow task configuration
• CUMULUS-3474
  • Fixed overridden changes to rules.buildPayload' to restore changes from ticket CUMULUS-2969which limited the definition object tonameandarn` to account for AWS character limits.
• CUMULUS-3479
  • Fixed typo in s3-replicator resource declaration where var.lambda_memory_size is supposed to be var.lambda_memory_sizes
• CUMULUS-3510
  • Fixed @cumulus/api validateAndUpdateSqsRule method to allow 0 retries and 0 visibilityTimeout in rule's meta. This fix from CUMULUS-2863 was not in release 16 and later.
• CUMULUS-3562
  • updated crypto-js to 4.2.0
  • updated aws-sdk/client-api-gateway to 3.499 to avoid older crypto-js dependency

[v16.1.3] 2024-1-15

Please note changes in 16.1.3 may not yet be released in future versions, as this is a backport/patch release on the 16.x series of releases. Updates that are included in the future will have a corresponding CHANGELOG entry in future releases.

Changed

• **CUMULUS_3499
  • Update AWS-SDK dependency pin to "2.1490" to prevent SQS issue. Dependency pin expected to be changed with the resolution to CUMULUS-2900

Fixed

• CUMULUS-3474
  • Fixed overriden changes to rules.buildPayload' to restore changes from ticket CUMULUS-2969which limited the definition object tonameandarn` to account for AWS character limits.
• CUMULUS-3501
  • Updated CreateReconciliationReport lambda to save report record to Elasticsearch.
  • Created docker image cumuluss/async-operation:48 from v16.1.2, and used it as default async_operation_image.
• CUMULUS-3510
  • Fixed @cumulus/api validateAndUpdateSqsRule method to allow 0 retries and 0 visibilityTimeout in rule's meta. This fix from CUMULUS-2863 was not in release 16 and later.
• CUMULUS-3540
  • stubbed cmr interfaces in integration tests allow integration tests to pass
  • needed while cmr is failing to continue needed releases and progress
  • this change should be reverted ASAP when cmr is working as needed again

[v16.1.2] 2023-11-01

Please note changes in 16.1.2 may not yet be released in future versions, as this is a backport/patch release on the 16.x series of releases. Updates that are included in the future will have a corresponding CHANGELOG entry in future releases.

Added

• CUMULUS-3218
  • Added optional maxDownloadTime field to provider schema
  • Added max_download_time column to PostgreSQL providers table
  • Updated @cumulus/ingest/lock to check expired locks based on provider.maxDownloadTime

Fixed

• @aws-sdk upgrade
  • Fixed TS compilation error on aws-client package caused by @aws-sdk/client-dynamodb 3.433.0 upgrade
  • Updated mapping for collection Elasticsearch records to prevent dynamic field for keys under meta.
• CUMULUS-3286
  • Fixed @cumulus/cmrjs/cmr-utils/getGranuleTemporalInfo and @cumulus/message/Granules/getGranuleCmrTemporalInfo to handle non-existing cmr file.
  • Updated mapping for granule and deletedgranule Elasticsearch records to prevent dynamic field for keys under queryFields.
• CUMULUS-3293
  • Process Dead Letter Archive is fixed to properly copy objects from /sqs/ to /failed-sqs/ location
• CUMULUS-3393
  • Fixed PUT collection endpoint to update collection configuration in S3.
• CUMULUS-3467
  • Added childWorkflowMeta to QueueWorkflow task configuration

[v18.1.0] 2023-10-25

MIGRATION notes

Rules API Endpoint Versioning

As part of the work on CUMULUS-3095, we have added a required header for the rules PUT/PATCH endpoints -- to ensure that older clients/utilities do not unexpectedly make destructive use of those endpoints, a validation check of a header value against supported versions has been implemented.

Moving forward, if a breaking change is made to an existing endpoint that requires user updates, as part of that update we will set the current version of the core API and require a header that confirms the client is compatible with the version required or greater.

In this instance, the rules PUT/PATCH endpoints will require a Cumulus-API-Version value of at least 2.

 curl --request PUT https://example.com/rules/repeat_test\
 --header 'Cumulus-API-Version: 2'\
 --header 'Content-Type: application/json'\
 --header 'Authorization: Bearer ReplaceWithToken'\
 --data ...

Users/clients that do not make use of these endpoints will not be impacted.

Breaking Changes

• CUMULUS-3427
  • Changed the naming conventions for memory size and timeouts configuration to simply the lambda name

Notable Changes

• CUMULUS-3095
  • Added PATCH rules endpoint to update rule which works as the existing PUT endpoint.
  • Updated PUT rules endpoint to replace rule.

Added

• CUMULUS-3218
  • Added optional maxDownloadTime field to provider schema
  • Added max_download_time column to PostgreSQL providers table
  • Updated @cumulus/ingest/lock to check expired locks based on provider.maxDownloadTime

Changed

• CUMULUS-3095
  • Updated @cumulus/api-client/rules to havereplaceRule and updateRule methods.
  • Updated mapping for rule Elasticsearch records to prevent dynamic field for keys under meta and payload, and fixed rule field mapping.
• CUMULUS-3351
  • Updated constructOnlineAccessUrls() to group CMR online access URLs by link type.
• CUMULUS-3377
  • Added configuration option to cumulus-tf/terraform.tfvars to include sns:Subscribe access policy for executions, granules, collections, and PDRs report topics.
• CUMULUS-3392
  • Modify cloudwatch rule by deleting custom
• CUMULUS-3434
  • Updated @cumulus/orca-recovery-adapter task to output both input granules and recovery output.
  • Updated example/cumulus-tf/orca.tf to use v9.0.0.

Fixed

• CUMULUS-3095
  • Added back rule schema validation which is missing after RDS phase 3.
  • Fixed a bug for creating rule with tags.
• CUMULUS-3286
  • Fixed @cumulus/cmrjs/cmr-utils/getGranuleTemporalInfo and @cumulus/message/Granules/getGranuleCmrTemporalInfo to handle non-existing cmr file.
  • Updated mapping for granule and deletedgranule Elasticsearch records to prevent dynamic field for keys under queryFields.
  • Updated mapping for collection Elasticsearch records to prevent dynamic field for keys under meta.
• CUMULUS-3393
  • Fixed PUT collection endpoint to update collection configuration in S3.
• CUMULUS-3427
  • Fixed issue where some lambda and task memory sizes and timeouts were not configurable
• @aws-sdk upgrade
  • Fixed TS compilation error on aws-client package caused by @aws-sdk/client-dynamodb 3.433.0 upgrade

[v18.0.0] 2023-08-28

Notable Changes

• CUMULUS-3270
  • update python lambdas to use python3.10
  • update dependencies to use python3.10 including cumulus-message-adapter, cumulus-message-adapter-python and cumulus-process-py
• CUMULUS-3259
  • Updated Terraform version from 0.13.6 to 1.5.3. Please see the instructions to upgrade your deployments.

Changed

• CUMULUS-3366
  • Added logging to the collectionRuleMatcher Rules Helper, which is used by the sqs-message-consumer and message-consumer Lambdas, to report when an incoming message's collection does not match any rules.

[v17.0.0] 2023-08-09

MIGRATION notes

• This release updates the hashicorp/aws provider required by Cumulus to ~> 5.0 which in turn requires updates to all modules deployed with Core in the same stack to use a compatible provider version.
• This update is not compatible with prior stack states - Terraform will not allow redeployment of a prior version of Cumulus using an older version of the provider. Please be sure to validate the install changeset is what you expect prior to upgrading to this version.
• Upgrading Cumulus to v17 from prior versions should only require the usual terraform init/apply steps. As always be sure to inspect the terraform plan or terraform apply changeset to ensure the changes between providers are what you're expecting for all modules you've chosen to deploy with Cumulus

Notable Changes

• CUMULUS-3258
  • @cumulus/api is now compatible only with Orca >= 8.1.0. Prior versions of Orca are not compatible with Cumulus 17+
  • Updated all hashicorp terraform AWS provider configs to ~> 5.0
    • Upstream/downstream terraform modules will need to utilize an AWS provider that matches this range

Breaking Changes

• CUMULUS-3258

  • Update @cumulus/api/lib/orca/getOrcaRecoveryStatusByGranuleCollection to @cumulus/api/lib/orca/getOrcaRecoveryStatusByGranuleIdAndCollection and add collectionId to arguments to support Orca v8+ required use of collectionId

  • Updated all terraform AWS providers to ~> 5.0

Changed

• CUMULUS-3258
  • Update all Core integration tests/integrations to be compatible with Orca >= v8.1.0 only

Fixed

• CUMULUS-3319
  • Removed @cumulus/api/models/schema and changed all references to @cumulus/api/lib/schema in docs and related models
  • Removed @cumulus/api/models/errors.js
  • Updated API granule write logic to cause postgres schema/db write failures on an individual granule file write to result in a thrown error/400 return instead of a 200 return and a 'silent' update of the granule to failed status.
  • Update api/lib/_writeGranule/_writeGranulefiles logic to allow for schema failures on individual granule writes via an optional method parameter in _writeGranules, and an update to the API granule write calls.
  • Updated thrown error to include information related to automatic failure behavior in addition to the stack trace.

[v16.1.1] 2023-08-03

Notable Changes

• The async_operation_image property of cumulus module should be updated to pull the ECR image for cumuluss/async-operation:47

Added

• CUMULUS-3298
  • Added extra time to the buffer for replacing the launchpad token before it expires to alleviate CMR error messages
• CUMULUS-3220
  • Created a new send-pan task
• CUMULUS-3287
  • Added variable to allow the aws_ecs_task_definition health check to be configurable.
  • Added clarity to how the bucket field needs to be configured for the move-granules task definition

Changed

• Security upgrade node from 14.19.3-buster to 14.21.1-buster
• CUMULUS-2985
  • Changed onetime rules RuleTrigger to only execute when the state is ENABLED and updated documentation to reflect the change
  • Changed the invokeRerun function to only re-run enabled rules
• CUMULUS-3188
  • Updated QueueGranules to support queueing granules that meet the required API granule schema.
  • Added optional additional properties to queue-granules input schema
• CUMULUS-3252
  • Updated example/cumulus-tf/orca.tf to use orca v8.0.1
  • Added cumulus task @cumulus/orca-copy-to-archive-adapter, and add the task to tf-modules/ingest
  • Updated tf-modules/cumulus module to take variable orca_lambda_copy_to_archive_arn and pass to tf-modules/ingest
  • Updated example/cumulus-tf/ingest_and_publish_granule_with_orca_workflow.tf CopyToGlacier (renamed to CopyToArchive) step to call orca_copy_to_archive_adapter_task
• CUMULUS-3253
  • Added cumulus task @cumulus/orca-recovery-adapter, and add the task to tf-modules/ingest
  • Updated tf-modules/cumulus module to take variable orca_sfn_recovery_workflow_arn and pass to tf-modules/ingest
  • Added example/cumulus-tf/orca_recovery_adapter_workflow.tf, OrcaRecoveryAdapterWorkflow workflow has OrcaRecoveryAdapter task to call the ORCA recovery step-function.
  • Updated example/data/collections/ collection configuration meta.granuleRecoveryWorkflow to use OrcaRecoveryAdapterWorkflow
• CUMULUS-3215
  • Create reconciliation reports will properly throw errors and set the async operation status correctly to failed if there is an error.
  • Knex calls relating to reconciliation reports will retry if there is a connection terminated unexpectedly error
  • Improved logging for async operation
  • Set default async_operation_image_version to 47
• CUMULUS-3024
  • Combined unit testing of @cumulus/api/lib/rulesHelpers to a single test file api/tests/lib/test-rulesHelpers and removed extraneous test files.
• CUMULUS-3209
  • Apply brand color with high contrast settings for both (light and dark) themes.
  • Cumulus logo can be seen when scrolling down.
  • "Back to Top" button matches the brand color for both themes.
  • Update "note", "info", "tip", "caution", and "warning" components to new admonition styling.
  • Add updated arch diagram for both themes.
• CUMULUS-3203
  • Removed ACL setting of private on S3.multipartCopyObject() call
  • Removed ACL setting of private for s3PutObject()
  • Removed ACL confguration on sync-granules task
  • Update documentation on dashboard deployment to exclude ACL public-read setting
• CUMULUS-3245
  • Update SQS consumer logic to catch ExecutionAlreadyExists error and delete SQS message accordingly.
  • Add ReportBatchItemFailures to event source mapping start_sf_mapping
• CUMULUS-3357
  • @cumulus/queue-granules is now written in TypeScript
  • @cumulus/schemas can now generate TypeScript interfaces for the task input, output and config.
• Added missing name to throttle_queue_watcher Cloudwatch event in throttled-queue.tf

Fixed

• CUMULUS-3258
  • Fix un-prefixed s3 lifecycle configuration ID from CUMULUS-2915
• CUMULUS-2625
  • Optimized heap memory and api load in queue-granules task to scale to larger workloads.
• CUMULUS-3265
  • Fixed @cumulus/api getGranulesForPayload function to query cloud metrics es when needed.
• CUMULUS-3389
  • Updated runtime of send-pan and startAsyncOperation lambdas to nodejs16.x