@cumulus/test-data - npm Package Versions

published 9.9.2 • 3 years ago

Changelog

[v9.9.2] 2021-02-10 [BACKPORT]

Please note changes in 9.9.2 may not yet be released in future versions, as this is a backport and patch release on the 9.9.x series of releases. Updates that are included in the future will have a corresponding CHANGELOG entry in future releases.### Added

CUMULUS-2775
- Added a configurable parameter group for the RDS serverless database cluster deployed by tf-modules/rds-cluster-tf. The allowed parameters for the parameter group can be found in the AWS documentation of allowed parameters for an Aurora PostgreSQL cluster. By default, the following parameters are specified:
  - shared_preload_libraries: pg_stat_statements,auto_explain
  - log_min_duration_statement: 250
  - auto_explain.log_min_duration: 250
CUMULUS-2840
- Added an index on granule_cumulus_id to the RDS files table.

Changed

CUMULUS-2847
- Move DyanmoDb table name into API keystore and initialize only on lambda cold start
CUMULUS-2781
- Add api_config secret to hold API/Private API lambda configuration values
CUMULUS-2775
- Changed the timeout_action to ForceApplyCapacityChange by default for the RDS serverless database cluster tf-modules/rds-cluster-tf

9.4.1

published 9.4.1 • 3 years ago

Changelog

[v9.4.1] 2022-02-14 [BACKPORT]

Please note changes in 9.4.1 may not yet be released in future versions, as this is a backport and patch release on the 9.4.x series of releases. Updates that are included in the future will have a corresponding CHANGELOG entry in future releases.

CUMULUS-2847
- Update dynamo configuration to read from S3 instead of System Manager Parameter Store
- Move api configuration initialization outside the lambda handler to eliminate unneded S3 calls/require config on cold-start only
- Moved ssh2 package from @cumulus/common to @cumulus/sftp-client and upgraded package from ^0.8.7 to ^1.0.0 to address security vulnerability issue in previous version.
- Fixed hyrax task package.json dev dependency
- Update CNM lambda dependencies for Core tasks
  - cumulus-cnm-response-task: 1.4.4
  - cumulus-cnm-to-granule: 1.5.4
- Whitelist ssh2 re: https://github.com/advisories/GHSA-652h-xwhf-q4h6

9.9.1

published 9.9.1 • 3 years ago

Changelog

[v9.9.1] 2021-02-10 [BACKPORT]

Please note changes in 9.9.1 may not yet be released in future versions, as this is a backport and patch release on the 9.9.x series of releases. Updates that are included in the future will have a corresponding CHANGELOG entry in future releases.

Fixed

CUMULUS-2775
- Updated @cumulus/api-client to not log an error for 201 response from updateGranule

Changed

Updated version of @cumulus/cumulus-message-adapter-js from 2.0.3 to 2.0.4 for all Cumulus workflow tasks
CUMULUS-2775
- Changed @cumulus/api-client/invokeApi() to accept a single accepted status code or an array of accepted status codes via expectedStatusCodes
CUMULUS-2837
- Update process-s3-dead-letter-archive to unpack SQS events in addition to Cumulus Messages
- Update process-s3-dead-letter-archive to look up execution status using getCumulusMessageFromExecutionEvent (common method with sfEventSqsToDbRecords)
- Move methods in api/lib/cwSfExecutionEventUtils to @cumulus/message/StepFunctions

10.0.1

published 10.0.1 • 3 years ago

Changelog

[v10.0.1] 2022-02-03

Fixed

Fixed IAM permissions issue with <prefix>-postgres-migration-async-operation Lambda which prevented it from running a Fargate task for data migration.

10.0.0

published 10.0.0 • 3 years ago

Changelog

[v10.0.0] 2022-02-01

Migration steps

Please read the documentation on the updates to the granule files schema for our Cumulus workflow tasks and how to upgrade your deployment for compatibility.
(Optional) Update the task-config for all workflows that use the sync-granule task to include workflowStartTime set to {$.cumulus_meta.workflow_start_time}. See here for an example.

BREAKING CHANGES

NDCUM-624
- Functions in @cumulus/cmrjs renamed for consistency with isCMRFilename and isCMRFile
  - isECHO10File -> isECHO10Filename
  - isUMMGFile -> isUMMGFilename
  - isISOFile -> isCMRISOFilename
CUMULUS-2388
- In order to standardize task messaging formats, please note the updated input, output and config schemas for the following Cumulus workflow tasks:
  - add-missing-file-checksums
  - files-to-granules
  - hyrax-metadata-updates
  - lzards-backup
  - move-granules
  - post-to-cmr
  - sync-granule
  - update-cmr-access-constraints
  - update-granules-cmr-metadata-file-links The primary focus of the schema updates was to standardize the format of granules, and particularly their files data. The granule files object now matches the file schema in the Cumulus database and thus also matches the files object produced by the API with use cases like applyWorkflow. This includes removal of name and filename in favor of bucket and key, removal of certain properties such as etag and duplicate_found and outputting them as separate objects stored in meta.
- Checksum values calculated by @cumulus/checksum are now converted to string to standardize checksum formatting across the Cumulus library.

Notable changes

CUMULUS-2718
- The sync-granule task has been updated to support an optional configuration parameter workflowStartTime. The output payload of sync-granule now includes a createdAt time for each granule which is set to the provided workflowStartTime or falls back to Date.now() if not provided. Workflows using sync-granule may be updated to include this parameter with the value of {$.cumulus_meta.workflow_start_time} in the task_config.
Updated version of @cumulus/cumulus-message-adapter-js from 2.0.3 to 2.0.4 for all Cumulus workflow tasks
CUMULUS-2783
- A bug in the ECS cluster autoscaling configuration has been resolved. ECS clusters should now correctly autoscale by adding new cluster instances according to the policy configuration.
- Async operations that are started by these endpoints will be run as ECS tasks with a launch type of Fargate, not EC2:
  - POST /deadLetterArchive/recoverCumulusMessages
  - POST /elasticsearch/index-from-database
  - POST /granules/bulk
  - POST /granules/bulkDelete
  - POST /granules/bulkReingest
  - POST /migrationCounts
  - POST /reconciliationReports
  - POST /replays
  - POST /replays/sqs

Added

Upgraded version of dependencies on knex package from 0.95.11 to 0.95.15
Added Terraform data sources to example/cumulus-tf module to retrieve default VPC and subnets in NGAP accounts
- Added vpc_tag_name variable which defines the tags used to look up a VPC. Defaults to VPC tag name used in NGAP accounts
- Added subnets_tag_name variable which defines the tags used to look up VPC subnets. Defaults to a subnet tag name used in NGAP accounts
Added Terraform data sources to example/data-persistence-tf module to retrieve default VPC and subnets in NGAP accounts
- Added vpc_tag_name variable which defines the tags used to look up a VPC. Defaults to VPC tag name used in NGAP accounts
- Added subnets_tag_name variable which defines the tags used to look up VPC subnets. Defaults to a subnet tag name used in NGAP accounts
Added Terraform data sources to example/rds-cluster-tf module to retrieve default VPC and subnets in NGAP accounts
- Added vpc_tag_name variable which defines the tags used to look up a VPC. Defaults to VPC tag name used in NGAP accounts
- Added subnets_tag_name variable which defines the tags used to look up VPC subnets. Defaults to tag names used in subnets in for NGAP accounts
CUMULUS-2299
- Added support for SHA checksum types with hyphens (e.g. SHA-256 vs SHA256) to tasks that calculate checksums.
CUMULUS-2439
- Added CMR search client setting to the CreateReconciliationReport lambda function.
- Added cmr_search_client_config tfvars to the archive and cumulus terraform modules.
- Updated CreateReconciliationReport lambda to search CMR collections with CMRSearchConceptQueue.
CUMULUS-2441
- Added support for 'PROD' CMR environment.
CUMULUS-2456
- Updated api lambdas to query ORCA Private API
- Updated example/cumulus-tf/orca.tf to the ORCA release v4.0.0-Beta3
CUMULUS-2638
- Adds documentation to clarify bucket config object use.
CUMULUS-2684
- Added optional collection level parameter s3MultipartChunksizeMb to collection's meta field
- Updated move-granules task to take in an optional config parameter s3MultipartChunksizeMb
CUMULUS-2747
- Updated data management type doc to include additional fields for provider configurations
CUMULUS-2773
- Added a document to the workflow-tasks docs describing deployment, configuration and usage of the LZARDS backup task.

Changed

Made vpc_id variable optional for example/cumulus-tf module
Made vpc_id and subnet_ids variables optional for example/data-persistence-tf module
Made vpc_id and subnets variables optional for example/rds-cluster-tf module
Changes audit script to handle integration test failure when USE\_CACHED\_BOOTSTRAP is disabled.
Increases wait time for CMR to return online resources in integration tests
CUMULUS-1823
- Updates to Cumulus rule/provider schemas to improve field titles and descriptions.
CUMULUS-2638
- Transparent to users, remove typescript type BucketType.
CUMULUS-2718
- Updated config for SyncGranules to support optional workflowStartTime
- Updated SyncGranules to provide createdAt on output based on workflowStartTime if provided, falling back to Date.now() if not provided.
- Updated task_config of SyncGranule in example workflows
CUMULUS-2735
- Updated reconciliation reports to write formatted JSON to S3 to improve readability for large reports
- Updated TEA version from 102 to 121 to address TEA deployment issue with the max size of a policy role being exceeded
CUMULUS-2743
- Updated bamboo Dockerfile to upgrade pip as part of the image creation process
CUMULUS-2744
- GET executions/status returns associated granules for executions retrieved from the Step Function API
CUMULUS-2751
- Upgraded all Cumulus (node.js) workflow tasks to use @cumulus/cumulus-message-adapter-js version 2.0.3, which includes an update cma-js to better expose CMA stderr stream output on lambda timeouts as well as minor logging enhancements.
CUMULUS-2752
- Add new mappings for execution records to prevent dynamic field expansion from exceeding Elasticsearch field limits
  - Nested objects under finalPayload.* will not dynamically add new fields to mapping
  - Nested objects under originalPayload.* will not dynamically add new fields to mapping
  - Nested keys under tasks will not dynamically add new fields to mapping
CUMULUS-2753
- Updated example/cumulus-tf/orca.tf to the latest ORCA release v4.0.0-Beta2 which is compatible with granule.files file schema
- Updated /orca/recovery to call new lambdas request_status_for_granule and request_status_for_job.
- Updated orca integration test
PR #2569
- Fixed TypeError thrown by @cumulus/cmrjs/cmr-utils.getGranuleTemporalInfo when a granule's associated UMM-G JSON metadata file does not contain a ProviderDates element that has a Type of either "Update" or "Insert". If neither are present, the granule's last update date falls back to the "Create" type provider date, or undefined, if none is present.
CUMULUS-2775
- Changed @cumulus/api-client/invokeApi() to accept a single accepted status code or an array of accepted status codes via expectedStatusCodes
PR #2611
- Changed @cumulus/launchpad-auth/LaunchpadToken.requestToken and validateToken to use the HTTPS request option https.pfx instead of the deprecated pfx option for providing the certificate.
CUMULUS-2836
- Updates cmr-utils/getGranuleTemporalInfo to search for a SingleDateTime element, when beginningDateTime value is not found in the metadata file. The granule's temporal information is returned so that both beginningDateTime and endingDateTime are set to the discovered singleDateTimeValue.
CUMULUS-2756
- Updated _writeGranule() in write-granules.js to catch failed granule writes due to schema validation, log the failure and then attempt to set the status of the granule to failed if it already exists to prevent a failure from allowing the granule to get "stuck" in a non-failed status.

Fixed

CUMULUS-2775
- Updated @cumulus/api-client to not log an error for 201 response from updateGranule
CUMULUS-2783
- Added missing lower bound on scale out policy for ECS cluster to ensure that the cluster will autoscale correctly.
CUMULUS-2835
- Updated hyrax-metadata-updates task to support reading the DatasetId from ECHO10 XML, and the EntryTitle from UMM-G JSON; these are both valid alternatives to the shortname and version ID.

9.7.1

published 9.7.1 • 3 years ago

Changelog