@cumulus/common - npm Package Versions

[v9.2.1] 2021-07-29 - [BACKPORT]

Fixed

• CUMULUS-2626
  • Update PDR migration to correctly find Executions by a Dynamo PDR's execution field

[v9.3.0] 2021-07-26

BREAKING CHANGES

• All API requests made by @cumulus/api-client will now throw an error if the status code does not match the expected response (200 for most requests and 202 for a few requests that trigger async operations). Previously the helpers in this package would return the response regardless of the status code, so you may need to update any code using helpers from this package to catch or to otherwise handle errors that you may encounter.
• The Cumulus API Lambda function has now been configured with reserved concurrency to ensure availability in a high-concurrency environment. However, this also caps max concurrency which may result in throttling errors if trying to reach the Cumulus API multiple times in a short period. Reserved concurrency can be configured with the archive_api_reserved_concurrency terraform variable on the Cumulus module and increased if you are seeing throttling errors. The default reserved concurrency value is 8.

Notable changes

• cmr_custom_host variable for cumulus module can now be used to configure Cumulus to integrate with a custom CMR host name and protocol (e.g. http://custom-cmr-host.com). Note that you must include a protocol (http:// or `https://) if specifying a value for this variable.
• The cumulus module configuration valuerds_connetion_heartbeat and it's behavior has been replaced by a more robust database connection 'retry' solution. Users can remove this value from their configuration, regardless of value. See the Changed section notes on CUMULUS-2528 for more details.

Added

• Added user doc describing new features related to the Cumulus dead letter archive.

• CUMULUS-2327

  • Added reserved concurrency setting to the Cumulus API lambda function.
  • Added relevant tfvars to the archive and cumulus terraform modules.

• CUMULUS-2460

  • Adds POST /executions/search-by-granules for retrieving executions from a list of granules or granule query
  • Adds searchExecutionsByGranules to @cumulus/api-client/executions

• CUMULUS-2475

  • Adds GET endpoint to distribution API

• CUMULUS-2463

  • PUT /granules reingest action allows a user to override the default execution to use by providing an optional workflowName or executionArn parameter on the request body.
  • PUT /granules/bulkReingest action allows a user to override the default execution/workflow combination to reingest with by providing an optional workflowName on the request body.

• Adds workflowName and executionArn params to @cumulus/api-client/reingestGranules

• CUMULUS-2476

  • Adds handler for authenticated HEAD Distribution requests replicating current behavior of TEA

• CUMULUS-2478

  • Implemented bucket map.
  • Implemented /locate endpoint
  • Cumulus distribution API checks the file request against bucket map:
    • retrieves the bucket and key from file path
    • determines if the file request is public based on the bucket map rather than the bucket type
    • (EDL only) restricts download from PRIVATE_BUCKETS to users who belong to certain EDL User Groups
    • bucket prefix and object prefix are supported
  • Add 'Bearer token' support as an authorization method

• CUMULUS-2486

  • Implemented support for custom headers
  • Added 'Bearer token' support as an authorization method

• CUMULUS-2487

  • Added integration test for cumulus distribution API

• CUMULUS-2569

  • Created bucket map cache for cumulus distribution API

• CUMULUS-2568

  • Add deletePdr/PDR deletion functionality to @cumulus/api-client/pdrs
  • Add removeCollectionAndAllDependencies to integration test helpers
  • Added example/spec/apiUtils.waitForApiStatus to wait for a record to be returned by the API with a specific value for status
  • Added example/spec/discoverUtils.uploadS3GranuleDataForDiscovery to upload granule data fixtures to S3 with a randomized granule ID for discover-granules based integration tests
  • Added example/spec/Collections.removeCollectionAndAllDependencies to remove a collection and all dependent objects (e.g. PDRs, granules, executions) from the database via the API
  • Added helpers to @cumulus/api-client:
    • pdrs.deletePdr - Delete a PDR via the API
    • replays.postKinesisReplays - Submit a POST request to the /replays endpoint for replaying Kinesis messages

• @cumulus/api-client/granules.getGranuleResponse to return the raw endpoint response from the GET /granules/<granuleId> endpoint

Changed

• Moved functions from @cumulus/integration-tests to example/spec/helpers/workflowUtils:
  • startWorkflowExecution
  • startWorkflow
  • executeWorkflow
  • buildWorkflow
  • testWorkflow
  • buildAndExecuteWorkflow
  • buildAndStartWorkflow
• example/spec/helpers/workflowUtils.executeWorkflow now uses waitForApiStatus to ensure that the execution is completed or failed before resolving
• example/spec/helpers/testUtils.updateAndUploadTestFileToBucket now accepts an object of parameters rather than positional arguments
• Removed PDR from the payload in the input payload test fixture for reconciliation report integration tests
• The following integration tests for PDR-based workflows were updated to use randomized granule IDs:
  • example/spec/parallel/ingest/ingestFromPdrSpec.js
  • example/spec/parallel/ingest/ingestFromPdrWithChildWorkflowMetaSpec.js
  • example/spec/parallel/ingest/ingestFromPdrWithExecutionNamePrefixSpec.js
  • example/spec/parallel/ingest/ingestPdrWithNodeNameSpec.js
• Updated the @cumulus/api-client/CumulusApiClientError error class to include new properties that can be accessed directly on the error object:
  • statusCode - The HTTP status code of the API response
  • apiMessage - The message from the API response
• Added params.pRetryOptions parameter to @cumulus/api-client/granules.deleteGranule to control the retry behavior
• Updated cmr_custom_host variable to accept a full protocol and host name (e.g. http://cmr-custom-host.com), whereas it previously only accepted a host name
• CUMULUS-2482
  • Switches the default distribution app in the example/cumulus-tf deployment to the new Cumulus Distribution
  • TEA is still available by following instructions in example/README.md
• CUMULUS-2463
  • Increases the duration of allowed backoff times for a successful test from 0.5 sec to 1 sec.
• CUMULUS-2528
  • Removed rds_connection_heartbeat as a configuration option from all Cumulus terraform modules
  • Removed dbHeartBeat as an environmental switch from @cumulus/db.getKnexClient in favor of more comprehensive general db connect retry solution
  • Added new rds_connection_timing_configuration string map to allow for configuration and tuning of Core's internal database retry/connection timeout behaviors. These values map to connection pool configuration values for tarn (https://github.com/vincit/tarn.js/) which Core's database module / knex(https://www.npmjs.com/package/knex) use for this purpose:
    • acquireTimeoutMillis
    • createRetryIntervalMillis
    • createTimeoutMillis
    • idleTimeoutMillis
    • reapIntervalMillis Connection errors will result in a log line prepended with 'knex failed on attempted connection error' and sent from '@cumulus/db/connection'
  • Updated @cumulus/db and all terraform mdules to set default retry configuration values for the database module to cover existing database heartbeat connection failures as well as all other knex/tarn connection creation failures.

Fixed

• Fixed bug where cmr_custom_host variable was not properly forwarded into archive, ingest, and sqs-message-remover modules from cumulus module
• Fixed bug where parse-pdr set a granule's provider to the entire provider record when a NODE_NAME is present. Expected behavior consistent with other tasks is to set the provider name in that field.
• CUMULUS-2568
  • Update reconciliation report integration test to have better cleanup/failure behavior
  • Fixed @cumulus/api-client/pdrs.getPdr to request correct endpoint for returning a PDR from the API
• CUMULUS-2620
  • Fixed a bug where a granule could be removed from CMR but still be set as published: true and with a CMR link in the Dynamo/PostgreSQL databases. Now, the CMR deletion and the Dynamo/PostgreSQL record updates will all succeed or fail together, preventing the database records from being out of sync with CMR.
  • Fixed @cumulus/api-client/pdrs.getPdr to request correct endpoint for returning a PDR from the API

[v9.2.0] 2021-06-22

Added

• CUMULUS-2475
  • Adds GET endpoint to distribution API
• CUMULUS-2476
  • Adds handler for authenticated HEAD Distribution requests replicating current behavior of TEA

Changed

• CUMULUS-2482
  • Switches the default distribution app in the example/cumulus-tf deployment to the new Cumulus Distribution
  • TEA is still available by following instructions in example/README.md

Fixed

• CUMULUS-2520
  • Fixed error that prevented /elasticsearch/index-from-database from starting.
• CUMULUS-2558
  • Fixed issue where executions original_payload would not be retained on successful execution

[v9.1.0] 2021-06-03

BREAKING CHANGES

• @cumulus/api-client/granules.getGranule now returns the granule record from the GET /granules/<granuleId> endpoint, not the raw endpoint response
• CUMULUS-2434
  • To use the updated update-granules-cmr-metadata-file-links task, the granule UMM-G metadata should have version 1.6.2 or later, since CMR s3 link type 'GET DATA VIA DIRECT ACCESS' is not valid until UMM-G version 1.6.2
• CUMULUS-2488
  • Removed all EMS reporting including lambdas, endpoints, params, etc as all reporting is now handled through Cloud Metrics
• CUMULUS-2472
  • Moved existing EarthdataLoginClient to @cumulus/oauth-client/EarthdataLoginClient and updated all references in Cumulus Core.
  • Rename EarthdataLoginClient property from earthdataLoginUrl to `loginUrl for consistency with new OAuth clients. See example in oauth-client README

Added

• HYRAX-439 - Corrected README.md according to a new Hyrax URL format.
• CUMULUS-2354
  • Adds configuration options to allow /s3credentials endpoint to distribute same-region read-only tokens based on a user's CMR ACLs.
  • Configures the example deployment to enable this feature.
• CUMULUS-2442
  • Adds option to generate cloudfront URL to lzards-backup task. This will require a few new task config options that have been documented in the task README.
• CUMULUS-2470
  • Added /s3credentials endpoint for distribution API
• CUMULUS-2471
  • Add /s3credentialsREADME endpoint to distribution API
• CUMULUS-2473
  • Updated tf-modules/cumulus_distribution module to take earthdata or cognito credentials
  • Configured example/cumulus-tf/cumulus_distribution.tf to use CSDAP credentials
• CUMULUS-2474
  • Add S3ObjectStore to aws-client. This class allows for interaction with the S3 object store.
  • Add object-store package which contains abstracted object store functions for working with various cloud providers
• CUMULUS-2477
  • Added /, /login and /logout endpoints to cumulus distribution api
• CUMULUS-2479
  • Adds /version endpoint to distribution API
• CUMULUS-2497
  • Created isISOFile() to check if a CMR file is a CMR ISO file.
• CUMULUS-2371
  • Added helpers to @cumulus/ingest/sqs:
    • archiveSqsMessageToS3 - archives an incoming SQS message to S3
    • deleteArchivedMessageFromS3 - deletes a processed SQS message from S3
  • Added call to archiveSqsMessageToS3 to sqs-message-consumer which archives all incoming SQS messages to S3.
  • Added call to deleteArchivedMessageFrom to sqs-message-remover which deletes archived SQS message from S3 once it has been processed.

Changed

• PR2224
• CUMULUS-2208
  • Moved all @cumulus/api/es/* code to new @cumulus/es-client package
• Changed timeout on sfEventSqsToDbRecords Lambda to 60 seconds to match timeout for Knex library to acquire database connections
• CUMULUS-2517
  • Updated postgres-migration-count-tool default concurrency to '1'
• CUMULUS-2489
  • Updated docs for Terraform references in FAQs, glossary, and in Deployment sections
• CUMULUS-2434
  • Updated @cumulus/cmrjs updateCMRMetadata and related functions to add both HTTPS URLS and S3 URIs to CMR metadata.
  • Updated update-granules-cmr-metadata-file-links task to add both HTTPS URLs and S3 URIs to the OnlineAccessURLs field of CMR metadata. The task configuration parameter cmrGranuleUrlType now has default value both.
  • To use the updated update-granules-cmr-metadata-file-links task, the granule UMM-G metadata should have version 1.6.2 or later, since CMR s3 link type 'GET DATA VIA DIRECT ACCESS' is not valid until UMM-G version 1.6.2
• CUMULUS-2472
  • Renamed @cumulus/earthdata-login-client to more generic @cumulus/oauth-client as a parent class for new OAuth clients.
  • Added @cumulus/oauth-client/CognitoClient to interface with AWS cognito login service.
• CUMULUS-2497
  • Changed the @cumulus/cmrjs package:
    • Updated @cumulus/cmrjs/cmr-utils.getGranuleTemporalInfo() so it now returns temporal info for CMR ISO 19115 SMAP XML files.
    • Updated @cumulus/cmrjs/cmr-utils.isCmrFilename() to include isISOFile().
• CUMULUS-2532
  • Changed integration tests to use api-client/granules functions as opposed to granulesApi from @cumulus/integration-tests.

Fixed

• CUMULUS-2519
  • Update @cumulus/integration-tests.buildWorkflow to fail if provider/collection API response is not successful
• CUMULUS-2518
  • Update sf-event-sqs-to-db-records to not throw if a collection is not defined on a payload that has no granules/an empty granule payload object
• CUMULUS-2512
  • Updated ingest package S3 provider client to take additional parameter remoteAltBucket on download method to allow for per-file override of provider bucket for checksum
  • Updated @cumulus/ingest.fetchTextFile's signature to be parameterized and added remoteAltBucketto allow for an override of the passed in provider bucket for the source file
  • Update "eslint-plugin-import" to be pinned to 2.22.1
• CUMULUS-2520
  • Fixed error that prevented /elasticsearch/index-from-database from starting.
• CUMULUS-2532
  • Fixed integration tests to have granule deletion occur before provider and collection deletion in test cleanup.
• 2231
  • Fixes broken relative path links in docs/README.md

Removed

• CUMULUS-2502
  • Removed outdated documentation regarding Kibana index patterns for metrics.

[v8.1.1] 2021-04-30 -- Patch Release

Please note changes in 8.1.1 may not yet be released in future versions, as this is a backport/patch release on the 8.x series of releases. Updates that are included in the future will have a corresponding CHANGELOG entry in future releases.

Added

• CUMULUS-2497
  • Created isISOFile() to check if a CMR file is a CMR ISO file.

Fixed

• CUMULUS-2512
  • Updated ingest package S3 provider client to take additional parameter remoteAltBucket on download method to allow for per-file override of provider bucket for checksum
  • Updated @cumulus/ingest.fetchTextFile's signature to be parameterized and added remoteAltBucketto allow for an override of the passed in provider bucket for the source file
  • Update "eslint-plugin-import" to be pinned to 2.22.1

Changed

• CUMULUS-2497

  • Changed the @cumulus/cmrjs package:
    • Updated @cumulus/cmrjs/cmr-utils.getGranuleTemporalInfo() so it now returns temporal info for CMR ISO 19115 SMAP XML files.
    • Updated @cumulus/cmrjs/cmr-utils.isCmrFilename() to include isISOFile().

• 2216

  • Removed "node-forge", "xml-crypto" from audit whitelist, added "underscore"

[v9.0.1] 2021-05-07

Migration Steps

Please review the migration steps for 9.0.0 as this release is only a patch to correct a failure in our build script and push out corrected release artifacts. The previous migration steps still apply.

Changed

• Corrected @cumulus/db configuration to correctly build package.

[v9.0.0] 2021-05-03

Migration steps

• This release of Cumulus enables integration with a PostgreSQL database for archiving Cumulus data. There are several upgrade steps involved, some of which need to be done before redeploying Cumulus. See the documentation on upgrading to the RDS release.

BREAKING CHANGES

• CUMULUS-2185 - RDS Migration Epic
  • CUMULUS-2191
    • Removed the following from the @cumulus/api/models.asyncOperation class in favor of the added @cumulus/async-operations module:
      • start
      • startAsyncOperations
  • CUMULUS-2187
    • The async-operations endpoint will now omit output instead of returning none when the operation did not return output.
  • CUMULUS-2309
    • Removed @cumulus/api/models/granule.unpublishAndDeleteGranule in favor of @cumulus/api/lib/granule-remove-from-cmr.unpublishGranule and @cumulus/api/lib/granule-delete.deleteGranuleAndFiles.
  • CUMULUS-2385
    • Updated sf-event-sqs-to-db-records to write a granule's files to PostgreSQL only after the workflow has exited the Running status. Please note that any workflow that uses sf_sqs_report_task for mid-workflow updates will be impacted.
    • Changed PostgreSQL file schema and TypeScript type definition to require bucket and key fields.
    • Updated granule/file write logic to mark a granule's status as "failed"
  • CUMULUS-2455
    • API move granule endpoint now moves granule files on a per-file basis
    • API move granule endpoint on granule file move failure will retain the file at it's original location, but continue to move any other granule files.
    • Removed the move method from the @cumulus/api/models.granule class. logic is now handled in @cumulus/api/endpoints/granules and is accessible via the Core API.

Added

• CUMULUS-2185 - RDS Migration Epic
  • CUMULUS-2130
    • Added postgres-migration-count-tool lambda/ECS task to allow for evaluation of database state
    • Added /migrationCounts api endpoint that allows running of the postgres-migration-count-tool as an asyncOperation
  • CUMULUS-2394
    • Updated PDR and Granule writes to check the step function workflow_start_time against the createdAt field for each record to ensure old records do not overwrite newer ones for legacy Dynamo and PostgreSQL writes
  • CUMULUS-2188
    • Added data-migration2 Lambda to be run after data-migration1
    • Added logic to data-migration2 Lambda for migrating execution records from DynamoDB to PostgreSQL
  • CUMULUS-2191
    • Added @cumulus/async-operations to core packages, exposing startAsyncOperation which will handle starting an async operation and adding an entry to both PostgreSQL and DynamoDb
  • CUMULUS-2127
    • Add schema migration for collections table
  • CUMULUS-2129
    • Added logic to data-migration1 Lambda for migrating collection records from Dynamo to PostgreSQL
  • CUMULUS-2157
    • Add schema migration for providers table
    • Added logic to data-migration1 Lambda for migrating provider records from Dynamo to PostgreSQL
  • CUMULUS-2187
    • Added logic to data-migration1 Lambda for migrating async operation records from Dynamo to PostgreSQL
  • CUMULUS-2198
    • Added logic to data-migration1 Lambda for migrating rule records from DynamoDB to PostgreSQL
  • CUMULUS-2182
    • Add schema migration for PDRs table
  • CUMULUS-2230
    • Add schema migration for rules table
  • CUMULUS-2183
    • Add schema migration for asyncOperations table
  • CUMULUS-2184
    • Add schema migration for executions table
  • CUMULUS-2257
    • Updated PostgreSQL table and column names to snake_case
    • Added translateApiAsyncOperationToPostgresAsyncOperation function to @cumulus/db
  • CUMULUS-2186
    • Added logic to data-migration2 Lambda for migrating PDR records from DynamoDB to PostgreSQL
  • CUMULUS-2235
    • Added initial ingest load spec test/utility
  • CUMULUS-2167
    • Added logic to data-migration2 Lambda for migrating Granule records from DynamoDB to PostgreSQL and parse Granule records to store File records in RDS.
  • CUMULUS-2367
    • Added granules_executions table to PostgreSQL schema to allow for a many-to-many relationship between granules and executions
      • The table refers to granule and execution records using foreign keys defined with ON CASCADE DELETE, which means that any time a granule or execution record is deleted, all of the records in the granules_executions table referring to that record will also be deleted.
    • Added upsertGranuleWithExecutionJoinRecord helper to @cumulus/db to allow for upserting a granule record and its corresponding granules_execution record
  • CUMULUS-2128
    • Added helper functions:
      • @cumulus/db/translate/file/translateApiFiletoPostgresFile
      • @cumulus/db/translate/file/translateApiGranuletoPostgresGranule
      • @cumulus/message/Providers/getMessageProvider
  • CUMULUS-2190
    • Added helper functions:
      • @cumulus/message/Executions/getMessageExecutionOriginalPayload
      • @cumulus/message/Executions/getMessageExecutionFinalPayload
      • @cumulus/message/workflows/getMessageWorkflowTasks
      • @cumulus/message/workflows/getMessageWorkflowStartTime
      • @cumulus/message/workflows/getMessageWorkflowStopTime
      • @cumulus/message/workflows/getMessageWorkflowName
  • CUMULUS-2192
    • Added helper functions:
      • @cumulus/message/PDRs/getMessagePdrRunningExecutions
      • @cumulus/message/PDRs/getMessagePdrCompletedExecutions
      • @cumulus/message/PDRs/getMessagePdrFailedExecutions
      • @cumulus/message/PDRs/getMessagePdrStats
      • @cumulus/message/PDRs/getPdrPercentCompletion
      • @cumulus/message/workflows/getWorkflowDuration
  • CUMULUS-2199
    • Added translateApiRuleToPostgresRule to @cumulus/db to translate API Rule to conform to Postgres Rule definition.
  • CUMUlUS-2128
    • Added "upsert" logic to the sfEventSqsToDbRecords Lambda for granule and file writes to the core PostgreSQL database
  • CUMULUS-2199
    • Updated Rules endpoint to write rules to core PostgreSQL database in addition to DynamoDB and to delete rules from the PostgreSQL database in addition to DynamoDB.
    • Updated create in Rules Model to take in optional createdAt parameter which sets the value of createdAt if not specified during function call.
  • CUMULUS-2189
    • Updated Provider endpoint logic to write providers in parallel to Core PostgreSQL database
    • Update integration tests to utilize API calls instead of direct api/model/Provider calls
  • CUMULUS-2191
    • Updated cumuluss/async-operation task to write async-operations to the PostgreSQL database.
  • CUMULUS-2228
    • Added logic to the sfEventSqsToDbRecords Lambda to write execution, PDR, and granule records to the core PostgreSQL database in parallel with writes to DynamoDB
  • CUMUlUS-2190
    • Added "upsert" logic to the sfEventSqsToDbRecords Lambda for PDR writes to the core PostgreSQL database
  • CUMUlUS-2192
    • Added "upsert" logic to the sfEventSqsToDbRecords Lambda for execution writes to the core PostgreSQL database
  • CUMULUS-2187
    • The async-operations endpoint will now omit output instead of returning none when the operation did not return output.
  • CUMULUS-2167
    • Change PostgreSQL schema definition for files to remove filename and name and only support file_name.
    • Change PostgreSQL schema definition for files to remove size to only support file_size.
    • Change PostgresFile to remove duplicate fields filename and name and rename size to file_size.
  • CUMULUS-2266
    • Change sf-event-sqs-to-db-records behavior to discard and not throw an error on an out-of-order/delayed message so as not to have it be sent to the DLQ.
  • CUMULUS-2305
    • Changed DELETE /pdrs/{pdrname} API behavior to also delete record from PostgreSQL database.
  • CUMULUS-2309
    • Changed DELETE /granules/{granuleName} API behavior to also delete record from PostgreSQL database.
    • Changed Bulk operation BULK_GRANULE_DELETE API behavior to also delete records from PostgreSQL database.
  • CUMULUS-2367
    • Updated granule_cumulus_id foreign key to granule in PostgreSQL files table to use a CASCADE delete, so records in the files table are automatically deleted by the database when the corresponding granule is deleted.
  • CUMULUS-2407
    • Updated data-migration1 and data-migration2 Lambdas to use UPSERT instead of UPDATE when migrating dynamoDB records to PostgreSQL.
    • Changed data-migration1 and data-migration2 logic to only update already migrated records if the incoming record update has a newer timestamp
  • CUMULUS-2329
    • Add write-db-dlq-records-to-s3 lambda.
    • Add terraform config to automatically write db records DLQ messages to an s3 archive on the system bucket.
    • Add unit tests and a component spec test for the above.
  • CUMULUS-2380
    • Add process-dead-letter-archive lambda to pick up and process dead letters in the S3 system bucket dead letter archive.
    • Add /deadLetterArchive/recoverCumulusMessages endpoint to trigger an async operation to leverage this capability on demand.
    • Add unit tests and integration test for all of the above.
  • CUMULUS-2406
    • Updated parallel write logic to ensure that updatedAt/updated_at timestamps are the same in Dynamo/PG on record write for the following data types:
      • async operations
      • granules
      • executions
      • PDRs
  • CUMULUS-2446
    • Remove schema validation check against DynamoDB table for collections when migrating records from DynamoDB to core PostgreSQL database.
  • CUMULUS-2447
    • Changed translateApiAsyncOperationToPostgresAsyncOperation to call JSON.stringify and then JSON.parse on output.
  • CUMULUS-2313
    • Added postgres-migration-async-operation lambda to start an ECS task to run a the data-migration2 lambda.
    • Updated async_operations table to include Data Migration 2 as a new operation_type.
    • Updated cumulus-tf/variables.tf to include optional_dynamo_tables that will be merged with dynamo_tables.
  • CUMULUS-2451
    • Added summary type file packages/db/src/types/summary.ts with MigrationSummary and DataMigration1 and DataMigration2 types.
    • Updated data-migration1 and data-migration2 lambdas to return MigrationSummary objects.
    • Added logging for every batch of 100 records processed for executions, granules and files, and PDRs.
    • Removed RecordAlreadyMigrated logs in data-migration1 and data-migration2
  • CUMULUS-2452
    • Added support for only migrating certain granules by specifying the granuleSearchParams.granuleId or granuleSearchParams.collectionId properties in the payload for the <prefix>-postgres-migration-async-operation Lambda
    • Added support for only running certain migrations for data-migration2 by specifying the migrationsList property in the payload for the <prefix>-postgres-migration-async-operation Lambda
  • CUMULUS-2453
    • Created storeErrors function which stores errors in system bucket.
    • Updated executions and granulesAndFiles data migrations to call storeErrors to store migration errors.
    • Added system_bucket variable to data-migration2.
  • CUMULUS-2455
    • Move granules API endpoint records move updates for migrated granule files if writing any of the granule files fails.
  • CUMULUS-2468
    • Added support for doing DynamoDB parallel scanning for executions and granules migrations to improve performance. The behavior of the parallel scanning and writes can be controlled via the following properties on the event input to the <prefix>-postgres-migration-async-operation Lambda:
      • granuleMigrationParams.parallelScanSegments: How many segments to divide your granules DynamoDB table into for parallel scanning
      • granuleMigrationParams.parallelScanLimit: The maximum number of granule records to evaluate for each parallel scanning segment of the DynamoDB table
      • granuleMigrationParams.writeConcurrency: The maximum number of concurrent granule/file writes to perform to the PostgreSQL database across all DynamoDB segments
      • executionMigrationParams.parallelScanSegments: How many segments to divide your executions DynamoDB table into for parallel scanning
      • executionMigrationParams.parallelScanLimit: The maximum number of execution records to evaluate for each parallel scanning segment of the DynamoDB table
      • executionMigrationParams.writeConcurrency: The maximum number of concurrent execution writes to perform to the PostgreSQL database across all DynamoDB segments
  • CUMULUS-2468 - Added @cumulus/aws-client/DynamoDb.parallelScan helper to perform parallel scanning on DynamoDb tables
  • CUMULUS-2507
    • Updated granule record write logic to set granule status to failed in both Postgres and DynamoDB if any/all of its files fail to write to the database.

Deprecated

• CUMULUS-2185 - RDS Migration Epic
  • CUMULUS-2455
    • @cumulus/ingest/moveGranuleFiles

[v8.1.0] 2021-04-29

Added

• CUMULUS-2348
  • The @cumulus/api /granules and /granules/{granuleId} endpoints now take getRecoveryStatus parameter to include recoveryStatus in result granule(s)
  • The @cumulus/api-client.granules.getGranule function takes a query parameter which can be used to request additional granule information.
  • Published @cumulus/api@7.2.1-alpha.0 for dashboard testing
• CUMULUS-2469
  • Added tf-modules/cumulus_distribution module to standup a skeleton distribution api

[v8.0.0] 2021-04-08

BREAKING CHANGES

• CUMULUS-2428
  • Changed /granules/bulk to use queueUrl property instead of a queueName property for setting the queue to use for scheduling bulk granule workflows

Notable changes

• Bulk granule operations endpoint now supports setting a custom queue for scheduling workflows via the queueUrl property in the request body. If provided, this value should be the full URL for an SQS queue.

Added

• CUMULUS-2374

  • Add cookbok entry for queueing PostToCmr step
  • Add example workflow to go with cookbook

• CUMULUS-2421

  • Added experimental ecs_include_docker_cleanup_cronjob boolean variable to the Cumulus module to enable cron job to clean up docker root storage blocks in ECS cluster template for non-device-mapper storage drivers. Default value is false. This fulfills a specific user support request. This feature is otherwise untested and will remain so until we can iterate with a better, more general-purpose solution. Use of this feature is NOT recommended unless you are certain you need it.

• CUMULUS-1808

  • Add additional error messaging in deleteSnsTrigger to give users more context about where to look to resolve ResourceNotFound error when disabling or deleting a rule.

Fixed

• CUMULUS-2281
  • Changed discover-granules task to write discovered granules directly to logger, instead of via environment variable. This fixes a problem where a large number of found granules prevents this lambda from running as an activity with an E2BIG error.