@cumulus/test-data - npm Package Versions

published 9.1.0 • 4 years ago

Changelog

[v9.1.0] 2021-06-03

BREAKING CHANGES

@cumulus/api-client/granules.getGranule now returns the granule record from the GET /granules/<granuleId> endpoint, not the raw endpoint response
CUMULUS-2434
- To use the updated update-granules-cmr-metadata-file-links task, the granule UMM-G metadata should have version 1.6.2 or later, since CMR s3 link type 'GET DATA VIA DIRECT ACCESS' is not valid until UMM-G version 1.6.2
CUMULUS-2488
- Removed all EMS reporting including lambdas, endpoints, params, etc as all reporting is now handled through Cloud Metrics
CUMULUS-2472
- Moved existing EarthdataLoginClient to @cumulus/oauth-client/EarthdataLoginClient and updated all references in Cumulus Core.
- Rename EarthdataLoginClient property from earthdataLoginUrl to `loginUrl for consistency with new OAuth clients. See example in oauth-client README

Added

HYRAX-439 - Corrected README.md according to a new Hyrax URL format.
CUMULUS-2354
- Adds configuration options to allow /s3credentials endpoint to distribute same-region read-only tokens based on a user's CMR ACLs.
- Configures the example deployment to enable this feature.
CUMULUS-2442
- Adds option to generate cloudfront URL to lzards-backup task. This will require a few new task config options that have been documented in the task README.
CUMULUS-2470
- Added /s3credentials endpoint for distribution API
CUMULUS-2471
- Add /s3credentialsREADME endpoint to distribution API
CUMULUS-2473
- Updated tf-modules/cumulus_distribution module to take earthdata or cognito credentials
- Configured example/cumulus-tf/cumulus_distribution.tf to use CSDAP credentials
CUMULUS-2474
- Add S3ObjectStore to aws-client. This class allows for interaction with the S3 object store.
- Add object-store package which contains abstracted object store functions for working with various cloud providers
CUMULUS-2477
- Added /, /login and /logout endpoints to cumulus distribution api
CUMULUS-2479
- Adds /version endpoint to distribution API
CUMULUS-2497
- Created isISOFile() to check if a CMR file is a CMR ISO file.
CUMULUS-2371
- Added helpers to @cumulus/ingest/sqs:
  - archiveSqsMessageToS3 - archives an incoming SQS message to S3
  - deleteArchivedMessageFromS3 - deletes a processed SQS message from S3
- Added call to archiveSqsMessageToS3 to sqs-message-consumer which archives all incoming SQS messages to S3.
- Added call to deleteArchivedMessageFrom to sqs-message-remover which deletes archived SQS message from S3 once it has been processed.

Changed

PR2224
CUMULUS-2208
- Moved all @cumulus/api/es/* code to new @cumulus/es-client package
Changed timeout on sfEventSqsToDbRecords Lambda to 60 seconds to match timeout for Knex library to acquire database connections
CUMULUS-2517
- Updated postgres-migration-count-tool default concurrency to '1'
CUMULUS-2489
- Updated docs for Terraform references in FAQs, glossary, and in Deployment sections
CUMULUS-2434
- Updated @cumulus/cmrjs updateCMRMetadata and related functions to add both HTTPS URLS and S3 URIs to CMR metadata.
- Updated update-granules-cmr-metadata-file-links task to add both HTTPS URLs and S3 URIs to the OnlineAccessURLs field of CMR metadata. The task configuration parameter cmrGranuleUrlType now has default value both.
- To use the updated update-granules-cmr-metadata-file-links task, the granule UMM-G metadata should have version 1.6.2 or later, since CMR s3 link type 'GET DATA VIA DIRECT ACCESS' is not valid until UMM-G version 1.6.2
CUMULUS-2472
- Renamed @cumulus/earthdata-login-client to more generic @cumulus/oauth-client as a parent class for new OAuth clients.
- Added @cumulus/oauth-client/CognitoClient to interface with AWS cognito login service.
CUMULUS-2497
- Changed the @cumulus/cmrjs package:
  - Updated @cumulus/cmrjs/cmr-utils.getGranuleTemporalInfo() so it now returns temporal info for CMR ISO 19115 SMAP XML files.
  - Updated @cumulus/cmrjs/cmr-utils.isCmrFilename() to include isISOFile().
CUMULUS-2532
- Changed integration tests to use api-client/granules functions as opposed to granulesApi from @cumulus/integration-tests.

Fixed

CUMULUS-2519
- Update @cumulus/integration-tests.buildWorkflow to fail if provider/collection API response is not successful
CUMULUS-2518
- Update sf-event-sqs-to-db-records to not throw if a collection is not defined on a payload that has no granules/an empty granule payload object
CUMULUS-2512
- Updated ingest package S3 provider client to take additional parameter remoteAltBucket on download method to allow for per-file override of provider bucket for checksum
- Updated @cumulus/ingest.fetchTextFile's signature to be parameterized and added remoteAltBucketto allow for an override of the passed in provider bucket for the source file
- Update "eslint-plugin-import" to be pinned to 2.22.1
CUMULUS-2520
- Fixed error that prevented /elasticsearch/index-from-database from starting.
CUMULUS-2532
- Fixed integration tests to have granule deletion occur before provider and collection deletion in test cleanup.
2231
- Fixes broken relative path links in docs/README.md

Removed

CUMULUS-2502
- Removed outdated documentation regarding Kibana index patterns for metrics.

8.1.1

published 8.1.1 • 4 years ago

Changelog

[v8.1.1] 2021-04-30 -- Patch Release

Please note changes in 8.1.1 may not yet be released in future versions, as this is a backport/patch release on the 8.x series of releases. Updates that are included in the future will have a corresponding CHANGELOG entry in future releases.

Added

CUMULUS-2497
- Created isISOFile() to check if a CMR file is a CMR ISO file.

Fixed

CUMULUS-2512
- Updated ingest package S3 provider client to take additional parameter remoteAltBucket on download method to allow for per-file override of provider bucket for checksum
- Updated @cumulus/ingest.fetchTextFile's signature to be parameterized and added remoteAltBucketto allow for an override of the passed in provider bucket for the source file
- Update "eslint-plugin-import" to be pinned to 2.22.1

Changed

CUMULUS-2497
- Changed the @cumulus/cmrjs package:
  - Updated @cumulus/cmrjs/cmr-utils.getGranuleTemporalInfo() so it now returns temporal info for CMR ISO 19115 SMAP XML files.
  - Updated @cumulus/cmrjs/cmr-utils.isCmrFilename() to include isISOFile().
2216
- Removed "node-forge", "xml-crypto" from audit whitelist, added "underscore"

9.0.1

published 9.0.1 • 4 years ago

Changelog

[v9.0.1] 2021-05-07

Migration Steps

Please review the migration steps for 9.0.0 as this release is only a patch to correct a failure in our build script and push out corrected release artifacts. The previous migration steps still apply.

Changed

Corrected @cumulus/db configuration to correctly build package.

9.0.0

published 9.0.0 • 4 years ago

Changelog

[v9.0.0] 2021-05-03

Migration steps

This release of Cumulus enables integration with a PostgreSQL database for archiving Cumulus data. There are several upgrade steps involved, some of which need to be done before redeploying Cumulus. See the documentation on upgrading to the RDS release.

BREAKING CHANGES

CUMULUS-2185 - RDS Migration Epic
- CUMULUS-2191
  - Removed the following from the @cumulus/api/models.asyncOperation class in favor of the added @cumulus/async-operations module:
    - start
    - startAsyncOperations
- CUMULUS-2187
  - The async-operations endpoint will now omit output instead of returning none when the operation did not return output.
- CUMULUS-2309
  - Removed @cumulus/api/models/granule.unpublishAndDeleteGranule in favor of @cumulus/api/lib/granule-remove-from-cmr.unpublishGranule and @cumulus/api/lib/granule-delete.deleteGranuleAndFiles.
- CUMULUS-2385
  - Updated sf-event-sqs-to-db-records to write a granule's files to PostgreSQL only after the workflow has exited the Running status. Please note that any workflow that uses sf_sqs_report_task for mid-workflow updates will be impacted.
  - Changed PostgreSQL file schema and TypeScript type definition to require bucket and key fields.
  - Updated granule/file write logic to mark a granule's status as "failed"
- CUMULUS-2455
  - API move granule endpoint now moves granule files on a per-file basis
  - API move granule endpoint on granule file move failure will retain the file at it's original location, but continue to move any other granule files.
  - Removed the move method from the @cumulus/api/models.granule class. logic is now handled in @cumulus/api/endpoints/granules and is accessible via the Core API.

Added

CUMULUS-2185 - RDS Migration Epic
- CUMULUS-2130
  - Added postgres-migration-count-tool lambda/ECS task to allow for evaluation of database state
  - Added /migrationCounts api endpoint that allows running of the postgres-migration-count-tool as an asyncOperation
- CUMULUS-2394
  - Updated PDR and Granule writes to check the step function workflow_start_time against the createdAt field for each record to ensure old records do not overwrite newer ones for legacy Dynamo and PostgreSQL writes
- CUMULUS-2188
  - Added data-migration2 Lambda to be run after data-migration1
  - Added logic to data-migration2 Lambda for migrating execution records from DynamoDB to PostgreSQL
- CUMULUS-2191
  - Added @cumulus/async-operations to core packages, exposing startAsyncOperation which will handle starting an async operation and adding an entry to both PostgreSQL and DynamoDb
- CUMULUS-2127
  - Add schema migration for collections table
- CUMULUS-2129
  - Added logic to data-migration1 Lambda for migrating collection records from Dynamo to PostgreSQL
- CUMULUS-2157
  - Add schema migration for providers table
  - Added logic to data-migration1 Lambda for migrating provider records from Dynamo to PostgreSQL
- CUMULUS-2187
  - Added logic to data-migration1 Lambda for migrating async operation records from Dynamo to PostgreSQL
- CUMULUS-2198
  - Added logic to data-migration1 Lambda for migrating rule records from DynamoDB to PostgreSQL
- CUMULUS-2182
  - Add schema migration for PDRs table
- CUMULUS-2230
  - Add schema migration for rules table
- CUMULUS-2183
  - Add schema migration for asyncOperations table
- CUMULUS-2184
  - Add schema migration for executions table
- CUMULUS-2257
  - Updated PostgreSQL table and column names to snake_case
  - Added translateApiAsyncOperationToPostgresAsyncOperation function to @cumulus/db
- CUMULUS-2186
  - Added logic to data-migration2 Lambda for migrating PDR records from DynamoDB to PostgreSQL
- CUMULUS-2235
  - Added initial ingest load spec test/utility
- CUMULUS-2167
  - Added logic to data-migration2 Lambda for migrating Granule records from DynamoDB to PostgreSQL and parse Granule records to store File records in RDS.
- CUMULUS-2367
  - Added granules_executions table to PostgreSQL schema to allow for a many-to-many relationship between granules and executions
    - The table refers to granule and execution records using foreign keys defined with ON CASCADE DELETE, which means that any time a granule or execution record is deleted, all of the records in the granules_executions table referring to that record will also be deleted.
  - Added upsertGranuleWithExecutionJoinRecord helper to @cumulus/db to allow for upserting a granule record and its corresponding granules_execution record
- CUMULUS-2128
  - Added helper functions:
    - @cumulus/db/translate/file/translateApiFiletoPostgresFile
    - @cumulus/db/translate/file/translateApiGranuletoPostgresGranule
    - @cumulus/message/Providers/getMessageProvider
- CUMULUS-2190
  - Added helper functions:
    - @cumulus/message/Executions/getMessageExecutionOriginalPayload
    - @cumulus/message/Executions/getMessageExecutionFinalPayload
    - @cumulus/message/workflows/getMessageWorkflowTasks
    - @cumulus/message/workflows/getMessageWorkflowStartTime
    - @cumulus/message/workflows/getMessageWorkflowStopTime
    - @cumulus/message/workflows/getMessageWorkflowName
- CUMULUS-2192
  - Added helper functions:
    - @cumulus/message/PDRs/getMessagePdrRunningExecutions
    - @cumulus/message/PDRs/getMessagePdrCompletedExecutions
    - @cumulus/message/PDRs/getMessagePdrFailedExecutions
    - @cumulus/message/PDRs/getMessagePdrStats
    - @cumulus/message/PDRs/getPdrPercentCompletion
    - @cumulus/message/workflows/getWorkflowDuration
- CUMULUS-2199
  - Added translateApiRuleToPostgresRule to @cumulus/db to translate API Rule to conform to Postgres Rule definition.
- CUMUlUS-2128
  - Added "upsert" logic to the sfEventSqsToDbRecords Lambda for granule and file writes to the core PostgreSQL database
- CUMULUS-2199
  - Updated Rules endpoint to write rules to core PostgreSQL database in addition to DynamoDB and to delete rules from the PostgreSQL database in addition to DynamoDB.
  - Updated create in Rules Model to take in optional createdAt parameter which sets the value of createdAt if not specified during function call.
- CUMULUS-2189
  - Updated Provider endpoint logic to write providers in parallel to Core PostgreSQL database
  - Update integration tests to utilize API calls instead of direct api/model/Provider calls
- CUMULUS-2191
  - Updated cumuluss/async-operation task to write async-operations to the PostgreSQL database.
- CUMULUS-2228
  - Added logic to the sfEventSqsToDbRecords Lambda to write execution, PDR, and granule records to the core PostgreSQL database in parallel with writes to DynamoDB
- CUMUlUS-2190
  - Added "upsert" logic to the sfEventSqsToDbRecords Lambda for PDR writes to the core PostgreSQL database
- CUMUlUS-2192
  - Added "upsert" logic to the sfEventSqsToDbRecords Lambda for execution writes to the core PostgreSQL database
- CUMULUS-2187
  - The async-operations endpoint will now omit output instead of returning none when the operation did not return output.
- CUMULUS-2167
  - Change PostgreSQL schema definition for files to remove filename and name and only support file_name.
  - Change PostgreSQL schema definition for files to remove size to only support file_size.
  - Change PostgresFile to remove duplicate fields filename and name and rename size to file_size.
- CUMULUS-2266
  - Change sf-event-sqs-to-db-records behavior to discard and not throw an error on an out-of-order/delayed message so as not to have it be sent to the DLQ.
- CUMULUS-2305
  - Changed DELETE /pdrs/{pdrname} API behavior to also delete record from PostgreSQL database.
- CUMULUS-2309
  - Changed DELETE /granules/{granuleName} API behavior to also delete record from PostgreSQL database.
  - Changed Bulk operation BULK_GRANULE_DELETE API behavior to also delete records from PostgreSQL database.
- CUMULUS-2367
  - Updated granule_cumulus_id foreign key to granule in PostgreSQL files table to use a CASCADE delete, so records in the files table are automatically deleted by the database when the corresponding granule is deleted.
- CUMULUS-2407
  - Updated data-migration1 and data-migration2 Lambdas to use UPSERT instead of UPDATE when migrating dynamoDB records to PostgreSQL.
  - Changed data-migration1 and data-migration2 logic to only update already migrated records if the incoming record update has a newer timestamp
- CUMULUS-2329
  - Add write-db-dlq-records-to-s3 lambda.
  - Add terraform config to automatically write db records DLQ messages to an s3 archive on the system bucket.
  - Add unit tests and a component spec test for the above.
- CUMULUS-2380
  - Add process-dead-letter-archive lambda to pick up and process dead letters in the S3 system bucket dead letter archive.
  - Add /deadLetterArchive/recoverCumulusMessages endpoint to trigger an async operation to leverage this capability on demand.
  - Add unit tests and integration test for all of the above.
- CUMULUS-2406
  - Updated parallel write logic to ensure that updatedAt/updated_at timestamps are the same in Dynamo/PG on record write for the following data types:
    - async operations
    - granules
    - executions
    - PDRs
- CUMULUS-2446
  - Remove schema validation check against DynamoDB table for collections when migrating records from DynamoDB to core PostgreSQL database.
- CUMULUS-2447
  - Changed translateApiAsyncOperationToPostgresAsyncOperation to call JSON.stringify and then JSON.parse on output.
- CUMULUS-2313
  - Added postgres-migration-async-operation lambda to start an ECS task to run a the data-migration2 lambda.
  - Updated async_operations table to include Data Migration 2 as a new operation_type.
  - Updated cumulus-tf/variables.tf to include optional_dynamo_tables that will be merged with dynamo_tables.
- CUMULUS-2451
  - Added summary type file packages/db/src/types/summary.ts with MigrationSummary and DataMigration1 and DataMigration2 types.
  - Updated data-migration1 and data-migration2 lambdas to return MigrationSummary objects.
  - Added logging for every batch of 100 records processed for executions, granules and files, and PDRs.
  - Removed RecordAlreadyMigrated logs in data-migration1 and data-migration2
- CUMULUS-2452
  - Added support for only migrating certain granules by specifying the granuleSearchParams.granuleId or granuleSearchParams.collectionId properties in the payload for the <prefix>-postgres-migration-async-operation Lambda
  - Added support for only running certain migrations for data-migration2 by specifying the migrationsList property in the payload for the <prefix>-postgres-migration-async-operation Lambda
- CUMULUS-2453
  - Created storeErrors function which stores errors in system bucket.
  - Updated executions and granulesAndFiles data migrations to call storeErrors to store migration errors.
  - Added system_bucket variable to data-migration2.
- CUMULUS-2455
  - Move granules API endpoint records move updates for migrated granule files if writing any of the granule files fails.
- CUMULUS-2468
  - Added support for doing DynamoDB parallel scanning for executions and granules migrations to improve performance. The behavior of the parallel scanning and writes can be controlled via the following properties on the event input to the <prefix>-postgres-migration-async-operation Lambda:
    - granuleMigrationParams.parallelScanSegments: How many segments to divide your granules DynamoDB table into for parallel scanning
    - granuleMigrationParams.parallelScanLimit: The maximum number of granule records to evaluate for each parallel scanning segment of the DynamoDB table
    - granuleMigrationParams.writeConcurrency: The maximum number of concurrent granule/file writes to perform to the PostgreSQL database across all DynamoDB segments
    - executionMigrationParams.parallelScanSegments: How many segments to divide your executions DynamoDB table into for parallel scanning
    - executionMigrationParams.parallelScanLimit: The maximum number of execution records to evaluate for each parallel scanning segment of the DynamoDB table
    - executionMigrationParams.writeConcurrency: The maximum number of concurrent execution writes to perform to the PostgreSQL database across all DynamoDB segments
- CUMULUS-2468 - Added @cumulus/aws-client/DynamoDb.parallelScan helper to perform parallel scanning on DynamoDb tables
- CUMULUS-2507
  - Updated granule record write logic to set granule status to failed in both Postgres and DynamoDB if any/all of its files fail to write to the database.

Deprecated

CUMULUS-2185 - RDS Migration Epic
- CUMULUS-2455
  - @cumulus/ingest/moveGranuleFiles

8.1.0

published 8.1.0 • 4 years ago

Changelog

[v8.1.0] 2021-04-29

Added

CUMULUS-2348
- The @cumulus/api /granules and /granules/{granuleId} endpoints now take getRecoveryStatus parameter to include recoveryStatus in result granule(s)
- The @cumulus/api-client.granules.getGranule function takes a query parameter which can be used to request additional granule information.
- Published @cumulus/api@7.2.1-alpha.0 for dashboard testing
CUMULUS-2469
- Added tf-modules/cumulus_distribution module to standup a skeleton distribution api

8.0.0

published 8.0.0 • 4 years ago

Changelog

[v8.0.0] 2021-04-08

BREAKING CHANGES

CUMULUS-2428
- Changed /granules/bulk to use queueUrl property instead of a queueName property for setting the queue to use for scheduling bulk granule workflows

Notable changes

Bulk granule operations endpoint now supports setting a custom queue for scheduling workflows via the queueUrl property in the request body. If provided, this value should be the full URL for an SQS queue.

Added

CUMULUS-2374
- Add cookbok entry for queueing PostToCmr step
- Add example workflow to go with cookbook
CUMULUS-2421
- Added experimental ecs_include_docker_cleanup_cronjob boolean variable to the Cumulus module to enable cron job to clean up docker root storage blocks in ECS cluster template for non-device-mapper storage drivers. Default value is false. This fulfills a specific user support request. This feature is otherwise untested and will remain so until we can iterate with a better, more general-purpose solution. Use of this feature is NOT recommended unless you are certain you need it.
CUMULUS-1808
- Add additional error messaging in deleteSnsTrigger to give users more context about where to look to resolve ResourceNotFound error when disabling or deleting a rule.

Fixed

CUMULUS-2281
- Changed discover-granules task to write discovered granules directly to logger, instead of via environment variable. This fixes a problem where a large number of found granules prevents this lambda from running as an activity with an E2BIG error.

6.0.1

published 6.0.1 • 4 years ago

7.2.0

published 7.2.0 • 4 years ago

Changelog

[v7.2.0] 2021-03-23

Added

CUMULUS-2346
- Added orca API endpoint to @cumulus/api to get recovery status
- Add CopyToGlacier step to example IngestAndPublishGranuleWithOrca workflow

Changed

HYRAX-357
- Format of NGAP OPeNDAP URL changed and by default now is referring to concept id and optionally can include short name and version of collection.
- addShortnameAndVersionIdToConceptId field has been added to the config inputs of the hyrax-metadata-updates task

7.1.0

published 7.1.0 • 4 years ago

Changelog