@cumulus/common

@cumulus/common

Common libraries used in Cumulus.

What is Cumulus?

Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.

Cumulus Documentation

General Utilities

• @cumulus/common/aws Utilities for working with AWS. For ease of setup, testing, and credential management, code should obtain AWS client objects from helpers in this module.
• @cumulus/common/concurrency Implementations of distributed concurrency primitives (mutex, semaphore) using DynamoDB
• @cumulus/common/errors Classes for thrown errors
• @cumulus/common/log Log helpers. Code should use this instead of console.* directly to enable tagging, timestamping, muting or potentially shipping logs to alternative locations
• @cumulus/common/string Utilities for manipulating strings
• @cumulus/common/util Other misc general utilities
• @cumulus/common/test-utils Utilities for writing tests
• @cumulus/common/local-helpers: Provides methods for setting up message payloads for use in development / local testing

Contributing

See Cumulus README

Changelog

[v1.11.2] - 2019-2-15

Added

• CUMULUS-1169
  • Added a @cumulus/common/StepFunctions module. It contains functions for querying the AWS StepFunctions API. These functions have the ability to retry when a ThrottlingException occurs.
  • Added @cumulus/common/aws.retryOnThrottlingException(), which will wrap a function in code to retry on ThrottlingExceptions.
  • Added @cumulus/common/test-utils.throttleOnce(), which will cause a function to return a ThrottlingException the first time it is called, then return its normal result after that.
• CUMULUS-1103 Compare the collection holdings in CMR with Cumulus' internal data store
• CUMULUS-1099 Add support for UMMG JSON metadata versions > 1.4.
  • If a version is found in the metadata object, that version is used for processing and publishing to CMR otherwise, version 1.4 is assumed.
• CUMULUS-678
  • Added support for UMMG json v1.4 metadata files. reconcileCMRMetadata added to @cumulus/cmrjs to update metadata record with new file locations. @cumulus/common/errors adds two new error types CMRMetaFileNotFound and InvalidArgument. @cumulus/common/test-utils adds new function randomId to create a random string with id to help in debugging. @cumulus/common/BucketsConfig adds a new helper class BucketsConfig for working with bucket stack configuration and bucket names. @cumulus/common/aws adds new function s3PutObjectTagging as a convenience for the aws s3().putObjectTagging function. @cumulus/cmrjs Adds: - isCMRFile - Identify an echo10(xml) or UMMG(json) metadata file. - metadataObjectFromCMRFile Read and parse CMR XML file from s3. - updateCMRMetadata Modify a cmr metadata (xml/json) file with updated information. - publish2CMR Posts XML or UMMG CMR data to CMR service. - reconcileCMRMetadata Reconciles cmr metadata file after a file moves.
• Adds some ECS and other permissions to StepRole to enable running ECS tasks from a workflow
• Added Apache logs to cumulus api and distribution lambdas
• CUMULUS-1119 - Added @cumulus/integration-tests/api/EarthdataLogin.getEarthdataLoginRedirectResponse helper for integration tests to handle login with Earthdata and to return response from redirect to Cumulus API
• CUMULUS-673 Added @cumulus/common/file/getFileChecksumFromStream to get file checksum from a readable stream

Fixed

• CUMULUS-1123
  • Cloudformation template overrides now work as expected

Changed

• CUMULUS-1169

  • Deprecated the @cumulus/common/step-functions module.
  • Updated code that queries the StepFunctions API to use the retry-enabled functions from @cumulus/common/StepFunctions

• CUMULUS-1121

  • Schema validation is now strongly enforced when writing to the database. Additional properties are not allowed and will result in a validation error.

• CUMULUS-678 tasks/move-granules simplified and refactored to use functionality from cmrjs. ingest/granules.moveGranuleFiles now just moves granule files and returns a list of the updated files. Updating metadata now handled by @cumulus/cmrjs/reconcileCMRMetadata. move-granules.updateGranuleMetadata refactored and bugs fixed in the case of a file matching multiple collection.files.regexps. getCmrXmlFiles simplified and now only returns an object with the cmrfilename and the granuleId. @cumulus/test-processing - test processing task updated to generate UMM-G metadata

• CUMULUS-1043

  • @cumulus/api now uses express as the API engine.
  • All @cumulus/api endpoints on ApiGateway are consolidated to a single endpoint the uses {proxy+} definition.
  • All files under packages/api/endpoints along with associated tests are updated to support express's request and response objects.
  • Replaced environment variables internal, bucket and systemBucket with system_bucket.
  • Update @cumulus/integration-tests to work with updated cumulus-api express endpoints

• @cumulus/integration-tests - buildAndExecuteWorkflow and buildWorkflow updated to take a meta param to allow for additional fields to be added to the workflow meta

• CUMULUS-1049 Updated Retrieve Execution Status API in @cumulus/api: If the execution doesn't exist in Step Function API, Cumulus API returns the execution status information from the database.

• CUMULUS-1119

  • Renamed DISTRIBUTION_URL environment variable to DISTRIBUTION_ENDPOINT
  • Renamed DEPLOYMENT_ENDPOINT environment variable to DISTRIBUTION_REDIRECT_ENDPOINT
  • Renamed API_ENDPOINT environment variable to TOKEN_REDIRECT_ENDPOINT

Removed

• Functions deprecated before 1.11.0:
  • @cumulus/api/models/base: static Manager.createTable() and static Manager.deleteTable()
  • @cumulus/ingest/aws/S3
  • @cumulus/ingest/aws/StepFunction.getExecution()
  • @cumulus/ingest/aws/StepFunction.pullEvent()
  • @cumulus/ingest/consumer.Consume
  • @cumulus/ingest/granule/Ingest.getBucket()

Deprecated

@cmrjs/ingestConcept, instead use the CMR object methods. @cmrjs/CMR.ingestGranule or @cmrjs/CMR.ingestCollection @cmrjs/searchConcept, instead use the CMR object methods. @cmrjs/CMR.searchGranules or @cmrjs/CMR.searchCollections @cmrjs/deleteConcept, instead use the CMR object methods. @cmrjs/CMR.deleteGranule or @cmrjs/CMR.deleteCollection