@cumulus/common
Advanced tools
Common libraries used in Cumulus.
npm install @cumulus/common
Cumulus is a cloud-based data ingest, archive, distribution and management prototype for NASA's future Earth science data streams.
To make a contribution, please see our contributing guidelines.
[v2.0.0] 2020-07-23
@cumulus/api-client package
CumulusApiClientError class must now be imported using
const { CumulusApiClientError } = require('@cumulus/api-client/CumulusApiClientError')@cumulus/sftp-client/SftpClient class must now be imported using
const { SftpClient } = require('@cumulus/sftp-client');@cumulus/ingest/SftpProviderClient no longer implicitly connect
when download, list, or sync are called. You must call connect on the
provider client before issuing one of those calls. Failure to do so will
result in a "Client not connected" exception being thrown.@cumulus/ingest/SftpProviderClient no longer implicitly
disconnect from the SFTP server when list is called.@cumulus/sftp-client/SftpClient must now be explicitly closed
by calling .end()@cumulus/sftp-client/SftpClient no longer implicitly connect to
the server when download, unlink, syncToS3, syncFromS3, and list are
called. You must explicitly call connect before calling one of those
methods.@cumulus/common package
cloudwatch-event.getSfEventMessageObject() now returns undefined if the
message could not be found or could not be parsed. It previously returned
null.S3KeyPairProvider.decrypt() now throws an exception if the bucket
containing the key cannot be determined.S3KeyPairProvider.decrypt() now throws an exception if the stack cannot be
determined.S3KeyPairProvider.encrypt() now throws an exception if the bucket
containing the key cannot be determined.S3KeyPairProvider.encrypt() now throws an exception if the stack cannot be
determined.sns-event.getSnsEventMessageObject() now returns undefined if it could
not be parsed. It previously returned null.aws module has been removed.BucketsConfig.buckets property is now read-only and privatetest-utils.validateConfig() function now resolves to undefined
rather than true.test-utils.validateInput() function now resolves to undefined rather
than true.test-utils.validateOutput() function now resolves to undefined
rather than true.S3KeyPairProvider.retrieveKey() function has been removed.@cumulus/cmrjs package
@cumulus/cmrjs.constructOnlineAccessUrl() and
@cumulus/cmrjs/cmr-utils.constructOnlineAccessUrl() previously took a
buckets parameter, which was an instance of
@cumulus/common/BucketsConfig. They now take a bucketTypes parameter,
which is a simple object mapping bucket names to bucket types. Example:
{ 'private-1': 'private', 'public-1': 'public' }@cumulus/cmrjs.reconcileCMRMetadata() and
@cumulus/cmrjs/cmr-utils.reconcileCMRMetadata() now take a required
bucketTypes parameter, which is a simple object mapping bucket names to
bucket types. Example: { 'private-1': 'private', 'public-1': 'public' }@cumulus/cmrjs.updateCMRMetadata() and
@cumulus/cmrjs/cmr-utils.updateCMRMetadata() previously took an optional
inBuckets parameter, which was an instance of
@cumulus/common/BucketsConfig. They now take a required bucketTypes
parameter, which is a simple object mapping bucket names to bucket types.
Example: { 'private-1': 'private', 'public-1': 'public' }cumuluss/cumulus-ecs-task Docker image must be updated to
cumuluss/cumulus-ecs-task:1.7.0. This can be done by updating the image
property of any tasks defined using the cumulus_ecs_service Terraform
module.@cumulus/aws-client/S3
getObjectSize function has changed. It now takes a
params object with three properties:
getObjectSize function will no longer retry if the object does not
exist@cumulus/message/Collections.getCollectionIdFromMessage now throws a
CumulusMessageError if collectionName and collectionVersion are missing
from meta.collection. Previously this method would return
'undefined___undefined' instead@cumulus/integration-tests/addCollections now returns an array of collections that
were added rather than the count of added collections@cumulus/common/util.uuid() function has been removed@cumulus/aws-client/S3.multipartCopyObject now returns an object with the
AWS etag of the destination object@cumulus/ingest/S3ProviderClient.list now sets a file object's path
property to undefined instead of null when the file is at the top level
of its bucketsync methods of the following classes in the @cumulus/ingest package
now return an object with the AWS s3uri and etag of the destination file
(they previously returned only a string representing the S3 URI)
FtpProviderClientHttpProviderClientS3ProviderClientSftpProviderClient@cumulus/cmr-js/cmr-utils were made
async, and added distributionBucketMap as a parameter:
DiscoverPdrs task now expects provider_path to be provided at
event.config.provider_path, not event.config.collection.provider_pathevent.config.provider_path is now a required parameter of the
DiscoverPdrs taskevent.config.collection is no longer a parameter to the DiscoverPdrs
taskprovider_path property. The tasks that
relied on that property are now referencing config.meta.provider_path.
Workflows should be updated accordingly./bulkDelete to
/granules/bulkDeletefile.hdf is the filename of the given resource) as the resource description instead of "File to download"Due to an issue with the AWS API Gateway and how the Thin Egress App Cloudformation template applies updates, you may need to redeploy your
thin-egress-app-EgressGateway manually as a one time migration step. If your deployment fails with an
error similar to:
Error: Lambda function (<stack>-tf-TeaCache) returned error: ({"errorType":"HTTPError","errorMessage":"Response code 404 (Not Found)"})
Then follow the AWS
instructions
to Redeploy a REST API to a stage for your egress API and re-run terraform apply.
CUMULUS-2081
CUMULUS-1902
CUMULUS-2058
lambda_processing_role_name as an output from the cumulus module
to provide the processing role nameCUMULUS-1417
checksumFor property to collection files config. Set this
property on a checksum file's definition matching the regex of the target
file. More details in the 'Data Cookbooks
Setup'
documentation.checksumFor validation to collections model.CUMULUS-1956
@cumulus/earthata-login-client package/s3credentials endpoint that is deployed as part of distribution now
supports authentication using tokens created by a different application. If
a request contains the EDL-ClientId and EDL-Token headers,
authentication will be handled using that token rather than attempting to
use OAuth.@cumulus/earthata-login-client.getTokenUsername() now accepts an
xRequestId argument, which will be included as the X-Request-Id header
when calling Earthdata Login.s3Credentials endpoint is invoked with an EDL token and an
X-Request-Id header, that X-Request-Id header will be forwarded to
Earthata Login.CUMULUS-1957
EDL-Client-Name header
is set, @the-client-name will be appended to the end of the Earthdata
Login username that is used as the RoleSessionName of the temporary IAM
credentials. This value will show up in the AWS S3 server access logs.CUMULUS-1958
bucket_map_key to the cumulus
terraform module as an override for the default .yaml values that are passed
to TEA by Core. Using this option requires that each configured
Cumulus 'distribution' bucket (e.g. public/protected buckets) have a single
TEA mapping. Multiple maps per bucket are not supported.distribution_bucket_map.CUMULUS-1961
CUMULUS-1970
add-missing-file-checksums workflow task@cumulus/aws-client/S3.calculateObjectHash() function@cumulus/aws-client/S3.getObjectReadStream() functionCUMULUS-1887
CUMULUS-2019
infix search to es query builder @cumulus/api/es/es/queries to
support partial matching of the keywordsCUMULUS-2032
httpListTimeout to set the default timeout for discovery HTTP/HTTPS
requests, and updates the default for the provider to 5 minutes (300 seconds).CUMULUS-176
CUMULUS-1861
sfEventSqsToDbRecords and
sfEventSqsToDbRecordsInputQueue. Previously failure to write a database
record would result in lambda success, and an error log in the CloudWatch
logs. The lambda has been updated to manually add a record to
the sfEventSqsToDbRecordsDeadLetterQueue if the granule, execution, or
pdr record fails to write, in addition to the previous error logging.CUMULUS-1956
/s3credentials endpoint that is deployed as part of distribution now
supports authentication using tokens created by a different application. If
a request contains the EDL-ClientId and EDL-Token headers,
authentication will be handled using that token rather than attempting to
use OAuth.CUMULUS-1977
/granules/bulk now returns a 202 status on a successful
response instead of a 200 response/granules/<granule-id> now returns a 404 status if the
granule record was already deleted@cumulus/api/models/Granule.update() now returns the updated granule
record/granules/bulkDelete API endpoint to support deleting
granules specified by ID or returned by the provided query in the request
body. If the request is successful, the endpoint returns the async operation
ID that has been started to remove the granules.
@cumulus/api/models/Granule.getRecord() method to return raw record
from DynamoDB@cumulus/api/models/Granule.delete() method which handles deleting
the granule record from DynamoDB and the granule files from S3CUMULUS-1982
globalConnectionLimit property of providers is now optional and
defaults to "unlimited"CUMULUS-1997
launchpad configuration to @cumulus/hyrax-metadata-updates task config schema.CUMULUS-1991
@cumulus/cmrjs/src/cmr-utils/constructOnlineAccessUrls() now throws an error if cmrGranuleUrlType = "distribution" and no distribution endpoint argument is providedCUMULUS-2011
CUMULUS-2016
CUMULUS-1991
DISTRIBUTION_ENDPOINT environment variable for API lambdas. This environment variable is required for API requests to move granules.CUMULUS-1961
@cumulus/api-client@cumulus/aws-client/S3.calculateS3ObjectChecksum()
@cumulus/aws-client/S3.getS3ObjectReadStream()
@cumulus/common/log.convertLogLevel()
@cumulus/collection-config-store
@cumulus/common/util.sleep()
CUMULUS-1930
@cumulus/common/log.convertLogLevel()@cumulus/common/util.isNull()@cumulus/common/util.isUndefined()@cumulus/common/util.negate()@cumulus/common/util.noop()@cumulus/common/util.isNil()@cumulus/common/util.renameProperty()@cumulus/common/util.lookupMimeType()@cumulus/common/util.thread()@cumulus/common/util.mkdtempSync()@cumulus/common.bucketsConfigJsonObject function has been
removed@cumulus/common.CollectionConfigStore class has been removed@cumulus/common.concurrency module has been removed@cumulus/common.constructCollectionId function has been
removed@cumulus/common.launchpad module has been removed@cumulus/common.LaunchpadToken class has been removed@cumulus/common.Semaphore class has been removed@cumulus/common.stringUtils module has been removed@cumulus/common/aws.cloudwatchlogs function has been removed@cumulus/common/aws.deleteS3Files function has been removed@cumulus/common/aws.deleteS3Object function has been removed@cumulus/common/aws.dynamodb function has been removed@cumulus/common/aws.dynamodbDocClient function has been
removed@cumulus/common/aws.getExecutionArn function has been removed@cumulus/common/aws.headObject function has been removed@cumulus/common/aws.listS3ObjectsV2 function has been removed@cumulus/common/aws.parseS3Uri function has been removed@cumulus/common/aws.promiseS3Upload function has been removed@cumulus/common/aws.recursivelyDeleteS3Bucket function has
been removed@cumulus/common/aws.s3CopyObject function has been removed@cumulus/common/aws.s3ObjectExists function has been removed@cumulus/common/aws.s3PutObject function has been removed@cumulus/common/bucketsConfigJsonObject function has been
removed@cumulus/common/CloudWatchLogger class has been removed@cumulus/common/collection-config-store.CollectionConfigStore
class has been removed@cumulus/common/collection-config-store.constructCollectionId
function has been removed@cumulus/common/concurrency.limit function has been removed@cumulus/common/concurrency.mapTolerant function has been
removed@cumulus/common/concurrency.promiseUrl function has been
removed@cumulus/common/concurrency.toPromise function has been
removed@cumulus/common/concurrency.unless function has been removed@cumulus/common/config.parseConfig function has been removed@cumulus/common/config.resolveResource function has been
removed@cumulus/common/DynamoDb.get function has been removed@cumulus/common/DynamoDb.scan function has been removed@cumulus/common/FieldPattern class has been removed@cumulus/common/launchpad.getLaunchpadToken function has been
removed@cumulus/common/launchpad.validateLaunchpadToken function has
been removed@cumulus/common/LaunchpadToken class has been removed@cumulus/common/message.buildCumulusMeta function has been
removed@cumulus/common/message.buildQueueMessageFromTemplate
function has been removed@cumulus/common/message.getCollectionIdFromMessage function
has been removed@cumulus/common/message.getMaximumExecutions function has
been removed@cumulus/common/message.getMessageExecutionArn function has
been removed@cumulus/common/message.getMessageExecutionName function has
been removed@cumulus/common/message.getMessageFromTemplate function has
been removed@cumulus/common/message.getMessageGranules function has been
removed@cumulus/common/message.getMessageStateMachineArn function
has been removed@cumulus/common/message.getQueueName function has been
removed@cumulus/common/message.getQueueNameByUrl function has been
removed@cumulus/common/message.hasQueueAndExecutionLimit function
has been removed@cumulus/common/Semaphore class has been removed@cumulus/common/string.globalReplace function has been removed@cumulus/common/string.isNonEmptyString function has been
removed@cumulus/common/string.isValidHostname function has been
removed@cumulus/common/string.match function has been removed@cumulus/common/string.matches function has been removed@cumulus/common/string.replace function has been removed@cumulus/common/string.toLower function has been removed@cumulus/common/string.toUpper function has been removed@cumulus/common/testUtils.getLocalstackEndpoint function has been removed@cumulus/common/util.setErrorStack function has been removed@cumulus/common/util.uuid function has been removed@cumulus/common/workflows.getWorkflowArn function has been
removed@cumulus/common/workflows.getWorkflowFile function has been
removed@cumulus/common/workflows.getWorkflowList function has been
removed@cumulus/common/workflows.getWorkflowTemplate function has
been removed@cumulus/aws-client/StepFunctions.toSfnExecutionName()@cumulus/aws-client/StepFunctions.fromSfnExecutionName()@cumulus/aws-client/StepFunctions.getExecutionArn()@cumulus/aws-client/StepFunctions.getExecutionUrl()@cumulus/aws-client/StepFunctions.getStateMachineArn()@cumulus/aws-client/StepFunctions.pullStepFunctionEvent()@cumulus/common/test-utils/throttleOnce()@cumulus/integration-tests/api/distribution.invokeApiDistributionLambda()@cumulus/integration-tests/api/distribution.getDistributionApiRedirect()@cumulus/integration-tests/api/distribution.getDistributionApiFileStream()FAQs
Common utilities used across tasks
The npm package @cumulus/common receives a total of 210 weekly downloads. As such, @cumulus/common popularity was classified as not popular.
We found that @cumulus/common demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket researchers uncover a malicious npm package posing as a tool for detecting vulnerabilities in Etherium smart contracts.
Security News
Research
A supply chain attack on Rspack's npm packages injected cryptomining malware, potentially impacting thousands of developers.
Research
Security News
Socket researchers discovered a malware campaign on npm delivering the Skuld infostealer via typosquatted packages, exposing sensitive data.