@cumulus/logger
Advanced tools
Changelog
[v2.0.5] 2020-09-15 - [BACKPORT]
thin_egress_stack_name
variable to cumulus
and distribution
Terraform modules to allow overriding the default Cloudformation stack name used for the thin-egress-app
. Please note that if you change/set this value for an existing deployment, it will destroy and re-create your API gateway for the thin-egress-app
.Changelog
[v2.0.3] 2020-09-02 - [BACKPORT]
CUMULUS-1961
activeCollections
query only returning 10 resultsCUMULUS-2039
@cumulus/aws-client/S3.getS3ObjectReadStreamAsync
function has been
removed. It read the entire S3 object into memory before returning a read
stream, which could cause Lambdas to run out of memory. Use
@cumulus/aws-client/S3.getObjectReadStream
instead.Changelog
v2.0.2+ Backport releases
Release v2.0.1 was the last release on the 2.0.x release series.
Changes after this version on the 2.0.x release series are limited security/requested feature patches and will not be ported forward to future releases unless there is a corresponding CHANGELOG entry.
For up-to-date CHANGELOG for the maintenance release branch see CHANGELOG.md from the 2.0.x branch.
For the most recent release information for the maintenance branch please see the release page
Changelog
[v2.0.1] 2020-07-28
multiple sort keys
support to @cumulus/api
@cumulus/message/Queue.getQueueUrl
to get the queue URL specified in a Cumulus workflow message, if any.@cumulus/launchpad-auth
causing Launchpad token requests to failChangelog
[v2.0.0] 2020-07-23
@cumulus/api-client
package
CumulusApiClientError
class must now be imported using
const { CumulusApiClientError } = require('@cumulus/api-client/CumulusApiClientError')
@cumulus/sftp-client/SftpClient
class must now be imported using
const { SftpClient } = require('@cumulus/sftp-client');
@cumulus/ingest/SftpProviderClient
no longer implicitly connect
when download
, list
, or sync
are called. You must call connect
on the
provider client before issuing one of those calls. Failure to do so will
result in a "Client not connected" exception being thrown.@cumulus/ingest/SftpProviderClient
no longer implicitly
disconnect from the SFTP server when list
is called.@cumulus/sftp-client/SftpClient
must now be explicitly closed
by calling .end()
@cumulus/sftp-client/SftpClient
no longer implicitly connect to
the server when download
, unlink
, syncToS3
, syncFromS3
, and list
are
called. You must explicitly call connect
before calling one of those
methods.@cumulus/common
package
cloudwatch-event.getSfEventMessageObject()
now returns undefined
if the
message could not be found or could not be parsed. It previously returned
null
.S3KeyPairProvider.decrypt()
now throws an exception if the bucket
containing the key cannot be determined.S3KeyPairProvider.decrypt()
now throws an exception if the stack cannot be
determined.S3KeyPairProvider.encrypt()
now throws an exception if the bucket
containing the key cannot be determined.S3KeyPairProvider.encrypt()
now throws an exception if the stack cannot be
determined.sns-event.getSnsEventMessageObject()
now returns undefined
if it could
not be parsed. It previously returned null
.aws
module has been removed.BucketsConfig.buckets
property is now read-only and privatetest-utils.validateConfig()
function now resolves to undefined
rather than true
.test-utils.validateInput()
function now resolves to undefined
rather
than true
.test-utils.validateOutput()
function now resolves to undefined
rather than true
.S3KeyPairProvider.retrieveKey()
function has been removed.@cumulus/cmrjs
package
@cumulus/cmrjs.constructOnlineAccessUrl()
and
@cumulus/cmrjs/cmr-utils.constructOnlineAccessUrl()
previously took a
buckets
parameter, which was an instance of
@cumulus/common/BucketsConfig
. They now take a bucketTypes
parameter,
which is a simple object mapping bucket names to bucket types. Example:
{ 'private-1': 'private', 'public-1': 'public' }
@cumulus/cmrjs.reconcileCMRMetadata()
and
@cumulus/cmrjs/cmr-utils.reconcileCMRMetadata()
now take a required
bucketTypes
parameter, which is a simple object mapping bucket names to
bucket types. Example: { 'private-1': 'private', 'public-1': 'public' }
@cumulus/cmrjs.updateCMRMetadata()
and
@cumulus/cmrjs/cmr-utils.updateCMRMetadata()
previously took an optional
inBuckets
parameter, which was an instance of
@cumulus/common/BucketsConfig
. They now take a required bucketTypes
parameter, which is a simple object mapping bucket names to bucket types.
Example: { 'private-1': 'private', 'public-1': 'public' }
cumuluss/cumulus-ecs-task
Docker image must be updated to
cumuluss/cumulus-ecs-task:1.7.0
. This can be done by updating the image
property of any tasks defined using the cumulus_ecs_service
Terraform
module.@cumulus/aws-client/S3
getObjectSize
function has changed. It now takes a
params object with three properties:
getObjectSize
function will no longer retry if the object does not
exist@cumulus/message/Collections.getCollectionIdFromMessage
now throws a
CumulusMessageError
if collectionName
and collectionVersion
are missing
from meta.collection
. Previously this method would return
'undefined___undefined'
instead@cumulus/integration-tests/addCollections
now returns an array of collections that
were added rather than the count of added collections@cumulus/common/util.uuid()
function has been removed@cumulus/aws-client/S3.multipartCopyObject
now returns an object with the
AWS etag
of the destination object@cumulus/ingest/S3ProviderClient.list
now sets a file object's path
property to undefined
instead of null
when the file is at the top level
of its bucketsync
methods of the following classes in the @cumulus/ingest
package
now return an object with the AWS s3uri
and etag
of the destination file
(they previously returned only a string representing the S3 URI)
FtpProviderClient
HttpProviderClient
S3ProviderClient
SftpProviderClient
@cumulus/cmr-js/cmr-utils
were made
async, and added distributionBucketMap as a parameter:
DiscoverPdrs
task now expects provider_path
to be provided at
event.config.provider_path
, not event.config.collection.provider_path
event.config.provider_path
is now a required parameter of the
DiscoverPdrs
taskevent.config.collection
is no longer a parameter to the DiscoverPdrs
taskprovider_path
property. The tasks that
relied on that property are now referencing config.meta.provider_path
.
Workflows should be updated accordingly./bulkDelete
to
/granules/bulkDelete
file.hdf
is the filename of the given resource) as the resource description instead of "File to download"Due to an issue with the AWS API Gateway and how the Thin Egress App Cloudformation template applies updates, you may need to redeploy your
thin-egress-app-EgressGateway
manually as a one time migration step. If your deployment fails with an
error similar to:
Error: Lambda function (<stack>-tf-TeaCache) returned error: ({"errorType":"HTTPError","errorMessage":"Response code 404 (Not Found)"})
Then follow the AWS
instructions
to Redeploy a REST API to a stage
for your egress API and re-run terraform apply
.
CUMULUS-2081
CUMULUS-1902
CUMULUS-2058
lambda_processing_role_name
as an output from the cumulus
module
to provide the processing role nameCUMULUS-1417
checksumFor
property to collection files
config. Set this
property on a checksum file's definition matching the regex
of the target
file. More details in the 'Data Cookbooks
Setup'
documentation.checksumFor
validation to collections model.CUMULUS-1956
@cumulus/earthata-login-client
package/s3credentials
endpoint that is deployed as part of distribution now
supports authentication using tokens created by a different application. If
a request contains the EDL-ClientId
and EDL-Token
headers,
authentication will be handled using that token rather than attempting to
use OAuth.@cumulus/earthata-login-client.getTokenUsername()
now accepts an
xRequestId
argument, which will be included as the X-Request-Id
header
when calling Earthdata Login.s3Credentials
endpoint is invoked with an EDL token and an
X-Request-Id
header, that X-Request-Id
header will be forwarded to
Earthata Login.CUMULUS-1957
EDL-Client-Name
header
is set, @the-client-name
will be appended to the end of the Earthdata
Login username that is used as the RoleSessionName
of the temporary IAM
credentials. This value will show up in the AWS S3 server access logs.CUMULUS-1958
bucket_map_key
to the cumulus
terraform module as an override for the default .yaml values that are passed
to TEA by Core. Using this option requires that each configured
Cumulus 'distribution' bucket (e.g. public/protected buckets) have a single
TEA mapping. Multiple maps per bucket are not supported.distribution_bucket_map
.CUMULUS-1961
CUMULUS-1970
add-missing-file-checksums
workflow task@cumulus/aws-client/S3.calculateObjectHash()
function@cumulus/aws-client/S3.getObjectReadStream()
functionCUMULUS-1887
CUMULUS-2019
infix
search to es query builder @cumulus/api/es/es/queries
to
support partial matching of the keywordsCUMULUS-2032
httpListTimeout
to set the default timeout for discovery HTTP/HTTPS
requests, and updates the default for the provider to 5 minutes (300 seconds).CUMULUS-176
CUMULUS-1861
sfEventSqsToDbRecords
and
sfEventSqsToDbRecordsInputQueue
. Previously failure to write a database
record would result in lambda success, and an error log in the CloudWatch
logs. The lambda has been updated to manually add a record to
the sfEventSqsToDbRecordsDeadLetterQueue
if the granule, execution, or
pdr record fails to write, in addition to the previous error logging.CUMULUS-1956
/s3credentials
endpoint that is deployed as part of distribution now
supports authentication using tokens created by a different application. If
a request contains the EDL-ClientId
and EDL-Token
headers,
authentication will be handled using that token rather than attempting to
use OAuth.CUMULUS-1977
/granules/bulk
now returns a 202 status on a successful
response instead of a 200 response/granules/<granule-id>
now returns a 404 status if the
granule record was already deleted@cumulus/api/models/Granule.update()
now returns the updated granule
record/granules/bulkDelete
API endpoint to support deleting
granules specified by ID or returned by the provided query in the request
body. If the request is successful, the endpoint returns the async operation
ID that has been started to remove the granules.
@cumulus/api/models/Granule.getRecord()
method to return raw record
from DynamoDB@cumulus/api/models/Granule.delete()
method which handles deleting
the granule record from DynamoDB and the granule files from S3CUMULUS-1982
globalConnectionLimit
property of providers is now optional and
defaults to "unlimited"CUMULUS-1997
launchpad
configuration to @cumulus/hyrax-metadata-updates
task config schema.CUMULUS-1991
@cumulus/cmrjs/src/cmr-utils/constructOnlineAccessUrls()
now throws an error if cmrGranuleUrlType = "distribution"
and no distribution endpoint argument is providedCUMULUS-2011
CUMULUS-2016
CUMULUS-1991
DISTRIBUTION_ENDPOINT
environment variable for API lambdas. This environment variable is required for API requests to move granules.CUMULUS-1961
@cumulus/api-client
@cumulus/aws-client/S3.calculateS3ObjectChecksum()
@cumulus/aws-client/S3.getS3ObjectReadStream()
@cumulus/common/log.convertLogLevel()
@cumulus/collection-config-store
@cumulus/common/util.sleep()
CUMULUS-1930
@cumulus/common/log.convertLogLevel()
@cumulus/common/util.isNull()
@cumulus/common/util.isUndefined()
@cumulus/common/util.negate()
@cumulus/common/util.noop()
@cumulus/common/util.isNil()
@cumulus/common/util.renameProperty()
@cumulus/common/util.lookupMimeType()
@cumulus/common/util.thread()
@cumulus/common/util.mkdtempSync()
@cumulus/common.bucketsConfigJsonObject
function has been
removed@cumulus/common.CollectionConfigStore
class has been removed@cumulus/common.concurrency
module has been removed@cumulus/common.constructCollectionId
function has been
removed@cumulus/common.launchpad
module has been removed@cumulus/common.LaunchpadToken
class has been removed@cumulus/common.Semaphore
class has been removed@cumulus/common.stringUtils
module has been removed@cumulus/common/aws.cloudwatchlogs
function has been removed@cumulus/common/aws.deleteS3Files
function has been removed@cumulus/common/aws.deleteS3Object
function has been removed@cumulus/common/aws.dynamodb
function has been removed@cumulus/common/aws.dynamodbDocClient
function has been
removed@cumulus/common/aws.getExecutionArn
function has been removed@cumulus/common/aws.headObject
function has been removed@cumulus/common/aws.listS3ObjectsV2
function has been removed@cumulus/common/aws.parseS3Uri
function has been removed@cumulus/common/aws.promiseS3Upload
function has been removed@cumulus/common/aws.recursivelyDeleteS3Bucket
function has
been removed@cumulus/common/aws.s3CopyObject
function has been removed@cumulus/common/aws.s3ObjectExists
function has been removed@cumulus/common/aws.s3PutObject
function has been removed@cumulus/common/bucketsConfigJsonObject
function has been
removed@cumulus/common/CloudWatchLogger
class has been removed@cumulus/common/collection-config-store.CollectionConfigStore
class has been removed@cumulus/common/collection-config-store.constructCollectionId
function has been removed@cumulus/common/concurrency.limit
function has been removed@cumulus/common/concurrency.mapTolerant
function has been
removed@cumulus/common/concurrency.promiseUrl
function has been
removed@cumulus/common/concurrency.toPromise
function has been
removed@cumulus/common/concurrency.unless
function has been removed@cumulus/common/config.parseConfig
function has been removed@cumulus/common/config.resolveResource
function has been
removed@cumulus/common/DynamoDb.get
function has been removed@cumulus/common/DynamoDb.scan
function has been removed@cumulus/common/FieldPattern
class has been removed@cumulus/common/launchpad.getLaunchpadToken
function has been
removed@cumulus/common/launchpad.validateLaunchpadToken
function has
been removed@cumulus/common/LaunchpadToken
class has been removed@cumulus/common/message.buildCumulusMeta
function has been
removed@cumulus/common/message.buildQueueMessageFromTemplate
function has been removed@cumulus/common/message.getCollectionIdFromMessage
function
has been removed@cumulus/common/message.getMaximumExecutions
function has
been removed@cumulus/common/message.getMessageExecutionArn
function has
been removed@cumulus/common/message.getMessageExecutionName
function has
been removed@cumulus/common/message.getMessageFromTemplate
function has
been removed@cumulus/common/message.getMessageGranules
function has been
removed@cumulus/common/message.getMessageStateMachineArn
function
has been removed@cumulus/common/message.getQueueName
function has been
removed@cumulus/common/message.getQueueNameByUrl
function has been
removed@cumulus/common/message.hasQueueAndExecutionLimit
function
has been removed@cumulus/common/Semaphore
class has been removed@cumulus/common/string.globalReplace
function has been removed@cumulus/common/string.isNonEmptyString
function has been
removed@cumulus/common/string.isValidHostname
function has been
removed@cumulus/common/string.match
function has been removed@cumulus/common/string.matches
function has been removed@cumulus/common/string.replace
function has been removed@cumulus/common/string.toLower
function has been removed@cumulus/common/string.toUpper
function has been removed@cumulus/common/testUtils.getLocalstackEndpoint
function has been removed@cumulus/common/util.setErrorStack
function has been removed@cumulus/common/util.uuid
function has been removed@cumulus/common/workflows.getWorkflowArn
function has been
removed@cumulus/common/workflows.getWorkflowFile
function has been
removed@cumulus/common/workflows.getWorkflowList
function has been
removed@cumulus/common/workflows.getWorkflowTemplate
function has
been removed@cumulus/aws-client/StepFunctions.toSfnExecutionName()
@cumulus/aws-client/StepFunctions.fromSfnExecutionName()
@cumulus/aws-client/StepFunctions.getExecutionArn()
@cumulus/aws-client/StepFunctions.getExecutionUrl()
@cumulus/aws-client/StepFunctions.getStateMachineArn()
@cumulus/aws-client/StepFunctions.pullStepFunctionEvent()
@cumulus/common/test-utils/throttleOnce()
@cumulus/integration-tests/api/distribution.invokeApiDistributionLambda()
@cumulus/integration-tests/api/distribution.getDistributionApiRedirect()
@cumulus/integration-tests/api/distribution.getDistributionApiFileStream()
Changelog
[v1.24.0] 2020-06-03
CUMULUS-1969
DiscoverPdrs
task now expects provider_path
to be provided at
event.config.provider_path
, not event.config.collection.provider_path
event.config.provider_path
is now a required parameter of the
DiscoverPdrs
taskevent.config.collection
is no longer a parameter to the DiscoverPdrs
taskprovider_path
property. The tasks that
relied on that property are now referencing config.meta.provider_path
.
Workflows should be updated accordingly.CUMULUS-1997
@cumulus/cmr-client/CMRSearchConceptQueue
parameters have been changed to take a cmrSettings
object containing clientId, provider, and auth information. This can be generated using @cumulus/cmrjs/cmr-utils/getCmrSettings
. The cmrEnvironment
variable has been removed.syncChecksumFiles
to the
SyncGranule task. This setting is false
by default, but when set to
true
, all checksum files associated with data files that are downloaded
will be downloaded as well.launchpad
configuration to @cumulus/hyrax-metadata-updates
task config schema.@cumulus/api/launchpadSaml
to support multiple userGroup attributes from the SAML responseChangelog
[v1.23.2] 2020-05-22
Updates to the Cumulus archive API:
401
response instead of a 403
for any request where the JWT passed as a Bearer token is invalid./refresh
and DELETE /token/<token>
endpoints now return a 401
response for requests with expired tokensCUMULUS-1894
@cumulus/ingest/granule.handleDuplicateFile()
copyOptions
parameter has been removedACL
parameter has been added@cumulus/ingest/granule.renameS3FileWithTimestamp()
undefined
CUMULUS-1896 Updated all Cumulus core lambdas to utilize the new message adapter streaming interface via cumulus-message-adapter-js v1.2.0. Users of this version of Cumulus (or later) must utilize version 1.3.0 or greater of the cumulus-message-adapter to support core lambdas.
CUMULUS-1912
@cumulus/api
reconciliationReports list endpoint returns a list of reconciliationReport records instead of S3Uri.CUMULUS-1969
DiscoverGranules
task now expects provider_path
to be provided at
event.config.provider_path
, not event.config.collection.provider_path
config.provider_path
is now a required parameter of the DiscoverGranules
taskTo take advantage of the new TTL-based access token expiration implemented in CUMULUS-1777 (see notes below) and clear out existing records in your access tokens table, do the following:
<prefix>-AccessTokensTable
DynamoDB tabledata-persistence
module, which should re-create the <prefix>-AccessTokensTable
DynamoDB tableThis release requires the Cumulus Message Adapter layer deployed with Cumulus Core to be at least 1.3.0, as the core lambdas have updated to cumulus-message-adapter-js v1.2.0 and the new CMA interface. As a result, users should:
cumulus-message-adapter-js
, you must update your lambda to use cumulus-message-adapter-js v1.2.0 and follow the migration instructions in the release notes. Prior versions of cumulus-message-adapter-js
are not compatible with CMA >= 1.3.0.Migrate existing s3 reconciliation report records to database (CUMULUS-1911):
data persistence
module and Cumulus resources, run the command:./node_modules/.bin/cumulus-api migrate --stack `<your-terraform-deployment-prefix>` --migrationVersion migration5
Added a limit for concurrent Elasticsearch requests when doing an index from database operation
Added the es_request_concurrency
parameter to the archive and cumulus Terraform modules
CUMULUS-1995
es_index_shards
parameter to the archive and cumulus Terraform modules to configure the number of shards for the ES index
CUMULUS-1894
@cumulus/aws-client/S3.moveObject()
CUMULUS-1911
@cumulus/api
package, tf-modules/archive
and tf-modules/data-persistence
Terraform modulesCUMULUS-1916
expirationTime
property is now a required field of the access tokens model.AccessTokens
table to set a TTL on the expirationTime
field in tf-modules/data-persistence/dynamo.tf
. As a result, access token records in this table whose expirationTime
has passed should be automatically deleted by DynamoDB.AccessTokens
table to set the expirationTime
field value in seconds from the epoch.@cumulus/api-client/reconciliationReports
@cumulus/common/util.deprecate()
so that only a single deprecation notice is printed for each name/version combinationSyncGranule
task can now handle files larger than 5 GBRemove granule from CMR
operation in @cumulus/api
now passes token to CMR when fetching granule metadata, allowing removal of private granulessqs-message-consumer
Lambda will now only schedule workflows for rules matching the queue and the collection information in each queue message (if any)
CUMULUS-1894
@cumulus/ingest/granule.copyGranuleFile()
@cumulus/ingest/granule.moveGranuleFile()
CUMULUS-1987 - Deprecated the following functions:
@cumulus/cmrjs/getMetadata(cmrLink)
-> @cumulus/cmr-client/CMR.getGranuleMetadata(cmrLink)
@cumulus/cmrjs/getFullMetadata(cmrLink)