@cumulus/logger
Advanced tools
Changelog
[v1.15.0] - 2019-11-04
CUMULUS-1644 - When a workflow execution begins or ends, the workflow
payload is parsed and any new or updated PDRs or granules referenced in that
workflow are stored to the Cumulus archive. The defined interface says that a
PDR in payload.pdr
will be added to the archive, and any granules in
payload.granules
will also be added to the archive. In previous releases,
PDRs found in meta.pdr
and granules found in meta.input_granules
were also
added to the archive. This caused unexpected behavior and has been removed.
Only PDRs from payload.pdr
and granules from payload.granules
will now be
added to the Cumulus archive.
CUMULUS-1449 - Cumulus now uses a universal workflow template when
starting a workflow that contains general information specific to the
deployment, but not specific to the workflow. Workflow task configs must be
defined using AWS step function parameters. As part of this change,
CumulusConfig
has been retired and task configs must now be defined under
the cma.task_config
key in the Parameters section of a step function
definition.
Migration instructions:
NOTE: These instructions require the use of Cumulus Message Adapter v1.1.x+.
Please ensure you are using a compatible version before attempting to migrate
workflow configurations. When defining workflow steps, remove any
CumulusConfig
section, as shown below:
ParsePdr:
CumulusConfig:
provider: "{$.meta.provider}"
bucket: "{$.meta.buckets.internal.name}"
stack: "{$.meta.stack}"
Instead, use AWS Parameters to pass task_config
for the task directly into
the Cumulus Message Adapter:
ParsePdr:
Parameters:
cma:
event.$: "$"
task_config:
provider: "{$.meta.provider}"
bucket: "{$.meta.buckets.internal.name}"
stack: "{$.meta.stack}"
In this example, the cma
key is used to pass parameters to the message
adapter. Using task_config
in combination with event.$: '$'
allows the
message adapter to process task_config
as the config
passed to the Cumulus
task. See example/workflows/sips.yml
in the core repository for further
examples of how to set the Parameters.
Additionally, workflow configurations for the QueueGranules
and QueuePdrs
tasks need to be updated:
queue-pdrs
config changes:
parsePdrMessageTemplateUri
replaced with parsePdrWorkflow
, which is
the workflow name (i.e. top-level name in config.yml
, e.g. 'ParsePdr').internalBucket
and stackName
configs now required to look up
configuration from the deployment. Brings the task config in line with
that of queue-granules
.queue-granules
config change: ingestGranuleMessageTemplateUri
replaced
with ingestGranuleWorkflow
, which is the workflow name (e.g.
'IngestGranule').CUMULUS-1396 - Workflow steps at the beginning and end of a workflow
using the SfSnsReport
Lambda have now been deprecated (e.g. StartStatus
,
StopStatus
) and should be removed from your workflow definitions. These
steps were used for publishing ingest notifications and have been replaced by
an implementation using Cloudwatch events for Step Functions to trigger a
Lambda that publishes ingest notifications. For further detail on how ingest
notifications are published, see the notes below on CUMULUS-1394. For
examples of how to update your workflow definitions, see our
example workflow definitions.
CUMULUS-1470
AWS::ApplicationAutoScaling::ScalableTarget
,
AWS::ApplicationAutoScaling::ScalingPolicy
, and AWS::CloudWatch::Alarm
resources should be defined in a kes overrides file. See
this example
for an example.CUMULUS-1100
CUMULUS-1396
@cumulus/common/sfnStep
:
LambdaStep
- A class for retrieving and parsing input and output to Lambda steps in AWS Step FunctionsActivityStep
- A class for retrieving and parsing input and output to ECS activity steps in AWS Step FunctionsCUMULUS-1574
GET /token
endpoint for SAML authorization when cumulus is protected by Launchpad.
This lets a user retrieve a token by hand that can be presented to the API.CUMULUS-1625
sf_start_rate
variable to the ingest
Terraform module, equivalent to sqs_consumer_rate
in the old model, but will not be automatically applied to custom queues as that was.CUMULUS-1513
sqs
-type rule support in the Cumulus API @cumulus/api
sqsMessageConsumer
lambda which processes messages from the SQS queues configured in the sqs
rules.CUMULUS-1639
NoSuchKey
response even if that object has been created. The
@cumulus/common/aws.getS3Object()
function will now retry up to 10 times
if a NoSuchKey
response is returned by S3. This can behavior can be
overridden by passing { retries: 0 }
as the retryOptions
argument.CUMULUS-1449
queue-pdrs
& queue-granules
config changes. Details in breaking changes section.CumulusConfig
to a task_config
AWS Parameter.CUMULUS-1452
devicemapper
CUMULUS-1453
@cumulus/sf-sns-report
task@cumulus/sf-sns-report
to always assume that it is running as an intermediate step in a workflow, not as the first or last stepCumulusConfig
as part of step function definitions, as this is an artifact of the way Kes parses workflow definitions that was not possible to migrate to Terraform. Use AWS Parameters and the task_config
key instead. See change note above.CUMULUS-1620 - Fixed bug where message_adapter_version
does not correctly inject the CMA
CUMULUS-1396 - Updated @cumulus/common/StepFunctions.getExecutionHistory()
to recursively fetch execution history when nextToken
is returned in response
CUMULUS-1571 - Updated @cumulus/common/DynamoDb.get()
to throw any errors encountered when trying to get a record and the record does exist
CUMULUS-1452
devicemapper
Changelog
[v1.14.4] - 2019-10-28
aws-elasticsearch-connector
package in @cumulus/api
to version 8.1.3
, since 8.2.0
includes breaking changesChangelog
[v1.14.3] - 2019-10-18
CUMULUS-1620 - Fixed bug where message_adapter_version
does not correctly inject the CMA
CUMULUS-1572 - A granule is now included in discovery results even when
none of its files has a matching file type in the associated collection
configuration. Previously, if all files for a granule were unmatched by a file
type configuration, the granule was excluded from the discovery results.
Further, added support for a boolean
property
ignoreFilesConfigForDiscovery
, which controls how a granule's files are
filtered at discovery time.
Changelog
[v1.14.2] - 2019-10-08
Your Cumulus Message Adapter version should be pinned to v1.0.13
or lower in your app/config.yml
using message_adapter_version: v1.0.13
OR you should use the workflow migration steps below to work with CMA v1.1.1+.
SfSnsReport
Lambda requires additional environment variables for integration with the new ingest notification SNS topics. Therefore, you must update the definition of SfSnsReport
in your lambdas.yml
like so:SfSnsReport:
handler: index.handler
timeout: 300
source: node_modules/@cumulus/sf-sns-report/dist
tables:
- ExecutionsTable
envs:
execution_sns_topic_arn:
function: Ref
value: reportExecutionsSns
granule_sns_topic_arn:
function: Ref
value: reportGranulesSns
pdr_sns_topic_arn:
function: Ref
value: reportPdrsSns
CUMULUS-1447 - The newest release of the Cumulus Message Adapter (v1.1.1) requires that parameterized configuration be used for remote message functionality. Once released, Kes will automatically bring in CMA v1.1.1 without additional configuration.
Migration instructions
Oversized messages are no longer written to S3 automatically. In order to utilize remote messaging functionality, configure a ReplaceConfig
AWS Step Function parameter on your CMA task:
ParsePdr:
Parameters:
cma:
event.$: "$"
ReplaceConfig:
FullMessage: true
Accepted fields in ReplaceConfig
include MaxSize
, FullMessage
, Path
and TargetPath
.
See https://github.com/nasa/cumulus-message-adapter/blob/master/CONTRACT.md#remote-message-configuration for full details.
As this change is backward compatible in Cumulus Core, users wishing to utilize the previous version of the CMA may opt to transition to using a CMA lambda layer, or set message_adapter_version
in their configuration to a version prior to v1.1.0.
CUMULUS-1394 - Ingest notifications are now provided via 3 separate SNS topics for executions, granules, and PDRs, instead of a single sftracker
SNS topic. Whereas the sftracker
SNS topic received a full Cumulus execution message, the new topics all receive generated records for the given object. The new topics are only published to if the given object exists for the current execution. For a given execution/granule/PDR, two messages will be received by each topic: one message indicating that ingest is running and another message indicating that ingest has completed or failed. The new SNS topics are:
reportExecutions
- Receives 1 message per executionreportGranules
- Receives 1 message per granule in an executionreportPdrs
- Receives 1 message per PDRCUMULUS-639
<prefix>-UsersTable
GET /saml/login
- starting point for SAML SSO creates the login request url and redirects to the SAML Identity Provider Service (IDP)POST /saml/auth
- SAML Assertion Consumer Service. POST receiver from SAML IDP. Validates response, logs the user in, and returns a SAML-based JWT.POST /refresh
ensureAuthorized
now presumes the bearer token is a JWT and tries to validate. If the token is malformed, it attempts to validate the token against Launchpad. This allows users to bring their own token as described here https://wiki.earthdata.nasa.gov/display/CUMULUS/Cumulus+API+with+Launchpad+Authentication. But it also allows dashboard users to manually authenticate via Launchpad SAML to receive a Launchpad-based JWT.CUMULUS-1394
Granule.generateGranuleRecord()
method to granules model to generate a granule database record from a Cumulus execution messagePdr.generatePdrRecord()
method to PDRs model to generate a granule database record from a Cumulus execution message@cumulus/common/message
:
getMessageExecutionName()
- Get the execution name from a Cumulus execution messagegetMessageStateMachineArn()
- Get the state machine ARN from a Cumulus execution messagegetMessageExecutionArn()
- Get the execution ARN for a Cumulus execution messagegetMessageGranules()
- Get the granules from a Cumulus execution message, if any.@cumulus/common/cloudwatch-event/isFailedSfStatus()
to determine if a Step Function status from a Cloudwatch event is a failed statusCUMULUS-1308
CUMULUS-1375
@cumulus/api
CUMULUS-1485 Update @cumulus/cmr-client
to return error message from CMR for validation failures.
CUMULUS-1394
Execution.generateDocFromPayload()
to Execution.generateRecord()
on executions model. The method generates an execution database record from a Cumulus execution message.CUMULUS-1432
logs
endpoint takes the level parameter as a string and not a numberCUMULUS-1447
CUMULUS-1448 Refactor workflows that are mutating cumulus_meta to utilize meta field
CUMULUS-1451
auto_create_index
will be set to false. This had been causing issues in the bootstrap lambda on deploy.CUMULUS-1456
@cumulus/api
endpoints default error handler uses boom
package to format errors, which is consistent with other API endpoint errors.logs
endpoint filter correctly filters logs by leveluseMessageAdapter
now does not set CUMULUS_MESSAGE_ADAPTER_DIR when true
sfTracker
SNS topic. Replaced by three new SNS topics for granule, execution, and PDR ingest notifications.@cumulus/common/aws
:
getGranuleS3Params()
setGranuleStatus()
Changelog
[v1.14.1] - 2019-08-29
CUMULUS-1455
CUMULUS-1211
Changelog
[v1.13.5] - 2019-08-29 - [BACKPORT]
Changelog
[v1.14.0] - 2019-08-22
We have encountered transient lambda service errors in our integration testing. Please handle transient service errors following these guidelines. The workflows in the example/workflows
folder have been updated with retries configured for these errors.
CUMULUS-799 added additional IAM permissions to support reading CloudWatch and API Gateway, so you will have to redeploy your IAM stack.
CUMULUS-800 Several items:
Delete existing API Gateway stages: To allow enabling of API Gateway logging, Cumulus now creates and manages a Stage resource during deployment. Before upgrading Cumulus, it is necessary to delete the API Gateway stages on both the Backend API and the Distribution API. Instructions are included in the documentation under Delete API Gateway Stages.
Set up account permissions for API Gateway to write to CloudWatch: In a one time operation for your AWS account, to enable CloudWatch Logs for API Gateway, you must first grant the API Gateway permission to read and write logs to CloudWatch for your account. The AmazonAPIGatewayPushToCloudWatchLogs
managed policy (with an ARN of arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs
) has all the required permissions. You can find a simple how to in the documentation under Enable API Gateway Logging.
Configure API Gateway to write logs to CloudWatch To enable execution logging for the distribution API set config.yaml
apiConfigs.distribution.logApigatewayToCloudwatch
value to true
. More information Enable API Gateway Logs
Configure CloudWatch log delivery: It is possible to deliver CloudWatch API execution and access logs to a cross-account shared AWS::Logs::Destination. An operator does this by adding the key logToSharedDestination
to the config.yml
at the default level with a value of a writable log destination. More information in the documentation under Configure CloudWatch Logs Delivery.
Additional Lambda Logging: It is now possible to configure any lambda to deliver logs to a shared subscriptions by setting logToSharedDestination
to the ARN of a writable location (either an AWS::Logs::Destination or a Kinesis Stream) on any lambda config. Documentation for Lambda Log Subscriptions
Configure S3 Server Access Logs: If you are running Cumulus in an NGAP environment you may configure S3 Server Access Logs to be delivered to a shared bucket where the Metrics Team will ingest the logs into their ELK stack. Contact the Metrics team for permission and location.
CUMULUS-1368 The Cumulus distribution API has been deprecated and is being replaced by ASF's Thin Egress App. By default, the distribution API will not deploy. Please follow the instructions for deploying and configuring Thin Egress.
To instead continue to deploy and use the legacy Cumulus distribution app, add the following to your config.yml
:
deployDistributionApi: true
If you deploy with no distribution app your deployment will succeed but you may encounter errors in your workflows, particularly in the MoveGranule
task.
BREAKING CHANGES
below for details.CUMULUS-642
CUMULUS-1418
cumulus-ecs-task
and a CMA layer instead of kes CMA injection.PR1125 - Adds layers
config option to support deploying Lambdas with layers
PR1128 - Added useXRay
config option to enable AWS X-Ray for Lambdas.
CUMULUS-1345
cmr
.cmrEnvironment
values are SIT
, UAT
, or OPS
with UAT
as the default.cmrLimit
and cmrPageSize
have been added as configurable options.CUMULUS-1273
CUMULUS-1226
elasticsearch/index-from-database
to index to an Elasticsearch index from the database for recovery purposes and elasticsearch/indices-status
to check the status of Elasticsearch indices via the API.CUMULUS-824
reportToEms
to configure whether the collection is reported to EMSCUMULUS-1357
ems
that generates EMS reports.CUMULUS-1241
meta.queueExecutionLimits
)CUMULUS-1311
@cumulus/common/message
with various message parsing/preparation helpersCUMULUS-812
CUMULUS-1337
cumulus.stackName
value to the instanceMetadata
endpoint.CUMULUS-1368
cmrGranuleUrlType
to the @cumulus/move-granules
task. This determines what kind of links go in the CMR files. The options are distribution
, s3
, or none
, with the default being distribution. If there is no distribution API being used with Cumulus, you must set the value to s3
or none
.Added packages/s3-replicator
Terraform module to allow same-region s3 replication to metrics bucket.
CUMULUS-1392
tf-modules/report-granules
Terraform module which processes granule ingest notifications received via SNS and stores granule data to a database. The module includes:
CUMULUS-1393
tf-modules/report-pdrs
Terraform module which processes PDR ingest notifications received via SNS and stores PDR data to a database. The module includes:
@cumulus/api/models/pdrs.createPdrFromSns()
CUMULUS-1400
tf-modules/report-executions
Terraform module which processes workflow execution information received via SNS and stores it to a database. The module includes:
@cumulus/common/sns-event
which contains helpers for SNS events:
isSnsEvent()
returns true if event is from SNSgetSnsEventMessage()
extracts and parses the message from an SNS eventgetSnsEventMessageObject()
extracts and parses message object from an SNS event@cumulus/common/cloudwatch-event
which contains helpers for Cloudwatch events:
isSfExecutionEvent()
returns true if event is from Step FunctionsisTerminalSfStatus()
determines if a Step Function status from a Cloudwatch event is a terminal statusgetSfEventStatus()
gets the Step Function status from a Cloudwatch eventgetSfEventDetailValue()
extracts a Step Function event detail field from a Cloudwatch eventgetSfEventMessageObject()
extracts and parses Step Function detail object from a Cloudwatch eventCUMULUS-1429
tf-modules/data-persistence
Terraform module which includes resources for data persistence in Cumulus:
CUMULUS-1379 CMR Launchpad Authentication
launchpad
configuration to @cumulus/deployment/app/config.yml
, and cloudformation templates, workflow message, lambda configuration, api endpoint configuration@cumulus/common/LaunchpadToken
and @cumulus/common/launchpad
to provide methods to get token and validate tokenCUMULUS-1232
@cumulus/cmr-client
updateToken()
CUMULUS-1245 CUMULUS-795
ems
configuration parameters for sending the ingest reports to EMSCUMULUS-1241
sqs2sfThrottle
lambda:
cumulus_meta.queueName
meta.queueExecutionLimits[queueName]
, where queueName
is cumulus_meta.queueName
sfSemaphoreDown
lambda to only attempt decrementing semaphores when:
cumulus_meta.queueName
exists on the Cumulus message ANDcumulus_meta.queueName
) exists in the the object meta.queueExecutionLimits
on the Cumulus messageCUMULUS-1338
sfSemaphoreDown
lambda to be triggered via AWS Step Function Cloudwatch events instead of subscription to sfTracker
SNS topicCUMULUS-1311
@cumulus/queue-granules
to set cumulus_meta.queueName
for queued execution messages@cumulus/queue-pdrs
to set cumulus_meta.queueName
for queued execution messagessqs2sfThrottle
lambda to immediately decrement queue semaphore value if dispatching Step Function execution throws an errorCUMULUS-1362
processingStartTime
and processingEndTime
will be set to the execution start time and end time respectively when there is no sync granule or post to cmr task present in the workflowCUMULUS-1400
@cumulus/ingest/aws/getExecutionArn
. Use @cumulus/common/aws/getExecutionArn
instead.CUMULUS-1439
CUMULUS-796
CUMULUS-1319
CUMULUS-1356
Collection
model's delete
method now removes the specified item
from the collection config store that was inserted by the create
method.
Previously, this behavior was missing.CUMULUS-1374
cmaDir
key to configuration will cause CUMULUS_MESSAGE_ADAPTER_DIR
to be set by default to /opt
for any Lambda not setting useCma
to true, or explicitly setting the CMA environment variable. In lambdas that package the CMA independently of the Cumulus packaging. Lambdas manually packaging the CMA should have their Lambda configuration updated to set the CMA path, or alternately if not using the CMA as a Lambda layer in this deployment set cmaDir
to ./cumulus-message-adapter
.CUMULUS-1337
PR1130
@cumulus/common/step-functions
. Use @cumulus/common/StepFunctions
instead.@cumulus/api/lib/testUtils.fakeFilesFactory
. Use @cumulus/api/lib/testUtils.fakeFileFactory
instead.@cumulus/cmrjs/cmr
functions: searchConcept
, ingestConcept
, deleteConcept
. Use the functions in @cumulus/cmr-client
instead.@cumulus/ingest/aws.getExecutionHistory
. Use @cumulus/common/StepFunctions.getExecutionHistory
instead.Changelog
[v1.13.4] - 2019-07-29
Changelog
[v1.13.3] - 2019-07-26
cmr
.
cmrEnvironment
values are SIT
, UAT
, or OPS
with UAT
as the default.cmrLimit
and cmrPageSize
have been added as configurable options.