@cumulus/logger
A logging library for use on the Cumulus project.
The @cumulus/logger
library exports a Logger
class, which generates
JSON-formated log events.
Log events always have the following keys:
- level (string) - the priority level of the message
- message (string) - the message itself
- sender (string) - the resource that created the message
- timestamp (string) - an ISO-8601 timestamp
Log events may have the following options keys:
- executions (string)
- version (string)
Log events may also contain user-specified keys.
Example log event:
{
"level": "info",
"message": "hello world",
"sender": "greetingFunction",
"timestamp": "2018-10-19T19:12:47.501Z"
}
Install
$ npm install @cumulus/logger
Usage
const Logger = require('@cumulus/logger');
const log = new Logger({ sender: 'example' });
log.info('hello, world');
API
new Logger({ [sender], [executions], [version] })
sender
Type: string
The sender of the log event. Typically a Lambda Function Name or ECS Task Name.
Defaults to "unknown".
executions
Type: string
An optional description of the executions.
version
Type: string
An optional version.
log.debug([...messageArgs])
Writes a log event to stdout with level set to "debug".
args
Type: ...any
See console.log().
log.error([...messageArgs][, error])
Writes a log event to stderr with level set to "error".
args
Type: ...any
See console.log().
error
Type: Error
If the last argument is an Error then the following additional properties will be set on the log event:
- error (Object)
- name (string)
- message (string)
- stack (Array<string>) - the lines of the stack trace
log.fatal([...messageArgs])
Writes a log event to stdout with level set to "fatal".
args
Type: ...any
See console.log().
log.info([...messageArgs])
Writes a log event to stdout with level set to "info".
args
Type: ...any
See console.log().
log.infoWithAdditionalKeys(additionalKeys, ...messageArgs)
Writes a log event to stdout with level set to "info". In addition to the
standard keys, additional keys will be added to the event. If an additional key
is specified with the same name as a standard key, the value standard key will
be displayed.
additionalKeys
Type: Object
Additional key/value pairs to be added to the event.
args
Type: ...any
See console.log().
log.trace([...messageArgs])
Writes a log event to stdout with level set to "trace".
args
Type: ...any
See console.log().
log.warn([...messageArgs])
Writes a log event to stdout with level set to "debug".
args
Type: ...any
See console.log().
What is Cumulus?
Cumulus is a cloud-based data ingest, archive, distribution and management
prototype for NASA's future Earth science data streams.
Cumulus Documentation
Contributing
See Cumulus README
[v1.14.0] - 2019-08-22
PLEASE NOTE
-
We have encountered transient lambda service errors in our integration testing. Please handle transient service errors following these guidelines. The workflows in the example/workflows
folder have been updated with retries configured for these errors.
-
CUMULUS-799 added additional IAM permissions to support reading CloudWatch and API Gateway, so you will have to redeploy your IAM stack.
-
CUMULUS-800 Several items:
-
Delete existing API Gateway stages: To allow enabling of API Gateway logging, Cumulus now creates and manages a Stage resource during deployment. Before upgrading Cumulus, it is necessary to delete the API Gateway stages on both the Backend API and the Distribution API. Instructions are included in the documentation under Delete API Gateway Stages.
-
Set up account permissions for API Gateway to write to CloudWatch: In a one time operation for your AWS account, to enable CloudWatch Logs for API Gateway, you must first grant the API Gateway permission to read and write logs to CloudWatch for your account. The AmazonAPIGatewayPushToCloudWatchLogs
managed policy (with an ARN of arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs
) has all the required permissions. You can find a simple how to in the documentation under Enable API Gateway Logging.
-
Configure API Gateway to write logs to CloudWatch To enable execution logging for the distribution API set config.yaml
apiConfigs.distribution.logApigatewayToCloudwatch
value to true
. More information Enable API Gateway Logs
-
Configure CloudWatch log delivery: It is possible to deliver CloudWatch API execution and access logs to a cross-account shared AWS::Logs::Destination. An operator does this by adding the key logToSharedDestination
to the config.yml
at the default level with a value of a writable log destination. More information in the documentation under Configure CloudWatch Logs Delivery.
-
Additional Lambda Logging: It is now possible to configure any lambda to deliver logs to a shared subscriptions by setting logToSharedDestination
to the ARN of a writable location (either an AWS::Logs::Destination or a Kinesis Stream) on any lambda config. Documentation for Lambda Log Subscriptions
-
Configure S3 Server Access Logs: If you are running Cumulus in an NGAP environment you may configure S3 Server Access Logs to be delivered to a shared bucket where the Metrics Team will ingest the logs into their ELK stack. Contact the Metrics team for permission and location.
-
CUMULUS-1368 The Cumulus distribution API has been deprecated and is being replaced by ASF's Thin Egress App. By default, the distribution API will not deploy. Please follow the instructions for deploying and configuring Thin Egress.
To instead continue to deploy and use the legacy Cumulus distribution app, add the following to your config.yml
:
deployDistributionApi: true
If you deploy with no distribution app your deployment will succeed but you may encounter errors in your workflows, particularly in the MoveGranule
task.
- CUMULUS-1418 Users who are packaging the CMA in their Lambdas outside of Cumulus may need to update their Lambda configuration. Please see
BREAKING CHANGES
below for details.
Added
-
CUMULUS-642
-
CUMULUS-1418
- Adds usage docs/testing of lambda layers (introduced in PR1125), updates Core example tasks to use the updated
cumulus-ecs-task
and a CMA layer instead of kes CMA injection. - Added Terraform module to publish CMA as layer to user account.
-
PR1125 - Adds layers
config option to support deploying Lambdas with layers
-
PR1128 - Added useXRay
config option to enable AWS X-Ray for Lambdas.
-
CUMULUS-1345
- Adds new variables to the app deployment under
cmr
. cmrEnvironment
values are SIT
, UAT
, or OPS
with UAT
as the default.cmrLimit
and cmrPageSize
have been added as configurable options.
-
CUMULUS-1273
- Added lambda function EmsProductMetadataReport to generate EMS Product Metadata report
-
CUMULUS-1226
- Added API endpoint
elasticsearch/index-from-database
to index to an Elasticsearch index from the database for recovery purposes and elasticsearch/indices-status
to check the status of Elasticsearch indices via the API.
-
CUMULUS-824
- Added new Collection parameter
reportToEms
to configure whether the collection is reported to EMS
-
CUMULUS-1357
- Added new BackendApi endpoint
ems
that generates EMS reports.
-
CUMULUS-1241
- Added information about queues with maximum execution limits defined to default workflow templates (
meta.queueExecutionLimits
)
-
CUMULUS-1311
- Added
@cumulus/common/message
with various message parsing/preparation helpers
-
CUMULUS-812
- Added support for limiting the number of concurrent executions started from a queue. See the data cookbook for more information.
-
CUMULUS-1337
- Adds
cumulus.stackName
value to the instanceMetadata
endpoint.
-
CUMULUS-1368
- Added
cmrGranuleUrlType
to the @cumulus/move-granules
task. This determines what kind of links go in the CMR files. The options are distribution
, s3
, or none
, with the default being distribution. If there is no distribution API being used with Cumulus, you must set the value to s3
or none
.
-
Added packages/s3-replicator
Terraform module to allow same-region s3 replication to metrics bucket.
-
CUMULUS-1392
- Added
tf-modules/report-granules
Terraform module which processes granule ingest notifications received via SNS and stores granule data to a database. The module includes:
- SNS topic for publishing granule ingest notifications
- Lambda to process granule notifications and store data
- IAM permissions for the Lambda
- Subscription for the Lambda to the SNS topic
-
CUMULUS-1393
- Added
tf-modules/report-pdrs
Terraform module which processes PDR ingest notifications received via SNS and stores PDR data to a database. The module includes:
- SNS topic for publishing PDR ingest notifications
- Lambda to process PDR notifications and store data
- IAM permissions for the Lambda
- Subscription for the Lambda to the SNS topic
- Added unit tests for
@cumulus/api/models/pdrs.createPdrFromSns()
-
CUMULUS-1400
- Added
tf-modules/report-executions
Terraform module which processes workflow execution information received via SNS and stores it to a database. The module includes:
- SNS topic for publishing execution data
- Lambda to process and store execution data
- IAM permissions for the Lambda
- Subscription for the Lambda to the SNS topic
- Added
@cumulus/common/sns-event
which contains helpers for SNS events:
isSnsEvent()
returns true if event is from SNSgetSnsEventMessage()
extracts and parses the message from an SNS eventgetSnsEventMessageObject()
extracts and parses message object from an SNS event
- Added
@cumulus/common/cloudwatch-event
which contains helpers for Cloudwatch events:
isSfExecutionEvent()
returns true if event is from Step FunctionsisTerminalSfStatus()
determines if a Step Function status from a Cloudwatch event is a terminal statusgetSfEventStatus()
gets the Step Function status from a Cloudwatch eventgetSfEventDetailValue()
extracts a Step Function event detail field from a Cloudwatch eventgetSfEventMessageObject()
extracts and parses Step Function detail object from a Cloudwatch event
-
CUMULUS-1429
- Added
tf-modules/data-persistence
Terraform module which includes resources for data persistence in Cumulus:
- DynamoDB tables
- Elasticsearch with optional support for VPC
- Cloudwatch alarm for number of Elasticsearch nodes
-
CUMULUS-1379 CMR Launchpad Authentication
- Added
launchpad
configuration to @cumulus/deployment/app/config.yml
, and cloudformation templates, workflow message, lambda configuration, api endpoint configuration - Added
@cumulus/common/LaunchpadToken
and @cumulus/common/launchpad
to provide methods to get token and validate token - Updated lambdas to use Launchpad token for CMR actions (ingest and delete granules)
- Updated deployment documentation and added instructions to setup CMR client for Launchpad authentication
Changed
-
CUMULUS-1232
- Added retries to update
@cumulus/cmr-client
updateToken()
-
CUMULUS-1245 CUMULUS-795
- Added additional
ems
configuration parameters for sending the ingest reports to EMS - Added functionality to send daily ingest reports to EMS
-
CUMULUS-1241
- Removed the concept of "priority levels" and added ability to define a number of maximum concurrent executions per SQS queue
- Changed mapping of Cumulus message properties for the
sqs2sfThrottle
lambda:
- Queue name is read from
cumulus_meta.queueName
- Maximum executions for the queue is read from
meta.queueExecutionLimits[queueName]
, where queueName
is cumulus_meta.queueName
- Changed
sfSemaphoreDown
lambda to only attempt decrementing semaphores when:
- the message is for a completed/failed/aborted/timed out workflow AND
cumulus_meta.queueName
exists on the Cumulus message AND- An entry for the queue name (
cumulus_meta.queueName
) exists in the the object meta.queueExecutionLimits
on the Cumulus message
-
CUMULUS-1338
- Updated
sfSemaphoreDown
lambda to be triggered via AWS Step Function Cloudwatch events instead of subscription to sfTracker
SNS topic
-
CUMULUS-1311
- Updated
@cumulus/queue-granules
to set cumulus_meta.queueName
for queued execution messages - Updated
@cumulus/queue-pdrs
to set cumulus_meta.queueName
for queued execution messages - Updated
sqs2sfThrottle
lambda to immediately decrement queue semaphore value if dispatching Step Function execution throws an error
-
CUMULUS-1362
- Granule
processingStartTime
and processingEndTime
will be set to the execution start time and end time respectively when there is no sync granule or post to cmr task present in the workflow
-
CUMULUS-1400
- Deprecated
@cumulus/ingest/aws/getExecutionArn
. Use @cumulus/common/aws/getExecutionArn
instead.
Fixed
-
CUMULUS-1439
- Fix bug with rule.logEventArn deletion on Kinesis rule update and fix unit test to verify
-
CUMULUS-796
- Added production information (collection ShortName and Version, granuleId) to EMS distribution report
- Added functionality to send daily distribution reports to EMS
-
CUMULUS-1319
- Fixed a bug where granule ingest times were not being stored to the database
-
CUMULUS-1356
- The
Collection
model's delete
method now removes the specified item
from the collection config store that was inserted by the create
method.
Previously, this behavior was missing.
-
CUMULUS-1374
- Addressed audit concerns (https://www.npmjs.com/advisories/782) in api package
BREAKING CHANGES
Changed
- CUMULUS-1418
- Adding a default
cmaDir
key to configuration will cause CUMULUS_MESSAGE_ADAPTER_DIR
to be set by default to /opt
for any Lambda not setting useCma
to true, or explicitly setting the CMA environment variable. In lambdas that package the CMA independently of the Cumulus packaging. Lambdas manually packaging the CMA should have their Lambda configuration updated to set the CMA path, or alternately if not using the CMA as a Lambda layer in this deployment set cmaDir
to ./cumulus-message-adapter
.
Removed
-
CUMULUS-1337
- Removes the S3 Access Metrics package added in CUMULUS-799
-
PR1130
- Removed code deprecated since v1.11.1:
- Removed
@cumulus/common/step-functions
. Use @cumulus/common/StepFunctions
instead. - Removed
@cumulus/api/lib/testUtils.fakeFilesFactory
. Use @cumulus/api/lib/testUtils.fakeFileFactory
instead. - Removed
@cumulus/cmrjs/cmr
functions: searchConcept
, ingestConcept
, deleteConcept
. Use the functions in @cumulus/cmr-client
instead. - Removed
@cumulus/ingest/aws.getExecutionHistory
. Use @cumulus/common/StepFunctions.getExecutionHistory
instead.