@cumulus/logger
A logging library for use on the Cumulus project.
The @cumulus/logger
library exports a Logger
class, which generates
JSON-formated log events.
Log events always have the following keys:
- level (string) - the priority level of the message
- message (string) - the message itself
- sender (string) - the resource that created the message
- timestamp (string) - an ISO-8601 timestamp
Log events may have the following options keys:
- executions (string)
- granules (string [stringified array of objects])
- parentArn (string)
- stackName (string)
- version (string)
- asyncOperationId (string)
Log events may also contain user-specified keys.
Example log event:
{
"level": "info",
"message": "hello world",
"sender": "greetingFunction",
"timestamp": "2018-10-19T19:12:47.501Z"
}
Install
npm install @cumulus/logger
Usage
const Logger = require('@cumulus/logger');
const log = new Logger({ sender: 'example' });
log.info('hello, world');
API
new Logger({ [asyncOperationId], [executions], [granules], [parentArn], [sender], [stackName], [version] })
asyncOperationId
Type: string
An optional async operation id associated with a workflow.
executions
Type: string
An optional description of the executions.
granules
type: string
An optional granules string. Generally a stringified array of granule objects.
parentArn
type: string
An optional stepfunction ARN of the parent workflow that triggered the current execution.
sender
Type: string
The sender of the log event. Typically a Lambda Function Name or ECS Task Name.
Defaults to "unknown".
stackName
type: string
Cumulus stack name.
version
Type: string
An optional version.
log.debug([...messageArgs])
Writes a log event to stdout with level set to "debug".
args
Type: ...any
See console.log().
log.error([...messageArgs][, error])
Writes a log event to stderr with level set to "error".
args
Type: ...any
See console.log().
error
Type: Error
If the last argument is an Error then the following additional properties will be set on the log event:
- error (Object)
- name (string)
- message (string)
- stack (Array<string>) - the lines of the stack trace
log.fatal([...messageArgs])
Writes a log event to stdout with level set to "fatal".
args
Type: ...any
See console.log().
log.info([...messageArgs])
Writes a log event to stdout with level set to "info".
args
Type: ...any
See console.log().
log.infoWithAdditionalKeys(additionalKeys, ...messageArgs)
Writes a log event to stdout with level set to "info". In addition to the
standard keys, additional keys will be added to the event. If an additional key
is specified with the same name as a standard key, the value standard key will
be displayed.
additionalKeys
Type: Object
Additional key/value pairs to be added to the event.
args
Type: ...any
See console.log().
log.trace([...messageArgs])
Writes a log event to stdout with level set to "trace".
args
Type: ...any
See console.log().
log.warn([...messageArgs])
Writes a log event to stdout with level set to "debug".
args
Type: ...any
See console.log().
About Cumulus
Cumulus is a cloud-based data ingest, archive, distribution and management
prototype for NASA's future Earth science data streams.
Cumulus Documentation
Contributing
To make a contribution, please see our contributing guidelines.
[v14.1.0] 2023-02-27
MIGRATION notes
PostgreSQL compatibility update
From this release forward Core will be tested against PostgreSQL 11 Existing
release compatibility testing was done for release 11.1.8/14.0.0+. Users
should migrate their datastores to Aurora PostgreSQL 11.13+ compatible data stores
as soon as possible.
Users utilizing the cumulus-rds-tf
module will have upgraded/had their
database clusters forcibly upgraded at the next maintenance window after 31 Jan
2023. Our guidance to mitigate this issue is to do a manual (outside of
terraform) upgrade. This will result in the cluster being upgraded with a
manually set parameter group not managed by terraform.
If you manually upgraded and the cluster is now on version 11.13, to continue
using the cumulus-rds-tf
module once upgraded update following module
configuration values if set, or allow their defaults to be utilized:
parameter_group_family = "aurora-postgresql11"
engine_version = 11.13
When you apply this update, the original PostgreSQL v10 parameter group will be
removed, and recreated using PG11 defaults/configured terraform values and
update the database cluster to use the new configuration.
Added
- CUMULUS-3193
- Add a Python version file
- CUMULUS-3121
- Added a map of variables in terraform for custom configuration of cloudwatch_log_groups' retention periods.
Please refer to the [Cloudwatch-Retention] (https://nasa.github.io/cumulus/docs/configuration/cloudwatch-retention)
section of the Cumulus documentation in order for more detailed information and an example into how to do this.
- CUMULUS-3071
- Added 'PATCH' granules endpoint as an exact duplicate of the existing
PUT
endpoint. In future releases the PUT
endpoint will be replaced with valid PUT logic
behavior (complete overwrite) in a future release. The existing PUT
implementation is deprecated and users should move all existing usage of
PUT
to PATCH
before upgrading to a release with CUMULUS-3072
.
Fixed
-
CUMULUS-3033
- Fixed
granuleEsQuery
to properly terminate if body.hit.total.value
is 0.
-
The getLambdaAliases
function has been removed from the @cumulus/integration-tests
package
-
The getLambdaVersions
function has been removed from the @cumulus/integration-tests
package
-
CUMULUS-3117
- Update
@cumulus/es-client/indexer.js
to properly handle framework write
constraints for queued granules. Queued writes will now be properly
dropped from elasticsearch writes along with the primary datastore(s) when
write constraints apply
-
CUMULUS-3134
- Get tests working on M1 Macs
-
CUMULUS-3148:
- Updates cumulus-rds-tf to use defaults for PostgreSQL 11.13
- Update IngestGranuleSuccessSpec as test was dependant on file ordering and
PostgreSQL 11 upgrade exposed dependency on database results in the API return
- Update unit test container to utilize PostgreSQL 11.13 container
-
CUMULUS-3149
- Updates the api
/granules/bulkDelete
endpoint to take the
following configuration keys for the bulkDelete:
- concurrency - Number of concurrent bulk deletions to process at a time.
Defaults to 10, increasing this value may improve throughput at the cost
of additional database/CMR/etc load.
- maxDbConnections - Defaults to
concurrency
, and generally should not be
changed unless troubleshooting performance concerns.
- Updates all bulk api endpoints to add knexDebug boolean query parameter to
allow for debugging of database connection issues in the future. Defaults
to false.
- Fixed logic defect in bulk deletion logic where an information query was
nested in a transaction call, resulting in transactions holding knex
connection pool connections in a blocking way that would not resolve,
resulting in deletion failures.
-
CUMULUS-3142
- Fix issue from CUMULUS-3070 where undefined values for status results in
unexpected insertion failure on PATCH.
-
CUMULUS-3181
- Fixed
sqsMessageRemover
lambda to correctly retrieve ENABLED sqs rules.
-
CUMULUS-3189
- Upgraded
cumulus-process
and cumulus-message-adapter-python
versions to
support pip 23.0
-
CUMULUS-3196
- Moved
createServer
initialization outside the s3-credentials-endpoint
lambda
handler to reduce file descriptor usage
-
README shell snippets better support copying
-
CUMULUS-3111
- Fix issue where if granule update dropped due to write constraints for writeGranuleFromMessage, still possible for granule files to be written
- Fix issue where if granule update is limited to status and timestamp values due to write constraints for writeGranuleFromMessage, Dynamo or ES granules could be out of sync with PG
Breaking Changes
- CUMULUS-3072
- Removed original PUT granule endpoint logic (in favor of utilizing new PATCH
endpoint introduced in CUMULUS-3071)
- Updated PUT granule endpoint to expected RESTful behavior:
- PUT will now overwrite all non-provided fields as either non-defined or
defaults, removing existing related database records (e.g. files,
granule-execution linkages ) as appropriate.
- PUT will continue to overwrite fields that are provided in the payload,
excepting collectionId and granuleId which cannot be modified.
- PUT will create a new granule record if one does not already exist
- Like PATCH, the execution field is additive only - executions, once
associated with a granule record cannot be unassociated via the granule
endpoint.
- /granule PUT and PATCH endpoints now require a header with values
{ version: 2 }
- PUT endpoint will now only support /:collectionId/:granuleId formatted
queries
- `@cumulus/api-client.replaceGranule now utilizes body.collectionId to
utilize the correct API PUT endpoint
- Cumulus API version updated to
2
Changed
- Snyk Security
- Upgraded jsonwebtoken from 8.5.1 to 9.0.0
- CUMULUS-3160: Upgrade knex from 0.95.15 to 2.4.1
- Upgraded got from 11.8.3 to ^11.8.5
- Dependabot Security
- Upgraded the python package dependencies of the example lambdas
- CUMULUS-3043
- Organize & link Getting Started public docs for better user guidance
- Update Getting Started sections with current content
- CUMULUS-3046
- Update 'Deployment' public docs
- Apply grammar, link fixes, and continuity/taxonomy standards
- CUMULUS-3071
- Updated
@cumulus/api-client
packages to use PATCH
protocol for existing
granule PUT
calls, this change should not require user updates for
api-client
users.
@cumulus/api-client/granules.updateGranule
@cumulus/api-client/granules.moveGranule
@cumulus/api-client/granules.updateGranule
@cumulus/api-client/granules.reingestGranule
@cumulus/api-client/granules.removeFromCMR
@cumulus/api-client/granules.applyWorkflow
- CUMULUS-3097
- Changed
@cumulus/cmr-client
package's token from Echo-Token to Earthdata Login (EDL) token in updateToken method - Updated CMR header and token tests to reflect the Earthdata Login changes
- CUMULUS-3144
- Increased the memory of API lambda to 1280MB
- CUMULUS-3140
- Update release note to include cumulus-api release
- CUMULUS-3193
- Update eslint config to better support typing
- Improve linting of TS files
Removed
- CUMULUS-2798
- Removed AsyncOperations model
Removed
- CUMULUS-3009
- Removed Dynamo PDRs table