bunyan-elasticsearch-bulk

bunyan-elasticsearch-bulk

• A Bunyan stream for saving logs into Elasticsearch.
• Saves logs in memory instead of sending them to the Elasticsearch server one by one.
• The logs are accumulated in memory until a certain number of logs reached or a certain amount of time passed without sending any logs.
• The goal is to save resources on both your application and Elastic Search server (ex: lowering costs in AWS).

Installation

npm i bunyan-elasticsearch-bulk

Usage with Node

const bunyan = require('bunyan')
const createESStream = require('bunyan-elasticsearch-bulk')

const config = {
  name: 'Application Name',
  streams: [{
    level: 'debug',
    stream: createESStream({
      indexPattern: '[my-app-]YYYY.MM.DD',
      node: 'http://localhost:9200'
    })
  }]
}

const log = bunyan.createLogger(config)

// From here, you can log things according to the best practices of Bunyan.
// Please familiarize yourself with it here: https://github.com/trentm/node-bunyan

log.info('Log this message!')
log.info({ otherInfo: 'What else do you need to log here?' }, 'Log this message!')

List of Configuration Parameters

Parameters Specific to Elasticsearch Official Client

These Parameters are passed to the client without any change.

You need to provide the following one. The rest are up to you.

Field	Required?	Example	Description
node or nodes	Yes	http://localhost:9200	The Elasticsearch endpoint to use. No trailing slash!

Parameters Specific to this Module

Field	Default	Description
indexPattern	[my-app-]YYYY[-]MM[-]DD	Pattern for the daily segmentation of log indices. This will be passed to moment.utc().format(indexPattern) from moment.js
type	logs	Type of the logs for Elasticsearch servers prior to v7.x
limit	100	How many logs to collect before submitting them to ES
interval	5000	Time in milliseconds before submitting logs even if their count has not reached the limit.
client	Client(options)	If you don't want to use the included version of the client v7.x, you can configure the one you want and pass it here.

Emitted Events

Chech server.js for example usages:

Event	Fires when ..
log_received	we receive a log
log_submitted	we bulk-submit logs to ES server. With a reason limit_exceeded or timeout
error	an error happens in the whole bulk submission or when saving an individual message fails

Changes since 1.0.x

For you as a client, you no longer need to use the keyword new with this module. There is a factory function now. Example above is updated.

Development

• A docker-compose.yaml file has been added to make it easier to develop and test locally.

Author(s)

• Milad Kawas Cale

License

MIT