@samsystem/winston-elasticsearch

winston-elasticsearch

An elasticsearch transport for the winston logging toolkit.

Features

• logstash compatible message structure.
• Thus consumable with kibana.
• Date pattern based index names.
• Custom transformer function to transform logged data into a different message structure.

Compatibility

For Elasticsearch 5.0 and later, use the 0.5.x series. For earlier versions, use the 0.4.x series.

Unsupported / Todo

• Querying.
• Real buffering of messages in case of unavailable ES.

Installation

npm install --save winston winston-elasticsearch

Usage

var winston = require('winston');
var Elasticsearch = require('winston-elasticsearch');

var esTransportOpts = {
  level: 'info'
};
winston.add(winston.transports.Elasticsearch, esTransportOpts);

// - or -

var logger = new winston.Logger({
  transports: [
    new Elasticsearch(esTransportOpts)
  ]
});

Options

• level [info] Messages logged with a severity greater or equal to the given one are logged to ES; others are discarded.
• index [none] the index to be used. This option is mutually exclusive with indexPrefix.
• indexPrefix [logs] the prefix to use to generate the index name according to the pattern <indexPrefix>-<indexSuffixPattern>.
• indexSuffixPattern [YYYY.MM.DD] a Moment.js compatible date/ time pattern.
• messageType [log] the type (path segment after the index path) under which the messages are stored under the index.
• transformer [see below] a transformer function to transform logged data into a different message structure.
• ensureMappingTemplate [true] If set to true, the given mappingTemplate is checked/ uploaded to ES when the module is sending the fist log message to make sure the log messages are mapped in a sensible manner.
• mappingTemplate [see file index-template-mapping.json file] the mapping template to be ensured as parsed JSON.
• flushInterval [2000] distance between bulk writes in ms.
• client An elasticsearch client instance. If given, all following options are ignored.
• clientOpts An object hash passed to the ES client. See its docs for supported options.
• waitForActiveShards [1] Sets the number of shard copies that must be active before proceeding with the bulk operation.
• pipeline [none] Sets the pipeline id to pre-process incoming documents with. See the bulk API docs.

Important

When changing the indexPrefix and/ or the transformer, make sure to provide a matching mappingTemplate.

Transformer

The transformer function allows to transform the log data structure as provided by winston into a structure more appropriate for indexing in ES.

The default transformer function's transformation is shown below.

Input:

{
  "message": "Some message",
  "level": "info",
  "meta": {
    "method": "GET",
    "url": "/sitemap.xml",
    ...
    }
  }
}

Output:

{
  "@timestamp": "2017-09-30T05:09:08.282Z",
  "message": "Some message",
  "severity": "info",
  "fields": {
    "method": "GET",
    "url": "/sitemap.xml",
    ...
  }
}

The @timestamp is generated in the transformer. Note that in current logstash versions, the only "standard fields" are @timestamp and @version, anything else ist just free.

A custom transformer function can be provided in the options hash.

Events

• error: in case of any error.

Example

An example assuming default settings.

Log Action

logger.info('Some message', <req meta data>);

Generated Message

The log message generated by this module has the following structure:

{
  "@timestamp": "2017-09-30T05:09:08.282Z",
  "message": "Some log message",
  "severity": "info",
  "fields": {
    "method": "GET",
    "url": "/sitemap.xml",
    "headers": {
      "host": "www.example.com",
      "user-agent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
      "accept": "*/*",
      "accept-encoding": "gzip,deflate",
      "from": "googlebot(at)googlebot.com",
      "if-modified-since": "Tue, 30 Sep 2017 11:34:56 GMT",
      "x-forwarded-for": "66.249.78.19"
    }
  }
}

Target Index

This message would be POSTed to the following endpoint:

http://localhost:9200/logs-2017.09.30/log/

So the default mapping uses an index pattern logs-*.