fluent-multi-format-parser

Multi format parser plugin for Fluentd

Parse format mixed logs.

Requirements

fluent-plugin-multi-format-parser	fluentd	ruby
>= 1.0.0	>= v0.14.0	>= 2.1
< 1.0.0	>= v0.12.0	>= 1.9

Installation

Use RubyGems:

fluent-gem install fluent-plugin-multi-format-parser

Configuration

This plugin is a parser plugin. After installed, you can use multi_format in <parse> supported plugins. Use multiple <pattern>s to specify multiple parser formats.

<source>
  @type udp
  tag logs.multi

  <parse>
    @type multi_format
    <pattern>
      format apache
    </pattern>
    <pattern>
      format json
      time_key timestamp
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</source>

multi_format tries pattern matching from top to bottom and returns parsed result when matched.

Available format patterns and parameters are depends on Fluentd parsers. See parser plugin document for more details.

For v1.0

Put <pattern>s inside <parse>.

<filter app.**>
  @type parser
  key_name message
  <parse>
    @type multi_format
    <pattern>
      format json
    </pattern>
    <pattern>
      format regexp
      expression /...your regexp pattern.../
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</filter>

For v0.12

Use format instead of <parse></parse>.

<filter app.**>
  @type parser
  key_name message

  format multi_format
  <pattern>
    format json
  </pattern>
  <pattern>
    format /...your regexp pattern.../
  </pattern>
  <pattern>
    format none
  </pattern>
</filter>

Adding format identity field

Sometimes it may be useful to know which pattern was used. Since pareser usage may not support retagging, there is an option to add a format name field and/or index field.

Example:

<filter app.**>
  @type parser
  key_name message
  <parse>
    @type multi_format

    # if set, add this key to record with value being pattern format name
    # (format_name key)
    format_key 'format'

    # If set, add index of the matching pattern as this key (0-based)
    index_key 'parser_index'

    <pattern>
      format json
      # set format name for this pattern. If unset, uses format name
      # followed by index (in this case would be 'json#0')
      format_name 'json'
    </pattern>
    <pattern>
      format regexp
      format_name 'MyRefex'
      expression /...your regexp pattern.../
    </pattern>
    <pattern>
      format none
      format_name 'unparsed'
    </pattern>
  </parse>
</filter>

NOTE

This plugin doesn't work with multiline parsers because parser itself doesn't store previous lines.

Copyright

Author	Masahiro Nakagawa
Copyright	Copyright (c) 2014- Masahiro Nakagawa
License	Apache License (2.0)