fluent-plugin-redis-store

Redis Output Plugin For fluentd

Fluentd output plugin to upload/publish event data to Redis storage.

Background

This is folked project from fluent-plugin-redisstore.

Features

Supported Redis commands

Currently the plugin supports following Redis commands:

• set by string type (of the plugin)
• lpush/rpush by list type
• sadd by set type
• zadd by zset type
• publish by publish type

Supported value format

• plain(as is)
• JSON
• MessagePack

key string for Redis storage

Redis commands require key and value.
For key, the plugin supports either way;

1. Specify a fixed key.
   You can do this simply using key option in td-agent configuration file.

   type redis_store
   key userdata
   

2. Lookup a key string in every event data by a lookup path.
   If event data have structured data like

   { "user": { "name": "Kei" } }
   

   and you want to use each name of user, you can use key_path option.

   type redis_store
   key_path user.name
   

   With the above data, Kei will be a key.

In addition, key_prefix and key_suffix are useful in some cases. Both are available either key and key_path

type redis_store
key_path user.name
key_prefix ouruser.
key_suffix .accesslog

With the previous data, key will be outuser.Kei.accesslog.

value data for Redis storage

To determine what value in every event data to be srtored, you have two options;

1. Store extracted data in event data, by a lookup path with value_path option.
   It works like key_path.
2. Store whole data.
   This is default behavior. To do it, simply omit value_path option.

Installation

fluent-gem install fluent-plugin-redis-store

# or if you are using td-agent:
td-agent-gem install fluent-plugin-redis-store

Configuration

Redis connection

Key	Type	Required?	Default	Description
host	string	Optional	127.0.0.1	host name of Redis server
port	int	Optional	6379	port number of Redis server
password	string	Optional		password for Redis connection
path	string	Optional		To connect via Unix socket, try '/tmp/redis.sock'
db	int	Optional	0	DB number of Redis
timeout	float	Optional	5.0	connection timeout in seconds

common options for storages

Key	Type	Default	Description
key	string		Fixed key used to store(publish) in Redis
key_path	string		path to lookup for key in the event data
key_prefix	string		prefix of key
key_suffix	string		suffix of key
value_path	string	(whole event data)	path to lookup for value in the event data
store_type	string	zset	string/list/set/zset/publish
format_type	string	plain	format type for value (plain/json/msgpack)
key_expire	int	-1	If set, the key will be expired in specified seconds
flush_interval	time	1	Time interval which events will be flushed to Redis

Note: either key or key_path is required.

string storage specific options

Key	Type	Default	Description
type	string		Fixed key used to store(publish) in Redis
No more options than common options.

list storage specific options

Key	Type	Default	Description
order	string	asc	asc: rpush, desc: lpush

set storage specific options

No more options than common options.

zset storage specific options

Key	Type	Default	Description
score_path	string	(time of log event)	path to lookup for score in the event data
collision_policy	string	(nil)	Only update elements that already exist (XX) or add a new element (NX)
value_expire	int		value expiration in seconds

If value_expire is set, the plugin assumes that the score in the SortedSet is based on timestamp and it deletes expired members every after new event data arrives.

publish storage specific options

No more options than common options.

Contributors

moaikids
HANAI tohru
Mohit Khanna
yamada-shinji
Heitor de Souza
Hiroshi Hatake
Gabriel Bordeaux

Copyright

Copyright (c) 2013 moaikids
Copyright (c) 2014 HANAI Tohru

License

Apache License, Version 2.0