pmacct-to-elasticsearch

pmacct-to-elasticsearch

pmacct-to-elasticsearch is a python script designed to read output from pmacct daemons, to process it and to store it into ElasticSearch. It works with both memory and print plugins and, optionally, it can perform manipulations on data (such as to add fields on the basis of other values).

.. image:: img/data_flow.png :align: center

1. pmacct daemons collect IP accounting data and process them with their plugins;
2. data are stored into in-memory-tables (memory plugins), JSON or CSV files (print plugins);
3. crontab jobs (memory plugins) or trigger scripts (print plugins) are invoked to execute pmacct-to-elasticsearch;
4. pmacct's output records are finally processed by pmacct-to-elasticsearch, which reads them from stdin (memory plugins) or directly from file.

Optionally, some data transformations can be configured, to allow pmacct-to-elasticsearch to add or remove fields to/from the output documents that are sent to ElasticSearch for indexing. These additional fields may be useful to enhance graphs and reports legibility, or to add a further level of aggregation or filtering.

Installation

Install the program using pip:

.. code:: bash

pip install pmacct-to-elasticsearch

Then clone the repository and run the ./install script to setup your system:

.. code:: bash

cd /usr/local/src/ git clone https://github.com/pierky/pmacct-to-elasticsearch.git cd pmacct-to-elasticsearch/ ./install

Configuration

Please refer to the CONFIGURATION.md_ file. The TRANSFORMATIONS.md_ file contains details about data transformations configuration.

.. _CONFIGURATION.md: CONFIGURATION.md .. _TRANSFORMATIONS.md: TRANSFORMATIONS.md

A simple tutorial on pmacct integration with ElasticSearch/Kibana using pmacct-to-elasticsearch can be found at http://blog.pierky.com/integration-of-pmacct-with-elasticsearch-and-kibana.

Future work

• Add support of more pmacct output formats (Apache Avro, ...).

Author

Pier Carlo Chiodi - https://pierky.com/

Blog: https://blog.pierky.com/ Twitter: @pierky <https://twitter.com/pierky>_

Changelog

0.3.2

• Code linting and fixing import and pip3 errors.

  Thanks to @palisadoes (Peter Harrison) <https://github.com/palisadoes> for these fixes.

0.3.1

• Fix: ES 6.2 compatibility.

  Issue #9 <https://github.com/pierky/pmacct-to-elasticsearch/issues/9>.

0.3.0

• New: CSV output support.

  The InputFormat option in the plugin configuration file can be used to instruct pmacct-to-elasticsearch to parse CSV output from pmacct.

• New: Multithreading support.

  The ReaderThreads option in the plugin configuration file sets the number of threads used to process pmacct's output.

• New: More command line arguments.

  The command line arguments under the Configuration options group can be used to override settings done on the plugin configuration file.

• Fix issue with index creation on ElasticSearch 5.x.

  Thanks to Kristoffer Olsson and Daniel Lindberg for reporting this and for their extensive support.

• Improved template for index creation.

• Fix an issue with transformations.

0.2.0

• New feature: HTTP Authentication support for ES API.

0.1.0

First release.