logstash-input-event-hub

Logstash input plugin for data from Event Hubs

Summary

This plugin reads data from specified Azure Event Hubs.

Installation

You can install this plugin using the Logstash "plugin" or "logstash-plugin" (for newer versions of Logstash) command:

logstash-plugin install logstash-input-azureeventhub

For more information, see Logstash reference Working with plugins.

Configuration

Required Parameters

key

The shared access key to the target event hub.

username

The name of the shared access policy.

namespace

Event Hub namespace.

eventhub

Event Hub name.

partitions

Partition count of the target event hub.

Optional Parameters

domain

Domain of the target Event Hub. Default value is "servicebus.windows.net".

port

Port of the target Event Hub. Default value is 5671.

receive_credits

The credit number to limit the number of messages to receive in a processing cycle. Value must be between 10 and 999. Default is 999.

consumer_group

Name of the consumer group. Default value is "$default".

time_since_epoch_millis

Specifies the point of time after which the messages are received. Default value is the time when this plugin is initialized:

Time.now.utc.to_i * 1000

thread_wait_sec

Specifies the time (in seconds) to wait before another try if no message was received.

partition_receiver_epochs

A map from partition (string) to epoch (integer). By default each partition doesn't have an epoch defined. For more information read https://blogs.msdn.microsoft.com/gyan/2014/09/02/event-hubs-receiver-epoch/ .

Examples

• Bare-bone settings

input
{
    azureeventhub
    {
        key => "VGhpcyBpcyBhIGZha2Uga2V5Lg=="
        username => "receivepolicy"
        namespace => "mysbns"
        eventhub => "myeventhub"
        partitions => 4
        partition_receiver_epochs => { '2' => 42 '0' => 15 }
    }
}

• Example for WAD (Azure Diagnostics)

input
{
    azureeventhub
    {
        key => "VGhpcyBpcyBhIGZha2Uga2V5Lg=="
        username => "receivepolicy"
        namespace => "mysbns"
        eventhub => "myeventhub"
        partitions => 4
        partition_receiver_epochs => { '2' => 42 '0' => 15 }
    }
}
filter {
    split {field => 'records'} #split the records array in individual events
}
output {
    stdout { 
        codec => rubydebug
    }
}

More information

The source code of this plugin is hosted in GitHub repo Microsoft Azure Diagnostics with ELK. We welcome you to provide feedback and/or contribute to the project.

Please also see Analyze Diagnostics Data with ELK template for quick deployment of ELK to Azure.