github.com/elastic/go-elasticsearch/_examples/bulk/kafka

Example: Bulk indexing from a Kafka topic

This example demonstrates using the BulkIndexer component to ingest data consumed from a Kafka topic.

The provided docker-compose.yml file launches a realistic environment with Zookeeper, Kafka, Confluent Control Center, Elasticsearch and Kibana, and allows to inspect data flows, indexer metrics, and see the ingested data in a dashboard.

First, launch the environment and wait until it's ready:

make setup

Then, launch the Kafka producers and consumers and the Elasticsearch indexer:

make run

Open the Kibana dashboard to see the results and Confluent Control Center to inspect the Kafka cluster and see details about the topic and performance of consumers.

See the producer/producer.go file for the Kafka producer, consumer/consumer.go for the Kafka consumer, and the kafka.go file for the main workflow. The default configuration will launch one producer, four consumers, one indexer, and will send 1,000 messages per second; see go run kafka.go --help for changing the defaults.