🚀 Big News: Socket Acquires Coana to Bring Reachability Analysis to Every Appsec Team.Learn more
Socket
DemoInstallSign in
Socket

github.com/elastic/go-elasticsearch/_examples/bulk/kafka

Package Overview
Dependencies
Alerts
File Explorer
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

github.com/elastic/go-elasticsearch/_examples/bulk/kafka

v0.0.0-20250521161930-13b824712130
Source
Go
Version published
Created
Source

Example: Bulk indexing from a Kafka topic

This example demonstrates using the BulkIndexer component to ingest data consumed from a Kafka topic.

The provided docker-compose.yml file launches a realistic environment with Zookeeper, Kafka, Confluent Control Center, Elasticsearch and Kibana, and allows to inspect data flows, indexer metrics, and see the ingested data in a dashboard.

Screenshot

First, launch the environment and wait until it's ready:

make setup

Then, launch the Kafka producers and consumers and the Elasticsearch indexer:

make run

Open the Kibana dashboard to see the results and Confluent Control Center to inspect the Kafka cluster and see details about the topic and performance of consumers.

See the producer/producer.go file for the Kafka producer, consumer/consumer.go for the Kafka consumer, and the kafka.go file for the main workflow. The default configuration will launch one producer, four consumers, one indexer, and will send 1,000 messages per second; see go run kafka.go --help for changing the defaults.

FAQs

Package last updated on 21 May 2025

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts