github.com/elastic/go-elasticsearch/_examples/bulk

Example: Bulk Indexing

default.go

The default.go example demonstrates how to properly operate the Elasticsearch's Bulk API.

The example intentionally doesn't use any abstractions or helper functions, to demonstrate the low-level mechanics of working with the Bulk API:

• iterating over a slice of data and preparing the meta/data pairs,
• filling a buffer with the payload until the configured threshold for a single batch is reached,
• sending a batch to Elasticsearch,
• checking for a request failure or a failed response,
• checking for individual errors in the response,
• updating a counter of indexed and failed documents,
• printing a report.

go run default.go -count=100000 -batch=25000

# Bulk: documents [100,000] batch size [25,000]
# ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
# → Generated 100,000 articles
# → Sending batch [1/4] [2/4] [3/4] [4/4]
# ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
# Successfully indexed [100,000] documents in 3.423s (29,214 docs/sec)

indexer.go

The indexer.go example demonstrates how to use the esutil.BulkIndexer helper for efficient indexing in parallel.

go run indexer.go -count=100000 -flush=1000000

# BulkIndexer: documents [100,000] workers [8] flush [1.0 MB]
# ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
# → Generated 100,000 articles
# ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
# Successfully indexed [100,000] documents in 1.909s (52,383 docs/sec)

The helper allows you to Add() bulk indexer items, and flushes each batch based on the configured threshold.

client, err := elasticsearch.New()
if err != nil {
	log.Fatalf("Error creating the client: %s", err)
}
defer client.Close(context.Background())

indexer, err := esutil.NewBulkIndexer(esutil.BulkIndexerConfig{Client: client})
if err != nil {
	log.Fatalf("Error creating the indexer: %s", err)
}
defer indexer.Close(context.Background())

if err := indexer.Add(
	context.Background(),
	esutil.BulkIndexerItem{
		Action: "index",
		Body:   strings.NewReader(`{"title":"Test"}`),
	},
); err != nil {
	log.Fatalf("Unexpected error: %s", err)
}

typed.go

The typed.go example demonstrates how to use the typed Bulk API to build a bulk request by appending operations (for example IndexOp) and then executing it with Do().

go run -tags bulk_typed typed.go -count=100000 -batch=1000

# Typed Bulk: documents [100,000] batch size [1,000]

base64.go

The base64.go example demonstrates indexing dense vectors using base64 encoding with the typed Bulk API and a public Elasticsearch Rally track dataset.

go run base64.go -count=20000 -batch=500

# Bulk Base64: documents [20,000] batch size [500]
# ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
# → Loaded 20,000 documents
# ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
# Successfully indexed [20,000] documents in 3.102s (6,446 docs/sec)

Please refer to the benchmarks folder for performance tests with different types of payload.

See the kafka folder for an end-to-end example of using the bulk helper for indexing data from a Kafka topic.