github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter

Elasticsearch Exporter

Status
Stability	beta: traces, logs
Distributions	contrib
Issues
Code Owners	@JaredTan95, @ycombinator

This exporter supports sending OpenTelemetry logs and traces to Elasticsearch.

Configuration options

• endpoints: List of Elasticsearch URLs. If endpoints and cloudid are missing, the ELASTICSEARCH_URL environment variable will be used.
• cloudid (optional): ID of the Elastic Cloud Cluster to publish events to. The cloudid can be used instead of endpoints.
• num_workers (optional): Number of workers publishing bulk requests concurrently.
• index (DEPRECATED, please use logs_index for logs, traces_index for traces): The index or data stream name to publish events to. The default value is logs-generic-default.
• logs_index: The index or data stream name to publish events to. The default value is logs-generic-default
• logs_dynamic_index (optional): takes resource or log record attribute named elasticsearch.index.prefix and elasticsearch.index.suffix resulting dynamically prefixed / suffixed indexing based on logs_index. (priority: resource attribute > log record attribute)
  • enabled(default=false): Enable/Disable dynamic index for log records
• traces_index: The index or data stream name to publish traces to. The default value is traces-generic-default.
• traces_dynamic_index (optional): takes resource or span attribute named elasticsearch.index.prefix and elasticsearch.index.suffix resulting dynamically prefixed / suffixed indexing based on traces_index. (priority: resource attribute > span attribute)
  • enabled(default=false): Enable/Disable dynamic index for trace spans
• logstash_format (optional): Logstash format compatibility. Traces or Logs data can be written into an index in logstash format.
  • enabled(default=false): Enable/Disable Logstash format compatibility. When logstash_format.enabled is true, the index name is composed using traces/logs_index or traces/logs_dynamic_index as prefix and the date, e.g: If traces/logs_index or traces/logs_dynamic_index is equals to otlp-generic-default your index will become otlp-generic-default-YYYY.MM.DD. The last string appended belongs to the date when the data is being generated.
  • prefix_separator(default=-): Set a separator between logstash_prefix and date.
  • date_format(default=%Y.%m.%d): Time format (based on strftime) to generate the second part of the Index name.
• pipeline (optional): Optional Ingest pipeline ID used for processing documents published by the exporter.
• flush: Event bulk indexer buffer flush settings
  • bytes (default=5242880): Write buffer flush size limit.
  • interval (default=30s): Write buffer flush time limit.
• retry: Elasticsearch bulk request retry settings
  • enabled (default=true): Enable/Disable request retry on error. Failed requests are retried with exponential backoff.
  • max_requests (default=3): Number of HTTP request retries.
  • initial_interval (default=100ms): Initial waiting time if a HTTP request failed.
  • max_interval (default=1m): Max waiting time if a HTTP request failed.
• mapping: Events are encoded to JSON. The mapping allows users to configure additional mapping rules.
  • mode (default=none): The fields naming mode. valid modes are:
    • none: Use original fields and event structure from the OTLP event.
    • ecs: Try to map fields defined in the OpenTelemetry Semantic Conventions to Elastic Common Schema (ECS). :warning: This mode's behavior is unstable, it is currently undergoing changes
    • raw: Omit the Attributes. string prefixed to field names for log and span attributes as well as omit the Events. string prefixed to field names for span events.
  • fields (optional): Configure additional fields mappings.
  • file (optional): Read additional field mappings from the provided YAML file.
  • dedup (default=true): Try to find and remove duplicate fields/attributes from events before publishing to Elasticsearch. Some structured logging libraries can produce duplicate fields (for example zap). Elasticsearch will reject documents that have duplicate fields.
  • dedot (default=true): When enabled attributes with . will be split into proper json objects.
• sending_queue
  • enabled (default = false)
  • num_consumers (default = 10): Number of consumers that dequeue batches; ignored if enabled is false
  • queue_size (default = 1000): Maximum number of batches kept in queue; ignored if enabled is false;

HTTP settings

• read_buffer_size (default=0): Read buffer size of HTTP client.
• write_buffer_size (default=0): Write buffer size of HTTP client.
• timeout (default=90s): HTTP request time limit.
• headers (optional): Headers to be sent with each HTTP request.

Security and Authentication settings

• user (optional): Username used for HTTP Basic Authentication.
• password (optional): Password used for HTTP Basic Authentication.
• api_key (optional): Authorization API Key in "encoded" format.

TLS settings

• ca_file (optional): Root Certificate Authority (CA) certificate, for verifying the server's identity, if TLS is enabled.
• cert_file (optional): Client TLS certificate.
• key_file (optional): Client TLS key.
• insecure (optional): In gRPC when set to true, this is used to disable the client transport security. In HTTP, this disables verifying the server's certificate chain and host name.
• insecure_skip_verify (optional): Will enable TLS but not verify the certificate.

Node Discovery

The Elasticsearch Exporter will check Elasticsearch regularly for available nodes and updates the list of hosts if discovery is enabled. Newly discovered nodes will automatically be used for load balancing.

• discover:
  • on_start (optional): If enabled the exporter queries Elasticsearch for all known nodes in the cluster on startup.
  • interval (optional): Interval to update the list of Elasticsearch nodes.

Example

exporters:
  elasticsearch/trace:
    endpoints: [https://elastic.example.com:9200]
    traces_index: trace_index
  elasticsearch/log:
    endpoints: [http://localhost:9200]
    logs_index: my_log_index
    sending_queue:
      enabled: true
      num_consumers: 20
      queue_size: 1000
······
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [elasticsearch/log]
    traces:
      receivers: [otlp]
      exporters: [elasticsearch/trace]
      processors: [batch]