net.logstash.logback:logstash-logback-encoder
Advanced tools
Provides logback encoders, layouts, and appenders to log in JSON and other formats supported by Jackson
!! This document applies to the next version under development.
Provides logback encoders, layouts, and appenders to log in JSON and other formats supported by Jackson.
Supports both regular LoggingEvents (logged through a Logger) and AccessEvents (logged via logback-access).
Originally written to support output in logstash's JSON format, but has evolved into a highly-configurable, general-purpose, structured logging mechanism for JSON and other Jackson dataformats. The structure of the output, and the data it contains, is fully configurable.
Maven style:
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>8.0</version>
<!-- Use runtime scope if the project does not have any compile-time usage of logstash-logback-encoder,
such as usage of StructuredArguments/Markers or implementations such as
JsonProvider, AppenderListener, JsonFactoryDecorator, JsonGeneratorDecorator, etc
<scope>runtime</scope>
-->
</dependency>
<!-- Your project must also directly depend on either logback-classic or logback-access. For example: -->
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.5.6</version>
<!-- Use runtime scope if the project does not have any compile-time usage of logback,
such as implementations of Appender, Encoder, Layout, TurboFilter, etc
<scope>runtime</scope>
-->
</dependency>
If you get ClassNotFoundException/NoClassDefFoundError/NoSuchMethodError at runtime,
then ensure the required dependencies (and appropriate versions) as specified in the pom file
from the maven repository exist on the runtime classpath.
Specifically, the following need to be available on the runtime classpath:
uuid provider is used)Older versions than the ones specified in the pom file might work, but the versions in the pom file are what testing has been performed against. Support for logback versions prior to 1.3.0 was removed in logstash-logback-encoder 7.4.
If you are using logstash-logback-encoder in a project (such as spring-boot) that also declares dependencies on any of the above libraries, you might need to tell maven explicitly which versions to use to avoid conflicts. You can do so using maven's dependencyManagement feature. For example, to ensure that maven doesn't pick different versions of logback-core, logback-classic, and logback-access, add this to your project's pom.xml
<properties>
<logback-classic.version>1.5.6</logback-classic.version>
<logback-access.version>2.0.2</logback-access.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
<version>${logback-classic.version}</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>${logback-classic.version}</version>
</dependency>
<dependency>
<groupId>ch.qos.logback.access</groupId>
<artifactId>common</artifactId>
<version>${logback-access.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
| logstash-logback-encoder | Minimum Java Version supported |
|---|---|
| 8.x | 11 |
| 7.x | 8 |
| 6.x | 8 |
| 5.x | 7 |
| <= 4.x | 6 |
To log using JSON format, you must configure logback to use either:
The appenders, encoders, and layouts provided by the logstash-logback-encoder library are as follows:
| Format | Protocol | Function | LoggingEvent | AccessEvent |
|---|---|---|---|---|
| Logstash JSON | Syslog/UDP | Appender | LogstashUdpSocketAppender | LogstashAccessUdpSocketAppender |
| Logstash JSON | TCP | Appender | LogstashTcpSocketAppender | LogstashAccessTcpSocketAppender |
| any | any | Appender | LoggingEventAsyncDisruptorAppender | AccessEventAsyncDisruptorAppender |
| Logstash JSON | any | Encoder | LogstashEncoder | LogstashAccessEncoder |
| Logstash JSON | any | Layout | LogstashLayout | LogstashAccessLayout |
| General JSON | any | Encoder | LoggingEventCompositeJsonEncoder | AccessEventCompositeJsonEncoder |
| General JSON | any | Layout | LoggingEventCompositeJsonLayout | AccessEventCompositeJsonLayout |
These encoders/layouts can generally be used by any logback appender (such as RollingFileAppender).
The general composite JSON encoders/layouts can be used to output any JSON format/data by configuring them with various JSON providers. The Logstash encoders/layouts are really just extensions of the general composite JSON encoders/layouts with a pre-defined set of providers.
The logstash encoders/layouts are easier to configure if you want to use the standard logstash version 1 output format. Use the composite encoders/layouts if you want to heavily customize the output, or if you need to use logstash version 0 output.
The *AsyncDisruptorAppender appenders are similar to logback's AsyncAppender,
except that a LMAX Disruptor RingBuffer
is used as the queuing mechanism, as opposed to a BlockingQueue.
These async appenders can delegate to any other underlying logback appender.
To output JSON for LoggingEvents to a syslog/UDP channel,
use the LogstashUdpSocketAppender with a LogstashLayout or LoggingEventCompositeJsonLayout
in your logback.xml, like this:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stash" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
<host>MyAwesomeSyslogServer</host>
<!-- port is optional (default value shown) -->
<port>514</port>
<!-- layout is required -->
<layout class="net.logstash.logback.layout.LogstashLayout"/>
</appender>
<root level="all">
<appender-ref ref="stash" />
</root>
</configuration>
You can further customize the JSON output by customizing the layout as described in later sections.
For example, to configure global custom fields, you can specify
<appender name="stash" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
<host>MyAwesomeSyslogServer</host>
<!-- port is optional (default value shown) -->
<port>514</port>
<layout class="net.logstash.logback.layout.LogstashLayout">
<customFields>{"appname":"myWebservice"}</customFields>
</layout>
</appender>
To output JSON for AccessEvents over UDP, use a LogstashAccessUdpSocketAppender
with a LogstashAccessLayout or AccessEventCompositeJsonLayout
in your logback-access.xml, like this:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stash" class="net.logstash.logback.appender.LogstashAccessUdpSocketAppender">
<host>MyAwesomeSyslogServer</host>
<!-- port is optional (default value shown) -->
<port>514</port>
<layout class="net.logstash.logback.layout.LogstashAccessLayout">
<customFields>{"appname":"myWebservice"}</customFields>
</layout>
</appender>
<appender-ref ref="stash" />
</configuration>
To receive syslog/UDP input in logstash, configure a syslog or udp input with the json codec in logstash's configuration like this:
input {
syslog {
codec => "json"
}
}
To output JSON for LoggingEvents over TCP, use a LogstashTcpSocketAppender
with a LogstashEncoder or LoggingEventCompositeJsonEncoder
in your logback.xml, like this:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>127.0.0.1:4560</destination>
<!-- encoder is required -->
<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>
<root level="DEBUG">
<appender-ref ref="stash" />
</root>
</configuration>
To output JSON for AccessEvents over TCP, use a LogstashAccessTcpSocketAppender
with a LogstashAccessEncoder or AccessEventCompositeJsonEncoder
in your logback-access.xml, like this:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stash" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender">
<destination>127.0.0.1:4560</destination>
<!-- encoder is required -->
<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder" />
</appender>
<appender-ref ref="stash" />
</configuration>
The TCP appenders use an encoder, rather than a layout as the UDP appenders .
You can use a Logstash*Encoder, *EventCompositeJsonEncoder, or any other logback encoder.
All of the output formatting options are configured at the encoder level.
Internally, the TCP appenders are asynchronous (using the LMAX Disruptor RingBuffer).
All the encoding and TCP communication is delegated to a single writer thread.
There is no need to wrap the TCP appenders with another asynchronous appender
(such as AsyncAppender or LoggingEventAsyncDisruptorAppender).
All the configuration parameters (except for sub-appender) of the async appenders are valid for TCP appenders. For example, waitStrategyType and ringBufferSize.
By default the TCP appenders will never block the logging thread - if the RingBuffer is full (e.g. due to slow network, etc), then events will be dropped. If desired, the appender can also be configured to block and wait for free space, see RingBuffer Full for more information.
The TCP appenders will automatically reconnect if the connection breaks. Multiple destinations can be configured to increase availability and reduce message lost. See Multiple Destinations for more information.
To receive TCP input in logstash, configure a tcp input with the json_lines codec in logstash's configuration like this:
input {
tcp {
port => 4560
codec => json_lines
}
}
In order to guarantee that logged messages have had a chance to be processed by the TCP appender, you'll need to cleanly shut down logback when your application exits.
If events occur infrequently, and the connection breaks consistently due to a server-side idle timeout,
then you can enable keep alive functionality by configuring a keepAliveDuration like this:
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
...
<keepAliveDuration>5 minutes</keepAliveDuration>
</appender>
This setting accepts a Logback Duration value - see the section dedicated to Duration Property for more information about the valid values.
When the keepAliveDuration is set, then a keep alive message will be sent if an event has not occurred for the length of the duration.
The keep alive message defaults to unix line ending (\n), but can be changed by setting the keepAliveMessage property to the desired value. The following values have special meaning:
<empty string>: no keep aliveSYSTEM: system's line separatorUNIX: unix line ending (\n)WINDOWS: windows line ending (\r\n)Any other value will be used as-is.
The keep alive message is encoded in UTF-8 by default. This can be changed by setting the keepAliveCharset property to the name of the desired charset.
The TCP appenders can be configured to try to connect to one of several destinations like this:
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>destination1.domain.com:4560</destination>
<destination>destination2.domain.com:4560</destination>
<destination>destination3.domain.com:4560</destination>
...
</appender>
or this:
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>
destination1.domain.com:4560,
destination2.domain.com:4560,
destination3.domain.com:4560
</destination>
...
</appender>
Destinations are expressed using the following format: host[:port] where:
host can be a hostname (eg. localhost) , an IPv4 address (eg. 192.168.1.1) or an IPv6 address enclosed between brackets (eg. [2001:db8::1]).port is optional and, if specified, must be prefixed by a colon (:). It must be a valid integer value between 0 and 65535.The appender uses a connectionStrategy to determine:
Logs are only sent to one destination at a time (i.e. not all destinations). By default, the appender will stay connected to the connected destination until it breaks, or until the application is shut down. Some connection strategies can force a reconnect (see below). If a connection breaks, then the appender will attempt to connect to the next destination selected by the connection strategy.
The available connection strategies are as follows:
| Strategy | Description |
|---|---|
| preferPrimary | (default)
The first destination is considered the primary destination.
Each additional destination is considered a secondary destination.
This strategy prefers the primary destination, unless it is down.
The appender will attempt to connect to each destination in the order in which they are configured.
If a connection attempt fails, thes the appender will attempt to connect to the next destination.
If a connection succeeds, and then closes before the minConnectionTimeBeforePrimary
has elapsed, then the appender will attempt to connect to the next destination.
If a connection succeeds, and then closes after the minConnectionTimeBeforePrimary
has elapsed, then the appender will attempt to connect
to the destinations in the order in which they are configured,
starting at the first/primary destination.
The secondaryConnectionTTL can be set to gracefully close connections to secondary destinations after a specific duration. This will force the the appender to reattempt to connect to the destinations in order again. The secondaryConnectionTTL value does not affect connections to the primary destination. The minConnectionTimeBeforePrimary (10 seconds by default) specifies the minimum amount of time that a sucessfully established connection must remain open before the next connection attempt will try the primary. i.e. If a connection stays open less than this amount of time, then the next connection attempt will attempt the next destination (instead of the primary). This is used to prevent a connection storm to the primary in the case the primary accepts a connection, and then immediately closes it. Example: <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>destination1.domain.com:4560</destination>
<destination>destination2.domain.com:4560</destination>
<destination>destination3.domain.com:4560</destination>
<connectionStrategy>
<preferPrimary>
<secondaryConnectionTTL>5 minutes</secondaryConnectionTTL>
</preferPrimary>
</connectionStrategy>
</appender>
|
| roundRobin |
This strategy attempts connections to the destination in round robin order.
If a connection fails, the next destination is attempted.
The connectionTTL can be set to gracefully close connections after a specific duration. This will force the the appender to reattempt to connect to the next destination. Example: <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>destination1.domain.com:4560</destination>
<destination>destination2.domain.com:4560</destination>
<destination>destination3.domain.com:4560</destination>
<connectionStrategy>
<roundRobin>
<connectionTTL>5 minutes</connectionTTL>
</roundRobin>
</connectionStrategy>
</appender>
|
| random |
This strategy attempts connections to the destination in a random order.
If a connection fails, the next random destination is attempted.
The connectionTTL can be set to gracefully close connections after a specific duration. This will force the the appender to reattempt to connect to the next random destination. Example: <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>destination1.domain.com:4560</destination>
<destination>destination2.domain.com:4560</destination>
<destination>destination3.domain.com:4560</destination>
<connectionStrategy>
<random>
<connectionTTL>5 minutes</connectionTTL>
</random>
</connectionStrategy>
</appender>
|
You can also use your own custom connection strategy by implementing the DestinationConnectionStrategy interface,
and configuring the appender to use it like this:
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>destination1.domain.com:4560</destination>
<destination>destination2.domain.com:4560</destination>
<destination>destination3.domain.com:4560</destination>
<connectionStrategy class="your.package.YourDestinationConnectionStrategy">
</connectionStrategy>
</appender>
By default, the TCP appender will wait 30 seconds between connection attempts to a single destination. The time between connection attempts to each destination is tracked separately.
This amount of time to delay can be changed by setting the reconnectionDelay field.
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
...
<reconnectionDelay>1 second</reconnectionDelay>
</appender>
This setting accepts a Logback Duration value - see the section dedicated to Duration Property for more information about the valid values.
By default, a connection timeout of 5 seconds is used when connecting to a remote destination.
You can adjust this by setting the appender's connectionTimeout configuration property to the desired value.
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
...
<connectionTimeout>5 seconds</connectionTimeout>
</appender>
A value of 0 means "don't use a timeout and wait indefinitely" which often really means "use OS defaults".
This setting accepts a Logback Duration value - see the section dedicated to Duration Property for more information about the valid values.
By default, a buffer size of 8192 bytes is used to buffer socket output stream writes.
You can adjust this by setting the appender's writeBufferSize.
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
...
<writeBufferSize>16384</writeBufferSize>
</appender>
Buffering can be disabled by setting the writeBufferSize to 0.
Consider disabling the write buffer if you are concerned about losing data from the buffer for flaky connections.
Disabling the buffer can potentially slow down the writer thread due to increased system calls,
but in some environments, this does not seem to affect overall performance.
See this discussion.
If a destination stops reading from its socket input, but does not close the connection, then writes from the TCP appender will eventually backup, causing the ring buffer to backup, causing events to be dropped.
To detect this situation, you can enable a write timeout, so that "stuck" writes will eventually timeout, at which point the connection will be re-established. When the write buffer is enabled, any buffered data will be lost when the connection is reestablished.
By default there is no write timeout. To enable a write timeout, do the following:
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
...
<writeTimeout>1 minute</writeTimeout>
</appender>
Note that since the blocking java socket output stream used to send events does not have a concept of a write timeout, write timeouts are detected using a task scheduled periodically with the same frequency as the write timeout. For example, if the write timeout is set to 30 seconds, then a task will execute every 30 seconds to see if 30 seconds has elapsed since the start of the current write operation. Therefore, it is recommended to use longer write timeouts (e.g. > 30s, or minutes), rather than short write timeouts, so that this task does not execute too frequently. Also, this approach means that it could take up to two times the write timeout before a write timeout is detected.
The write timeout must be >0. A timeout of zero is interpreted as an infinite timeout which effecively means "no write timeout".
This setting accepts a Logback Duration value - see the section dedicated to Duration Property for more information about the valid values.
The appender starts writing the events stored in the queue as soon as the connection is established. In some cases you may want to add an extra delay before sending the first events after the connection is established. This may come in handy in situations where the appender connects to an intermediate proxy that needs some time to establish a connection to the final destination. If the appender starts writing immediately, events may be lost in-flight if the proxy ultimately fails to connect to the final destination.
To enable this feature, set the initialSendDelay to the desired delay before the first event is sent after the connection is established. If the connection is lost before the delay expires, no event will be lost. The default value is 0 which means no delay and start flusing pending events immediately.
The following example configures a delay of 5 secondes before writing in the new connection:
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
...
<initialSendDelay>5 secondes</initialSendDelay>
</appender>
This setting accepts a Logback Duration value - see the section dedicated to Duration Property for more information about the valid values.
To use SSL, add an <ssl> sub-element within the <appender> element for the LogstashTcpSocketAppender
or LogstashAccessTcpSocketAppender.
See the logback manual for how to configure SSL.
SSL for the Logstash*TcpSocketAppenders are configured the same way as logback's SSLSocketAppender.
For example, to enable SSL using the JVM's default keystore/truststore, do the following:
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
...
<!-- Enable SSL using the JVM's default keystore/truststore -->
<ssl/>
</appender>
To use a different truststore, do the following:
<appender name="stash" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender">
...
<!-- Enable SSL and use a different truststore -->
<ssl>
<trustStore>
<location>classpath:server.truststore</location>
<password>${server.truststore.password}</password>
</trustStore>
</ssl>
</appender>
All the customizations that logback offers
(such as configuring cipher specs, protocols, algorithms, providers, etc.)
are supported by the Logback*TcpSocketAppenders.
See the logstash documentation for the tcp input for how to configure it to use SSL.
The *AsyncDisruptorAppender appenders are similar to logback's AsyncAppender,
except that a LMAX Disruptor RingBuffer
is used as the queuing mechanism, as opposed to a BlockingQueue.
These async appenders can delegate to any other underlying logback appender.
For example:
<appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
...
</appender>
<appender name="async" class="net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender">
<appender-ref ref="file" />
</appender>
Warning Since Logback 1.3 it is not allowed anymore to declare an
<appender>inside another<appender>. The nested appender should instead be declared outside and<appender-ref>must be used to refer to it.See LOGBACK-1674 for more information.
Logging events are first enqueued in a ring buffer before they are delivered to their final destination by a separate handler thread.
The buffer size is fixed, it does not grow or shrink at runtime. Its size is determined by the ringBufferSize configuration property set to 8192 by default.
If the handler thread is not as fast as the producing threads, then the ring buffer will eventually fill up, at which point events will be dropped (the default) or the producing threads are blocked depending on configured appendTimeout (see RingBuffer Full.
The async appenders will by default never block the logging thread. If the RingBuffer is full (e.g. due to slow network, etc), then events will be dropped.
Alternatively, you can configure the appender to wait until space becomes available instead of dropping the events immediately. This may come in handy when you want to rely on the buffering and the async nature of the appender but don't want to loose any event in case of large logging bursts that exceed the size of the RingBuffer.
The behaviour of the appender when the RingBuffer is controlled by the appendTimeout configuration property:
appendTimeout | Behaviour when RingBuffer is full |
|---|---|
< 0 | disable timeout and wait until space is available |
0 | no timeout, give up immediately and drop event (this is the default) |
> 0 | retry during the specified amount of time |
Logging threads waiting for space in the RingBuffer wake up periodically at a frequency starting at 1ns and increasing exponentially up to appendRetryFrequency (default 5ms).
Only one thread is allowed to retry at a time. If a thread is already retrying, additional threads are waiting on a lock until the first is finished. This strategy should help to limit CPU consumption while providing good enough latency and throughput when the ring buffer is at (or close) to its maximal capacity.
When the appender drops an event, it emits a warning status message every droppedWarnFrequency consecutive dropped events (1000 by default, use 0 to turn off warnings). Another status message is emitted when the drop period is over and a first event is succesfully enqueued reporting the total number of events that were dropped.
In order to guarantees that logged messages have had a chance to be processed by asynchronous appenders (including the TCP appender) and ensure background threads have been stopped, you'll need to cleanly shut down logback when your application exits.
When gracefully stopped, async appenders wait until all events in the buffer are processed and the buffer is empty.
The maximum time to wait is configured by the shutdownGracePeriod parameter and is set to 1 minute by default.
Events still in the buffer after this period is elapsed are dropped and the appender is stopped.
By default, the BlockingWaitStrategy is used by the worker thread spawned by this appender.
The BlockingWaitStrategy minimizes CPU utilization, but results in slower latency and throughput.
If you need faster latency and throughput (at the expense of higher CPU utilization), consider
a different wait strategy offered by the disruptor.
!! Whichever wait strategy you choose, be sure to test and monitor CPU utilization, latency, and throughput to ensure it meets your needs. For example, in some configurations,
SleepingWaitStrategycan consume 90% CPU utilization at rest.
The wait strategy can be configured on the async appender using the waitStrategyType parameter, like this:
<appender name="async" class="net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender">
<waitStrategyType>sleeping</waitStrategyType>
...
</appender>
The supported wait strategies are as follows:
| Wait Strategy | Parameters | Implementation |
|---|---|---|
| blocking | none | BlockingWaitStrategy |
| busySpin | none | BusySpinWaitStrategy |
| liteBlocking | none | LiteBlockingWaitStrategy |
| yielding | none | YieldingWaitStrategy |
sleeping{
retries,
sleepTimeNs
}
e.g.sleeping or sleeping{500,1000} |
| SleepingWaitStrategy |
phasedBackoff{
spinTime,
yieldTime,
timeUnit,
fallbackStrategy
}
e.g.phasedBackoff{10,60,seconds,blocking} |
| PhasedBackoffWaitStrategy |
timeoutBlocking{
timeout,
timeUnit
}
e.g.timeoutBlocking{1,minutes} |
| TimeoutBlockingWaitStrategy |
liteTimeoutBlocking{
timeout,
timeUnit
}
e.g.liteTimeoutBlocking{1,minutes} |
| LiteTimeoutBlockingWaitStrategy |
See AsyncDisruptorAppender
for other configuration parameters (such as ringBufferSize, threadNamePrefix, daemon, and droppedWarnFrequency)
Listeners can be registered to an appender to receive notifications for the appender lifecycle and event processing.
See the two listener interfaces for the types of notifications that can be received:
AppenderListener - basic notifications for the async appenders and UDP appenders.TcpAppenderListener - extension of AppenderListener with additional TCP-specific notifications. Only works with the TCP appenders.Some example use cases for a listener are:
A FailureSummaryLoggingAppenderListener
is provided that will log a warning on the first success after a series of consecutive append/send/connect failures.
The message includes summary details of the failures that occurred (such as the number of failures, duration of the failures, etc).
To register it:
<appender name="stash" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender">
<listener class="net.logstash.logback.appender.listener.FailureSummaryLoggingAppenderListener">
<loggerName>net.logstash.logback.appender.listener.FailureSummaryLoggingAppenderListener</loggerName>
</listener>
</appender>
You may also create your own listener by implementing the *Listener interface and register it to an appender using the listener xml element like this:
<appender name="stash" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender">
...
<listener class="your.package.YourListenerClass">
<yourListenerProperty>propertyValue</yourListenerProperty>
</listener>
</appender>
Multiple listeners can be registered by supplying multiple listener xml elements.
You can use any of the encoders/layouts provided by the logstash-logback-encoder library with other logback appenders.
For example, to output LoggingEvents to a file, use the LogstashEncoder
with the RollingFileAppender in your logback.xml like this:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stash" class="ch.qos.logback.core.rolling.RollingFileAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>info</level>
</filter>
<file>/some/path/to/your/file.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>/some/path/to/your/file.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>
<root level="all">
<appender-ref ref="stash" />
</root>
</configuration>
To log AccessEvents to a file, configure your logback-access.xml like this:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stash" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/some/path/to/your/file.log</file>
<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder" />
</appender>
<appender-ref ref="stash" />
</configuration>
The LogstashLayout and LogstashAccessLayout can be configured the same way as
the LogstashEncoder and LogstashAccessEncoder. All the other examples
in this document use encoders, but the same options apply to the layouts as well.
To receive file input in logstash, configure a file input in logstash's configuration like this:
input {
file {
path => "/some/path/to/your/file.log"
codec => "json"
}
}
The following sections describe the fields included in the JSON output by default for LoggingEvents written by the
LogstashEncoderLogstashLayout, andIf you are using the composite encoders/layouts, then the fields written will vary by the providers you configure.
These fields will appear in every LoggingEvent unless otherwise noted. The field names listed here are the default field names. The field names can be customized (see Customizing Standard Field Names).
| Field | Description |
|---|---|
@timestamp | Time of the log event (ISO_OFFSET_DATE_TIME) - see Customizing Timestamp |
@version | Logstash format version (e.g. 1) - see Customizing Version |
message | Formatted log message of the event - see Customizing Message |
logger_name | Name of the logger that logged the event |
thread_name | Name of the thread that logged the event |
level | String name of the level of the event |
level_value | Integer value of the level of the event |
stack_trace | (Only if a throwable was logged) The stacktrace of the throwable. Stackframes are separated by line endings. |
tags | (Only if tags are found) The names of any markers not explicitly handled. (e.g. markers from MarkerFactory.getMarker will be included as tags, but the markers from Markers will not.) This can be fully disabled by specifying <includeTags>false</includeTags>, in the encoder/layout/appender configuration. |
By default, LogstashEncoder/LogstashLayout will write each
Mapped Diagnostic Context (MDC) (org.slf4j.MDC)
entry to the output.
To disable writing MDC entries, add <includeMdc>false</includeMdc>
to the LogstashEncoder/LogstashLayout configuration.
You can also configure specific entries in the MDC to be included or excluded as follows:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<includeMdcKeyName>key1ToInclude</includeMdcKeyName>
<includeMdcKeyName>key2ToInclude</includeMdcKeyName>
</encoder>
or
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<excludeMdcKeyName>key1ToExclude</excludeMdcKeyName>
<excludeMdcKeyName>key2ToExclude</excludeMdcKeyName>
</encoder>
When key names are specified for inclusion, then all other fields will be excluded. When key names are specified for exclusion, then all other fields will be included. It is a configuration error to specify both included and excluded key names.
By default, the MDC key is used as the field name in the output.
To use an alternative field name in the output for an MDC entry,
specify <mdcKeyFieldName>mdcKeyName=fieldName</mdcKeyFieldName>:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<mdcKeyFieldName>key1=alternateFieldNameForKey1</mdcKeyFieldName>
</encoder>
You can also manipulate the MDC entry values written to the JSON output. By default, no manipulations are done and all MDC entry values are written as text.
Currently, the following MDC entry writers are supported:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<!--
Writes long values (instead of String values) for any MDC values
that can be parsed as a long (radix 10).
e.g. Writes 1234 instead of "1234"
-->
<mdcEntryWriter class="net.logstash.logback.composite.loggingevent.mdc.LongMdcEntryWriter"/>
<!--
Writes double values (instead of String values) for any MDC values
that can be parsed as a double, except NaN and positive/negative Infinity.
e.g. 1234.5678 instead of "1234.5678"
-->
<mdcEntryWriter class="net.logstash.logback.composite.loggingevent.mdc.DoubleMdcEntryWriter"/>
<!--
Writes boolean values (instead of String values) for any MDC values
that equal "true" or "false", ignoring case.
e.g. Writes true instead of "true"
-->
<mdcEntryWriter class="net.logstash.logback.composite.loggingevent.mdc.BooleanMdcEntryWriter"/>
<!--
Composite MDC entry writer that delegates writing MDC entries to a list of `MdcEntryWriter`
if the key of an MDC entry does match a given include pattern and does not match a given
exclude pattern.
Omitting the 'includeMdcKeyPattern' means to include all MDC keys.
Omitting the 'excludeMdcKeyPattern' means to exclude no MDC keys.
The elements with the key patterns are optional. If provided, use a regular expression
with the syntax of `java.util.regex.Pattern`.
-->
<mdcEntryWriter class="net.logstash.logback.composite.loggingevent.mdc.RegexFilteringMdcEntryWriter">
<includeMdcKeyPattern>keyPatternToInclude</includeMdcKeyPattern>
<excludeMdcKeyPattern>keyPatternToExclude</excludeMdcKeyPattern>
<mdcEntryWriter class="net.logstash.logback.composite.loggingevent.LongMdcEntryWriter"/>
</mdcEntryWriter>
</encoder>
To add your own MDC entry writer for other types or apply the manipulations only for specific fields
you can write your own implementation of MdcEntryWriter.
You can also replace the default MDC JSON provider with your own class extending from
MdcJsonProvider.
Configuring your class as a Custom JSON Provider will then replace
the default MdcJsonProvider.
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<provider class="mypackagenames.MyCustomMdcJsonProvider"/>
</encoder>
Slf4j 2's fluent API supports attaching key value pairs to the log event.
LogstashEncoder/LogstashLayout will write each key value pair as a field in the output by default.
To disable writing key value pairs, add <includeKeyValuePairs>false</includeKeyValuePairs>
to the LogstashEncoder/LogstashLayout configuration.
You can also configure specific key value pairs to be included or excluded as follows:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<includeKeyValueKeyName>key1ToInclude</includeKeyValueKeyName>
<includeKeyValueKeyName>key2ToInclude</includeKeyValueKeyName>
</encoder>
or
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<excludeKeyValueKeyName>key1ToExclude</excludeKeyValueKeyName>
<excludeKeyValueKeyName>key2ToExclude</excludeKeyValueKeyName>
</encoder>
When key names are specified for inclusion, then all other keys will be excluded. When key names are specified for exclusion, then all other keys will be included. It is a configuration error to specify both included and excluded key names.
By default, the key is used as the field name in the output.
To use an alternative field name in the output for an key value pair,
specify<keyValuePairsKeyFieldName>keyName=fieldName</keyValuePairsKeyFieldName>:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<keyValueKeyFieldName>key1=alternateFieldNameForKey1</keyValueKeyFieldName>
</encoder>
By default, each property of Logback's Context (ch.qos.logback.core.Context)
will appear as a field in the LoggingEvent.
This can be disabled by specifying <includeContext>false</includeContext>
in the encoder/layout/appender configuration.
Note that logback versions prior to 1.1.10 included a HOSTNAME property by default in the context.
As of logback 1.1.10, the HOSTNAME property is lazily calculated (see LOGBACK-1221), and will no longer be included by default.
The encoder/layout/appender do not contain caller info by default. This can be costly to calculate and should be switched off for busy production environments.
To switch it on, add the includeCallerData property to the configuration.
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<includeCallerData>true</includeCallerData>
</encoder>
If the encoder is included inside an asynchronous appender, such as
AsyncAppender, LoggingEventAsyncDisruptorAppender, or LogstashTcpSocketAppender, then
includeCallerData must be set to true on the appender as well.
When switched on, the following fields will be included in the log event:
| Field | Description |
|---|---|
caller_class_name | Fully qualified class name of the class that logged the event |
caller_method_name | Name of the method that logged the event |
caller_file_name | Name of the file that logged the event |
caller_line_number | Line number of the file where the event was logged |
In addition to the fields above, you can add other fields to the LoggingEvent either globally, or on an event-by-event basis.
Add custom fields that will appear in every LoggingEvent like this :
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<customFields>{"appname":"myWebservice","roles":["customerorder","auth"],"buildinfo":{"version":"Version 0.1.0-SNAPSHOT","lastcommit":"75473700d5befa953c45f630c6d9105413c16fe1"}}</customFields>
</encoder>
or in an AccessEvent like this :
<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder">
<customFields>{"appname":"myWebservice","roles":["customerorder","auth"],"buildinfo":{"version":"Version 0.1.0-SNAPSHOT","lastcommit":"75473700d5befa953c45f630c6d9105413c16fe1"}}</customFields>
</encoder>
When logging a message, you can add additional fields to the JSON output by using
StructuredArguments, ORMarkersThe difference between the two is that
StructuredArguments are included in a the log event's formatted message
(when the message has a parameter for the argument) AND in the JSON output.
StructuredArguments will be included in the JSON output if using LogstashEncoder/Layout
or if using composite encoders/layouts with the arguments provider.Markers are only written to the JSON output, and NEVER to the log event's formatted message.
Markers will be included in the JSON output if using LogstashEncoder/Layout
or if using composite encoders/layouts with the logstashMarkers provider.You can use StructuredArguments even if the message does not contain a parameter
for the argument. However, in this case, the argument will only be written to the JSON output
and not the formatted message (which is effectively the same behavior that the Markers provide).
In general, you should use StructuredArguments, unless you have a static analyzer
that flags parameter count / argument count mismatches.
Both StructuredArguments and Markers require constructing additional objects.
Therefore, it is best practice to surround the log lines with logger.isXXXEnabled(),
to avoid the object construction if the log level is disabled.
Examples using StructuredArguments:
import static net.logstash.logback.argument.StructuredArguments.*;
/*
* Add "name":"value" to the JSON output,
* but only add the value to the formatted message.
*
* The formatted message will be `log message value`
*/
logger.info("log message {}", value("name", "value"));
/*
* Add "name":"value" to the JSON output,
* and add name=value to the formatted message.
*
* The formatted message will be `log message name=value`
*/
logger.info("log message {}", keyValue("name", "value"));
/*
* Add "name":"value" ONLY to the JSON output.
*
* Since there is no parameter for the argument,
* the formatted message will NOT contain the key/value.
*
* If this looks funny to you or to static analyzers,
* consider using Markers instead.
*/
logger.info("log message", keyValue("name", "value"));
/*
* Add multiple key value pairs to both JSON and formatted message
*/
logger.info("log message {} {}", keyValue("name1", "value1"), keyValue("name2", "value2")));
/*
* Add "name":"value" to the JSON output and
* add name=[value] to the formatted message using a custom format.
*/
logger.info("log message {}", keyValue("name", "value", "{0}=[{1}]"));
/*
* In the JSON output, values will be serialized by Jackson's ObjectMapper.
* In the formatted message, values will follow the same behavior as logback
* (formatting of an array or if not an array `toString()` is called).
*
* Add "foo":{...} to the JSON output and add `foo.toString()` to the formatted message:
*
* The formatted message will be `log message <result of foo.toString()>`
*/
Foo foo = new Foo();
logger.info("log message {}", value("foo", foo));
/*
* Add "name1":"value1","name2":"value2" to the JSON output by using a Map,
* and add `myMap.toString()` to the formatted message.
*
* Note the values can be any object that can be serialized by Jackson's ObjectMapper
* (e.g. other Maps, JsonNodes, numbers, arrays, etc)
*/
Map myMap = new HashMap();
myMap.put("name1", "value1");
myMap.put("name2", "value2");
logger.info("log message {}", entries(myMap));
/*
* Add "array":[1,2,3] to the JSON output,
* and array=[1,2,3] to the formatted message.
*/
logger.info("log message {}", array("array", 1, 2, 3));
/*
* Add fields of any object that can be unwrapped by Jackson's UnwrappableBeanSerializer to the JSON output.
* i.e. The fields of an object can be written directly into the JSON output.
* This is similar to the @JsonUnwrapped annotation.
*
* The formatted message will contain `myobject.toString()`
*/
logger.info("log message {}", fields(myobject));
/*
* In order to normalize a field object name, static helper methods can be created.
* For example:
* public static StructuredArgument foo(Foo foo) {
* return StructuredArguments.value("foo", foo);
* }
*/
logger.info("log message {}", foo(foo));
Abbreviated convenience methods are available for all the structured argument types.
For example, instead of keyValue(key, value), you can use kv(key, value).
Examples using Markers:
import static net.logstash.logback.marker.Markers.*;
/*
* Add "name":"value" to the JSON output.
*/
logger.info(append("name", "value"), "log message");
/*
* Add "name1":"value1","name2":"value2" to the JSON output by using multiple markers.
*/
logger.info(append("name1", "value1").and(append("name2", "value2")), "log message");
/*
* Add "name1":"value1","name2":"value2" to the JSON output by using a map.
*
* Note the values can be any object that can be serialized by Jackson's ObjectMapper
* (e.g. other Maps, JsonNodes, numbers, arrays, etc)
*/
Map myMap = new HashMap();
myMap.put("name1", "value1");
myMap.put("name2", "value2");
logger.info(appendEntries(myMap), "log message");
/*
* Add "array":[1,2,3] to the JSON output
*/
logger.info(appendArray("array", 1, 2, 3), "log message");
/*
* Add "array":[1,2,3] to the JSON output by using raw json.
* This allows you to use your own json seralization routine to construct the json output
*/
logger.info(appendRaw("array", "[1,2,3]"), "log message");
/*
* Add any object that can be serialized by Jackson's ObjectMapper
* (e.g. Maps, JsonNodes, numbers, arrays, etc)
*/
logger.info(append("object", myobject), "log message");
/*
* Add fields of any object that can be unwrapped by Jackson's UnwrappableBeanSerializer.
* i.e. The fields of an object can be written directly into the json output.
* This is similar to the @JsonUnwrapped annotation.
*/
logger.info(appendFields(myobject), "log message");
The following sections describe the fields included in the JSON output by default for AccessEvents written by the
LogstashAccessEncoder,LogstashAccessLayout, andIf you are using the composite encoders/layouts, then the fields written will vary by the providers you configure.
These fields will appear in every AccessEvent unless otherwise noted. The field names listed here are the default field names. The field names can be customized (see Customizing Standard Field Names).
| Field | Description |
|---|---|
@timestamp | Time of the log event. (yyyy-MM-dd'T'HH:mm:ss.SSSZZ) See customizing timestamp. |
@version | Logstash format version (e.g. 1) See customizing version. |
message | Message in the form ${remoteHost} - ${remoteUser} [${timestamp}] "${requestUrl}" ${statusCode} ${contentLength} |
method | HTTP method |
protocol | HTTP protocol |
status_code | HTTP status code |
requested_url | Request URL |
requested_uri | Request URI |
remote_host | Remote host |
remote_user | Remote user |
content_length | Content length |
elapsed_time | Elapsed time in millis |
Request and response headers are not logged by default, but can be enabled by specifying a field name for them, like this:
<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder">
<fieldNames>
<requestHeaders>request_headers</requestHeaders>
<responseHeaders>response_headers</responseHeaders>
</fieldNames>
</encoder>
See Customizing Standard Field Names) for more details.
To write the header names in lowercase (so that header names that only differ by case are treated the same),
set lowerCaseFieldNames to true, like this:
<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder">
<fieldNames>
<requestHeaders>request_headers</requestHeaders>
<responseHeaders>response_headers</responseHeaders>
</fieldNames>
<lowerCaseHeaderNames>true</lowerCaseHeaderNames>
</encoder>
Headers can be filtered via configuring the requestHeaderFilter and/or the responseHeaderFilter
with a HeaderFilter, such as the
IncludeExcludeHeaderFilter.
The IncludeExcludeHeaderFilter can be configured like this:
<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder">
<fieldNames>
<requestHeaders>request_headers</requestHeaders>
</fieldNames>
<requestHeaderFilter>
<include>Content-Type</include>
</requestHeaderFilter>
</encoder>
Custom filters implementing HeaderFilter
can be used by specifying the filter class like this:
<requestHeaderFilter class="your.package.YourFilterClass"/>
Logstash-logback-encoder uses Jackson to encode log and access events.
Logstash-logback-encoder provides sensible defaults for Jackson, but gives you full control over the Jackson configuration.
For example, you can:
JsonFactory and JsonGeneratorJSON is used by default, but other data formats supported by Jackson can be used.
:warning: When using non-JSON data formats, you must include the appropriate jackson dataformat library on the runtime classpath, typically via a maven/gradle dependency (e.g. for Smile, include
jackson-dataformat-smile).
Decorators are provided for the following data formats:
cbor - CborJsonFactoryDecoratorsmile - SmileJsonFactoryDecoratoryaml - YamlJsonFactoryDecoratorTo use one these formats, specify the <jsonFactoryDecorator> like this:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<jsonFactoryDecorator class="net.logstash.logback.decorate.smile.SmileJsonFactoryDecorator"/>
</encoder>
Other data formats can be used by implementing a custom
net.logstash.logback.decorate.JsonFactoryDecorator.
The following decorators can be used to configure data-format-specific generator features:
SmileFeatureJsonGeneratorDecoratorCborFeatureJsonGeneratorDecoratorYamlFeatureJsonGeneratorDecoratorFor example:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<jsonFactoryDecorator class="net.logstash.logback.decorate.smile.SmileJsonFactoryDecorator"/>
<jsonGeneratorDecorator class="net.logstash.logback.decorate.smile.SmileFeatureJsonGeneratorDecorator">
<disable>WRITE_HEADER</disable>
</jsonGeneratorDecorator>
</encoder>
The JsonFactory and JsonGenerator used to write output can be customized by instances of:
For example, you could enable pretty printing by using the PrettyPrintingJsonGeneratorDecorator
Or customize object mapping like this:
public class ISO8601DateDecorator implements JsonFactoryDecorator {
@Override
public JsonFactory decorate(JsonFactory factory) {
ObjectMapper codec = (ObjectMapper) factory.getCodec();
codec.setDateFormat(new ISO8601DateFormat());
return factory;
}
}
and then specify the decorators in the logback.xml file like this:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<jsonGeneratorDecorator class="net.logstash.logback.decorate.PrettyPrintingJsonGeneratorDecorator"/>
<jsonFactoryDecorator class="your.package.ISO8601DateDecorator"/>
</encoder>
JsonFactory and JsonGenerator features can be enabled/disabled by using the
FeatureJsonFactoryDecorator and FeatureJsonGeneratorDecorator, respectively.
For example:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<jsonFactoryDecorator class="net.logstash.logback.decorate.FeatureJsonFactoryDecorator">
<disable>USE_THREAD_LOCAL_FOR_BUFFER_RECYCLING</disable>
</jsonFactoryDecorator>
<jsonGeneratorDecorator class="net.logstash.logback.decorate.FeatureJsonGeneratorDecorator">
<enable>WRITE_NUMBERS_AS_STRINGS</enable>
</jsonGeneratorDecorator>
</encoder>
See the net.logstash.logback.decorate package and sub-packages for other decorators.
By default, Jackson modules are dynamically registered via
ObjectMapper.findAndRegisterModules().
Therefore, you just need to add jackson modules (e.g. jackson-datatype-jdk8) to the classpath, and they will be dynamically registered.
To disable automatic discovery, set <findAndRegisterJacksonModules>false</findAndRegisterJacksonModules> on the encoder/layout.
If you have a module that Jackson is not able to dynamically discover,
you can register it manually via a JsonFactoryDecorator.
By default, when a string is written as a JSON string value, any character not allowed in a JSON string will be escaped.
For example, the newline character (ASCII 10) will be escaped as \n.
To customize these escape sequences, use the net.logstash.logback.decorate.CharacterEscapesJsonFactoryDecorator.
For example, if you want to use something other than \n as the escape sequence for the newline character, you can do the following:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<jsonFactoryDecorator class="net.logstash.logback.decorate.CharacterEscapesJsonFactoryDecorator">
<escape>
<targetCharacterCode>10</targetCharacterCode>
<escapeSequence>\u2028</escapeSequence>
</escape>
</jsonFactoryDecorator>
</encoder>
You can also disable all the default escape sequences by specifying <includeStandardAsciiEscapesForJSON>false</includeStandardAsciiEscapesForJSON> on the CharacterEscapesJsonFactoryDecorator.
If you do this, then you will need to register custom escapes for each character that is illegal in JSON string values. Otherwise, invalid JSON could be written.
The MaskingJsonGeneratorDecorator
can be used to mask sensitive values (e.g. personally identifiable information (PII) or financial data).
Data to be masked can be identified by path and/or by value.
Paths of fields to mask can be specified in several ways, as shown in the following example:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<jsonGeneratorDecorator class="net.logstash.logback.mask.MaskingJsonGeneratorDecorator">
<!-- The default mask string can optionally be specified by <defaultMask>.
When the default mask string is not specified, **** is used.
-->
<defaultMask>****</defaultMask>
<!-- Field paths to mask added via <path> will use the default mask string -->
<path>singleFieldName</path>
<path>/absolute/path/to/mask</path>
<path>partial/path/to/mask</path>
<path>partial/path/with/*/wildcard</path>
<path>tilde~0slash~1escapedPath</path>
<!-- Multiple field paths can be specified as a comma separated string in the <paths> element. -->
<paths>path1, path2, path3</paths>
<!-- Field paths to mask added via <pathMask> can use a non-default mask string -->
<pathMask>
<path>some/path</path>
<path>some/other/path</path>
<mask>[masked]</mask>
</pathMask>
<pathMask>
<paths>anotherFieldName,anotherFieldName2</paths>
<mask>**anotherCustomMask**</mask>
</pathMask>
<!-- Field paths to mask can be supplied dynamically with an implementation
of MaskingJsonGeneratorDecorator.PathMaskSupplier
-->
<pathMaskSupplier class="your.custom.PathMaskSupplierA"/>
<!-- Custom implementations of net.logstash.logback.mask.FieldMasker
can be used for more advanced masking behavior
-->
<fieldMasker class="your.custom.FieldMaskerA"/>
<fieldMasker class="your.custom.FieldMaskerB"/>
</jsonGeneratorDecorator>
</encoder>
See PathBasedFieldMasker
for the path string format and more examples. But in general:
/ and are absolute to the root of the JSON output event (e.g. /@timestamp would mask the default timestamp field)/ and match anywhere that path sequence is seen in the output./ characters) will match all occurrences of a field with the given name*) will match anything at that location within the path~1 to escape / within a token~0 to escape ~ within a tokenSpecific values to be masked can be specified in several ways, as seen in the following example:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<jsonGeneratorDecorator class="net.logstash.logback.mask.MaskingJsonGeneratorDecorator">
<!-- The default mask string can optionally be specified by <defaultMask>.
When the default mask string is not specified, **** is used.
-->
<defaultMask>****</defaultMask>
<!-- Values to mask added via <value> will use the default mask string -->
<value>^foo$</value>
<value>bar</value>
<!-- Multiple values can be specified as a comma separated string in the <values> element. -->
<values>
^baz$,
^blah$
</values>
<!-- Values to mask added via <valueMask> can use a non-default mask string
The mask string here can reference regex capturing groups if needed
-->
<valueMask>
<value>^(foo)-.*$</value>
<value>^(bar)-.*$</value>
<mask>$1****</mask>
</valueMask>
<!-- Values to mask can be supplied dynamically with an implementation of
MaskingJsonGeneratorDecorator.ValueMaskSupplier
-->
<valueMaskSupplier class="your.custom.ValueMaskSupplierA"/>
<!-- Custom implementations of net.logstash.logback.mask.ValueMasker
can be used for more advanced masking behavior
-->
<valueMasker class="your.custom.ValueMaskerA"/>
<valueMasker class="your.custom.ValueMaskerB"/>
</jsonGeneratorDecorator>
</encoder>
Identifying data to mask by value is much more expensive than identifying data to mask by path. Therefore, prefer identifying data to mask by path.
The value to mask is passed through every value masker, with the output of one masker passed as input to the next masker. This allows each masker to mask specific substrings within the value. The order in which the maskers are executed is not defined, and should not be relied upon.
When using regexes to identify strings to mask, all matches within each string field value will be replaced.
If you want to match the full string field value, then use the beginning of line (^) and end of line ($) markers.
The standard field names above for LoggingEvents and AccessEvents can be customized by using the fieldNamesconfiguration element in the encoder or appender configuration.
For example:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<fieldNames>
<timestamp>time</timestamp>
<message>msg</message>
<stackTrace>stacktrace</stackTrace>
...
</fieldNames>
</encoder>
Prevent a field from being output by setting the field name to [ignore].
For LoggingEvents, see LogstashFieldNames
for all the field names that can be customized. Each java field name in that class is the name of the xml element that you would use to specify the field name (e.g. logger, levelValue). Additionally, a separate set of shortened field names can be configured like this:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<fieldNames class="net.logstash.logback.fieldnames.ShortenedFieldNames"/>
</encoder>
For LoggingEvents, log the caller info, MDC properties, and context properties
in sub-objects within the JSON event by specifying field
names for caller, mdc, and context, respectively.
For AccessEvents, see LogstashAccessFieldNames
for all the field names that can be customized. Each java field name in that class is the name of the xml element that you would use to specify the field name (e.g. fieldsMethod, fieldsProtocol).
The version field value by default is the string value 1.
The value can be changed like this:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<version>2</version>
</encoder>
The value can be written as a number (instead of a string) like this:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<writeVersionAsInteger>true</writeVersionAsInteger>
</encoder>
By default, timestamps are written as string values in the format specified by
DateTimeFormatter.ISO_OFFSET_DATE_TIME
(e.g. 2019-11-03T10:15:30.123+01:00), in the default TimeZone of the host Java platform.
You can change the pattern like this:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<timestampPattern>yyyy-MM-dd'T'HH:mm:ss.SSS</timestampPattern>
</encoder>
The value of the timestampPattern can be any of the following:
[UNIX_TIMESTAMP_AS_NUMBER] - timestamp written as a JSON number value of the milliseconds since unix epoch[UNIX_TIMESTAMP_AS_STRING] - timestamp written as a JSON string value of the milliseconds since unix epoch[ constant ] - (e.g. [ISO_OFFSET_DATE_TIME]) timestamp written using the given DateTimeFormatter constantyyyy-MM-dd'T'HH:mm:ss.SSS) timestamp written using a DateTimeFormatter created from the given patternThe provider uses a standard Java DateTimeFormatter under the hood. However, special optimisations are applied when using one of the following standard ISO formats that make it nearly 7x faster and more GC friendly:
[ISO_OFFSET_DATE_TIME][ISO_ZONED_DATE_TIME][ISO_LOCAL_DATE_TIME][ISO_DATE_TIME][ISO_INSTANT]With logback 1.3+ the timestamp will have millisecond precision.
The formatter uses the default TimeZone of the host Java platform by default. You can change it like this:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<timeZone>UTC</timeZone>
</encoder>
The value of the timeZone element can be any string accepted by java's TimeZone.getTimeZone(String id) method.
For example America/Los_Angeles, GMT+10 or UTC.
Use the special value [DEFAULT] to use the default TimeZone of the system.
By default, LoggingEvent messages are written as JSON strings. Any characters not allowed in a JSON string, such as newlines, are escaped. See the Customizing Character Escapes section for details.
You can also write messages as JSON arrays instead of strings, by specifying a messageSplitRegex to split the message text.
This configuration element can take the following values:
SYSTEM (uses the system-default line separator)UNIX (uses \n)WINDOWS (uses \r\n)If you split the log message by the origin system's line separator, the written message does not contain any embedded line separators. The target system can unambiguously parse the message without any knowledge of the origin system's line separators.
For example:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<messageSplitRegex>SYSTEM</messageSplitRegex>
</encoder>
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<messageSplitRegex>\r?\n</messageSplitRegex>
</encoder>
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<messageSplitRegex>#+</messageSplitRegex>
</encoder>
By default, AccessEvent messages are written in the following format:
%clientHost - %user [%date] "%requestURL" %statusCode %bytesSent
To customize the message pattern, specify the messagePattern like this:
<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder">
<messagePattern>%clientHost [%date] "%requestURL" %statusCode %bytesSent</messagePattern>
</encoder>
The pattern can contain any of the AccessEvent conversion words.
For LoggingEvents, you can shorten the logger name field length similar to the normal pattern style of %logger{36}.
Examples of how it is shortened can be found here.
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<shortenedLoggerNameLength>36</shortenedLoggerNameLength>
</encoder>
The algorithm will shorten the logger name and attempt to reduce its size down to a maximum of number of characters. It does so by reducing each part between dots to their first letter and gradually expand them starting from the right most element until the maximum size is reached.
To enable this feature, set the shortenedLoggerNameLength property to the desired value.
Setting the length to zero constitutes an exception and returns only the part of the logger name after last dot.
Use -1 to disable shortening entirely.
The next table provides examples of the abbreviation algorithm in action.
| LENGTH | LOGGER NAME | SHORTENED |
|---|---|---|
| 0 | org.company.stack.Sample | Sample |
| 5 | org.company.stack.Sample | o.c.s.Sample |
| 16 | org.company.stack.Sample | o.c.stack.Sample |
| 22 | org.company.stack.Sample | o.company.stack.Sample |
| 25 | org.company.stack.Sample | org.company.stack.Sample |
When logging exceptions,
stack traces are formatted using logback's ExtendedThrowableProxyConverter by default.
However, you can configure the encoder to use any ThrowableHandlingConverter to format stacktraces.
Note that the ThrowableHandlingConverter only applies to the
exception passed as an extra argument
to the log method, the way you normally log an exception in slf4j.
Do NOT use structured arguments or markers for exceptions.
A powerful ShortenedThrowableConverter is included in the logstash-logback-encoder library to format stacktraces with the features listed in the next sections.
This converter can even be used within a PatternLayout to format stacktraces in any non-JSON logs you may have.
Nested stacktraces often contain redudant frames that can safely be omitted without loosing any valuable information.
The following example shows a standard stack trace of an exception with a single root cause:
java.lang.RuntimeException: Unable to invoke service
at org.company.stack.gen.StackGenerator.causedBy(StackGenerator.java:40)
at org.company.stack.gen.StackGenerator.generateCausedBy(StackGenerator.java:34)
at org.company.stack.framework.Dispatcher.invoke(Dispatcher.java:11)
at org.company.stack.framework.Dispatcher.dispatch(Dispatcher.java:8)
at org.company.stack.Sample.execute(Sample.java:36)
at org.company.stack.Sample.omitCommonFrames(Sample.java:22)
at org.company.stack.Sample.main(Sample.java:18)
Caused by: java.lang.RuntimeException: Destination unreachable
at org.company.stack.gen.StackGenerator.two(StackGenerator.java:58)
at org.company.stack.gen.StackGenerator.one(StackGenerator.java:55)
at org.company.stack.gen.StackGenerator.causedBy(StackGenerator.java:38)
at org.company.stack.gen.StackGenerator.generateCausedBy(StackGenerator.java:34)
at org.company.stack.framework.Dispatcher.invoke(Dispatcher.java:11)
at org.company.stack.framework.Dispatcher.dispatch(Dispatcher.java:8)
at org.company.stack.Sample.execute(Sample.java:36)
at org.company.stack.Sample.omitCommonFrames(Sample.java:22)
at org.company.stack.Sample.main(Sample.java:18)
As we can see, the exception and the root cause have the first 7 frames in common. The overall stack trace length can be reduced by omitting these redundant frames from the root cause, as shown below:
java.lang.RuntimeException: Unable to invoke service
at org.company.stack.gen.StackGenerator.causedBy(StackGenerator.java:40)
at org.company.stack.gen.StackGenerator.generateCausedBy(StackGenerator.java:34)
at org.company.stack.framework.Dispatcher.invoke(Dispatcher.java:11)
at org.company.stack.framework.Dispatcher.dispatch(Dispatcher.java:8)
at org.company.stack.Sample.execute(Sample.java:36)
at org.company.stack.Sample.omitCommonFrames(Sample.java:22)
at org.company.stack.Sample.main(Sample.java:18)
Caused by: java.lang.RuntimeException: Destination unreachable
at org.company.stack.gen.StackGenerator.two(StackGenerator.java:58)
at org.company.stack.gen.StackGenerator.one(StackGenerator.java:55)
at org.company.stack.gen.StackGenerator.causedBy(StackGenerator.java:38)
... 6 common frames omitted
Common frames are omitted and replaced with a message indicating the number of frames dropped. Note that the last common frame remains as a visual cue to the reader.
This feature is enabled by default and can be disabled if desired by setting the omitCommonFrames property to false.
It is possible to use regular expressions to truncate the stacktrace after the first matching stacktrace element. The strings being matched against are in the form "fullyQualifiedClassName.methodName".
For example, to suppress everything below org.company.stack.framework package or after a call to the StackGenerator.one() method, configure the following:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<truncateAfter>^org\.company\.stack\.framework\..*</truncateAfter>
<truncateAfter>^\.StackGenerator\.one</truncateAfter>
</throwableConverter>
</encoder>
This will produce something similar to the following:
java.lang.RuntimeException: Unable to invoke service
at org.company.stack.gen.StackGenerator.causedBy(StackGenerator.java:40)
at org.company.stack.gen.StackGenerator.generateCausedBy(StackGenerator.java:34)
at org.company.stack.framework.Dispatcher.invoke(Dispatcher.java:11)
... 4 frames truncated
Caused by: java.lang.RuntimeException: Destination unreachable
at org.company.stack.gen.StackGenerator.two(StackGenerator.java:58)
at org.company.stack.gen.StackGenerator.one(StackGenerator.java:55)
... 7 frames truncated (including 6 common frames)
Note how the stacktrace is truncated after the matching frames.
Alternatively, multiple regex patterns can be specified at once using the <truncateAfters> configuration keyword. This property accepts an optional comma separated list of patterns. The previous example configuration can also be written as follows:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<truncateAfters>
^org\.company\.stack\.framework\..*,
^\.StackGenerator\.one
</truncateAfters>
</throwableConverter>
</encoder>
Using the <truncateAfters> configuration option can be useful when using an environment variable to specify the actual patterns at deployment time.
Sometimes portions of the stacktrace are not worthy of interest and you want to exclude them to make the overall stacktrace shorter. The encoder allows to filter out consecutive unwanted stacktrace elements based on regular expressions and replace them with a single message indicating "something" has been removed.
To enable this feature, simply define the regex patterns matching the frames you want to exclude using one or more <exclude> configuration keywords.
Take the following stacktrace as an example:
java.lang.RuntimeException: Destination unreachable
at org.company.stack.gen.StackGenerator.two(StackGenerator.java:58)
at org.company.stack.gen.StackGenerator.one(StackGenerator.java:55)
at org.company.stack.gen.StackGenerator.threeSingle$SpringCGLIB(StackGenerator.java:14)
at org.company.stack.gen.StackGenerator.twoSingle$SpringCGLIB(StackGenerator.java:11)
at org.company.stack.gen.StackGenerator.oneSingle$SpringCGLIB(StackGenerator.java:8)
at org.company.stack.gen.StackGenerator.generateSingle(StackGenerator.java:5)
at org.company.stack.framework.Dispatcher.invoke(Dispatcher.java:11)
at org.company.stack.framework.Dispatcher.dispatch(Dispatcher.java:8)
at org.company.stack.Sample.execute(Sample.java:107)
at org.company.stack.Sample.exclude(Sample.java:94)
at org.company.stack.Sample.main(Sample.java:19)
Suppose the last three frames are common to all your exceptions (they come from the application bootstrap) and you want to reduce them to a single line for brevetiy. Also, you are not interested in frames with package.classname ending with $SpringCGLIB because they are generated by the Spring runtime... To do this, you will use the following configuration:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<exclude>\$SpringCGLIB$</exclude>
<exclude>^org\.company\.stack\.Sample\..*</exclude>
</throwableConverter>
</encoder>
And your stacktrace would be rendered as follows:
java.lang.RuntimeException: Destination unreachable
at org.company.stack.gen.StackGenerator.two(StackGenerator.java:58)
at org.company.stack.gen.StackGenerator.one(StackGenerator.java:55)
... 3 frames excluded
at org.company.stack.gen.StackGenerator.generateSingle(StackGenerator.java:5)
at org.company.stack.framework.Dispatcher.invoke(Dispatcher.java:11)
at org.company.stack.framework.Dispatcher.dispatch(Dispatcher.java:8)
... 3 frames excluded
Note that the converter effectively removes stack trace elements only if at least TWO consecutive frames match the configured regex patterns. This is to avoid replacing a single frame with "... 1 frames excluded" that doesn't shorten the stacktrace at all...
In addition, the first frame of the stacktrace is always output and cannot be excluded.
Alternatively, multiple exclusion patterns can be specified at once using the <exclusions> configuration keyword. This property accepts an optional comma separated list of patterns. The previous example configuration can also be written as follows:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<exclusions>
\$SpringCGLIB$,
^org\.company\.stack\.Sample\..*
</exclusions>
</throwableConverter>
</encoder>
Using the <exclusions> configuration option can be useful when using an environment variable to specify the actual patterns at deployment time.
The maxDepthPerThrowable property is used to limit the depth of each individual throwable nested inside the original exception, caused-bys and suppressed exceptions included. Beyond this limit, additional elements are omitted and a message indicating the number elements removed is added instead.
For example, the following configuration limits the stacktrace to 2 elements:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<maxDepthPerThrowable>2</maxDepthPerThrowable>
</throwableConverter>
</encoder>
This would produce something similar to the following:
java.lang.RuntimeException: Unable to invoke service
at org.company.stack.gen.StackGenerator.causedBy(StackGenerator.java:40)
at org.company.stack.gen.StackGenerator.generateCausedBy(StackGenerator.java:34)
... 5 frames truncated
Caused by: java.lang.RuntimeException: Destination unreachable
at org.company.stack.gen.StackGenerator.two(StackGenerator.java:58)
at org.company.stack.gen.StackGenerator.one(StackGenerator.java:55)
... 7 frames truncated (including 6 common frames)
Note how the maximum depth applies to each individual throwables. The last message indicates that 7 frames were truncated of which 6 are common to both the exception and the cause.
The special value -1 can be used to disable the feature and allow for an unlimited depth (no limit), which is the default.
The maxLength property is used to set a limit on the size of the overall trace, all throwables combined.
For example, use the following configuration to limit the size to 256 characters:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<maxLength>256</maxLength>
</throwableConverter>
</encoder>
The overall trace will be limited to 256 characters like this:
java.lang.RuntimeException: Unable to invoke service
at org.company.stack.gen.StackGenerator.causedBy(StackGenerator.java:40)
at org.company.stack.gen.StackGenerator.generateCausedBy(StackGenerator.java:34)
at org.company.stack.framework.Dispatcher....
The special value -1 can be used to disable the feature and allow for an unlimited length (no limit), which is the default.
Class names can be abbreviated in a way similar to the Logback layout feature. The algorithm will shorten the full class name (package + class) and attempt to reduce its size down to a maximum of number of characters. It does so by reducing the package elements to their first letter and gradually expand them starting from the right most element until the maximum size is reached.
To enable this feature, set the shortenedClassNameLength property to the desired value.
Setting the length to zero constitutes an exception and returns the "simple" class name without package name.
Set length to -1 to disable shortening entirely.
The next table provides examples of the abbreviation algorithm in action.
| LENGTH | CLASSNAME | SHORTENED |
|---|---|---|
| 0 | org.company.stack.Sample | Sample |
| 5 | org.company.stack.Sample | o.c.s.Sample |
| 16 | org.company.stack.Sample | o.c.stack.Sample |
| 22 | org.company.stack.Sample | o.company.stack.Sample |
| 25 | org.company.stack.Sample | org.company.stack.Sample |
For example, use the following configuration to try to shorten the class names down to 25 characters:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<shortenedClassNameLength>25</shortenedClassNameLength>
</throwableConverter>
</encoder>
This will produce an output similar to this:
j.lang.RuntimeException: Unable to invoke service
at o.c.s.gen.StackGenerator.causedBy(StackGenerator.java:40)
at o.c.s.gen.StackGenerator.generateCausedBy(StackGenerator.java:34)
at o.c.s.f.Dispatcher.invoke(Dispatcher.java:11)
at o.c.s.f.Dispatcher.dispatch(Dispatcher.java:8)
at org.company.stack.Sample.execute(Sample.java:97)
at org.company.stack.Sample.classNameShortening(Sample.java:77)
at org.company.stack.Sample.main(Sample.java:19)
Caused by: j.lang.RuntimeException: Destination unreachable
at o.c.s.gen.StackGenerator.two(StackGenerator.java:58)
at o.c.s.gen.StackGenerator.one(StackGenerator.java:55)
at o.c.s.gen.StackGenerator.causedBy(StackGenerator.java:38)
... 6 common frames omitted
Note that the exception name is also shortened, as are the individual frames.
Alternatively you can specify your own custom abbreviation strategy with the <classNameAbbreviator> configuration property as shown below:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<throwableConverter>
<classNameAbbreviator class="your.own.CustomAbbreviator">
<param1>aValue</param1>
</classNameAbbreviator>
</throwableConverter>
</encoder>
Note The value of
<shortenedClassNameLength>property is ignored when a custom abbreviator is explicitly specified.
Stacktrace elements are sperated by the SYSTEM line separator by default.
The linesSeparator property can be used to specify a different value. The line separator can be specified as:
SYSTEM (uses the system default)UNIX (uses \n)WINDOWS (uses \r\n), orFor example, to use a pipe (|) as separator between stacktrace elements you would use the following configuration:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<lineSeparator>|</lineSeparator>
</throwableConverter>
</encoder>
The stacktrace will be rendered on a single line with | between frames as follows (the line is truncated for readability):
java.lang.RuntimeException: Unable to invoke service| at org.company.stack.gen.StackGenerator.causedBy(StackGenerator.java:40)| at org.c
Stacktraces are usually rendered with the root cause appearing last.
You can invert the order and have the root cause output first by setting the rootCauseFirst property to true (false by default).
Sample output:
java.lang.RuntimeException: Destination unreachable
at org.company.stack.gen.StackGenerator.two(StackGenerator.java:58)
at org.company.stack.gen.StackGenerator.one(StackGenerator.java:55)
at org.company.stack.gen.StackGenerator.causedBy(StackGenerator.java:38)
... 6 common frames omitted
Wrapped by: java.lang.RuntimeException: Unable to invoke service
at org.company.stack.gen.StackGenerator.causedBy(StackGenerator.java:40)
at org.company.stack.gen.StackGenerator.generateCausedBy(StackGenerator.java:34)
at org.company.stack.framework.Dispatcher.invoke(Dispatcher.java:11)
at org.company.stack.framework.Dispatcher.dispatch(Dispatcher.java:8)
at org.company.stack.Sample.execute(Sample.java:79)
at org.company.stack.Sample.rootCauseFirst(Sample.java:66)
at org.company.stack.Sample.main(Sample.java:18)
Standard Logback EventEvaluators can be used to determine if the stacktrace should be rendered.
EventEvaluators are used to skip generation of the stack trace for matching ILoggingEvents. In other words, an evaluator must evaluate to false (do not skip) to include the stacktrace...
The following sample configuration leverage the Logback JaninoEventEvaluator event evaluator to output the stacktrace only if the log message contains the word billing:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<evaluator class="ch.qos.logback.classic.boolex.JaninoEventEvaluator">
<expression>return !message.contains("billing");</expression>
</evaluator>
</throwableConverter>
</encoder>
Multiple evaluators can be registered and are evaluated in the order in which they are registered. The stacktrace is only generated if all evaluators returned false.
To Be Documented
Computing and inlining hexadecimal hashes for each exception stack using the inlineHash or stackHash provider (more info).
To use this with a PatternLayout, you must configure a new "conversionRule" as described here.
For example:
<!-- Define a new conversion rule named "stack" -->
<conversionRule conversionWord="stack"
converterClass="net.logstash.logback.stacktrace.ShortenedThrowableConverter" />
This configuration registers the ShortenedThrowableConverter under the name stack. From there the converter can be used in a PatternLayout using the syntax %stack{options} with optional configuration options between {}, each separated by a comma.
The first three options must appear in the following order:
full or short or an integer valuefull or short or an integer valuefull or short or an integer valueThe remaining options can appear in any order and are interpreted as follows:
rootFirst - indicating that stacks should be printed root-cause firstinlineHash - indicating that hexadecimal error hashes should be computed and inlinedinline - indicating that the whole stack trace should be inlined, using \\n as separatoromitCommonFrames - indicating that common frames should be omittedkeepCommonFrames - indicating that common frames should be preservedFor example,
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>[%thread] - %msg%n%stack{5,1024,10,rootFirst,omitCommonFrames,regex1,regex2,evaluatorName}</pattern>
</encoder>
</appender>
Note that it is not possible to configure the truncateAfter feature when the converter is used within a pattern layout.
LogstashEncoder, LogstashAccessEncoder and their "layout" counterparts all come with a predefined set of encoders. You can register additional JsonProviders using the <provider> configuration property as shown in the following example:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<!-- Add a new provider after those than come with the LogstashEncoder -->
<provider class="net.logstash.logback.composite.loggingevent.LoggingEventPatternJsonProvider">
<pattern>
{
"message": "%mdc{custom_value} %message"
}
</pattern>
</provider>
<!-- Disable the default message provider -->
<fieldNames>
<message>[ignore]</message>
</fieldNames>
</encoder>
You can add several additional JsonProviders using multiple <provider> entries. They will appear just after the default providers registered by the LogstashEncoder.
In this example, the pattern provider produces a "message" JSON field that will conflict with the message field produced by the MessageJsonProvider already registered by the LogstashEncoder itself. Different options to avoid the conflict:
You can specify a prefix (written before the JSON object), suffix (written after the JSON object), and/or line separator (written after suffix), which may be required for the log pipeline you are using, such as:
@cee: prefix.For example, to add standard syslog headers for syslog over UDP, configure the following:
<configuration>
<conversionRule conversionWord="syslogStart" converterClass="ch.qos.logback.classic.pattern.SyslogStartConverter"/>
<appender name="stash" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
<host>MyAwesomeSyslogServer</host>
<!-- port is optional (default value shown) -->
<port>514</port>
<layout>
<prefix class="ch.qos.logback.classic.PatternLayout">
<pattern>%syslogStart{USER}</pattern>
</prefix>
</layout>
</appender>
...
</configuration>
When using the LogstashEncoder, LogstashAccessEncoder or a composite encoder, the prefix is an Encoder, not a Layout, so you will need to wrap the prefix PatternLayout in a LayoutWrappingEncoder like this:
<configuration>
...
<appender ...>
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
...
<prefix class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>@cee:</pattern>
</layout>
</prefix>
</encoder>
</appender>
</configuration>
Note that logback's xml configuration reader will trim whitespace from xml element values. Therefore, if you want to end the prefix or suffix pattern with whitespace, first add the whitespace, and then add something like %mdc{keyThatDoesNotExist} after it. For example <pattern>your pattern %mdc{keyThatDoesNotExist}</pattern>. This will cause logback to output the whitespace as desired, and then a blank string for the MDC key that does not exist.
The line separator, which is written after the suffix, can be specified as:
SYSTEM (uses the system default)UNIX (uses \n)WINDOWS (uses \r\n), orFor example:
<configuration>
...
<appender ...>
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
...
<lineSeparator>UNIX</lineSeparator>
</encoder>
</appender>
</configuration>
If you want greater flexibility in the JSON format and data included in LoggingEvents and AccessEvents, use the LoggingEventCompositeJsonEncoder and AccessEventCompositeJsonEncoder (or the corresponding layouts).
These encoders/layouts are composed of one or more JSON providers that contribute to the JSON output. No providers are configured by default in the composite encoders/layouts. You must add the ones you want.
For example:
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<mdc/>
<pattern>
<pattern>
{
"timestamp": "%date{ISO8601}",
"myCustomField": "fieldValue",
"relative": "#asLong{%relative}"
}
</pattern>
</pattern>
<stackTrace>
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<maxDepthPerThrowable>30</maxDepthPerThrowable>
<maxLength>2048</maxLength>
<shortenedClassNameLength>20</shortenedClassNameLength>
<exclude>^sun\.reflect\..*\.invoke</exclude>
<exclude>^net\.sf\.cglib\.proxy\.MethodProxy\.invoke</exclude>
<evaluator class="myorg.MyCustomEvaluator"/>
<rootCauseFirst>true</rootCauseFirst>
</throwableConverter>
</stackTrace>
</providers>
</encoder>
The logstash-logback-encoder library contains many providers out-of-the-box,
and you can even plug-in your own by extending JsonProvider.
Each provider has its own configuration options to further customize it.
These encoders/layouts make use of an internal buffer to hold the JSON output during the rendering process.
The size of this buffer is set to 1024 bytes by default. A different size can be configured by setting the minBufferSize property to the desired value.
The buffer automatically grows above the minBufferSize when needed to accommodate with larger events. However, only the first minBufferSize bytes will be reused by subsequent invocations. It is therefore strongly advised to set the minimum size at least equal to the average size of the encoded events to reduce unnecessary memory allocations and reduce pressure on the garbage collector.
The table below lists the providers available to both LoggingEvents and AccessEvents. The provider name is the xml element name to use when configuring.
| Provider | Description/Properties |
|---|---|
| context | Outputs entries from logback's context.
|
| nestedField |
Nests a JSON object under the configured fieldName. The nested object is populated by other providers added to this provider. See Nested JSON provider.
|
| pattern |
Outputs fields from a configured JSON Object string, while substituting patterns supported by logback's PatternLayout.
|
| sequence |
Event sequence number. With Logback 1.3+ the sequence number is obtained from the event itself as long as the LoggerContext is configured with a `SequenceNumberGenerator` (which is not by default). If no SequenceNumberGenerator is configured, the provider emits a warning and reverts to a locally generated incrementing number starting at 1.
|
| threadName | Name of the thread from which the event was logged.
|
| timestamp | Event timestamp.
|
| uuid |
Outputs random UUID as field value. Handy when you want to provide unique identifier for log lines.
Note: The com.fasterxml.uuid:java-uuid-generator optional dependency must be added to applications that use the `uuid` provider. |
| version | Logstash JSON format version.
|
The common providers mentioned above, and the providers listed in the table below, are available for LoggingEvents. The provider name is the xml element name to use when configuring. Each provider's configuration properties are shown, with default configuration values in parenthesis.
| Provider | Description/Properties |
|---|---|
| arguments |
Outputs fields from the event arguments array. See Event-specific Custom Fields.
|
| callerData | Outputs data about from where the logger was called (class/method/file/line).
|
| contextName | Outputs the name of logback's context.
|
| loggerName | Name of the logger that logged the message.
|
| logLevel | Logger level text (INFO, WARN, etc).
|
| logLevelValue | Logger level numerical value.
|
| logstashMarkers | Used to output Logstash Markers as specified in Event-specific Custom Fields. |
| mdc |
Outputs entries from the Mapped Diagnostic Context (MDC). Will include all entries by default. When key names are specified for inclusion, then all other fields will be excluded. When key names are specified for exclusion, then all other fields will be included. It is a configuration error to specify both included and excluded key names.
|
| keyValuePairs |
Outputs key value pairs added via slf4j's fluent api. Will include all key value pairs by default. When key names are specified for inclusion, then all other keys will be excluded. When key names are specified for exclusion, then all other keys will be included. It is a configuration error to specify both included and excluded key names.
|
| message | Formatted log event message.
|
| rawMessage | Raw log event message, as opposed to formatted log where parameters are resolved.
|
| rootStackTraceElement | (Only if a throwable was logged) Outputs a JSON Object containing the class and method name from which the outer-most exception was thrown.
|
| stackHash | (Only if a throwable was logged) Computes and outputs a hexadecimal hash of the throwable stack. This helps identifying several occurrences of the same error (more info).
|
| stackTrace | Stacktrace of any throwable logged with the event. Stackframes are separated by newline chars.
|
| tags | Outputs logback markers as a comma separated list.
|
| throwableClassName | (Only if a throwable was logged) Outputs a field that contains the class name of the thrown Throwable.
|
| throwableMessage | (Only if a throwable was logged) Outputs a field that contains the message of the thrown Throwable.
|
| throwableRootCauseClassName | (Only if a throwable was logged and a root cause could be determined) Outputs a field that contains the class name of the root cause of the thrown Throwable.
|
| throwableRootCauseMessage | (Only if a throwable was logged and a root cause could be determined) Outputs a field that contains the message of the root cause of the thrown Throwable.
|
The common providers mentioned above, and the providers listed in the table below, are available for AccessEvents. The provider name is the xml element name to use when configuring. Each provider's configuration properties are shown, with default configuration values in parenthesis.
| Provider | Description/Properties |
|---|---|
| contentLength | Content length.
|
| elapsedTime | Elapsed time in milliseconds.
|
| message | Message in the form `${remoteHost} - ${remoteUser} [${timestamp}] "${requestUrl}" ${statusCode} ${contentLength}`.
|
| method | HTTP method.
|
| protocol | HTTP protocol.
|
| remoteHost | Remote Host.
|
| remoteUser | Remote User.
|
| requestedUri | Requested URI.
|
| requestedUrl | Requested URL.
|
| requestHeaders | Include the request headers.
|
| responseHeaders | Include the response headers.
|
| statusCode | HTTP status code.
|
Use the nestedField provider to create a sub-object in the JSON event output.
For example...
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<timestamp/>
<nestedField>
<fieldName>fields</fieldName>
<providers>
<logLevel/>
</providers>
</nestedField>
</providers>
</encoder>
...will produce something like...
{
"@timestamp": "...",
"fields": {
"level": "DEBUG"
}
}
When used with a composite JSON encoder/layout, the pattern JSON provider can be used to
define a template for a portion of the logged JSON output.
The encoder/layout will populate values within the template.
Every value in the template is treated as a pattern for logback's standard PatternLayout so it can be a combination
of literal strings (for some constants) and various conversion specifiers (like %d for date).
The pattern string (configured within the pattern provider) must be a JSON Object. The contents of the JSON object are included within the logged JSON output.
This example...
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<!-- provides the timestamp -->
<timestamp/>
<!-- provides the version -->
<version/>
<!-- provides the fields in the configured pattern -->
<pattern>
<!-- the pattern that defines what to include -->
<pattern>
{ "level": "%level" }
</pattern>
</pattern>
</providers>
</encoder>
... will produce something like...
{
"@timestamp": "...",
"@version": "1",
"level": "DEBUG"
}
The real power comes from the fact that there are lots of standard conversion specifiers so you
can customise what is logged and how. For example, you could log a single specific value from MDC with %mdc{mykey}.
Or, for access logs, you could log a single request header with %i{User-Agent}.
You can use nested objects and arrays in your pattern.
If you use a null, number, or a boolean constant in a pattern, it will keep its type in the
resulting JSON. However, only the text values are searched for conversion patterns.
And, as these patterns are sent through PatternLayout, the result is always a string
even for something which you may feel should be a number - like for %b (bytes sent, in access logs).
You can either deal with the type conversion on the logstash side or you may use special operations provided by this encoder. The operations are:
#asLong{...} - evaluates the pattern in curly braces and then converts resulting string to a Long (or a null if conversion fails).#asDouble{...} - evaluates the pattern in curly braces and then converts resulting string to a Double (or a null if conversion fails).#asBoolean{...}- evaluates the pattern in curly braces and then converts resulting string to a Boolean. Conversion is case insensitive. true, yes, y and 1 (case insensitive) are converted to a boolean true, a null or empty string is converted to null, anything else returns false.asNullIfEmpty{...} - evaluates the pattern in curly braces and the converts resulting string into null if it is empty.#asJson{...} - evaluates the pattern in curly braces and then converts resulting string to json (or a null if conversion fails).#tryJson{...} - evaluates the pattern in curly braces and then converts resulting string to json (or just the string if conversion fails).So this example...
<pattern>
{
"line_str": "%line",
"line_long": "#asLong{%line}",
"has_message": "#asBoolean{%mdc{hasMessage}}",
"json_message": "#asJson{%message}"
}
</pattern>
... and this logging code...
MDC.put("hasMessage", "true");
LOGGER.info("{\"type\":\"example\",\"msg\":\"example of json message with type\"}");
...will produce something like...
{
"line_str": "97",
"line_long": 97,
"has_message": true,
"json_message": {"type":"example","msg":"example of json message with type"}
}
Note that the value that is sent for line_long is a number even though in your pattern it is a quoted text.
And the json_message field value is a json object, not a string.
You can escape an operation by prefixing it with \ if you don't want it to be interpreted.
The pattern provider can be configured to omit fields with the following empty values:
null"")[]){})To omit fields with empty values, configure omitEmptyFields to true (default is false), like this:
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<pattern>
<omitEmptyFields>true</omitEmptyFields>
<pattern>
{
"logger": "%logger",
"level": "%level",
"thread": "%thread",
"message": "%message",
"traceId": "%mdc{traceId}"
}
</pattern>
</pattern>
</providers>
</encoder>
If the MDC did not contain a traceId entry, then a JSON log event from the above pattern would not contain the traceId field...
{
"logger": "com.example...",
"level": "DEBUG",
"thread": "exec-1",
"message": "Hello World!"
}
For LoggingEvents, conversion specifiers from logback-classic's
PatternLayout are supported.
For example:
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<timestamp/>
<pattern>
<pattern>
{
"custom_constant": "123",
"tags": ["one", "two"],
"logger": "%logger",
"level": "%level",
"thread": "%thread",
"message": "%message",
...
}
</pattern>
</pattern>
</providers>
</encoder>
Note that the %property{key} conversion specifier behaves slightly differently when used in the context of the Pattern Json provider. If the property cannot be found in the logger context or the System properties, it returns an empty string instead of null as it would normally do. For example, assuming the "foo" property is not defined, %property{foo} would return "" (an empty string) instead of "null" (a string whose content is made of 4 letters).
The property conversion specifier also allows you to specify a default value to use when the property is not defined. The default value is optional and can be specified using the :- operator as in Bash shell. For example, assuming the "foo" property is not defined, %property{foo:-bar} will return bar.
For AccessEvents, conversion specifiers from logback-access's
PatternLayout are supported.
For example:
<encoder class="net.logstash.logback.encoder.AccessEventCompositeJsonEncoder">
<providers>
<pattern>
<pattern>
{
"custom_constant": "123",
"tags": ["one", "two"],
"remote_ip": "%a",
"status_code": "%s",
"elapsed_time": "%D",
"user_agent": "%i{User-Agent}",
"accept": "%i{Accept}",
"referer": "%i{Referer}",
"session": "%requestCookie{JSESSIONID}",
...
}
</pattern>
</pattern>
</providers>
</encoder>
There is also a special operation that can be used with this AccessEvents:
#nullNA{...} - if the pattern in curly braces evaluates to a dash (-), it will be replaced with a null value.You may want to use it because many of the PatternLayout conversion words from logback-access will evaluate to -
for non-existent value (for example for a cookie, header or a request attribute).
So the following pattern...
<pattern>
{
"default_cookie": "%requestCookie{MISSING}",
"filtered_cookie": "#nullNA{%requestCookie{MISSING}}"
}
</pattern>
...will produce...
{
"default_cookie": "-",
"filtered_cookie": null
}
You can create your own JSON provider by implementing the JsonProvider interface (or extending one of the existing classes that implements the JsonProvider interface).
Then, add the provider to a LoggingEventCompositeJsonEncoder like this:
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
...
<provider class="your.provider.YourJsonProvider">
<!-- Any properties exposed by your provider can be set here -->
</provider>
...
</providers>
</encoder>
or a LogstashEncoder like this:
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
...
<provider class="your.provider.YourJsonProvider">
<!-- Any properties exposed by your provider can be set here -->
</provider>
...
</encoder>
You can do something similar for AccessEventCompositeJsonEncoder and LogstashAccessEncoder as well, if your JsonProvider handles IAccessEvents.
During execution, the encoders/appenders/layouts provided in logstash-logback-encoder
will add logback status messages to the logback StatusManager.
These status messages are typically reported via a logback StatusListener.
Since the async appenders (especially the tcp appenders) report warnings and errors via the status manager, a default status listener that outputs WARN and ERROR level status messages to standard out will be registered on startup if a status listener has not already been registered. To disable the automatic registering of the default status listener by an appender, do one of the following:
<addDefaultStatusListener>false</addDefaultStatusListener in each async appender.Configuring Logback using XML is handled by Logback's Joran configuration system. This section is a short description of the high level data types supported by Joran. For more information, please refer to the official documentation.
Duration represents a laps of time.
It can be specified as an integer value representing a number of milliseconds, or a string such as "20 seconds", "3.5 minutes" or "5 hours" that will be automatically converted by logback's configuration system into Duration instances.
The recognized units of time are the millisecond, second, minute, hour and day. The unit name may be followed by an "s". Thus, "2000 millisecond" and "2000 milliseconds" are equivalent. In the absence of a time unit specification, milliseconds are assumed.
The following examples are therefore equivalent:
<duration>2000</duration>
<duration>2000 millisecond</duration>
<duration>2000 milliseconds</duration>
When specified, some properties accept a comma-separated list of values.
Leading and trailing whistespace characters are removed from each value during the decoding process, including tabs (\t), carriage return (\r) and line feeds (\n) - see the String.trim() function for more information. It is therefore safe to add an extra blank after the comma or write the values on multiple lines for better readability.
The examples below are equivalent and produce a list containing the values valueA, valueB, and valueC:
<property>valueA, valueB, valueC</property>
<property>
valueA,
valueB,
valueC
</property>
Also, multiple consecutive comma are treated as a single delimiter. This allows you to write a generic configuration as follows where the actual value comes from an external environment variable whose value may be empty.
<property>valueA, ${ENV_VAR}, valueC</property>
If needed, the comma delimiter may be escaped by prefixing it with a backslash (\) to treat it as being part of the value instead of considered as an actual delimiter. The example below defines a list containing one single element whose value is the string foo,bar:
<property>foo\,bar</property>
Memory usage and performance of logstash-logback-encoder have been improved by addressing issues discovered with the help of the YourKit Java Profiler.
YourKit, LLC has graciously donated a free license of the YourKit Java Profiler to this open source project.
FAQs
Provides logback encoders, layouts, and appenders to log in JSON and other formats supported by Jackson
We found that net.logstash.logback:logstash-logback-encoder demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.