axum-observability

axum-observability

Focused Axum middleware for request IDs, W3C trace correlation, request-scoped structured events, and one terminal access record per response.

Why this package exists

Managed platforms such as Cloud Run already collect container output. Applications should only need to write structured JSON to standard output (stdout); the platform can handle ingestion and delivery.

Compared with sending logs through an in-process cloud logging client, this reduces container CPU, memory, and network use by removing logging API calls, authentication, buffering, batching, and retry work from the application. It also avoids the dependency and maintenance cost of a cloud logging SDK, including its credentials, configuration, and upgrades.

This crate turns that simple pipeline into useful production observability. It provides validated request IDs, strict W3C trace correlation, request-scoped fields, and one structured terminal access record. Application and access logs can share the same correlation metadata, making records from a request easier to find and understand.

Field conventions map the same contract to provider-oriented fields without coupling application code to a cloud SDK. The crate focuses on structured logging and request correlation: it does not create or export distributed tracing spans, configure OpenTelemetry, export metrics, or ship logs. It uses an internal request span only to attach validated correlation fields to application events.

Why newline-delimited JSON

JsonLayer emits newline-delimited JSON (NDJSON, also called JSON Lines): each application or access event is one compact, self-contained JSON object followed by one LF (\n). The output is a stream of objects, never a JSON array.

NDJSON is deliberate for production logging:

• Agents such as Vector, Fluent Bit, and Datadog can parse entries as a stream with bounded memory instead of waiting for a closing array bracket.
• Append-only output needs no array brackets, commas, whole-file rewrites, or trailing-comma coordination. Each event is submitted as one complete encoded line. JsonLayer serializes writer creation and all partial writes for one record, so concurrent events through the same layer cannot interleave.
• A crash or interrupted final write can damage the incomplete last line, while previously completed lines remain independently parseable.
• Analytics systems can split large inputs on newline boundaries and process independent records in parallel.
• Standard tools work directly on the stream, for example head -n 20 app.log | jq -r '.message'.

Standard JSON arrays are suited to complete documents; NDJSON retains JSON's structured fields while providing framing designed for continuous log streams.

Package scope

ObservabilityLayer is one Tower layer for Axum 0.8. The application keeps control of its tracing subscriber, writer, filter, panic recovery, listener, and deployment policy. JsonLayer is composable and never installs global state itself.

This is an independently maintained crate, not official Axum middleware.

Requirements and installation

The minimum supported Rust version is 1.97.0. Version 2.0.0 supports the Axum 0.8 release line, including Axum 0.8.0 with only its matched-path feature.

[dependencies]
axum = "0.8"
axum-observability = "2.0.0"
tracing = "0.1.44"
tracing-subscriber = { version = "0.3.23", features = ["env-filter"] }

Version 2.0.0 is the clean contract described by this checkout. Version 1 remains available for applications that need its API, defaults, or structured fields; v2 provides no compatibility aliases for those surfaces.

Complete setup

When this documentation shows one configuration, it uses GCP. Complete provider-neutral, GCP, AWS, and Azure configuration examples are available in examples and EXAMPLES.md.

use axum::{Router, routing::get};
use axum_observability as obs;
use tracing_subscriber::prelude::*;

let config = obs::ObservabilityConfig::default().with_field_convention(obs::FieldConvention::Gcp);

tracing_subscriber::registry()
    .with(
        tracing_subscriber::EnvFilter::try_from_default_env()
            .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
    )
    .with(config.json_layer(std::io::stdout))
    .init();

let app = Router::new()
    .route("/health", get(|| async { "ok" }))
    .layer(obs::ObservabilityLayer::new(config));
# let _: Router = app;

JsonLayer writes one complete JSON object per line and holds a record-level lock across the writer's full write_all operation. Keep axum_observability::request=info enabled in RUST_LOG when application events need correlation from the request span. Terminal access records carry their own validated correlation fields, so a surviving WARN or ERROR access event stays complete even when the INFO request span is filtered.

Handlers can extract the validated context directly:

use axum_observability::RequestContext;

async fn handler(context: RequestContext) {
    tracing::info!(item_count = 3_u64, "loaded items");
    assert!(!context.request_id().is_empty());
}

Event fields cannot overwrite package-owned request correlation values. Without ObservabilityLayer, extraction rejects with the public MissingRequestContext error, status 500, and the fixed body request context unavailable. This makes middleware misconfiguration diagnosable without reflecting request data.

Middleware placement

Observability must wrap response-producing middleware so it sees the final status and body. Axum applies the last layer first, so add it last:

use std::time::Duration;

use axum::{Router, http::StatusCode};
use axum_observability::{ObservabilityConfig, ObservabilityLayer};
use tower_http::{
    catch_panic::CatchPanicLayer,
    cors::CorsLayer,
    limit::RequestBodyLimitLayer,
    timeout::TimeoutLayer,
};

let app = Router::new()
    // Add routes and application middleware first.
    .layer(RequestBodyLimitLayer::new(1024 * 1024))
    .layer(CorsLayer::new())
    .layer(TimeoutLayer::with_status_code(
        StatusCode::REQUEST_TIMEOUT,
        Duration::from_secs(30),
    ))
    .layer(CatchPanicLayer::new())
    .layer(ObservabilityLayer::new(ObservabilityConfig::default()));
# let _: Router = app;

Integration tests cover this order for recovered panics and timeouts. The crate does not recover panics itself. If a panic must become a final 500 response, install recovery middleware inside ObservabilityLayer as shown.

Request and trace context

The default header is X-Request-ID. Under the default policy, exactly one incoming value is accepted when it contains 1-128 ASCII URI-unreserved characters: A-Z, a-z, 0-9, -, ., _, and ~. Missing, empty, duplicate, oversized, non-ASCII, or otherwise invalid values are replaced. The fallback is 128 random bits encoded as 32 lowercase hexadecimal characters.

Application configuration and tests can validate the baseline grammar through RequestId::parse, FromStr, or TryFrom; invalid values return the public, non-sensitive InvalidRequestId error. RequestId has no unchecked public constructor. A RequestId extracted from middleware represents the value selected by the configured policy, so a custom validator can admit a typed value that does not pass the baseline parser.

The selected value is available from:

• the validated RequestId in the RequestContext extractor and request extension;
• exactly one canonical configured header before downstream service code runs;
• request_id() and correlation_id();
• application events inside an enabled package request span;
• the terminal access record; and
• the configured response header, unless disabled.

Configuration can change the request/response header name, disable the response header, broaden or narrow caller-input validation within Axum's native text header boundary, or supply a fallible generator. A custom validator can admit broader RFC 9110 field content such as id:42, internal space or tab, and values longer than 128 bytes. Edge whitespace, controls, non-text bytes, and values Axum cannot re-emit exactly are rejected before the callback. Generated values use RequestId and retain the baseline grammar. The custom validator is never applied to generated values. A generator is invoked once before the package-owned fallback is used, and callback failure never produces an invalid ID or alters traffic.

traceparent parsing rejects duplicates, uppercase hexadecimal, zero trace or parent IDs, invalid framing and flags, and unsafe native field content. Version 00 must use its exact framing. Versions 01 through fe validate the known prefix and treat a delimiter plus any remaining extension bytes as opaque, including a trailing delimiter, without a package-invented length ceiling. W3C Trace Context Level 1 is the default; select TraceContextLevel::Level2 explicitly when Level 2 key grammar and version-00 random trace-ID flag projection are required. Higher versions preserve sampling without assigning meaning to the random bit. Repeated tracestate values are combined in wire order and accepted only when their selected-level grammar, unique-key rule, and 32-member limit pass. The crate can propagate at least 512 characters and admits a valid 513-character value; 512 is not a package rejection ceiling. Empty members are retained and count toward the member limit. An invalid tracestate is discarded without invalidating a valid traceparent.

With valid trace context, correlation_id is the trace ID; otherwise it is the request ID. The incoming parent ID belongs to the caller. The crate does not claim it as a span created by this service, manufacture a current span ID, or mutate outbound trace headers.

Structured log contract

Every JSON event produced by JsonLayer contains timestamp, target, and level (severity on GCP). GCP maps TRACE and DEBUG to DEBUG, and WARN to Cloud Logging's canonical WARNING; INFO and ERROR are unchanged. Events that record a message keep it under message. Typed tracing fields remain JSON numbers, booleans, and strings. Application errors recorded through tracing::field::Visit::record_error use their display text.

During a request, events inside the enabled package span also contain request_id and correlation_id. Valid W3C context adds trace_id, parent_id, trace_flags, and trace_sampled. Configured Level 2 additionally adds trace_id_random for version 00; Level 1 and higher versions omit it.

The terminal record always has message = "request completed". Its common semantic fields are:

Field	JSON type	Presence and meaning
request_id	string	Always; the validated or generated request ID
correlation_id	string	Always; trace ID when valid W3C context exists, otherwise request ID
trace_id, parent_id	string	Only with valid W3C context
trace_flags	string	Only with valid W3C context; exactly two lowercase hexadecimal characters
trace_sampled	boolean	Only with valid W3C context
trace_id_random	boolean	Only with valid version-00 W3C context in configured Level 2 mode
method	string	Always; HTTP method
path_template	string	When Axum's MatchedPath is available
path	string	Only with with_raw_path(true); exact nonempty query-free http::Uri path, including *
operation_id	string	When an OperationId request or response extension exists
status	number	When a response status is known
duration_ms	number	Always; non-negative handling and streaming time
peer_ip	string	Only with the peer-ip feature, with_peer_ip(true), and ConnectInfo
user_agent	string	Only with with_user_agent(true) and one UTF-8 RFC 9110 field-content value
terminal_reason	string	Only for body_error, service_error, or response_dropped

Optional values are omitted; the formatter does not emit null placeholders. Normal completion omits terminal_reason; abnormal records do not invent an error summary when the original failure is unavailable. The default level is ERROR for every abnormal terminal reason or a normal 5xx, WARN for a normal 4xx, and INFO otherwise. The status-level mapper applies only to normal completion. Application events cannot replace package correlation, envelope, provider aliases owned by the active field convention, or internal-control fields. Access-only names, aliases owned only by an inactive field convention, other unowned provider fields, and names that merely use an obs. or _obs_ prefix remain ordinary application data. Access enrichment cannot replace exact terminal access fields; package-owned fields win without reserving speculative namespaces. Application error fields remain native application data and are not synthesized on access lines.

path_template is the default low-cardinality aggregation key. Concrete path can have unbounded cardinality and may contain identifying data, so it is off by default.

Axum 0.8 MatchedPath already uses the portable whole-segment {name} and terminal {*name} forms. Different concrete parameter or catch-all values therefore retain one template, and unmatched routes omit route and operation identity rather than substituting the request path.

Operation IDs

An outer Tower layer cannot inspect request extensions inserted after the request has been consumed by route middleware. Route-specific operation IDs should therefore use Axum's native response-extension path:

use axum::{Extension, http::StatusCode};
use axum_observability::OperationId;

async fn list_items() -> (Extension<OperationId>, StatusCode) {
    (Extension(OperationId::from_static("list-items")), StatusCode::OK)
}

An OperationId already present before the request reaches observability is also supported. A response extension takes precedence because it is closest to the selected route.

Field conventions

Select one convention on the shared ObservabilityConfig, finalize that value, then construct json_layer and the terminal middleware from the same unchanged configuration. json_layer snapshots the convention at construction time; later builder calls create a different configuration and do not update an existing layer.

• Generic is the provider-neutral default and uses level.
• Gcp replaces level with severity, adds logging.googleapis.com/trace, logging.googleapis.com/trace_sampled, and a structured httpRequest access object. httpRequest maps enabled path, peer_ip, and user_agent to requestUrl, remoteIp, and userAgent; method, status, and latency use requestMethod, numeric status, and a seconds string. The trace field is the bare validated 32-character W3C trace ID. The crate never emits a fake logging.googleapis.com/spanId.
• Aws adds xray_trace_id in 1-8hex-24hex form. It does not create an X-Ray segment or parse X-Amzn-Trace-Id.
• Azure adds operation_Id and operation_ParentId. It does not initialize an Azure SDK or parse legacy Request-Id headers.

Provider trace fields are omitted without valid W3C context and never change which request metadata is captured. They correlate logs; trace creation, sampling policy, and export remain application concerns. Google Cloud's current preferred trace field format is the bare trace ID.

Diagnostics and failure boundaries

The body wrapper owns a one-shot terminal guard:

• an already-ended body completes before its EOF state is exposed;
• a streaming body completes when it returns EOF;
• a body error emits one body_error record and passes the original body error to the consumer;
• an inner service error emits one service_error record and returns the original service error;
• dropping an unfinished response or service future emits one response_dropped record at ERROR; and
• once the guard completes, later polling or drop cannot emit a duplicate.

Status and duration reflect the latest trustworthy state. If the response was never produced, status is omitted. The monotonic clock is saturating, so a bad custom clock cannot produce a negative duration.

Custom generator, validator, level-mapper, access-enricher, and clock panics are contained with safe fallback behavior. An initial clock failure uses the package monotonic clock; a finish-time failure falls back to the request start. This containment requires Rust's default panic = "unwind"; Rust code cannot recover from panic = "abort". Formatter serialization and writer errors do not replace the HTTP response. Writer failures can still mean a log record was not delivered; applications remain responsible for choosing and monitoring the output destination.

Configuration

ObservabilityConfig is builder-based:

Method	Default	Purpose
with_field_convention	FieldConvention::Generic	Select one provider field convention
with_trace_context_level	TraceContextLevel::Level1	Select W3C Trace Context Level 1 or Level 2
with_request_id_header	x-request-id	Set the request and response correlation header
with_response_header	true	Enable or disable response-header injection
with_raw_path	false	Opt into query-free concrete path capture
with_peer_ip	false	With the peer-ip feature, opt into trusted socket-peer capture
with_user_agent	false	Opt into one unambiguous text User-Agent value
with_request_id_generator	random 128-bit ID	Supply a fallible typed generator, invoked once per replacement
with_request_id_validator	accepts baseline	Broaden or narrow caller IDs within Axum's native text-header boundary
with_status_level_mapper	5xx/4xx/other mapping	Map final status to a tracing::Level
with_clock	Instant::now	Supply a monotonic clock, primarily for deterministic tests
with_access_enricher	no extra fields	Add synchronous application-owned terminal fields

trace_context_level() exposes the resolved W3C grammar level.

Unknown options do not exist: configuration is compile-time checked. The header setter accepts a validated http::HeaderName; use HeaderName::from_static or HeaderName::try_from at the configuration boundary:

use axum::http::HeaderName;
use axum_observability::ObservabilityConfig;

let config = ObservabilityConfig::default()
    .with_request_id_header(HeaderName::from_static("x-correlation-id"));
# let _: ObservabilityConfig = config;

Enrichment values must be safe to log; the crate does not redact application-owned fields.

Proxy trust and privacy

peer_ip comes only from Axum ConnectInfo<SocketAddr> when the peer-ip feature and runtime opt-in are both enabled. The crate never parses Forwarded or X-Forwarded-For, because trusting caller-controlled forwarding headers without a known proxy boundary permits spoofing. Configure trusted proxy handling before constructing ConnectInfo if the original client address is required.

Raw paths, peer IPs, and User-Agent values are independently off by default; enabling any of them changes the application's privacy posture. The terminal schema never logs query strings, request or response bodies, cookies, authorization values, arbitrary headers, or forwarded-IP headers. The GCP request URL uses the same query-free concrete path.

There is no automatic redaction of application tracing fields or access enrichment. Applications remain responsible for keeping credentials, personal data, and secrets out of those values.

Troubleshooting

Symptom	Cause	Correction
No access record	Its selected level is filtered, or the writer failed	Enable the axum_observability::access level and verify the writer
WARN/ERROR access record lacks application span fields	INFO request span is filtered	Terminal correlation remains complete; enable axum_observability::request=info for correlated application events
Timeout or recovered panic has the wrong status	Observability is inside response-producing middleware	Add ObservabilityLayer last so it wraps recovery and timeout layers
operation_id is absent	A route middleware inserted it only on the consumed request	Return Extension(OperationId) on the response
peer_ip is absent	The feature or runtime opt-in is disabled, or no ConnectInfo<SocketAddr> exists	Enable peer-ip, call with_peer_ip(true), and provide a trusted peer extension
Caller request ID is replaced	It is missing, duplicate, invalid, or rejected by custom policy	Send one baseline value, or align the caller value with the configured validator
GCP trace link is absent	traceparent is missing or invalid	Send one valid lowercase W3C traceparent; do not provide a project-qualified value
Duplicate framework access lines	Another access logger remains enabled	Disable the competing access logger when this crate owns terminal records

Compatibility

The crate supports Rust 1.97.0 or newer and the Axum 0.8 release line. The public ObservabilityService is the nameable Tower service produced by ObservabilityLayer. Within each major release line, exported APIs, configuration defaults, structured fields, and supported runtime versions are Semantic Versioning contracts. Breaking changes require a new major version, explicit changelog coverage, and migration guidance.

Version 2 intentionally removes v1 compatibility shims and changes the request-ID validator, trace-context, and formatter-construction APIs. Read the migration guidance before upgrading. Build both the middleware and JSON formatter from one ObservabilityConfig.

Development and validation

Development uses just. On Homebrew, install the system prerequisites and repository tools before running validation:

brew install just rust llvm actionlint zizmor
just install

The Homebrew Rust and LLVM versions must match. The coverage recipes detect an active Homebrew Rust compiler and select Homebrew's llvm-cov and llvm-profdata automatically.

just qa

just qa runs formatting, Clippy with warnings denied, tests, doctests, dependency policy, the RustSec audit, actionlint, and zizmor. Maintainers should follow the public release architecture and guide.

Property and mutation testing

Stable property tests generate valid W3C trace context and exercise equivalent multi-header tracestate layouts as part of the normal test suite. Mutation testing remains an explicit maintainer campaign:

just mutation

Mutation testing runs outside just qa. Add a behavioral test when a surviving mutant exposes a real contract gap. Equivalent transformations do not need artificial assertions.

Consumer image

Run just e2e-image observability-e2e-local:manual to build a production-shaped consumer image from the exact checkout. The recipe prefers Podman and falls back to Docker.

Building the image verifies packaging and integration only. It does not run the image, validate emitted logs, compare implementations, or approve a release. Optional independent tooling may exercise the package's documented public contract. Any audit result is informational and is never a publication requirement.

References

• Axum middleware documents middleware placement and ordering.
• Extension and IntoResponseParts document request and response extensions.
• http-body::Body defines frame polling, EOF, and size-hint behavior.
• tracing-subscriber::Layer and EnvFilter define formatter composition and filtering.
• W3C Trace Context Level 1 Recommendation defines the default traceparent and tracestate contract.
• W3C Trace Context Level 2 Candidate Recommendation Draft defines the explicit Level 2 key grammar and random trace-ID flag.
• Google Cloud trace and log integration documents the bare trace ID as the preferred trace field format.
• Google Cloud Trace release notes record when the bare trace ID became the preferred form while the full project resource name remained supported.
• Google Cloud structured logging documents severity, message, httpRequest, and special trace fields.
• AWS X-Ray trace IDs documents conversion from W3C to 1-8hex-24hex form.
• Azure Application Insights data model defines operation_Id as the root-operation identifier and operation_ParentId as the immediate-parent identifier.

License

MIT