Faizo

Writing · Operations · Caddy · HTTP

Three ways a reverse proxy invents a 502

Caddy logged 502s for requests my backend had answered. Unread request bodies, container restarts and HTTP/3 clients hanging up, and how to fix each.

My production stack is simple: Caddy terminates TLS and reverse-proxies to a handful of Rust services in containers. Any error-level log line is forwarded to a Discord channel on my phone, so a 502 is an alert. Over a few months I was paged for 502s that were not failures of my backend at all. Caddy had produced each one itself, for three different reasons. Each is easy to fix once you can recognise it.

1. The backend answered without reading the request body

Symptom: an occasional 502 on a path you don't even serve, usually a POST from a vulnerability scanner (POST /lib/vendor/phpunit/… was mine). Caddy's error log mentions write: broken pipe.

What happens: the Rust web framework I use, axum, answers an unknown route with 404 straight away, without reading the request body. The HTTP server can't reuse a connection whose body wasn't consumed, so it closes it. Caddy is still busy streaming the body upstream, gets a broken pipe, throws away the perfectly good 404 and reports a 502 of its own, at error level.

Fix: make the fallback handler read (and discard) the body before answering. The connection stays healthy, the client gets the honest 404, and nothing is logged anywhere:

async fn not_found(body: axum::body::Bytes) -> StatusCode {
    // Reading the body (bounded by the router's body-size limit) keeps the
    // connection reusable, so the proxy delivers this 404 instead of a 502.
    drop(body);
    StatusCode::NOT_FOUND
}

let app = Router::new()
    .route("/api/…", post(handler))
    .fallback(not_found)          // register before the layers that should wrap it
    .layer(DefaultBodyLimit::max(256 * 1024))
    .layer(TraceLayer::new_for_http());

Two details. Buffering into Bytes is bounded by the body-size limit, so a huge body is rejected rather than read forever. And in axum, Router::layer only wraps what was added above it, so the fallback must be registered before the layers you want it to pass through.

2. The upstream was restarting

Symptom: a burst of 502s at deploy time, with dial tcp …: connect: connection refused in Caddy's log.

What happens: recreating a container leaves a gap of a second or so where its address refuses connections. Caddy dials it, gets refused, and answers 502. I measured it: during one forced recreate of a service, 6 of 200 probe requests failed. For a Discord app that's worse than an alert. An interaction that lands in that gap shows the member "This interaction failed".

Fix: let Caddy retry dials for a short window. A snippet keeps it consistent across every upstream:

(upstream_retry) {
	lb_try_duration 2s
	lb_try_interval 200ms
}

api.example.com {
	reverse_proxy app:8080 {
		import upstream_retry
	}
}

After the change, the same test gave 0 of 200 failures and no error lines at all, since a retry that succeeds logs nothing. This is safe because of what is being retried: a dial failure means no connection was made, so the upstream never saw the request and can't have processed it. An upstream that answers 500 still passes straight through. And a service that is really down still produces a 502, just about two seconds later, so a real outage still pages. I keep the window well under three seconds because Discord interactions must be answered within three.

One caveat: this only covers hops Caddy makes. My API server also calls an internal dispatcher service directly over the container network, so that client has its own short re-dial loop for refused connections, with the same "dial failures only" rule.

3. An HTTP/3 client hung up

Symptom: 502s on ordinary GET requests (/api/capabilities, /auth/me) over HTTP/3.0, mostly from Firefox and iOS Safari, each taking almost exactly two seconds, with an error like:

reading: Application error 0x100 (remote)

What happens: 0x100 is H3_NO_ERROR: the browser closed the stream because it no longer needed the response, typically a navigation or a closed tab. Caddy only recognises a client going away when Go reports a plain context cancellation. The QUIC library it uses cancels the request context with the QUIC close as the cause, which is neither, so Caddy treats it as an upstream failure. Since it's a GET, Caddy's default retry policy kicks in: it retries every 200 ms against a client that is gone until the two-second try window runs out, then logs a 502 at error level. The two-second duration in the log is the retry window, not backend latency.

Fix: this one I handle in the alerting, not the proxy. Disabling GET retries would undo fix number two. The alert forwarder drops these lines when all of the following hold:

  • the error is an HTTP/3 application close marked (remote), meaning the client closed it. A (local) close is my side aborting, which still pages;
  • the request's duration is short (I use 15 seconds as the cut-off). A client that gave up after waiting a long time for an answer my backend never sent is a real problem, and it still pages.

Chromium can hang up with a different spelling. When its network stack drops all its connections at once, the log reads INTERNAL_ERROR (remote): 16:net error. I mute that exact string under the same duration rule. Any other transport-level error code from a browser could mean my QUIC stack actually broke the protocol, so it still pages.

A note on the alert forwarder itself

Muting by text is dangerous if you do it in the wrong order. My forwarder originally checked for the word "panicked" before it had parsed the log level. My crash-report endpoint is public and logs some fields verbatim at info level, so anyone could have triggered a "PANIC" page by sending an error message that contained the word. Now every content match runs after the level is parsed, and info, debug and trace lines can never page. A genuine Rust panic is printed to stderr without any log prefix, so it still matches.

The forwarder also has an offline test mode that runs the classifier over a file of real production lines, and I run it on the server before restarting the service. Log formats drift. In one case a change to the span prefix meant crash reports silently stopped paging for four weeks, and nobody noticed because silence looks like health.

Checklist

  • Drain request bodies in your fallback handler.
  • Retry upstream dials, briefly, at the proxy. Nothing else.
  • Learn your proxy's spelling of "the client went away" for each protocol you serve, and filter it with a duration bound.
  • Parse the log level before matching any content.
  • Test the alert classifier against real lines whenever you change it.