Writing · Operations · DNS · Rust
A dropped DNS packet that looked like a Discord outage
A 5-second "connect timed out" to discord.com was a DNS stall inside Docker. How one lost UDP packet stalls lookups, and how serving stale answers fixes it.
One evening my phone showed an error from DWEEB's API server:
GET /api/guilds/…/webhooks error=could not reach Discord: client error (Connect): operation timed out latency=5001 ms
Discord was fine. Its status page was green, other requests a second earlier and later had succeeded, and nothing in the request was unusual. The number that gave it away was 5001 ms: almost exactly my HTTP client's five-second connect timeout. Something had eaten the entire connect budget before a TCP connection was even attempted. That something was DNS.
How one lost packet becomes a five-second stall
The server runs in Docker on a small VPS. A lookup for discord.com from inside a container passes through four resolvers before anything leaves the machine, and none of the first three does what you'd hope:
| Layer | Caches? | What it does on a lost packet |
|---|---|---|
| glibc in the container | No | Waits, then retries (default timeout 5 s) |
| Docker's embedded DNS | No | Forwards to the host; abandons the forwarded query after 4 s |
| systemd-resolved on the host | Only within the record's TTL | On the version I run, re-sends an unanswered UDP query only after a fixed 5 s |
| The provider's two resolvers | Yes | Answer in milliseconds, when the packet arrives |
So when one UDP datagram is dropped somewhere between the host and the provider's resolver, systemd-resolved won't ask again for five seconds, but Docker gives up at four. The lookup fails or stalls past four seconds, and the application's five-second connect timeout, which covers DNS and TCP and TLS, is spent by the lookup alone. The error says "connect timed out", which points you at the network or at Discord, when the real fault is a resolver on your own host.
Once I knew what to look for, the evidence was in the Docker daemon's journal a second before the alert:
[resolver] failed to query external DNS server … 127.0.0.53:53: i/o timeout question=discord.com
Counting those lines showed 61 stalls in eight days, seven of them for discord.com. Healthy lookups took 4 to 29 ms. Stalls were all four seconds or more. Nothing in between.
Fix one: make the host resolver less fragile
The first fix belongs to the host: give systemd-resolved additional upstream servers so a single lost packet to one of them isn't a five-second wait. That reduced the rate. It can't make it zero, and it does nothing for the next machine I deploy to. The application needed to survive this on its own.
I also considered the two quick application fixes and rejected both. Raising the connect timeout past the stall turns every stall into a slow request instead of a failed one, and in this incident the stall outlasted glibc's own retry too. Reclassifying connect failures as "Discord's fault" would silence a real problem: if my host genuinely can't resolve or reach Discord, I want to know.
Fix two: serve stale
DNS already has an answer for this situation. RFC 8767, "Serving Stale Data to Improve DNS Resiliency", describes resolvers answering from expired cache entries when the authoritative servers can't be reached. I applied the same idea inside the application, as the HTTP client's custom resolver:
- Every connection still starts a fresh lookup, and a fresh answer always wins.
- If no answer arrives within 1 second, or the lookup fails outright, connect to the address from the last successful lookup for that host, provided it is at most 24 hours old. Log that at info level.
- Let the slow lookup keep running in the background, so a late answer still refreshes the remembered address.
- If nothing is remembered (the process just started), keep waiting, but give up at 4 seconds with an error that says what happened.
A simplified version of the resolver, using reqwest's Resolve trait and Tokio:
const FRESH_WAIT: Duration = Duration::from_secs(1);
const DEADLINE: Duration = Duration::from_secs(4);
const MAX_STALE: Duration = Duration::from_secs(24 * 60 * 60);
const CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
// The ordering is the design. Check it at compile time.
const _: () = assert!(FRESH_WAIT.as_millis() < DEADLINE.as_millis());
const _: () = assert!(DEADLINE.as_millis() < CONNECT_TIMEOUT.as_millis());
async fn resolve(&self, host: String) -> Result<Vec<SocketAddr>, BoxError> {
// Spawned, so a slow lookup outlives this call and still updates memory.
let lookup = tokio::spawn(self.clone().lookup_and_remember(host.clone()));
let mut lookup = std::pin::pin!(lookup);
if let Ok(done) = tokio::time::timeout(FRESH_WAIT, &mut lookup).await {
match done? {
Ok(addrs) => return Ok(addrs),
Err(err) => return self.remembered(&host).ok_or(err),
}
}
if let Some(addrs) = self.remembered(&host) {
tracing::info!(target: "dns", %host, "no answer in 1s; dialing last good address");
return Ok(addrs);
}
match tokio::time::timeout(DEADLINE - FRESH_WAIT, lookup).await {
Ok(done) => done?,
Err(_) => Err(format!(
"no answer for {host} within 4s, and no earlier answer to fall back on"
).into()),
}
}
(remembered returns the stored addresses only when they are younger than MAX_STALE.)
Why connecting to an old address is safe
The obvious worry with a stale address is connecting to the wrong machine. HTTPS takes care of that: TLS verifies the server's certificate against the hostname on every connection, whatever IP address the socket happens to reach. If Discord had moved discord.com to a new address and the old one now belonged to someone else, the handshake would fail, and the request would fail with a certificate error, which is the same outcome as the stall but faster. A stale address can fail to connect. It can't connect you to an impostor.
The 24-hour cap is a sanity limit rather than a security one. Big API providers keep their front-end addresses stable for much longer than a day, but if a host has been unreachable through DNS for a whole day, a fresh failure is more useful than a stale success.
What changed
- A DNS stall for a host the process has talked to before now costs about one second instead of failing the request, and leaves an info line I can grep for.
- A stall on a cold process still fails, still pages (it really is my host's problem), but at four seconds and with a message that names DNS, instead of a generic timeout that sent me looking at Discord.
- The timeouts are documented by an assertion, not a comment: wait for fresh answer, then fall back, then give up, all inside the connect timeout.
I only put this resolver on the client that talks to Discord. Other clients in the system either have long enough budgets to ride out a stall late, or are short-lived plugin clients where the right fix is the host resolver. If you run anything latency-sensitive in Docker on a single host, it's worth grepping your daemon journal for failed to query external DNS
before you blame the API on the other end.
