Writing · Discord API · Rust · Reliability
Nested deadlines: answering Discord interactions inside three seconds
Routing Discord button clicks through a dispatcher to plugin services inside the 3-second window: failure classes, safe retries and nested timeouts.
Every button click, select menu pick and modal submit in a Discord app arrives as an interaction, and Discord gives your app three seconds to send the first response. Miss it and the person who clicked sees This interaction failed
(or <app> didn't respond in time
), whatever your server did afterwards. Three seconds sounds generous until you put a few network hops between Discord and the code that actually does the work.
This is how the interaction path in DWEEB is built, the three mistakes I made in it, and the rules I ended up with. The code is Rust, but nothing here depends on the language.
The shape of the problem
DWEEB lets people put working buttons and menus into webhook messages: a ticket panel, a giveaway, a poll, a self-assignable role menu. Each of those features is its own small service (I call them plugins), and Discord only knows one URL for the whole application: the interactions endpoint. So a single public service, the dispatcher, receives every interaction, verifies Discord's signature, looks at the component's custom_id prefix (giveaway:, poll:, tickets: and so on) and forwards the request to the plugin that owns it over a private network.
That gives every click this path:
- Discord POSTs the interaction to the dispatcher.
- The dispatcher forwards it to a plugin.
- The plugin may call Discord's REST API (read a role, post a message, edit a channel) before it can decide what to answer.
- The plugin's answer travels back through the dispatcher to Discord.
Each arrow is a place time can go. The design question is who is allowed to spend how much of it, and what happens when they overrun.
Mistake one: an error message that says nothing
The first alert this path ever sent me read, in full, forward failed prefix="selfrole:" upstream=http://self-role:8092 err=error sending request for url (…)
. That sentence is what the Rust HTTP client reqwest prints for every transport failure. A refused connection, a DNS failure, our own deadline expiring and a connection dying half-way through all render identically. The real reason sits in the error's source() chain, which nothing was printing.
Those four cases want opposite handling, so the first fix was to stop treating them as one thing. Every failed forward is now classified:
enum ForwardFailure {
/// We never got a connection. Nothing reached the plugin.
Dial,
/// Connected, sent the request, no answer before the deadline.
Timeout,
/// The connection died after the request was written.
Broken,
}
fn classify(err: &reqwest::Error) -> ForwardFailure {
// Order matters: a *connect* timeout answers true to both
// is_connect() and is_timeout(), and it is a dial failure.
if err.is_connect() {
ForwardFailure::Dial
} else if err.is_timeout() {
ForwardFailure::Timeout
} else {
ForwardFailure::Broken
}
}
The log line then carries the flattened cause chain instead of the top-level message, so the same failure now reads client error (Connect): tcp connect error: Connection refused (os error 111)
. One detail worth copying: print the chain or the client's own message, not both. The top-level message only repeats the URL, and my alert forwarder truncates long lines from the end, which is exactly where the useful part sits.
What is safe to retry, and what never is
The dispatcher reaches plugins directly over the container network, so when a plugin container is being recreated during a deploy there is a short window where its address refuses connections. That used to fail the click. The obvious fix is a retry, and the obvious retry is wrong.
A retry is only safe when you can prove the first attempt had no effect. For a dial failure you can: no connection was made, so the plugin never saw the request. For a connection that broke after the request was written you can't. The plugin may have toggled the role, entered the member into the giveaway or recorded the vote, and sending the interaction again would do it twice. A self-assignable role that gets toggled twice is a role that silently didn't change.
So the rule is narrow: retry dial failures, never anything else. The HTTP library already retries the one other safe case on its own (a pooled keep-alive connection that turns out to be closed before anything was sent), so what reaches the Broken branch is precisely the set of failures that must not be repeated.
The retry also has to respect the overall budget. Each attempt gets only what is left of the time the hop started with, and the client has a short connect timeout so that one attempt against an address that silently drops packets can't eat the whole window:
let started = Instant::now();
let mut attempt_err = None;
loop {
let left = FORWARD_BUDGET.saturating_sub(started.elapsed());
if left.is_zero() {
break Err(attempt_err.expect("at least one attempt ran"));
}
match client.post(&upstream).timeout(left).body(body.clone()).send().await {
Ok(response) => break Ok(response),
Err(err) if matches!(classify(&err), ForwardFailure::Dial) => {
attempt_err = Some(err);
tokio::time::sleep(DIAL_RETRY_GAP.min(left)).await;
}
Err(err) => break Err(err), // Timeout or Broken: never resend
}
}
A dial that recovers inside the window logs at info level. A dial that never recovers means the plugin is genuinely unreachable, which is my infrastructure's fault, and that one pages me.
Mistake two: two deadlines with the same number
This is the one I'd most like other people to avoid, because it looks fine in review. The dispatcher gave each forward 2,500 ms. Every plugin gave its own Discord REST calls 2,500 ms too. Both numbers are reasonable on their own, and both sit comfortably inside Discord's three seconds.
The problem is that the inner clock starts later. The plugin's REST call begins some milliseconds after the dispatcher's timer started, so whenever Discord's API was slow enough to bring the inner call close to its limit, the outer timer fired first, every time. The dispatcher gave up and told the member the plugin didn't respond
, then the plugin's call succeeded a moment later. The member saw a failure for a click that had worked, and my alerting saw a plugin timeout, which pointed me at the wrong component.
The rule I took from it: an inner deadline must always be strictly smaller than the outer deadline waiting on it, with enough margin for the work around it. Then the failure is reported by the component that actually knows what went wrong. The plugins' Discord clients now use 2,200 ms, and the ordering is checked by the compiler rather than by a comment that will drift:
const FORWARD_BUDGET_MS: u64 = 2_500;
const PLUGIN_DISCORD_TIMEOUT_MS: u64 = 2_200;
const DISCORD_INTERACTION_WINDOW_MS: u64 = 3_000;
const _: () = assert!(PLUGIN_DISCORD_TIMEOUT_MS < FORWARD_BUDGET_MS);
const _: () = assert!(FORWARD_BUDGET_MS < DISCORD_INTERACTION_WINDOW_MS);
In my setup the plugins live in separate crates, so the assertion pins the plugin value as a constant in the dispatcher. That's still better than nothing: changing one number without the other now needs a conscious edit in two places.
Mistake three: one log level for every failure
With the failures classified, the next step was making the alerting follow fault rather than symptom:
| Failure | Whose fault | Level |
|---|---|---|
| Dial failed for the whole window | Mine: the plugin is down | Error, pages |
| Connection broke after the request was sent | Unknown and unexpected | Error, pages |
| Plugin accepted the request but ran out of time | Usually a slow Discord API behind it | Warning, does not page |
The timeout row deserves a note. A plugin that is merely slow because Discord was slow is not something I can act on at 4 a.m., and it heals itself. A plugin that is genuinely wedged is a different story, and that is caught by a separate health check that probes each plugin directly every two minutes. Health checks can tell slow from stuck. The forward path can't.
The member-facing copy follows the same split. For a timeout the message says to check whether the action went through before clicking again, because the plugin may well have completed it after the dispatcher stopped listening. Try again
would be exactly wrong for a toggle.
When to defer instead
Discord also lets you buy time: answer with a deferred response within three seconds, then follow up within fifteen minutes. It is tempting to defer everything. I don't, because a deferral changes what the person sees (a "thinking" state or a loading message) and adds a second request that can fail on its own.
The rule I use is simple. If the work is one or two fast reads that can run concurrently, answer directly: the Directory plugin, which renders a live list of a server's roles and channels, makes three concurrent read requests and never defers. If the work is a multi-step flow that writes to Discord several times (creating a ticket channel, setting its permissions, posting a welcome message, logging it), defer immediately and do the work in the background, rolling back cleanly if a step fails. A test asserts that every response the Directory plugin can produce is a final one, because a stray deferral there would show up as a spinner that never resolves, and nobody would notice it in code review.
The checklist I use now
- Classify transport failures before deciding anything. Check "couldn't connect" before "timed out", since a connect timeout is both.
- Log the error's cause chain, not the client's top-level message.
- Retry only failures that provably had no effect. For a forwarded interaction that means dial failures, nothing else.
- Give each retry the remaining budget, not a fresh one.
- Make every inner deadline strictly smaller than the outer one, and have the compiler check it.
- Page on faults that are yours. Watch the rest with health checks that can tell slow from dead.
- Word failure messages for the case where the action may have happened anyway.
None of this is specific to Discord. Any system where a caller gives up on a fixed clock and a callee keeps working has the same traps. Discord just makes the clock short enough that you hit them in the first month.
