Writing · Discord API · Rust · Rate limits
A short page is not the last page: counting a Discord bot's servers
My bot's server count sat at 199 for weeks, then jumped to 344. Paginating GET /users/@me/guilds correctly, and a per-token rate limit I raced myself.
DWEEB's backend asks Discord which servers its bot is in. It needs that to show whether the bot is present in a server before offering features that require it, and it publishes the total to the app's listing page on Top.gg. For weeks the number sat at 198 or 199. Then, on a day with no deploy, it jumped from 199 to 344 in a single refresh.
The number had never been 199. My pagination had been wrong the whole time.
The endpoint
GET /users/@me/guilds returns the servers the current user (or bot) is in, at most 200 per request, ordered by ID. You page forward with after set to the last ID you received. The documentation notes that 200 is the most servers a regular user can join, so for user tokens one request is enough. Bots have no such limit, so for a bot you have to page.
The bug: stopping on a short page
My loop stopped when a page came back with fewer than 200 entries, which is the usual way to detect the end of a list:
// Wrong: a page shorter than the limit was taken to mean "no more pages".
loop {
let page = get_guilds(after, 200).await?;
ids.extend(page.iter().map(|g| g.id));
if page.len() < 200 { break; }
after = page.last().map(|g| g.id);
}
The endpoint doesn't guarantee that. It can return fewer than the limit while there are more results after the page. That would explain both the plateau and the sudden jump: the first page kept coming back one or two short of 200, and the loop never asked for the second page until one day a full page came back. During those weeks, about 145 servers that had the bot showed "bot not present" in DWEEB, and the public listing showed a number short by more than 40%.
The fix costs one extra request per walk: only an empty page ends the list.
loop {
let page = get_guilds(after, 200).await?;
let Some(last) = page.last() else { break }; // empty page: done
after = Some(last.id);
ids.extend(page.iter().map(|g| g.id));
}
I've found other Discord endpoints that are tempting to paginate the same way. Unless the documentation explicitly promises that a short page is the last one, I page until empty.
If the read fails, publish nothing
The server count is pushed to Top.gg every 30 minutes, because Top.gg shows whatever number a bot last posted. Two rules keep that honest:
- A failed read is skipped, never posted as zero. An earlier helper folded errors into an empty set, which is fine for "is the bot in this one server?" but would publish "0 servers" to a public page if used for the count. A stale count lags. A zero is visibly wrong and drops the listing to the bottom of every sort.
- The reporter never logs at error level. A Top.gg outage isn't worth waking anyone. Failures are logged at info and retried on the next tick. Only a rejected token earns a warning.
The same endpoint, the user side: a rate-limit race
The web app also calls this endpoint with each user's OAuth token, to list the servers they can manage. My logs showed about 1,100 "Discord rate-limited" warnings in 29 days, each followed by a wait of about one second, always on routes that load when a page opens.
The cause was in my own code. Two different authorization checks each needed the user's server list: one for "servers you can manage" and one for "servers you're a member of". They are two views of the same response from Discord, but each fetched and cached it separately. Opening the server library triggered both at once, both called Discord with the same token, and the second one got a 429 and slept for the Retry-After time. Discord holds this route to roughly one call per second per token, so a page load was racing itself.
The fix was to make the server list one fetch, one cache entry, shared by every view that derives from it, with concurrent callers waiting for the same in-flight request. The login callback also warms that cache, so the first page load after signing in doesn't fetch at all. One caveat about my own evidence: I worked this out from the code and the shape of the logs, not from Discord's rate-limit headers, because I wasn't logging those yet. That's the next section.
Logging rate limits without leaking secrets
Discord's responses include rate-limit headers that tell you what you hit: X-RateLimit-Bucket, X-RateLimit-Scope (user, global or shared) and, on a 429, how long to wait. Logging them makes the next investigation take minutes instead of a day. The catch is that the obvious companion field, the request path, can contain a secret: a webhook's execute URL is /webhooks/{id}/{token}.
So the warning logs a route shape built from an allow-list of path patterns, never the raw path:
fn route_shape(path: &str) -> &'static str {
// Only known shapes are named. Anything else is logged as "other",
// so a token-bearing path can never reach the logs.
match path.split('/').filter(|s| !s.is_empty()).collect::<Vec<_>>().as_slice() {
["users", "@me", "guilds"] => "/users/@me/guilds",
["guilds", _, "channels"] => "/guilds/{id}/channels",
["guilds", _, "roles"] => "/guilds/{id}/roles",
["webhooks", ..] => "/webhooks/{id}/{token}",
_ => "other",
}
}
The same caution applies to error messages. Some HTTP clients append the request URL to their error text (reqwest's Display adds for url (…)). One of my scheduled-post workers stored that text as the failure reason shown in the UI, which put a webhook token in plain text into a field the rest of the server could read. Error messages that leave the process are now built from the error's cause chain with the URL removed, and stored reasons are scrubbed on startup.
Summary
- Page
GET /users/@me/guildswithafteruntil a page is empty, not until a page is short. - Never publish a count from a failed read.
- Fetch a user's server list once and derive every view from it. Don't let two checks race for the same per-token budget.
- Log Discord's rate-limit bucket and scope, with the route reduced to an allow-listed shape.
- Strip URLs from error text before it is logged or stored.
