rate-limiting-and-backpressure

Overview

When demand exceeds capacity, the only question is which work fails and how quickly it finds out. A system with no explicit answer still answers: it queues everything, latency grows without bound, and every caller times out on work that will be completed and then discarded. Choosing the victim deliberately is the entire discipline.

When to use

One caller, tenant, or job is degrading the service for everyone
Designing quotas, throttles, or the response a caller gets when over the limit
A queue whose depth grows through the day, or that drains long after the traffic ended
Timeouts, pool exhaustion, or a cascading failure spreading between services
Retry behavior amplifying a partial failure — error rate rising after a dependency starts recovering
Not for: capacity planning and scaling decisions; this covers what happens in the window where capacity is already insufficient
Not for: load that is the same expensive answer computed repeatedly — that is caching-strategy's problem, and the cheapest request is the one the origin never sees. Reach here once the work is already irreducible, or once a cache's own stampede is the load

Shed, queue, or slow down

Response	Trade	Correct when
Shed — reject immediately	Preserves latency for accepted work, discards the rest	A human is waiting, the work is not durable, or the result expires quickly
Queue — accept and defer	Preserves work, spends latency, and hides the overload until the queue is full	Work is durable and asynchronous, and the bound is set from an acceptable wait
Slow down — backpressure to the producer	Preserves both, requires a producer that can be slowed	The producer is under your control: an internal caller, a stream, a batch job, a replication feed

Rule: never accept work you cannot drain faster than it arrives under the worst arrival rate you have actually measured. Queueing is not a capacity increase; it is a loan against latency, repaid by whoever is still waiting when the queue finally drains.

Algorithms

Mechanism	Burst behavior	State	Failure mode
Token bucket	Allows a burst up to bucket size, then steady refill rate	Two numbers per key	Burst size chosen without thought lets one caller consume a full window instantly
Leaky bucket / GCRA	No burst; output perfectly smooth	Two numbers per key	Legitimate bursty clients are punished; queueing variant hides overload
Fixed window counter	Up to 2x the limit across a boundary	One counter per key per window	The boundary spike is real traffic and it will find you
Sliding window log	Exact	One timestamp per request	Memory grows with traffic — the limiter becomes the bottleneck
Sliding window counter	Approximate, smooth	Two counters per key	Slight inaccuracy at boundaries; the right default for most services
Concurrency limit / semaphore	Bounds simultaneous work, not arrival rate	One counter per resource	Slow callers hold slots; requires a timeout on hold time or it deadlocks under a slow dependency

Rate limits and concurrency limits solve different problems and most systems need both. A rate limit protects fairness between callers; only a concurrency limit protects a finite resource — a connection pool, a thread pool, memory — because the resource is consumed by requests in flight, not by requests per second. If exactly one control is affordable, bound concurrency: it degrades gracefully under a dependency slowdown, while a rate limit calibrated for a healthy dependency admits exactly as much work when everything is already stuck.

Choosing the dimension

Limit on the dimension that owns the cost: per API key, per tenant, per account, per endpoint class, or per resource. Getting this wrong punishes the wrong caller, and the caller who gets punished is usually the innocent one.

Per IP breaks on NAT, corporate egress, and mobile carriers — one limit shared by thousands of unrelated users — while an attacker rotates addresses freely. Use it only as a pre-authentication backstop.
Global only means the heaviest tenant sets everyone's experience, and the limit is invisible to the caller who triggered it.
Per endpoint with equal weights is wrong whenever endpoints differ in cost by more than about 10x. Charge a weight — a report costing 50 tokens and a lookup costing 1 — so the limit tracks work rather than request count.
Keep a global backstop below the sum of per-tenant limits, because that sum is not your capacity and never was. Every tenant simultaneously behaving within its limit is a normal Tuesday, not a hypothetical.
Scope limits to what the caller can control. A limit whose denominator includes work triggered by other users produces a caller who cannot comply no matter what they do.

A fixed shared capacity split across non-fungible types needs a floor per type

Everything above divides throughput over time among symmetric consumers — callers, tenants, keys — who are substitutable for each other and differ only in identity. A different failure shows up when a fixed, one-shot capacity (a prompt-assembly budget, a config size limit, a page of shared storage) is instead split across several non-fungible content types with no reservation per type: whichever type is least disciplined about its own size determines how much room is left for every other type, down to zero. "Biggest wins" is not even a defensible allocation policy here, because the types are not substitutes — a large rule set consuming the whole shared budget doesn't mean rules needed the room more than the tool definitions it evicted, only that nothing stopped it from taking that room.

Give each non-fungible type its own reserved slice instead of one undivided pool. An unbounded type should degrade only its own contribution — truncate on a clean boundary and report what got cut, by name — and never consume the room reserved for a type it has nothing to do with.

Backpressure

Backpressure means the constraint reaches the producer, not that the consumer buffers harder. Concretely: bounded queues that reject or block on insert, not accept; stopping reads from the socket rather than draining into memory; withholding acknowledgement of the next message until the current one is done; a semaphore held for the full life of the request, not just its CPU phase; flow-control windows on streams.

The unbounded queue is how load becomes latency instead of visible failure. It absorbs exactly the signal you needed, and it keeps absorbing until every item in it is already past the deadline of whoever submitted it — at which point the system spends 100% of its capacity producing results nobody is waiting for. Two defenses: propagate a deadline with every request, and discard work whose deadline has passed before starting it. Dropping dead work at the head of the queue is often the single change that recovers an overloaded system.

Sizing is arithmetic, not intuition. By Little's Law, wait time equals queue length divided by service rate, so pick the maximum acceptable wait and set the bound to service rate × that wait. A queue of 10,000 in front of a consumer serving 100 per second is a 100-second wait, which is a decision someone should have made on purpose.

Deadline-first starves whatever can wait

Serving whatever expires soonest is the natural way to drain a queue with deadlines, and it has one failure that no queue-length panel shows. When the urgent stream never stops — a steady arrival of short-deadline items, each holding a worker for a while — the pool is always busy with the next urgent item, and work whose deadline is far enough away is never picked at all. The patient class does not run slowly; it does not run. A batch that is joined in full before the next pick makes it worse, because every round is chosen by the same rule against the same stream.

The diagnostic is a count of items served per class over a window, not queue depth and not a fairness quota — a quota attached to a pick path the hot loop does not take reports fairness while nothing patient is served. When the two classes differ a lot in service time, give the patient class its own small bounded worker rather than a share of the urgent pool: a share of one pool is what the urgent stream takes, every round.

Start the clock when the work does

A per-item deadline is only meaningful if the item's clock starts when the item does. Most timeout primitives fix their deadline at construction, so building every wrapper up front — the natural shape of a fan-out — starts every clock simultaneously, including for items queued behind their own siblings on a concurrency-limited backend. Items several waves deep then spend their entire budget waiting for a slot and time out having done no work at all. Either construct the timeout inside the unit of work, downstream of whatever gate grants the item its slot, so the clock starts when the turn actually begins — or issue the work in bounded waves and create each wave's timeouts immediately before awaiting that wave. fanning-out-independent-work is where the wave shape gets decided; this is the constraint that decides how wide a wave may be.

The diagnostic: every item times out while a single item, run alone, finishes comfortably. That is queue time inside the deadline, not slow work. The failure is also quieter than it looks — a timeout that fired before any work began is indistinguishable downstream from work that ran and produced a neutral result, so an aggregate over partial results should record how many contributors actually ran. "No agreement" among workers that never started is not disagreement.

Bound the resource, not the batch

The resource decides the scope of its limit: a backend shared process-wide needs a process-wide semaphore; one shared across processes needs the bound somewhere every caller passes through — a shared pool, a proxy, the backend's own admission control. A bound scoped to the unit of work reads as correct in isolation, and it is: each batch keeps its in-flight work within the backend's capacity, and three independent schedulers running three well-behaved batches still oversubscribe the same backend threefold. Every caller is disciplined; the process is not. This is the concurrency side of the global-backstop rule above — the sum of local limits is not the capacity, and never was.

The diagnostic: the same saturation symptom returns after a local bound fixed it. The instinct is to suspect the fix regressed; instead, count the independent callers that each hold a full copy of the backend's capacity. The corollary is arithmetic, however wrong it feels: parallelism against an already-saturated backend buys no throughput — the backend was the bottleneck either way — and spends the callers' deadlines on waiting instead. Serializing callers is often strictly better, not a reluctant compromise.

Watch age, not just depth

Queue depth alone is ambiguous — 10,000 items is healthy at 5,000 per second and an outage at 5 per second. Alert on age of the oldest unstarted item, which is directly comparable to the latency objective. It is also a signal that has to be emitted deliberately: depth is what a queue reports about itself for free, and age is what instrumenting-for-observability calls the measurement you have to decide to take before the incident, not during it. Supporting indicators, all leading rather than lagging: in-use concurrency versus the limit, time spent waiting for a slot, rejections broken down by reason, and the ratio of arrival rate to service rate — sustained above 1.0, everything else is a countdown.

Retries

Exponential backoff with full jitter: delay = random(0, min(cap, base × 2^attempt)). Backoff without jitter re-synchronizes every client that failed together, so they all return at the same instant and rebuild the herd that caused the failure. Jitter is not a refinement of backoff; without it, backoff schedules the next outage.
Cap attempts and total elapsed time. Three attempts within a bounded budget is a reasonable default. The deadline matters more than the count: a retry issued after the caller has given up is pure load.
Retry budgets: permit retries only up to a fraction of successful traffic — 10% is a common ceiling — and fail fast beyond it. This is the control that stops a partial outage from becoming total, because when the dependency is failing, retries triple the offered load precisely when capacity has fallen.
Retry only what is safe: idempotent operations, on retryable classes only — timeouts, connection failures, explicit 429 or 503 with a retry hint. Never retry a rejection caused by the request itself. "Idempotent" here is a property someone had to build, not one to assume from the verb — designing-for-idempotency covers the keys and dedup windows that make a retried write safe, and a retry policy laid over an operation that never got them is an amplifier for duplicates rather than a recovery mechanism.
Retry at one layer. Three layers retrying three times each is 27x amplification, and each layer looks reasonable in isolation. Choose the layer that owns the deadline and make the others propagate.

Removing retries at one layer does not lower the rate if the layer above re-drives the failure on its own tick: that layer is now the retry loop, and it needs a verdict it reads (or state every caller shares) to stop. Count per-attempt side effects before and after, because anything accounted per unit of the outer loop fires more often when the inner loop gets shorter.

Breakers and shedding

A circuit breaker trips on error ratio over a rolling window with a minimum sample count — tripping on one failure out of one is a self-inflicted outage — stays open for a cooldown, then half-opens to a limited probe before restoring. Its purpose is to stop spending capacity and latency on calls that are already known to fail, and to give the dependency room to recover.

Shed by value, not uniformly. Rank traffic and reject in ascending order of worth: bulk and batch work first, then retries, then anonymous and free-tier traffic, then interactive user work; never health checks, authentication refresh, or the control plane that lets you fix the incident. Shed at the edge, before a connection, thread, or transaction has been claimed — rejection after the expensive resource is held provides no relief at all. Uniform shedding is how a background job crowds out the paying user, because that job retries harder and cares less.

Tell the caller

A limit a client cannot see before hitting it forces every client to discover it by failing. Return a distinct, machine-readable status for "over your quota" versus "we are overloaded" — the first is the caller's to fix, the second is not and should be retried. That is exactly the split modeling-errors asks for before a mechanism gets chosen, arriving at the point where a client has to act on it: one class is a permanent answer about this request, the other is a temporary answer about the system. Include a Retry-After-equivalent hint with jitter already applied, and headers or fields naming the limit, the remaining allowance, and the reset time. Document the limit, the window, the dimension, and the burst allowance. Keep the error shape identical to other errors so client code handles it on an existing path rather than a new one.

Common mistakes

Symptom	Real cause
Latency grows without bound, nothing errors	Unbounded queue absorbing the overload signal
Load spikes every 30 seconds after an outage begins	Backoff without jitter synchronizing all clients
A dependency recovers and immediately falls over again	No retry budget or breaker; full retry backlog lands at once
Pool exhausted though the rate limit was never breached	Rate limited but not concurrency limited; a slow dependency held every slot
One tenant degrades everyone despite per-tenant limits	Sum of per-tenant limits exceeds real capacity; no global backstop
Legitimate users throttled while abuse continues	Limiting per IP behind NAT while the attacker rotates addresses
Queue drains hours after traffic stops	Work with expired deadlines never discarded
Every item in a fan-out times out, any single item alone finishes fast	Timeout clocks started at construction, so queue time behind siblings consumed the budget
The saturation symptom returns after a per-batch bound fixed it	The bound is scoped to the batch, so several independent callers each hold a full copy of the backend's capacity
Rejecting requests does not reduce load	Shedding after the connection, transaction, or thread is already claimed
Clients retry into the limit in a tight loop	No retry hint, no remaining-quota signal, limit undiscoverable in advance
One content type's growth silently evicted an unrelated type from a shared budget	A fixed one-shot capacity was split across non-fungible types with no reserved floor per type

Red flags

"We will just make the queue bigger"
A queue with no maximum size anywhere in the design
Retries added at a new layer without checking what the layers below already do
A limit chosen as a round number with no measured capacity behind it
A breaker that trips on a single failure, or that never half-opens
Alerting on queue depth but not on queue age
Timeout wrappers for a whole batch constructed up front, then awaited a few at a time
A semaphore created per batch, per request, or per caller for a resource they all share
"The client should not be sending that much" instead of a limit that makes it observable
Several unrelated content types sharing one undivided capacity with no reserved floor for any of them