Anurag’s Read

Using load shedding to avoid overload.

A deep, practical guide to keeping a service useful under overload, with concrete tactics for queues, deadlines and testing. Many engineers meet these problems in production.
Score
9.1
Verdict
Must read
Blog and date
AWS,
Original length and level
22 min, advanced
Topics
load shedding, overload, reliability, queues
Read the original

Context

The author built shared service frameworks for AWS teams, which needed default limits for connections and timeouts. A fixed max-connections value failed because workloads and dependency speed change. Too low a limit rejects work with spare capacity, and too high a limit makes servers slow.

Architecture

Protection works in layers: automatic scaling first, then load shedding, then monitoring and continuous testing. Edge layers such as API Gateway and WAF drop the most extreme traffic cheaply. Load balancers and proxies get high limits and leave decisions to the server.

The server uses local information to shed, with priority rules and deadline checks. Dequeue logic discards old requests.

Mechanisms

M1

Goodput versus throughput

Throughput counts all requests sent to the server. Goodput counts requests answered without error and in time for the client. When median latency reaches the client timeout, about half of requests fail. Retries at each layer of a deep call graph multiply load.

M2

Deadline propagation

Clients send their remaining time with each request. Servers check the deadline after they dequeue a request and drop doomed work. Durations need a monotonic timer, while absolute times need synchronized clocks. Each hop passes on the time that remains.

M3

Request priority

Load balancer health pings rank first, because failed pings shrink the fleet. Calls that finish started work outrank calls that start new work. Later page requests outrank first-page requests. Human traffic outranks crawlers, and over-quota requests rank lower.

M4

Queue control

Hidden queues sit in thread pools, memory and TCP buffers. Servers record queue age at dequeue and discard requests past a limit. LIFO order helps where the protocol allows it, as HTTP/2 does.

Design decisions

No.ChoseInstead ofWhy
D1Fast-fail excess requests at the load balancerQueue them, as Classic Load Balancer didQueued requests caused brownouts because servers could not see their age.
D2Set load balancer connection limits highTight proxy limits such as NGINX max_connsProxies cannot rank requests, and in-flight counts mislead.

Key numbers

  • Median latency equal to timeout gives about 50% availability
  • Shedding 60% of traffic can hide poor latency in mixed metrics

Failures and limits

  • Shedding at the same CPU level as scaling can stop scaling from triggering.
  • Shedding hides true headroom, so AZ failure capacity needs testing.
  • Fast rejection can cost more than holding the request, so rejections may be slowed.

Lessons you can use

  • Load test far beyond the breaking point and watch client-side availability.
  • Keep rejected requests out of success latency metrics.
  • Target a zero false-positive rejection rate and alarm on rejections.
  • Bound the work per request, for example with pagination.

Glossary

GoodputThe rate of requests that finish correctly and in time.
BrownoutA state where a service is slow or partly failing but not down.
LIFO queueA queue that serves the newest request first.