System Design · Lesson 2

Rate Limiting

Rate limiting controls how much traffic one client, user, token or tenant may send during a time period.

Why systems rate limit

Rate limits protect capacity, reduce abuse, contain bugs and make shared systems fairer. They can be applied at an API gateway, reverse proxy, application layer or directly around expensive operations.

Common algorithms

Fixed window is simple but allows bursts near window boundaries. Sliding window smooths those boundaries at higher storage cost. Token bucket supports controlled bursts while preserving an average rate.

Worked example

Limit an AI analysis endpoint to 20 requests/minute

Your endpoint triggers an expensive model call. A mobile retry bug could accidentally send hundreds of requests and burn budget.

LIMIT = 20
WINDOW = 60

async def allow(user_id: str) -> bool:
    key = f"rate:{user_id}:{int(time.time() // WINDOW)}"
    count = await redis.incr(key)
    if count == 1:
        await redis.expire(key, WINDOW + 1)
    return count <= LIMIT

Where to key it

Authenticated user ID is usually better than IP when users may share networks or switch between mobile connections.

What to return

Use HTTP 429 and include enough information for the client to back off instead of immediately retrying.

Client → API gateway → rate-limit check ├─ allowed → expensive AI call └─ blocked → 429 + retry-after

Trade-off: a fixed window is easy and often good enough. Move to token bucket or sliding-window logic when burst behavior materially affects users.

Distributed rate limits

When many app instances serve traffic, the counter must live in a shared store or at a shared gateway. Redis is common because atomic increment and expiry operations are fast.

Failure choice

Decide whether your limiter fails open or closed when the rate-limit store is unavailable. For a public read API, failing open may preserve availability. For expensive or sensitive writes, failing closed may be safer.