Nobody expects you to design a transformer in an interview. Assume the model exists and is served by an inference engine. What you are being asked is how to build a product around a dependency with properties no normal backend has: it takes seconds rather than milliseconds, it streams its answer, it runs on hardware that costs tens of thousands of dollars per node, and every request has a directly measurable marginal cost.
Why this is unlike any API you have designed
| Property | Typical REST API | LLM inference |
|---|---|---|
| Latency | 10–100 ms | 1–60 s, proportional to output length |
| Response delivery | One complete response | Token stream — first token matters most |
| Cost per request | Effectively zero | Cents to dollars; directly metered |
| Scarce resource | Database connections, CPU | GPU memory and GPU time |
| Scaling out | Seconds, elastic | Minutes; capacity is procured, not summoned |
| Concurrency limit | Thousands per instance | Tens per GPU, bounded by memory |
| Statefulness | Stateless | Per-request KV cache growing with the conversation |
| Determinism | Same input, same output | Sampled; retries produce different answers |
Every architectural decision follows from row one and row five. When your bottleneck cannot be autoscaled in the moment, the design must be about rationing: queueing, admission control, batching and prioritisation. That is a genuinely different discipline from the elastic stateless world described in What happens when 10M users hit your API.
Requirements and numbers
Functional scope worth claiming, and what to exclude:
- Send a message in a conversation and receive a streamed reply.
- Multi-turn context; conversation history persisted and listable.
- Stop generation mid-stream; regenerate a response.
- Free and paid tiers with different models and quotas.
- Out of scope: training, fine-tuning, tool use and RAG (mention them as extensions).
100M weekly users, 10M DAU
Average 8 messages/day per active user = 80M messages/day
≈ 925 messages/sec average
Peak 3x ≈ 2,800 messages/sec
Per message:
input ~1,500 tokens (history + system prompt)
output ~500 tokens
Output token demand at peak:
2,800 msg/s × 500 tokens = 1.4M output tokens/sec
One A100/H100-class node serving a mid-size model with batching:
~2,000–5,000 output tokens/sec
→ 1.4M / 3,000 ≈ 470 GPU nodes just for peak generation
→ this number, not RPS, is the entire capacity conversation
Latency targets:
time to first token < 1 s ← what users perceive as "fast"
inter-token latency < 50 ms ← must beat reading speed
total unbounded, because it streamsTwo things to notice. Capacity is measured in tokens per second per GPU, so a request generating 2,000 tokens costs four times one generating 500 — request-rate limiting alone is meaningless here. And time-to-first-token is the perceived-latency metric; total latency barely matters once text is flowing, because the user is reading.
The architecture
Client (web/mobile) │ POST /conversations/:id/messages (SSE response) ▼ Edge / API gateway ── auth, per-user rate limits, abuse signals ▼ Chat orchestrator (stateless, ordinary web tier) ├─▶ Conversation store (messages, metadata — durable) ├─▶ Context assembler (history selection + truncation/summary) ├─▶ Safety: input classification ├─▶ Quota service (tokens remaining, tier, concurrency) ├─▶ Prompt/response cache (exact + semantic) ▼ Inference queue (priority by tier, with admission control) ▼ Inference tier: model-sharded GPU pools, continuous batching │ token stream ▼ Orchestrator ── incremental safety filtering ── SSE to client ▼ Async: persist final message, meter tokens for billing, log for evals
The important separation: the orchestrator is a boring stateless service that scales elastically, and all the difficulty is isolated in the queue and the GPU pool behind it. Keep them separate so a GPU shortage degrades generation without taking down conversation history, listing, or login.
Streaming: SSE and first-token latency
A 20-second wait for a complete response feels broken; the same 20 seconds with text appearing after 600ms feels fast. Streaming is a product requirement, not an optimisation.
| Transport | Fit | Notes |
|---|---|---|
| Server-Sent Events | Best default | One-directional, plain HTTP, auto-reconnect, proxy-friendly |
| WebSocket | Good if you need duplex | More infrastructure; needed for voice or live interruption |
| Long polling | Fallback only | Wasteful and high latency |
| HTTP chunked without SSE | Works | You reimplement event framing and reconnection yourself |
GET/POST /conversations/:id/messages
Content-Type: text/event-stream
Cache-Control: no-cache
X-Accel-Buffering: no ← without this, nginx buffers and streaming dies
data: {"type":"start","messageId":"m_91","model":"gpt-x"}
data: {"type":"delta","text":"Rate"}
data: {"type":"delta","text":" limiting"}
data: {"type":"delta","text":" is"}
: keepalive ← comment frame every ~15s so proxies/LBs
do not kill an idle-looking connection
data: {"type":"done","usage":{"in":1502,"out":486},"finish":"stop"}Operational realities that break naive streaming implementations:
- Buffering proxies. Any layer that buffers destroys the effect. Disable buffering explicitly at the CDN, load balancer and reverse proxy — see nginx explained.
- Long-lived connections change your capacity model. Thousands of open connections for 30+ seconds each means connection count, not requests per second, is what your web tier must be sized for. Use an async runtime; thread-per-request will die here.
- Disconnects must cancel generation. If a user closes the tab and you keep generating, you are burning GPU seconds for nobody. Propagate cancellation all the way to the inference engine — this is real money.
- Resumability. Mobile networks drop. Persist tokens as they are produced, keyed by message ID, so a reconnect can replay from an offset instead of regenerating.
The inference tier
Generation happens in two distinct phases with completely different performance profiles, and understanding this is what makes the rest of the design make sense.
- Prefill. The input prompt is processed in parallel — compute-bound, fast per token, and it determines time-to-first-token. A long conversation history makes prefill expensive, which is why context management is a latency lever.
- Decode. Output tokens are generated one at a time, each depending on the last. Memory-bandwidth-bound and inherently sequential. This is why you cannot make a single response much faster — you can only serve more responses concurrently.
Because decode is bandwidth-bound rather than compute-bound, a single request wastes most of the GPU. Running many requests together costs barely more per step than running one, so batching is not a nice-to-have — it is the difference between 200 and 3,000 tokens per second per node.
Continuous batching and the KV cache
Naive (static) batching waits for N requests, runs them together, and waits for the longest one to finish before starting the next batch — so a request generating 20 tokens is held hostage by one generating 2,000. Continuous batching instead admits new requests into the running batch at every decoding step and evicts finished ones. It is the single most important throughput technique in LLM serving.
Static batching [==A (20 tok)....idle...........................] [==B (2000 tok) ................................] A finished long ago but its slot is wasted until B completes. Continuous batching step t: A B C D step t+1: A B C D A completes → slot freed immediately step t+2: E B C D E admitted mid-flight GPU stays saturated; throughput improves several-fold.
The constraint on batch size is GPU memory, and specifically the KV cache: each in-flight request stores the attention keys and values for every token so far, so its memory footprint grows as it generates.
KV cache per token ≈ 2 (K,V) × layers × hidden_dim × bytes_per_element
For a mid-size model, roughly ~0.5 MB per token of context.
4,000-token conversation ≈ 2 GB of KV cache for ONE request
80 GB GPU, ~40 GB free after weights → ~20 concurrent long conversations
Consequences:
- concurrency per GPU is tens, not thousands
- long conversations consume disproportionate capacity
- out-of-memory means preempting a request mid-generation
- paged attention (vLLM-style) allocates KV in blocks, cutting
fragmentation and roughly doubling effective concurrency
- prefix caching reuses the KV for a shared system prompt across
all users — free, large, and the first optimisation to implementPractical levers, roughly in order of value: paged attention, prefix caching of the system prompt, quantisation to shrink weights and free memory for KV, tensor parallelism to fit larger models, and routing short requests to smaller models.
Conversation state and context windows
The model is stateless; the product is not. Every turn resends the relevant history, so context assembly is both a correctness concern and your main cost lever — input tokens are billed and they dominate prefill time.
conversations(id, user_id, title, model, created_at, updated_at)
index (user_id, updated_at DESC) → sidebar listing
messages(id, conversation_id, role, content, token_count,
created_at, parent_id)
index (conversation_id, created_at) → load a thread
parent_id supports regenerate/branching
Hot path: last N messages of the active conversation in Redis,
so assembling context is one lookup, not a database scan.
Cold: old conversations archived to object storage; blobs are
large and rarely read, and this is mostly append-only data.Strategies when history exceeds the context window, each with a real trade-off:
| Strategy | How | Cost |
|---|---|---|
| Truncate oldest | Keep system prompt + last N turns | Model forgets earlier context — users notice |
| Rolling summary | Summarise old turns into a compact note | An extra inference call; lossy |
| Retrieval over history | Embed turns, retrieve the relevant few | Vector store; may retrieve the wrong turns |
| Hierarchical summary | Summaries of summaries | Complexity; compounding loss |
Always pin the system prompt and the most recent turns, and keep the prompt prefix byte-identical across requests so prefix caching can hit. A dynamic timestamp injected at the top of the system prompt silently invalidates that cache for every user — a common and expensive mistake. If you add retrieval over external documents, the trade-offs are covered in GraphRAG vs vector RAG.
Queueing, admission control and fairness
When GPUs are saturated you cannot scale out in the moment, so the queue is a first-class component, not an implementation detail. What you do with a full queue is the design.
- Priority by tier. Paid traffic gets a higher-priority class and a reserved share of capacity; free traffic uses the remainder. Reserve, do not merely prioritise — pure priority starves free users completely at peak, and that is a product decision someone should make consciously.
- Admission control, not unbounded queueing. If the estimated wait exceeds what a user will tolerate, reject immediately with a clear message. A 90-second queue produces a timeout, a retry and double the load — the cascade described in the 10M-user walkthrough.
- Per-user concurrency caps. One user with a script must not occupy twenty GPU slots. Cap concurrent generations per account, independently of the request-rate limit.
- Model-based routing. Send short or simple requests to a smaller, cheaper model. Routing even 40% of traffic to a model that is 5x cheaper transforms your unit economics, and most users cannot tell.
- Graceful degradation ladder. Under pressure: shorten max output length, drop to a smaller model, disable optional features, then finally shed load with an honest "at capacity" message and an estimated wait. Never a blank spinner.
- Regional pinning. GPU capacity is regional and scarce; route to the region with capacity, accepting the extra network latency, since 100ms of RTT is irrelevant next to a multi-second generation.
Cost, quotas and caching
This is the dimension that makes LLM systems unlike anything else: an enthusiastic user can cost you more than they pay, so metering is part of the architecture rather than a billing afterthought.
Quota keyed by (user, window), measured in tokens:
free: 50k tokens/day, small model, 1 concurrent generation
plus: 2M tokens/month, large model, 3 concurrent
team: pooled quota with per-seat caps
Enforcement:
- before generating: reserve an estimate (input + max_output)
- while streaming: decrement actual tokens
- after finishing: settle the difference, release the reservation
- hard-stop mid-stream when the quota is exhausted, with an
explicit "quota reached" event rather than a silent truncation
Request-count rate limiting alone is useless here: one request can
cost 100x another. Weight the cost by tokens — the same weighted
approach as in the rate limiter deep dive.The savings that actually matter, in order of impact:
- Prefix caching of the shared system prompt — enormous and nearly free.
- Exact-match response cache for identical prompts; small hit rate, zero risk.
- Semantic cache via embeddings for near-duplicate questions — good for FAQ-shaped traffic, risky for anything personalised, so scope it carefully.
- Model routing to smaller models by task complexity.
- Output length caps — the most direct control, since cost scales with generated tokens.
- Cancellation on disconnect — pure waste otherwise.
Rate limiting mechanics, including the weighted-cost variant, are in Design a rate limiter for 10M users.
Safety and abuse
- Input classification before inference — cheap, fast classifiers reject clearly disallowed requests without spending GPU time.
- Output filtering on the stream, which is genuinely awkward: you have already shown the user tokens. Buffer a small sliding window before emitting, so you can stop and replace mid-generation rather than retracting text on screen.
- Prompt injection. Any untrusted content in the context — a pasted document, a fetched page, a tool result — may contain instructions. Treat it as data, keep tool permissions minimal, and never let retrieved text escalate privileges.
- Abuse detection on patterns rather than single messages: scripted traffic, shared accounts, systematic jailbreak probing.
- Privacy. Conversations are unusually sensitive. Encrypt at rest, support deletion and export, keep an explicit training-data opt-out, and redact logs — do not log full prompts by default.
Reliability with a flaky, slow dependency
| Failure | Handling |
|---|---|
| GPU node dies mid-generation | Stream a recoverable error; retry once on another node from the last persisted token |
| Model returns garbage or loops | Repetition detection and a hard max-token cap |
| Inference tier saturated | Admission control with an honest wait estimate; degrade to a smaller model |
| Whole region out of capacity | Route cross-region; extra RTT is negligible against generation time |
| Client disconnects | Cancel generation immediately to stop burning GPU time |
| Safety filter unavailable | Fail closed — this is the one place not to fail open |
| Retry produces a different answer | Non-determinism is expected; never retry a partially streamed message blindly — resume from persisted tokens |
How to answer this in an interview
"I'll assume the model exists and focus on serving it, because the hard parts are that requests take seconds, stream their output, and run on hardware I can't autoscale in the moment. Numbers first: 10M DAU × 8 messages is ~900 msg/s, 2,800 at peak, and at 500 output tokens each that's 1.4M output tokens/sec. A GPU node does maybe 3,000 with good batching, so ~470 nodes. Capacity here is tokens/sec per GPU, not RPS — that's the whole conversation. Architecture: a stateless orchestrator handling auth, conversation storage, context assembly, quotas and safety, then a priority queue in front of GPU pools. I keep those separate so a GPU shortage doesn't take down history or login. Streaming over SSE, because time-to-first-token under a second is what feels fast — and I'd disable proxy buffering, send keepalives, and cancel generation on disconnect, since that's literally money. On the GPUs, continuous batching rather than static, because otherwise a 20-token response waits on a 2,000-token one. Concurrency is bounded by KV cache memory — a 4,000-token conversation is gigabytes — so paged attention and prefix caching of the shared system prompt, which means keeping the prompt prefix byte-identical rather than injecting a timestamp. Context management is the cost lever: pin the system prompt and recent turns, summarise or retrieve over older history. Rationing, since I can't scale out on demand: reserved capacity per tier rather than pure priority so free users aren't starved, per-user concurrency caps, admission control that rejects fast instead of queueing into timeouts, and routing simple requests to a smaller model. Quotas metered in tokens, not requests, because one request can cost 100x another — reserve an estimate, decrement while streaming, settle at the end. Trade-offs I'm accepting: cross-region routing for capacity, a smaller model under load, and truncated or summarised history — all degradations users tolerate far better than a spinner. And safety filtering fails closed, unlike everything else."
Summary
- Capacity is tokens per second per GPU, not requests per second.
- GPU capacity is not elastic, so the design is about rationing: queueing, batching, admission control.
- Stream over SSE; time-to-first-token is the perceived-latency metric, and buffering proxies break it.
- Continuous batching plus paged attention and prefix caching drive most of the throughput.
- KV cache memory bounds concurrency to tens per GPU, and grows with conversation length.
- Context assembly is the main cost lever: pin, truncate, summarise or retrieve.
- Meter quotas in tokens, cancel on disconnect, and route simple requests to cheaper models.
- Degrade gracefully — smaller model, shorter output, honest wait — and fail closed on safety.