"10M users" is not a number you can design against. Ten million users who open the app once a week is a small system; ten million users refreshing a live scoreboard is a very large one. So the first move — in production and in an interview — is to convert users into requests per second, bytes and connections. Then walk the request path and find the hop with the lowest ceiling, because that hop is your capacity.
Turn users into numbers first
A rough model that is good enough to design against:
10,000,000 registered users × 20% daily active = 2,000,000 DAU × 30 API requests per active session = 60,000,000 requests/day 60,000,000 / 86,400 ≈ 700 requests/sec average peak factor 5x (evening spike) ≈ 3,500 requests/sec peak Read/write ratio, say 90:10 → 3,150 reads/s, 350 writes/s Average response 8 KB → ~28 MB/s egress at peak p99 latency target 300 ms
Those five numbers change everything that follows. 3,500 requests per second is comfortably served by a handful of application instances and one well-indexed database with a cache. 350,000 requests per second is a different architecture. Always state the numbers before drawing boxes — that is also the core advice in How to start any system design interview.
The request path, hop by hop
Edge: DNS, TLS and CDN
The edge is where you make most traffic never reach your servers, which is the cheapest scaling you will ever do.
- DNS rarely breaks under load because resolvers cache it, but a short TTL during a traffic spike multiplies lookups, and a single-provider outage takes you down entirely. Use a managed anycast provider, sane TTLs, and health-checked failover.
- TLS handshakes are the expensive part of a new connection. Terminate TLS at the edge, enable session resumption and HTTP/2 or HTTP/3 so clients reuse connections instead of paying the handshake repeatedly.
- CDN cache hit ratio is the single biggest lever. Static assets should be near 100%. Then push further: cache GET API responses that are the same for everyone (product catalogue, public feeds, config) at the edge for even 5–30 seconds. At 3,500 rps, a 10-second edge cache on one popular endpoint can remove tens of thousands of origin requests per minute.
Load balancer and app tier
The load balancer is limited by concurrent connections and new-connections-per-second, not by bandwidth. Two things bite here: keep-alive settings that make idle connections accumulate, and health checks that are too aggressive, so a briefly slow instance gets pulled out, shifting its load onto the rest and taking them out too.
The app tier is limited by concurrency, and how much concurrency you have depends on the model:
| Concurrency model | Effective limit | What blocking IO does |
|---|---|---|
| Thread/process per request (classic) | Threads per instance (tens to low hundreds) | One slow dependency consumes all threads |
| Async event loop (Node, Go, async Python) | Thousands of in-flight requests | CPU-bound work blocks everything instead |
| Serverless functions | Concurrency quota per account/region | Cold starts and downstream connection explosion |
Two invariants make the app tier scalable: it must be stateless (session in a shared store or a signed token, never in instance memory) and it must be fast to start, because autoscaling that takes four minutes does not help a spike that lasts two.
The real ceiling: connection pools
This is the failure mode people miss, and the one interviewers are usually fishing for. App instances scale horizontally; databases do not. Every instance carries its own pool, so the database's connection count scales with your instance count:
40 app instances × pool size 20 = 800 connections wanted PostgreSQL max_connections = 200 Result: connection errors, retries, more instances launched by the autoscaler, more connections wanted — a self-reinforcing loop. Fix: PgBouncer in transaction mode. 800 client connections → ~40 real server connections Plus: pool size per instance must shrink as instance count grows.
The same logic applies to every downstream: third-party APIs with rate limits, Redis connections, file descriptors. Whenever you multiply a per-instance resource by an autoscaling instance count, you have built a way to overwhelm something.
The data tier
At 3,150 reads and 350 writes per second, a single well-tuned primary is fine — provided the queries are indexed and a cache absorbs the hot reads. What actually helps, in order:
- Cache the hot reads. Read-through cache on the entity-by-id path. A 90% hit rate turns 3,150 reads/s into 315 database reads/s.
- Fix the queries. Indexes and killing N+1 patterns buy more headroom than any hardware upgrade. See Your database is at 100% CPU.
- Read replicas for reads that tolerate lag: search, listings, analytics. Keep read-your-own-writes on the primary.
- Make writes asynchronous where the user does not need the result: counters, activity feeds, notifications, audit logs. Enqueue, acknowledge, process behind the scenes.
- Partition, then shard only when a single primary genuinely cannot take the write volume.
How it becomes an outage
Systems rarely fail gracefully at the limit. They fail like this:
- One dependency slows from 20ms to 2s — a hot partition, a GC pause, a noisy neighbour.
- Requests to it hold worker threads for 100x longer, so the app tier's usable concurrency collapses.
- Queues build, latency rises, clients time out — and retry, tripling the offered load.
- The autoscaler adds instances, each opening more database connections, worsening the bottleneck.
- Health checks fail, instances are removed, the survivors get more load, and everything is down — including endpoints that never touched the slow dependency.
The defences are the resilience patterns, and they matter more than raw capacity:
| Defence | What it prevents |
|---|---|
| Aggressive timeouts on every call | Threads held hostage by a slow dependency |
| Retries with exponential backoff + jitter | Retry storms that multiply load |
| Circuit breaker | Hammering a dependency that is already down |
| Bulkheads (separate pools per dependency) | One slow dependency starving unrelated endpoints |
| Load shedding / admission control | Total collapse — serve 90% instead of failing 100% |
| Rate limiting per client | One abusive caller consuming the system |
Each has a dedicated deep dive: circuit breaker, bulkhead, and rate limiting.
The scaling playbook
- Push work to the edge. Cache anything identical across users, even briefly.
- Keep the app tier stateless so it scales horizontally and restarts freely.
- Pool and cap connections — this is the ceiling that surprises people.
- Cache the hot read path before adding database capacity.
- Make non-critical writes asynchronous through a queue.
- Isolate failure with timeouts, breakers and bulkheads so degradation is partial.
- Load test to failure so you know which hop breaks first — and instrument that hop.
How to answer this in an interview
Do not list technologies. Convert users to numbers, walk the path, name the ceiling at each hop, and finish with the cascade story — that last part is what makes you sound like someone who has been paged.
"10M users tells me nothing until I know DAU and requests per session. Say 2M DAU, 30 requests each, 5x peak — about 3,500 rps peak, 90% reads. At that volume the app tier is easy: stateless, autoscaled. The ceilings are elsewhere. Edge caching removes most read traffic. Then the one that actually bites is database connections — 40 instances × pool 20 exceeds max_connections, so I'd put a transaction-mode pooler in front and cap per-instance pools. And I'd assume something will get slow, so timeouts, jittered retries, circuit breakers and per-dependency bulkheads — otherwise one slow dependency takes the whole API down, not just the endpoints that use it."
Summary
- Convert users into requests per second, read/write ratio and payload size before designing.
- Capacity equals the smallest ceiling on the request path, not the average.
- The edge is the cheapest scaling: cached requests never reach you.
- App tiers scale horizontally; connection pools and databases do not — pool and cap.
- Cache hot reads, make non-critical writes async, add replicas before sharding.
- Outages come from cascades: timeouts, jittered retries, breakers and bulkheads are the fix.