What serverless actually means
Serverless is a set of operational properties, not a technology: no capacity you provision, billing tied to actual use, scale to zero, and the provider owning patching, placement and availability of the runtime. Functions-as-a-service is the famous part, but a serverless architecture is usually mostly managed services — object storage, managed queues, serverless databases — glued by functions.
The shape of a serverless system
upload -> object storage
| (event)
v
[fn: validate] --invalid--> dead-letter queue
|
v
queue (durable, retries)
|
v
[fn: transcode] (fan-out: 1 msg per output format)
|
v
write results -> storage + metadata table
|
v
[fn: notify] -> webhook / push
HTTP API: gateway -> [fn: handler] -> serverless DB / cache
Scheduled: cron -> [fn: reconcile]Note that every arrow is a managed, durable boundary. That is what makes the model work: each function is small, stateless and independently retryable, and the platform holds the state between them.
Cold starts, honestly
A cold start is the time to provision a new execution environment and initialise your runtime and dependencies before the first request is served. It hits when traffic starts, when it scales up, and after idle eviction.
| Factor | Effect | Mitigation |
|---|---|---|
| Runtime | Interpreted/JIT-light (JS, Python) start fastest; heavy JVM/.NET slowest | Pick the runtime for the latency path |
| Bundle size and deps | Every MB and every module init costs milliseconds | Tree-shake, lazy-require, minimal layers |
| VPC / private networking | Historically added seconds; much better now but non-zero | Only attach when required |
| Init work | Config fetch, secrets, DB pool at module scope | Cache across invocations; avoid per-request re-init |
| Idle traffic | Environments get reclaimed | Provisioned concurrency or a warmer, at a cost |
The practical rule: cold starts are a p99 problem, not a p50 problem. If your product has a hard interactive latency SLO on a low-traffic endpoint, either pay for provisioned concurrency or keep that endpoint on a long-running service.
Statelessness and the database problem
A function may run in a thousand concurrent environments. Two consequences dominate design:
- No local state survives. In-memory caches are per-environment and unpredictable; the filesystem is ephemeral. Anything durable goes to a managed store.
- Connection pools invert. Traditional pooling assumes few long-lived processes. A thousand functions each opening connections will exhaust a Postgres instance instantly. The fixes: an external connection pooler/proxy, an HTTP-based data API, or a database designed for serverless concurrency.
Reuse across invocations is real but not guaranteed — initialise clients at module scope so warm invocations reuse them, and never assume warmth for correctness.
Events, retries and idempotency
- At-least-once is the default. Platform retries, queue redelivery and client retries all mean handlers run twice. Every handler needs an idempotency key and a dedupe store.
- Dead-letter queues are mandatory. Without one, a poison message either disappears or retries forever, both of which cost money.
- Timeouts are hard limits. Long jobs must be decomposed into steps or moved to a workflow/step-function orchestrator with durable state between steps.
- Concurrency limits protect downstreams. Serverless scales faster than your database. Cap function concurrency explicitly, or the auto-scaling becomes an attack on your own datastore.
Cost: where the curve crosses
Serverless bills per invocation and per GB-second. That is dramatically cheaper for spiky, low-duty-cycle workloads and dramatically more expensive at sustained high throughput, where a reserved container running at 70% utilisation wins.
Serverless: invocations x (per-request fee + duration x memory price) Containers: replicas x hourly price (paid whether busy or idle) Crossover typically appears when average utilisation of an equivalent container fleet would exceed ~40-60%. Also count: API gateway per-request fees, NAT egress, per-request logging ingestion (often the real surprise).
When to use it and when not to
- Great fit: event processing, media pipelines, webhooks, cron and reconciliation jobs, glue between managed services, low-traffic internal tools, spiky public APIs, and anything embarrassingly parallel.
- Poor fit: sustained high-throughput hot paths, hard sub-50 ms tail latency, long-running or stateful compute, workloads needing GPUs and large model weights, and systems requiring heavy per-instance caching.
- Watch for: vendor lock-in through proprietary event formats and IAM models, and local-development pain that slows the team.
Interview framing
Strong answer
"Traffic is bursty and event-shaped, so I'd use serverless for the ingestion pipeline: storage event triggers a validation function, work goes onto a durable queue with a DLQ, and transcoding functions consume it with a concurrency cap so we do not overwhelm the metadata database. Handlers are idempotent on the upload ID. The interactive API has a 100 ms p99 requirement, so that stays on a long-running service — cold starts would put us over the budget for the same money. Database connections go through a pooler rather than direct from functions."
Follow-ups
- What happens when the function times out mid-write?
- How do you stop a traffic spike from destroying the database?
- At what traffic level would you move off serverless, and how would you know?