Distributed Locks: What They Are and How Not to Get Burned

What a distributed lock is

A distributed lock is a mechanism that lets many processes, on many machines, agree that only one of them may run a particular piece of work at a time. The agreement is stored somewhere outside all of them — Redis, ZooKeeper, etcd, a database row — because they share no memory and cannot see each other's variables.

Concretely: a monthly payout job runs on six application pods. Every pod's scheduler fires at 02:00. Without coordination, six pods pay every vendor. With a lock on the key payout:2026-09, one pod wins and the other five skip the run.

Locks are used for two very different reasons, and conflating them is the root of most production incidents:

• Efficiency. You want to avoid duplicate work — sending the same email twice, recomputing the same report. A rare double-run is annoying, not fatal.
• Correctness. A double-run corrupts data or moves money twice. Here a lock alone is never sufficient; you need fencing or idempotency underneath it.

How it differs from a normal lock

An in-process mutex is enforced by the operating system and CPU. If the holder dies, the kernel cleans up. There is no network in the path, and the "is it held?" question has exactly one authoritative answer that cannot be stale.

Ways to implement one

1. Redis single instance (SET NX PX)

The workhorse. Acquire with a unique owner token and a TTL, release only if you still own it. The release must be atomic, hence Lua — a GET-then-DEL can delete a lock someone else already acquired after yours expired.

-- acquire
SET lock:payout:2026-09 <owner-uuid> NX PX 30000

-- release (Lua, atomic)
if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
else
  return 0
end

• Fast, trivial to operate, good enough for efficiency locks.
• A single Redis node is a single point of failure; a replica failover can hand the same lock to two owners because replication is asynchronous.

2. Redlock across N independent Redis nodes

Acquire on a majority of 5 independent masters within a small time budget, subtract elapsed time from the validity window, release everywhere on failure. It removes the single point of failure but remains timing-dependent, and it is a well-known point of disagreement among distributed-systems practitioners. Do not use it as your only defence for money or inventory.

3. ZooKeeper / etcd — consensus-backed

The correct tool when correctness matters. Both give you a linearizable store with session semantics, so lock release on client death is handled by the system rather than by a guessed TTL.

• ZooKeeper: create an ephemeral sequential znode under /locks/payout. Lowest sequence number holds the lock; each waiter watches only its predecessor, which avoids the herd. The ephemeral node vanishes when the session expires.
• etcd: a lease plus a key, with the client keeping the lease alive. etcdctl lock and the concurrency package implement exactly this. The lease revision doubles as a fencing token.
• Cost: another stateful cluster to run, and higher latency per acquisition.

4. Your relational database

Often the best answer, because you already have it and it is already the system of record.

SELECT pg_try_advisory_lock(hashtext('payout:2026-09'));
-- ... work ...
SELECT pg_advisory_unlock(hashtext('payout:2026-09'));

INSERT INTO locks (name, owner, expires_at)
VALUES ('payout:2026-09', $1, now() + interval '30 seconds')
ON CONFLICT (name) DO UPDATE
  SET owner = $1, expires_at = now() + interval '30 seconds'
WHERE locks.expires_at < now()
RETURNING owner;

And the strongest form of all: SELECT … FOR UPDATE on the row you are about to change. That is not a distributed lock at all — it is the database's own concurrency control, and it cannot get out of sync with the data it protects.

5. Cloud primitives

DynamoDB conditional writes with a TTL attribute, Google Cloud Storage generation preconditions, Azure Blob leases, Consul sessions. All are the same lease-with-token idea on managed infrastructure.

Designing a system with distributed locks

Take a concrete brief: a payments platform must run a nightly settlement per merchant. Each merchant's settlement must run exactly once per day, must not overlap with itself, and may take anywhere from 5 seconds to 20 minutes.

1. Choose the lock granularity. One global lock serialises 50,000 merchants into a single thread. Lock per merchant per day: settle:{merchantId}:2026-09-16. Granularity is a throughput decision, and it is the one most often got wrong.
2. Make the lock a lease with a short TTL. 30 seconds, not 20 minutes. A long TTL means a crashed holder blocks the key until it expires.
3. Renew in a heartbeat from the worker while the job runs — extend the TTL every 10 seconds, and only if you still own the key. If three renewals fail in a row, stop the work yourself.
4. Attach a fencing token. Every acquisition returns a monotonically increasing number. Pass it to every downstream write.
5. Make the protected work idempotent anyway. Key each payout on (merchantId, date) with a unique constraint. Now a double-run is a constraint violation instead of a duplicate payment. See idempotency: the same request twice must charge once.
6. Fail closed and back off. If a worker cannot acquire, it does nothing and retries with jittered backoff — it must never assume the other holder finished.
7. Instrument it. Acquisition latency, contention rate, lock hold time percentiles, renewal failures, and expiries-while-working. That last metric is your correctness alarm.

const lease = await lock.acquire(key, { ttlMs: 30_000 });
if (!lease) return;                       // someone else owns it; leave quietly

const stop = lease.startHeartbeat(10_000); // renews only if still owner
try {
  // every write carries lease.fence; storage rejects stale fences
  await settle(merchantId, { fence: lease.fence, idempotencyKey: `${merchantId}:${date}` });
} finally {
  stop();
  await lease.release();                   // no-op if we already lost it
}

Failure modes and how to handle them

Lock expires while the work is still running

The most common real incident. The job took longer than the TTL, the lock expired, a second worker acquired it, and now two workers are inside the critical section.

• Heartbeat renewal, with the worker aborting when renewal fails.
• Fencing tokens so the storage layer rejects the stale writer.
• Set the TTL from measured p99 hold time, and alert when hold time approaches the TTL.

Process pause: GC, VM freeze, CPU starvation

A worker stops for 45 seconds mid-critical-section and resumes believing it still holds a 30-second lock. No amount of clever locking prevents this — the process was not running to notice. Only fencing at the write path saves you.

-- worker with fence 41 arrives after worker with fence 42 already wrote
UPDATE settlements
   SET status = 'paid', fence = 41
 WHERE merchant_id = $1 AND date = $2
   AND fence < 41;      -- 0 rows updated: the stale writer is silently rejected

Clock skew and drift

Never compare wall-clock timestamps across machines to decide expiry. Let the store own expiry (Redis TTL, etcd lease) and measure durations locally with a monotonic clock. Subtract the time spent acquiring from the validity you believe you have.

Network partition and split brain

The lock store says the holder's session is gone; the holder cannot reach the store and assumes it is still fine. Two believers, one lock. Handle it by making the holder self-fence: if it cannot confirm ownership within the TTL, it must stop writing. On the store side, use a quorum system so a minority partition cannot grant locks. This is precisely the CAP trade-off — a lock service must choose consistency over availability, or it is not a lock service.

Releasing someone else's lock

Always store an owner token and check it atomically on release and renewal. A bare DEL key is a latent outage.

Deadlock and lock ordering

Multiple locks per operation reintroduce deadlock. Acquire in a globally fixed order (sort keys), always use TTLs so nothing is held forever, and prefer a single coarser lock over two fine ones when the ordering rule gets hard to guarantee.

Thundering herd on release

A thousand waiters polling the same key at the same interval will hammer the store the instant it frees. Use jittered exponential backoff, or watch-based waiting (ZooKeeper predecessor watches, etcd watches) so each waiter is notified individually.

Store outage

Decide the policy explicitly and write it down. For correctness locks, no lock means no work — degrade to not running. For efficiency locks, running duplicated work may be acceptable. Never let the default be "the timeout swallowed the error and we proceeded".

When not to use a lock at all

Half the distributed locks in production should not exist. Cheaper and safer alternatives:

• Partition the work. Hash the key to a single owner shard or Kafka partition. One consumer per partition means exclusivity by construction — no lock, no expiry, no fencing. See how Kafka gives you this for free.
• Use the database's concurrency control. A unique constraint, an atomic UPDATE … WHERE version = $n, or SELECT … FOR UPDATE is stronger than any external lock.
• Make it idempotent and let it run twice. Often the whole problem dissolves.
• Elect a leader once instead of locking per operation.

Interview framing

"I'd lock per merchant per day rather than globally, so throughput scales with merchants. The lock is a 30-second lease in etcd, heartbeated by the worker; the lease revision is my fencing token and every settlement write is conditional on fence < current, so a GC-paused worker that wakes up late is rejected by storage rather than by trust. Settlements are also keyed on (merchant, date) with a unique constraint, so the lock is an optimisation and correctness does not depend on it. If etcd is unreachable, settlement does not run and we page — a correctness lock fails closed."

The line that earns the most credit: a lock is not a correctness mechanism unless the resource itself enforces the fence. Say it early.

Learn system design properly

Distributed locks only make sense once consistency, replication, leader election and failure detection are solid in your head. That is exactly what our System Design (HLD) module covers — a structured path through fundamentals, networking, load balancing, caching, consistency and real case studies, with diagrams and progress tracking rather than a pile of blog posts.

• The HLD roadmap — what to learn, in what order.
• Start the lessons — fundamentals, networking and load balancing first.
• Practise with structured rubrics and AI feedback, then read the top system design interview questions.