Your Redis Cache Suddenly Became Useless

The pager goes off: database CPU at 95%, API latency up 8x, nothing deployed to the database. You look at the cache and the hit rate has fallen from 94% to 30%. The cache did not go down — it is up, responding, healthy. It just stopped being useful, which is worse than being down, because nothing alerted.

The important framing: a cache is not storage, it is a load-shedding device for your database. When hit rate drops from 94% to 30%, the database's read load does not rise by 64% — it rises by more than tenfold, because it was only ever seeing 6% of reads.

10,000 reads/sec at the API

hit rate 94%  →  database sees   600 reads/sec
hit rate 70%  →  database sees 3,000 reads/sec   (5x)
hit rate 30%  →  database sees 7,000 reads/sec   (11.7x)
hit rate  0%  →  database sees 10,000 reads/sec  (16.7x)

Hit rate is not a performance metric. It is a capacity multiplier.

The symptom and the metric that matters

Hit rate is the metric. If you take one operational lesson from this article: alert on cache hit rate, not on cache availability. A cache that is up and missing everything is the incident you will actually have.

Alongside it, track: evicted keys per second, expired keys per second, memory used against maxmemory, p99 command latency, and connected clients.

Diagnose in three commands

redis-cli INFO stats
  keyspace_hits:  1,204,553
  keyspace_misses:  830,221      → hit rate 59%  (bad)
  evicted_keys:   9,441,203      → EVICTION PRESSURE
  expired_keys:     221,884

redis-cli INFO memory
  used_memory_human: 15.94G
  maxmemory_human:   16.00G      → full; you are evicting to make room
  maxmemory_policy:  allkeys-lru
  mem_fragmentation_ratio: 1.4

redis-cli --bigkeys        # is one giant key eating the memory?
redis-cli --hotkeys        # is one key taking all the traffic?
redis-cli SLOWLOG GET 10   # are individual commands slow (KEYS, big SMEMBERS)?

Now read the numbers as a decision tree:

The seven ways a cache dies

1. A deploy changed the key format

Someone renamed a field, changed the serialiser, or added a tenant prefix. Every read is now a miss against a cache full of unreachable old data, so you are simultaneously at 0% hit rate and out of memory. This is the most common self-inflicted version.

Prevent it by versioning keys deliberately — v3:user:{id}:profile — and treating a version bump as a planned cold-cache event: roll it out gradually, or warm the new namespace before switching reads.

2. Eviction pressure

The working set grew past maxmemory so Redis evicts to make room, and it evicts the key you were about to read. Under LRU with a full keyspace, hit rate degrades non-linearly: a little over capacity is fine, well over capacity collapses. Fix by adding memory, shortening TTLs on low-value data, caching smaller values, or splitting hot and cold namespaces into separate instances so a bulk job cannot evict your hot entities.

Also check maxmemory-policy. noeviction turns memory pressure into write errors; volatile-lru only evicts keys with a TTL, so keys written without one are immortal and will fill the instance. Set TTLs on everything.

3. Synchronised TTLs

You warm 200,000 keys at deploy time with a flat 10-minute TTL. Ten minutes later they all expire in the same second and the full read volume hits the database at once. The signature is a sawtooth hit-rate graph with a period equal to your TTL.

// bad: every key created together dies together
await redis.set(key, value, { EX: 600 });

// good: spread expiry over a window
const ttl = 600 + Math.floor(Math.random() * 120);   // 600–720s
await redis.set(key, value, { EX: ttl });

4. Cold start after restart or failover

A failover, a scale-up, or an eviction-policy change gives you an empty cache and full traffic at the same moment — a stampede by construction. Mitigate with persistence or replicas so a failover keeps the dataset, and a warm-up phase that loads the top-N keys before the instance takes traffic.

5. Hot key

One key — a global config, a trending post, a feature-flag blob — takes a disproportionate share of traffic. Redis is single-threaded per shard, so one hot key saturates one core and latency rises for everyone on that shard. Sharding by key does not help, because it is one key. The fix is a local in-process cache with a very short TTL (1–5 seconds) in front of Redis, which collapses millions of requests into a handful.

6. Big keys and blocking commands

A 200MB list, a set with 5 million members, an HGETALL on a huge hash, or KEYS * in production. Redis processes commands one at a time; one slow command stalls every other client. Use SCAN instead of KEYS, paginate large structures, and check --bigkeys regularly.

7. Wrong granularity or broken invalidation

Caching an entire dashboard response means one changed field invalidates everything; caching per tiny field means dozens of round trips per request. And if invalidation is write-path-dependent and one write path forgot to invalidate, you serve stale data until the TTL saves you — which looks like a correctness bug, not a cache bug.

Stampedes and how to stop them

A stampede is the specific event where a popular key expires and a thousand concurrent requests all miss, all query the database, and all write the same value back. The database does a thousand times the necessary work at the worst possible moment.

async function getWithCoalescing(key, loader, ttl) {
  const hit = await redis.get(key);
  if (hit !== null) return JSON.parse(hit);

  // only one caller wins the lock and loads from the database
  const gotLock = await redis.set(`lock:${key}`, "1", { NX: true, EX: 10 });
  if (!gotLock) {
    await sleep(50);
    return getWithCoalescing(key, loader, ttl);   // bounded retries in practice
  }

  try {
    const value = await loader();
    const jitter = Math.floor(Math.random() * ttl * 0.2);
    await redis.set(key, JSON.stringify(value), { EX: ttl + jitter });
    return value;
  } finally {
    await redis.del(`lock:${key}`);
  }
}

Two more techniques worth naming:

• Probabilistic early recomputation. As a key approaches expiry, a small random fraction of readers refresh it in the background, so it is almost never actually missing.
• Serve stale while revalidating. Store value plus a soft_expiry, with the physical TTL much longer. Past soft expiry, return the stale value immediately and refresh asynchronously. Reads never block on the database, and a database outage degrades to slightly stale data instead of an error.

Invalidation done properly

The pragmatic default: TTL as the safety net, plus explicit invalidation as the fast path. Invalidation will occasionally be missed; the TTL bounds how long that hurts. Never rely on invalidation alone. The mechanics of the read path itself are covered in the cache-aside pattern, and the data-structure and operational side of Redis in Redis explained.

Designing a cache that degrades gracefully

• Alert on hit rate, with a threshold based on what the database can survive.
• Version every key and treat a version change as a controlled cold start.
• TTL on everything, always jittered.
• Coalesce misses so one key can only cause one database query.
• Serve stale on refresh failure rather than propagating the error.
• Local cache in front of Redis for hot keys, with a 1–5s TTL.
• Separate namespaces so a batch job cannot evict interactive data.
• Treat cache unavailability as a fallback, not an error — but rate-limit the resulting database load, or you have traded a cache outage for a database outage.
• Load test with the cache disabled so you know whether the database can survive 0% hit rate at all. Usually it cannot, which is itself worth knowing.

How to answer this in an interview

"First I'd confirm it's a hit-rate problem, not a Redis problem: INFO stats for
 keyspace hits vs misses and evicted_keys, and INFO memory for maxmemory pressure.

 That splits the diagnosis. Evictions climbing with memory full means the working
 set outgrew the instance. Misses high with no evictions means keys aren't being
 found — usually a deploy changed the key format, which gives you 0% hit rate and
 a cache full of unreachable data at the same time. A sawtooth hit rate means
 synchronised TTLs all expiring together.

 To stabilise I'd rate-limit or shed the expensive read path so the database
 survives, then warm the cache before restoring full traffic — otherwise I just
 re-create the stampede.

 Structurally: versioned keys, jittered TTLs, single-flight loading so one key
 causes at most one database query, and stale-while-revalidate so a refresh
 failure serves slightly old data instead of an error. And I'd alert on hit rate,
 because a cache that's up and missing everything doesn't page anyone today."

Summary

• Hit rate is a capacity multiplier — a drop from 94% to 30% is over 10x database load.
• INFO stats and INFO memory distinguish eviction pressure from key-not-found.
• The usual causes: changed key format, eviction, synchronised TTLs, cold start, hot key, big key, broken invalidation.
• Jitter every TTL and coalesce misses so one key causes one database query.
• Stale-while-revalidate makes the cache an availability layer, not just a speed-up.
• Alert on hit rate; cache availability alone will not warn you.

Part of Top 10 System Design Interview Questions.