Elasticsearch Explained: Inverted Indexes, Sharding, Relevance and Sizing

Why a search engine, not a database

WHERE description LIKE '%wireless headphone%' is a full table scan that cannot rank results, cannot handle typos, cannot stem "running" to "run", and cannot tell you that a match in the title matters more than one in a footer. Elasticsearch exists because relevance-ranked full-text retrieval is a fundamentally different problem from row lookup.

• Full-text search with stemming, synonyms, fuzziness and phrase matching.
• Relevance ranking — results ordered by score, not insertion order.
• Aggregations — faceted counts, histograms, percentiles over billions of docs.
• Horizontal scale — shards spread across nodes; queries fan out and merge.

The inverted index and analyzers

A forward index maps document → words. An inverted index maps word → documents. That inversion is the whole trick: finding every doc containing "wireless" becomes a single dictionary lookup instead of a scan.

char filters   : strip HTML/punctuation noise
tokenizer      : ["Running", "Shoes", "Waterproof"]
lowercase      : ["running", "shoes", "waterproof"]
stop words     : (removes "the", "and", ...)
stemmer        : ["run", "shoe", "waterproof"]

inverted index
  run        -> [doc1, doc7, doc9]
  shoe       -> [doc1, doc3, doc7]
  waterproof -> [doc1, doc5]

Mappings: text vs keyword

"title":  { "type": "text", "fields": { "raw": { "type": "keyword" } } }

That multi-field pattern — title for searching, title.raw for sorting and faceting — is the single most useful mapping idiom. Also: turn off dynamic mapping in production, or one bad payload will create thousands of fields and blow up your cluster state.

Shards, replicas and segments

• An index is split into primary shards; each shard is a complete Lucene index.
• Each primary can have replica shards — redundancy plus extra read throughput.
• Each shard is made of immutable segments. Updates write a new doc and tombstone the old one; deletes are logical until a merge.
• Documents route by hash(routing_key) % primary_shards, which is why the primary count is fixed at creation.

Near-real-time: refresh, flush, merge

1. Index — the doc lands in an in-memory buffer plus the durable translog.
2. Refresh (default every 1s) — buffer becomes a searchable segment. This is why writes are visible in ~1s, not instantly.
3. Flush — segments fsync to disk, translog truncated.
4. Merge — background compaction of small segments, physically removing deleted docs.

How a search executes

Scatter-gather in two phases. Query phase: the coordinating node fans the query to one copy of every shard; each returns the top-N doc IDs and scores. Fetch phase: the coordinator merges, keeps the global top-N, then fetches those documents' source.

This is also why deep pagination is deadly: from: 10000, size: 10 makes every shard return 10,010 hits to be merged. Use search_after with a tiebreaker sort for deep paging, and the point-in-time API for stable snapshots.

{
  "query": {
    "bool": {
      "must":   [{ "multi_match": { "query": "wireless headphones",
                                     "fields": ["title^3", "description"] } }],
      "filter": [{ "term":  { "in_stock": true } },
                 { "range": { "price": { "lte": 200 } } }]
    }
  }
}

Relevance and BM25

• Term frequency — more occurrences score higher, with diminishing returns.
• Inverse document frequency — rare terms carry more signal than common ones.
• Field length normalisation — a match in a 5-word title beats one in a 5,000-word body.

Tune with field boosts (title^3), function_score for business signals like recency or popularity, and phrase matching in a should clause to reward exact order. Always debug with the _explain API rather than guessing at score changes.

Aggregations

"aggs": {
  "by_brand":  { "terms": { "field": "brand.raw", "size": 10 } },
  "price_p95": { "percentiles": { "field": "price", "percents": [95] } },
  "per_day":   { "date_histogram": { "field": "created_at", "calendar_interval": "day" } }
}

Bucket, metric and pipeline aggregations turn the search cluster into an analytics engine. Two caveats: terms counts are approximate across shards, and high-cardinality aggregations are memory-hungry — that is what circuit breakers are protecting you from.

Sizing and operations

The dominant failure mode is over-sharding: hundreds of tiny shards, each with its own overhead, bloating cluster state and slowing every query. Fewer, bigger shards is almost always the right correction.

Common mistakes

• Using Elasticsearch as the system of record. It has no transactions and no joins — keep the source of truth in a database and index into ES.
• Deep from/size pagination instead of search_after.
• Scoring things that should be filters — filters are cached, queries are not.
• Dynamic mapping in production, causing field explosion.
• Exposing the cluster directly to clients; queries are executable and expensive.
• Reindexing by hand instead of using aliases — always read/write through an alias so you can swap indices atomically.

Interview framing

"Postgres stays the system of record and we stream changes into Elasticsearch, so search availability never threatens write correctness. Documents are analyzed once at index time into an inverted index; queries use the same analyzer, with binary conditions in filter context so they are cached and unscored. Search is scatter-gather across shards, so we avoid deep pagination with search_after, size shards at 10–50 GB, and use rolling indices behind an alias so a reindex is an atomic alias swap. Writes are visible after the 1-second refresh — that near-real-time window is stated as a product requirement, not discovered as a bug."