Entity Resolution: The Hardest Part of Any Knowledge Graph

Why this is the make-or-break stage

A knowledge graph's value comes from paths. A path only exists if the node in the middle is shared by both ends. Miss one merge and the path disappears — silently. The query returns zero rows, nobody sees an error, and the team concludes graphs are overhyped.

The opposite error is worse. Over-merge two different people into one node and you have manufactured a false connection, which in fraud, compliance or healthcare is not a bug, it is an incident. Entity resolution is therefore a precision/recall trade-off with asymmetric costs, and the first design question is which direction your domain wants to err.

The standard pipeline

Normalisation

The cheapest wins live here. Deterministic clean-up before any fuzzy matching typically removes a third of your duplicate pairs at zero risk.

• Unicode NFKC, case-fold, collapse whitespace and punctuation.
• Strip legal suffixes for organisations: Ltd, Inc, GmbH, Pvt, LLC.
• Standardise structured fields: E.164 phone numbers, ISO dates, lower-cased email domains, postal codes via a proper address parser.
• Keep the original value. Always. Normalisation feeds matching; display uses the source.

const STRONG_KEYS = ["taxId", "email", "phoneE164", "isin"];

function deterministicKey(r: Record) {
  for (const k of STRONG_KEYS) {
    if (r[k]) return `${k}:${normalise(r[k])}`;
  }
  return null; // falls through to probabilistic matching
}

Blocking: escaping O(n²)

Ten million records is 5×10¹³ pairs. You will not score those. Blocking restricts comparisons to candidates that share a cheap key, and choosing that key well is the difference between a job that finishes in minutes and one that never does.

Use several blocking keys in parallel and take the union of candidate pairs. Multiple weak keys give far better recall than one clever key, because different keys fail on different errors.

Two numbers to track

• Pair completeness — the fraction of true duplicates that end up in at least one block. Below ~0.95 you are capping your recall before scoring even starts.
• Reduction ratio — how much of the full pair space you eliminated. You want >0.999 at scale.

Similarity scoring

Score each candidate pair field by field, then combine. Field comparators matter: use Jaro-Winkler for person names (it rewards matching prefixes), token-set Jaccard for organisation names (word order varies), edit distance for identifiers, and geo distance for addresses.

function score(a: Record, b: Record) {
  if (a.taxId && b.taxId && a.taxId !== b.taxId) return 0;      // veto
  if (a.dob && b.dob && a.dob !== b.dob) return 0;              // veto

  const parts = [
    { w: 0.35, s: jaroWinkler(a.nameNorm, b.nameNorm) },
    { w: 0.20, s: a.emailNorm && a.emailNorm === b.emailNorm ? 1 : 0 },
    { w: 0.15, s: tokenSetRatio(a.addressNorm, b.addressNorm) },
    { w: 0.15, s: cosine(a.embedding, b.embedding) },
    { w: 0.15, s: sharedNeighbourRatio(a.id, b.id) },           // graph signal
  ];
  return parts.reduce((t, p) => t + p.w * p.s, 0);
}

The last term is the one people forget and the one a graph gives you for free: two nodes that share several neighbours are more likely to be the same entity, even when their strings disagree. Structural evidence often beats string evidence on messy real data.

Rules, probabilistic, or learned?

From pairs to clusters

Pairwise decisions are not transitive: A≈B and B≈C does not make A≈C. Naive union-find over all matched pairs is how you get one giant blob node containing 40,000 "entities" — the classic transitive-closure explosion.

1. Build a graph of pairs weighted by score, dropping anything below the low threshold.
2. Cluster with a method that can split weak links — correlation clustering, or connected components followed by a modularity-based split.
3. Cap cluster size. A cluster of 500 people is a bug, not a discovery; send it to review instead of merging.
4. Pick a canonical record per cluster: most complete, most recently updated, or highest-trust source.

Merge policy and reversibility

Never destroy the inputs. The durable pattern is a canonical node plus preserved source nodes:

MERGE (c:Canonical:Customer {clusterId: $clusterId})
  SET c.name = $canonicalName, c.updatedAt = datetime()
WITH c
UNWIND $sourceKeys AS sk
  MATCH (s:SourceCustomer {key: sk})
  MERGE (s)-[r:SAME_AS]->(c)
    SET r.score = $score, r.method = $method, r.decidedAt = datetime();

• Queries traverse Canonical nodes; audits traverse back down SAME_AS.
• Unmerging is deleting one SAME_AS edge and re-clustering — not a data recovery exercise.
• Record method and score on the edge. "Why are these one customer?" must be answerable in one query.
• Human overrides live in a separate table of forced-merge and forced-split pairs, applied after scoring and never overwritten by a re-run.

Measuring it

You cannot tune what you have not labelled. Sample 300–500 candidate pairs stratified by score band and label them once; that gold set will serve you for a year.

• Pairwise precision/recall for threshold tuning.
• Cluster-level metrics (B-cubed or V-measure) — pairwise numbers hide blob clusters.
• Grey-band volume — how many pairs land in review. This is your human cost per million records.
• Drift alarms — merge rate per ingest batch. A sudden jump usually means an upstream format change, not a real change in the world.

Doing it at scale

• Incremental, not batch. New records only compare against their blocks; re-clustering is local to affected clusters.
• Idempotent runs. Same input, same clusters — otherwise downstream ids churn and every consumer breaks.
• Stable canonical ids. Derive from the cluster's oldest member, not from a hash of the whole membership, or ids change on every merge.
• Backpressure the review queue. If the grey band grows faster than humans clear it, tighten blocking rather than raising thresholds blindly.

Checklist

1. Decide the error asymmetry for your domain before setting thresholds.
2. Exhaust deterministic keys before any fuzzy matching.
3. Use multiple blocking keys; measure pair completeness.
4. Add graph structural similarity to your score — it is free and it is strong.
5. Cluster with size caps; never raw transitive closure.
6. Merge non-destructively with score, method and timestamp on every link.
7. Keep a labelled gold set and re-measure on every threshold change.

Further reading: Splink for a production-grade probabilistic implementation you can read, the dedupe library docs for active learning, and Neo4j's node similarity algorithms for the structural signals.