Entity Resolution at Scale: One Patient, Fifty Records
The same patient appears differently across EMR, lab, pharmacy, and claims systems. Probabilistic entity resolution running as a continuous process — not a one-time ETL — is the foundation every downstream metric depends on.
One patient. Fifty records. Not a hypothetical. This is the operational reality at every healthcare organization integrating data from more than a handful of source systems.
A single patient generates records across EMR systems, laboratory information systems, pharmacy dispensing platforms, claims clearinghouses, radiology archives, and patient portals. Each system assigns its own identifier. Each system captures a slightly different version of the patient's name, address, and date of birth. Each system has its own data model, its own update frequency, and its own concept of what constitutes a "patient record."
The result is identity fragmentation at scale. And identity fragmentation is not a data quality problem. It is a clinical operations problem. Every metric, every care gap, every AI response, every campaign, and every care coordination workflow depends on one foundational question: do we know who this patient is?
Why Simple Matching Fails
The intuitive approach to entity resolution is deterministic matching: if two records share the same first name, last name, and date of birth, they are the same patient. This works in demos. It fails in production for predictable reasons.
Name variation. "Rajesh Kumar" in the EMR, "R. Kumar" in the lab system, "Rajesh K." in the claims feed. Transliterations, abbreviations, maiden names, legal name changes, and data entry errors mean that the same person's name can appear in a dozen variations across systems.
Date of birth errors. Date of birth is treated as a reliable identifier, but entry errors are common. A transposed digit — 03/15/1978 vs 03/15/1987 — breaks deterministic matching entirely. Some systems store estimated dates for patients who do not know their exact birthdate.
Address instability. Patients move. Address is a weak identifier in healthcare populations, where the average patient changes address every 3-5 years. Homeless patients may have no stable address at all.
Identifier collisions. Medical record numbers are unique within a single system but meaningless across systems. Two patients at two hospitals can share the same MRN. Using MRN as a cross-system identifier produces false positives.
Duplicate records within a single system. Even before cross-system matching, most EMRs contain internal duplicates — the same patient registered twice with slightly different demographics. Resolution must handle intra-system and inter-system duplicates simultaneously.
Why Deterministic Matching Fails
Simple name-and-DOB matching breaks down against the realities of healthcare data
Name Variation
Abbreviations, transliterations, legal changes, and entry errors produce dozens of name variants for a single patient.
DOB Entry Errors
Transposed digits, estimated dates, and format inconsistencies make date of birth unreliable as a sole matching key.
Address Instability
Patients move frequently. Address is one of the weakest demographic identifiers in healthcare populations.
Identifier Collisions
Medical record numbers are system-local. Cross-system MRN matching produces false positives at scale.
Intra-System Duplicates
Most EMRs already contain internal duplicates that must be resolved before cross-system matching begins.
Temporal Decay
Demographics change over time. A match that was valid two years ago may no longer hold after name or address changes.
Probabilistic Matching: Trading Certainty for Accuracy
Probabilistic matching takes a fundamentally different approach. Instead of requiring exact agreement on a set of fields, it computes a likelihood score based on partial agreement across many fields. Two records that agree on last name, date of birth, and phone number but disagree on first name might still score above the match threshold — especially if the first names are phonetically similar.
The mathematics involve assigning weights to each field based on its discriminating power. Agreement on a rare last name like "Zygomalas" is stronger evidence of a match than agreement on "Smith." Agreement on phone number is stronger than agreement on gender. The system computes a composite score and classifies each pair as a match, a non-match, or a candidate for manual review.
This approach requires careful calibration. Set the threshold too low, and you merge distinct patients — a dangerous clinical error. Set it too high, and you leave duplicates unresolved, fragmenting the patient's longitudinal record.
The calibration is not a one-time exercise. As data volumes grow, as source systems change, and as patient populations evolve, the matching thresholds must be re-evaluated. A resolution engine that was calibrated for a single hospital will produce different error rates when applied to a multi-facility health system with overlapping catchment areas.
Resolution Is a Continuous Process
The traditional approach to entity resolution is batch ETL: run a matching job overnight, produce a master patient index, and use it until the next run. This model is inadequate for modern healthcare operations.
New data arrives continuously. A patient registers at an urgent care facility at 2 PM. A lab result arrives at 3 PM from an external reference lab. A pharmacy claim posts at 4 PM. If entity resolution runs overnight, there is a 12-hour window during which these records are unlinked — invisible to care coordinators, missing from dashboards, excluded from AI context.
Continuous entity resolution processes new records as they arrive, evaluating them against the existing identity graph in near real-time. When a new record arrives, the resolution engine:
- Extracts demographic attributes from the incoming record
- Generates candidate matches from the existing identity graph using blocking strategies (narrowing the search space without exhaustive comparison)
- Scores each candidate pair using the probabilistic matching model
- Either links the record to an existing identity, creates a new identity, or flags the record for manual review
- Updates all downstream systems that depend on the identity graph
This is not a batch job. It is a streaming pipeline that maintains the identity graph as a living data structure.
Continuous Entity Resolution Pipeline
Identity resolution as a streaming process, not an overnight batch job
The Downstream Impact
Identity resolution is not a data engineering curiosity. It is the foundation on which every downstream computation depends. When identity is wrong, everything built on top of it is wrong.
Care gaps become unreliable. If a patient's lab results are linked to one identity and their diagnosis history is linked to another, the care gap engine evaluates an incomplete record. It may flag a care gap that has already been addressed — or miss one that is genuinely open.
AI hallucinations have a new source. When an AI agent retrieves a patient's longitudinal record to answer a clinical question, an incomplete record due to unresolved identity fragments is indistinguishable from a genuinely sparse history. The model answers confidently based on incomplete data. No amount of prompt engineering fixes a fragmented identity graph.
Population metrics are inaccurate. A hospital that cannot resolve identities across its facilities double-counts patients, inflating census numbers, distorting readmission rates, and misrepresenting quality metrics to payers and regulators.
Campaign outreach is duplicated. A patient who exists as three unresolved identities receives three copies of every campaign message. This is not just wasteful — in healthcare, it erodes trust and can trigger compliance concerns.
Care coordination breaks. When a care coordinator reviews a patient's record and sees only the encounters from one facility, they make decisions without the full clinical picture. The referral that happened at another facility, the medication that was prescribed by an outside provider, the lab result from an external lab — all invisible.
Building Resolution Into the Platform
Entity resolution cannot be an afterthought — a tool that runs on top of the data layer after the data has already been stored and consumed. It must be a core layer of the data platform, positioned between ingestion and every downstream consumer.
Every record that enters the platform passes through the resolution engine before it is available to any application, any API, any dashboard, or any AI model. This is a strong architectural constraint, and it is the only way to guarantee that every consumer operates on resolved identities.
The resolution engine must also handle the inverse: when two previously distinct identities are discovered to be the same patient (a merge), every downstream reference must be updated. And when a merge is determined to be incorrect (an unmerge), the operation must be reversible without data loss.
This bidirectional resolution — linking and unlinking, merging and unmerging — requires the identity graph to be a versioned, auditable data structure where every decision is traceable. Who merged these records? When? Based on what evidence? Can it be reversed?
The Bottom Line
Entity resolution is not a solved problem you can handle with a matching algorithm and a batch job. It is continuous, probabilistic, and operationally critical -- the foundation of every healthcare data platform. Get identity right, and everything downstream works. Get it wrong, and no amount of analytics, AI, or workflow automation compensates for not knowing who your patients are.
The fifty records are not the problem. The problem is that they do not know they belong to the same person. Solving that -- continuously, accurately, at scale -- is where healthcare data engineering actually begins.
THB's DataCloud includes a continuous entity resolution engine that maintains a live master patient index across all source systems, with probabilistic matching, automated merge/unmerge workflows, and real-time identity graph updates. Explore the DataCloud architecture.