AI Glossary / Graphs and symbolic AI / Intermediate
Record linkage
Definition
Record linkage is the process of deciding which records refer to the same real-world entity across or within datasets. The entity might be a person, organization, product, or location. Linkage uses identifiers and other evidence to connect records despite differences or missing information, while managing the possibility of both incorrect links and missed matches.
What it means
Two systems may describe the same supplier differently. One stores a trading name and an old address; another stores a legal name and a current address. Joining on one exact text field can miss the connection, while joining on a common name can accidentally combine different suppliers.
Record linkage treats identity as a decision supported by evidence. It asks whether records concern the same entity, rather than whether their text merely looks similar. The answer also depends on the entity definition: two branches can belong to one company while remaining distinct delivery locations. Establishing that definition prevents technically plausible matches from answering the wrong business question.
How it works
Prepare relevant fields, retaining original values so transformations can be checked. Generate candidate pairs, often using blocking rules that avoid comparing every record with every other record. Compare the candidate fields and decide whether to link, reject, or refer the pair for review.
Deterministic approaches use explicit matching rules. Probabilistic approaches assess how strongly agreements and disagreements support a match. Agreement on a rare value can be more informative than agreement on a common one. Missing values require deliberate handling rather than automatic treatment as agreement.
Validate accepted and rejected candidates, including cases excluded by candidate generation. Document the rules, thresholds, assumptions, and corrections. A linkage score is evidence for a decision; it is not necessarily a calibrated probability that the records belong together.
Three practical examples
- Supplier management: an organization connects purchasing records with its approved-supplier register despite variations in naming and formatting.
- Research preparation: analysts link repeated survey records to study changes over time while evaluating linkage errors.
- Catalog maintenance: a retailer identifies product records that describe the same model even when descriptions use different abbreviations.
Think of it this way
Think of sorting several sets of labeled folders after their labels have been typed differently. Matching labels help, but similar labels alone do not prove that the contents belong together. You also inspect other clues and keep uncertain folders separate. The analogy is limited because automated linkage needs reproducible rules and quantified checks rather than an undocumented feeling of familiarity.
Worked example
Consider a fictional validation exercise with a trusted reference set containing 100 true record pairs. A linkage system accepts 90 pairs. Inspection finds that 81 are correct and nine connect different entities.
Precision is 81/90 = 90%: nine out of ten accepted links are correct. Recall is 81/100 = 81%: the system finds 81 of the 100 true links and misses 19. These answer different questions.
Suppose a revised candidate-generation stage adds eight accepted links, seven correct and one incorrect. The totals become 98 accepted, 88 correct, and ten incorrect. Precision is 88/98, about 89.8%, while recall rises to 88%. The change improves coverage with a small precision reduction.
These are invented counts, not a benchmark or a recommended trade-off. Crucially, reviewing only accepted links could estimate precision but would not reveal all 19 original missed matches. Recall assessment needs evidence about true links beyond the accepted set. Whether the revision is useful depends on the consequences of each error.
Illustrative case study
Fictional case: Cedar Parts merges two supplier lists before preparing a purchasing dashboard. Staff first define the target entity as a legal supplier, with branch addresses retained separately. Exact trusted identifiers establish straightforward links, while uncertain name-and-address combinations enter a review queue.
Reviewers discover two similarly named suppliers at different addresses and one supplier that changed its trading name. They preserve the first pair as separate entities and document the evidence supporting the second link. A sample of unlinked records is also examined for missed matches.
The team stores the crosswalk between source records and linked entities rather than silently overwriting the source lists. This fictional process illustrates traceability; it does not claim a measured improvement.
Common misconceptions and mistakes
Text similarity is not identity. Two people can share a name, and one organization can have several legitimate names.
A high overall match rate is not proof of quality: a system can increase that rate by accepting incorrect pairs. Equally, very strict rules can produce attractive precision while missing many true links.
Blocking is another common blind spot. If a true pair never becomes a candidate, a perfect final classifier cannot recover it in that pass. Also avoid forcing a one-to-one relationship unless the data and entity definition justify it. Duplicate records and changing organisational structures can make that assumption inappropriate.
Business use
Record linkage can improve reporting consistency, data integration, and the preparation of training data. Its business value depends on whether linked identities are dependable enough for the intended use.
Define acceptable error consequences before choosing thresholds. A dashboard count and an operational account merge may require different review policies. Retain linkage evidence, correction routes, and versioned crosswalks so downstream users can trace changes. Limit access to identifying fields according to the task's actual needs rather than copying every available attribute into every output.
People also ask
Is record linkage just a database join?
An exact join can implement a linkage rule when reliable shared keys exist. Linkage addresses the harder identity decision when keys are absent, inconsistent, or uncertain.
Must it use machine learning?
No. Explicit rules, statistical methods, learned models, and manual review can all contribute.
Should uncertain records always be merged?
No. Keeping a pair unresolved can be preferable to creating a false identity connection. The appropriate action depends on purpose and evidence.
Check your understanding
A system accepts 90 pairs, of which 81 are correct, from 100 true pairs. Is its recall 90%?
Answer: No. Precision is 90%, but recall is 81%, because recall uses all 100 true pairs as its denominator.
Related terms
- Blocking
- Precision
- Recall
- entity resolution
- deduplication
- fuzzy matching
- data integration
- probabilistic matching
Learn next
design a small linkage evaluation with known matches and non-matches. Separate candidate-generation failures from decision-stage failures, then examine how changing a threshold affects both incorrect links and missed links.
More in Graphs and symbolic AI
- Reciprocity - Reciprocity measures how often directed connections are matched by connections in the reverse direction.
- Recombination - Recombination is an evolutionary-computation operation that creates candidate solutions by combining information from parent candidates.
- Reconstruction - Reconstruction, in computer vision, is the process of estimating a scene's three-dimensional structure from observations such as photographs.
- Rectangular grid - A rectangular grid is a two-dimensional arrangement of locations organized into rows and columns.
- Rectified Linear Unit - A Rectified Linear Unit, usually abbreviated ReLU, is a neural-network activation function defined by ReLU(x) = max(0, x).
- Recursive best-first search - Recursive best-first search, or RBFS, is a heuristic search algorithm that pursues promising paths while limiting memory use.
Sources
Primary sources: ONS explains deterministic and probabilistic linkage: https://www.ons.gov.uk/methodology/methodologicalpublications/generalmethodology/onsworkingpaperseries/developingstandardtoolsfordatalinkagefebruary2021 . Its Census methodology illustrates blocking and review: https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/methodologies/linkagemethodsforcensus2021inenglandandwales . Statistics Canada describes linkage planning, documentation, and validation: https://www150.statcan.gc.ca/n1/pub/12-605-x/12-605-x2017001-eng.htm . Teaching examples are original.
Previous: Reconstruction loss Next: Recordset