Does this method require a trained neural network?

The original Criminisi approach uses patch sampling and a designed priority rule; it does not require neural-network training.

Where do the replacement pixels come from?

They are copied from the permitted source region, usually selected from the available image content.

Can it reconstruct a unique missing face accurately?

Visual filling alone cannot establish a person's original hidden appearance, especially when no suitable examples exist.

AI Glossary / Computer vision and multimodal AI / Intermediate

Exemplar inpainting (Criminisi)

Definition

Exemplar inpainting, in the Criminisi approach, fills a selected missing or unwanted image region by copying suitable patches from an available source region. A priority rule combines confidence in surrounding pixels with the strength and direction of image structure. This guides the order of filling so that texture and boundaries can extend into the hole together.

What it means

An exemplar is an existing example used to supply replacement image content. If a photograph contains a gap across a patterned wall, other visible parts of that wall may provide useful patches.

The method does more than choose a color for each empty pixel. It considers small neighborhoods, their known content, and their position along the current boundary of the hole. The order matters because newly filled areas become context for later choices.

The reconstructed area is inferred content. A convincing wall continuation does not establish what was hidden behind an object. This matters when an image serves as a factual record.

How it works

Begin with an image, a target mask identifying the region to fill, and a permitted source region containing example patches. Establish confidence values for known pixels and identify the fill front, the boundary between known and unknown content.

For patches centered on that front, compute a priority from a confidence term and a data term. Confidence reflects how much reliable surrounding information supports the patch. The data term encourages continuation of strong image structure into the gap.

Select the highest-priority target patch. Search the source region for a patch that best matches its already known pixels, then copy source values into the unknown positions. Update confidence and the fill front before repeating.

Early decisions change later context. Patch size, examples, and target geometry influence the result.

Three practical examples

1. Photo retouching: An editor removes a small distracting object from an illustrative image of a textured surface, using nearby material to fill the selected area.

2. Archive presentation: A conservator creates a clearly labeled viewing copy with a scratch filled, while preserving the unaltered scan for historical and analytical purposes.

3. Training demonstration: A computer-vision instructor masks part of an image containing a straight boundary and compares structure-guided filling with a simple inward filling order.

Think of it this way

Think of repairing a patterned quilt with pieces taken from spare matching fabric. A good patch must agree with the surrounding pattern, and repairing an important stripe first may help align later pieces. The analogy stops at physical authenticity: image patches are copied, and their visual plausibility does not reveal the original hidden scene.

Worked example

Suppose two patches lie on the fill front. Patch A has confidence 0.8 and data term 0.4, giving priority 0.8 × 0.4 = 0.32. Patch B has confidence 0.5 and data term 0.9, giving priority 0.45. Patch B is selected despite its lower confidence because its structure term is stronger.

For a simplified matching step, imagine that three known grayscale values in B are 10, 20, and 30. Source candidate X has corresponding values 11, 19, and 30. Its sum of squared differences is 1 + 1 + 0 = 2.

Candidate Y has values 10, 22, and 33, giving 0 + 4 + 9 = 13. Under this simplified distance calculation, X is the better match. Values from X are copied only into B's unknown positions; the known target values remain intact.

These small numbers isolate the two decisions: which patch to fill next and which source patch to use. A real implementation handles full image patches, color information, gradients, confidence updates, and boundary conditions. A low matching error still does not guarantee that copied content is semantically appropriate.

Illustrative case study

Fictional case study: A museum prepares an online display using a scan of a decorative poster. A tear interrupts both a flat colored region and a printed border. The digitization team retains the original scan and makes a separate restoration draft.

An exemplar-based fill continues much of the border convincingly, but it also repeats a distinctive ornament. The editor narrows the allowed source region and reviews the repaired section at full resolution before accepting the display copy.

The published description identifies the image as digitally reconstructed and keeps the archival version available internally. The exercise shows why patch matching and human review can complement each other; it does not claim that the algorithm recovered the poster's exact lost design.

Common misconceptions and mistakes

Exemplar inpainting is not the same as recovering original hidden pixels. Several plausible completions may fit the visible boundary.

Confidence in the algorithm is a propagated quantity used for filling decisions, not a calibrated probability that a reconstruction is historically true.

A high-priority patch is selected because of the combined rule, not simply because it has the strongest edge or the most known pixels alone.

Larger patches are not universally better. They may capture broader structure but make suitable source matches harder to find. Repetition, perspective changes, and missing unique objects can expose the method's limits.

Business use

The approach is useful for controlled restoration and retouching workflows where the surrounding image contains suitable material. It also provides an understandable baseline for studying image completion before introducing learned generative methods.

Evaluation should cover visible seams, repeated patterns, structural continuity, operator correction time, and whether the output meets the intended editorial purpose. Keep the source image, target mask, and approved reconstruction together. For images used as evidence or records, ensure that an altered presentation copy cannot be mistaken for the unmodified capture.

Check your understanding

Patch C has confidence 0.6 and data term 0.7. Patch D has confidence 0.9 and data term 0.4. Which has higher priority?

Answer: C, because 0.42 exceeds 0.36. The comparison concerns filling order, not proof of reconstruction accuracy.

Related terms

Texture synthesis
Image patch
Isophote
Image gradient
Image inpainting
exemplar-based learning
image restoration

Learn next

Study patch similarity and image gradients, then trace a few iterations on a small masked image. Compare results when the source region contains a matching texture but lacks the structure needed to bridge the gap.

Sources

Criminisi, Pérez, and Toyama, Region Filling and Object Removal by Exemplar-Based Image Inpainting, original paper: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/criminisi_tip2004.pdf

The primary paper was opened and the priority and patch-filling mechanism checked. Numerical examples and the museum scenario are original teaching illustrations.

Previous: Execution plan Next: Exhaustive decomposition