AI Glossary / Computer vision and multimodal AI / Intermediate
Hays - Efros scene completion
Definition
Hays - Efros scene completion is a research approach that fills missing image regions by retrieving compatible content from a very large photograph collection. Introduced in Scene Completion Using Millions of Photographs, it searches for plausible scene fragments and blends selected candidates into the incomplete image.
What it means
A hole in an image may be too large or structurally complex to fill using nearby texture alone. A missing section of a beach, street or mountain scene can require content that is absent from the remaining image. Hays and Efros explored whether a large external photo collection could provide suitable fragments. The method is data-driven through retrieval, rather than a modern text-prompted generative model. It can offer several plausible completions for a user to choose from. Those alternatives are visual inventions assembled from other images, not recovered evidence of what the hidden part of the original scene actually contained.
How it works
The system searches a large database for photographs whose broad scene appearance is compatible with the visible query image. Candidate content is evaluated near the missing region so that both scene-level plausibility and local compatibility matter. Suitable fragments are positioned and blended to reduce seams. The retrieval stage expands the available source material beyond the original photograph.
Database coverage is central: a large collection increases opportunities for matching but does not guarantee the needed viewpoint, geometry or object arrangement. Local blending can hide a boundary while leaving a globally implausible result. The original work is therefore best understood as a particular retrieval-and-compositing research system, with user choice among candidates, rather than a universal scene-recovery algorithm.
Three practical examples
- A graphics researcher studies how external-image retrieval can fill a large gap that simple texture propagation cannot handle convincingly.
- A designer explores alternative backgrounds for a damaged or intentionally edited scene, treating the results as creative composites.
- A computer-vision course compares retrieval-based completion with patch-based inpainting and learned generation to clarify their different sources of image content.
Think of it this way
Think of repairing a torn landscape collage by searching a large box of photographs for a piece with a compatible horizon and texture. The box provides possibilities that the remaining picture does not contain. The analogy is close to the retrieval idea, but the actual system uses numerical representations, automated ranking and blending rather than only visual browsing with scissors.
Worked example
Suppose a photograph has a missing rectangle covering part of a beach and horizon. Candidate A contains similar sand texture but places the horizon too high. Candidate B has a compatible horizon and sea color but introduces a distant boat. Candidate C matches the broad layout yet creates a visible lighting seam. A retrieval-and-completion workflow evaluates scene layout and local compatibility, blends promising candidates and presents alternatives for review.
This decision example does not assign invented scores to the published algorithm. It demonstrates why the nearest texture match is not necessarily the best scene completion, and why a seamless candidate can still add content, such as the boat, that was never observed in the original scene.
Illustrative case study
Fictional example: Cedar Design tests retrieval-based completion on an incomplete travel image. Several candidates produce plausible coastlines, but each places different details in the missing region. The team selects one for a clearly edited promotional composition and keeps the original image. It does not describe the result as restoration of an exact historical view. The experiment highlights the method's ability to propose alternatives and the editor's responsibility to understand what those alternatives represent.
Common misconceptions and mistakes
The method does not reconstruct hidden ground truth simply because it finds a visually convincing match. A huge database does not guarantee a suitable candidate. Semantic compatibility in the research sense does not mean the system has complete human understanding of the scene. Blending cannot repair every perspective or lighting mismatch. The approach is distinct from using only patches inside the same image and from training a generative model to synthesize new pixels. Finally, access to source photographs does not by itself settle the rights needed for a production composite.
Business use
Businesses can learn from this work when designing visual retrieval and editing workflows. Evaluate both the quality of candidate search and the effort needed to review and blend results. Maintain provenance for external source content and use material appropriate to the intended publication rights. Distinguish creative completion from documentary reconstruction. For technical comparisons, report database scope and test conditions, since retrieval quality depends strongly on available examples. A useful workflow gives reviewers control over alternatives rather than presenting one plausible result as uniquely correct.
People also ask
Does the method use only the incomplete image?
No. Its defining idea is to retrieve compatible material from a large external photograph collection.
Is it the same as modern prompt-based generative fill?
No. The original method is based on image retrieval and compositing, with a different source of candidate content.
Why offer several completions?
Missing content is ambiguous, and multiple retrieved fragments can produce plausible but different scenes.
Check your understanding
Two retrieved fragments both make a missing region look natural. Does that mean one is the recovered original?
Answer: No. Both are possible visual completions. The method does not establish which content, if any, occupied the original hidden region.
Related terms
- Image retrieval
- Content-aware fill
- Semantic similarity
- scene completion
- image inpainting
- image compositing
- patch matching
Learn next
Learn the difference between texture synthesis and scene-level completion. Then study how global image retrieval, local compatibility and blending contribute separately to a convincing result, while preserving provenance and ambiguity.
More in Computer vision and multimodal AI
- Harris corner detector - The Harris corner detector identifies image locations where brightness changes strongly in more than one direction.
- Hartley transform - The Hartley transform is a real-valued transform closely related to the Fourier transform.
- Hausdorff distance - Hausdorff distance measures the greatest nearest-neighbor separation between two sets of points.
- HDR (high dynamic range) - HDR, or high dynamic range, describes imaging representations or systems that handle a broad range between dark and bright values.
- HDR merging - HDR merging combines multiple measurements of the same scene to produce an image representation with useful information across a wider brightness range.
- HDR-VDP - HDR-VDP is a family of visual-difference prediction methods designed to estimate how people perceive differences between images across a broad luminance range.
Sources
James Hays and Alexei Efros, Scene Completion Using Millions of Photographs, authors' project: https://graphics.cs.cmu.edu/projects/scene-completion/ Primary paper: https://graphics.cs.cmu.edu/projects/scene-completion/scene-completion.pdf Primary project and paper excerpts consulted; full implementation and editorial review remain pending. The candidate comparison is an original conceptual example.
Previous: Hawthorne effect Next: Hazard rate