Why are both directions necessary?

A result can be locally plausible while omitting source content, or retain source content while adding unsupported artifacts. The directions address those different failures.

Does it require every patch to match a unique patch?

Not generally. Nearest-match objectives can reuse matches, so the relationship is not automatically one-to-one.

Does it understand which object matters most?

Not by definition. Importance must come from the representation, weights, constraints or additional semantic information.

AI Glossary / Computer vision and multimodal AI / Advanced

Bidirectional similarity

Definition

Bidirectional similarity is an image-comparison objective that checks patch matches in both directions between a source and a result. It encourages the result to retain source information while also ensuring that its own local content can be supported by the source.

What it means

A smaller version of an image can look plausible while omitting an important feature. Conversely, a result can preserve several source features but arrange them into visually implausible local structures. Bidirectional similarity addresses these two concerns through complementary patch comparisons.

The source-to-result direction is commonly associated with completeness: source patches should find similar counterparts in the result. The result-to-source direction is associated with coherence: result patches should resemble something available in the source.

The concept was developed by Simakov, Caspi, Shechtman and Irani for visual summarization and related editing tasks. It operates on visual evidence and matching rules rather than guaranteeing semantic understanding of what a person considers most important.

How it works

The method compares local patches, potentially across multiple scales. For each source patch, it finds a sufficiently close match in the target and accumulates the mismatch cost. It also searches from each target patch back to the source and accumulates that cost.

The objective combines the two directions, often with normalization and weights. An optimization procedure changes the result to reduce the combined cost. Approximate nearest-neighbor searches and coarse-to-fine processing can make large patch-matching problems more practical.

The details matter: patch size, distance measure, scale handling and relative weights influence the result. The objective does not impose a one-to-one correspondence between every source and target patch. Repeated or rearranged content can satisfy many local matches, so global meaning and spatial relationships may still need additional constraints or review.

Three practical examples

A visual-retargeting method reduces an image's dimensions while trying to preserve representative local content.
An image-completion system assesses whether generated patches resemble available source material and remain locally coherent.
A research workflow compares editing results for missing source structures and newly introduced patch artifacts.

Think of it this way

Think of checking a summary in two passes: first ask whether important material from the original appears somewhere, then ask whether every statement in the summary is supported by the original. The analogy is limited: bidirectional visual similarity compares image patches, not meanings or factual claims, and its notion of completeness depends on the chosen representation and weighting.

Worked example

Suppose a simplified source has three patch types: brick, window and sign. A target contains only brick and window patches. Every target patch may have an excellent source match, giving a low coherence cost. However, the source sign has no good target match, so completeness suffers.

Now suppose another target contains all three source patch types but introduces a strange checkerboard patch absent from the source. Completeness may improve, but the unsupported target patch raises the coherence cost.

For a toy numerical example, let average completeness mismatch be 4 and average coherence mismatch be 1. With equal weights, their summed objective is 5. Changing the weights changes that trade-off. These invented costs illustrate the two directions rather than reproducing a particular implementation's exact normalization.

Illustrative case study

Fictional example: Patchwork Gallery experiments with reducing wide exhibition photographs for a narrow digital display. A first patch-based result preserves believable wall and floor texture but loses a small sign that helps viewers understand the scene.

The team examines source-to-result matches and adjusts the editing constraints so the sign remains represented. It then checks the reverse direction for unusual local combinations introduced by the change. Human reviewers still assess whether the final arrangement communicates the intended exhibition layout.

The gallery retains the original and marks the result as an adapted composition. This fictional case shows why local visual similarity and meaningful communication require separate consideration.

Common misconceptions and mistakes

Bidirectional similarity is not simply comparing an image twice with the same scalar formula. The two directed searches penalize different kinds of failure.

Completeness does not guarantee that every object remains in its original location or size. Coherence does not guarantee that the whole scene is physically possible. A method can produce locally familiar patches in a globally misleading arrangement. The objective also differs from semantic embedding similarity, which may compare learned representations of overall meaning. Neither should be substituted for the other without examining the task.

Business use

The approach is useful for understanding content-aware resizing, patch-based editing and visual-synthesis objectives. It makes an explicit trade-off between retaining source material and avoiding unsupported local appearance.

For business use, identify content that must remain intact, such as text, logos or spatial relationships, and supply additional constraints where needed. Review the final output for both omissions and introduced artifacts. Record patch settings and source material so results can be reproduced. A low objective value should be treated as evidence about the selected visual measure, not a universal quality certificate.

Check your understanding

A target uses only texture patches already found in the source but removes a distinctive sign. Which direction is most directly responsible for noticing the missing sign?

Answer: The source-to-target completeness direction, because the source sign patch needs an adequate counterpart in the result.

Related terms

Completeness
Image retargeting
Nearest neighbor search
Patch matching
visual summarisation
coherence
image completion
content-aware editing

Learn next

Create a simple source collage with three distinct patch types. Compare a target that omits one type with a target that adds an unfamiliar type. Explain the effect on each directed cost separately. Then consider a rearrangement that preserves all local patches but changes the scene's meaning, revealing the objective's limits.

Sources

Simakov, Caspi, Shechtman and Irani, Summarizing Visual Data Using Bidirectional Similarity, original project: https://www.wisdom.weizmann.ac.il/~vision/VisualSummary.html Original paper: https://projectsweb.cs.washington.edu/research/VACE/VisionResearchGroup/cvpr08/502.pdf The patch types, costs and gallery case are original illustrations. No semantic-preservation or global-optimality guarantee is claimed.

Previous: Bidirectional search Next: Bidirectional Transformer Language Model