AI Glossary / Computer vision and multimodal AI / Advanced

Content-based image retrieval

Definition

Content-based image retrieval, or CBIR, finds images using information extracted from their visual content. A query image or a description of desired visual features is compared with stored image representations. The results are ranked by a chosen similarity rule rather than relying solely on filenames, captions or manually assigned keywords.

What it means

Imagine having a picture of a patterned tile and wanting to find visually related tiles in a catalog. The search needs to compare what the images contain, not just whether someone wrote the same product description. However, “related” needs a clear meaning: similar color, pattern, shape and product identity are different objectives. A useful retrieval system reflects the user's objective and lets the user inspect the evidence. A high similarity score is not automatically a statement that two products are interchangeable.

How it works

A system extracts features, stores their representations in a searchable index and applies a compatible comparison method to each query. Features may describe color, texture or shape; learned representations provide another option. The system ranks candidate images and returns a selected set. Indexing can reduce the work required to search a large collection. The representation and comparison rule jointly determine what counts as similar. IBM's QBIC research is an early example of organizing image search around visual features. Modern implementations should be evaluated for their own task and collection.

Three practical examples

• A furniture retailer lets a customer submit a chair photograph and review catalog items with visually related designs, while displaying actual dimensions separately.
• A design team searches a licensed image library for photographs with a similar color arrangement to a reference, then checks subject relevance and usage rights.
• A maintenance team retrieves visually comparable equipment photographs to support investigation. A technician decides whether the retrieved items actually depict the same component or fault.

Think of it this way

Think of arranging fabric swatches by appearance. You could put similar colors together, then sort by pattern size. Someone seeking blue material would appreciate the first arrangement, while someone seeking a particular weave might need another. An image search system makes comparable choices through its features and similarity rule. Its ordering is useful only when those choices fit the searcher's question.

Worked example

In an original simplified example, represent images by two normalized visual features. A query has vector q=(1,0). Candidate A has vector (0.8,0.6), and candidate B has vector (0.6,0.8). All three vectors have length one, so their cosine similarities with q equal their dot products: A scores 0.8 and B scores 0.6. A ranks first under this rule. Suppose the first feature mainly represents a color preference: A may be a better color match while B is the correct product type. The arithmetic is valid, but the ranking objective may be wrong. Real image representations are more complex, and these numbers are not probabilities of product identity.

Illustrative case study

A fictional homeware business tests visual search on its own catalog. Staff prepare queries photographed under several lighting conditions and label which catalog results would satisfy each shopping request. They discover that the first prototype often retrieves similarly colored objects from the wrong category. The team adds category filtering and evaluates whether shoppers can locate suitable products in the first few results. It also checks newly added catalog images and unusually cropped photographs. The team reports task-specific retrieval findings rather than presenting visual resemblance as proof of identical materials, dimensions or availability. No performance improvement is asserted by this illustrative scenario.

Common misconceptions and mistakes

CBIR is not limited to exact duplicate detection. Conversely, a visually similar result need not depict the same object. A similarity score should not be relabeled as a confidence percentage without a justified interpretation. Another mistake is judging a system from a few attractive examples while ignoring difficult queries. Cropping, viewpoint and backgrounds can change what the representation emphasizes. Search results also inherit collection limitations: an item that is absent cannot be retrieved. Image rights and product facts require their own records; they do not follow from resemblance.

Business use

Potential applications include catalog discovery, media organization and assisted visual investigation. A business should define the desired result before selecting a representation: finding the same item, suggesting alternatives and matching a mood board require different evaluation labels. Useful measures might include relevant results among the first several returned and the time users need to complete a search. Operational planning should cover catalog updates, image quality and the cost of reviewing incorrect matches. Keep visual ranking separate from commercial attributes that require verified product information.

People also ask

Does CBIR need written image labels?

It can compare visual representations without relying solely on labels, although metadata can improve a practical search workflow.

Is every returned image a duplicate?

No. Similarity can refer to broader visual properties.

Can it find something outside the collection?

Retrieval returns available indexed candidates; missing items require another source or collection update.

Check your understanding

Why did candidate A rank above B?

Answer: Its cosine similarity to the query was larger under the stipulated representation.

Why might B still be more useful?

Answer: Product relevance may not match the feature being emphasized.

What should a business define first?

Answer: The practical meaning of a successful retrieval for its users.

Related terms

• Similarity Search
• Feature extraction
• Nearest neighbor search
• Semantic gap
• Image embedding
• image classification

Learn next

Create a small image collection and write three different retrieval goals for the same query picture. Compare how color, shape and product category would change the ideal ordering. Then design relevance labels before evaluating a larger search system.

More in Computer vision and multimodal AI

• Constrained least squares - Constrained least squares finds model parameters that minimize a sum of squared residuals while obeying specified restrictions.
• Constraint satisfaction - Constraint satisfaction is the task of assigning values to variables so that all specified restrictions are met.
• Content-aware fill - Content-aware fill is an image-editing technique that replaces a selected region with content chosen or synthesized to fit the surrounding image.
• Content-preserving warp - A content-preserving warp is a spatially varying image transformation that balances desired movement against distortion of visible structures.
• Context encoders - Context encoders are neural networks trained to predict missing image regions from the surrounding visible content.
• Contrast-detection autofocus - Contrast-detection autofocus adjusts a camera's focus using image-derived contrast or sharpness measurements.

Sources

IBM Research, The Query By Image Content (QBIC) System: https://research.ibm.com/publications/the-query-by-image-content-qbic-system IBM Research, Query by image content using multiple objects and multiple features: https://research.ibm.com/publications/query-by-image-content-using-multiple-objects-and-multiple-features-user-interface-issues The vectors, commercial examples and homeware case are original educational illustrations. Historical references establish the feature-based retrieval context, not the performance of a modern product.

Previous: Content-aware fill Next: Content determination