AI Glossary / Deep learning / Intermediate

Contrastive Learning

Definition

Contrastive learning trains a representation by comparing examples and encouraging relationships that match a defined notion of similarity. A common approach brings representations of positive pairs closer while distinguishing them from negative examples. Positive pairs may be different views of the same item, related items, or examples sharing a label. The choice of what should count as similar is central: the model learns the relationships supplied by the training design.

What it means

Instead of asking only “Which class is this?”, a contrastive task may ask “Which other example belongs with this one?” The model learns an embedding, a numeric representation in which these relationships can be measured.

For images, two suitable transformations of the same photograph can form a positive pair. In a supervised version, labels can define multiple positives from the same class. Contrastive learning is therefore not exclusively unsupervised. The resulting representation may support classification or retrieval, but its usefulness for those tasks must be measured separately from success at the comparison task.

How it works

An encoder maps each input to a vector. A similarity measure compares vectors, and a loss rewards the desired relationships. Many formulations use a temperature parameter to control how strongly score differences affect the loss.

SimCLR, for example, learns from differently augmented views of images and applies its contrastive loss after a projection network. Supervised contrastive learning uses label information to define positive relationships across examples.

The details vary across methods: how pairs are formed, which negatives are used, how examples are sampled, and which representation is retained. Avoid assuming that every self-supervised method is contrastive or that every contrastive system must use the same batch size, augmentation, or architecture.

Three practical examples

1. Product-image retrieval: a team learns representations intended to keep different suitable views of the same product close, then tests whether users retrieve the correct item among visually similar alternatives.

2. Document matching: researchers train on known related document pairs and evaluate whether the learned representation supports finding relevant material outside the training collection.

3. Label-efficient image study: students pretrain on paired image views, then measure how well a small labeled dataset can train a classifier using the learned representation.

Think of it this way

Think of arranging cards on a table so cards that belong together are nearby and others are distinguishable. The arrangement can become useful without writing a detailed description of every card. The analogy is limited: the computer's space can have many dimensions, and “belong together” is defined by the training procedure. A bad pairing rule can organize the cards neatly around the wrong property.

Worked example

Consider one anchor example, one positive, and two negatives. Suppose their similarity scores to the anchor are 0.8, 0.2, and 0.1. With temperature 0.5, dividing by temperature gives 1.6, 0.4, and 0.2.

Exponentiating gives approximately 4.953, 1.492, and 1.221. The positive's normalized share is 4.953/(4.953 + 1.492 + 1.221), approximately 0.646. A simple contrastive cross-entropy loss for this comparison is −ln(0.646), approximately 0.437.

If the positive score were only 0.2 while the negative scores stayed unchanged, its share would become 1.492/(1.492 + 1.492 + 1.221), approximately 0.355. The loss would rise to about 1.036.

These stipulated scores illustrate why the objective rewards distinguishing the positive. They are not measured model results or a complete training batch. The normalized share is part of this comparison loss, not automatically a calibrated probability that two real-world items are semantically equivalent.

Illustrative case study

Fictional case: Pair Harbour trains an image representation using random crops as positive views. The intended task is to distinguish packages whose identities differ mainly through a small printed symbol. Some crops remove that symbol, so training encourages agreement between views that omit important identity information.

The team reviews its positive-pair policy and tests retrieval on near-identical packages. It chooses transformations according to which properties the application should preserve, rather than copying an augmentation recipe without examination. No accuracy improvement is claimed. The case illustrates how a seemingly reasonable similarity rule can conflict with the distinctions needed downstream.

Common misconceptions and mistakes

Contrastive learning does not mean increasing photographic contrast. It concerns comparisons within a learning objective.

Positive and negative describe relationships for the task, not good and bad data. An example can be a positive for one anchor and a negative for another.

Different source items are not always true negatives. They may share the meaning or category the application wants to preserve, creating misleading training pressure.

Finally, a low training loss does not establish useful retrieval or classification. A model may exploit shortcuts in the pair construction instead of learning the intended relationships.

Business use

The main business question is whether useful relationships can be specified more easily than detailed labels. Paired records, multiple views, or existing category information may support representation learning, but their quality needs inspection.

Define a downstream evaluation before investing in training. Include difficult near-matches and cases where superficial similarity is misleading. Record the pairing and sampling rules so model behavior can be traced to the intended similarity definition. Compare against a suitable existing representation, accounting for training cost, retrieval quality, and maintenance requirements rather than assuming contrastive pretraining is automatically beneficial.

People also ask

Does contrastive learning require class labels?

Not always. Some methods construct pairs from transformations or other relationships; supervised variants deliberately use labels.

Are all self-supervised methods contrastive?

No. Self-supervision also includes objectives that do not use this positive-versus-negative comparison structure.

Why do augmentations matter?

They specify which changes the representation should tolerate. Removing a task-critical distinction can teach an unsuitable invariance.

Check your understanding

Two different catalog photographs show the same product. Is treating them as negatives always appropriate?

Answer: No. If the task should recognize product identity across photographs, they may be a false-negative pair. Pair construction should reflect the relationship the application needs.

Related terms

• Embedding
• Self-supervised learning
• Similarity measure
• Temperature
• positive pair
• negative sample
• SimCLR
• supervised contrastive learning

Learn next

calculate the worked loss with a different temperature and observe how the positive share changes. Then write a pairing policy for a concrete task, listing transformations that preserve identity and transformations that could erase important distinctions.

• Temperature

More in Deep learning

• Confabulation (neural networks) - Confabulation in neural-network discussions can refer to generated patterns that resemble learned material but are not faithful reconstructions of a...
• Continual Learning - Continual learning is the study and practice of enabling a model to learn from successive experiences while retaining useful abilities acquired earlier.
• Contrastive Hebbian learning - Contrastive Hebbian learning is a neural learning approach that adjusts connections using differences between activity relationships in two network states.
• Convolutional deep belief network - A convolutional deep belief network, or CDBN, is a hierarchical probabilistic model that combines deep belief network ideas with convolutional weight sharing.
• Convolutional layer - A convolutional layer is a neural-network component that applies learned filters across local regions of an input.
• Convolutional neural network - A convolutional neural network, or CNN, is a neural network that uses convolutional layers to learn patterns through local connections and shared filters.

Sources

Chen and colleagues, A Simple Framework for Contrastive Learning of Visual Representations: https://arxiv.org/html/2002.05709v3 — primary text consulted for paired views, encoder, projection head, and temperature-scaled loss.

Khosla and colleagues, Supervised Contrastive Learning: https://arxiv.org/html/2004.11362v5 — primary text consulted for label-defined positive relationships. Examples, loss calculation, and fictional case are original teaching material. This entry covers common comparison-based representation learning rather than every method bearing the word contrastive.

Previous: Contrastive Hebbian learning Next: Contrastive search