Must the method use a momentum encoder?

No; SimSiam is a counterexample.

Does a negative loss value imply negative example pairs?

No; negative cosine similarity can be used with positive views only.

Are labels forbidden?

No; pretraining can be self-supervised while downstream training or evaluation uses labels.

AI Glossary / Probabilistic machine learning / Advanced

Negative-free representation learning

Definition

Negative-free representation learning learns useful features without explicitly treating other examples as negative pairs that must be pushed apart. In a common self-supervised image setting, it encourages compatible representations of two transformed views of the same image. Additional architectural, optimization, or statistical mechanisms are needed to make this agreement informative rather than allowing every input to receive an identical representation.

What it means

A model can learn from relationships within unlabeled data. For example, a crop and a differently colored version of one photograph still share much of their content. Encouraging their representations to agree can teach the model which variations to ignore.
In contrastive approaches, other images often provide explicit negative examples. Negative-free approaches avoid that particular training requirement. They still need data, suitable transformations, carefully chosen objectives, and evaluation. The name does not mean there are no negative numbers in a loss, no difficult examples, or no comparisons of any kind. It identifies the absence of explicit negative example pairs in the representation-learning objective being discussed.

How it works

Create two views through transformations appropriate to the intended task. Process them through encoders and any projection or prediction heads required by the method. Train the network so that one view's output predicts or agrees with an output associated with the other view.
BYOL uses an online network with a predictor and a target network whose encoder and projector weights follow an exponential moving average of their online counterparts. The target outputs are treated as constants during backpropagation. SimSiam instead uses a shared encoder, a predictor, and stop-gradient on the target side of each directional comparison, without requiring a momentum target encoder.
Stop-gradient leaves the forward value unchanged while blocking gradients through a specified path. It does not freeze all learning: other paths still update the shared or online parameters. These systems' complete training arrangements matter. Simply deleting negatives from an arbitrary contrastive objective does not reproduce them, and stop-gradient alone is not a universal guarantee against collapse. Other negative-free families can use different mechanisms.

Three practical examples

A manufacturer pretrains an image encoder on unlabeled component photographs before fitting a classifier with a smaller labeled set. Crops must retain information relevant to component identity; a transformation that removes a distinctive connector could teach the wrong invariance.
A wildlife archive studies whether augmented views can improve an encoder for later image organization. Different species should remain distinguishable, even though the pretraining objective does not explicitly label other photographs as negatives.
A document-image project explores representations that tolerate scanner brightness changes. It should not automatically encourage invariance to rotations or edits that alter the document's meaning. The transformation policy is part of the learning problem, not a harmless preprocessing detail.

Think of it this way

Imagine recognizing the same object in two photographs without being handed a list of objects it must differ from. Agreement is useful, but responding “object” to every photograph would also be consistent. A successful learning system must preserve useful distinctions while becoming insensitive to selected changes in appearance.

Worked example

Consider one directional comparison between an already normalized prediction vector p = (0.6, 0.8) and an already normalized target vector z = (1, 0). Both have length one. Their cosine similarity is 0.6 × 1 + 0.8 × 0 = 0.6.
A normalized squared-distance loss gives (0.6 − 1)² + (0.8 − 0)² = 0.16 + 0.64 = 0.8. The equivalent expression 2 − 2 × cosine also gives 0.8. A negative-cosine objective assigns −0.6 to the same comparison. These are different numerical scales for the same directional alignment preference, not evidence that one model is better.
This calculation covers one direction only. SimSiam averages its two directional negative-cosine terms, while the original BYOL formulation sums two normalized squared-distance terms. Averaging across training examples and other implementation choices must also be stated when comparing reported losses.
Now suppose every prediction and every target, for every input, becomes the same unit vector (1, 0). Every squared-distance comparison is zero and every negative-cosine comparison is −1. Yet the representation has zero variation across inputs and cannot distinguish them. A minimum alignment loss therefore does not, by itself, establish successful representation learning. The example illustrates why collapse monitoring and downstream evaluation are necessary; it does not simulate a trained BYOL or SimSiam system.

Illustrative case study

Fictional company Willow Components proposes pretraining an image encoder using a negative-free method. Its team compares a complete published training recipe against training from random initialization under the same labeled-data budget. It reserves separate physical components for evaluation to avoid near-duplicate photographs making the task artificially easy.
During training, the team records variation across representation dimensions and checks whether nearest neighbors remain meaningful. Later, it evaluates a classifier and retrieval task on held-out images. A falling alignment loss alone is not accepted as success. This is an invented evaluation plan; no improvement in accuracy, labeling cost, or production performance is claimed.

Common misconceptions and mistakes

Negative-free does not mean supervision-free in every sense: transformations, architecture, and training design provide learning signals and assumptions. It also does not mean the model never uses information shared across a batch, since normalization and other operations may do so.
Another mistake is assuming a target network never changes. In BYOL its parameters change through the moving average, rather than through target-side gradients. Finally, preventing complete constant-output collapse does not guarantee that every feature dimension is informative or that the learned invariances suit the downstream task.

Business use

Consider negative-free pretraining when substantial unlabeled data are available and a learned representation could support later prediction or search. Compare total training cost, labeled-data needs, and performance under realistic deployment conditions. Preserve a clear record of the augmentations and evaluation split, because these choices can affect usefulness as much as the method's name.

Check your understanding

What is the squared-distance loss in the example?

Answer: 0.8.

Why can zero loss be unhelpful?

Answer: All inputs might share one constant representation.

Does stop-gradient prevent every network parameter from updating?

Answer: No; it blocks only the specified gradient path.

Related terms

Self-supervised learning
Representation collapse
Contrastive Learning
BYOL
SimSiam
stop-gradient
momentum encoder

Learn next

Compare complete BYOL and SimSiam training procedures, then study representation variance and downstream evaluation alongside alignment loss.

Sources

Grill and colleagues, Bootstrap Your Own Latent (2020), Section 3: https://proceedings.neurips.cc/paper/2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf . Chen and He, Exploring Simple Siamese Representation Learning (2020 preprint; CVPR 2021): https://arxiv.org/pdf/2011.10566 . The vector calculation and Willow Components scenario are original illustrations. Both primary objectives independently consulted; directional scaling verified.

Previous: Negative class Next: Negative relationship