AI Glossary / Natural language and speech / Advanced

Unlikelihood training

Definition

Unlikelihood training adds an objective that lowers the probability of selected undesirable outputs. For a negative candidate with model probability p, the basic penalty is −log(1 − p), which becomes larger when the model assigns that candidate more probability.

What it means

Maximum-likelihood training rewards observed target text. Unlikelihood training supplements that positive signal with an explicit instruction, expressed mathematically, to avoid particular candidates in particular contexts.

In the cited work on text degeneration, negative candidates identify unwanted repetition. A token-level version uses previous context tokens except the correct next token. A sequence-level version constructs penalties from repeated sequences in model-generated continuations.

The word undesirable depends on the candidate-selection rule. The loss does not discover that rule by itself, and a token that is undesirable in one context may be necessary in another. This is a training intervention that changes model parameters, distinct from applying a repetition penalty only while decoding.

How it works

For each training context, identify a set of negative candidate tokens. Calculate −log(1 − p(c | context)) for each candidate c and sum the penalties.

Combine that unlikelihood term with the usual negative log probability of the correct target, using a weight to control its influence. Minimizing the combined loss encourages the target while discouraging the selected alternatives.

In the original token-level repetition setup, the true next token is removed from the negative set even if it occurred earlier. This avoids simultaneously rewarding and penalizing that target under the same selection rule.

For sequence-level training, the model first generates a continuation, and repeated n-grams can identify positions to penalize. That adds generation cost and changes the contexts on which the model learns. The negative-selection policy and loss weighting therefore deserve as much attention as the formula.

Three practical examples

• A language-model experiment adds penalties for incorrect token repetitions drawn from the preceding context.
• A fine-tuning run generates continuations and penalizes tokens participating in repeated n-grams.
• An evaluation checks that reduced repetition has not damaged necessary reuse of names, terms or structured labels.

Think of it this way

Think of a writing coach who provides both a good example and specific notes about an unwanted habit. The learner receives a signal about what to produce and another about what to avoid.

The analogy explains complementary supervision. A numerical loss cannot decide whether the coach's criticism is sensible. If the negative examples are poorly chosen, training can suppress wording that the task actually requires.

Worked example

Suppose a correct next token has probability 0.6, while one selected negative candidate has probability 0.3. The remaining vocabulary shares probability 0.1. Using natural logarithms, the target loss is −log(0.6), approximately 0.511.

The unlikelihood penalty is −log(1 − 0.3) = −log(0.7), approximately 0.357. With weight 0.5, the combined loss is 0.511 + 0.5 × 0.357, approximately 0.689.

For comparison, consider a different valid probability distribution with the same target probability 0.6 but negative probability 0.1. The unlikelihood term becomes −log(0.9), approximately 0.105, and the combined loss becomes approximately 0.564.

This shows how the objective distinguishes two distributions with the same target probability. It does not predict the exact effect of one optimizer step, because neural parameters couple many outputs and training examples.

Illustrative case study

The fictional PineParagraph team studies repeated phrases in generated summaries. It defines a negative-candidate rule and checks examples before launching a fine-tuning experiment.

Some repeated phrases are unwanted loops, while others are product names that must remain consistent. Reviewers therefore include legitimate repetition in the evaluation set and inspect whether the selection rule penalizes the correct target.

The team compares repetition, factual consistency, fluency and held-out predictive quality separately. This invented case illustrates the need for a sound negative-example policy. It reports no measured benefit and does not suggest that unlikelihood training is a complete solution to factual errors.

Common misconceptions and mistakes

• Unlikelihood means training the model to produce unlikely text. It lowers probabilities of specifically selected negative candidates while retaining positive learning signals.
• Every repeated token should be penalized. Repetition can be correct or necessary, so context and target handling matter.
• It is just a decoding setting. The method updates model parameters through an additional training objective.

Business use

Unlikelihood training is useful when an unwanted output pattern can be identified reliably enough to provide training feedback. It makes the negative signal explicit and testable.

Document how candidates are selected, whether they come from reference contexts or generated continuations, and how the penalty is weighted. Inspect false positives in the negative set. Compare against the original model using consistent decoding settings and assess qualities beyond the targeted pattern. A lower repetition rate can coexist with worse wording or missing information. Keep the scope precise: the loss penalizes selected probabilities, not all possible forms of undesirable behavior.

People also ask

Does the penalty grow when a negative candidate becomes more likely?

Yes. As p approaches one, −log(1 − p) becomes very large.

Can unlikelihood be combined with maximum likelihood?

Yes. The cited token-level objective adds a weighted unlikelihood term to ordinary target likelihood training.

Is it the same as negative sampling?

No. Negative sampling is a different objective family; the presence of negative candidates alone does not make the methods identical.

Check your understanding

A negative candidate's probability falls from 0.4 to 0.2. Does its unlikelihood penalty increase or decrease?

Answer: It decreases, from −log(0.6), approximately 0.511, to −log(0.8), approximately 0.223.

Related terms

• Cross-entropy
• Negative sampling
• Maximum likelihood
• negative candidate
• repetition
• sequence-level training
• decoding penalty
• language-model fine-tuning

Learn next

Read maximum likelihood and decoding-time repetition penalties. Compare where each intervention acts, then examine a negative-candidate rule on examples containing both unwanted loops and legitimate repeated terminology.

More in Natural language and speech

• Two-stream self-attention - Two-stream self-attention is XLNet's mechanism for maintaining separate content and query representations at token positions.
• Unicode - Unicode is a standard for representing and processing text across writing systems.
• Unit production - A unit production is a context-free grammar rule whose right-hand side consists of exactly one nonterminal symbol, such as A→B.
• Unvoiced sound - An unvoiced sound, also called a voiceless sound, is produced without the regular vocal-fold vibration that supplies voicing.
• UTF-8 - UTF-8 is a Unicode encoding that represents each Unicode scalar value using one to four bytes.
• Utterance - An utterance is a particular instance or segment of language produced in communication, commonly a stretch of speech by a speaker in context.

Sources

Sean Welleck and colleagues, Neural Text Degeneration with Unlikelihood Training: https://arxiv.org/html/1908.04319v2

The original token-level loss, exclusion of the true target and sequence-level repetition construction were reviewed. Numerical loss comparisons and the fictional review case are original. Historical experiment findings are not presented as universal guarantees about model quality.

Previous: Unlabeled example Next: Unmatched count