Is this the same as temperature scaling after training?

No. It changes the training loss and probability transformation through two distinct parameters.

Does it work only for binary classification?

No; the formulation extends to multiple classes.

Are both temperatures always different from one?

No. Special choices recover ordinary components or the complete standard logistic loss.

AI Glossary / Probabilistic machine learning / Advanced

Bi-tempered loss

Definition

Bi-tempered loss is a generalization of logistic classification loss that uses two temperature parameters: one changes how prediction errors are penalized, and the other changes how model scores become class probabilities. The Bregman-divergence formulation can combine a bounded loss with a heavier-tailed probability transformation to improve robustness to noisy labels.

What it means

A classifier usually converts scores into probabilities using softmax and then penalizes disagreement with the supplied label using cross-entropy. A badly mislabeled example can receive a very large loss when the model confidently predicts a different class.
Bi-tempered loss changes this training objective rather than correcting labels directly. Its two controls address different aspects of the loss and probability mapping. They offer a way to reduce sensitivity to some noisy examples, but they do not identify which labels are wrong. A difficult, correctly labeled example can also receive less influence under a robust objective.

How it works

The tempered logarithm is log_t(u) = (u^(1 − t) − 1)/(1 − t), with the ordinary logarithm recovered as t approaches one. Choosing a loss temperature t1 below one, in the usual nonnegative range, gives bounded per-example loss.
A second temperature, t2, controls a tempered exponential used to transform scores. Values above one produce a heavier tail than the ordinary exponential on the relevant negative-score side. A shared normalization offset is chosen so that the resulting class probabilities sum to one; simply dividing tempered exponentials by their sum is generally a different transformation.
The Bregman loss also contains a correction term involving powers of the predicted probabilities. Replacing only the logarithm in cross-entropy does not reproduce the full method. Setting both temperatures to one recovers ordinary softmax logistic loss. Suitable temperatures require validation for the particular data and task.

Three practical examples

1. A document-classification team could compare training objectives when historical category labels contain inconsistent annotations, evaluating on a separately reviewed test set.
2. An image-recognition researcher could deliberately corrupt a controlled fraction of training labels to study sensitivity to noise while keeping evaluation labels clean.
3. A service-ticket team could investigate whether label disagreements disproportionately affect certain categories, then compare robust training with annotation improvements and a standard baseline.
These examples are possible investigations, not claims that changing the loss necessarily improves accuracy.

Think of it this way

Imagine a coach reviewing a stack of marked exercises. One control limits how much a single disputed mark can dominate the lesson. Another changes how sharply the coach turns competing answers into confidence scores. Both controls can make training less reactive, but neither establishes whether the disputed mark is an error or an important exception.

Worked example

Consider an original two-class example with target class one and scores a1 = −9 and a2 = −1/9. Set t2 = 2. On the relevant domain, exp_2(u) = 1/(1 − u). With normalization offset zero, the transformed probabilities are 0.1 and 0.9, which already sum to one.
Ordinary softmax applied to these same scores instead gives class one probability approximately 0.000138. This illustrates that changing the transformation can materially change probabilities for fixed scores; it is not a comparison between two trained models.
Now set t1 = 0.5. For target class one and probabilities (q, 1 − q), the full loss becomes L = 2(1 − sqrt(q)) − (2/3)[1 − q^1.5 − (1 − q)^1.5]. At q = 0.1, this is approximately 1.291170. The second term is essential: omitting it would produce a different loss.
For comparison at the same probabilities, ordinary cross-entropy is −ln(0.1), approximately 2.302585. These numbers use different objective scales, so the smaller value does not establish better classification.
As q decreases to 0.01 and 0.0001, the tempered losses are approximately 1.790692 and 1.979901. They approach two as q approaches zero, whereas cross-entropy keeps increasing without bound. This isolates boundedness for the stated binary example. It does not prove that the target label is wrong or that downweighting the example will improve generalization.

Illustrative case study

In a fictional scenario, Juniper Archive trains a classifier using document labels assembled by several contractors. Some disagreements appear to reflect annotation mistakes, while others concern ambiguous category boundaries.
The team compares standard training with bi-tempered loss, keeping the data split and evaluation procedure consistent. It reviews performance by category and checks whether rare but legitimate examples are being neglected. The team also inspects confident errors instead of treating every high-loss training case as mislabeled.
The fictional lesson is to evaluate robustness alongside the consequences of reducing difficult examples' influence. No accuracy gain or successful deployment is claimed.

Common misconceptions and mistakes

The temperatures are not physical temperatures, and they do not merely divide logits by a single calibration constant. Bounded loss does not mean that all errors have equal loss or that training is effortless. The method does not guarantee robustness to every noise pattern. A lower training loss cannot be compared directly across different objectives as evidence of better performance. Heavy-tailed output probabilities also do not guarantee that probabilities are calibrated on new data.

Business use

Bi-tempered loss may be worth evaluating when classification labels are expensive to curate and known to contain noise. A useful comparison keeps the model, data access, and evaluation conditions aligned while examining task performance, calibration, and results for smaller categories. Label-quality improvements remain valuable. The relevant business question is whether the trained system makes better decisions under realistic conditions, rather than whether the chosen objective looks numerically smaller.

Check your understanding

Why is zero a valid normalization offset in the example?

Answer: The two tempered probabilities already sum to one.

What distinguishes the loss from a tempered negative logarithm alone?

Answer: The additional probability-power correction term.

Why can robust training harm a useful minority pattern?

Answer: Difficult correct examples may be treated as less influential even though their labels are valid.

Related terms

Learn next

Compare robust objectives with label auditing and sample weighting, then examine how evaluation separates label noise from difficult but valid examples.

Sources

Amid, Warmuth, Anil, and Koren, Robust Bi-Tempered Logistic Loss Based on Bregman Divergences (2019), especially Section 1.1: https://proceedings.neurips.cc/paper/2019/file/8cd7775f9129da8b5bf787a063d8426e-Paper.pdf . The calculations and Juniper Archive scenario are original illustrations.

Previous: Bhattacharyya distance Next: Bias