Is cross-entropy loss the same as accuracy?

No. It assesses assigned probabilities; accuracy usually counts the most likely predicted labels that match observations.

Can the loss reach zero?

With one-hot targets, zero is the limiting ideal when the correct class receives probability one. Finite softmax logits do not produce exact probability one.

Is it always the right objective?

No. Suitability depends on the prediction task, target representation, data quality, and decision requirements.

AI Glossary / Statistics and probability / Intermediate

Cross-entropy loss

Definition

Cross-entropy loss is a training objective that penalizes a model for assigning low probability to the correct outcome. For one observed class k, the ordinary categorical loss is -ln(q_k), where q_k is the predicted probability of that class. Training seeks parameters that reduce the average loss across examples.

What it means

The loss provides more information than a correct-or-incorrect score. Predicting the correct class with probability 0.51 and with probability 0.99 can give the same classification accuracy but different losses. A confidently wrong prediction receives a much larger penalty than an uncertain mistake.

Cross-entropy is the underlying information measure; cross-entropy loss is its use as an objective for fitting or assessing predictions. The name alone does not specify the target encoding, weights, averaging convention, or whether software expects probabilities or raw scores. Those details affect implementation and comparisons.

How it works

For mutually exclusive classes, a model often produces logits z_j. Softmax converts them to probabilities q_j = exp(z_j)/sum exp(z_i). With a one-hot target y, the loss is -sum y_j ln(q_j). Its derivative with respect to each logit is q_j - y_j for an unweighted single example.

Backpropagation passes that derivative through the model, and an optimizer updates its parameters. Stable implementations combine operations using log-sum-exp rather than naively exponentiating large scores.

For a binary target y and probability q, binary cross-entropy is -y ln(q) - (1-y) ln(1-q). Multilabel tasks often apply binary losses separately to labels that can coexist. This differs from forcing all labels into a single mutually exclusive softmax distribution.

Three practical examples

An email classifier learns probabilities for mutually exclusive routing categories from reviewed examples, using categorical cross-entropy during training.
An image tagging system uses separate binary terms for labels such as outdoor and vehicle, because several labels can be true simultaneously.
A next-token model learns from the probability assigned to each observed continuation, while padding positions are excluded from the training objective.

Think of it this way

Think of a tutor who asks for both an answer and a probability. A learner who repeatedly says “almost certain” before being wrong receives strong corrective feedback. The lesson is to allocate probability responsibly, not merely choose a winning option. The analogy describes the scoring rule; it does not guarantee that a trained model will become well calibrated.

Worked example

Consider an original three-class example with logits [ln(2),0,0]. Softmax gives probabilities [0.50,0.25,0.25]. The true class is the second, so the loss is -ln(0.25) = 1.38629 nats.

The target vector is [0,1,0], making the logit gradient [0.50,-0.75,0.25]. Treating these three logits as independently adjustable parameters, one gradient-descent step with learning rate 0.1 produces [ln(2)-0.05,0.075,-0.025].

The new probabilities are approximately [0.48095,0.27249,0.24656]. Loss falls to -ln(0.27249), approximately 1.30015 nats. The most likely class remains the first, so classification accuracy on this example has not improved even though the objective has.

This is a checked teaching update on free logits, not a full neural-network training run. Shared network parameters can affect many examples at once, and an arbitrary learning rate need not reduce a real training objective.

Illustrative case study

In this fictional case, Clearpath Support trains a model to route requests among billing, delivery, and technical help. Its analysts initially compare a total training loss from one run with a mean validation loss from another and conclude that the model improved dramatically.

They correct the comparison by using the same reduction, category mapping, and held-out dataset. They then inspect classification accuracy, per-category mistakes, probability calibration, and the costs of misrouting urgent requests alongside cross-entropy.

The team retains the loss as a useful training signal but sets routing thresholds using operational evidence. This invented case illustrates measurement discipline and does not claim a documented commercial result.

Common misconceptions and mistakes

A lower training loss does not establish better generalization. Overfitting, label errors, and distribution changes can separate training performance from deployment results.

Do not apply softmax twice when a library expects logits. Conversely, do not supply arbitrary logits to a function explicitly expecting probabilities. Read the interface rather than inferring it from the loss name.

Class weighting and label smoothing modify the objective. Scores from differently weighted or smoothed objectives are not automatically comparable. A highly confident wrong label can dominate ordinary cross-entropy, so inspect label quality rather than assuming every large loss identifies a model defect.

Business use

Business teams can use cross-entropy to compare probabilistic classifiers under a consistent evaluation protocol. Keep label definitions, data periods, weighting, and averaging fixed, and record whether the score is measured on training or held-out data.

Translate deployment decisions into the relevant outcomes: missed cases, unnecessary reviews, delays, or cost. An improvement in the training objective does not itself quantify savings. For imbalanced tasks, examine performance in consequential minority categories even when the overall average looks favorable.

Check your understanding

Check your understanding: In the worked example, why did loss improve while accuracy stayed unchanged?

Answer: The correct class received more probability, but another class remained most likely.

Why must binary multilabel output be distinguished from softmax output?

Answer: Multiple labels may be true simultaneously, whereas a softmax distribution allocates one total probability mass across alternatives.

Related terms

Cross-entropy
Softmax
Logit
Gradient descent
Binary classification
Calibration
negative log-likelihood
label smoothing
class imbalance

Learn next

Derive the binary loss from a Bernoulli likelihood, then compare held-out loss and classification accuracy for two models whose confidence differs. Inspect how class weights and excluded positions change a batch average before interpreting training dashboards.

Sources

Sources: Jurafsky and Martin, Speech and Language Processing, logistic regression chapter, https://web.stanford.edu/~jurafsky/slp3/4.pdf, for likelihood-based learning and binary loss. PyTorch, CrossEntropyLoss documentation, https://docs.pytorch.org/docs/2.14/generated/torch.nn.CrossEntropyLoss.html, for logits, targets, weighting, and reduction conventions. Worked calculations and the fictional case are original teaching material.

Previous: Cross-entropy Next: Cross-entropy method