Can a correct prediction have positive hinge loss?

Yes. A positive signed margin below one is correct under the sign rule but still incurs loss.

Is hinge loss differentiable everywhere?

No. It has a corner when the signed margin equals one.

Does a lower hinge loss always mean fewer classification mistakes?

No. It can improve margins without changing any decisions, and aggregate comparisons can trade off different examples.

AI Glossary / Mathematics and optimization / Intermediate

Hinge loss

Definition

Hinge loss is a margin-based loss used in classification. For a binary label y in {−1, +1} and a real-valued prediction score s, it is max(0, 1 − ys). The product ys is the signed margin. The loss is zero when that margin is at least one, positive for a correct but insufficiently separated prediction, and larger when the score points toward the wrong class.

What it means

A classifier's sign decision and its training loss answer different questions. A positive score usually predicts the positive class and a negative score the negative class, but hinge loss also asks how far the score has moved in the correct direction on its chosen scale. A correctly classified observation can therefore still contribute loss. The score is not required to be a probability. Hinge loss is associated with support vector machines, where it is combined with a regularization term that controls the model's size and margin tradeoff.

How it works

Compute the signed margin m = ys, subtract it from one, and retain the result only if it is positive. As a function of the score, the loss is linear while m < 1 and flat while m > 1. It has a corner at m = 1, so it is convex but not differentiable there. For a linear score s = wᵀx + b, a violating example has gradient −yx with respect to w away from the corner; a sufficiently separated example contributes zero loss gradient. Training methods must handle the corner through an appropriate subgradient or other supported optimization approach.

Three practical examples

A binary linear-classification experiment can use hinge loss to encourage correct decisions with a specified score margin.
A model evaluator can compare hinge loss on decision scores with ordinary classification error to see which correctly classified examples remain close to the boundary.
A technical review can distinguish plain hinge loss from squared hinge loss and check that the chosen training and reporting conventions match the implementation.

Think of it this way

Think of requiring a delivery to arrive before a deadline with a safety buffer. Arriving on the correct side of the deadline can still fall short of the desired buffer. The analogy explains why correct classification can incur loss, but the numerical margin is a model-score convention, not elapsed time or a calibrated confidence level. Regularization and feature scaling influence its practical meaning.

Worked example

Consider four observations with labels y = (+1, +1, −1, −1) and scores s = (1.5, 0.25, −0.5, 0.75). Their signed margins are 1.5, 0.25, 0.5 and −0.75. The hinge losses are therefore 0, 0.75, 0.5 and 1.75. Total hinge loss is 3 and mean hinge loss is 0.75. Using the sign of each nonzero score for classification, only the fourth observation is wrong, giving error rate 1/4 = 0.25. The second and third are correctly classified but still incur loss because their margins are below one. If every score is doubled, the margins become 3, 0.5, 1 and −1.5, and losses become 0, 0.5, 0 and 2.5. The mean remains 0.75 in this particular example: improved margins for correct examples are offset by a more strongly wrong fourth score. Score scaling therefore changes hinge contributions without necessarily changing classification decisions. For squared hinge loss on the original scores, the mean would be (0 + 0.75² + 0.5² + 1.75²)/4 = 0.96875, demonstrating that the squared variant is a different loss.

Illustrative case study

In a fictional ticket-routing study, Hazel Support uses a linear classifier with a margin-based objective. Analysts discover that a report passes hard class labels into the loss calculation instead of the real decision scores. They correct the evaluation and separately report classification errors and hinge loss. The team also checks label encoding and class order before comparing model versions. The case illustrates reliable metric calculation, not a claim that hinge loss is the best objective for every routing task.

Common misconceptions and mistakes

Hinge loss is not the same as zero-one classification loss or a probability error. Feeding probabilities or hard labels into a formula intended for decision scores changes its meaning. Labels coded 0 and 1 must be handled according to the implementation's convention rather than inserted blindly into the ±1 formula. Zero loss does not imply a uniquely optimal model, and driving scores larger is not a substitute for regularization. The loss has a nondifferentiable corner. Multiclass hinge losses have several formulations, so the binary equation should not be presented as their complete definition.

Business use

Document label encoding, score orientation and whether the reported value is a mean, sum or weighted mean. Check the metric on a few hand-calculated examples, including correct low-margin and incorrect predictions. Distinguish the unregularized hinge metric from the full training objective. If the application needs probabilities, assess a suitable calibration procedure rather than treating raw margins as probabilities. Evaluate errors and their business consequences alongside the surrogate loss. Preserve feature scaling and model settings when comparing runs, because the score scale affects margin-based quantities.

Check your understanding

Why does the worked third observation have loss 0.5 even though its score correctly predicts the negative class?

Answer: Its label-score product is (−1)(−0.5) = 0.5, below the required margin one, so the loss is 1 − 0.5 = 0.5.

Related terms

Support vector machine
Squared hinge loss
Regularization
signed margin
decision function
zero-one loss

Learn next

plot max(0, 1 − m) against margin m and mark the four worked observations. Compare its values with zero-one error and squared hinge. Then inspect the documented class ordering and score convention of a classifier before calculating the loss on its output.

Sources

Sources: Jung and colleagues, The Aalto Dictionary of Machine Learning, hinge loss, https://aaltodictionaryofml.github.io/ADictML_Main.pdf ; scikit-learn, hinge_loss reference, https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hinge_loss.html . The four-observation calculation, score-scaling comparison and fictional ticket-routing review are original; the binary formula is distinguished from multiclass variants.

Previous: Hindsight optimization Next: Hinton diagrams