Is cross-entropy always only a surrogate?

It can also be the direct probabilistic objective of interest; its role depends on the task being optimized.

Does convexity guarantee easy neural-network training?

No. A convex score-level loss can become nonconvex after composition with a neural network.

Can a lower surrogate loss accompany lower accuracy?

Yes, as the two-model example shows.

AI Glossary / Probabilistic machine learning / Intermediate

Surrogate loss

Definition

A surrogate loss is an alternative training objective used in place of a task loss that is difficult to optimize directly. It provides a more useful mathematical or computational signal while aiming to support good performance on the original task. Logistic loss and hinge loss are common surrogates for binary classification error.

What it means

A classifier's error count changes only when its decision crosses a boundary. Small improvements in its score may leave that count unchanged, giving an optimizer little guidance. A surrogate can distinguish a confidently wrong prediction, a nearly correct one, and a confident correct one.

The surrogate and the task metric are different quantities. Lower surrogate loss can support better learning under suitable conditions, but it does not guarantee that every update or finite-model comparison improves accuracy, recall, revenue, or another operational outcome. Choosing a surrogate therefore involves both tractability and alignment with the intended decision problem.

How it works

Specify the target task loss and the model's output representation. For binary labels y in {minus 1, plus 1} and real score f(x), the margin is y times f(x). Positive margins support correct classification; negative margins support the wrong class.

Logistic loss is log[1 + exp(minus margin)]. Hinge loss is the maximum of zero and one minus the margin. Both provide richer signals than a simple error indicator. Hinge loss is convex in the margin but has a corner, so differentiability everywhere is not required for a useful surrogate.

Classification calibration connects minimizing population surrogate risk with the correct classification decision under appropriate conditions. It does not remove restrictions from a finite model family, limited data, optimization error, or distribution shift. Also, a convex loss in model scores need not make a neural-network training problem convex in its parameters. The whole composition determines optimization behavior.

Three practical examples

1. Classification training: optimize cross-entropy rather than directly minimizing the number of misclassified records. Evaluate the final decision thresholds and error costs separately.

2. Ranking development: use a tractable pairwise or listwise training objective for a ranking task. Check the ranking metric and user-relevant outcomes because a chosen surrogate may emphasize different mistakes.

3. Structured prediction: replace an expensive combinatorial task loss with an objective that can be optimized more efficiently. Investigate whether its assumptions and decision rule fit the actual output structure rather than importing a binary-classification guarantee unchanged.

Think of it this way

Think of practicing parking using the distance from a target space rather than receiving only a final pass-or-fail signal. Distance offers guidance before the maneuver succeeds. But improving that guide is not enough if the actual test also cares about obstacles, orientation, or safety. A useful surrogate must support the real task rather than replace its meaning.

Worked example

Consider an original two-example binary classification comparison. Both true labels are positive, and a predicted positive probability greater than 0.5 is classified as positive. Model A assigns probability 0.6 to each example. It classifies both correctly and has mean log loss minus ln(0.6), approximately 0.5108 nats.

Model B assigns probabilities 0.99 and 0.49. It classifies the first correctly and the second incorrectly, so its accuracy is only 50 percent. Its mean log loss is [minus ln(0.99) minus ln(0.49)] divided by 2, approximately 0.3617 nats, lower than Model A's.

The example shows that two finite predictors can be ranked differently by log loss and thresholded accuracy. Model B receives a substantial reward for its confident correct first prediction and only a moderate penalty for its near-threshold error on the second. This does not contradict population calibration results, which concern different assumptions and optimization limits.

For another distinction, a correctly classified margin of 0.2 has hinge loss 0.8, while a margin of 1.2 has hinge loss zero. Both have zero classification error, but the surrogate distinguishes their margins. The appropriate metric depends on what the application needs to assess.

Illustrative case study

Fictional case: Cedar Routing trains a model to direct incoming requests. Its team uses cross-entropy for training but evaluates missed urgent requests, overall accuracy, and probability calibration separately. Validation data determine an operating threshold consistent with the workflow's costs.

When a new model lowers validation cross-entropy but worsens the urgent-request metric, analysts inspect the affected records instead of declaring an automatic improvement. They consider whether class weighting or the decision rule should change and reserve an independent test set for final assessment. This fictional case describes a decision process and reports no measured business benefit.

Common misconceptions and mistakes

A surrogate is not necessarily an upper bound on the original loss without a specified scaling or construction. It need not be smooth, and it need not be convex. Classification calibration is not identical to calibrated predicted probabilities. Lower training loss does not prove generalization, while a good surrogate choice cannot correct mislabeled data or an inappropriate evaluation population by itself.

Business use

The practical value is turning a difficult learning objective into a tractable training process. The risk is treating progress on the training objective as progress on every business outcome. Teams should state the target metric, surrogate, weighting, regularization, and decision rule. Track both surrogate and task performance on held-out data, especially when errors have asymmetric costs or the deployment distribution differs from training.

Check your understanding

Answer: 1.

Which model has lower log loss in the example?

Answer: Model B, at approximately 0.3617 nats.

Answer: 2.

Which model has higher thresholded accuracy?

Answer: Model A, with both examples correct.

Answer: 3.

Why can hinge loss be useful despite its corner?

Answer: Subgradient-based optimization can handle that nondifferentiability, and the loss still provides margin information.

Related terms

Loss function
Cross-entropy
Hinge loss
Empirical risk minimization
Regularization
classification calibration
decision threshold
task metric

Learn next

Plot classification error, logistic loss, and hinge loss against the margin. Then compare their rankings on finite predictions and study the assumptions connecting surrogate-risk minimization to task performance.

Sources

Bartlett, Jordan and McAuliffe, Convexity, Classification, and Risk Bounds, analyzes surrogate risk and classification calibration; author technical-report version: https://www.stat.berkeley.edu/~jordan/638.pdf

The two-model and margin calculations and Cedar Routing case are original teaching examples; the case is fictional.

Independent review consulted the author report’s surrogate-risk discussion, classification-calibration definition and margin-loss examples.

Previous: Surrogate key Next: Surrogate model