How does it extend binary logistic regression?

It models a probability distribution over several classes rather than only two outcomes.

Can it represent nonlinear boundaries?

It can use nonlinear feature transformations, although its class scores remain linear in the supplied features.

Should the largest probability always determine the action?

Not necessarily. Different error costs, abstention rules or human-review requirements can justify another decision rule.

AI Glossary / Statistics and probability / Intermediate

Multi-class logistic regression

Definition

Multi-class logistic regression is a classification method that estimates probabilities for more than two mutually exclusive outcome categories. Its joint multinomial form, often called softmax regression, assigns a score to each class and converts the scores into probabilities that sum to one. The scores are usually linear functions of the input features, although the features themselves can include transformations. Another multiclass approach fits separate one-versus-rest logistic models; that procedure should not be confused with jointly fitting a multinomial probability model.

What it means

The model answers a question such as which single category best describes a request, while retaining a probability estimate for every category. Unlike ordinary linear regression, its output probabilities remain within their valid range. The categories need not have a natural order. A model for ordered outcomes may use a different formulation, and a task allowing several labels at once needs a multilabel approach. Logistic probabilities are estimates based on the fitted relationship and data; they are not automatically well calibrated or sufficient for every decision.

How it works

Represent each example with features and one observed class label. For class k, compute a score zk = bk + wk dot x. The softmax probability is exp(zk) divided by the sum of exponentiated scores across classes. Fit coefficients by optimizing the multinomial likelihood, commonly expressed as cross-entropy loss, with regularization when appropriate. A reference-class parameterization uses K - 1 sets of coefficients because adding the same score to every class does not change probabilities. Evaluate classification quality, calibration and relevant subgroup behavior on suitable held-out data. Apply a decision rule suited to the consequences of errors.

Three practical examples

Request routing: classify an incoming message into one of several mutually exclusive support categories.
Document processing: assign a form to a defined document type using its observed features.
Outcome modeling: estimate probabilities for several unordered survey or operational outcomes. A good application checks that the categories are defined consistently and that one label per observation is appropriate.

Think of it this way

Think of several competing explanations receiving scores. Softmax converts those relative scores into shares of a total probability budget. Raising one score while holding the others fixed increases its share and reduces the others. Adding the same amount to every score changes none of the shares, because the relative comparison is unchanged.

Worked example

Suppose three artificial class scores are 0, log(2) and log(3). Their exponentials are 1, 2 and 3, with total 6. The resulting probabilities are 1/6, 2/6 and 3/6: approximately 0.1667, 0.3333 and 0.5. An equal-cost largest-probability decision selects the third class, although the other classes together still have probability 0.5. If the observed label is the third class, the example's cross-entropy loss is -log(0.5), approximately 0.6931 using natural logarithms. Adding 5 to every score leaves the probabilities unchanged because the common exp(5) factor cancels. With three classes, four input features and an intercept, a reference-class formulation has (3 - 1) times (4 + 1) = 10 coefficients. These original calculations illustrate normalization and parameterization, not a fitted model's real-world accuracy.

Illustrative case study

In a fictional case, Cedar Documents builds a classifier for three types of incoming forms. The team uses a multinomial logistic baseline before testing more complex alternatives. Reviewers find that some forms legitimately belong to two categories, so the original single-label definition is unsuitable for that subset. They revise the task rather than forcing ambiguous records into one category. For the remaining task, they evaluate calibration and define a review policy when no class receives sufficient support for an automated action.

Common misconceptions and mistakes

Multiclass does not mean multilabel. A joint softmax model is not identical to independent binary classifiers, even if both produce one final class. The linear scores do not imply probabilities change linearly with inputs. Coefficients describe relative log-odds under a chosen parameterization, not direct percentage-point probability changes. Predictors do not need to be normally distributed. Class imbalance, separation, correlated features and limited samples can complicate estimation or interpretation. The standard multinomial logit structure also imposes restrictions on relative odds, which may be inappropriate for some choice problems.

Business use

Businesses can use the method as an efficient, inspectable baseline when a single categorical outcome is well defined. Record feature preprocessing and category order, because both affect interpretation and deployment. Compare overall metrics with class-specific errors and assess whether probability calibration supports the intended decision thresholds. Regularization can improve stability but requires appropriate tuning. Include unknown or out-of-scope cases in the operational design. A clear model and well-defined labels may be more useful than additional complexity applied to an ambiguous classification task.

Check your understanding

If the same constant is added to all softmax scores, do the probabilities change?

Answer: No. The common multiplicative factor in the exponentials cancels during normalization.

Related terms

Softmax
Cross-entropy
Calibration
Multinomial logit
reference category
class probability
one-versus-rest
multilabel classification

Learn next

Review binary logistic regression and log-odds, then calculate a small softmax example. Compare joint multinomial fitting with one-versus-rest classification and examine how decision costs affect the final action.

Sources

Penn State STAT 504, Multinomial Logistic Regression Models: https://online.stat.psu.edu/stat504/Lesson08 Scikit-learn developers, Linear Models, multinomial logistic regression: https://scikit-learn.org/stable/modules/linear_model.html

Previous: Multi-class classification Next: Multi-class multi-output