Can one backbone have several heads?

Yes, for separate tasks or outputs.

Does a head always produce one label?

No. Multilabel designs can select several categories.

AI Glossary / Natural language and speech / Advanced

Classifier head

Definition

A classifier head is the part of a neural model that converts learned representations into scores for a classification task. It sits after a feature-producing component, often called the encoder or backbone, and maps its output to a specified set of labels. A simple head is a learned linear transformation; others contain additional layers.

What it means

A text encoder may represent a message as a vector containing hundreds of numbers. Those numbers are not themselves labels such as complaint, question, or praise. A classifier head learns how to turn that representation into evidence for the categories required by a particular application.
The division makes it possible to reuse a representation model for different tasks while changing the output component. However, attaching a new head does not make its predictions useful automatically. A newly initialized head generally needs task-specific training, and the underlying representation must contain information relevant to the task.
The word “head” describes a component's role, not one universal architecture. It is also different from an attention head inside a transformer: attention heads help compute representations, whereas a classifier head produces task-level classification scores.

How it works

For a simple linear head, a representation h is transformed into a score vector z using z=Wh+b. The rows of W correspond to output classes, and b supplies one bias per class. These scores are commonly called logits. They can be negative and need not sum to one.
For mutually exclusive classes, softmax can convert logits into a probability distribution. The highest-scoring class is a common prediction rule. For a multilabel task, separate sigmoid outputs can instead support multiple labels, with decision thresholds chosen for the task. These alternatives should match the label definition and training objective.
During training, a loss compares predictions with target labels. The training setup may update only the head while keeping the backbone's parameters fixed, or update both. Sequence classification typically uses a representation for the whole input, while token classification applies predictions at token positions. Padding, pooling, and token alignment still require appropriate handling.

Three practical examples

A support classifier maps a message representation to one of several routing categories, using a head trained on the organization's labeled examples.
A document tagging model uses a multilabel head because a report can concern both procurement and sustainability rather than exactly one topic.
A named-entity model assigns category scores to token representations, then assembles compatible token predictions into entity spans.

Think of it this way

Think of the backbone as producing a structured set of observations and the head as a specialized scoring sheet. The same observations can support different scoring sheets. Changing the sheet changes the question being answered, but it cannot recover information absent from the observations.

Worked example

Consider an original two-feature, three-class teaching model. Its labels, in order, are question, complaint, and praise. Suppose the encoder supplies h=[2,1]. These values are invented for arithmetic and are not claimed to be interpretable features from a real language model.
Let the head's weight rows be [1,0], [0,1], and [−1,0], with biases [0,0,1]. Multiplication gives:
Question: 1×2+0×1+0=2.
Complaint: 0×2+1×1+0=1.
Praise: −1×2+0×1+1=−1.
The logits are therefore [2,1,−1]. Question has the largest score. Applying softmax gives approximately [0.7054,0.2595,0.0351]. The probabilities sum to one, but their normalization alone does not establish that they are well calibrated on real data.
If complaint is the correct training label, the unweighted cross-entropy for this example is −ln(0.2595), approximately 1.3490 using the unrounded probability. A training update can use this error to adjust the parameters permitted by the training configuration. The calculation does not demonstrate that one update will correct the prediction.
This head has six weights and three biases, for nine parameters. Adding a fourth class to the same two-feature linear design would require eight weights and four biases, for twelve parameters. Merely adding the output slot would not teach the model the new category.
Finally, adding the same constant to every logit leaves the softmax probabilities unchanged. Replacing [2,1,−1] with [7,6,4] therefore preserves both the distribution and the predicted label. Absolute logit values should not be read as percentages.

Illustrative case study

Fictional example: TicketHarbor reuses a text encoder when replacing three broad support categories with a more detailed set. The team creates a head with the new number of outputs and trains it using examples labeled under the revised definitions.
An integration review discovers that the application still uses the old ordering of label names. Numerically valid outputs are consequently displayed with incorrect meanings. The team versions the label mapping alongside the model and checks representative messages end to end. This scenario illustrates a compatibility failure, not a measured gain from a particular architecture.

Common misconceptions and mistakes

A classifier head is not necessarily just the final softmax operation. Its learned transformation may produce logits, with probability conversion performed elsewhere. Conversely, some architectures include several layers within the head.
A high softmax value does not prove that the input resembles the training distribution or that the prediction is correct. Freezing the backbone also does not eliminate the cost of computing its representations. When using a library loss that expects logits, applying softmax first changes the input the loss receives and can produce an unintended training objective.

Business use

Document the head's label inventory, label ordering, input representation, and decision rules as part of the deployed model. Evaluate minority categories and realistic mixed-topic inputs, not just overall accuracy. When categories change, review training data, output dimensions, downstream routing, and monitoring together. A small component can still control a consequential business decision.

Check your understanding

How many parameters are in the example head?

Answer: Nine.

Which label has the highest logit?

Answer: Question.

Why can a correct score vector produce an incorrect displayed label?

Answer: The application may use the wrong label ordering.

Related terms

Learn next

Recalculate the example for a different representation, then compare single-label and multilabel decision rules. Trace how numerical output positions become named categories in an application.

Sources

Hugging Face, BERT model documentation, sequence and token classification heads: https://huggingface.co/docs/transformers/model_doc/bert PyTorch, CrossEntropyLoss, expected logits and loss formulation: https://docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html PyTorch, linear-layer documentation: https://docs.pytorch.org/docs/main/generated/torch.nn.Linear.html

Previous: Classifier guidance Next: Clause