Can one model have several output heads?

Yes. A shared representation can feed separate heads for different tasks, although the training setup must manage their objectives.

Is an attention head the same as a language-modeling head?

No. Attention heads operate inside attention layers; the language-modeling head produces token scores from final representations.

Can a new head work without training?

A randomly initialized task head generally has no reason to produce useful task predictions. Whether training only the head is sufficient depends on the task and representations.

AI Glossary / Generative AI and language / Intermediate

Head

Definition

A head is a model component that performs a particular computation or produces a particular kind of output. In language-model discussions, two common meanings are an attention head inside a transformer and an output or task head attached to a model's representations. An attention head computes a weighted combination of information; a language-modeling or classification head maps representations to predictions. The surrounding context is essential because these are different architectural roles.

What it means

The word is shorthand, not a complete specification. Saying a model has “eight heads” usually refers to attention heads within a layer, while saying it has “a classification head” refers to its task output. A shared model body can support different output heads, and a transformer can contain many attention heads across many layers. Counting heads without identifying which type and where they sit can therefore give a misleading picture of the model.

How it works

In a standard attention head, learned projections create queries, keys, and values. Query-key comparisons produce weights used to combine values; multiple heads provide several such computations before their outputs are combined. An output head instead takes a representation produced by the model body and transforms it into task scores. A language-modeling head commonly produces vocabulary logits, which can be normalized into next-token probabilities. A classification head produces class scores. Exact designs vary and can include additional transformations, pooling, or shared parameters.

Three practical examples

Sentiment classification: attach a trained output head that maps a text representation to sentiment-category scores.
Language generation: use a language-modeling head to convert a token's final hidden representation into scores over possible next tokens.
Architecture analysis: inspect attention heads to understand patterns of information aggregation, while avoiding the assumption that each head has one fixed, human-readable function.

Think of it this way

Think of a workshop with shared preparation equipment and different finishing stations. An output head resembles a finishing station tailored to one product, while attention heads are more like parallel processing stations within the shared production line. The analogy clarifies location and role, but the learned numerical components do not have conscious responsibilities or necessarily interpretable specialties.

Worked example

Consider a toy classification output head receiving a two-number representation h = [2, 1]. It has three class-weight rows: [1, 0], [0, 1], and [-1, -1], with zero biases. The resulting scores are 2, 1, and -3. Applying softmax gives approximate probabilities 0.7275, 0.2676, and 0.0049, so the first class receives the largest score. This is an output-head calculation, not an attention-head calculation.
For a separate architecture example, suppose a transformer layer represents each token using 512 features and uses eight attention heads with 64 output features each. Concatenating the eight head outputs gives 8 times 64 = 512 features before the output projection. That does not mean the classifier has eight classes or that the model has eight separate full copies of its entire representation pipeline. If the transformer has six such layers, it has 48 attention-head instances across those layers under this simplified design, while it could still have only one classification output head. These examples show why both the type and location of a head must be stated when interpreting counts or dimensions.

Illustrative case study

A fictional document team adapts a pretrained text model for routing messages into three departments. It initially loads a model body without a trained routing head and expects meaningful labels. The team discovers that a newly initialized head needs task training and evaluation. It also finds that a configuration field describing attention-head count has nothing to do with the three department labels. Clear architecture notes prevent these two uses of “head” from being confused during implementation and review.

Common misconceptions and mistakes

More heads do not automatically mean greater intelligence or better quality. Attention heads are not necessarily independent experts, and their patterns can change across inputs and training. A task head can be small yet still require appropriate labeled data. A language-modeling head predicts tokens rather than verifying facts. Replacing an output head does not ensure that the model body already contains every feature needed for the new task. The same word can also have meanings in linguistics or other software contexts, so labels should be expanded when ambiguity matters.

Business use

When comparing models, ask whether head counts describe attention structure, output tasks, or another component. For adaptation projects, identify which components are pretrained, newly initialized, frozen, or updated. Check that output dimensions match the task's classes or vocabulary and that evaluation measures the intended behavior. Use attention visualizations as exploratory evidence rather than a complete explanation of a decision. Resource estimates should follow actual parameter and tensor shapes, not the head count alone.

Check your understanding

A model uses eight attention heads and predicts three document classes. How many classification outputs should its routing head normally produce?

Answer: Three class scores for this task. The attention-head count describes an internal computation and does not determine the number of task labels.

Related terms

Learn next

Draw a model body, its internal attention layers, and the final task head as separate components. Label the dimensions of one attention output and one classification output. Study scaled dot-product attention and linear classification next, then inspect a real architecture's naming conventions to determine exactly what each use of “head” refers to.

Dot-product attention

Sources

Vaswani and colleagues, Attention Is All You Need: https://arxiv.org/abs/1706.03762 Hugging Face Transformers, model structure documentation: https://github.com/huggingface/transformers/blob/main/docs/source/en/models.md Jurafsky and Martin, Speech and Language Processing, language-modeling head discussion: https://web.stanford.edu/~jurafsky/slp3/ed3bookfeb3_2024.pdf

Previous: HDR-VDP Next: Head/tail breaks