Are positional embeddings always learned?

No. Some are learned parameters, while others are generated by fixed mathematical rules. Position information can also enter attention through relative or rotary mechanisms.

Does the first token always have position one?

No. Indexing conventions vary, and zero-based position identifiers are common. Follow the specific model's expected convention.

Can position information alone make a model understand grammar?

No. It supplies useful structure, but understanding depends on the learned computations, training data and task. It is one component of the system.

AI Glossary / Retrieval and knowledge systems / Intermediate

Positional embeddings

Definition

Positional embeddings are numerical representations that supply information about where tokens or other elements occur in a sequence. They help a model distinguish arrangements that contain the same elements in different orders. In Transformer terminology, the phrase is often used broadly alongside positional encoding, although individual architectures introduce position information in different ways.

What it means

A token embedding represents the token's identity, while position information represents its location or relationship to other locations. The distinction matters because "the dog chased the cat" and "the cat chased the dog" contain the same words but describe different events.

Plain self-attention without position signals or order-dependent constraints does not by itself assign an intrinsic sequence order to its inputs. A model therefore needs an appropriate mechanism for using order. Positional embeddings are one family of mechanisms, rather than a claim that every Transformer must use the same additive vector table.

How it works

In a common absolute-position design, each sequence position has a vector with the same dimension as the token embedding. The vectors are added before further processing. Position vectors may be learned during training or computed from a fixed rule, such as the original Transformer's sinusoidal encoding.

Relative-position approaches instead represent relationships such as the distance between two tokens within attention computations. Rotary position embedding applies position-dependent rotations to query and key representations; it is not simply another vector added to the input embedding.

Implementation details must match the trained architecture. Position numbering, padding, special tokens and cached generation can affect which positions are assigned. Increasing a configured maximum length does not automatically teach the model to use longer sequences reliably. Length extension should be evaluated for the specific position method and task.

Three practical examples

1. A language model uses order information to distinguish who performed an action from who received it, rather than relying only on a collection of words.

2. A document encoder represents the sequence positions of tokens within its supported input window, helping later layers use local and distant relationships.

3. A vision Transformer can use spatial position representations for image patches, since the same visual pieces arranged differently may describe different images.

Think of it this way

Think of a theater ticket with both a person's name and a seat number. The name identifies who is attending; the seat number locates them in an arrangement. Keeping the same people but changing their seats changes the layout. Position representations give a model numerical information about that layout.

Worked example

Consider a toy additive model with two-dimensional vectors. Let token A have embedding (1,0), token B have embedding (0,1), and the first and second positions have vectors (0.1,0.2) and (0.3,0.4).

For sequence [A,B], the combined inputs are (1.1,0.2) at the first position and (0.3,1.4) at the second. For [B,A], they become (0.1,1.2) and (1.3,0.4). The same token now receives a different positional contribution when it moves.

These additions show how the inputs carry both token and position information. They do not prove that the model understands the sequence correctly. Notice also that summing the two combined vectors gives (1.4,1.6) in both cases: a later operation that simply discards order by summing can still lose the distinction. The surrounding architecture must use the position-sensitive representations effectively.

Illustrative case study

Fictional case study: SequenceWorks adapts a document model to a new input pipeline. Its team changes padding placement but accidentally leaves position identifiers inconsistent with the intended model convention. Some examples still produce plausible outputs, hiding the issue during a brief demonstration.

The team inspects token positions, special tokens and attention masks on a small set of known inputs. It corrects the pipeline and compares the same documents under the supported formatting rules before evaluating the larger dataset.

This fictional case illustrates why position handling is part of model compatibility. It does not establish a universal rule for left or right padding, because the correct handling depends on the model and execution path.

Common misconceptions and mistakes

Positional embeddings are not the same as vocabulary embeddings, attention masks or token-type embeddings. These can coexist and carry different information. Absolute position does not mean a real-world timestamp unless the representation is specifically designed that way.

Another mistake is assuming that a longer position table or a mathematically computable encoding guarantees useful long-context behavior. The model's training and other architectural limits still matter. Finally, relative and rotary methods should not be described as identical to adding learned absolute vectors; they inject position information through different computations.

Business use

For businesses building document, language or multimodal systems, positional representations usually matter through model selection and integration. Errors can arise when changing tokenization pipelines, batching, sequence lengths or generation caching, even when the model weights remain unchanged.

Validate the complete input path using examples whose interpretation depends on order. For long documents, test performance across positions and distances rather than only checking that the system accepts the input. A model can process a large sequence without using every part of it equally well. Chunking and retrieval choices also affect what positional relationships remain available.

Check your understanding

A token keeps the same vocabulary identifier but moves from the first to the second position in an additive absolute-position model. What changes?

Answer: Its vocabulary embedding stays the same, but the positional vector added to it changes. Consequently, its combined input representation generally changes.

Related terms

Absolute position
Self-attention
Attention mask
token embedding
relative position representation
rotary position embedding
context window and sequence modelling

Learn next

inspect token and position identifiers for a short sequence before and after padding. Then compare additive absolute encoding with relative attention or rotary encoding, focusing on where the position information enters the computation.

Sources

Primary sources: Vaswani and colleagues, Attention Is All You Need, section 3.5: https://arxiv.org/html/1706.03762v7

Su and colleagues, RoFormer: Enhanced Transformer with Rotary Position Embedding: https://arxiv.org/abs/2104.09864

Hugging Face Transformers glossary, position identifiers and positional embedding variants: https://huggingface.co/docs/transformers/v4.26.1/en/glossary

Previous: Pose Next: Positional encoding