Does absolute position always require learned vectors?

No. Learned tables are common, but fixed index-based encodings are also possible.

Does adding a position vector change the token's identity?

It changes the representation supplied to the model, while the token ID itself remains the same.

Is absolute position a physical location?

No. Here it is an index within a sequence representation.

AI Glossary / Natural language and speech / Intermediate

Absolute position

Definition

Absolute position identifies where a token occurs within a sequence using its own position index, such as first, second, or third. In the transformer setting, an absolute positional representation supplies information about that index so the model can distinguish identical tokens appearing in different places. A common implementation adds a position-specific vector to each token's embedding.

What it means

A token's identity and its location answer different questions. A vocabulary identifier tells the model which token it is; a position index tells the model where that occurrence sits within the current sequence. The same word can therefore use the same token embedding but receive different positional information at different occurrences.

The supplied source describes learned absolute position embeddings: one trainable vector for each supported position. More broadly, absolute positions can also be represented through fixed functions such as sinusoidal encodings. Relative-position approaches instead emphasize relationships or distances between positions, although implementation details vary across architectures.

How it works

In a learned-table design, create a matrix with one row per supported sequence position and one column per embedding dimension. Look up the token embedding and the position embedding, then add them. Their dimensions must match. The resulting vector enters the transformer alongside other required inputs and masks.

Training adjusts the position vectors along with other model parameters. Positions seen less often may receive less useful training. A table also has a finite range of defined rows, so longer sequences require an explicit extension or different treatment.

Position indices are coordinates within the model's sequence convention. They are not timestamps, character offsets, or real-world ordering guarantees. Special tokens, padding, concatenated examples, and cached generation can all affect which index should be assigned. The attention mask is a separate mechanism controlling which positions can interact.

Three practical examples

1. Sentence encoding: distinguish an occurrence of a token near the beginning from another occurrence later in the same input.

2. Document processing: represent where a token falls within the model's current input window, while keeping that window index distinct from its page or paragraph location.

3. Autoregressive generation: assign successive position indices to newly generated tokens so their representations remain consistent with the prefix already processed.

Think of it this way

Think of numbered seats in a row. A person's name identifies who is present, while the seat number identifies their location. Relative position would describe how far one person sits from another. The analogy has limits: a learned position vector is a numerical representation, not a literal seat label, and the model must learn how to use it.

Worked example

Suppose a toy model uses two-dimensional token embeddings. The token “blue” has embedding e = (1, 2). Its learned position vector at index 0 is p_0 = (0.1, 0.2), while at index 1 it is p_1 = (−0.3, 0.4).

An occurrence of “blue” at index 0 enters the next stage as e + p_0 = (1.1, 2.2). At index 1, the same token enters as e + p_1 = (0.7, 2.4). The token identity is unchanged, but its initial representation carries different position information.

Now compare positions 1 and 3 with positions 4 and 6. Both pairs have a separation of two. An absolute table uses four position-specific entries; a simple relative-distance representation could associate the same distance value with both pairs. This illustrates the distinction, not an assertion that complete attention computations must be identical.

For a separate storage calculation, a learned table supporting 2,048 positions with embedding dimension 768 contains 2,048 × 768 = 1,572,864 parameters. Stored as 32-bit values, the raw table requires 6,291,456 bytes, or 6 MiB, excluding training-state and implementation overhead.

The table has no row for index 2,048 if indexing runs from 0 through 2,047. A system must not silently treat a missing row as evidence that the model supports arbitrary lengths. These examples are invented and hand-checked, not descriptions of a particular released model.

Illustrative case study

Fictional case: Birch Search compares document embeddings produced individually with embeddings produced in padded batches. Some results differ because the two paths assign different position indices around padding and special tokens.

The team inspects the tokenizer output, position indices, and attention masks together. It defines a consistent convention and verifies matched examples under the intended inference settings. It also separates harmless numerical variation from systematic indexing differences. The case illustrates representation debugging and does not claim a measured retrieval improvement.

Common misconceptions and mistakes

Absolute position is not the same as a vocabulary ID. A frequent token can appear at many positions, and a position can contain many different tokens across examples. Nor is a token position necessarily a word number: tokenization can split one word into several pieces.

A fixed sinusoidal formula can be evaluated at new indices, but this alone does not prove useful model generalization to longer contexts. Learned tables likewise need a deliberate policy for extension.

Do not assume an absolute embedding enforces causality. In a decoder, the causal mask supplies the relevant restriction on access to later tokens. Positional information and access control play different roles.

Business use

For product teams using language models, positional conventions matter when changing tokenizers, batching, input windows, or serving runtimes. Preserve the model's expected relationship among token IDs, position IDs, and masks.

When evaluating long inputs, measure performance at different lengths and locations instead of assuming that a supported maximum length means equal reliability everywhere. Position representation is one factor among many, including training coverage and attention behavior.

Check your understanding

Two token pairs have equal separation but occur at different sequence indices. Do they necessarily use the same absolute position vectors?

Answer: No. Absolute vectors correspond to individual indices. Equal separation is a relative-position property.

Related terms

Positional embeddings
Transformer
Attention mask
Context window
Tokenization
positional embedding
token embedding
relative position

Learn next

compare learned absolute tables with sinusoidal and relative-position schemes, then inspect how padding and cached generation affect position indices.

Sources

Primary source: Jurafsky and Martin, Speech and Language Processing, Chapter 7, positional embeddings: https://web.stanford.edu/~jurafsky/slp3/7.pdf . The learned-table description and comparison with alternatives were read. The vector and storage examples are original. This entry preserves the sequence-position scope of the supplied term.

Previous: Absolute frequency Next: Abstract word