Are both streams initialized from the same token embedding?

No. In the cited design, content uses the token embedding while the query stream starts from a learned vector and gains target-position information through the architecture.

Can the query use words to the right in the original sentence?

Yes, if those positions occur earlier in the sampled prediction order and are therefore allowed context.

Does two-stream mean twice as many independent model parameters?

No. The streams use shared parameters in the described mechanism, although maintaining additional representations has computational costs.

AI Glossary / Natural language and speech / Advanced

Two-stream self-attention

Definition

Two-stream self-attention is XLNet's mechanism for maintaining separate content and query representations at token positions. The content stream includes a token's identity, while the query stream predicts that token using allowed context and its position without seeing the token itself.

What it means

Permutation language modeling creates a particular challenge. When predicting a token, the model needs to know which original position is being requested, but it must not see the answer already stored at that position. Later predictions, however, should be able to use that token as context.

A single representation cannot simply include and exclude the same token content for both purposes. XLNet therefore maintains two sets of hidden states with different access rules. The content states carry known token information, while query states support target-aware prediction.

The streams share model parameters in the cited design. They are not two independent language models, and they do not mean one stream reads left-to-right while another reads right-to-left.

How it works

For a sampled prediction order, a content representation at a position can attend to content states from earlier positions in that order and to itself. It therefore represents the permitted context plus its own token.

A query representation can use its target position and content states from strictly earlier positions in the prediction order. It cannot attend to its own token's content state. The prediction distribution is calculated from this query representation.

Initially, content states come from token embeddings, while query states begin from a learned vector. Positional information makes the query target-aware. Across layers, both streams follow masks that preserve their distinct information boundaries.

The distinction must hold through all paths, not merely by hiding one direct connection. Earlier content states must themselves exclude later token content, otherwise information could leak indirectly into a target query.

Three practical examples

An XLNet implementation uses a query state to predict an interior token while allowing its content state to support later predictions.
A researcher draws separate masks for content and query streams to check which diagonal connections are permitted.
A model review verifies that shared parameters do not imply shared access to the target token's identity.

Think of it this way

Think of a numbered answer card with two views. A question view shows its number and previously revealed clues, while an information view includes the answer for use after that card is revealed.

The analogy captures the need for different access rights at the same position. The streams are neural representations, not independent documents, and attention masks must enforce the restriction consistently through successive layers.

Worked example

Take four original token positions and the sampled prediction order 2, 4, 1, 3. Ignore any external memory for this toy example.

When predicting position 1, the query stream may use the content of positions 2 and 4, because they come earlier in the sampled order. It also knows that the requested target is position 1. It must not use token x1 itself or position 3's content.

The content stream at position 1 may use positions 2, 4 and 1. This allows x1 to become part of the context for the later prediction at position 3.

Across all four positions, content access sets have sizes 1, 2, 3 and 4 in prediction order, for ten permitted position-pair connections per layer. Query access sets have sizes 0, 1, 2 and 3, for six such connections. The difference is the four self-content connections. These counts illustrate the masks, not total attention runtime or full implementation details.

Illustrative case study

The fictional HazelModels team builds a teaching implementation of XLNet-style pretraining. Its first version uses the content representation directly to predict the token at the same position.

The loss quickly becomes suspiciously low because the representation contains the answer. The team introduces a separate query stream and checks a four-token example layer by layer. It confirms that earlier context states cannot carry later target content through an indirect path.

The revised experiment separates information-flow correctness from the eventual model-quality comparison. This invented case reports no real training result; it illustrates why target awareness and target-content exclusion are different requirements.

Common misconceptions and mistakes

The two streams are forward and backward language models. Their distinction is content visibility, not two fixed reading directions.
Removing the direct self-attention link always prevents leakage. Indirect paths through other representations must also respect the prediction order.
The query stream has no positional information. Knowing the target position is essential to distinguish which token is being requested.

Business use

Two-stream self-attention is useful for understanding how architectural details support a training objective. It shows that preventing answer leakage can require separate representations, not merely a new loss formula.

For implementation reviews, specify the prediction order, initial states, positional signals and masks. Trace a tiny sequence through multiple layers, including any recurrence or memory used by the actual model. Keep the distinction between query and content states clear in diagrams and documentation. A structurally valid implementation still needs evaluation on the intended downstream tasks; the attention design alone does not guarantee a performance benefit.

Check your understanding

Under prediction order 3, 1, 2, which current-sequence token contents may the query for position 1 access?

Answer: Only position 3's content. Position 1's own content is excluded, and position 2 has not yet become available in that order.

Related terms

Permutation language modeling
Attention mask
Positional encoding
XLNet
content stream
query stream
target leakage
Transformer-XL

Learn next

Read permutation language modeling first if prediction order is unfamiliar. Then construct content and query visibility sets for three positions and check that the query cannot recover its own answer through any allowed context state.

Permutation language modeling

Sources

Zhilin Yang and colleagues, XLNet: Generalized Autoregressive Pretraining for Language Understanding: https://arxiv.org/abs/1906.08237

Original paper, Section 2.3 and Appendix A.2: https://arxiv.org/pdf/1906.08237

The original content/query definitions, initialization and shared-parameter update equations were reviewed. The four-position visibility sets and connection counts are original, explicitly excluding external memory and other implementation costs.

Previous: Two-sided test Next: Two-Tower Embedding Model