Does document rotation change token counts?

The pure cyclic transformation does not; additional preprocessing could, so it should be documented separately.

Must the new start be a sentence boundary?

Not in the cited BART definition, which selects a token position uniformly.

Is the target the rotated text?

No. Denoising trains the model to reconstruct the original document from the rotated input.

AI Glossary / Natural language and speech / Advanced

Document-rotation denoising

Definition

Document-rotation denoising is a language-model pretraining task in which a document is cyclically shifted to start at a randomly chosen token, and the model learns to reconstruct the original document. It tests the model's ability to infer where the original sequence should begin.

What it means

Imagine cutting a sequence into two pieces and moving the first piece to the end. The tokens remain present, and their circular order remains unchanged, but the visible start of the document moves.

In the BART paper, document rotation is one of several experimental corruption methods. A token is chosen uniformly at random as the new starting point. An encoder receives the rotated sequence, and an autoregressive decoder learns to produce the original sequence.

This is not rotating a page image, changing a screen orientation or randomly shuffling all words. It is a particular way to construct a self-supervised training pair from text. The original sequence supplies the target, so an additional human-written label is not required for each example.

How it works

Represent a document as tokens x1 through xn. Choose a start index j. The corrupted input becomes xj through xn followed by x1 through x(j−1). The target remains x1 through xn.

The model encodes the corrupted input and is trained to predict the original output token by token. During standard teacher-forced training, each decoder prediction receives the preceding target tokens, while the loss rewards correct reconstruction.

Rotation preserves the number and identity of tokens. It changes the linear boundary: what was the end can now be adjacent to what was the beginning. A cut can occur inside a sentence because the source method chooses a token, not necessarily a sentence boundary.

Some examples may be ambiguous or trivial. Choosing the original first token leaves the sequence unchanged. Repetitive text can make several starts look alike. Successful use of this objective therefore depends on the broader data and training design.

Three practical examples

A pretraining researcher creates rotated text inputs to compare boundary reconstruction with token masking.
A teaching exercise asks learners to recover the opening of a short document whose first paragraph has been moved to the end.
A model-analysis team checks whether a rotation-only objective teaches useful downstream representations or mainly a narrow reordering skill.

Think of it this way

Think of a strip of text joined into a loop, then cut open at a different point. The writing has not disappeared, but a reader must work out where the original opening belonged.

The analogy captures cyclic order and the moved boundary. A neural model does not literally inspect a paper loop, and text alone may not uniquely reveal the intended beginning. The training target provides that reference even when the input is ambiguous.

Worked example

Use six labeled tokens: A, B, C, D, E, F. Choose token D, at position 4, as the new start.

The corrupted input is D, E, F, A, B, C. The target is A, B, C, D, E, F. All six tokens remain, and each appears exactly once. The model must undo the cyclic shift rather than invent missing tokens.

There are six possible starting positions when selection is uniform over these six tokens. Each has probability 1/6, including position 1, which produces the unchanged sequence.

If all six token labels are distinct, these choices produce six distinct rotations. If the sequence were A, B, A, B, A, B, different start positions could produce the same visible rotation. That distinction matters when reasoning about the number of unique corrupted examples, even though the sampling procedure still selects among six positions.

Illustrative case study

The fictional JuniperText team compares several denoising tasks for a document model. Rotation looks attractive because it preserves all input content and is easy to inspect.

The team first verifies the transformation with labeled tokens. It then tests real documents, including lists and repeated boilerplate where the original beginning may be difficult to identify. Downstream evaluation checks whether any learned boundary sensitivity transfers to the intended application.

Reviewers resist describing rotation as a proven improvement merely because it is included in a published model study. They compare it against masking and other corruptions under controlled conditions. This invented case reports no training result or business benefit.

Common misconceptions and mistakes

Rotation is random sentence permutation. A cyclic shift preserves circular token order; sentence permutation can rearrange sentences much more freely.
Denoising always removes or masks information. This corruption changes sequence boundaries while keeping the tokens.
Every corruption tested in BART became its preferred pretraining recipe. Experimental options must be distinguished from the configuration chosen for the main model.

Business use

Document rotation is most useful as a concept for understanding self-supervised objective design and controlled pretraining experiments. It makes clear that a training target can ask a model to repair structure rather than simply fill missing words.

For a practical experiment, define tokenization, special-token handling and the allowed rotation positions. Check that target sequences remain untouched and that truncation does not silently turn rotation into deletion. Evaluate downstream behavior instead of relying only on reconstruction loss. The original BART comparison found rotation alone weak relative to stronger corruption choices in its tested setting, so its inclusion is not a general recommendation.

Check your understanding

A sequence 1, 2, 3, 4, 5 is rotated to begin at 3. What are the input and reconstruction target?

Answer: The corrupted input is 3, 4, 5, 1, 2. The target remains 1, 2, 3, 4, 5.

Related terms

Self-supervised learning
Denoising autoencoder
BART
sequence-to-sequence model
text corruption
sentence permutation
token masking
pretraining objective

Learn next

Compare document rotation with sentence permutation and span infilling. For each transformation, state which information remains visible, what the decoder must reconstruct and whether the corrupted sequence keeps the original length.

Sources

Mike Lewis and colleagues, BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, ACL 2020: https://aclanthology.org/2020.acl-main.703/

Original paper, Sections 2.2 and 4: https://aclanthology.org/2020.acl-main.703.pdf

The token-level rotation definition, reconstruction objective and comparative findings were reviewed. The six-token calculation and fictional experiment are original. Rotation is presented as an evaluated corruption, not as the preferred BART training recipe.

Previous: Document retrieval Next: Document structuring