Does the model receive the original sentence numbers?

Not in the basic text-reconstruction setup. The reordered text is the input, and the original document supplies the target.

Can sentence permutation leave a document unchanged?

Yes, if the sampled permutation is the identity or if repeated identical sentences make different permutations look the same.

Is this the same as correcting a real document's order?

It is a synthetic pretraining task. Applying a model to reorder an actual document would require a separate task definition and evaluation.

AI Glossary / Natural language and speech / Advanced

Sentence-permutation denoising

Definition

Sentence-permutation denoising is a pretraining task that shuffles a document's sentences and trains a model to reconstruct their original order. It preserves the words within each sentence while disrupting the document's larger-scale sequence.

What it means

Many documents depend on sentence order. An introduction supplies context, a later sentence refers back to it, and a conclusion follows supporting evidence. Shuffling sentences creates a reconstruction problem involving those relationships.

In the BART study, sentence permutation is one of the input-corruption methods used to investigate denoising pretraining. The original text is the target, and the reordered sentences form the input. The paper describes dividing documents into sentences using full stops before shuffling them.

This differs from permutation language modeling, where prediction order changes but original text positions remain intact. Here the input document itself is reordered. It also differs from document rotation, which moves a boundary through a sequence without creating an arbitrary sentence ordering.

How it works

Split a document into sentence units using a documented segmentation rule. Sample a permutation of their indices and concatenate the sentences in that order to create the corrupted input.

An encoder processes that reordered text. An autoregressive decoder is trained with a reconstruction loss to output the original document. The target includes the original order and wording, rather than merely a list of sentence indices.

The pure transformation retains sentence content and token count, apart from any additional formatting or special-token handling. It may be combined with other corruptions, such as span infilling, which create a different and potentially harder input.

Not all documents provide a unique ordering signal. Independent bullet-like statements can remain plausible in several orders. Segmentation mistakes also change the task: an abbreviation mistaken for a sentence boundary can split a unit that should have stayed together.

Three practical examples

A pretraining experiment shuffles three paragraphs' sentences and asks a model to reconstruct the original text.
A researcher compares sentence shuffling alone with a combination of shuffling and masked spans.
A data-quality review checks whether sentence boundaries were detected correctly before generating denoising examples.

Think of it this way

Think of cutting a short article into sentence strips and mixing the strips before asking someone to rebuild it. Pronouns, time markers and logical transitions can help identify the original order.

The analogy describes sentence-level disruption. A language model is optimized through token prediction, and a coherent reconstruction is not necessarily the exact original. Some texts allow multiple sensible arrangements even when only one is the training target.

Worked example

Consider three invented sentences: S1 says, The team received a sample. S2 says, They inspected it carefully. S3 says, The inspection revealed a crack.

A sampled order (3, 1, 2) gives the input: The inspection revealed a crack. The team received a sample. They inspected it carefully. The target remains S1, S2, S3.

With three distinct sentences, there are 3! = 6 possible permutations, including the unchanged order. Under uniform sampling over all six, the unchanged order has probability 1/6. The (3, 1, 2) order is also a cyclic rotation, but other permutations, such as (2, 1, 3), are not.

The words inside each sentence remain in place. If the model restores only a plausible order but changes crack to scratch, it has not exactly reconstructed the target. Conversely, correct wording in the shuffled order still misses the ordering part of the task.

Illustrative case study

The fictional AspenCorpus team prepares a denoising experiment using technical reports. A simple sentence splitter treats some abbreviated units and initials as sentence endings, producing fragments that are then shuffled independently.

The team inspects examples and revises its preprocessing rules. It also includes documents with weak ordering cues so that reconstruction difficulty is not confused with segmentation failure.

When evaluating the model, it separates this pretraining task from the downstream question-answering task it ultimately cares about. This invented case highlights how a corruption method depends on its text units. It reports no measured improvement or claim that sentence shuffling alone is sufficient pretraining.

Common misconceptions and mistakes

Sentence permutation shuffles every word. It changes sentence order while preserving the internal ordering of each sentence unit.
Any coherent output is the training target. The reconstruction objective rewards the original document, not an arbitrary plausible rewrite.
A published corruption option is always beneficial alone. The BART comparison found sentence shuffling alone weaker than stronger corruption configurations in its evaluated setting.

Business use

Sentence-permutation denoising helps explain how pretraining can target document organization rather than only local word prediction. It is relevant to experiments on discourse-sensitive representations and reconstruction objectives.

Define sentence boundaries and record whether permutations include the unchanged order. Inspect inputs after all transformations, especially when combining shuffling with masking or truncation. Compare downstream performance under consistent training budgets. The BART paper's main recipe combined sentence permutation with text infilling; that historical configuration should not be read as proof that the same combination is optimal for every corpus, tokenizer or task.

Check your understanding

A document's sentences are reordered, but each sentence's words remain unchanged. Has document content been deleted by the pure permutation operation?

Answer: No. The operation changes order. Deletion or masking would be additional corruption steps.

Related terms

Discourse
Reconstruction loss
Permutation language modeling
BART
denoising pretraining
sentence segmentation
document rotation
text infilling

Learn next

Compare the six permutations of three distinct sentences with the three possible sentence-level cyclic rotations. Then inspect how adding a missing span changes the information available to the reconstruction model.

Sources

Mike Lewis and colleagues, BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, ACL 2020: https://aclanthology.org/2020.acl-main.703/

Original paper, Sections 2.2 and 4: https://aclanthology.org/2020.acl-main.703.pdf

The sentence-permutation definition, reconstruction framework and comparison with combined corruptions were reviewed. The three-sentence example and permutation count are original. Findings are scoped to the cited study.

Previous: Sentence extraction Next: Sentence separation