Must a corpus be very large?

No. A small, carefully selected corpus can suit a narrow study, although conclusions remain limited by its coverage and sample size.

Does a corpus need annotations?

No. Some contain raw or minimally processed text; others add detailed linguistic or task-specific labels.

Is a corpus the same as a knowledge base?

Not necessarily. A corpus collects language material, whereas a knowledge base organizes explicit information or claims. A system may derive one from the other.

AI Glossary / Natural language and speech / Beginner

Text corpus

Definition

A text corpus is an organized collection of texts assembled for a defined purpose, such as studying language, training a model or evaluating a language-processing system. A corpus can contain written documents, transcribed speech or both, often with metadata and annotations.

What it means

Corpus is singular; corpora is the usual plural. The important feature is not merely that many files exist, but that their contents and selection can be described. A collection of customer enquiries, a historical newspaper archive and a bilingual translation dataset are different corpora with different uses.

A corpus provides evidence about the language included in it. It does not automatically represent all speakers, all topics or present-day usage. Size alone cannot repair systematic gaps in how material was collected.

The British National Corpus is a historical example: its project documentation describes a broad sample of late twentieth-century British English, including written material and transcribed speech. That intended scope matters when deciding which questions the collection can answer.

How it works

Corpus construction begins with a purpose and a sampling plan. The plan states which sources, languages, genres, dates and populations should be included. Collection records preserve where each item came from and the conditions under which it can be used.

Processing can standardize file formats, remove unintended duplicates and identify document boundaries. Further steps may add sentence segmentation, tokenization or linguistic labels. Each change should be documented because it can affect later measurements.

Metadata links text to information such as source, date or genre. It allows analyses to compare relevant subsets instead of combining unlike material blindly. For machine learning, training, development and test partitions need suitable separation.

A release should include a description of scope, preprocessing, known limitations and version. Keeping the underlying collection reproducible helps explain why a result changed when the corpus changed.

Three practical examples

A help-desk team builds a corpus of authorized, appropriately prepared support messages to test enquiry categories.
A language researcher compares word use across genres in a documented collection.
A translation team assembles aligned source and target documents to train or evaluate bilingual processing.

Think of it this way

Think of a carefully labeled sample collection. Its usefulness depends on what was sampled, how the samples were handled and whether their labels remain trustworthy.

The analogy emphasizes selection rather than size. A large box of one kind of sample still cannot describe every possible environment, just as millions of similar documents need not represent a diverse language community.

Worked example

Seeing how duplicates change a corpus statistic

Imagine a small corpus of ten support messages. Four distinct messages mention a delayed delivery and six distinct messages concern other issues. The document-level delayed-delivery rate is 4/10 = 40%.

Now suppose each of the four delay messages is accidentally stored twice, while the other messages remain single copies. The collection contains 14 records, eight of which mention delay. The apparent rate becomes 8/14, approximately 57.1%.

The customers' original messages have not changed. The statistic changed because the collection contains uneven duplication. Removing the accidental duplicates restores the original ten-message denominator and 40% rate.

These figures describe mentions in the corpus, not the true rate of delivery failures among all orders. Customers who write to support are a selected group, and one order might produce several legitimate messages. Deduplication policy should therefore distinguish accidental copies from repeated real interactions.

Illustrative case study

A fictional learning-platform company builds an evaluation corpus from learner questions. Most available examples come from one popular introductory course, so an apparently broad test actually measures performance mainly on that course's vocabulary.

The team documents course, language and collection period, then reports performance separately for different subsets. It adds examples from underrepresented courses and keeps related conversation turns within the same evaluation partition.

The project does not claim that equal-sized subsets make the corpus universally representative. It states which learner interactions the evaluation covers and which remain weakly sampled. The case shows how a useful corpus description changes the meaning of a model-performance claim.

Common misconceptions and mistakes

Equating corpus size with representativeness. A large collection can still omit important groups or genres.
Treating publicly readable text as unrestricted project data. Collection and reuse conditions need to be established.
Splitting near-duplicate documents across training and test sets. That can inflate apparent generalization.
Assuming token counts are word counts under every scheme. Different tokenization and markup conventions produce different totals.
Forgetting that cleaning changes evidence. Removing punctuation or repetitions may discard information relevant to some tasks.

Business use

A well-designed corpus gives a language project a stable foundation for development and evaluation. Start with the intended decision: testing support-message routing requires different sampling from studying general vocabulary or fine-tuning a translation model.

Ask for source coverage, collection dates, duplication handling, access controls and the split strategy. Keep annotation quality and document quality visible as separate concerns. An accurately labeled collection of unrepresentative texts still gives a narrow test.

For recurring business use, version the corpus and record additions or removals. Compare models on a fixed evaluation set when appropriate, while maintaining a separate process to check whether the operational population has changed. This makes improvements easier to interpret and prevents silent shifts in the benchmark.

Check your understanding

A million documents all come from the same product forum. Does the large count establish that the corpus represents general everyday language?

Answer: No. The source restricts the topics, participants and communication style. Representativeness must be assessed against a stated target population.

Related terms

Metadata
Tokenization
Sampling bias
Parallel corpus
Dataset
corpus linguistics
annotation
train-test split

Learn next

Read sampling bias and data leakage. Write a short corpus description naming the intended population, included sources, excluded material, collection dates and rule for keeping related documents out of different data splits.

Sources

British National Corpus project, About the BNC: https://www.natcorp.ox.ac.uk/corpus/. Project documentation on corpus counts: https://www.natcorp.ox.ac.uk/corpus/index.xml?ID=numbers. These primary resources illustrate scope, metadata and counting distinctions; the support-message arithmetic and learning-platform case are original examples.

Previous: Text classification Next: Text graph