What is the difference between a corpus and a dataset?

Dataset is broader; corpus usually refers to a collection of language material, although usage varies.

Can a corpus include speech?

Yes. Spoken-language corpora can include audio, transcripts and associated annotations.

Does every corpus need labels?

No. Many analyses and learning methods use unlabeled language material.

AI Glossary / Graphs and symbolic AI / Beginner

Corpus

Definition

A corpus is a collection of language material assembled for analysis, research or computational use. It may contain written text, transcribed speech or other recorded language, often accompanied by metadata and annotations. The plural is corpora.

What it means

A corpus provides examples of language in use. It can support vocabulary analysis, model training, evaluation, retrieval or linguistic research. Its usefulness depends on what it contains and how it was collected, not just on its size.

A general corpus may include several genres, while a domain corpus focuses on material such as product manuals or support conversations. A parallel corpus aligns material across languages. An annotated corpus adds information such as part-of-speech labels, entities or syntactic structure.

A corpus is not automatically a representative sample of all language or all users. A large collection of formal documents may poorly reflect short conversational messages. The collection's time period, geography, language varieties and selection rules shape the patterns an analyst can observe.

It is also distinct from a trained model. The corpus is data; a model learns or computes from it. A retrieval system may search a corpus without changing model weights.

How it works

Building a useful corpus begins with a purpose and sampling plan. Specify the kinds of material, inclusion criteria, unit of collection and intended use. Preserve document identifiers and relevant metadata so later analysis can trace results back to their context.

Preparation may include decoding files, separating documents, removing duplicates and tokenizing text. These operations change what counts as an item or word. A token can be a word, punctuation mark or subword unit depending on the tool; token totals are meaningful only with the counting convention stated.

Annotations need definitions and quality checks. Two annotators can disagree because the task is ambiguous, not simply because one was careless. Retaining original material separately from derived versions makes revisions easier to investigate.

For model evaluation, create splits that reflect the intended generalization question. Randomly distributing near-duplicate messages from the same conversation across training and testing can make the test less independent than it appears. Splitting by document, conversation, source or time may be more appropriate.

Version the corpus and record transformations. Otherwise, two teams can claim to use the same collection while analyzing different subsets or preprocessing outputs.

Three practical examples

A terminology team studies a corpus of product manuals to identify common phrases and inconsistent naming.
A support-model team builds a labeled corpus of conversations, preserving conversation boundaries when designing evaluation splits.
A bilingual search project uses aligned documents to study how equivalent concepts are expressed in two languages.

Think of it this way

Think of a carefully assembled library shelf chosen to answer a particular research question. The books on that shelf determine which language patterns can be observed.

The analogy does not imply that every corpus is curated like a library or that its contents are automatically authoritative. A corpus can include informal, erroneous or duplicated material, and its documentation should make those characteristics visible.

Worked example

Counting material without confusing units

Suppose a teaching corpus contains three documents with 100, 200 and 700 word tokens under one stated tokenizer. The corpus contains 3 documents and 1,000 tokens. If a term occurs 2, 4 and 14 times respectively, it occurs 20 times overall, or 20 per 1,000 tokens.

The same term appears in all three documents, so its document frequency is 3 out of 3. Token frequency and document frequency answer different questions: repeated mentions within one document increase the first without necessarily increasing the second.

Now suppose the longest document is accidentally included twice. The collection becomes 4 stored documents and 1,700 tokens, with 34 term occurrences. The relative token frequency remains 20 per 1,000 in this constructed example, so that single statistic would not reveal duplication.

Other vocabulary and evaluation results could still be distorted. Corpus auditing therefore needs identifiers and duplication checks as well as aggregate counts. A convenient-looking frequency rate is not sufficient evidence that the underlying collection is sound.

Illustrative case study

Fictional case: Birch Support trains a classifier on a corpus assembled from exported tickets. Its evaluation looks strong, but many replies repeat the original customer message, and parts of the same conversation appear in both training and test sets.

The team reconstructs conversation groups, records which text comes from customers versus staff and redesigns the split around those groups. It also checks whether the test period resembles the intended deployment period.

The lesson is that corpus construction is part of the experiment. File count alone does not reveal how many independent examples the system has seen.

Common misconceptions and mistakes

A corpus must contain millions of documents. Small purpose-built collections can be useful.
More text always improves a model. Relevance, duplication, quality and evaluation design matter.
Tokens and words always mean the same thing. Tokenization conventions differ.
Public availability settles every reuse question. Intended use and applicable permissions still need to be established.
A corpus is objective merely because it is collected data. Selection and annotation choices affect what it represents.

Business use

A well-documented corpus can become a reusable foundation for search, analytics and language-system development. Business teams should know which populations, channels and time periods it covers before interpreting model performance or language statistics.

Maintain a clear inventory of sources, permitted uses, collection dates, transformations and known gaps. Separate training, evaluation and retrieval roles where their requirements differ. When the content changes, record a new version and assess whether earlier results remain comparable. Corpus maintenance is often as important as the first collection effort.

Check your understanding

A word appears 50 times in one document and nowhere else in a ten-document corpus. Are its token frequency and document frequency the same?

Answer: No. Its occurrence count is 50, while its document frequency is one document, or 10 percent of the documents.

Related terms

Text corpus
Tokenization
Parallel corpus
Data leakage
Dataset
annotation
document frequency
corpus versioning

Learn next

Create a small collection with document identifiers and a stated tokenizer. Count both term occurrences and documents containing each term. Add a duplicate document and observe which statistics change, then explain what the collection can and cannot represent.

Sources

NLTK authors, Accessing Text Corpora and Lexical Resources, covering corpus organisation, genres and access methods: https://www.nltk.org/book/ch02.html

NLTK authors, Processing Raw Text, covering text preparation and tokenisation: https://www.nltk.org/book/ch03.html

The frequency calculation and fictional support-data review are original illustrations.

Previous: Coronal sound Next: Correct sampling