Common Crawl

Definition

Common Crawl is a nonprofit organization and the large, openly available archive of web pages it collects by crawling the public internet and releases for anyone to download. It is not a single AI model and not a product you subscribe to. Many language models use it, in whole or in filtered form, as one source of training text, though each developer decides how much to use and how to clean it.

What it means

Language models learn from enormous amounts of text. Someone has to gather that text first. Common Crawl does the gathering step in public: it visits web pages at scale, saves copies, and publishes the results in batches so researchers and companies do not each have to build their own crawler.

For a business reader, the useful idea is that an AI model's behavior depends partly on what it read. A general web archive reflects the web: broad, uneven, full of useful facts and also errors, spam, duplicates and bias. Knowing that Common Crawl exists helps you ask a sharper question of any AI vendor: what data was this model trained on, and how was it filtered?

How it works

A web crawler is a program that starts from a list of pages, downloads each one, finds the links on it, and follows those links to more pages. Common Crawl runs this process repeatedly and stores the raw results in large files. The files typically hold the page content as it was fetched, along with metadata such as the address and the date. Each release is a snapshot of what the crawler reached in that period, so it is a sample of the web and not a complete copy of it.

The raw archive is not ready to train on. Developers who use it usually apply a cleaning pipeline. Typical steps include stripping menus and page furniture to keep the main text, removing duplicate pages, filtering out low-quality or unwanted material, and detecting the language. The exact steps vary by developer and are often only partly documented, so two models trained on the same archive can behave quite differently.

The crawler is also meant to respect site owners' stated wishes, such as instructions in a site's robots.txt file about which pages automated visitors may fetch. Owners can also ask crawlers not to collect their content. Whether and how that content can be used for training is a separate legal and policy question that differs by jurisdiction and is still being debated.

Three practical examples

Model developer: A small team building a language model needs billions of words and has no crawler of its own. It downloads a Common Crawl snapshot, removes duplicates and low-quality pages, and uses what remains as part of its training mix alongside licensed and curated sources.
Researcher: A university group studies how often certain topics appear on the public web over time. It uses several Common Crawl snapshots from different periods to compare word frequencies, since the archive keeps dated, repeatable samples.
Compliance team: A bank's risk group is reviewing a vendor's AI tool. Its questionnaire asks whether the model was trained on web-scale data such as Common Crawl, how personal data and copyrighted material were handled, and what the vendor can tell the bank about filtering.

Think of it this way

Think of Common Crawl as a huge public library photocopy room where someone has photocopied a very large share of the pages on the open shelves and left the stacks out for anyone to borrow. The analogy stops working because a library chooses its books with care, while a crawl copies what it can reach, good and bad alike, so the quality work happens afterward.

Worked example

A fictional team, Northfield Labs, downloads a snapshot of 1,000,000 raw pages to train a small model. Step 1: remove pages that are mostly menus or boilerplate. This drops 30 percent, which is 1,000,000 x 0.30 = 300,000 pages, leaving 700,000. Step 2: remove duplicates. This drops 20 percent of the remaining pages, which is 700,000 x 0.20 = 140,000 pages, leaving 560,000. Step 3: apply a quality filter that keeps 50 percent, which is 560,000 x 0.50 = 280,000 pages. The team keeps 280,000 of 1,000,000 pages, or 28 percent. If the average kept page holds 600 words, the training set is 280,000 x 600 = 168,000,000 words. The lesson: the headline size of a raw archive says little about the size or quality of what a model actually learns from.

Illustrative case study

Fictional case: Harbor & Pine, a made-up regional insurance broker, wanted a model to draft plain-language explanations of policy terms. A contractor proposed fine-tuning an open model whose training mix included a filtered web archive. In testing, the drafts sometimes repeated phrases from competitor marketing pages and occasionally stated outdated rules as current. The team realized the model had absorbed web text of mixed age and reliability. They changed the approach: they kept the general model for fluent writing but added retrieval from the broker's own approved policy documents, required every draft to cite the document it drew from, and had a licensed agent review each output before sending. Error reports in a month of review fell from 14 flagged drafts out of 200 to 3 out of 200, and the team added a quarterly check that the source documents were current.

Common misconceptions and mistakes

Common Crawl is an AI model you can use. It is a data archive. Models are trained on data like it, but the archive itself does not answer questions.

Everything in it is free to use for any purpose. Being publicly downloadable is not the same as being cleared for every use. Copyright, privacy and terms-of-use questions still apply and vary by jurisdiction.

It contains the whole internet. It is a sample of publicly reachable pages, and it misses content behind logins, paywalls and blocked crawlers.

More web data always makes a better model. Quality, filtering and balance matter at least as much as volume, and unfiltered web text can add noise and bias.

Business use

Most businesses will never download Common Crawl directly. The practical use is as a due-diligence topic. When you evaluate a vendor or a model, ask what the training data sources were, whether web-scale archives were used, how personal data and copyrighted content were handled, and whether the vendor offers any indemnity or documentation. For your own tasks, plan to ground a general model in your own trusted documents rather than relying on whatever it absorbed from the web.

Measure what matters to your use: accuracy against a test set of your own questions, how often outputs need correction, and whether sources can be traced. Costs and risks are mostly indirect. They include legal and reputational exposure if a model reproduces protected or sensitive text, bias inherited from uneven web content, and the effort of governance and review. Teams that train their own models face real costs for storage, computing and cleaning, which can be large and should be estimated before starting.

Check your understanding

What is Common Crawl, in one sentence?

Answer: A nonprofit's open archive of web pages collected by crawling, often used as one source of training text for language models.

In the worked example, how many pages remain after removing boilerplate and duplicates?

Answer: 560,000 pages: 1,000,000 minus 300,000 leaves 700,000, and minus 140,000 more leaves 560,000.

Why is raw crawl data not used directly for training?

Answer: It contains duplicates, boilerplate, spam and low-quality or unwanted material, so developers filter and clean it first.

What practical question should you ask an AI vendor about this topic?

Answer: What data the model was trained on, how it was filtered, and how personal and copyrighted content were handled.

Related terms

Training Data
Pretraining
Large Language Model
Web Crawler
Data Quality
Retrieval-Augmented Generation
Fine-Tuning
Data Provenance

Learn next

Next, read about training data and pretraining to see how raw text becomes a working model, then retrieval-augmented generation to see how businesses add their own trusted documents. A good practical step is to write a short vendor question list about data sources and review it with your legal or compliance contact.

Sources

Google, Machine Learning Glossary: https://developers.google.com/machine-learning/glossary Jurafsky and Martin, Speech and Language Processing: https://web.stanford.edu/~jurafsky/slp3/ UK Information Commissioner's Office, Guidance on AI and data protection: https://ico.org.uk/ The sources were consulted for general definitions of training data and for data protection considerations; the examples, analogy, worked example and case are original educational illustrations. Checked 5 Nov 2026.

Previous: Common cause Next: Common data model