Is pre-training always done from scratch?

Initial pre-training often starts from untrained parameters, but continued pre-training extends an existing checkpoint. Reports should distinguish the two.

Does pre-training require manually labeled answers?

Not necessarily. Common language objectives create prediction targets from the text itself, although training approaches vary.

Is fine-tuning required afterward?

Not always. Some pretrained models can be used directly for supported purposes, while others need adaptation or additional system components for the desired application.

AI Glossary / Generative AI and language / Beginner

Pre-training

Definition

Pre-training is an earlier stage of model training that learns broadly useful patterns before the model is used directly or adapted to more specific tasks. In language AI, it commonly involves learning from substantial collections of text using objectives such as predicting tokens from context. The resulting pretrained model provides a starting point for later work. Pre-training is a learning process, not simply uploading documents or writing instructions in a prompt.

What it means

A language application needs more than a list of task-specific answers. It benefits from representations of language and regularities learned across many examples. Pre-training develops these capabilities before a particular organization adapts or deploys the model.

The word describes a stage relative to later use. It is not restricted to language models: image and other models can also be pretrained. This entry focuses on the language setting appropriate to the supplied category.

Pre-training alone does not guarantee a helpful conversational assistant. Instruction-following, application behavior, and other requirements may involve additional training or system design. Nor does a broad training collection make the resulting model an authoritative source of facts.

How it works

A training team selects and prepares data, chooses a model architecture and objective, and converts inputs into the model's expected representation. Training repeatedly computes predictions, measures loss, and updates parameters to improve the objective.

In a next-token objective, preceding tokens provide context for predicting the next token. A masked-language objective instead asks the model to recover hidden text from available context. These are examples, not an exhaustive list of pre-training methods.

The team monitors learning on held-out data and saves checkpoints. A checkpoint may then support downstream fine-tuning, representation extraction, or another supported use. Data composition, duplication, contamination of evaluation material, and mismatch with intended applications influence how the resulting model should be assessed.

Three practical examples

1. A research team pretrains a text encoder, then adapts separate copies for document classification and information extraction.

2. A language-model developer trains on broad text before conducting a separate instruction-tuning stage with examples of requested behavior.

3. A domain project continues training an existing checkpoint on relevant text, then evaluates whether that additional stage helps its specialized task without unacceptable regressions elsewhere.

Think of it this way

Think of pre-training as preparing a reusable set of building materials before constructing a particular product. Later work can draw on what has already been prepared instead of starting with nothing.

The analogy has limits. A model learns distributed numerical relationships rather than neatly labeled facts or components. Later adaptation can change behavior in unexpected ways, and the learned patterns may include errors or biases present in the training material.

Worked example

Consider a tiny next-token teaching example: the three tokens “orders arrive today.” Ignore punctuation, start markers, and end markers for simplicity. Two prediction tasks are visible: predict “arrive” from “orders,” and predict “today” from “orders arrive.”

Suppose the model assigns the correct tokens probabilities of 0.25 and 0.50. Using natural logarithms, their losses are approximately 1.386 and 0.693. The average is about 1.040.

If a later checkpoint assigns probabilities 0.50 and 0.80, the corresponding average is about 0.458. Lower loss means the model assigned more probability to these observed targets. This tiny calculation explains the learning signal; it does not establish generalization, factual understanding, or suitability for a customer-facing task. Real evaluation needs separate, representative examples.

Illustrative case study

Fictional case: Lake Research wants a model that handles technical maintenance language. It begins from an existing pretrained checkpoint and considers further training on approved manuals.

Before doing so, the team separates manuals used for development from documents used for evaluation and checks for near-duplicate passages. It then measures both domain-language prediction and a downstream extraction task.

An improvement in text-prediction loss is treated as encouraging but insufficient. The team also checks whether extracted equipment identifiers remain accurate. Its decision depends on the application outcome, because better performance on a pre-training objective does not automatically mean fewer operational mistakes.

Common misconceptions and mistakes

Pre-training is not the same as prompting. Ordinary prompting supplies input at inference time; it does not itself update the model's stored parameters.

It is also different from retrieval. A retrieval system obtains relevant documents for a request, whereas pre-training changes parameters through learning. The two can be combined in an application.

Another mistake is treating “unlabeled text” as “no learning target.” Self-supervised objectives derive targets from the data itself. Finally, training on more text is not automatically beneficial: quality, diversity, repetition, and relevance matter alongside quantity.

Business use

Most application teams can assess existing pretrained models before considering an expensive pre-training project. The relevant question is whether available models can satisfy the task through suitable application design or adaptation.

When additional pre-training is considered, define the expected benefit and evaluation plan first. Include data preparation, computing resources, licensing, documentation, and downstream testing in the cost. Preserve the provenance of the starting checkpoint and training stages so later teams can distinguish original capabilities from changes introduced by the project.

Check your understanding

A team adds company documents to a retrieval index but does not update model parameters. Has it pretrained the language model on those documents?

Answer: No. It has changed the information available through retrieval. Pre-training would involve a learning process that updates parameters using a training objective.

Related terms

Self-supervised learning
Masked language modeling
Checkpoint
Loss function
Fine-tuning
Instruction tuning
Pretrained language model
next-token prediction
continued pretraining
retrieval

Learn next

Trace one model's documented lifecycle from data and training objective through checkpoint and downstream adaptation. Write down which stage changes parameters and which merely supplies inputs. Then design separate measures for the training objective and the intended business task, so progress on one is not mistaken for proof of success on the other.

Checkpoint

Sources

Sources: Google Machine Learning Glossary, https://developers.google.com/machine-learning/glossary — primary terminology for pre-training and later adaptation. Devlin and colleagues, BERT, https://arxiv.org/abs/1810.04805 — original author abstract on pretraining bidirectional language representations. Jurafsky and Martin, Speech and Language Processing, language-model chapter, https://web.stanford.edu/~jurafsky/slp3/3.pdf — primary textbook treatment of language probabilities and evaluation. The loss calculation and fictional case are original educational examples.

Previous: Pre-trained model Next: Precedence constraints