Does probing normally retrain the original model?

In the common diagnostic setup described here, the original model stays fixed and the separate probe is trained.

Does a successful probe prove the model uses that information?

No. Recoverability and causal use are different claims.

Must probes always be linear?

No. Probe design depends on the question; linear probes are one useful restricted choice, not the definition of probing.

AI Glossary / Natural language and speech / Intermediate

Probing

Definition

Probing is a method for investigating what information can be recovered from a model's internal representations. In a common NLP setup, researchers keep the original model fixed and train a separate predictor, called a probe, to infer a property such as part of speech from its activations. The probe's performance is evidence about recoverable information under that experimental setup.

What it means

The target of the investigation is the representation, not simply whether a new classifier can achieve a good score. A powerful probe may learn relationships that make interpretation difficult, while a restricted probe may fail to recover information that is present in a less accessible form.

Consequently, a probing result needs a clearly stated question. “Can a linear classifier recover this label?” is narrower than “Does the representation contain any information about this property?” Neither question, by itself, establishes whether the original model uses that information to produce its outputs.

How it works

Researchers select a property, a model layer or activation location, and labeled examples. They extract representations from the fixed model, train the probe on one portion of the data, and evaluate on held-out examples. The chosen labels, representation, split, and probe architecture all help define what the result means.

Control tasks and baselines test alternative explanations. For example, a probe may exploit word identity rather than contextual grammatical structure. Hewitt and Liang propose control tasks using random labels associated with word types to investigate such memorization, comparing performance on the linguistic and control tasks.

There is no single universally correct probe complexity. Pimentel and colleagues frame probing as information estimation and argue for strong predictors under that objective. The practical lesson is to match the probe and controls to the scientific claim. Results about predictive accessibility should be distinguished from causal claims, which require additional evidence about how changing information affects the original model's behavior.

Three practical examples

1. Layer comparison: Researchers test whether a particular grammatical label is recoverable from different layers using the same data and evaluation setup.

2. Representation diagnosis: A team investigates whether contextual features add useful predictive information beyond a baseline using word identity alone.

3. Interpretability review: Reviewers check whether a claim that a model “understands syntax” is actually supported only by a narrower probe-classification result.

Think of it this way

Think of a probe as a decoder trying to read one kind of information from a stored signal. Successful decoding shows that the chosen procedure can recover something useful. The analogy is limited because the decoder itself learns from examples, and recovering a signal does not show that another part of the system relies on it.

Worked example

Consider an original hypothetical experiment with 200 held-out labeled token examples. Probe A correctly predicts the linguistic label for 180 examples, giving 90% accuracy. On a separately constructed control task of the same size, it predicts 150 labels correctly, giving 75% accuracy.

The simple linguistic-minus-control accuracy difference is 90−75=15 percentage points. Probe B achieves 170/200=85% on the linguistic task and 100/200=50% on the control, a difference of 35 percentage points. Thus A has higher linguistic accuracy, while B has the larger difference under this stipulated comparison.

These invented numbers do not establish that B is universally better. The interpretation depends on what the control measures, how it was constructed, and the research question. The calculation also cannot show that the original model uses the probed property: neither experiment changes the model's internal information and observes the resulting output behavior. It compares predictive measurements, not causal interventions or general intelligence.

Illustrative case study

Fictional case: Birch Research presents a high-scoring grammatical probe as proof that its assistant reasons with grammar. An internal reviewer asks for the layer, data split, probe capacity, and comparison baselines. The team finds that its first report did not distinguish easily memorized word types from context-dependent uses.

It revises the experiment and narrows the claim to the information recoverable under the tested conditions. It plans separate work for causal questions. This fictional case illustrates better interpretation and reporting, not a finding about any real assistant or a promised outcome from adding controls.

Common misconceptions and mistakes

Do not treat probe accuracy as a direct measurement of the original model's task accuracy. The probe predicts an auxiliary label from a selected representation; the original system may perform a different task with additional computations.

A failed simple probe also does not prove that the information is entirely absent. It may be inaccessible to that probe or inadequately supported by the data. Conversely, success does not automatically prove causal use. Avoid slogans that the simplest or most accurate probe is always best without specifying the purpose of the experiment.

Business use

For business-facing model assessments, require the claim to match the evidence. A probing study can inform a research investigation, but it should not substitute for representative tests of the deployed task. Record the frozen model version, representation location, labels, train/test separation, baselines, and uncertainty so later reviewers can understand and reproduce the narrow conclusion.

Check your understanding

In the toy comparison, which probe has higher linguistic accuracy, and which has the larger linguistic-minus-control difference?

Answer: A has higher linguistic accuracy, 90% versus 85%. B has the larger difference, 35 versus 15 percentage points. The preferred interpretation depends on the experimental objective.

Related terms

Representation
Activation
Interpretability
linear probe
control task
selectivity
causal intervention
information theory

Learn next

Learn how a supervised classifier is evaluated, then study probing with frozen representations. Compare a linguistic task with carefully designed controls and baselines. Finally, distinguish questions about information accessibility from questions about causal use before interpreting a high score as an explanation of model behavior.

Sources

Jurafsky and Martin, Speech and Language Processing, chapter 10, probing section: https://web.stanford.edu/~jurafsky/slp3/10.pdf

Hewitt and Liang, Designing and Interpreting Probes with Control Tasks, primary abstract consulted: https://aclanthology.org/D19-1275/

Pimentel and colleagues, Information-Theoretic Probing for Linguistic Structure, primary abstract consulted: https://arxiv.org/abs/2004.03061

The numerical comparison and company case are original illustrations. They do not reproduce the cited papers' experiments or reported scores.

Previous: Probalistic imputation Next: Probit model