Must the original model have been trained with concept labels?

No. CAV analysis can use concept examples after the model has been trained.

Is the CAV itself the TCAV score?

No. The CAV is a direction; TCAV uses that direction to summarize output sensitivity over examples.

Can the same concept have different CAVs?

Yes. Changing the layer, model, examples, or comparison set can change the learned direction.

Concept activation vector

Definition

A concept activation vector, or CAV, is a direction in a neural network's internal activation space associated with a concept defined through example inputs. It is commonly obtained by training a linear classifier to distinguish activations from concept examples and comparison examples, then taking the separating direction. CAVs support analysis of whether a model's output is sensitive to a human-chosen concept. They do not prove that the model understands that concept as a person does.

What it means

A neural layer represents an input as many numbers. Rather than interpreting every number separately, an analyst can ask whether a direction through that representation corresponds to an idea such as a striped texture.

The concept is operationally defined by the examples supplied. If all the striped examples also have dark backgrounds, the direction may reflect that unintended association. A CAV therefore connects a specific collection of examples, a particular model, and a selected layer. It is not a universal vector that can be moved unchanged between unrelated models or treated as a dictionary definition.

How it works

Collect concept and comparison examples, pass them through the model, and record activations at a chosen layer. Fit a linear separator to these two activation sets. Its normal vector supplies the CAV direction, oriented consistently towards the concept.

One use is TCAV, Testing with Concept Activation Vectors. It evaluates a class output's directional sensitivity along the CAV across relevant examples. A common TCAV score counts the proportion with positive directional sensitivity.

The original method also checks consistency across repeated comparison sets to reduce spurious interpretations. Inspecting concept examples, layer choice, separator quality, and repeatability is necessary before giving the resulting number a meaningful label.

Three practical examples

1. Texture hypothesis: an image analyst tests whether a product classifier is sensitive to striped packaging, using examples designed to avoid making color or background the accidental defining feature.

2. Dataset investigation: researchers compare concepts associated with different photographic conditions to investigate whether a model may be responding to an unintended capture artifact.

3. Layer comparison: students learn concept directions at several layers and examine how the same chosen concept is represented differently, without assuming that one layer provides the complete explanation.

Think of it this way

Think of placing a labeled compass direction inside a map of internal representations. Moving in that direction corresponds approximately to becoming more like the supplied concept examples. The analogy is limited: the map is learned, high-dimensional, and model-specific. Moving along a direction inside it may not correspond to any realistic change that could be made to an actual image.

Worked example

Suppose a concept separator in a two-dimensional activation space produces direction (3, 4). Its length is 5, so the normalized CAV is (0.6, 0.8). Assume this orientation points towards the concept examples.

For one input, suppose the gradient of a selected class score with respect to these activations is (2, −1). The directional derivative is the dot product: 2 × 0.6 + (−1) × 0.8 = 0.4. It is positive, indicating that a sufficiently small move along this direction would locally increase that score to first order.

For a teaching collection of ten class examples, suppose seven have positive directional derivatives. The corresponding sign-based TCAV score is 7/10 = 0.7.

This does not mean the concept caused 70% of the predictions, contributed 70% of each score, or explains 70% of the model. The calculation summarizes a particular directional-sensitivity test, and the stipulated values do not replace concept validation or statistical assessment.

Illustrative case study

Fictional case: Texture Bay investigates whether a product-recognition model relies on glossy labels. Its first concept set consists mostly of studio photographs, while comparison images are taken in a warehouse. A seemingly clear direction separates the groups.

The analyst notices that gloss and photographic setting are entangled. The team collects examples that vary setting within both groups and repeats the analysis. It retains the original result as a hypothesis rather than a conclusion about the model's reasoning. No measured effect is claimed. The case illustrates how the examples used to define a concept can determine what a concept vector actually captures.

Common misconceptions and mistakes

A CAV is not the name of a single neuron that stores a concept. It is a direction involving the selected layer's representation.

A high TCAV score is not a causal attribution percentage. Directional sensitivity and real-world causal influence are different questions.

The ability to separate two example sets does not establish that the intended semantic distinction is what the separator learned. Confounding differences may explain it.

Finally, one run with one comparison set is weak evidence. Repeatability, concept quality, and meaningful controls matter when deciding whether an interpretation is useful.

Business use

CAVs can help technical teams investigate concrete hypotheses about a model, particularly when a domain expert can supply examples of a suspected concept. Start with a question that could change an evaluation or data-collection decision.

Document the example selection, model version, layer, comparison sets, and scoring procedure. Present the finding as evidence about the tested representation and sensitivity, with its limits. When the result suggests a concerning dependency, follow up with controlled data tests or model changes rather than treating the interpretability score alone as a complete explanation or remedy.

Check your understanding

A TCAV score is 0.8 for a concept and class. Does this establish that the concept supplied 80% of the reasoning behind every prediction?

Answer: No. In the common sign-based version, it means 80% of the evaluated examples had positive directional sensitivity under that test. It is neither an explanation share nor a causal percentage.

Related terms

Interpretability
Directional derivative
Confounding
TCAV
activation space
linear probe
concept examples
model inspection

Learn next

calculate the dot product in the worked example after reversing the concept direction and explain the sign change. Then design concept and comparison sets that vary an unwanted background feature independently of the intended concept.

Dot product

Sources

Kim and colleagues, Interpretability Beyond Feature Attribution: https://arxiv.org/html/1711.11279v3 — primary methods text consulted for CAV construction, directional sensitivity, and TCAV testing; publication record: https://arxiv.org/abs/1711.11279.

TCAV project documentation: https://github.com/tensorflow/tcav — primary implementation overview consulted. This entry explains the linear CAV formulation. Numerical examples and fictional case are original; interpretations are bounded to the tested concept and representation.

Previous: Concentration inequality Next: Concept drift