Is term discrimination the same as TF-IDF?

No. They are related indexing ideas with different definitions.

Can the value be negative?

Yes, under the similarity-based convention described here, a term that increases compactness can receive a negative value.

Does the score use relevance labels?

The geometric calculation need not, which is why separate relevance evaluation matters.

AI Glossary / Retrieval and knowledge systems / Intermediate

Term discrimination

Definition

Term discrimination is an information-retrieval concept that evaluates how much an indexing term helps distinguish document representations within a collection. In the classic discrimination-value model, a term is assessed through the change in document-space compactness when that term is removed or added. The concept concerns the structure of an indexed collection, rather than directly measuring whether a particular document answers a user's query.

What it means

An index term can make some documents easier to distinguish from others, or it can make many documents look alike. Term discrimination examines that effect using a specified representation and similarity measure.
The idea is related to term weighting and feature selection, but it is not identical to inverse document frequency. A term's value depends on how it changes the geometry of the represented collection, not just how many documents contain it. It is also unrelated to social discrimination: here, discrimination means distinguishing items for retrieval.

How it works

Represent documents as weighted term vectors and choose a measure of compactness. One possible measure uses similarities among document pairs; the classic literature also discusses a centroid-based calculation that reduces computation.
Let Q denote compactness with the term present and Q_without denote compactness after deleting it, using a measure where larger values mean greater similarity. Define the discrimination value as Q_without minus Q. A positive result means removal made documents more similar, so the term had helped separate them. A negative result means the term had increased compactness.
The subtraction order matters. If instead a measure expresses spread or distance, its direction differs from a similarity-based compactness measure. Formulas cannot be compared safely without checking what Q represents and how vectors are normalized.
Repeat the calculation for candidate terms, then assess whether proposed indexing changes improve retrieval on representative queries. The geometric score is a proxy, not a replacement for relevance evaluation. Removing terms can also create zero vectors, so implementations need an explicit policy for undefined similarities. Vocabulary, weighting, normalization, and collection composition all affect the result.

Three practical examples

A retrieval researcher investigates which terms most affect separation in a small document collection.
An indexing team compares candidate vocabularies, using geometric analysis alongside query-based relevance judgments.
An analyst studies why a very common term changes similarity scores without helping users distinguish the documents they need.

Think of it this way

Think of descriptions on storage boxes. A label that appears on every box may make their descriptions look alike, while a label that identifies a useful difference can help separate them. Whether the difference helps a searcher still depends on what the searcher wants.

Worked example

Consider an original illustration using mean cosine similarity over distinct document pairs as the compactness measure. This is an explicitly chosen pairwise version, not a reproduction of the paper's centroid calculation.
Three documents have two-coordinate vectors A=(1,1), B=(1,0), and C=(1,0). The second coordinate represents candidate term k. The similarities are cosine(A,B)=1/sqrt(2), cosine(A,C)=1/sqrt(2), and cosine(B,C)=1.
Their mean compactness is Q=(2/sqrt(2)+1)/3, approximately 0.804738. Removing k leaves A, B, and C all represented by the one-coordinate vector (1). Every pair now has cosine similarity 1, so Q_without=1.
The discrimination value is 1-0.804738, approximately 0.195262. It is positive under the stated convention: k separated A from the other documents.
For a separate counterexample, start with two documents (1,0) and (0,1), whose cosine similarity is 0. Add a third coordinate equal to 1 in both documents. Their new vectors are (1,0,1) and (0,1,1). The dot product is 1 and both norms are sqrt(2), so their similarity becomes 0.5.
For this added common term, Q_without-Q is 0-0.5=-0.5. The term makes these two representations more similar. Neither calculation establishes real-world relevance: a user might still need a common term for a specific phrase or a query about a shared characteristic.

Illustrative case study

Fictional case: Ash Archive studies a compact technical-document index. Its team finds that some common terms increase overall similarity and considers reducing their role.
Before changing the vocabulary, the team evaluates known queries and discovers that certain common words are important inside exact technical phrases. It retains those retrieval capabilities and treats discrimination analysis as one input to the decision. It also checks rare identifiers separately because a collection-wide score can undervalue a term that is essential for a narrow query.
This hypothetical case illustrates careful interpretation, not a reported deployment or a measured improvement.

Common misconceptions and mistakes

A high discrimination value is not a guarantee of high precision or recall. It describes a representation under a chosen geometric measure.
Do not equate vector-space compactness with the proportion of nonzero matrix entries. Matrix sparsity and similarity-based density are different ideas.
Rare terms are not automatically useless, and common terms are not automatically dispensable. Their retrieval value depends on queries and combinations as well as collection statistics. Also avoid reversing the sign of a formula while retaining the same interpretation of positive scores.

Business use

The concept is useful for understanding why indexing choices change document similarity and for interpreting historical retrieval research. It can inform experiments on vocabulary selection, but business search quality should be assessed with actual tasks and relevance judgments.
Document the exact compactness measure, normalization, and treatment of empty vectors. Compare the effects of proposed changes on common queries, specialized terms, and phrase searches. Recalculate when the collection changes substantially, since a term's contribution is relative to the documents represented.

Check your understanding

Why is k positive in the first example?

Answer: Deleting it makes all three document vectors identical in direction.

Why is the common term negative in the second example?

Answer: Adding it raises similarity from 0 to 0.5.

What must be checked before interpreting a formula?

Answer: Whether its quantity measures similarity or distance and which subtraction order it uses.

Related terms

Feature Selection
Centroid
Cosine similarity
Vector space model
term weighting
inverse document frequency
document similarity
automatic indexing

Learn next

Learn cosine similarity and vector normalization, then reproduce a small deletion experiment. Compare pairwise and centroid-based measures, and evaluate any vocabulary changes against real retrieval questions rather than geometric scores alone.

Sources

Primary research: Salton, Wong, and Yang, A Vector Space Model for Automatic Indexing, Communications of the ACM (1975), especially Section 4: https://openlib.org/home/krichel/courses/lis618/readings/salton75.pdf

Previous: Term-at-a-time Next: Term-document matrix