Is Klecka's tau a rank correlation?

No. It measures classification improvement relative to a specified random-assignment baseline.

Can two tau values accompany the same accuracy?

Yes, if their baseline probabilities differ.

Does a positive tau imply the classifier beats the majority-class rule?

No. That deterministic rule is a different comparator and must be evaluated separately.

AI Glossary / Statistics and probability / Advanced

Klecka's tau

Definition

Klecka's tau is a descriptive measure of how much a classification rule reduces errors relative to a specified random-assignment baseline. It is associated with the evaluation of discriminant analysis, but its arithmetic compares observed correct classifications with the number expected under chosen allocation probabilities. A value of 1 indicates perfect classification, 0 indicates no improvement over that baseline, and negative values indicate worse performance. It is not Kendall's rank-correlation tau, and the coefficient itself is not a p-value or a guarantee of out-of-sample accuracy.

What it means

Raw accuracy is easier to interpret when compared with what a simple baseline could achieve. Randomly assigning cases to groups can produce correct classifications even without predictive information. The expected success rate depends on both the actual group sizes and the probabilities used for random assignment. Klecka's tau expresses the improvement as a fraction of the errors that this baseline would make. Because the baseline is part of the definition, reporting tau without the allocation probabilities can hide an important assumption, particularly when class sizes are unequal.

How it works

Let N be the number of evaluated cases, C the number correctly classified, nᵢ the actual number in group i and pᵢ the probability that the baseline randomly assigns a case to group i. Expected correct classifications are E = Σnᵢpᵢ, with the pᵢ values summing to one. Then tau = (C − E)/(N − E), provided N − E is positive. Equivalently, it is one minus the observed error count divided by the baseline's expected error count. Equal allocation uses pᵢ = 1/g for g groups; proportional allocation uses pᵢ = nᵢ/N. These choices answer different comparisons and should not be silently interchanged.

Three practical examples

Discriminant-analysis reporting: supplement a classification table with improvement relative to an explicitly stated random baseline.
Imbalanced classification review: show how the interpretation of an accuracy result changes under equal versus proportional random allocation.
Evaluation documentation: compare training and held-out classification results using the same predefined baseline. The coefficient does not correct optimism caused by testing on the training data.

Think of it this way

Think of a baseline that is expected to make a certain number of mistakes. Tau asks what fraction of those mistakes the evaluated classifier avoids. If the baseline expects 50 errors and the classifier makes 20, the reduction is 30 out of 50, or 0.6. That is a relative error reduction, not an accuracy of 60% and not a 60-percentage-point increase in accuracy.

Worked example

Suppose a held-out set contains 100 cases in three groups of sizes 70, 20 and 10, and a classifier correctly labels 80. Under equal random allocation, expected correct classifications are 70/3 + 20/3 + 10/3 = 100/3. Tau is (80 − 100/3)/(100 − 100/3) = 0.70. Under proportional random allocation, expected correct classifications are 70(0.7) + 20(0.2) + 10(0.1) = 54. Tau becomes (80 − 54)/(100 − 54) = 13/23, approximately 0.5652. Both calculations use the same 80% accuracy. A separate always-predict-the-largest-class rule achieves 70% accuracy here, so the error reduction relative to that deterministic baseline is (80 − 70)/(100 − 70) = 1/3. That last comparison is useful, but it should be labeled separately from the chosen random-allocation tau.

Illustrative case study

Fictional case study: Laurel Classification Lab reports high accuracy for assigning records to three operational categories. A reviewer asks which baseline underlies the reported tau. The analyst finds that one report used equal assignment probabilities and another used observed class proportions. The team corrects the documentation, keeps the same baseline across model comparisons and adds a confusion matrix with class-specific errors. The revised report makes clear that a single overall improvement measure can coexist with poor performance on the smallest category.

Common misconceptions and mistakes

Common mistakes include confusing tau with Kendall's tau, treating it as a significance test, and interpreting a positive value as proof that the model beats every simple rule. Equal random assignment can be a weak comparator for imbalanced data. Proportional random assignment is also different from always selecting the largest group. A high aggregate coefficient can hide uneven errors across classes or costly mistakes in a rare class. If expected baseline accuracy is one, the denominator is zero and the usual formula is undefined. Changing class prevalence can change the coefficient even when other aspects of performance appear similar.

Business use

Businesses should use the coefficient as one descriptive comparison rather than a complete model scorecard. Report raw accuracy, the evaluation sample, actual class counts and the baseline probabilities alongside tau. Include class-specific performance and error costs when decisions have unequal consequences. Use genuinely held-out or appropriately cross-validated predictions to assess generalization, and consider uncertainty with a method suited to the evaluation design. The coefficient alone neither adjusts for repeated observations nor provides a universal threshold for acceptable operational performance.

Check your understanding

Why does the example's tau fall from 0.70 to about 0.5652 under proportional allocation?

Answer: The proportional baseline already expects more correct predictions because the groups are unequal. The classifier therefore removes a smaller fraction of that baseline's expected errors.

Related terms

Discriminant analysis
Confusion matrix
classification accuracy
chance agreement
class imbalance
majority-class baseline
out-of-sample evaluation

Learn next

compute tau from a small confusion matrix using explicitly chosen allocation probabilities, then compare it with raw accuracy and a majority-class baseline. Examine class-specific errors before deciding whether any aggregate improvement is practically useful.

Sources

Sources: BIOSTATS documentation, https://gannet.fish.washington.edu/seashell/bu-github/project-oyster-oa/analyses/DNR_Preliminary_Analyses_20170321/biostats%20help.pdf (tau implementation). F. A. McKinney et al., Factors Relating to the Job Placement of Former Secondary Vocational Education Students, https://files.eric.ed.gov/fulltext/ED209477.pdf, printed page 56 (application and formula attributed to Klecka, 1980). Examples and fictional case are original.

Previous: KL annealing Next: Kleene +