AI Glossary / Causal inference and experimentation / Intermediate

Hamming loss

Definition

Hamming loss measures the fraction of individual label decisions that are incorrect. In ordinary multilabel classification with N examples and L possible labels, it counts mismatches between predicted and reference binary label indicators, then divides by N times L. Lower is better, and zero means every evaluated label decision matches. It gives partial credit when some labels for an example are correct, unlike complete-set accuracy, which requires the entire predicted label set to match.

What it means

A multilabel task allows several labels to apply to the same example. An image might contain both a bicycle and a person, or a document might have several topics. Each label is a separate present-or-absent decision. Hamming loss averages errors over those decisions, including both omitted true labels and incorrectly added labels. This makes it easy to interpret locally, but a large number of correctly absent labels can make the overall loss look small even when the model misses important positive labels.

How it works

Represent reference and predicted label sets with the same ordered list of possible labels. Compare each example-label position, assigning one error when the binary indicators differ and zero when they match. Sum errors and divide by the number of evaluated positions for the ordinary unweighted form. If probabilities are produced, a declared thresholding rule is needed before calculating this discrete loss. Document weighting, missing-label policies, and the treatment of labels that are unknown rather than truly absent. Compare with other measures that reflect positive-label retrieval and whole-example success.

Three practical examples

• Document tagging: count incorrect topic assignments and missing required topics across a fixed label vocabulary.
• Image annotation: evaluate each object's presence-or-absence label while separately examining rare-object recall.
• Attribute prediction: measure how often individual binary product attributes are wrong, with critical attributes reviewed separately.

Think of it this way

Think of a checklist with several boxes on every form. Hamming loss counts wrongly checked and wrongly unchecked boxes across all forms. The analogy explains partial credit. It also exposes a limitation: if most boxes should be empty, leaving everything unchecked can look good overall while failing to identify the few boxes that matter most.

Worked example

Suppose three documents can have four labels: A, B, C, and D. Their reference sets are {A, C}, {B}, and {A, D}. The predicted sets are {A}, {B, C}, and {D}. Document 1 misses C, document 2 incorrectly adds C, and document 3 misses A. There are three incorrect label decisions across 3 times 4 = 12 positions, so Hamming loss is 3/12 = 0.25.
None of the three complete predicted sets matches its reference set, so complete-set accuracy is 0%, even though 75% of individual label decisions are correct. The measures answer different questions. Now consider a different example with 100 possible labels and exactly two true labels per document. Predicting no labels makes only two mismatches per document, giving Hamming loss 2/100 = 0.02, yet positive-label recall is zero. A low loss can therefore coexist with failure to retrieve every relevant label. The label vocabulary and prevalence belong alongside the reported result.

Illustrative case study

A fictional archive team initially uses Hamming loss to evaluate a topic tagger. As it expands the label vocabulary, many newly added labels are absent from most documents, making an all-negative baseline surprisingly competitive. The team reports that baseline and adds micro and macro precision-recall measures plus complete-set accuracy. It also checks frequently confused topic pairs. Hamming loss remains useful for per-label error frequency, but it no longer stands alone as evidence that documents receive useful tags.

Common misconceptions and mistakes

Hamming loss is not the fraction of examples with any mistake; that is a whole-example error measure. It is not necessarily one minus multilabel subset accuracy. In ordinary single-label classification, label mismatch rate can coincide with misclassification rate, but one-hot encoding a multiclass problem creates a different per-bit denominator and should not be confused with that convention. Unknown labels are not automatically negative. Equal treatment of label decisions also does not mean every label has equal operational importance.

Business use

Use the loss when each individual label decision is a meaningful evaluation unit. Publish the label vocabulary, threshold rule, weighting, and reference completeness. Include a simple baseline, especially for sparse label sets. Pair the result with measures of positive-label detection and with whole-record correctness if downstream work requires a complete set. Inspect errors by label and by example type. If some mistakes have substantially greater consequences, report them separately or define justified weights rather than allowing the average to conceal them.

People also ask

Can false positives and false negatives both increase Hamming loss?

Yes. An extra label and a missing true label each contribute one mismatch in the ordinary unweighted form.

Does it use probability confidence directly?

No. Standard Hamming loss compares discrete label decisions, so probabilities must first be converted under a stated rule.

Can it be low when every example has an error?

Yes. Each example may have one wrong label among many correct decisions, while no complete label set is entirely correct.

Check your understanding

Why is complete-set accuracy zero but Hamming loss only 0.25 in the worked example?

Answer: Every document has at least one mistake, so all fail the complete-set test. Only three of the twelve individual label decisions are wrong, so the per-label error fraction is one quarter.

Related terms

• Hamming distance
• Exact match
• False positive
• False negative
• Multilabel classification
• subset accuracy
• label sparsity
• micro F1
• macro F1
• thresholding

Learn next

Create a small binary label matrix and mark every mismatched cell. Calculate Hamming loss, then mark which rows match completely and calculate subset accuracy. Add an all-negative baseline and inspect positive-label recall. This exercise shows why per-label correctness, complete-set correctness, and retrieval of relevant labels need separate interpretation.

More in Causal inference and experimentation

• Gini impurity - Gini impurity measures how mixed the class labels are within a group of observations, commonly a node in a classification tree.
• Granger causality - Granger causality is a predictive relationship between time series: the past of one series helps predict another beyond the information already included in...
• Hamming - Hamming is a name attached to several distinct concepts, including Hamming distance, Hamming loss, Hamming codes, and the Hamming window.
• Heterogeneous treatment effect - A heterogeneous treatment effect is a causal effect that varies across individuals, groups or contexts.
• Highly adaptive lasso - Highly adaptive lasso, or HAL, is a flexible regression method that represents complex relationships using a large collection of basis functions and...
• Holdout cross-validation - Holdout cross-validation is an imprecise phrase commonly used for evaluating a model on a held-out portion of data, sometimes repeated across several random...

Work in finance? Get the free CFO AI Prompt Pack: 7 prompts for finance leaders.

Sources

Scikit-learn, hamming_loss: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.hamming_loss.html Scikit-learn, accuracy_score and subset accuracy: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html Scikit-learn, multilabel metrics guide: https://scikit-learn.org/stable/modules/model_evaluation.html

Previous: Hamming distance Next: Hampel test