Is intersection over union the same basic formula?

Yes. In segmentation and related uses, that name describes the same intersection-to-union ratio.

Do true negatives improve Jaccard?

No. Correct absences are outside the union and do not contribute to the basic score.

What happens when both sets are empty?

The raw formula becomes 0/0. Software or evaluation protocols apply a stated convention, which should be reported.

AI Glossary / Causal inference and experimentation / Intermediate

Jaccard similarity coefficient score

Definition

The Jaccard similarity coefficient score measures overlap between two sets by dividing the size of their intersection by the size of their union. In model evaluation, the sets may be predicted and reference labels, or predicted and reference foreground pixels. It is also called the Jaccard index or intersection over union. A higher score indicates greater overlap, but the exact averaging and empty-set conventions must be stated when summarizing many predictions.

What it means

The intersection contains items present in both sets. The union contains every distinct item present in either set. Jaccard therefore rewards shared positive selections while penalizing both missing and extra selections. Items absent from both sets do not enlarge the numerator or denominator.

This is helpful when the main question concerns overlap rather than all possible absences. For example, a document may have only a few applicable tags from a large vocabulary. Counting every correctly absent tag could make performance appear strong despite poor selection of the tags that matter. Jaccard focuses attention on the labels actually involved in the comparison.

How it works

Define the objects being compared and represent their membership consistently. Count shared members, then count all distinct members appearing in either set. Divide intersection by union when the union is nonempty.

For binary decisions, the same calculation is true positives divided by true positives plus false positives plus false negatives. True negatives do not enter this expression.

With multiple labels or examples, choose how to aggregate. Sample averaging first scores each example's label sets; macro averaging first scores each class; micro averaging pools counts. These procedures answer different questions and can produce different results. If both sets are empty, the basic fraction is undefined, so the evaluation must specify the convention used rather than silently assuming agreement on that case.

Three practical examples

1. Document tagging: a team compares the set of predicted topics with reference topics, giving partial credit when some but not all tags match.

2. Image segmentation: a researcher compares predicted foreground pixels with an annotated region to measure spatial overlap without rewarding surrounding background pixels.

3. Item extraction: a system extracts named items from a short record, and evaluators compare normalized predicted and reference item sets after defining exactly what counts as a match.

Think of it this way

Think of two guest lists: Jaccard asks what fraction of everyone named on either list appears on both. It does not count everyone in the city who appears on neither. The analogy is limited because model evaluation depends on a reference and a matching policy. Slightly different names, duplicate entries, and partial matches require decisions before the sets can be compared meaningfully.

Worked example

Suppose the reference tags are {delivery, refund, delay}, while predictions are {delivery, delay, packaging}. The intersection contains delivery and delay, so its size is 2. The union contains delivery, refund, delay, and packaging, so its size is 4. Jaccard is 2/4 = 0.50.

There are two true positives, one false positive, and one false negative, giving the same value: 2/(2 + 1 + 1). Precision and recall each equal 2/3, showing that Jaccard is a different measure.

For another example with one correctly predicted tag and no mistakes, Jaccard is 1. The sample-average score across these two examples is (0.50 + 1)/2 = 0.75. Pooling the counts instead gives micro Jaccard 3/(3 + 1 + 1) = 0.60. Neither aggregation should be left unnamed.

Illustrative case study

Fictional case: Copper Tags evaluates an automated filing assistant. A report shows high overall accuracy because most possible topics are absent from every document. Staff still find missing and unnecessary tags on the documents they actually handle.

The team adds Jaccard scores for each document and reviews the distribution across short and long tag sets. It also reports per-topic performance so rare topics are not hidden by an aggregate. Reviewers agree how to handle documents with no applicable tags before comparing system versions. The exercise changes the evaluation question from counting widespread absences to examining useful overlap, without claiming that one metric captures every filing cost.

Common misconceptions and mistakes

Do not confuse Jaccard with exact-match accuracy. A partly correct set can receive a positive Jaccard score even when the complete predicted set is wrong.

Do not count repeated occurrences as additional set members unless a different, explicitly defined measure is intended. Standard sets contain distinct elements.

Avoid comparing sample, macro, and micro averages as though they were interchangeable. Also specify thresholding when probabilities must become discrete selections. Changing that threshold changes the sets and therefore the score. Empty-set treatment can materially affect results when many examples contain no positives.

Business use

Jaccard offers a clear overlap measure for tagging, extraction, and region detection. It is useful when extra and missing selections both matter and correctly absent items should not dominate the assessment.

Business teams should still examine the costs of each error type. Missing a critical tag and adding a harmless tag contribute symmetrically to the basic denominator, although their consequences may differ. Pair overlap scores with appropriate error breakdowns and user-task measures. Keep matching, thresholding, and averaging rules stable between versions so an apparent improvement reflects changed predictions rather than changed bookkeeping.

Check your understanding

The reference set has three members, the predicted set has four, and two members are shared. What is Jaccard?

Answer: The union has 3 + 4 − 2 = 5 distinct members, so Jaccard is 2/5 = 0.40. The shared members are counted once in the union.

Related terms

Intersection over union
Precision
Recall
Segmentation
F1 score
multilabel classification
macro averaging
micro averaging
subset accuracy

Learn next

Create two small reference - prediction set pairs and calculate their intersections and unions. Compare the average of individual Jaccard scores with a score from pooled counts. Include one empty-set case and write the convention you would use before calculating the final summary.

Sources

Primary implementation source: scikit-learn, Jaccard similarity coefficient score, https://scikit-learn.org/stable/modules/model_evaluation.html#jaccard-similarity-coefficient-score . The set definition and averaging discussion were consulted. This entry describes the standard unweighted set coefficient; specialized weighted variants require their own definition. All tag sets, arithmetic, and the fictional filing project are original teaching illustrations.

Previous: Iteratively Reweighted Least Squares Next: Jackknife variance estimates for random forest