Is CLEVER an attack that generates adversarial examples?

No. It estimates robustness through local sensitivity analysis and statistical sampling.

Does a larger number always mean a better model?

It suggests greater estimated robustness only under comparable evaluation settings; accuracy and other requirements still matter.

Is the reported score a formal certificate?

The sampled statistical score is not automatically a certified bound. Its estimation assumptions and limitations must be considered.

AI Glossary / Deep learning / Advanced

CLEVER score

Definition

The CLEVER score is a statistical measure intended to estimate a neural classifier's local robustness to adversarial input changes. Its name expands to Cross Lipschitz Extreme Value for nEtwork Robustness. It combines the gap between competing class scores with an estimate of how rapidly that gap can change near an input. CLEVER is attack-independent in its construction, but its sampled estimate should not be confused with a formally certified robustness guarantee.

What it means

A classifier may label an image correctly yet change its answer after a small, carefully chosen alteration. CLEVER asks how much room may exist around the current input before a competing class can overtake the predicted class.

The answer depends on how input distance is measured and which alternative classes are considered. A targeted score concerns one chosen alternative; an untargeted score considers changing the prediction to any alternative. The number is meaningful only alongside these settings. It is not an accuracy percentage, a general intelligence score, or a measure of whether the original prediction was correct.

How it works

The method examines differences between the current class score and competing scores. It samples nearby inputs, evaluates gradient norms, and uses extreme value theory to estimate a local upper limit on the sensitivity of a score difference.

Intuitively, a larger score gap and slower possible change suggest a larger robust neighborhood. The estimated sensitivity enters a gap-to-sensitivity calculation, limited by the neighborhood radius being studied. Untargeted evaluation takes the most restrictive competing-class result.

Because finite sampling and statistical fitting replace exhaustive analysis, the result depends on assumptions and implementation details. The original work supplies the theoretical motivation; a practical estimate does not inherit the certainty of a bound computed from a valid, proven sensitivity limit.

Three practical examples

1. Model comparison: researchers compare two image classifiers under the same input scaling, distance norm, sampled neighborhood, and evaluation examples, while also testing their ordinary accuracy.

2. Training experiment: a team measures how a proposed training change affects CLEVER scores, then checks whether complementary adversarial tests support the interpretation.

3. Evaluation audit: an analyst reviews a robustness report and asks whether the numbers are targeted or untargeted, how sampling was configured, and whether preprocessing was included in the evaluated system.

Think of it this way

Think of estimating how far a hiker can move before reaching a boundary by combining the current height above it with an estimate of the steepest nearby slope. If the slope estimate misses a steep patch, the safe-distance estimate can be too generous. The analogy is limited because classifier boundaries occupy high-dimensional input spaces, and the chosen numerical distance may not match human perception.

Worked example

Suppose a classifier's current class score exceeds a particular target class score by 0.6. For an illustrative evaluation, let the estimated local sensitivity of this difference be 3, using a specified distance norm. Let the sampled neighborhood radius be 0.5.

The gap divided by estimated sensitivity is 0.6/3 = 0.2. Taking the smaller of this result and the neighborhood radius gives an illustrative targeted score of 0.2.

If a second alternative class yields a corresponding estimate of 0.1, then considering both alternatives makes the untargeted estimate no larger than 0.1. Another alternative could reduce it further.

These stipulated numbers illustrate the calculation, not an actual fitted extreme-value model. If the true relevant sensitivity were 6 rather than the estimated 3, the first ratio would be 0.1. This demonstrates why the quality of the sensitivity estimate matters and why a larger estimated radius is not automatically a certificate.

Illustrative case study

Fictional case: Robust Harbour compares two inspection-image classifiers and initially reports that one has a larger average CLEVER score. During review, an engineer discovers that the models used different image-value ranges, making the reported input distances difficult to compare.

The team standardizes preprocessing, documents the norm and sampling settings, and evaluates the same selected examples. It also reports low-scoring cases rather than only the mean, and runs separate attack-based tests. No winning model or numerical improvement is claimed. The case illustrates that a robustness number needs a reproducible measurement context before it supports a practical decision.

Common misconceptions and mistakes

“Attack-independent” does not mean free of assumptions, sampling uncertainty, or implementation errors. It means the score is not defined by the success of one particular attack algorithm.

A high score does not show that an input was classified correctly. A confidently wrong region can also resist small perturbations.

Different norms and input scales produce different distance interpretations. Their numbers should not be compared casually.

Gradient masking and related evaluation difficulties have prompted published criticism and responses. Treat suspiciously strong scores as a reason for further investigation, especially when the evaluated pipeline contains operations that complicate gradient analysis.

Business use

CLEVER can contribute to a research evaluation of model sensitivity, particularly when teams want evidence beyond a single adversarial test. It should sit within an evaluation plan that defines the allowed perturbations and the actual consequences of changed predictions.

Ask for per-example results, preprocessing details, sampling parameters, and complementary tests. Keep robustness to small numerical perturbations separate from robustness to lighting changes, unfamiliar products, missing data, or other operational shifts. A useful decision requires evidence aligned with the failure conditions the business expects to face.

Check your understanding

Two reports give CLEVER scores using different pixel scales and distance norms. Can the larger number establish greater robustness?

Answer: No. First align the input representation, norm, neighborhood, target definition, and evaluation examples. Otherwise the numbers describe different measurement conditions.

Related terms

Adversarial example
Lipschitz constant
gradient norm
extreme value theory
robustness certification
gradient masking
targeted attack
perturbation norm

Learn next

calculate gap-to-sensitivity ratios for two competing classes, then study the distinction between estimating a local sensitivity limit and proving one. Read both the original method and its published criticism before interpreting a robustness report.

Sources

Weng and colleagues, Evaluating the Robustness of Neural Networks: https://arxiv.org/html/1801.10578v1 — primary method and algorithms consulted.

Author implementation: https://github.com/huanzhang12/CLEVER — sampling, fitting, and author discussion of limitations and criticism.

Goodfellow, Gradient Masking Causes CLEVER to Overestimate Adversarial Perturbation Size: https://arxiv.org/abs/1804.07870 — primary critical publication record consulted. Worked numbers and fictional case are original illustrations, not experimental findings.

Previous: Clefts Next: Client-Server Architecture