Does it set coefficients exactly to zero?

As a continuous prior, it does not put positive probability on exact zero. A separate thresholding or selection procedure can produce an exact sparse summary.

Why have local and global scales?

The global scale encourages overall shrinkage while local scales allow effects to differ.

Is it always preferable to ridge or lasso?

No. Suitability depends on assumptions, data, computation and the decision being supported.

AI Glossary / Probabilistic machine learning / Advanced

Horseshoe distribution

Definition

The horseshoe distribution is a continuous probability distribution commonly used as a Bayesian shrinkage prior for sparse signals or coefficients. It combines strong concentration near zero with heavy tails, allowing many effects to be small while leaving room for a few large effects.

What it means

Sparse modeling assumes that only a relatively small part of a large set of possible effects is substantial. The horseshoe prior expresses that idea without assigning a separate binary inclusion indicator to every coefficient.

It is called a global-local shrinkage prior because a shared scale influences the whole collection while individual scales let particular coefficients behave differently. The goal is to avoid treating every coefficient with the same amount of shrinkage.

The name refers to a horseshoe-shaped distribution of shrinkage weights in a standardized setting. The coefficient density itself is sharply peaked at zero; it is not shaped like a literal horseshoe. The distribution is continuous, so a tall density peak does not mean positive probability at exactly zero.

How it works

One standard construction is beta_j given lambda_j and tau distributed normally with mean zero and variance tau squared times lambda_j squared. Each positive local scale lambda_j has a standard half-Cauchy distribution. Tau is the global scale, which may be specified or given its own prior.

Integrating out the local scale produces the horseshoe marginal prior for a coefficient at the chosen global scale. A small global scale encourages widespread shrinkage, while large local scales can permit individual coefficients to escape much of it.

Posterior inference combines this prior with the likelihood. The resulting shrinkage depends on the observed data, noise assumptions and full model. Predictor scaling and prior choices matter; the distribution's reputation for sparse problems is not a substitute for specifying those choices.

Three practical examples

A regression analyst uses global-local shrinkage when many candidate predictors are expected to have negligible effects.
A signal-processing study estimates many noisy components while allowing a few substantial signals.
A Bayesian neural-network research project investigates shrinkage of selected weights or groups under an explicitly defined horseshoe construction.

Think of it this way

Think of a system with a general tendency to turn down many uncertain signals, but separate controls that let strongly supported signals remain prominent.

The analogy should not be read as a literal threshold switch. The horseshoe prior does not automatically label a coefficient exactly on or off. Posterior uncertainty and any later selection rule must be handled explicitly.

Worked example

Reading a conditional shrinkage factor

Consider the normal-means teaching model y given beta distributed normally with mean beta and variance 1. Fix the global scale at tau = 1. Conditional on local scale lambda, let beta have normal prior variance lambda squared.

The conditional posterior mean is [lambda squared/(1 + lambda squared)]y. Write kappa = 1/(1 + lambda squared); then the mean is (1 − kappa)y.

If y = 2 and lambda = 0.5, lambda squared is 0.25. The multiplier is 0.25/1.25 = 0.2, giving conditional posterior mean 0.4. If lambda = 2, the multiplier is 4/5 = 0.8, giving 1.6.

These calculations hold the local scale fixed to show its role. A full horseshoe analysis infers uncertainty about lambda rather than selecting one of these illustrative values by hand. Its posterior mean averages the relevant shrinkage over the posterior distribution. Neither conditional result is exactly zero, and neither establishes whether the coefficient should be retained in a final decision rule.

Illustrative case study

Fictional case: a demand model with many predictors

A fictional retailer explores a demand regression containing many candidate effects. The analyst expects only some effects to be substantial and compares a horseshoe prior with simpler regularization choices.

The team standardizes predictors consistently, documents the global-scale prior and examines sensitivity to its assumptions. It evaluates predictions on held-out periods and reviews computation diagnostics rather than judging the model solely by how many coefficients look small.

When managers request a short predictor list, the analyst explains that continuous shrinkage is not itself an exact variable-selection rule. The team defines a separate selection and validation procedure. No measured forecasting gain is assumed in this fictional example.

Common misconceptions and mistakes

An infinite density at zero is not a point mass at zero. Confusing density with probability leads to incorrect claims that the horseshoe directly produces a posterior inclusion probability for exact zero.

Heavy tails do not guarantee that every large-looking observation is a genuine signal. The likelihood and data quality still matter, and unusual observations can have other explanations.

The horseshoe and regularized horseshoe are related but distinct. The regularized version adds control over large coefficients through a finite slab-like scale. Substituting one for the other changes the prior and should be recorded.

Business use

Use the horseshoe when a sparse-effect assumption is credible and uncertainty about coefficients is part of the analysis. Ask what the global scale implies about plausible effects after accounting for predictor units and sample size.

Assess posterior computation, sensitivity and out-of-sample performance. If several predictors are strongly related, inspect joint uncertainty rather than treating one shrunk coefficient as proof that its variable is irrelevant. Keep prediction, explanation and variable selection as separately justified claims; a shrinkage prior does not make causal relationships identifiable.

Check your understanding

In the worked normal-means model, lambda = 1 and y = 2. What is the conditional posterior mean?

Answer: The multiplier is 1/(1 + 1) = 0.5, so the mean is 1. This is conditional on the specified scale, not a complete horseshoe posterior calculation.

Related terms

Bayesian Regression
Sparsity
Shrinkage prior
half-Cauchy distribution
global-local shrinkage
spike-and-slab prior
regularized horseshoe

Learn next

Contrast a continuous shrinkage prior with a prior containing an actual point mass at zero. Then examine how predictor standardization changes the practical meaning of a global scale.

Sources

Primary source: Carvalho, Polson and Scott, Handling Sparsity via the Horseshoe, AISTATS 2009: https://proceedings.mlr.press/v5/carvalho09a/carvalho09a.pdf

Primary extension: Piironen and Vehtari, Sparsity information and regularization in the horseshoe and other shrinkage priors: https://arxiv.org/abs/1707.01694

The conditional calculation and retailer case are original illustrations.

Previous: Horner's method Next: Horvitz - Thompson estimator