Can replication probabilities be estimated sensibly?

Yes, within explicit models that account for uncertainty, design, prior information, and relevant differences between studies.

AI Glossary / Statistics and probability / Intermediate

P-rep

Definition

P-rep, usually written p_rep, is a proposed statistic intended to estimate the probability that a replication produces an effect with the same sign as an observed effect. It was introduced by Peter Killeen as an alternative way of discussing evidence. Its interpretation depends on a predictive model and assumptions; it is not a general, assumption-free probability of successful replication.

What it means

An estimated positive effect might be followed by another positive estimate in a new study, even if that second result is small or statistically nonsignificant. P-rep focuses on this directional agreement in its usual formulation.
That target is much narrower than reproducing a practically important benefit, reaching a significance threshold, or confirming the original explanation. The proposal generated methodological debate, including questions about prior assumptions and predictive interpretation. Readers should treat it as a specific historical statistical proposal rather than a universal replacement for careful replication analysis.

How it works

A simple normal-model illustration explains the intuition. Suppose an observed effect estimate d has known standard error s, and a future independent study estimates the same underlying effect with the same standard error.
With a flat prior for that common effect, the posterior distribution of the effect is normal with mean d and variance s squared. Integrating over this uncertainty gives a predictive distribution for the future estimate with mean d and variance 2s squared.
For positive d, the probability that the future estimate is positive is Phi[d/(sqrt(2) times s)]. For a negative observed estimate, directional agreement uses its magnitude analogously. This derivation is tied to the stated normal model and prior, not a guarantee about actual future studies.
A different prior, different replication precision, heterogeneity between studies, or selection of the original result can change the prediction. Unknown standard errors also require more careful treatment than this idealized calculation.
Define replication success before computing a probability. Same sign, a minimum meaningful effect, statistical significance, and closeness to the original estimate are different events with different probabilities.
Report effect sizes, uncertainty, study design, and the predictive assumptions. Genuine replication data provide information that cannot be replaced by relabeling a statistic from a single selected study.

Three practical examples

Research interpretation: a reader checks whether an older article's p_rep refers only to agreement in direction rather than a significant replicated result.
Experiment planning: an analyst compares predicted directional agreement with the probability of exceeding a useful business threshold.
AI evaluation: a team examines whether an apparent benchmark gain is likely to remain positive in a new evaluation, while explicitly modeling test-set changes and uncertainty.

Think of it this way

Think of predicting whether tomorrow's measured change points in the same direction as today's. Predicting the direction is less demanding than predicting the same magnitude or a commercially useful improvement. The forecast also depends on how much you believe today's observation reflects a stable underlying effect.

Worked example

Assume a fictional experiment estimates a positive effect of 2 units with known standard error 1. A replication will have the same standard error, and both estimates follow the shared-effect normal model with a flat prior described above.
The future estimate has predictive mean 2 and standard deviation sqrt(2). Its probability of being positive is Phi(2/sqrt(2)), approximately 0.92135, or 92.14%.
Now define success as a positive result exceeding the usual two-sided normal 5% significance threshold, using the rounded critical value 1.96 and replication standard error 1. The future estimate must exceed 1.96. Its predictive probability is Phi[(2 - 1.96)/sqrt(2)], approximately 0.5113, or 51.13%.
Thus a high directional probability does not imply a similarly high probability of a significant replication.
As a sensitivity check, replace the flat prior with a normal prior centered at zero with variance 1. The posterior effect mean becomes 1 and variance 0.5, giving predictive variance 1.5. Directional probability becomes Phi(1/sqrt(1.5)), approximately 79.29%. The data are unchanged; the predictive assumptions changed. These are model-based illustrations, not validated success rates.

Illustrative case study

Illustrative fictional case: Hazel Insights finds an old internal report claiming a high probability of replication for an AI-assisted workflow improvement. Its reviewers discover that the reported quantity concerns only the sign of a new estimate.
They define the minimum benefit needed to justify rollout, plan a fresh evaluation, and examine predictions under several plausible effect models. They keep directional agreement separate from meaningful benefit and statistical significance. The case is fictional and does not claim that the original proposal has been adopted as a current business standard.

Common misconceptions and mistakes

Do not read p_rep as the probability that the original hypothesis is true. Do not interpret same-sign replication as significant replication or exact reproduction of the original estimate.
Avoid treating a transformed p-value as a source of new independent evidence. Do not ignore publication selection, changing populations, model assumptions, or prior information. Criticism of a statistic also does not imply that predictive probabilities are inherently invalid; the predictive target and model must be made explicit.

Business use

The main business value is interpretive discipline. Teams reviewing research or historical reports should ask what replication means, which assumptions support the number, and whether the target matches the operational decision. For new work, specify a predictive framework and meaningful success criterion transparently instead of relying on the label alone.

Check your understanding

Why are the example's 92.14% and 51.13% different?

Answer: One asks for a positive estimate; the other asks for an estimate above 1.96.

Why did the sensitivity calculation change?

Answer: A more concentrated prior near zero altered the posterior predictive distribution.

Related terms

Replication (statistics)
Posterior predictive distribution
P-value
Effect size
Bayesian inference
Replication
statistical power
publication bias
predictive probability

Learn next

Next study posterior prediction and replication design. Practice defining meaningful success events, then assess sensitivity to prior assumptions, selection effects, and differences between original and replication settings.

Posterior (prediction)

Sources

Killeen, An Alternative to Null-Hypothesis Significance Tests (2005): https://labs.la.utexas.edu/gilden/files/2016/05/prep.pdf — original paper hosted by a university laboratory; same-sign definition and replication-variance discussion reviewed directly. PMC retrieval remained challenged. Lecoutre and colleagues, Killeen's probability of replication and predictive probabilities: https://pubmed.ncbi.nlm.nih.gov/20515237/ — primary abstract on predictive interpretation and confusion with significant replication. Iverson, Wagenmakers, and Lee, A model-averaging approach to replication: https://pubmed.ncbi.nlm.nih.gov/20515238/ — primary abstract on model-based critique. Calculations are original explicit-model illustrations.

Previous: p-hacking Next: P/T ratio