AI Glossary / Statistics and probability / Intermediate
Pearson correlation coefficient
Definition
The Pearson correlation coefficient measures the direction and strength of linear association between two quantitative variables. The sample coefficient, r, is the covariance standardized by the variables' standard deviations and ranges from -1 to +1 when defined. A population counterpart is commonly written rho. Correlation is unitless and does not by itself establish causation.
What it means
A positive coefficient means larger values of one variable tend to accompany larger values of the other in a linear pattern. A negative coefficient means larger values tend to accompany smaller ones. Values near zero indicate little linear association, which does not exclude a strong nonlinear relationship.
The magnitude is not a universal quality grade. Its practical meaning depends on the variables, population, measurement reliability, range, and decision. A moderate association can matter in one setting and be inadequate in another.
How it works
Collect correctly matched pairs of values and inspect a scatterplot. Check missing data, outliers, subgroup patterns, and whether the observations represent independent units or a dependent structure.
Subtract each variable's sample mean from its observations. Sum the products of corresponding deviations. Divide that cross-product sum by the square root of the product of the two squared-deviation sums.
The normalization removes measurement units. Adding a constant or multiplying a variable by a positive constant leaves r unchanged; multiplying by a negative constant reverses its sign. General nonlinear transformations can change the value.
If either variable is constant, its deviation sum is zero and correlation is undefined. Near-constant inputs can also create numerical difficulties. A software value of zero should not be substituted for an undefined result.
Computing r does not require normality, but inferential procedures need assumptions. The familiar exact null test uses an independent sample from a joint normal model with zero population correlation. Other testing or interval methods require their own sampling, resampling, or approximation conditions.
For simple ordinary least-squares regression with an intercept and one predictor, R squared equals r squared. This identity does not convert correlation into a causal explanation or a general measure of predictive accuracy outside the sample.
Account for dependence and selection. Time trends can create misleading correlations, repeated observations can overstate precision, and a restricted range can alter the coefficient. Report the data context and uncertainty rather than only a correlation matrix.
Three practical examples
- AI evaluation: researchers examine whether two numerical scoring measures rank performance similarly in a roughly linear manner.
- Operations: analysts explore the association between workload and response time before specifying a predictive model.
- Measurement review: a team checks linear association between two instruments while separately assessing bias and agreement.
Think of it this way
Think of points scattered around a straight path. Pearson correlation describes how consistently the points follow an upward or downward line. A perfectly curved path can have a weak or zero coefficient even though its points follow a very clear pattern.
Worked example
Use four original illustrative pairs: X = (1,2,3,4) and Y = (2,1,4,3). Both means are 2.5. The X deviations are (-1.5,-0.5,0.5,1.5), and the Y deviations are (-0.5,-1.5,1.5,0.5).
The paired deviation products are 0.75, 0.75, 0.75, and 0.75, summing to 3. Each variable's squared deviations sum to 5. Therefore r = 3/sqrt(5 times 5) = 0.60.
If X is converted to 10X + 100, its positive linear association with Y remains 0.60. If X is multiplied by -1, the coefficient becomes -0.60.
Now consider a separate set X = (-2,-1,0,1,2) and Y = (4,1,0,1,4). Here Y equals X squared exactly, yet the positive and negative cross-products cancel and r = 0.
The second example demonstrates why zero Pearson correlation is not proof of no relationship. Neither tiny constructed dataset supports a population significance claim; the purpose is to verify the calculation and distinguish linear association from general dependence.
Illustrative case study
Illustrative fictional case: Pine Metrics compares an AI-generated quality score with expert ratings. A correlation looks promising, but a scatterplot reveals a subgroup with consistently inflated AI scores.
The team reports the overall association, examines subgroup behavior, and adds agreement and calibration analyses. It also checks whether repeated ratings from the same reviewer require clustered uncertainty estimates. This fictional case illustrates why a single correlation coefficient is insufficient for validating a scoring system.
Common misconceptions and mistakes
Do not interpret correlation as causation. Do not interpret r = 0.60 as 60% accuracy or a 60% improvement. A coefficient of 1 can coexist with systematic measurement disagreement if one method consistently adds a fixed amount or uses a different scale.
Avoid ignoring outliers or combining distinct groups without inspection. Do not confuse statistical significance with a strong relationship. With many pairwise comparisons, some apparently notable correlations can arise by chance.
Business use
Pearson correlation is useful for exploratory analysis, checking redundant indicators, and communicating linear relationships. Business reports should include the sample size, relevant plot, missing-data policy, uncertainty, and any subgroup or dependence issues. For prediction or measurement validation, add task-specific performance measures rather than treating correlation as a complete scorecard.
People also ask
Can r be negative?
Yes; it indicates an inverse linear association.
Can r be zero despite a relationship?
Yes, as the squared example shows.
Is r affected by changing units?
Positive affine unit changes leave it unchanged.
Is high correlation the same as agreement?
No; agreement concerns how closely actual values match.
Check your understanding
What is the coefficient in the first example?
Answer: 0.60.
Why is the second coefficient zero?
Answer: Symmetric positive and negative contributions cancel despite an exact curved relationship.
What happens if a variable is constant?
Answer: Pearson correlation is undefined because its standard deviation is zero.
Related terms
- Covariance
- Coefficient of determination
- Partial correlation
- Confounding
- Spearman correlation
- Kendall correlation
- scatterplot
- agreement
Learn next
Next compare linear correlation with rank-based association and agreement measures. Then study uncertainty, outlier sensitivity, restricted range, and the risks of interpreting correlations in time-series or clustered data.
More in Statistics and probability
- Partial autocorrelation function - The partial autocorrelation function, or PACF, describes the linear association between a time series and its lagged values after accounting for the...
- Partialing - Partialing, also spelled partialling, is statistically adjusting for one or more variables when examining relationships among other variables.
- Partially open question - A partially open question is a survey question that combines structured response options with an opportunity to answer in the respondent's own words.
- Pearson's chi-square ( c 2 ) - Pearson's chi-square is a statistic that compares observed counts with counts expected under a specified null hypothesis.
- Per-comparison error rate - The per-comparison error rate is the Type I error probability associated with an individual hypothesis test: rejecting that test's null hypothesis when it...
- Per-word entropy - Per-word entropy expresses uncertainty in a word-generating process on an average-per-word basis.
Sources
SciPy, scipy.stats.pearsonr: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html — official developer documentation for the coefficient, formula, constant-input behavior, and inference assumptions. The full page was reviewed. The two numerical datasets, arithmetic checks, and fictional case are original; the supplied KNIME source was not separately verified for this entry.
Previous: Peak signal-to-noise ratio (PSNR) Next: Pearson's chi-square ( c 2 )