Which correlation should be used?

It depends on the measurement scales, relationship shape, and question; Pearson and rank-based measures summarize different properties.

Does negative correlation mean poor quality?

No; the sign describes direction, which may be desirable or undesirable depending on the context.

Can independent variables have nonzero sample correlation?

Yes, through sampling fluctuation; independence implies zero population Pearson correlation when the required moments exist.

AI Glossary / Statistics and probability / Intermediate

Correlation

Definition

Correlation describes a statistical association between variables, often summarized by a coefficient indicating its direction and strength. The most familiar form, Pearson correlation, measures linear association between two numerical variables. A positive value indicates a tendency for larger values to occur together; a negative value indicates a tendency for larger values of one variable to accompany smaller values of the other. Correlation alone does not establish causation.

What it means

Correlation helps answer whether variables vary together in the observed data or a specified population. It does not tell us how much one variable changes when the other is deliberately changed. The association could reflect a causal relationship, a common cause, selection, a shared time trend, or other features of the data.
The particular measure matters. Pearson correlation concerns linear patterns, while rank-based measures such as Spearman correlation can summarize monotonic patterns. A single number should be interpreted alongside the data's shape, measurement scale, and collection process.

How it works

Identify paired measurements that refer to the same observational units. Plot them to inspect direction, curvature, unusual observations, subgroups, and differences in spread. Verify units and coding, and decide how missing values will be handled before calculating a summary.
For Pearson correlation, subtract each variable's mean, multiply corresponding deviations, and add the products. Divide by the square root of the product of the two sums of squared deviations. The result lies between minus one and plus one when both variables have nonzero variation. A constant variable makes this coefficient undefined.
Values near either extreme indicate a strong linear pattern; zero means no linear association as measured by the coefficient. It does not rule out a nonlinear relationship. Changing units by positive linear rescaling leaves Pearson correlation unchanged, while reversing a scale reverses its sign.
Distinguish the sample correlation from the population quantity it estimates. Uncertainty depends on the sampling process and relevant assumptions. Repeated observations, clustering, or time dependence require appropriate analysis rather than treating all pairs as independent.
Examine whether aggregation, restricted ranges, outliers, or shared trends explain the pattern. In AI applications, check that an apparent relationship is available at prediction time and persists in an appropriate validation setting. A correlation found after searching many variables may reflect chance selection rather than a stable relationship.

Three practical examples

An operations team explores the association between order size and processing duration before building a model.
An AI team checks whether two input features contain similar linear information, while avoiding automatic deletion based solely on one coefficient.
A service analyst examines whether two performance measures move together and whether the relationship differs across customer segments.

Think of it this way

Think of correlation as a description of how two dancers move together. It can show that their movements are coordinated, but it does not establish who leads, whether both follow the same music, or whether the pattern will continue in another performance.

Worked example

Take five paired observations with X values 1, 2, 3, 4, 5 and Y values 2, 1, 4, 3, 5. Both means are three. The X deviations are -2, -1, 0, 1, 2, and the Y deviations are -1, -2, 1, 0, 2.
Their cross-products sum to 2 + 2 + 0 + 0 + 4 = 8. Each variable's squared deviations sum to ten. Pearson correlation is therefore 8/sqrt(10 x 10) = 0.8. This describes a positive linear association in these five pairs.
Now consider X values -2, -1, 0, 1, 2 and Y = X squared, giving 4, 1, 0, 1, 4. The cross-products of centered values sum to zero, so Pearson correlation is zero even though Y is determined exactly by X through a curved relationship.
Together, the examples show why correlation must be named and interpreted carefully. A coefficient of 0.8 does not mean eighty percent of cases are correctly predicted or that X causes Y. A coefficient of zero does not mean the variables have no relationship. Five observations also provide a very limited basis for population conclusions.

Illustrative case study

Fictional case: Birch Commerce finds that daily website visits and sales have a high correlation. The team initially treats visits as the sole driver of sales, but a closer review shows that promotions and seasonal demand affect both variables.
Analysts inspect the relationship within comparable periods, check time patterns, and separate forecasting usefulness from a causal claim about increasing visits. They also evaluate models on later data rather than assuming the historical correlation guarantees future accuracy. The association remains informative, but its practical meaning becomes more specific and better supported.

Common misconceptions and mistakes

Correlation is not a slope, a causal effect, or a measure of agreement between two instruments. Two measurements can correlate perfectly while one is consistently twice the other. A zero Pearson coefficient does not generally imply independence. Outliers and range restrictions can change the value substantially. Finally, statistical significance does not establish a large or useful association, especially with very large datasets.

Business use

Correlation is useful for exploration, monitoring, feature assessment, and identifying questions for further study. Pair the coefficient with a visualization, sample description, and uncertainty where appropriate. When the decision involves changing a process to improve an outcome, investigate causal evidence rather than translating an association directly into an intervention. When the goal is prediction, test usefulness on data that reflect deployment.

Check your understanding

What does r = 0.8 establish in the first example?

Answer: A positive linear association in the observed pairs.

Why is the second coefficient zero?

Answer: Positive and negative centered cross-products cancel despite the quadratic pattern.

Would converting X to a larger positive unit scale change Pearson r?

Answer: No.

Related terms

Kendall's tau
Covariance
Scatter plot
Confounding
Correlation coefficient
Pearson correlation
Spearman correlation
nonlinear association

Learn next

Compare scatter plots that share the same correlation but have different shapes. Then examine rank correlation, uncertainty intervals, and the difference between a useful predictor and a variable whose intervention has a causal effect.

Sources

NIST/SEMATECH e-Handbook, Scatter Plot: https://www.itl.nist.gov/div898/handbook/eda/section3/eda33q.htm NIST/SEMATECH e-Handbook, glossary, correlation: https://www.itl.nist.gov/div898/handbook/glossary.htm KNIME, Data Science Glossary: https://www.knime.com/blog/data-science-glossary