AI Glossary / Mathematics and optimization / Intermediate
Coefficient of determination
Definition
The coefficient of determination, usually written R², compares a regression model's squared prediction errors with the squared deviations of the observed outcomes from their mean. In its common form, R²=1−SSE/SST, where SSE is the residual sum of squares and SST is the total sum of squares around that mean.
What it means
For a nonconstant set of outcomes, R²=1 indicates perfect predictions on those observations. R²=0 means the predictions have the same total squared error as predicting their mean every time. A negative value means the predictions have a larger squared error than that reference.
The score is dimensionless, but it depends on the variability of the evaluated outcomes. It is often described as a proportion of variance explained, particularly in ordinary least-squares fitting with an intercept. For general predictions, the squared-error comparison is the clearest interpretation. R² does not measure the proportion of individual predictions that are correct.
How it works
First calculate the mean of the true outcomes in the evaluated dataset. Subtract this mean from each outcome, square the differences, and sum them to obtain SST. Separately, subtract each prediction from its corresponding outcome, square the residuals, and sum them to obtain SSE.
Divide SSE by SST and subtract from one. Using sums avoids ambiguity about whether a variance formula divides by n or n−1. A formulation using average squared errors must use consistent denominators to produce the same score.
The calculation can be applied to training or held-out predictions, but the interpretation of evidence differs. Training performance describes fit to already-used observations. Held-out evaluation is more informative about generalization when the split represents the intended use. The reference mean in the usual score comes from the evaluated outcomes; it is a scoring reference, not necessarily a deployable forecasting baseline.
Three practical examples
1. Demand prediction: compare regression models on the same held-out outcomes, alongside error measures expressed in units of demand.
2. Model diagnostics: identify predictions that perform worse in squared error than the evaluated sample's mean, producing a negative R².
3. Reporting: show fit relative to outcome variation while separately reporting whether large errors occur in business-critical cases.
Think of it this way
Think of a contest against a simple reference that always gives the same answer: the evaluated outcomes' mean. R² measures how much squared error the model removes relative to that reference.
The analogy is limited because the reference uses information from the evaluated outcomes. It is not always the baseline that would have been available before making predictions. It also treats errors through squares, which may differ from the actual costs of overprediction and underprediction.
Worked example
Suppose the true outcomes are 2, 4, and 6. Their mean is 4, so SST=(2−4)²+(4−4)²+(6−4)²=4+0+4=8.
Model A predicts 3, 4, and 5. Its SSE=(2−3)²+(4−4)²+(6−5)²=1+0+1=2. Therefore R²=1−2/8=0.75. It removes 75% of the reference squared error on this small evaluated dataset; it does not get 75% of the three predictions exactly right.
Model B predicts 4 for every observation. Its SSE is 8 and R²=0. Model C predicts 5, 7, and 9, giving three residuals of −3, SSE=27, and R²=1−27/8=−2.375.
Model C also illustrates why R² is not generally squared correlation between predictions and outcomes. Its predictions are the outcomes plus a constant, so their correlation is one, yet their systematic offset produces a strongly negative prediction R². Special regression identities should not be generalized to arbitrary prediction vectors.
Illustrative case study
Fictional case: an analytics team presents a high training R² for a forecasting model. Its operations manager asks what errors occur on later periods and whether those errors affect staffing decisions.
The team adds a time-appropriate held-out evaluation, reports errors in demand units, and compares against a practical forecast baseline. It keeps R² as one useful summary but stops treating it as a complete measure of usefulness. The revised report distinguishes good historical fit from reliable operational performance without claiming that a single threshold applies to every forecasting problem.
Common misconceptions and mistakes
R² is not classification accuracy, a causal explanation, or a guarantee of useful predictions. It can be negative and is not generally the square of correlation between arbitrary predictions and outcomes.
When all evaluated outcomes are identical, SST is zero and the ordinary ratio is undefined. Libraries may apply special conventions for this case, so reporting should identify the handling. Also distinguish ordinary R² from adjusted R², which modifies the measure to account for model size in specified settings.
Business use
Use R² to communicate relative squared-error performance, especially when models are evaluated on the same data. Pair it with an error metric in meaningful units and a baseline available in actual use.
Comparisons across datasets need care because different target variation changes the denominator. A high R² can coexist with errors too large for a business decision. A modest R² may still accompany useful improvements over the operational alternative. The evaluation question should guide the metric combination.
People also ask
Can R² be negative?
Yes. The predictions can have more squared error than the evaluated outcomes' mean reference.
Does R²=0.8 mean 80% accuracy?
No. It describes a reduction in squared error relative to the stated reference, not a fraction of correct predictions.
What if the true outcomes are constant?
The ordinary denominator is zero. Software-specific replacement rules must be distinguished from the mathematical formula.
Check your understanding
An evaluation has SST=50 and a model has SSE=15. What is R²?
Answer: R²=1−15/50=0.7. The model's squared error is 30% of the mean-reference squared error on that evaluation set.
Related terms
- Residual
- Mean Squared Error
- Correlation
- Baseline
- Regression
- total sum of squares
- adjusted R²
- held-out evaluation
Learn next
compare R² with mean absolute error and root mean squared error on identical predictions. Then examine why training and held-out scores differ, and how a practical forecasting baseline differs from the mean used in the scoring formula.
More in Mathematics and optimization
- Absolute error loss - Absolute error loss measures a numerical prediction's error as the absolute difference between its predicted and reference values: L(y,y-hat)=|y-y-hat|.
- Active-set method - An active-set method is an optimization approach that works with a changing subset of constraints believed to be tight at the solution.
- Adjoint state method - The adjoint state method computes how a chosen objective changes when the parameters of a mathematical model change.
- Σ-algebra
- Algebraic connectivity - Algebraic connectivity is the second-smallest eigenvalue of a graph's Laplacian matrix.
- Algorithm IMED - Algorithm IMED, short for Indexed Minimum Empirical Divergence, is a method for choosing actions in a stochastic multi-armed bandit problem.
Sources
Scikit-learn, r2_score documentation, including negative values and constant-target handling: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html . Aalto Dictionary of Machine Learning, R² entry: https://aaltodictionaryofml.github.io/ADictML_Main.pdf . Relevant passages were read. Editorial clarification: sums specify consistent normalization; calculations and the correlation counterexample are original.
Previous: Coefficient matrix Next: Coefficient of multiple correlation