R-squared (Coefficient of Determination)

R-squared is defined as the proportion of the variance in the dependent variable (y) that is predictable from the independent variable(s) (X). It is calculated as:

R² = 1 - (SS_res / SS_tot)

• SS_res (Sum of Squares Residual): The sum of squared differences between observed values and model-predicted values.
• SS_tot (Total Sum of Squares): The sum of squared differences between observed values and the mean of the dependent variable.

A value of 1 indicates perfect prediction, while 0 indicates the model explains none of the variance around the mean.

In ordinary least squares (OLS) linear regression, R-squared has a clear, bounded interpretation:

• Explained Variance: Directly represents the fraction of total variance 'explained' by the linear model.
• Goodness-of-Fit: A higher R-squared indicates a better fit of the model to the data.
• Caveat: It does not indicate whether:
  • The independent variables are causally related to the dependent variable.
  • The model is correctly specified (e.g., omitting a key variable).
  • The coefficient estimates are unbiased.

It is a descriptive, not a causal, measure of fit.

R-squared is frequently misinterpreted. Key limitations include:

• Non-Comparative Across Datasets: A high R-squared on data with high inherent variance is not comparable to a lower R-squared on data with low variance.
• Sensitivity to Outliers: A single outlier can artificially inflate or deflate R-squared.
• No Indication of Bias: A model can have a high R-squared but produce systematically biased predictions (poor calibration).
• Always Increases with Predictors: Adding any variable, even random noise, will never decrease R-squared in OLS, leading to overfitting. This is addressed by the Adjusted R-squared.

Adjusted R-squared penalizes the addition of non-informative predictors to counteract overfitting. It is calculated as:

Adjusted R² = 1 - [(1 - R²) * (n - 1) / (n - k - 1)]

• n: Number of observations.
• k: Number of independent variables.

Unlike standard R-squared, Adjusted R-squared can decrease when a new predictor adds less explanatory power than expected by chance, providing a more reliable metric for model comparison, especially with multiple predictors.

For non-linear models (e.g., polynomial regression, decision trees, neural networks), the interpretation of R-squared changes:

• It remains a measure of explained variance but loses its direct connection to OLS properties.
• It can be negative for models that fit worse than a simple horizontal line (the mean). This occurs when SS_res > SS_tot.
• In machine learning, it is often called the coefficient of determination or R² score (sklearn.metrics.r2_score). It is a useful metric for regression tasks but should be evaluated alongside Mean Squared Error (MSE) or Mean Absolute Error (MAE) to understand error magnitude.

When evaluating R-squared in practice:

• Context is Critical: An R-squared of 0.7 may be excellent in social sciences (high noise) but poor in physics experiments.
• Use with Other Metrics: Always pair with residual analysis, MSE/MAE, and prediction error plots to diagnose model flaws.
• Focus on Out-of-Sample Performance: A high training R-squared with a low validation R-squared signals overfitting.
• Prioritize Adjusted R-squared for multiple regression to compare models with different numbers of features.
• Remember its Domain: It is a variance-based metric; for probabilistic or classification-calibrated regression, also consider Log Loss or Brier Score.

Adjusted R-squared modifies the standard R-squared to account for the number of predictors in a model. Unlike R-squared, which always increases with added variables, Adjusted R-squared penalizes model complexity, increasing only if a new predictor improves the model more than would be expected by chance.

• Key Formula: 1 - [(1 - R²)(n - 1)/(n - k - 1)], where n is sample size and k is the number of independent variables.
• Primary Use: Comparing models with different numbers of predictors to prevent overfitting. It is the preferred metric for multiple regression model selection.
• Interpretation: A model with a higher Adjusted R-squared than another is generally considered to have a better specification, balancing fit and parsimony.

Mean Squared Error is a fundamental regression loss function that measures the average of the squares of the errors—i.e., the average squared difference between the estimated values and the actual value. It is the quantity that ordinary least squares regression directly minimizes.

• Calculation: MSE = (1/n) * Σ(actual - predicted)².
• Relationship to R-squared: R-squared is derived from MSE. It is calculated as 1 - (MSE of model / Variance of target). While R-squared is a relative, unitless measure of fit, MSE is an absolute measure of error in the units of the target variable squared.
• Key Property: It heavily penalizes large errors due to the squaring operation, making it sensitive to outliers.

Root Mean Squared Error is the square root of the Mean Squared Error. It is one of the most commonly used regression metrics because it is expressed in the same units as the target variable, making it more interpretable than MSE.

• Calculation: RMSE = √(MSE).
• Interpretation: It can be read as a "typical" magnitude of error. For example, an RMSE of 5 for a house price model in thousands of dollars means predictions are typically off by about $5,000.
• Comparison to R-squared: While R-squared explains the proportion of variance, RMSE provides a direct, tangible measure of prediction error. A model can have a high R-squared but still have a practically large RMSE if the underlying variance of the data is high.

The Regression F-statistic tests the overall significance of a linear regression model. It assesses whether there is a linear relationship between any of the independent variables and the dependent variable, as a group.

• Null Hypothesis: All regression coefficients are equal to zero (the model explains no variance).
• Relationship to R-squared: The F-statistic is directly related to R-squared: F = (R² / k) / ((1 - R²) / (n - k - 1)). A high R-squared will generally produce a high F-statistic, leading to rejection of the null hypothesis.
• Usage: It is the first test reported in regression output. A significant F-statistic (typically p-value < 0.05) is necessary to justify interpreting the R-squared value; otherwise, the observed fit may be due to chance.

Residual Analysis is the examination of the errors (residuals = actual - predicted) left after fitting a regression model. It is a critical diagnostic tool that validates the assumptions underlying R-squared and linear regression.

Key checks performed include:

• Independence: Residuals should not be correlated with each other (no autocorrelation).
• Homoscedasticity: The variance of residuals should be constant across all levels of the predicted value.
• Normality: Residuals should be approximately normally distributed, especially for inference on coefficients.
• Linearity: The relationship between variables should be linear; patterns in a residual vs. fitted plot indicate non-linearity.

A high R-squared value is misleading if residual analysis reveals violated assumptions, as the model's inferences and predictions become unreliable.

In the broadest statistical sense, the Coefficient of Determination is a measure of how well observed outcomes are replicated by a model, based on the proportion of total variation of outcomes explained by the model. R-squared is its specific implementation for linear regression.

• General Principle: It compares the variance of the model's errors to the variance of the data itself.
• Beyond OLS: The concept extends to other models:
  • Logistic Regression: Pseudo R-squared measures (e.g., McFadden's) approximate explained variance.
  • Nonlinear Models: Computed as the square of the correlation between observed and predicted values.
• Core Interpretation: Regardless of the model, it answers: "What percentage of the movement in the target variable can be accounted for by the model's inputs?" This makes it a universally sought-after, if sometimes misapplied, indicator of model utility.