Heteroscedasticity

The defining characteristic of heteroscedasticity is that the variance of the residuals (errors) changes systematically with the value of an independent variable or the predicted value. This violates the homoscedasticity assumption of ordinary least squares (OLS) regression.

• Visual Pattern: In a plot of residuals vs. predicted values or an independent variable, the spread of points forms a funnel shape (e.g., widening or narrowing), not a random band.
• Example: In a model predicting house prices based on square footage, the variability in price (error) is often much larger for multi-million dollar mansions than for modest homes, creating a fan-shaped residual plot.

While OLS coefficient estimates remain unbiased, heteroscedasticity invalidates the standard formulas for standard errors, t-statistics, and F-statistics.

• Consequence: Standard errors become biased, leading to incorrect confidence intervals and misleading hypothesis tests (p-values). You may falsely declare a variable significant (Type I error) or fail to detect a real effect (Type II error).
• Core Issue: OLS assumes a single, constant variance (σ²) for all errors. Heteroscedasticity means this assumption is false, so the classical covariance matrix of the coefficients is incorrect.

Several formal tests and visual diagnostics are used to detect heteroscedasticity.

• Visual Inspection: Plotting studentized residuals or standardized residuals against predicted values is the first diagnostic step. Look for systematic patterns.
• Breusch-Pagan Test: A Lagrange multiplier test that regresses squared residuals on the independent variables. A significant result indicates heteroscedasticity.
• White Test: A more general test that also includes cross-products of independent variables, detecting a wider range of heteroscedastic forms.
• Goldfeld-Quandt Test: Splits the data into two groups and compares the variance of residuals from separate regressions, useful when variance increases with a specific variable.

Heteroscedasticity often signals a deeper problem with the regression model itself, not just the error structure.

• Omitted Variables: The model may be missing a key predictor that is correlated with the scale of the errors.
• Incorrect Functional Form: Using a linear model for a non-linear relationship can manifest as heteroscedastic residuals. A log transformation of the dependent variable can sometimes stabilize variance.
• Skewed Data: Data with a highly skewed distribution (e.g., income, network latency) naturally exhibits changing variance. Weighted Least Squares (WLS) is a direct remedy, assigning less weight to observations with higher error variance.

The most common practical solution is to use heteroscedasticity-consistent standard errors (HCSE), such as White's robust standard errors or the more refined HC3 estimator.

• Mechanism: These methods compute a new covariance matrix for the coefficients that does not rely on the homoscedasticity assumption, providing valid inference even in the presence of heteroscedasticity.
• Advantage: Coefficient estimates remain the same (OLS), but their reported standard errors, t-statistics, and p-values become reliable. This is often implemented as a post-estimation correction in statistical software.

In predictive modeling, heteroscedasticity directly impacts error analysis and model selection.

• Loss Function Sensitivity: Metrics like Mean Squared Error (MSE) are highly sensitive to large errors in high-variance regions, potentially skewing model evaluation.
• Quantile Regression: An alternative to OLS that models different percentiles (e.g., the median, 90th percentile) of the dependent variable, providing a more complete picture when variance is not constant.
• Anomaly Detection Context: Heteroscedasticity complicates anomaly detection; a residual considered large in a low-variance region might be normal in a high-variance region. Models must account for this conditional variance.

Applying a mathematical transformation to the dependent variable (Y) or predictor variables (X) can stabilize the variance. Common transformations include:

• Logarithmic Transformation: log(Y) or log(X) is highly effective when the variance increases with the level of the variable.
• Square Root Transformation: sqrt(Y) is useful for count data.
• Box-Cox Transformation: A more generalized power transformation that finds an optimal lambda parameter to stabilize variance.

These transformations aim to make the relationship more linear and the error variance more constant, though they can complicate the interpretation of coefficients.

Weighted Least Squares is a direct generalization of OLS used when the variance of the errors is known or can be estimated. Instead of minimizing the sum of squared residuals, WLS minimizes a weighted sum, giving less influence to observations with higher error variance.

Process:

1. Estimate the error variance for different segments of the data (e.g., by grouping or using an auxiliary regression).
2. Define weights inversely proportional to the estimated variances: weight_i = 1 / variance_i.
3. Perform regression using these weights.

WLS provides Best Linear Unbiased Estimators (BLUE) under the new, known heteroscedasticity structure.

Also known as Heteroscedasticity-Consistent Standard Errors (e.g., White-Huber-Eicker standard errors), this method does not alter the OLS coefficient estimates but corrects the estimated standard errors and test statistics to be valid in the presence of heteroscedasticity of an unknown form.

Key Advantage: It is a post-estimation correction that protects against incorrect inferences (p-values, confidence intervals) without changing the model's functional form or requiring knowledge of the exact variance structure. It is the most commonly applied remedy in econometrics and many social sciences due to its simplicity and robustness.

Generalized Least Squares is the most comprehensive framework, of which WLS is a special case. GLS directly models the covariance structure of the errors. It transforms the original model to satisfy the homoscedasticity assumption.

Method: If the variance-covariance matrix of the errors is Ω, GLS applies a transformation using Ω⁻¹⁄² to the data, resulting in a model with spherical errors (constant variance and no correlation). The estimator is given by: β_GLS = (X'Ω⁻¹X)⁻¹X'Ω⁻¹y.

GLS is asymptotically efficient but requires specifying or estimating the full error covariance matrix Ω, which can be complex.

Heteroscedasticity often signals a model misspecification. Remedies involve fundamentally rethinking the model's functional form:

• Adding Omitted Variables: Heteroscedasticity may arise from leaving out a key predictor that interacts with the error term.
• Including Interaction Terms or Polynomials: If variance changes with X, the relationship between Y and X may be non-linear or involve interactions.
• Switching Model Type: For certain data types, alternative models inherently handle non-constant variance:
  • Generalized Linear Models (GLMs): For example, using a Poisson regression for count data or a Gamma regression for strictly positive, right-skewed data.
  • Quantile Regression: Models the conditional median or other quantiles, making it robust to heteroscedasticity and outliers.

Remedying heteroscedasticity is often an iterative process guided by diagnostics:

1. Test: Use tests like the Breusch-Pagan or White test to confirm its presence.
2. Visualize: Plot residuals against fitted values or predictors to identify the variance pattern (e.g., funnel shape).
3. Choose & Apply Remedy: Select a technique based on the diagnosed pattern (e.g., log transform for a proportional pattern, WLS for group-wise variance).
4. Re-diagnose: After applying a remedy, perform residual analysis again to check if heteroscedasticity has been mitigated. The goal is to achieve a plot of residuals that shows no systematic pattern in the spread.

Homoscedasticity is the assumption that the variance of the error term (or residuals) is constant across all levels of the independent variables. It is the ideal condition violated by heteroscedasticity.

• Constant variance is a core assumption of Ordinary Least Squares (OLS) regression.
• When homoscedasticity holds, estimators are efficient (have minimum variance).
• Most statistical tests for heteroscedasticity, like the Breusch-Pagan test, test the null hypothesis of homoscedasticity.

Weighted Least Squares is a corrective regression technique used when heteroscedasticity is present. It modifies the ordinary least squares approach to account for unequal error variances.

• Observations with higher variance (more noise) are given less weight in the fitting process.
• Observations with lower variance (more reliable) are given more weight.
• This reweighting produces more precise and reliable parameter estimates than standard OLS under heteroscedastic conditions.

Robust standard errors (e.g., Huber-White or sandwich estimators) are a post-estimation correction that provides valid inference even when heteroscedasticity is present.

• They do not change the coefficient estimates from OLS.
• They adjust the estimated standard errors of the coefficients to be consistent despite non-constant variance.
• This method is widely used in econometrics and social sciences as a practical fix for heteroscedasticity without altering the model.

Applying a logarithmic transformation to the dependent variable is a common remedial measure for heteroscedasticity, particularly when variance increases with the mean.

• It compresses the scale of large values, often stabilizing variance across the range of data.
• This transformation can also help linearize relationships.
• It is a form of variance-stabilizing transformation, alongside square root or reciprocal transformations, chosen based on the nature of the variance trend.

The Breusch-Pagan test is a formal statistical hypothesis test used to detect the presence of heteroscedasticity in a linear regression model.

• It tests whether the estimated variance of the residuals is dependent on the values of the independent variables.
• A significant p-value (typically < 0.05) leads to rejection of the null hypothesis of homoscedasticity.
• It is a Lagrange multiplier test that regresses the squared residuals on the original independent variables.