Locally Weighted Regression

LWR is a lazy learner that performs no explicit training phase. Instead, it stores the entire training dataset and defers all computation until a query is made. When a prediction is required for a specific point, the algorithm dynamically retrieves the relevant local neighborhood from memory, fits a simple model, and discards it. This contrasts with eager learners like neural networks that compress data into fixed parameters during training. The memory-based approach allows LWR to adapt its hypothesis complexity to the query point without retraining.

The core mechanism of LWR is a weighting function that assigns importance to each training point based on its distance from the query instance. The standard choice is the tricube kernel:

• Points exactly at the query location receive full weight (1.0)
• Points beyond a bandwidth parameter h receive zero weight
• The weight decays smoothly in between, creating a soft boundary This ensures that the local model is dominated by the most proximate data, enforcing the locality constraint critical to LIME's fidelity.

The bandwidth or smoothing parameter h controls the size of the local neighborhood and fundamentally governs the bias-variance trade-off:

• Small bandwidth: High variance, low bias. The model fits very local patterns but is sensitive to noise. Captures fine detail at the risk of overfitting.
• Large bandwidth: Low variance, high bias. The model smooths over more data, approaching a global linear regression. More stable but may miss local structure. Optimal bandwidth selection is typically performed via cross-validation or analytical methods like the Akaike Information Criterion (AIC).

Within the weighted neighborhood, LWR fits a low-degree polynomial—typically linear or quadratic—using weighted least squares. The objective function minimizes: Σ w(x_i, x_query) * (y_i - β₀ - β₁x_i - ...)² A degree-0 fit produces a locally constant estimate (kernel smoothing). A degree-1 fit (local linear regression) corrects boundary bias and is the most common choice. The prediction for the query point is simply the fitted polynomial evaluated at the query location. Higher-degree polynomials capture more curvature but increase variance.

LWR makes no assumptions about the global functional form of the underlying data-generating process. Unlike parametric models that impose a fixed structure (e.g., linear, exponential), LWR adapts to arbitrary nonlinear relationships by stitching together local approximations. This flexibility is the reason LIME uses LWR as its surrogate: a sparse linear model fitted with local weights can approximate any complex decision boundary in a small region. The trade-off is that LWR requires the full dataset at inference time and scales linearly with data size.

In the LIME algorithm, LWR is the engine that trains the interpretable surrogate model. The process maps directly to LWR mechanics:

• Perturbation samples become the training data points
• Cosine or Euclidean distance from the original instance determines weights via an exponential kernel
• A sparse linear model (Lasso) is fitted using these weighted samples
• The resulting coefficients become the feature importance scores in the explanation The kernel width in LIME directly corresponds to LWR's bandwidth, controlling how 'local' the explanation remains.

A distance-based weighting function that assigns higher importance to perturbed samples closer to the original instance. The kernel defines the effective neighborhood size by controlling how quickly sample weights decay with distance.

• Enforces the locality constraint of the explanation
• Typically uses a radial basis function (RBF) kernel
• Weight calculation: exp(-distance^2 / kernel_width^2)
• Samples far from the target instance receive near-zero weight

A hyperparameter controlling the effective size of the local neighborhood. It determines how quickly sample weights decay with distance from the original instance, directly balancing local fidelity against explanation stability.

• Too narrow: overfits to noise, unstable explanations
• Too wide: loses locality, approximates global behavior
• Optimal selection is critical for meaningful explanations
• OptiLIME automates this selection process

The process of generating a synthetic neighborhood of data points by randomly altering or masking features of the original instance. This creates the training set for the local surrogate model.

• Text data: token masking or removal
• Image data: superpixel masking (turning segments on/off)
• Tabular data: value perturbation around the instance
• Each perturbed sample receives a proximity weight from the kernel

An interpretable surrogate model that uses Lasso (L1) regularization to select only a small number of the most important features. This ensures the final explanation remains concise for human consumption.

• L1 penalty forces irrelevant coefficients to exactly zero
• Produces a linear combination of weighted features
• Balances fidelity-interpretability trade-off
• The non-zero coefficients become the explanation

A measure of how accurately the interpretable surrogate model approximates the behavior of the underlying black-box model in the immediate neighborhood of the instance being explained.

• Measured via weighted squared error on perturbed samples
• High fidelity: surrogate closely matches black-box locally
• Low fidelity: explanation may be misleading or incomplete
• The fidelity-interpretability trade-off governs surrogate complexity

The specific strategy for creating perturbed data points, which varies by data modality. Each domain requires a tailored approach to produce meaningful, realistic variations around the target instance.

• Text: token masking or word removal from documents
• Images: superpixel masking (contiguous region occlusion)
• Tabular: sampling from feature distributions or quantiles
• Feature discretization often precedes neighborhood generation