Post-Hoc Explanation

A defining property of many post-hoc methods is model-agnosticism. These techniques treat the original model as a black box, requiring only the ability to query inputs and receive outputs or prediction probabilities. They do not require access to internal parameters, gradients, or architectural details. This allows a single explanation framework to be applied across diverse model types—from gradient-boosted trees to deep neural networks—making it a universal auditing tool for heterogeneous enterprise AI portfolios.

Post-hoc explanations are generated after training is complete and the model is frozen. This retroactive nature means the explanation process does not influence the model's learning, weights, or inference logic. It is a purely analytical layer applied to a static artifact. This is critical for regulated industries where a model has already been validated and cannot be modified without re-certification. The explanation is a read-only operation on the decision function.

Post-hoc methods are categorized by their explanatory scope:

• Local Explanations: Explain a single, specific prediction (e.g., why a loan was denied for this applicant). Methods like LIME generate local surrogate models.
• Global Explanations: Explain the entire model's average behavior (e.g., what features are generally important). Methods like SHAP can aggregate local explanations for a global view. The choice of scope depends on whether the user is debugging an individual error or auditing systemic bias.

Many post-hoc techniques operate by training a separate, intrinsically interpretable surrogate model to mimic the black-box model's decision boundary. Common surrogates include sparse linear models or shallow decision trees. The surrogate is trained on a perturbed neighborhood of the input, learning a simple approximation of the complex local decision function. The fidelity of this surrogate—how accurately it mimics the black box—is the primary metric for explanation quality.

The primary output of most post-hoc methods is a feature attribution vector. This assigns a numerical importance score to each input feature, indicating its contribution to the final prediction. The sum of these attributions typically reconstructs the difference between the model's output and a baseline. This additive decomposition allows a complex, non-linear decision to be presented as a simple, human-readable bar chart of positive and negative influences.

Generating a post-hoc explanation is a separate computational task that adds latency to the prediction pipeline. For methods like LIME, this involves generating thousands of perturbation samples and training a surrogate model for each instance. For SHAP, it requires evaluating the model on all possible feature coalitions. This overhead is a key engineering trade-off: the cost of interpretability is paid in inference-time compute, not in training-time constraints.

A property of an explanation method indicating it can be applied to any machine learning model regardless of internal architecture. Post-hoc methods like LIME and SHAP are model-agnostic because they require only the ability to query the model's prediction function—no access to weights, gradients, or structure is needed.

• Operates via black-box access only
• Contrasts with model-specific methods like integrated gradients
• Enables uniform explanation workflows across heterogeneous model fleets

The complex, high-dimensional surface of the black-box model's prediction function in the immediate vicinity of a specific data instance. Post-hoc methods approximate this boundary with a simple surrogate to generate instance-level explanations.

• The surrogate model learns a linear separation of this local region
• Perturbation sampling explores the shape of this boundary
• Kernel width controls how "local" the approximation remains

A class of explanation methods that decompose a model's prediction into a sum of individual feature contributions relative to a baseline. Both LIME and SHAP belong to this family, producing explanations of the form: prediction = base_value + sum(feature_contributions).

• Provides a linear explanation of any output
• Each feature receives a signed importance score
• Enables direct comparison of feature effects across instances

The fundamental balancing act in local explanation methods between using a highly accurate but complex surrogate and a simple, human-understandable model that may lose some local precision.

• A deep surrogate achieves high fidelity but defeats the purpose of explanation
• A sparse linear model is interpretable but may miss non-linear interactions
• OptiLIME automates this trade-off by tuning kernel width

A type of model explanation that provides the specific reasons for a single, individual prediction, as opposed to a global explanation describing average model behavior. Post-hoc methods are inherently instance-level, answering "why did the model predict this for this specific input?"

• Contrasts with global explanation methods like feature importance plots
• Essential for debugging edge cases and auditing individual decisions
• Can be aggregated via Submodular Pick for global overviews

The property that a local explanation should remain consistent across multiple runs with different random seeds. Unstable explanations indicate that identified important features are artifacts of the perturbation sampling process rather than genuine model behavior.

• Measured by variance of feature importance scores across runs
• Bayesian LIME provides uncertainty estimates for explanations
• Larger neighborhood samples generally improve stability at computational cost