Interpretable Model

The defining characteristic of an interpretable model is that its internal mechanics are fully inspectable. Unlike black-box models that require surrogate explanations, every parameter, weight, and computation in a glass-box architecture can be examined directly.

• Decision trees expose a clear hierarchical path of if-then rules
• Linear regression reveals exact coefficient weights for each feature
• Generalized Additive Models (GAMs) show isolated shape functions per variable
• Rule-based systems provide explicit logical conditions

This transparency enables auditors to verify the model's logic without approximation.

A model is simulatable when a human can take the input data, mentally step through the entire computation, and arrive at the same prediction within a reasonable time. This property requires the model to be sufficiently compact.

• The entire reasoning process fits within human working memory
• No single computation step requires opaque mathematical transformations
• A domain expert can reproduce the output with pencil and paper
• Contrast with deep neural networks containing millions of interacting weights

Simulatability is the gold standard for high-stakes domains like medical diagnosis and credit adjudication.

An interpretable model exhibits decomposability when every component—each input feature, learned parameter, and intermediate calculation—carries an intuitively understandable meaning in isolation.

• Each coefficient in a linear model represents the marginal effect of one feature
• Each split in a decision tree corresponds to a single, named condition
• Each shape function in a GAM visualizes one variable's partial contribution
• No feature interactions are buried in entangled weight matrices

Decomposability allows domain experts to validate whether individual components align with established domain knowledge.

Beyond the trained parameters, the learning algorithm itself must be transparent. This means the training objective, optimization procedure, and convergence criteria are fully specified and auditable.

• The loss function defines exactly what the model optimizes
• The optimization method (e.g., CART for trees, OLS for regression) is deterministic
• No stochastic gradient descent with random initialization obscuring the path
• The hyperparameter search space is bounded and documented

Algorithmic transparency ensures reproducibility—retraining on identical data yields an identical or predictably similar model.

Interpretable models often encode causal structure rather than mere statistical correlation. When the model's architecture mirrors the known causal relationships in a domain, its predictions become mechanistically explainable.

• Structural causal models explicitly represent cause-effect relationships
• Bayesian networks encode conditional independence assumptions
• Decision trees can align with clinical decision protocols
• Linear models with domain-informed feature engineering capture known mechanisms

This causal alignment distinguishes genuine interpretability from models that merely output feature importance scores without explaining why relationships exist.

Interpretable models face a well-documented accuracy-interpretability trade-off. The constraints that make a model transparent—limited depth, additive structure, monotonicity—also restrict its capacity to capture complex non-linear patterns.

• Deep neural networks excel at raw perceptual tasks (images, audio) where interpretable models struggle
• Ensemble methods like random forests sacrifice simulatability for accuracy
• The trade-off is domain-dependent: tabular data with meaningful features often suits interpretable models
• Inherently interpretable neural architectures (e.g., prototype-based networks) are an active research frontier

For regulated industries, the cost of an unexplainable error often outweighs marginal accuracy gains.

A design philosophy prioritizing full internal transparency, where every parameter and computation is directly inspectable. Unlike black-box models, glass-box architectures like decision trees, linear regression, and generalized additive models (GAMs) allow auditors to trace the exact decision path from input to output without approximation. This enables deterministic verification of logic, making them preferred for high-stakes regulated domains.

A game-theoretic framework for feature attribution that assigns each input feature an importance value for a specific prediction. SHAP values guarantee local accuracy and consistency, meaning the sum of attributions equals the model output and features with larger impact receive higher values. While SHAP can explain any model, it is a post-hoc method—unlike natively interpretable models where explanations are intrinsic to the architecture.

A causal explanation method that identifies the minimal change to an input feature required to alter a model's prediction to a desired alternative outcome. For example: 'Your loan was denied because your income was $45K. If it were $52K, the decision would flip to approved.' Counterfactuals provide actionable recourse, directly supporting the right to explanation under GDPR and enabling users to understand how to achieve a different result.

A structured transparency document detailing a model's intended use, performance metrics, evaluation data, and known limitations. Standardized by Google, model cards transform interpretability from an internal property into an externally auditable artifact. For interpretable models, cards document the inherent explainability mechanism (e.g., decision paths, coefficient weights) rather than relying on surrogate explanations.

A technique for interrogating an opaque model's behavior by analyzing only its inputs and outputs without accessing internal weights. Auditors use statistical probes, sensitivity analysis, and adversarial testing to detect bias or regulatory violations. This method is necessary when interpretable models are not deployed, but it introduces uncertainty—unlike glass-box architectures where logic can be directly verified.

A legal principle codified in GDPR Article 22 and Recital 71, granting individuals the right to receive meaningful information about the logic involved in automated decisions. Interpretable models natively satisfy this requirement because their decision logic is self-documenting. Opaque models require supplementary post-hoc techniques, which may not meet the 'meaningful' threshold under strict regulatory interpretation.