Glossary
Algorithmic Explainability and Interpretability
Feature Attribution Methods
Terms related to algorithms that assign importance scores to input features for a specific model prediction. Target: CTOs and ML engineers auditing model decisions.
Integrated Gradients
An axiomatic attribution method that computes feature importance by accumulating gradients along a straight-line path from a baseline input to the actual input, satisfying the completeness axiom.
DeepLIFT
A backpropagation-based attribution method that explains predictions by comparing neuron activations to a reference activation, distributing contribution scores according to differences from that reference state.
Layer-wise Relevance Propagation
A decomposition technique that redistributes a model's prediction score backward through the network layers using local conservation rules to assign relevance scores to individual input features.
SmoothGrad
A sensitivity map sharpening technique that reduces visual noise in gradient-based attributions by averaging the gradients computed from multiple noisy copies of the same input image.
Expected Gradients
An extension of Integrated Gradients that removes the need for a single baseline by averaging attributions over multiple background samples drawn from the training data distribution.
Guided Backpropagation
A visualization technique that modifies the standard backpropagation of ReLU activations to only propagate positive gradients and positive activations, resulting in sharper feature visualizations.
Grad-CAM
A localization technique that uses the gradients of a target concept flowing into the final convolutional layer to produce a coarse heatmap highlighting the important regions in an image.
Score-CAM
A perturbation-based visual explanation method that eliminates gradient dependency by weighting activation maps using the model's confidence score increase when each map is used as a mask.
FullGrad
An attribution method that satisfies completeness by aggregating the gradient of the output with respect to both the input features and the biases at every layer of the network.
Saliency Maps
A foundational visualization technique that ranks input features by computing the absolute value of the partial derivative of the model's output with respect to each input dimension.
Occlusion Sensitivity
A perturbation-based attribution method that systematically occludes different regions of an input with a gray square and measures the resulting change in the model's prediction probability.
Permutation Feature Importance
A model-agnostic global explanation method that measures the drop in a model's performance score after randomly shuffling the values of a single feature, breaking its association with the target.
KernelSHAP
A model-agnostic implementation of Shapley additive explanations that uses a weighted linear regression on a sample of coalitions to estimate SHAP values for any black-box model.
TreeSHAP
A fast, exact algorithm for computing Shapley additive explanations specifically optimized for tree-based ensemble models like XGBoost, LightGBM, and random forests.
Partial Dependence Plots
A global visualization tool that shows the marginal effect of one or two features on the predicted outcome by averaging the model's predictions over the marginal distribution of all other features.
Individual Conditional Expectation
A disaggregated visualization technique that plots the functional relationship between a feature and the prediction for every single instance, revealing heterogeneous effects hidden by partial dependence plots.
Accumulated Local Effects
An unbiased alternative to partial dependence plots that computes feature effects by averaging local differences in predictions within conditional partitions, remaining valid under correlated features.
RISE
A black-box attribution method that generates importance maps by averaging thousands of randomly generated binary masks, weighted by the model's confidence score on the correspondingly masked input.
Extremal Perturbation
An optimization-based attribution method that finds the smallest smooth mask that maximally preserves or destroys a model's prediction, producing compact and interpretable saliency maps.
Influence Functions
A robust statistics technique adapted for deep learning that estimates the effect of removing a specific training point or perturbing a feature on a model's prediction by computing Hessian-vector products.
TracIn
A training data attribution method that estimates the influence of a training example on a prediction by tracking the cumulative change in loss on that example throughout the model's training checkpoints.
Activation Maximization
A feature visualization technique that synthesizes an input pattern that maximally activates a specific neuron, channel, or layer by performing gradient ascent in the input space.
Network Dissection
An analytical framework that quantifies the interpretability of individual hidden units in a CNN by measuring the alignment between unit activations and human-labeled semantic concepts.
Attribution Prior
A regularization technique that encodes domain-specific expectations about what features a model should focus on directly into the training objective by penalizing undesirable gradient patterns.
Faithfulness Metric
A quantitative evaluation criterion that measures how accurately an attribution map reflects the true importance of features by perturbing the most important inputs and measuring the prediction drop.
ROAR
An evaluation protocol that iteratively retrains a model after removing a fraction of the most important features according to an attribution method, measuring the resulting degradation in performance.
Insertion Metric
An evaluation metric that measures the quality of an attribution map by incrementally adding pixels from most to least important into a blurred baseline and recording the increase in model probability.
Deletion Metric
An evaluation metric that measures the quality of an attribution map by incrementally removing pixels from most to least important and recording the resulting decay in the model's prediction probability.
Causal Scrubbing
A mechanistic interpretability technique that tests a hypothesized circuit by resampling activations from a corrupted dataset and verifying that the model's behavior is fully mediated by the identified subgraph.
Axiomatic Attribution
A theoretical framework that defines desirable mathematical properties—such as completeness, sensitivity, and implementation invariance—that a feature attribution method must satisfy to be considered reliable.
Shapley Additive Explanations
Terms related to the game-theoretic SHAP framework for explaining model outputs by fairly distributing credit among input features. Target: Data scientists and compliance officers requiring consistent feature importance.
SHAP (SHapley Additive exPlanations)
A game-theoretic framework for explaining the output of any machine learning model by computing the Shapley value for each input feature.
Shapley Values
A solution concept from cooperative game theory that fairly distributes the total payout of a coalition to each player based on their marginal contribution across all possible coalitions.
Additive Feature Attribution
A class of explanation models that express a prediction as a linear sum of individual feature contributions relative to a baseline.
KernelSHAP
A model-agnostic SHAP implementation that uses a weighted linear regression with a specially designed kernel to approximate Shapley values efficiently.
TreeSHAP
A model-specific SHAP algorithm that computes exact Shapley values for tree-based models like XGBoost and random forests in polynomial time.
DeepSHAP
A high-speed approximation algorithm for computing SHAP values in deep learning models by combining DeepLIFT with Shapley value calculations.
Marginal Contribution
The difference in a model's prediction when a specific feature is added to a subset of other features, forming the basis of Shapley value computation.
Baseline Value
The expected model output across the background dataset, representing the prediction when no feature information is known.
Efficiency Property
A Shapley axiom ensuring that the sum of all feature attributions exactly equals the difference between the model's prediction and the baseline value.
Local Accuracy
The property guaranteeing that an explanation model matches the original model's output for a specific input instance.
Missingness
A SHAP property requiring that features not present in the original input are assigned an attribution of zero.
Consistency
A Shapley property stating that if a model changes so a feature's contribution increases, its SHAP value should not decrease.
SHAP Interaction Values
An extension of SHAP that captures pairwise feature interaction effects by distributing credit among all pairs of features.
Background Dataset
A representative sample of data used to compute the expected model output and to impute missing features during SHAP value estimation.
Interventional SHAP
A causal interpretation of SHAP that breaks feature correlations by sampling from the marginal distribution, reflecting the model's behavior under intervention.
Observational SHAP
A SHAP formulation that preserves feature correlations by conditioning on observed values, reflecting the model's behavior on the natural data manifold.
SHAP Summary Plot
A visualization combining feature importance with feature effects, displaying the distribution of SHAP values for each feature across a dataset.
SHAP Waterfall Plot
A visualization that decomposes a single prediction by showing how each feature pushes the model output from the baseline value to the final prediction.
SHAP Force Plot
An interactive visualization that stacks feature contributions to show the additive forces pushing a prediction higher or lower from the baseline.
SHAP Dependence Plot
A scatter plot showing the relationship between a feature's value and its SHAP value, optionally colored by an interacting feature to reveal interaction effects.
Coalitional Game Theory
The mathematical field studying how groups of players form coalitions and distribute payoffs, providing the theoretical foundation for Shapley values.
Feature Independence Assumption
The simplifying assumption that input features are uncorrelated, which reduces computational complexity but may produce unrealistic model evaluations.
Conditional Expectation
The statistical method of estimating missing feature values by conditioning on known features, used in observational SHAP to preserve feature correlations.
Shapley Value Estimation
The process of approximating exact Shapley values using sampling techniques when exhaustive computation over the power set is infeasible.
Variance Reduction
Techniques like antithetic and paired sampling used in SHAP estimation to decrease the number of model evaluations needed for convergence.
Causal SHAP
An extension of SHAP that incorporates a causal graph to compute feature attributions respecting the underlying causal structure of the data.
Global Feature Importance
A measure of overall feature impact derived by averaging the absolute SHAP values for each feature across all instances in a dataset.
Local Feature Importance
The attribution of a model's prediction for a single instance to its constituent input features, providing instance-level explanation.
Explanation Model
A simple, interpretable surrogate model, such as a linear function of binary variables, used to approximate the behavior of a complex model locally.
Shapley-Taylor Index
A generalization of SHAP interaction values that quantifies the importance of interactions among any specified number of features.
Local Interpretable Model Explanations
Terms related to the LIME technique for approximating complex model predictions locally with interpretable surrogate models. Target: Engineers debugging individual predictions.
Local Interpretable Model-agnostic Explanations (LIME)
A technique that explains the prediction of any classifier by approximating it locally with an interpretable surrogate model trained on perturbed samples around the instance of interest.
Surrogate Model
An intrinsically interpretable model, such as a linear regression or decision tree, trained to mimic the predictions of a complex black-box model within a specific local region.
Local Fidelity
A measure of how accurately an interpretable surrogate model approximates the behavior of the underlying black-box model in the immediate neighborhood of the instance being explained.
Interpretable Representation
A human-understandable transformation of the original input features, such as a bag-of-words for text or superpixels for images, used as the input space for the surrogate model.
Perturbation Sampling
The process of generating a synthetic neighborhood of data points by randomly altering or masking features of the original instance to create a training set for the local surrogate model.
Superpixel Segmentation
An image preprocessing step that groups contiguous pixels with similar perceptual characteristics into larger, semantically meaningful atomic regions used as the interpretable features for image explanations.
Exponential Kernel
A distance-based weighting function used in LIME to assign higher importance to perturbed samples that are closer to the original instance, enforcing the locality constraint of the explanation.
Sparse Linear Model
An interpretable surrogate model that uses Lasso regression or similar regularization to select only a small number of the most important features, ensuring the final explanation remains concise for human consumption.
Lasso Regression
A linear regression method with an L1 regularization penalty that forces the coefficients of less important features to exactly zero, creating a sparse and interpretable local explanation model.
Submodular Pick
An algorithm for selecting a diverse and representative set of individual instance explanations to provide a non-redundant global overview of model behavior for human review.
Neighborhood Generation
The specific strategy for creating perturbed data points, which varies by data modality—such as token masking for text, superpixel masking for images, or value perturbation for tabular data.
Feature Discretization
The process of converting continuous numerical features into categorical bins or quantiles to create a more human-friendly and interpretable domain for the local surrogate model.
Kernel Width
A hyperparameter controlling the effective size of the local neighborhood by determining how quickly sample weights decay with distance from the original instance, balancing local fidelity and explanation stability.
Explanation Stability
The property that a local explanation should remain consistent across multiple runs with different random seeds, indicating that the identified important features are robust rather than artifacts of the sampling process.
Fidelity-Interpretability Trade-off
The fundamental balancing act in local explanation methods between using a highly accurate but complex surrogate and a simple, human-understandable model that may lose some local precision.
Model-Agnostic
A property of an explanation method indicating that it can be applied to any machine learning model regardless of its internal architecture, requiring only the ability to query the model's prediction function.
Local Decision Boundary
The complex, high-dimensional surface of the black-box model's prediction function in the immediate vicinity of a specific data instance, which the local surrogate model attempts to approximate with a simple linear separation.
Anchor Explanations
A successor to LIME that provides high-precision if-then rules, called anchors, which sufficiently guarantee a prediction will remain unchanged regardless of changes to other feature values not in the rule.
Cosine Distance
A proximity measure used in LIME for text data that calculates the similarity between two documents based on the angle between their TF-IDF vector representations, ignoring differences in document length.
Locally Weighted Regression
A non-parametric regression method that fits a simple model to a localized subset of data, weighting points by their proximity to the target instance, forming the statistical backbone of the LIME algorithm.
Additive Feature Attribution
A class of explanation methods, including LIME and SHAP, that decompose a model's prediction into a sum of individual feature contributions relative to a baseline, providing a linear explanation of the output.
Post-Hoc Explanation
An explanation generated after a model has been trained, applied to an existing black-box system without requiring any modification to the model's internal architecture or training procedure.
Instance-Level Explanation
A type of model explanation that provides the specific reasons for a single, individual prediction, as opposed to a global explanation that describes the average behavior of the entire model.
Explanation Regularization
A technique that adds a penalty term to the surrogate model's training objective to enforce desirable properties like sparsity or smoothness, directly controlling the complexity of the generated explanation.
Feature Importance
A numerical score assigned to each input feature by the surrogate model, indicating the magnitude and direction of that feature's contribution to the specific prediction being explained.
OptiLIME
An optimization framework that automatically selects the optimal kernel width for a LIME explanation by balancing the trade-off between local fidelity and the stability of the explanation across different runs.
Bayesian LIME
An extension of LIME that uses Bayesian ridge regression as the surrogate model to provide uncertainty estimates alongside feature importance scores, indicating the confidence of the explanation itself.
Saliency Mask
A visual representation of an image explanation where the most important superpixels for a prediction are highlighted, creating a heatmap that shows which regions of the image drove the model's decision.
Explanation Pipeline
The end-to-end sequence of data transformation, perturbation, surrogate training, and feature selection steps required to generate a human-readable explanation from a raw data instance and a black-box model.
Black-Box Access
A model access scenario where the explanation system can only query the model with inputs and receive outputs or prediction probabilities, without any visibility into the model's internal parameters, gradients, or architecture.
Gradient-Based Sensitivity Analysis
Terms related to interpreting neural networks by computing the gradient of the output with respect to the input features. Target: Deep learning researchers analyzing model focus.
Saliency Map
A heatmap visualization that highlights the input features (pixels, words) that most influence a neural network's prediction, typically computed using the gradient of the output with respect to the input.
Integrated Gradients
An axiomatic attribution method that computes feature importance by accumulating the gradients of the model's output along a straight-line path from a baseline input to the actual input.
SmoothGrad
A technique that sharpens gradient-based saliency maps by averaging the gradients computed from multiple noisy copies of the same input, reducing visual noise.
Grad-CAM
A technique that produces coarse localization maps by using the gradients of a target concept flowing into the final convolutional layer of a CNN to weight the activation maps.
Guided Backpropagation
A modified backpropagation algorithm that restricts gradient flow to only positive gradients and positive activations, resulting in sharper feature visualizations by suppressing negative contributions.
DeepLIFT
An attribution method that explains a prediction by comparing the activation of each neuron to a reference activation and assigning contribution scores based on the difference.
Layer-wise Relevance Propagation
A decomposition technique that redistributes a model's prediction score backwards through the network layers using a conservation property, assigning a relevance score to each input feature.
Score-CAM
A class activation mapping method that eliminates the need for gradients by using the increase in the model's confidence score as the weight for each activation map.
Expected Gradients
An attribution method that computes feature importance by averaging the gradients of the model's output over a distribution of background samples, unifying Integrated Gradients and SHAP.
Gradient SHAP
An approximation of SHAP values that uses the expected gradients of a model with respect to a background distribution, adding Gaussian noise to the inputs for computational efficiency.
Axiomatic Attribution
A framework for evaluating feature attribution methods based on their adherence to mathematical axioms like completeness, sensitivity-n, and implementation invariance.
Completeness Axiom
The principle that the sum of the feature attributions for a given input must equal the difference between the model's output for that input and a baseline reference output.
Implementation Invariance
The axiom stating that two functionally equivalent neural networks, regardless of their internal architecture, should produce identical feature attributions for the same input.
Sensitivity-n
An axiom requiring that if a model's output is mathematically independent of a feature, that feature must receive an attribution of zero.
Gradient × Input
A simple attribution method that multiplies the gradient of the output with respect to the input by the input value itself, providing a first-order Taylor approximation of feature importance.
FullGrad
An attribution method that satisfies the completeness axiom by aggregating the input-gradient with the gradients of all intermediate network biases, providing a full decomposition of the output.
VarGrad
A technique that measures the uncertainty of a gradient-based saliency map by computing the variance of the gradients obtained from multiple noisy versions of the same input.
Integrated Hessians
An extension of Integrated Gradients that uses the second-order Hessian matrix to capture feature interactions and curvature, providing a more complete attribution for non-linear models.
Path Methods
A class of attribution techniques, including Integrated Gradients, that define feature importance by integrating gradients along a specified path in the input space from a baseline to the target input.
Deep Taylor Decomposition
A method for explaining neural network predictions by performing a Taylor expansion of the model's output at a root point, redistributing the function value to input features via propagation rules.
Extremal Perturbation
A technique that finds the smallest smooth mask over an input image that maximally preserves a model's prediction, identifying the most salient region for that class.
Shattered Gradient
A phenomenon where the gradient of a deep network with respect to its input resembles white noise, providing no visually coherent saliency map due to the network's highly non-linear loss surface.
Gradient Saturation
A problem in deep networks where the gradient of the output with respect to the input becomes near-zero for features that strongly activate the correct class, causing them to appear unimportant.
Attribution Flow
A theoretical framework that views the propagation of relevance scores through a network as a continuous flow, satisfying conservation laws and providing a unified view of decomposition methods.
Neuron Conductance
An extension of Integrated Gradients that measures the importance of a specific hidden neuron by computing the flow of the gradient through that neuron along the integration path.
Activation Maximization
A visualization technique that synthesizes an input pattern that maximally activates a specific neuron, filter, or class output by performing gradient ascent in the input space.
Feature Visualization
A set of techniques, including activation maximization, used to generate synthetic inputs that reveal what visual patterns or concepts a specific neuron or layer in a neural network has learned to detect.
Attribution Prior
A regularization term added to a model's training objective that encodes a desired property of the feature attributions, such as smoothness or sparsity, to make the model inherently more interpretable.
Infidelity Measure
A metric that quantifies the faithfulness of an attribution method by measuring the error between the model's output change and the dot product of the attribution with a significant input perturbation.
Local Lipschitz Estimate
A robustness metric for attributions that measures the maximum change in the explanation under small, adversarial input perturbations, quantifying the stability of the saliency map.
Counterfactual Explanations
Terms related to generating minimal changes to input data that would alter a model's prediction to a desired outcome. Target: CTOs designing recourse and fairness systems.
Counterfactual Explanation
A causal explanation that describes the minimal change to an input instance required to alter a model's prediction to a predefined, desired output.
Algorithmic Recourse
The process of providing an end-user with a set of actionable changes they can make to their input features to receive a favorable model decision.
Actionable Recourse
A subset of algorithmic recourse that constrains recommended changes to only those features an individual can realistically control or modify in the real world.
Recourse Feasibility
The degree to which a counterfactual recommendation respects real-world constraints, such as immutable features, causal relationships, and user capabilities.
Action Set
A formal specification of the permissible modifications a user can make to each feature, defining the boundary between actionable and non-actionable changes.
Immutable Feature
A protected input attribute, such as age or place of birth, that cannot be changed and must be held constant when generating counterfactual explanations.
Counterfactual Fairness
A causal definition of individual fairness stating that a decision is fair if it is the same in the actual world and a counterfactual world where a sensitive attribute was changed.
Structural Causal Model (SCM)
A formal framework representing variables and their causal dependencies through structural equations, enabling the computation of interventional and counterfactual queries.
Do-Calculus
A set of three inference rules developed by Judea Pearl for transforming expressions involving interventions into estimable statistical quantities from observational data.
Causal Graph
A directed acyclic graph (DAG) where nodes represent variables and edges represent direct causal relationships, used to encode assumptions for counterfactual inference.
Sparse Counterfactual
A counterfactual instance that alters the prediction by changing only a small number of features, making the explanation easier for a human to understand and act upon.
Diverse Counterfactuals
A set of multiple, distinct counterfactual instances that provide a user with alternative paths to achieve a desired outcome, avoiding a single, potentially infeasible recommendation.
Plausible Counterfactual
A counterfactual instance that lies within the high-density region of the training data distribution, ensuring the explanation is realistic and not an adversarial artifact.
Counterfactual Proximity
A metric quantifying the distance between the original input instance and the generated counterfactual, often measured using L1 or L2 norms to enforce minimal change.
Counterfactual Validity
A binary evaluation metric that verifies whether a generated counterfactual instance successfully flips the model's prediction to the target outcome.
Growing Spheres
A model-agnostic counterfactual generation algorithm that searches for the closest counterfactual by expanding a hypersphere around the query instance until the decision boundary is crossed.
Contrastive Explanation
An explanation that answers the question 'Why P and not Q?' by highlighting the minimal differences between the factual outcome and a presented counterfactual alternative.
Decision Boundary
The hypersurface in the feature space that separates different prediction classes, which counterfactual algorithms must cross to flip a model's output.
Recourse Robustness
The property that a counterfactual recommendation remains valid and flips the prediction even after the underlying model is retrained or slightly updated.
Feasibility Constraint
A hard rule encoded into a counterfactual generation algorithm that prevents the modification of immutable features or enforces causal monotonicity.
Mahalanobis Distance
A distance metric that accounts for feature correlations and variances in the data distribution, used to generate more plausible counterfactuals than standard Euclidean distance.
Counterfactual Inference
The computational process of estimating the outcome of an intervention in a hypothetical scenario that is contrary to what actually occurred, using a structural causal model.
Individual Fairness
The principle that similar individuals should receive similar predictions, often operationalized through counterfactual fairness by comparing an individual to their counterfactual self.
Counterfactual Evaluation
The systematic assessment of generated counterfactuals using metrics like validity, proximity, sparsity, diversity, and plausibility to benchmark different generation algorithms.
Gradient-Based Counterfactual
A model-specific generation technique that uses the gradient of the model's loss function with respect to the input to iteratively perturb features toward a target prediction.
Mechanistic Interpretability
Terms related to reverse-engineering the internal computations and learned algorithms within a neural network's weights. Target: AI safety researchers and advanced engineering leads.
Mechanistic Interpretability
The field of reverse-engineering a neural network's learned algorithms and internal computations from its weights and activations into human-understandable components.
Circuits
Sparse, interpretable subgraphs of a neural network consisting of connected attention heads and MLP neurons that implement a specific, human-understandable algorithm.
Superposition
A hypothesized phenomenon where a neural network represents more independent features than it has dimensions in a given layer, compressing sparse features into a lower-dimensional space.
Sparse Autoencoder
An unsupervised architecture trained to decompose a model's dense, polysemantic activations into a sparse set of interpretable, monosemantic features.
Monosemanticity
The property of a single neuron or feature direction in a neural network corresponding to exactly one natural, human-interpretable concept.
Polysemanticity
The observed phenomenon where a single neuron or feature direction in a neural network responds to multiple, seemingly unrelated input concepts.
Activation Patching
A causal intervention technique that replaces a model's internal activation at a specific layer and position with a value from a corrupted or alternative forward pass to isolate its function.
Path Patching
A causal method that isolates the direct effect of a specific computational path between two model components by patching activations along that path while freezing all other paths.
Causal Scrubbing
A systematic evaluation methodology that tests a hypothesized circuit by replacing all activations outside the circuit with corrupted values and verifying the model's performance is preserved.
Logit Lens
A technique that applies the unembedding matrix directly to intermediate residual stream activations to decode the model's next-token predictions at each layer.
Tuned Lens
An improvement on the logit lens that learns an affine transformation for each layer's residual stream to produce more accurate decoded next-token predictions.
Induction Heads
A specific attention head circuit that performs in-context copying by attending to a previous token that followed a similar current token, enabling sequence completion.
Residual Stream
The central information highway in a transformer architecture where each layer reads from and writes its output back to a shared, accumulating state vector.
Feature Visualization
An optimization-based technique that synthesizes input examples that maximally activate a specific neuron, channel, or feature direction to visualize what it detects.
Dictionary Learning
A method for decomposing a model's activations into a sparse linear combination of learned basis vectors, each representing a distinct, interpretable feature.
Linear Representation Hypothesis
The conjecture that high-level concepts are encoded as linear directions in the representation space of a neural network's activation vectors.
Probing Classifier
A simple supervised model trained on a network's internal activations to predict a specific linguistic or world property, testing if that information is linearly encoded.
Knowledge Neurons
Specific MLP neurons identified through activation analysis that store factual associations and are causally responsible for a model's expression of that knowledge.
Grokking
A phenomenon where a model abruptly transitions from memorizing a training dataset to perfectly generalizing a clean, general solution after a prolonged period of overfitting.
Automated Circuit Discovery
Algorithmic methods that automatically identify the minimal subgraph of a neural network responsible for a specific behavior without manual human inspection.
Direct Logit Attribution
A linear decomposition technique that attributes the final logit output for a token directly to the individual contributions of each attention head and MLP layer.
Activation Engineering
The practice of directly modifying a model's internal activations during a forward pass by adding steering vectors to control its behavior without prompt engineering.
Function Vector
A compact representation derived from in-context examples that can be injected into a model's activations to trigger a specific input-output task behavior without the original prompt.
Cross-Coder
A variant of a sparse autoencoder trained on the activations of two different models simultaneously to identify shared and distinct features between them.
Transcoder
A model component trained to decompose the output of a specific MLP layer or attention block into a sparse set of interpretable features, replacing the original component for analysis.
Disentanglement
The objective of learning a representation where individual latent dimensions correspond to independent, interpretable generative factors of the data.
Reverse Engineering
The manual process of hypothesizing, validating, and refining a human-understandable algorithm that explains the precise computational function of a neural network circuit.
Zero Ablation
A causal intervention that sets a neuron's or head's output to zero to measure its importance; contrasted with mean ablation which replaces it with an average value.
Memorization Localization
The process of identifying the specific weights, neurons, or layers within a model that are responsible for storing exact training data rather than generalizable patterns.
Locating and Editing Factual Associations
A causal tracing methodology to identify the specific MLP layers storing a fact and then surgically modifying those weights to update the stored association.
Transformer Mechanistic Analysis
Terms related to decoding the specific functional roles of attention heads and MLP layers in transformer architectures. Target: NLP engineers and model architects.
Mechanistic Interpretability
The field of reverse-engineering the internal algorithms and learned computations encoded within a neural network's weights and activations into human-understandable components.
Residual Stream
The primary information highway in a transformer model where each layer reads from and writes to a shared accumulating state, enabling the model to bypass layers and preserve information flow.
Attention Head
A specialized computational subunit within a transformer layer that computes a weighted sum of past token representations, allowing the model to move information between token positions.
Multi-Layer Perceptron (MLP) Layer
A position-wise feed-forward network within a transformer block that processes each token's representation independently to store and retrieve factual knowledge through non-linear transformations.
Induction Head
A specific type of attention head discovered through mechanistic analysis that performs in-context copying by attending to the token immediately following a previous occurrence of the current token.
Sparse Autoencoder (SAE)
An unsupervised technique used to decompose a model's dense, polysemantic internal activations into a sparse set of interpretable, monosemantic features using a learned overcomplete basis.
Superposition Hypothesis
The theory that neural networks represent more independent features than they have dimensions in a given layer by encoding them in almost-orthogonal directions within the activation space.
Activation Patching
A causal intervention technique that replaces a model's internal activation at a specific layer and token position with a cached activation from a different forward pass to isolate functional circuits.
Circuit Analysis
The process of identifying and validating the minimal subgraph of a neural network's computational graph that is necessary and sufficient to perform a specific behavior or task.
Logit Lens
A direct probing method that applies the unembedding matrix to intermediate residual stream states to interpret the model's next-token predictions before the final layer norm is applied.
Causal Scrubbing
A formal hypothesis-testing framework that systematically replaces activations to verify if a proposed circuit explains a model's behavior by checking if the circuit's components are faithful under resampling.
Rotary Position Embedding (RoPE)
A position encoding method that applies a rotation matrix to query and key vectors based on their absolute positions, ensuring the attention dot product depends only on the relative distance between tokens.
QK Circuit
The computational pathway formed by the query and key projection matrices that determines which previous tokens an attention head attends to by computing attention scores.
OV Circuit
The computational pathway formed by the output and value projection matrices that determines what information an attention head copies from the source token to the destination token.
Knowledge Neuron
A specific neuron within an MLP layer identified through causal tracing that is primarily responsible for expressing a particular piece of factual knowledge stored in the model.
Polysemantic Neuron
A single neuron that activates in response to multiple unrelated input features or concepts, making it difficult to assign a single, human-interpretable label to its function.
Dictionary Learning
A decomposition method that learns an overcomplete basis of vectors to represent activations as a sparse linear combination of interpretable features, often implemented via sparse autoencoders.
Causal Mediation Analysis
A statistical framework adapted for neural networks to quantify the contribution of a specific intermediate variable or neuron to a model's output by measuring the indirect effect through that mediator.
Function Vector
A compact representation derived from in-context learning examples that, when added to the model's activations, can steer the model to perform a specific task without the original prompt.
Attention Sink
A phenomenon where the first few tokens of a sequence receive disproportionately high attention scores from all heads, acting as a resting place for attention mass that is not allocated to semantically relevant tokens.
Grouped Query Attention (GQA)
An attention mechanism that uses a smaller number of key-value head groups than query heads, interpolating between multi-head and multi-query attention to reduce the KV cache size while preserving quality.
KV Cache
A memory buffer that stores the pre-computed key and value tensors from previous tokens during autoregressive generation to avoid redundant computation and accelerate inference.
Representation Engineering
A top-down approach to controlling model behavior by directly manipulating high-level representations in the residual stream using steering vectors rather than modifying individual weights.
Model Editing
The task of surgically updating a specific piece of factual knowledge stored within a pre-trained model's weights without affecting its performance on unrelated inputs or capabilities.
Rank-One Model Editing (ROME)
A precise model editing technique that treats an MLP layer as a linear associative memory and inserts a new fact by performing a rank-one update to the weight matrix of a specific layer.
Causal Tracing
A method for locating where factual knowledge is stored by running the model with corrupted input embeddings and then restoring clean hidden states one layer at a time to measure the causal effect on the output.
Gated Sparse Autoencoder
An SAE architecture that separates the selection of which features are active from the estimation of their magnitudes using a gating mechanism, improving the separation of feature activation and strength.
Cross-Layer Transcoding
A technique for interpreting features that span multiple layers by training a transcoder to translate the sparse feature representation of one layer into the feature representation of a subsequent layer.
Self-Repair
A phenomenon observed during ablation studies where the network dynamically compensates for the removal of a component by adjusting the activations of other, redundant components in later layers.
Singular Value Decomposition (SVD)
A matrix factorization technique used in mechanistic analysis to decompose weight matrices into interpretable rank-one components, revealing the principal directions of information transformation in OV and QK circuits.
Concept Activation Vectors
Terms related to testing how sensitive a model's predictions are to high-level, human-understandable concepts. Target: Researchers validating model alignment with domain knowledge.
Concept Activation Vector (CAV)
A vector in a neural network's activation space that represents a high-level, human-understandable concept, derived by training a linear classifier to distinguish between examples of the concept and random counterexamples.
Testing with CAVs (TCAV)
A technique that quantifies a model's sensitivity to a user-defined concept by measuring the directional derivative of a class prediction towards the Concept Activation Vector.
Sensitivity
In the context of concept-based explanations, a measure of how much a model's prediction for a class changes in response to perturbations along the direction of a concept vector.
Directional Derivative
The rate of change of a model's prediction score with respect to an infinitesimal shift in the input's activation along a specific concept vector direction.
Statistical Significance Testing
A procedure within TCAV that uses a two-sided t-test to determine if the sensitivity scores for a concept are significantly different from those obtained using random vectors, ensuring the concept is not an artifact.
Activation Space
The high-dimensional vector space formed by the outputs of a specific layer within a neural network, where individual directions can encode semantically meaningful concepts.
Concept Bottleneck Model (CBM)
An inherently interpretable architecture that first predicts a set of predefined human-understandable concepts from the input and then uses only those concept scores to make the final prediction.
Concept Whitening
A module that replaces a standard batch normalization layer, aligning the latent space axes with predefined concepts to produce a disentangled and interpretable representation.
Concept Relevance Propagation (CRP)
An extension of Layer-wise Relevance Propagation that decomposes a model's decision not just by input features but also through higher-level latent concepts, tracing their relevance flow.
ConceptSHAP
A method that applies Shapley values to quantify the importance of individual concepts for a model's prediction, providing a game-theoretic attribution for concept-based explanations.
Concept Completeness Score
A metric that evaluates how sufficient a set of discovered or defined concepts is for explaining a model's full behavior on a given task, measuring the fidelity of the concept-based explanation.
Concept Purity
A measure of how well the representations of a single concept are clustered together and separated from other concepts in the activation space, indicating the concept's internal consistency.
Concept Separability
The degree to which a linear or non-linear classifier can distinguish between the activation patterns of two different concepts, reflecting their distinct encoding in the network.
Concept Attribution
The process of assigning a relevance or importance score to a high-level concept for a specific model prediction, analogous to feature attribution but operating on semantic abstractions.
Concept Localization
The technique of identifying the specific spatial regions, network layers, or individual neurons that are most responsible for encoding a particular concept.
Concept Erasure
A technique for removing a specific, often sensitive, concept's information from a model's latent representation by projecting the activations onto a subspace orthogonal to the concept vector.
Concept Subspace Projection
The mathematical operation of decomposing an activation vector into components that align with a concept vector and components that are orthogonal to it, used for analysis or manipulation.
Concept Sensitivity Map
A visualization that highlights the regions of an input or the nodes in a computational graph that are most sensitive to perturbations along a specific concept direction.
Concept Influence
A measure of the causal effect that manipulating a concept's activation value has on the model's final output, often estimated through intervention.
Concept-Based Explanation
A model explanation that uses high-level, human-interpretable abstractions as the fundamental units of reasoning, rather than low-level input features like pixels or tokens.
Concept Importance
A global or local score that ranks the significance of different concepts for a model's decision-making process, often derived from concept attribution methods.
Concept Discovery
The automated process of identifying meaningful and coherent directions in a model's activation space that correspond to human-interpretable concepts without pre-defining them.
Automatic Concept Extraction (ACE)
An algorithm that automatically discovers visual concepts by clustering image patches that activate similar spatial patterns in a network and then testing their significance with TCAV.
Concept Embedding
A dense vector representation of a semantic concept, learned either from a concept's exemplar data or derived from a model's activation space, that captures its relational properties.
Concept Prototype
A representative example or a learned embedding in the input space that serves as the canonical instance of a concept, used in prototype-based interpretable models.
Concept Bank
A curated collection of pre-defined, labeled concept vectors and their associated probe datasets, used as a standard library for auditing and interpreting various models.
Concept Regularization
A training technique that adds a penalty term to the loss function to encourage a model's latent space to align with predefined concepts, improving interpretability.
Concept Intervention
The act of directly modifying a model's internal activations during inference to increase or decrease the presence of a concept, used to causally test its influence on the output.
Concept Vector Arithmetic
Performing linear operations, such as addition and subtraction, on concept vectors in the activation space to create new, semantically meaningful composite concepts.
Concept Selectivity
A measure of whether a specific neuron, channel, or direction in the activation space responds exclusively to a single concept and not to others.
Explainable Graph Neural Networks
Terms related to identifying critical nodes, edges, and subgraphs that drive predictions in graph-structured data. Target: Engineers working on molecular or social network models.
GNNExplainer
A model-agnostic framework that identifies a compact subgraph structure and a small subset of node features most influential for a Graph Neural Network's prediction.
SubgraphX
An explainability method that uses Monte Carlo Tree Search to efficiently explore and identify the most critical subgraph structures for GNN predictions.
Integrated Gradients on Graphs
A gradient-based attribution method adapted for graph data that computes the importance of edges or nodes by accumulating gradients along a path from a baseline graph to the input.
Grad-CAM on Graphs
An adaptation of the Gradient-weighted Class Activation Mapping technique that visualizes the node or edge regions most influential to a GNN's decision by using gradient signals flowing into the final convolutional layers.
Attention Flow
A technique for interpreting Graph Attention Networks by analyzing the propagation and aggregation of attention weights across layers to trace how information flows between nodes.
Counterfactual Subgraphs
The minimal structural perturbations to a graph, such as removing specific edges or nodes, that would alter a GNN's prediction to a different outcome.
GraphLIME
A local interpretable model explanation method for GNNs that approximates a node's prediction by training a simple, interpretable model on its local neighborhood features.
Shapley Values on Graphs
A game-theoretic approach that assigns a fair importance score to each node or edge by computing its marginal contribution to the GNN's prediction across all possible coalitions of graph components.
GNN-LRP
A Layer-wise Relevance Propagation technique adapted for Graph Neural Networks that decomposes the output prediction backwards through the network to assign relevance scores to input nodes and edges.
GraphSVX
A post-hoc explainer that decomposes GNN predictions using Shapley values on a coalition of nodes and features, providing both structural and feature-level explanations.
CF-GNNExplainer
A counterfactual explanation generator for GNNs that identifies the minimal edge deletions required to change a model's prediction, focusing on actionable recourse.
Graph Pruning
The process of systematically removing nodes or edges deemed irrelevant by an explainability metric to create a sparser, more interpretable subgraph that preserves the original prediction.
Saliency Maps on Graphs
A visualization technique that assigns an importance score to each node or edge based on the gradient of the target prediction with respect to the input adjacency matrix or feature matrix.
Perturbation Analysis
A fidelity assessment method that measures the change in a GNN's prediction after masking or altering the most important nodes or edges identified by an explainer.
Mutual Information Maximization
An explanation objective function that selects a subgraph by maximizing the mutual information between the subgraph's representation and the GNN's final prediction.
Graph Information Bottleneck
A principle for learning explainable GNNs by compressing the input graph into a minimal subgraph that retains maximal mutual information about the label, discarding irrelevant structural noise.
Disentangled Graph Representations
Learning node embeddings where each dimension corresponds to an independent, interpretable generative factor of the graph data, enabling fine-grained explanation of latent structures.
Graph Rationalization
A self-explainable GNN framework where a generator module extracts a concise, causal subgraph (the rationale) and a predictor module makes a decision based solely on that rationale.
GSAT
Graph Stochastic Attention, a method that injects stochasticity into the attention mechanism to automatically select a minimal, label-relevant subgraph for explanation and prediction.
GraphMask
A post-hoc explanation method that learns a sparse mask over the messages passed between nodes in each GNN layer to identify which connections are irrelevant to a prediction.
DiffPool
A differentiable graph pooling module that learns hierarchical cluster assignments for nodes, enabling the interpretation of GNN decisions at multiple levels of graph granularity.
Faithfulness Metric
A quantitative evaluation score that measures how accurately an explanation subgraph reflects the true reasoning process of the GNN, typically assessed by the drop in performance when the explanation is removed.
Fidelity Metric
An evaluation metric that measures how well an explanation mimics the original model's behavior, often calculated as the accuracy of the original model on the extracted explanatory subgraph.
Nettack
A targeted adversarial attack on GNNs that introduces small, imperceptible perturbations to the graph structure and node features to change a specific node's prediction.
Causal Graph Discovery
The process of inferring cause-and-effect relationships between variables in graph-structured data, moving beyond correlation to identify the true generative mechanisms for robust explanations.
Structural Causal Models
A formal framework representing causal relationships in a graph as a set of structural equations, used to perform intervention analysis and generate counterfactual explanations for GNNs.
Dynamic Graph Explanations
Techniques for attributing importance to temporal edges and nodes in time-evolving graphs, explaining how a GNN's prediction is influenced by the sequence of structural changes.
Meta-Path Explanations
An interpretability method for heterogeneous graphs that identifies the most relevant sequences of node types and relationships (meta-paths) contributing to a prediction.
Knowledge Graph Explanations
Methods for explaining link prediction or node classification in knowledge graphs by identifying the most salient reasoning paths and logical rules from the graph's triples.
Rule Extraction from GNNs
The process of distilling a trained GNN into a set of human-readable, symbolic logical rules that approximate its decision-making process on graph data.
Uncertainty Quantification
Terms related to measuring and decomposing a model's confidence in its predictions to distinguish between data noise and model ignorance. Target: CTOs deploying models in high-stakes environments.
Epistemic Uncertainty
Model uncertainty arising from a lack of knowledge or data, which can theoretically be reduced with more training samples or a better model architecture.
Aleatoric Uncertainty
Inherent and irreducible noise in the data generation process, such as measurement error or inherent stochasticity, which cannot be reduced by collecting more data.
Confidence Calibration
The process of aligning a model's predicted probability of correctness with the actual empirical frequency of being correct.
Expected Calibration Error (ECE)
A scalar summary statistic that measures the discrepancy between a model's confidence and its observed accuracy by partitioning predictions into bins.
Brier Score
A strictly proper scoring rule that measures the mean squared difference between the predicted probability and the actual binary outcome.
Temperature Scaling
A post-hoc calibration method that divides the logits by a single scalar parameter to soften the softmax output without changing the model's accuracy.
Monte Carlo Dropout
An approximate Bayesian inference technique that uses dropout at test time to generate multiple stochastic forward passes, producing a predictive distribution.
Deep Ensembles
A method for quantifying predictive uncertainty by training multiple neural networks with different random initializations and averaging their outputs.
Bayesian Neural Network (BNN)
A neural network where the weights are treated as probability distributions rather than point estimates, enabling principled uncertainty quantification.
Variational Inference
An optimization-based technique that approximates complex posterior distributions by finding the closest distribution within a simpler, tractable family.
Prior Distribution
The initial belief about a model parameter's value before observing any data, encoded as a probability distribution.
Posterior Distribution
The updated belief about a model parameter's value after combining the prior distribution with observed data via Bayes' theorem.
Evidence Lower Bound (ELBO)
A variational objective function that maximizes a lower bound on the log marginal likelihood, balancing data fit against the complexity of the approximate posterior.
Gaussian Process (GP)
A non-parametric Bayesian model defined by a kernel function that specifies a distribution over functions, providing calibrated uncertainty estimates.
Conformal Prediction
A distribution-free framework that wraps any model to produce prediction sets with a rigorous, finite-sample coverage guarantee.
Prediction Set
A set of possible labels output by a conformal predictor that is guaranteed to contain the true label with a user-specified probability.
Quantile Regression
A statistical technique that estimates the conditional quantiles of a response variable, directly modeling the lower and upper bounds of a prediction interval.
Prediction Interval
An interval estimate for a future observation that quantifies the range within which the value will fall with a specified probability.
Credible Interval
A Bayesian posterior interval that directly states the probability that the true parameter value lies within the interval, given the observed data.
Confidence Interval
A frequentist interval estimate for a population parameter that would contain the true value in a specified proportion of repeated samples.
Out-of-Distribution Detection
The task of identifying test inputs that are semantically or statistically different from the model's training data to prevent overconfident predictions.
Mahalanobis Distance
A distance metric that measures the number of standard deviations a point is from the mean of a distribution, accounting for feature correlations.
Evidential Deep Learning
A method that places a Dirichlet distribution directly over class probabilities, allowing a single forward pass to output both a prediction and its associated uncertainty.
Dirichlet Distribution
A multivariate probability distribution over the simplex of categorical probabilities, used as a conjugate prior in evidential deep learning.
Ensemble Disagreement
A measure of epistemic uncertainty computed by evaluating the variance or mutual information among the predictions of an ensemble of models.
Active Learning
A training paradigm where the learning algorithm strategically queries an oracle to label the most informative data points, often using uncertainty estimates.
Distributional Shift
A change in the joint probability distribution of inputs and labels between the training and deployment environments, degrading model performance.
Covariate Shift
A specific type of distributional shift where the input distribution changes but the conditional distribution of the label given the input remains constant.
Concept Drift
A change in the underlying statistical relationship between the input features and the target variable over time.
Sensitivity Analysis
The study of how the uncertainty in a model's output can be apportioned to different sources of uncertainty in its inputs.
Conformal Prediction
Terms related to a distribution-free framework for generating statistically rigorous prediction sets with guaranteed coverage. Target: Engineers requiring formal uncertainty guarantees.
Conformal Prediction
A distribution-free, model-agnostic framework that produces prediction sets with a rigorous, finite-sample marginal coverage guarantee, ensuring that the true label is included with a user-specified probability.
Nonconformity Measure
A heuristic function that quantifies how unusual a specific input-label pair appears relative to a training dataset, serving as the foundational engine for computing prediction sets in conformal prediction.
Prediction Set
A set-valued output from a conformal predictor that contains the true label with a predefined confidence level, providing a robust alternative to single-point predictions in high-stakes decision-making.
Marginal Coverage Guarantee
The core statistical promise of conformal prediction that the probability of the true label falling within the prediction set is at least the nominal confidence level, averaged over the randomness in both the calibration and test data.
Exchangeability
The fundamental assumption in standard conformal prediction that the joint distribution of the calibration and test data points is invariant to any permutation, a weaker condition than the independent and identically distributed (IID) assumption.
Split Conformal Prediction
An efficient variant of conformal prediction that partitions the training data into a proper training set for model fitting and a disjoint calibration set for computing nonconformity scores, avoiding the computational cost of full retraining.
Calibration Set
A held-out dataset, distinct from the training data, used exclusively to compute the empirical quantile of nonconformity scores, which directly determines the size of the final prediction sets.
Inductive Conformal Prediction
A class of conformal predictors that pre-computes a fixed prediction rule from a proper training set before evaluating nonconformity on a calibration set, enabling fast, real-time inference on new test points.
Conformalized Quantile Regression
A technique that wraps a standard quantile regression model with a conformal calibration step to correct the predicted quantile intervals, yielding finite-sample valid prediction intervals without assuming a specific error distribution.
Adaptive Conformal Inference
A method for online conformal prediction that dynamically adjusts the quantile threshold over time to maintain long-run coverage guarantees even when the data distribution shifts, without requiring knowledge of the shift mechanism.
Conditional Coverage
A stronger, more granular validity objective in conformal prediction that seeks to guarantee coverage for specific subgroups or feature values, which is statistically impossible to achieve exactly without additional assumptions but can be approximated.
Conformal Risk Control
An extension of conformal prediction that provides statistical guarantees for controlling any monotone loss function, enabling the creation of prediction sets with bounded false negative rates or other task-specific error metrics.
Jackknife+ Prediction
A leave-one-out cross-validation-based conformal method that provides a computationally efficient and theoretically valid alternative to full jackknife prediction, offering tighter prediction sets than split conformal methods.
Conformal Anomaly Detection
The application of conformal p-values to the unsupervised problem of outlier detection, flagging test points as anomalous if they are significantly nonconforming relative to a calibration set of normal data.
Conformal Predictive Distributions
A framework that extends conformal prediction from set-valued outputs to full cumulative distribution functions, providing a complete probabilistic description of the uncertainty for a test label.
Weighted Conformal Prediction
A variant of conformal prediction that applies importance weights to the calibration samples to maintain valid coverage guarantees under covariate shift, where the distribution of input features changes between training and test time.
Mondrian Conformal Prediction
A technique that applies conformal calibration independently within pre-defined data categories or classes to achieve label-conditional coverage guarantees, ensuring validity for each distinct group.
Conformal Time Series
The adaptation of conformal prediction to sequential data where the exchangeability assumption is violated, often using a sliding window of recent observations as the calibration set to handle temporal dependencies.
Conformal e-values
A modern formulation of conformal inference that produces e-values—nonnegative random variables with an expected value of at most one under the null hypothesis—for flexible sequential testing and anytime-valid uncertainty quantification.
Conformalized Bayesian Neural Networks
A hybrid uncertainty quantification approach that applies a conformal calibration wrapper to the posterior predictive distributions of a Bayesian neural network to correct for model misspecification and achieve finite-sample coverage.
Conformalized Survival Analysis
The integration of conformal prediction with survival models to produce lower prediction bounds for survival times with guaranteed coverage, accounting for the unique challenge of right-censored data.
Conformal Test-Time Adaptation
A method that combines conformal prediction with test-time training to maintain valid prediction sets when a model must adapt to a new, unlabeled target distribution online without source data.
Conformal OOD Detection
A statistical framework for out-of-distribution detection that uses conformal p-values to test whether a new input belongs to the training distribution, providing a rigorous false positive rate control mechanism.
Conformalized Deep Ensembles
A technique that applies a conformal calibration step to the aggregated predictions of a deep ensemble, transforming the ensemble's empirical variance into a statistically rigorous prediction set.
Conformalized Causal Inference
The application of conformal prediction to estimate valid confidence intervals for individual treatment effects, providing distribution-free uncertainty quantification for counterfactual predictions in causal models.
Conformalized LLM
The integration of conformal prediction with large language models to generate output sets with statistical guarantees, such as guaranteeing the inclusion of a correct translation or a factual answer within a generated set of responses.
Conformalized Reinforcement Learning
A framework that uses conformal prediction to quantify the uncertainty of a learned policy's value estimates or to construct safe action sets with guaranteed constraint satisfaction in offline RL settings.
Conformalized Gaussian Processes
A method that calibrates the credible intervals of a Gaussian process using a held-out conformal calibration set to correct for potential kernel misspecification and achieve exact marginal coverage.
Conformal Multivariate Prediction
An extension of conformal prediction to multi-target regression or structured output spaces, producing valid prediction regions for vector-valued labels using multivariate nonconformity measures.
Conformalized Autoencoders
The use of a conformal calibration step on the reconstruction error of an autoencoder to define a statistically valid threshold for anomaly detection, controlling the false positive rate in unsupervised settings.
Explainable Reinforcement Learning
Terms related to interpreting the policies and reward functions of agents that learn through interaction with an environment. Target: Engineers building autonomous decision systems.
Policy Visualization
A technique for rendering an agent's learned policy as a visual heatmap or graph to illustrate which actions it will take in specific states.
Reward Decomposition
The process of breaking down a scalar reward signal into constituent sub-rewards to explain which objectives are driving an agent's behavior.
Saliency Map
A heatmap that highlights the regions of an input state, such as pixels in a game frame, that most strongly influence a reinforcement learning agent's decision.
Q-Value Decomposition
A method for factoring an action-value function into additive components to attribute credit to specific sub-goals or entities within a state.
Policy Distillation
The process of compressing a complex teacher policy into a simpler, more interpretable student model, such as a decision tree, for explanation purposes.
Causal Policy Analysis
The application of causal inference tools, like intervention analysis, to determine whether a policy relies on spurious correlations or true causal relationships.
Feature Attribution
The assignment of an importance score to each dimension of the state space to explain which features are critical for a specific action selection.
Markov Decision Process (MDP)
The foundational mathematical framework for reinforcement learning, defined by states, actions, a transition function, and a reward function, used to model sequential decision-making.
Inverse Reinforcement Learning (IRL)
A technique for inferring the underlying reward function that an expert is implicitly optimizing, providing an explanation for demonstrated behavior.
Shielded Reinforcement Learning
A safety framework where a formal 'shield' monitor overrides an agent's unsafe actions with provably safe alternatives, making the system's safety constraints explicit.
Epistemic Uncertainty
The reducible uncertainty in a model's predictions caused by a lack of knowledge or data, which can be explained by identifying unexplored regions of the state space.
Distributional Reinforcement Learning
A class of algorithms that model the full probability distribution of returns, rather than just the expected value, to explain the risk and stochasticity of a policy.
Contrastive Explanations
An explanation format that answers 'Why action A instead of action B?' by highlighting the minimal state differences that caused the policy to diverge.
Shapley Value
A game-theoretic solution concept adapted to reinforcement learning to fairly distribute credit for a joint action among cooperating agents in a multi-agent system.
Integrated Gradients
A model-agnostic attribution method that computes the path integral of gradients from a baseline state to the actual input to identify influential features.
Decision Transformer
An architecture that reframes reinforcement learning as a sequence modeling problem, using a causal transformer to generate actions, which can be analyzed via attention rollout.
Counterfactual Policy Evaluation
A family of off-policy evaluation techniques that estimate how a new policy would perform using historical data, explaining potential outcomes without deployment.
World Model
An internal generative model of the environment learned by an agent, which can be probed and visualized to understand the agent's beliefs about state transitions.
Disentangled Representation
A latent state encoding where individual dimensions correspond to independent, meaningful generative factors, making the agent's internal state representation interpretable.
Grad-CAM
A gradient-based localization technique that produces a coarse saliency map by using the gradients of a target Q-value flowing into the final convolutional layer of a deep Q-network.
Feature Ablation
A causal interpretability method that systematically removes or occludes input features to measure the resulting change in policy output and determine feature necessity.
TCAV (Testing with Concept Activation Vectors)
A technique that quantifies the sensitivity of a trained policy to user-defined high-level concepts, providing explanations in human-friendly terms.
Rationale Generation
The task of training an agent to output a natural language justification alongside its action, explaining the 'why' behind a decision in a human-readable format.
Value Decomposition Network (VDN)
A multi-agent reinforcement learning architecture that additively decomposes the joint Q-value into per-agent Q-values to explain individual agent contributions.
Multi-Agent Credit Assignment
The challenge of determining which individual agent's action contributed to a shared team reward, solved by explainable methods like difference rewards or Shapley Q-values.
Decision Tree Extraction
A technique for distilling a neural network policy into a structurally interpretable decision tree that mimics the original policy's input-output mapping.
Sparse Policy
A policy regularized via L1 penalty or discrete masks to use only a minimal subset of input features, inherently explaining which features are sufficient for control.
Hierarchical Reinforcement Learning (HRL)
A framework that decomposes a task into a hierarchy of sub-policies and options, providing a temporal abstraction that explains long-horizon behavior.
Successor Representation
A cognitive model that decomposes the value function into a reward-independent predictive map of future states, explaining the agent's temporal abstraction of the environment.
Skill Discovery
The unsupervised learning of reusable, temporally extended behaviors (options), often via mutual information maximization, which can be inspected as interpretable primitives.
Algorithmic Fairness Auditing
Terms related to detecting, measuring, and mitigating unwanted bias in machine learning models to ensure equitable outcomes. Target: Compliance officers and ethical AI leads.
Algorithmic Bias
The systematic and repeatable error in a machine learning model that creates unfair outcomes, privileging one arbitrary group over another.
Demographic Parity
An independence-based fairness criterion requiring a model's positive prediction rate to be equal across all groups defined by a protected attribute.
Equalized Odds
A separation-based fairness metric requiring a classifier to have equal true positive rates and equal false positive rates across different sensitive groups.
Equal Opportunity
A fairness criterion that requires only the true positive rate to be equal across protected groups, ensuring equally qualified individuals have the same chance of being selected.
Predictive Parity
A sufficiency-based fairness metric requiring that the positive predictive value, or precision, be equal across groups, meaning a positive prediction implies the same probability of success regardless of group membership.
Disparate Impact
A legal doctrine and quantitative fairness metric that identifies facially neutral practices which disproportionately harm members of a protected group, often measured by the 80% rule.
Counterfactual Fairness
A causal definition of fairness where a prediction for an individual is considered fair if it remains the same in the actual world and a counterfactual world where the individual belonged to a different demographic group.
Individual Fairness
A fairness principle requiring that similar individuals receive similar predictions, formalized by a distance metric constraint on the input and output spaces.
Group Fairness
A class of fairness definitions that partition a population into groups defined by a protected attribute and require a statistical measure to be equal across these groups.
Adversarial Debiasing
An in-processing bias mitigation technique that uses an adversarial network to remove sensitive information from a model's learned representations while maximizing predictive accuracy.
Proxy Discrimination
A form of algorithmic bias where a non-protected feature, such as zip code, serves as a stand-in for a protected attribute like race, allowing disparate impact to occur indirectly.
Protected Attribute
A legally or ethically recognized characteristic of an individual, such as race, gender, or age, that must not be used as a basis for discriminatory decision-making.
Disparate Treatment
The intentional and explicit differential treatment of individuals based on their membership in a protected class, constituting direct discrimination.
Fairness Through Unawareness
A naive and often ineffective fairness intervention where a model is simply denied direct access to a protected attribute, ignoring the problem of proxy discrimination through correlated features.
Causal Fairness
A framework for defining and auditing fairness using causal inference and structural causal models to distinguish between discriminatory and legitimate causal pathways in a prediction.
Intersectional Fairness
A fairness paradigm that examines bias against subgroups defined by the combination of multiple protected attributes, such as race and gender, rather than treating each attribute in isolation.
Multicalibration
A strong fairness guarantee ensuring a predictor is calibrated not just on the overall population but simultaneously on a rich collection of computationally identifiable, potentially intersecting subgroups.
Bias Audit
A systematic, often third-party evaluation of an algorithmic system to detect and measure discriminatory outcomes against protected groups before or after deployment.
Model Card
A structured transparency document, pioneered by Google, that reports the intended use, evaluation results, and ethical considerations of a trained machine learning model.
Datasheet for Datasets
A standardized document that accompanies a dataset, detailing its motivation, composition, collection process, and recommended uses to enhance transparency and accountability.
Accuracy-Fairness Trade-off
The observed tension where enforcing strict fairness constraints on a model can lead to a measurable reduction in its overall predictive accuracy.
Representation Bias
A form of data bias arising from how a population is sampled and defined, leading to a training dataset that underrepresents or misrepates certain groups.
Historical Bias
A pernicious form of bias where the state of the world as captured in data reflects existing structural prejudices, causing a model to learn and perpetuate these historical injustices.
Feedback Loop
A phenomenon in deployed ML systems where a model's biased predictions influence the future data it is trained on, creating a self-reinforcing cycle that amplifies initial biases.
Algorithmic Recourse
The ability for an individual negatively affected by an algorithmic decision to understand the reasons and take actionable steps to reverse that decision in the future.
Reject Option Classification
A post-processing bias mitigation technique that defers decisions in a region of high uncertainty near a classifier's decision boundary, assigning favorable outcomes to a disadvantaged group in that zone.
Fairness Indicators
A suite of tools, often associated with TensorFlow, that enables the computation and visualization of common fairness metrics for binary and multi-class classifiers during model evaluation.
Slicing Analysis
A model debugging technique that computes performance metrics on semantically meaningful data subsets, or slices, to uncover performance disparities hidden by aggregate metrics.
Differential Fairness
A privacy-inspired definition of fairness that uses the concept of differential privacy to limit the influence of a protected attribute on a model's output distribution.
Fairness in LLMs
The specialized study of detecting and mitigating social biases, stereotypes, and toxic outputs encoded within the vast training corpora and parameters of large language models.
Chain-of-Thought Transparency
Terms related to eliciting and inspecting the intermediate reasoning steps generated by large language models. Target: CTOs validating LLM logic for complex tasks.
Chain-of-Thought Prompting
A prompting technique that elicits a series of intermediate reasoning steps from a large language model to improve performance on complex tasks requiring multi-step logic.
Zero-Shot CoT
A variant of chain-of-thought prompting that uses a simple trigger phrase like 'Let's think step by step' to elicit reasoning without providing any prior examples.
Few-Shot CoT
A chain-of-thought prompting method that provides the model with a small number of complete examples containing both the question and the detailed reasoning path before asking the target question.
Auto-CoT
An automated method for generating chain-of-thought demonstrations by clustering diverse questions and sampling their reasoning chains, eliminating the need for manual example creation.
Self-Consistency
A decoding strategy that samples multiple diverse reasoning paths for a single problem and selects the most consistent final answer, improving the reliability of chain-of-thought outputs.
Tree-of-Thoughts
A framework that extends chain-of-thought prompting by exploring multiple reasoning paths simultaneously in a tree structure, allowing the model to look ahead and backtrack during problem-solving.
Graph-of-Thoughts
An advanced reasoning framework that models the thought process as a directed graph, enabling complex merging, branching, and cycling of intermediate reasoning steps for intricate problem-solving.
ReAct
A prompting paradigm that interleaves reasoning traces and action steps, enabling a language model to generate task-specific actions and process external observations in a synergistic loop.
Faithful CoT
A reasoning trace that accurately reflects the true causal process by which the model arrived at its final answer, free from post-hoc rationalization or confabulation.
Process Supervision
A training methodology that provides feedback on each intermediate step of a model's reasoning chain, rewarding correct logical progression rather than just the final outcome.
Outcome Supervision
A training approach that provides feedback based solely on the correctness of the final answer, without evaluating the validity of the intermediate reasoning steps.
Process Reward Model
A specialized model trained to evaluate the correctness and logical validity of each step in a reasoning chain, used to guide and score the step-by-step generation of a policy model.
Chain-of-Verification
A technique where a language model generates an initial response, then systematically drafts and answers a series of independent verification questions to fact-check and correct its own output.
Self-Refine
An iterative prompting framework where a single language model generates an initial output, provides its own feedback on that output, and then uses the critique to produce a refined version.
Reflexion
An agentic framework that uses verbal reinforcement learning, where an agent reflects on task failure signals stored in episodic memory to improve its reasoning and decision-making on subsequent attempts.
STaR
A bootstrapping technique where a model is trained on its own correctly reasoned answers, using rationales that led to correct outcomes to iteratively improve its intrinsic reasoning capabilities.
Least-to-Most Prompting
A problem-solving strategy that decomposes a complex task into simpler sub-problems, solving each sequentially and feeding the solutions of prior sub-problems into the context for the next.
Program-of-Thoughts
A prompting approach that directs a language model to generate executable programming code as its reasoning trace, offloading the computational execution to an external interpreter for precise results.
Reasoning Token
A specialized token inserted into a model's output sequence to signal a pause for internal computation, allowing the model to perform additional processing before generating the next text token.
Activation Patching
A causal intervention technique in mechanistic interpretability that replaces a model's internal activation at a specific location with a corrupted or alternative activation to isolate its function.
Logit Lens
A technique for interpreting transformer models by applying the unembedding matrix directly to intermediate residual stream activations, revealing the model's next-token predictions at each layer.
Sparse Autoencoder
An unsupervised neural network trained to reconstruct its input through a bottleneck with a sparsity penalty, used in mechanistic interpretability to decompose model activations into monosemantic features.
Superposition Hypothesis
The theory that neural networks represent more independent features than they have dimensions in a given layer by encoding them in overlapping, nearly orthogonal directions in the activation space.
Hallucination Snowballing
A failure mode in language models where an initial factual error in a reasoning chain leads to a cascade of subsequent errors, as the model builds further logic on the incorrect premise.
Post-Hoc Rationalization
The phenomenon where a model generates a plausible-sounding but causally inaccurate justification for a decision after the decision has already been made, masking the true underlying heuristics.
Clever Hans Effect
A model's reliance on spurious statistical correlations in training data to make correct predictions for the wrong reasons, analogous to the horse that appeared to do math but was reading cues.
Multi-Hop Reasoning
The cognitive process of connecting multiple distinct pieces of information across a long context or multiple documents to derive a conclusion that is not explicitly stated in any single source.
Tool-Augmented Reasoning
A paradigm where a language model is given the ability to call external tools like calculators, search engines, or code interpreters to ground its reasoning in verifiable, external computation.
Semantic Entropy
A measure of uncertainty in language model outputs that clusters token-level predictions by their semantic meaning, distinguishing between high uncertainty from multiple plausible answers and lexical variation.
Faithfulness Metric
A quantitative score designed to measure the degree to which a generated reasoning trace accurately and causally represents the model's true computational process for arriving at an answer.
Model Probing and Decoding
Terms related to using diagnostic classifiers and techniques to extract encoded linguistic and world knowledge from model representations. Target: Researchers auditing model capabilities.
Linear Probing
A technique for training a simple linear classifier on top of a frozen model's internal representations to diagnose what information is encoded at a specific layer.
Sparse Autoencoder
An unsupervised neural network trained to reconstruct activations from a model while enforcing sparsity, used to decompose polysemantic neurons into interpretable, monosemantic features.
Feature Visualization
An optimization-based method that generates synthetic inputs to maximally activate a specific neuron, channel, or feature, revealing what pattern the model has learned to detect.
Logit Lens
A decoding technique that applies the unembedding matrix directly to intermediate residual stream states, allowing researchers to inspect a model's iterative token predictions before the final layer.
Activation Patching
A causal intervention method that replaces a model's internal activation at a specific location with a cached activation from a different input to localize where a computation occurs.
Causal Tracing
A method for identifying the specific hidden states within a transformer that are causally responsible for recalling a fact by systematically restoring clean activations from a corrupted forward pass.
Knowledge Neuron
A specific neuron in a feed-forward network whose activation is strongly correlated with the expression of a particular factual association, identified through activation analysis.
Steering Vector
A direction in a model's activation space that, when added to the residual stream during inference, reliably modifies the model's high-level behavior, such as its sentiment or truthfulness.
Superposition Hypothesis
The theory that neural networks represent more independent features than they have dimensions by encoding them in overlapping, nearly orthogonal directions within a shared activation space.
Monosemanticity
The property of a neuron or feature that activates exclusively for a single, human-interpretable concept, representing the ideal decomposition goal of mechanistic interpretability.
Polysemantic Neuron
A neuron that responds to multiple unrelated input patterns, complicating direct interpretation and motivating the use of sparse autoencoders for feature disentanglement.
Dictionary Learning
A sparse coding approach applied to model activations to find an overcomplete basis of interpretable feature directions that decompose the superimposed representations of a neural network.
Causal Mediation Analysis
A statistical framework for quantifying how much a model's output depends on a specific intermediate representation by measuring the effect of intervening on that representation.
Residual Stream
The core data pathway in a transformer architecture where each layer reads from and writes additive updates to a running hidden state, forming the primary object of probing analysis.
Probing Task
A carefully designed auxiliary classification or regression task used to test whether a model's learned representations contain specific linguistic or world knowledge.
Structural Probe
A diagnostic classifier designed to test whether a model's embeddings implicitly encode parse trees by predicting the depth or distance between tokens in a syntax tree.
Ablation
A causal technique that removes or zeroes out a model component, such as a neuron or attention head, to measure the resulting drop in performance and infer its functional importance.
Causal Abstraction
A framework for evaluating whether a neural network implements a specific high-level causal algorithm by aligning its internal components with the variables of a causal graph.
Interchange Intervention
A causal method that replaces the activation of a model component processing a base input with the activation from a source input to test if the component computes a specific variable.
Concept Erasure
A technique for removing a specific linear concept direction, such as a bias or protected attribute, from a model's representations to prevent it from influencing downstream predictions.
Representation Engineering
A top-down control paradigm that reads and writes high-level cognitive vectors in a model's activation space to monitor and steer behavior without retraining.
Contrastive Activation Addition
A method for computing a steering vector by subtracting the mean activations of a negative-prompt dataset from a positive-prompt dataset and adding the difference during inference.
DoLa Decoding
A contrastive decoding strategy that amplifies the knowledge from later transformer layers while subtracting the logits of earlier layers to reduce hallucinations in large language model outputs.
Contrastive Decoding
An inference-time search heuristic that maximizes the probability difference between an expert model and an amateur model to generate more factual and coherent text.
Inference-Time Intervention
A general class of techniques that modify a model's internal activations during the forward pass to alter its behavior without updating the original weights.
Activation Addition
A lightweight control method that adds a pre-computed bias vector to the residual stream at a specific layer to induce a desired behavior, such as a refusal or a specific sentiment.
Binding Problem
The challenge of understanding how a neural network dynamically associates distinct features, such as a color and a shape, to represent a specific composite object without confusion.
Model Inversion
An attack that reconstructs private training data or sensitive features by inverting the gradients or internal representations of a deployed machine learning model.
Membership Inference Attack
A privacy attack that determines whether a specific data record was part of a model's training set by analyzing the model's confidence scores or internal activations.
Unembedding Matrix
The final linear layer in a language model that maps the high-dimensional residual stream state to a probability distribution over the vocabulary of tokens.
Temporal Model Explainability
Terms related to attributing importance to specific time steps in sequence models used for forecasting and time-series analysis. Target: Engineers in finance and IoT analytics.
Temporal SHAP
A method that adapts Shapley value calculations to assign importance scores to individual time steps in a sequence model's prediction.
Temporal Integrated Gradients
A gradient-based attribution technique that computes the integral of gradients along a path from a baseline to the input, specifically adapted for time-series data to highlight influential time steps.
Attention Flow
A technique for tracking the propagation of attention weights across layers in a Transformer model to quantify how information from a specific time step influences the final prediction.
Layer-wise Relevance Propagation (LRP) for Sequences
A decomposition method that backpropagates a model's prediction score through the layers of a recurrent or temporal convolutional network to assign relevance scores to each input time step.
Time-Step Ablation
A perturbation-based method that systematically removes or masks individual time steps from a sequence to measure the resulting change in the model's output and determine their importance.
Temporal Occlusion Analysis
An interpretability technique that slides a masking window across a time series, occluding segments to generate a saliency map showing which temporal intervals are most critical for a prediction.
Counterfactual Temporal Trajectory
A generated alternative time-series path with minimal changes from the original input that would cause a forecasting model to produce a different, desired outcome.
Granger Causality Saliency
An attribution method that quantifies the predictive causality between time series by measuring how much past values of one series improve the forecast of another, often used to validate model attention.
Temporal Fusion Transformer Interpretability
The built-in variable selection and attention mechanisms within the Temporal Fusion Transformer architecture that natively provide feature and time-step importance for multi-horizon forecasting.
Shapelet Attribution
A technique that identifies and assigns importance to discriminative subsequences (shapelets) in a time series that are most representative of a particular class or prediction.
Dynamic Mode Decomposition Attribution
An interpretability method that decomposes a sequence model's hidden state dynamics into spatiotemporal coherent modes to attribute predictions to underlying system behaviors.
LSTM Gate Activation Analysis
A diagnostic process that visualizes and quantifies the activations of forget, input, and output gates in an LSTM network to explain how information is stored, updated, or discarded at each time step.
Temporal Surrogate Model
An interpretable proxy model, such as a decision tree, trained to approximate the predictions of a complex temporal model on a local or global scale to provide explanations.
Forecast Error Contribution
A decomposition technique that breaks down a model's total prediction error into additive components attributable to specific time steps, features, or sources of uncertainty.
Temporal Causal Attribution
The process of identifying which past time steps and features are the actual causal drivers of a model's forecast, often using structural causal models or intervention analysis.
Sequence Perturbation Testing
A robustness evaluation method that introduces small, controlled noise or distortions to a time series to analyze the stability and continuity of a model's explanations.
Change Point Detection Attribution
An explainability method that identifies points in a time series where the statistical properties shift and quantifies their disproportionate influence on a model's subsequent predictions.
Seasonality Decomposition Attribution
A technique that isolates the trend, seasonal, and residual components of a time series to attribute a model's prediction to these distinct underlying patterns.
Temporal Disentanglement
A representation learning approach that separates a model's latent space into independent factors corresponding to static and dynamic attributes, enabling attribution to time-invariant or time-varying concepts.
Conformal Time-Step Importance
A method that applies conformal prediction to produce statistically valid prediction intervals and then attributes the width of the interval at each horizon to the uncertainty contributed by specific time steps.
Temporal DeepLIFT
An attribution method that computes importance scores for time steps by comparing the activation of each neuron to a reference activation, satisfying the summation-to-delta property for recurrent networks.
Time-Series Counterfactual Generation
The algorithmic process of creating a realistic, alternative time series that is minimally different from the original but leads to a different model classification or forecast.
Sequence Influence Function
A robust statistical method that estimates the effect of removing a specific training sequence on a model's parameters and its prediction for a test sequence, identifying influential training examples.
Temporal Grad-CAM
An adaptation of the Gradient-weighted Class Activation Mapping technique that uses gradients flowing into the final convolutional layer of a temporal model to produce a heatmap of important time steps.
Temporal Anomaly Attribution
The process of decomposing an anomaly score generated by a time-series model to identify the specific time steps and features that contributed most to the detection of an unusual event.
Temporal Faithfulness Metric
A quantitative evaluation score that measures how accurately a temporal explanation reflects the true reasoning process of the underlying model by testing its correlation with model behavior under perturbation.
Time-Series Saliency Map
A visual representation, typically a heatmap overlaid on the input sequence, that highlights the relative importance of each time step for a specific model prediction.
Temporal Causal Discovery
The application of algorithms like PCMCI or VAR-LiNGAM to time-series data to infer a causal graph, which is then used to validate or generate feature and lag attributions for a predictive model.
Temporal Prototype Attribution
An example-based explanation method that identifies representative time-series prototypes in the training data and attributes a prediction to the model's similarity to these learned patterns.
Temporal Prediction Difference
An attribution method that quantifies the importance of a time step by measuring the difference in the model's output probability when the feature value at that step is marginalized out.
Adversarial Example Interpretability
Terms related to understanding why models fail on imperceptibly perturbed inputs to diagnose vulnerabilities. Target: Security engineers hardening ML systems.
Adversarial Example
An input to a machine learning model that has been intentionally perturbed in a way imperceptible to humans, causing the model to make an incorrect prediction with high confidence.
Adversarial Robustness
The measured resilience of a neural network against adversarial examples, quantifying the model's ability to maintain prediction accuracy when inputs are subjected to malicious perturbations.
Evasion Attack
An attack vector where an adversary modifies a malicious input sample at test time to cause misclassification by a trained model, effectively evading a security or classification boundary.
White-Box Attack
An adversarial attack methodology where the attacker has full knowledge of the target model's architecture, parameters, and gradients, enabling precise computation of worst-case perturbations.
Black-Box Attack
An adversarial attack executed without access to the model's internal weights or gradients, relying solely on querying the model's output scores or decisions to estimate the decision boundary.
Fast Gradient Sign Method (FGSM)
A single-step, white-box attack that generates an adversarial example by adding a small perturbation in the direction of the sign of the gradient of the loss function with respect to the input.
Projected Gradient Descent (PGD)
A powerful iterative extension of FGSM that applies multiple small gradient steps and projects the result back onto an epsilon-ball around the original input to find a strong adversarial example.
Carlini & Wagner Attack (C&W)
An optimization-based attack formulated to find minimally distorted adversarial perturbations by minimizing a loss that balances the Lp-norm distance and the misclassification objective.
Gradient Masking
A phenomenon where a defense causes the model's gradients to become useless or misleading to an attacker, often creating a false sense of security rather than true adversarial robustness.
Adversarial Training
A defensive technique that injects adversarial examples into the training dataset with correct labels, forcing the model to learn a smoother and more robust decision boundary.
Defensive Distillation
A defense mechanism that trains a second model using the soft probability outputs of a first model, aiming to smooth the loss surface and reduce the sensitivity of the model to small input variations.
Universal Adversarial Perturbation
A single, image-agnostic perturbation vector that, when added to most natural images from a dataset, causes a high rate of misclassification across the entire distribution.
Adversarial Patch
A localized, highly visible perturbation placed within a scene that causes a model to ignore the rest of the context and output a specific attacker-chosen target class.
Physical Adversarial Attack
An attack that remains effective after being printed and captured by a camera in the physical world, demonstrating that adversarial vulnerabilities persist across varying real-world conditions.
Feature Squeezing
A detection method that compares the model's prediction on an original input against its prediction on a squeezed version to identify adversarial examples by their high sensitivity to bit-depth reduction.
Adversarial Example Transferability
The property by which an adversarial example crafted to fool one specific model also causes misclassification in other independently trained models with different architectures.
Decision Boundary Analysis
The process of visualizing and probing the geometric surface that separates classes in the input space to understand why a model is vulnerable to adversarial perturbations in specific regions.
Loss Landscape Visualization
A technique for plotting the loss function in the high-dimensional input space around a data point to reveal sharp, non-smooth curvature that correlates with high adversarial vulnerability.
Robustness Certificate
A formal, verifiable guarantee that a model's prediction will remain constant for any perturbation of the input within a specified Lp-norm radius, providing a provable lower bound on robustness.
Randomized Smoothing
A technique for constructing a provably robust classifier by adding Gaussian noise to inputs and aggregating predictions, creating a smooth model that certifies robustness against L2-norm attacks.
Perturbation Budget
The maximum allowed magnitude of an adversarial perturbation, typically constrained by an Lp-norm such as L-infinity, defining the threat model's capacity to alter the input.
Imperceptibility Threshold
The perceptual limit below which a human observer cannot distinguish the adversarial example from the original input, serving as the constraint for generating stealthy attacks.
Adversarial Saliency Map
A visualization tool that identifies the specific input pixels an adversary must modify to achieve a targeted misclassification, highlighting the model's most sensitive input features.
Semantic Adversarial Example
An input that is modified in a semantically meaningful way, such as changing the color or rotation of an object, which causes misclassification without relying on pixel-level noise.
Adversarial Debiasing
The use of adversarial training techniques to remove unwanted biases from model representations by training a predictor to be invariant to protected attributes while maintaining task performance.
Model Distillation for Interpretability
Terms related to training a simpler, transparent student model to mimic a complex teacher model for explanation purposes. Target: Architects deploying interpretable systems.
Knowledge Distillation
A model compression technique where a smaller 'student' model is trained to replicate the behavior and predictive performance of a larger, more complex 'teacher' model.
Teacher-Student Architecture
A training framework consisting of a high-capacity teacher model that generates supervisory signals for a compact student model to mimic, transferring knowledge through softened outputs.
Soft Targets
The probability distributions over classes produced by a teacher model, typically smoothed by a high temperature parameter, which provide richer information about inter-class similarities than hard labels.
Temperature Scaling
A hyperparameter applied to the softmax function during distillation that controls the softness of the output probability distribution, revealing the 'dark knowledge' of the teacher's internal representations.
Distillation Loss
A composite objective function that combines the Kullback-Leibler divergence between teacher and student soft targets with the standard cross-entropy loss against ground-truth labels.
Kullback-Leibler Divergence
A statistical measure of how one probability distribution diverges from a second, reference distribution, used as the primary loss component to align student outputs with teacher soft targets.
Dark Knowledge
The implicit information about the generalization and similarity structure of the data encoded in the relative probabilities of incorrect classes within a teacher model's softmax output.
Surrogate Model
An inherently interpretable model, such as a decision tree or linear model, trained to approximate the predictions of a complex black-box model for the purpose of generating explanations.
Rule Extraction
The process of deriving a set of symbolic, human-readable if-then rules from a trained neural network by querying it and distilling its decision boundaries into a logical form.
Decision Tree Surrogate
A globally interpretable tree-based model trained on the input-output pairs of a black-box model to provide a faithful, high-level approximation of its overall decision logic.
Linear Proxy Model
A simple linear model, such as LASSO or logistic regression, trained to mimic a complex model's predictions locally or globally to provide feature-level importance scores.
Explainable Boosting Machine (EBM)
A glass-box model that combines additive feature functions with automatic interaction detection, often used as a high-fidelity, inherently interpretable student for distilling complex models.
Dataset Distillation
A technique that synthesizes a small set of informative training samples from a large dataset, such that a model trained on this synthetic set achieves performance comparable to one trained on the original data.
Self-Distillation
A process where a model is iteratively trained using its own previous predictions as soft targets, often improving generalization and calibration without requiring a separate teacher model.
Ensemble Distillation
The process of compressing the collective knowledge of a diverse ensemble of teacher models into a single, faster student model that mimics the ensemble's averaged predictions.
Born-Again Network
A neural network trained via self-distillation where a student model with an identical architecture to the teacher is trained on the teacher's outputs, often achieving superior performance.
Model Compression
A broad set of techniques, including distillation, pruning, and quantization, used to reduce the computational and memory footprint of a machine learning model for efficient deployment.
Fidelity-Evaluated Student
A student model whose quality is measured by its fidelity—the degree to which its predictions match those of the teacher model on unseen data, rather than solely by accuracy on ground-truth labels.
Interpretability-Accuracy Tradeoff
The fundamental design tension between a model's predictive performance and its transparency, which distillation aims to navigate by creating a simpler model that retains high fidelity to a complex one.
Post-Hoc Distillation
The standard distillation paradigm where an interpretable student model is trained to explain a pre-existing, fully trained black-box teacher model after the teacher's training is complete.
Global Surrogate
An interpretable model trained to approximate the entire decision boundary of a black-box model across the whole input space, providing a complete but approximate explanation of its behavior.
Local Surrogate
An interpretable model trained to approximate a black-box model's predictions in the immediate vicinity of a specific input instance, explaining a single prediction rather than the whole model.
Feature-Based Distillation
A distillation approach where the student model is trained to match the intermediate feature representations or activation maps of the teacher model, not just its final output logits.
Attention Transfer
A specific form of feature-based distillation where the student model is trained to mimic the spatial attention maps generated by the teacher's layers, transferring the focus mechanism.
Transparent-by-Design Student
A student model chosen for its inherent structural interpretability, such as a Generalized Additive Model or shallow decision tree, ensuring the distilled model is natively explainable.
Rule-Regularized Distillation
A training method that adds a penalty term to the distillation loss to encourage the student model's decision boundaries to be simple and amenable to extraction as a compact set of logical rules.
Distillation for Decision Trees
The specific application of knowledge distillation where the student model is a decision tree algorithm like CART or C4.5, used to create a human-readable flowchart of the teacher's logic.
Distillation for Logistic Regression
The process of using a complex model's soft targets to train a logistic regression student, yielding a globally interpretable linear model with calibrated probability estimates.
Distillation for Rule Lists
Training a student model that produces a decision set or rule list, such as those from the SIRUS algorithm, to provide a stable and sparse set of if-then explanations for a black-box teacher.
Distillation for Generalized Additive Models
The process of training an inherently interpretable GAM student to mimic a complex teacher, resulting in a model where the contribution of each feature can be visualized and understood via shape functions.
Automated Rationale Generation
Terms related to systems that produce natural language justifications alongside their predictions. Target: Product managers building human-AI interfaces.
Natural Language Explanations (NLE)
Human-readable justifications generated by a model alongside its prediction to articulate the reasoning behind a decision.
Rationale Generation
The automated process of producing a coherent, evidence-based textual justification that supports a specific model output.
Post-Hoc Rationalization
The technique of generating an explanation for a prediction after the model has made its decision, often using a secondary explainer model.
Faithful Rationales
Generated explanations that accurately reflect the true internal reasoning process of the model, not just a plausible-sounding story.
Plausible Rationales
Human-acceptable explanations that sound convincing but may not accurately represent the model's actual decision-making mechanics.
Self-Explaining Neural Networks (SENN)
Models architected with an intrinsic explainability component that generates interpretations as part of the forward pass.
Concept-Based Explanations
Justifications structured around high-level, human-understandable concepts rather than low-level input features.
Contrastive Explanations
Rationales that explain why a model predicted outcome A instead of a contrasting outcome B, often highlighting minimal necessary conditions.
Counterfactual Rationales
Natural language descriptions of the minimal changes to an input that would have resulted in a different, desired prediction.
Causal Rationales
Explanations grounded in cause-and-effect relationships rather than mere statistical correlations within the training data.
Chain-of-Thought Prompting
A technique that elicits step-by-step reasoning from large language models by providing few-shot examples of intermediate logical steps.
Rationale Consistency
A metric evaluating whether a model generates logically coherent and non-contradictory explanations across similar inputs.
Explanation Faithfulness
The degree to which a generated rationale accurately mirrors the true computational logic used by the model to arrive at a prediction.
Simulatability
The ability of a human observer to use a model's explanation to correctly anticipate the model's output on a new, unseen input.
Verbalized Uncertainty
The capability of a model to express its confidence level or doubt in natural language alongside its predictions and rationales.
Evidence Attribution
The mechanism of grounding generated explanations by explicitly pointing to specific segments of the source input data as proof.
Source Grounding
The process of linking claims within a generated rationale directly to verifiable external documents or specific training data points.
Citation Generation
The automated creation of precise references to source documents that support the factual assertions made in a model's explanation.
Factual Consistency
A metric ensuring that the content of a generated rationale does not contradict real-world knowledge or the provided source data.
Hallucination Detection
Techniques used to identify and flag generated explanations that contain fabricated, nonsensical, or unfaithful information.
Faithfulness Metrics
Quantitative measures designed to automatically score how accurately a generated explanation reflects the model's internal decision process.
Interactive Explanations
A dynamic interface allowing users to probe a model with follow-up questions or adjust parameters to explore the reasoning space.
User-Adaptive Explanations
Rationales that are dynamically tailored to the technical expertise, role, or specific needs of the individual end-user.
Minimal Sufficient Explanations
The practice of providing the shortest possible justification that is still complete enough to justify the model's decision.
Anchors
High-precision, if-then rules that sufficiently 'anchor' a prediction locally, ensuring the decision remains fixed regardless of changes to other feature values.
LLM-as-Explainer
The paradigm of using a large language model's generative capabilities to produce post-hoc rationales for itself or other black-box systems.
Explanation Policy
A governance framework defining the rules, format, and timing for when and how an AI system must provide justifications to users.
GDPR Right to Explanation
The regulatory requirement under the General Data Protection Regulation for providing meaningful information about the logic involved in automated decisions.
Model Cards
Structured transparency artifacts documenting a model's intended use, evaluation results, and limitations, often including example rationales.
Actionable Explanations
Rationales that not only explain a decision but also provide the user with clear steps to change the outcome in the future.
Multimodal Explainability
Terms related to interpreting models that jointly process and reason over multiple data types like text and images. Target: Engineers working on vision-language systems.
Multimodal Explainability
The field of techniques for interpreting predictions made by AI models that jointly process multiple data modalities, such as text and images, to identify which cross-modal interactions were most influential.
Cross-Modal Attention Maps
Visualizations of the attention weights between tokens from different modalities, such as image patches and text words, revealing how a vision-language model grounds linguistic concepts in visual regions.
Vision-Language Grounding
The process of identifying the specific correspondences between textual phrases and image regions that a multimodal model uses to align semantic concepts across modalities for a given prediction.
Multimodal Grad-CAM
An extension of the Gradient-weighted Class Activation Mapping technique that produces visual explanations for vision-language models by highlighting the image regions most influential for a text-conditioned prediction.
Modality Ablation
An explainability method that systematically removes or zeroes out one input modality to measure its causal contribution to the model's final output and assess cross-modal reliance.
Multimodal LIME
An adaptation of Local Interpretable Model-agnostic Explanations that perturbs inputs across multiple modalities to train a local surrogate model and explain individual multimodal predictions.
SHAP for Multimodal
The application of Shapley Additive Explanations to fairly distribute the credit for a multimodal model's prediction among the input features from all contributing modalities.
Joint Embedding Visualization
Techniques for projecting the shared, high-dimensional representation space of a multimodal model into 2D or 3D to inspect the alignment and clustering of concepts from different modalities.
CLIP Interpretability
The specific study of how Contrastive Language-Image Pretraining models like OpenAI's CLIP internally represent and associate visual and textual concepts to form their aligned multimodal understanding.
Multimodal Concept Activation Vectors (MCAV)
A method that extends TCAV to measure the sensitivity of a multimodal model's predictions to high-level, human-interpretable concepts that span both visual and textual inputs.
Fusion Layer Probing
The practice of training diagnostic classifiers on the hidden states of a multimodal model's fusion layers to decode what cross-modal information is represented at different stages of processing.
Multimodal Counterfactuals
Explanations that identify the minimal, synchronized changes to inputs in multiple modalities that would alter a multimodal model's prediction to a specified alternative outcome.
Cross-Modal Attribution
A class of methods that assign importance scores to input features in one modality based on their interaction with and influence on features from another modality within a multimodal model.
Modality Importance Weighting
A technique that quantifies the overall contribution of each input modality to a specific prediction, producing a scalar weight that indicates which data stream the model relied on most.
Multimodal Faithfulness
A metric that evaluates whether the features identified as important by a multimodal explanation truly influence the model's prediction when those cross-modal features are perturbed or removed.
Cross-Modal Attention Flow
A method for tracking the propagation and aggregation of attention weights across different modalities through the successive layers of a transformer-based multimodal architecture.
Multimodal Integrated Gradients
An attribution method that computes the path integral of the gradients for all input modalities from a neutral baseline to the actual input, satisfying the completeness axiom for multimodal models.
Modality Dropout Explainability
An analysis technique that randomly drops one modality during inference to study the resulting change in prediction confidence and diagnose the model's dependence on cross-modal information.
Multimodal Layer-wise Relevance Propagation (MLRP)
A technique that backpropagates a model's prediction score through the network layers, decomposing the relevance and assigning it to the input features of all modalities while conserving the total relevance.
Vision-Language Saliency Maps
Visual heatmaps that highlight the image pixels most relevant to a specific textual query or class, generated by computing the gradient of a text-conditioned vision model's output with respect to the input image.
Multimodal Causal Mediation Analysis
A technique from causal inference applied to multimodal models to identify specific cross-modal neurons or representations that function as causal mediators between an input concept and the model's output.
Cross-Modal Representation Similarity
The quantitative comparison of the internal representations of different modalities using metrics like Centered Kernel Alignment to understand how a model aligns and fuses distinct data streams.
Multimodal Prototype Networks
An interpretable architecture that learns representative cross-modal prototypes from the training data and makes predictions based on the similarity of a new multimodal input to these learned examples.
Vision-Language Rationale Extraction
The task of generating a textual justification that explains a vision-language model's prediction by referencing the specific visual evidence and cross-modal reasoning that supported the decision.
Cross-Modal Attention Rollout
A method that linearly combines attention matrices across transformer layers to trace how information from one modality propagates to another, producing a single map of cross-modal information flow.
Modality Fusion Entropy
A metric that measures the uncertainty or dispersion in how a model distributes its attention across different modalities at the fusion point, indicating whether it uses a balanced or biased integration strategy.
Multimodal Adversarial Robustness
The study of a multimodal model's susceptibility to adversarial perturbations in one or more modalities, used to expose brittle cross-modal correlations and understand failure modes.
Cross-Modal Concept Bottlenecks
An architectural intervention that forces a model to predict a set of human-interpretable concepts spanning multiple modalities before making a final prediction, enabling direct inspection of the cross-modal reasoning.
Multimodal Occlusion Sensitivity
A perturbation-based method that systematically occludes regions of an image or masks words in a text to measure the resulting change in a multimodal model's prediction, identifying critical cross-modal features.
Vision-Language Attention Entropy
A diagnostic metric that quantifies the focus or dispersion of a model's cross-modal attention heads, where high entropy indicates a broad, unfocused gaze and low entropy indicates a sharp, specific grounding.
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us