Interpretable Representation

The transformed features must map directly to concepts a human can reason about. For text, this is often a bag-of-words indicating the presence or absence of individual tokens. For images, superpixel segmentation groups pixels into perceptually meaningful atomic regions. For tabular data, feature discretization bins continuous values into interpretable categories like 'high', 'medium', or 'low'. The key is that a domain expert can look at the representation and immediately grasp what each feature signifies without needing to understand the underlying model.

A common strategy is to convert the original input into a binary vector where each element signals the presence or absence of a specific interpretable component. In text classification, a '1' indicates a word exists in the document regardless of frequency. For images, a binary mask shows which superpixels are 'turned on' or 'turned off'. This binary simplification strips away complex continuous relationships, allowing the surrogate model to learn clear, discrete rules like 'the word not is present' rather than grappling with raw pixel intensities or TF-IDF weights.

The interpretable representation must preserve enough information to accurately approximate the black-box model's decision boundary in the immediate neighborhood of the instance. This creates a fundamental fidelity-interpretability trade-off: overly simplistic representations lose predictive signal, while overly complex ones defeat the purpose of explanation. Effective representations balance this by capturing the features most salient to the local prediction while discarding irrelevant global noise. The representation's quality is measured by how well a simple linear model trained on it can mimic the complex model's output for nearby perturbed samples.

The construction of an interpretable representation is highly dependent on the data type:

• Text: Token masking creates a binary bag-of-words vector. Cosine distance between TF-IDF vectors measures neighborhood proximity.
• Images: Superpixel algorithms like SLIC or Quickshift group pixels into contiguous regions. The representation is a binary vector indicating which superpixels are present.
• Tabular Data: Continuous features are discretized into quantiles or bins. Categorical features may be one-hot encoded or grouped.
• Time Series: Shapelets or interval-based aggregations form interpretable temporal patterns.

An ideal interpretable representation enables the surrogate model to select only a small subset of features for the final explanation. This sparsity is enforced through Lasso regression (L1 regularization) or a dedicated feature selection step. A sparse explanation—highlighting only 5-10 key features—is far more digestible than a dense one listing hundreds of weak contributors. The representation must be structured so that individual features correspond to independently meaningful concepts, allowing the model to zero out irrelevant ones without losing coherence.

The representation must support systematic perturbation to generate a meaningful neighborhood of synthetic samples. For a binary bag-of-words, perturbation involves randomly flipping bits to simulate the removal or addition of words. For superpixels, perturbation means randomly turning regions on or off. The representation's design directly impacts the quality of this synthetic dataset: features must be independent enough that random perturbation creates plausible, near-instance variations rather than nonsensical inputs that the black-box model has never encountered.

Preprocessing step that groups contiguous pixels with similar perceptual characteristics into larger, semantically meaningful atomic regions. These superpixels replace raw pixels as the interpretable features for image explanations.

• Quickshift and SLIC are common algorithms
• Preserves object boundaries while reducing dimensionality
• Each superpixel becomes a binary feature (present/masked) for the surrogate model

Transforms text into a vector where each dimension corresponds to the presence or absence of a specific word from a vocabulary. This interpretable feature space allows the surrogate model to explain predictions in terms of individual word contributions.

• Strips grammar and word order for simplicity
• TF-IDF weighting often applied to downweight common words
• Enables explanations like 'the word terrible decreased the positive sentiment score'

Process of converting continuous numerical features into categorical bins or quantiles. Creates a more human-friendly domain for the local surrogate model by replacing precise values with interpretable ranges.

• Age → 'Young', 'Middle-aged', 'Senior'
• Income → 'Low', 'Medium', 'High'
• Balances granularity against interpretability

Fundamental perturbation strategy where interpretable components are toggled on or off to create the synthetic neighborhood for surrogate training. The representation must support meaningful removal of features.

• Text: word removal or replacement with UNK token
• Images: superpixel graying or blurring
• Tabular: binning into 'present' vs 'absent' states

Proximity measure used in LIME for text data that calculates similarity between two documents based on the angle between their TF-IDF vector representations. Ignores differences in document length, focusing purely on word overlap patterns.

• Values range from 0 (identical) to 1 (orthogonal)
• Weights perturbed samples by their distance from the original
• Enforces the locality constraint in high-dimensional text space

Visual representation of an image explanation where the most important superpixels for a prediction are highlighted. Creates a heatmap overlay showing which regions drove the model's decision.

• Green/red regions indicate positive/negative contributions
• Directly maps interpretable components back to original pixels
• Enables non-technical stakeholders to visually validate model reasoning