Black-Box Access

The fundamental constraint of black-box access is that the model is treated as an oracle. The explanation system can only submit inputs and receive outputs or prediction probabilities. There is no access to internal weights, gradients, activation maps, or computational graphs. This is the most restrictive access level and is common with third-party APIs like GPT-4 or Claude, where the model is hosted behind a proprietary endpoint.

Because the internal mechanics are invisible, any explanation method operating under black-box access must be model-agnostic. It cannot rely on backpropagation, layer-wise relevance, or attention weights. Instead, it must infer feature importance purely by observing how outputs change in response to input perturbations. LIME and SHAP's KernelExplainer are canonical examples of methods designed explicitly for this access scenario.

Black-box explanation methods depend heavily on prediction probabilities rather than hard class labels. The continuous probability score provides a richer signal for the surrogate model to approximate the local decision boundary. If only a hard label is returned, the explanation loses fidelity because it cannot distinguish between a high-confidence and a marginal prediction. Most commercial LLM APIs return logprobs or confidence scores to enable this.

The primary cost of black-box interpretability is inference latency and API expense. To build a local surrogate model, the system must generate hundreds or thousands of synthetic perturbed samples and query the black-box model for each one. This creates a direct trade-off between explanation fidelity and the query budget. Techniques like submodular pick and OptiLIME aim to reduce this sampling burden without sacrificing explanation quality.

Black-box access is often enforced as a security boundary to protect intellectual property and prevent model extraction attacks. However, this same boundary can be exploited by adversaries using black-box adversarial attacks, which probe the model's decision surface to craft evasion samples. Explanation systems operating in this mode must be careful not to inadvertently leak enough decision boundary information to enable model stealing or membership inference.

• White-Box Access: Full visibility into weights, gradients, and architecture. Enables gradient-based methods like Integrated Gradients.
• Gray-Box Access: Partial visibility, such as access to intermediate layer activations but not weights. Common in federated learning.
• Black-Box Access: Zero internal visibility. The only viable explanation strategies are perturbation-based or surrogate-model approaches. This is the default for all third-party model-as-a-service platforms.

A property of an explanation method indicating it can be applied to any machine learning model regardless of internal architecture. The method requires only the ability to query the model's prediction function—receiving outputs or probability distributions for given inputs. This is the foundational design principle that makes techniques like LIME and SHAP compatible with black-box access scenarios, as they treat the model as an opaque oracle.

An explanation generated after a model has been trained, applied to an existing black-box system without requiring any modification to the model's internal architecture or training procedure. This is the only viable strategy under black-box access, as internal weights and gradients are unavailable. Post-hoc methods work by analyzing the statistical relationship between perturbed inputs and their corresponding outputs to infer feature influence.

The maximum number of API calls or prediction requests an explanation system is permitted to make to the black-box model. This is a critical practical constraint in black-box access scenarios, especially with cloud-hosted models that charge per token or inference. Efficient perturbation sampling strategies and submodular pick algorithms are designed to maximize explanatory insight while minimizing the query budget consumed.

A specific level of black-box access where the model returns a full probability distribution over all classes rather than just the top predicted label. This richer output signal enables more nuanced local surrogate training, as the explanation system can observe how confidence shifts continuously as inputs are perturbed. It contrasts with hard-label access, where only the final class decision is visible, making decision boundary approximation significantly harder.

An intrinsically interpretable model—such as a linear regression, decision tree, or sparse linear model—trained to mimic the predictions of the complex black-box model within a specific local region. Under black-box access, the surrogate is trained on a synthetic dataset of perturbed samples and their queried outputs. The surrogate's transparent structure then serves as the explanation artifact presented to the human auditor.

The process of generating a synthetic neighborhood of data points by randomly altering or masking features of the original instance. These perturbed samples are sent as queries to the black-box model to collect input-output pairs. The strategy varies by data modality:

• Text: Token masking or deletion
• Images: Superpixel occlusion or blurring
• Tabular: Value jittering or categorical swapping This is the only mechanism to probe the local decision boundary without internal access.