Counterfactual Evaluation

The most fundamental binary metric: does the generated counterfactual instance actually flip the model's prediction to the desired target class? A counterfactual is considered valid only if f(x_cf) == y_target. This is a non-negotiable prerequisite before evaluating any other quality metric.

• Calculation: Simple boolean check against the model's output.
• Target: Must be 100% for any generated set to be useful.
• Failure mode: An invalid counterfactual provides no actionable recourse.

Quantifies the cost of change by measuring the distance between the original instance x and the counterfactual x_cf. The goal is to find the closest valid counterfactual to minimize user effort.

• L1 Norm (Manhattan): ||x - x_cf||_1. Encourages sparse changes and is measured in absolute feature units.
• L2 Norm (Euclidean): ||x - x_cf||_2. Penalizes larger individual deviations more heavily.
• Mahalanobis Distance: Accounts for feature correlation by using the inverse covariance matrix, keeping counterfactuals within the plausible data manifold.

Measures the number of features changed between the original instance and the counterfactual. A sparse explanation is cognitively easier to grasp and often more actionable, as a user only needs to modify a few attributes.

• L0 Norm: Counts the exact number of features with non-zero delta.
• Trade-off: High sparsity often conflicts with low proximity; changing one feature by a large amount might be less actionable than changing three features by small amounts.
• Interpretability: Humans can typically process only 2-4 changed features effectively.

Evaluates the dissimilarity among a set of generated counterfactuals. Providing a user with multiple, distinct paths to a favorable outcome is critical if the closest counterfactual is infeasible.

• Determinantal Point Process (DPP): A probabilistic measure used to select a subset of counterfactuals that are both high-quality and maximally diverse.
• Distance between CFs: Average pairwise distance between generated instances.
• Coverage: Ensures the set explores different regions of the decision boundary.

Checks if the counterfactual instance is realistic and grounded in the training data distribution. An implausible counterfactual (e.g., Age=200, Income=-$50k) is an adversarial artifact, not a valid explanation.

• Density Estimation: Uses techniques like Local Outlier Factor (LOF) to verify x_cf lies in a high-density region.
• Causal Constraints: Ensures the counterfactual does not violate known causal relationships (e.g., a child cannot have a PhD).
• Contrast: Proximity alone can find points in empty space; plausibility prevents this.

Measures whether a counterfactual explanation remains valid after model retraining. In production, models are updated frequently; a fragile counterfactual that fails post-update erodes user trust.

• Delta-Robustness: The counterfactual should flip the prediction even under small weight perturbations.
• Adversarial Training: Generation algorithms can be hardened to find counterfactuals that are consistently valid across a family of nearby models.
• Causal Stability: Counterfactuals grounded in structural causal mechanisms are inherently more robust to distribution shifts.

The most fundamental binary metric. A counterfactual is valid if and only if the model's prediction for the generated instance matches the target outcome. This is a hard pass/fail check that verifies the instance has successfully crossed the decision boundary. Without validity, all other metrics are meaningless. It is often the primary constraint in optimization-based generation algorithms.

Measures the distance between the original input and the counterfactual instance, enforcing the principle of minimal change. Common metrics include:

• L1 (Manhattan) distance: Promotes feature-level sparsity.
• L2 (Euclidean) distance: Penalizes large deviations.
• Mahalanobis distance: Accounts for feature correlations and data density, yielding more plausible paths. Lower proximity means the explanation is easier to interpret and the recourse is less costly.

Quantifies how many features were changed to achieve the counterfactual. A sparse counterfactual alters only a small subset of features, making the explanation cognitively manageable for a human. This is typically measured by the L0 norm—a direct count of features with non-zero change. High sparsity is critical for actionable recourse, as users can only realistically modify a limited number of attributes simultaneously.

Ensures the counterfactual instance is realistic and lies within the high-density region of the training data distribution. An implausible counterfactual is an adversarial artifact that provides no meaningful recourse. Plausibility is often evaluated by measuring the distance to the nearest neighbor in the training set or by using a generative model to score the likelihood of the instance. This metric enforces that recommendations respect real-world data constraints.

Instead of a single explanation, a set of diverse counterfactuals offers a user multiple distinct paths to a favorable outcome. Diversity is measured by the pairwise distance between generated counterfactuals. A diverse set avoids over-reliance on a single, potentially infeasible recommendation (e.g., 'reduce age'). It empowers user choice and provides a richer understanding of the model's decision boundary.

Evaluates whether a counterfactual remains valid after model retraining or minor updates. A fragile counterfactual that fails post-update undermines user trust and is not a reliable basis for action. Robustness is measured by the probability that the counterfactual still flips the prediction on a slightly perturbed or retrained model. This is a critical metric for deploying recourse systems in production environments with continuous model learning.