Decision Tree Extraction

The extracted tree must maintain a high degree of behavioral equivalence with the original neural policy. Fidelity is measured by the percentage of input-output pairs where the tree and the network agree.

• High-fidelity extraction: >95% agreement on the state-action mapping
• Fidelity-complexity trade-off: Deeper trees increase fidelity but reduce interpretability
• Local fidelity: Accuracy can vary across different regions of the state space

The goal is not perfect replication but a parsimonious approximation that captures the policy's critical decision boundaries while remaining auditable by a human operator.

Unlike stochastic neural policies that output probability distributions over actions, an extracted decision tree provides a single, traceable reasoning path for every input.

• Each decision node applies a binary threshold test on a single input feature
• The path from root to leaf is a conjunction of conditions that is logically verifiable
• No hidden state, no non-linear activation functions, no floating-point accumulation errors

This determinism enables formal verification tools to prove safety properties, such as 'the agent will never select action X when feature Y exceeds threshold Z.'

Effective decision tree extraction algorithms produce trees that use only a minimal subset of the available input features. This sparsity is a direct measure of interpretability.

• A tree with 5 features is cognitively manageable; one with 50 is not
• Regularization penalties during extraction encourage shallower, sparser trees
• Irrelevant features are automatically pruned, revealing which state dimensions actually drive the policy

The selected features serve as a sufficient statistic for the agent's behavior, explaining what the neural network learned to pay attention to during training.

A critical limitation: the extracted tree captures correlational patterns, not causal mechanisms. A split on feature X does not imply X causes the action — it may be a proxy for an unobserved confounder.

• The tree explains what the policy does, not why the environment responds that way
• Interventional analysis is required to verify causal claims
• Spurious correlations from the training data can be faithfully reproduced in the tree

This property distinguishes decision tree extraction from true causal policy analysis. The tree is a descriptive model of the policy, not a causal model of the world.

When the underlying neural policy exhibits monotonic relationships between certain inputs and action preferences, a well-extracted tree can preserve and make explicit these directional constraints.

• Example: 'As inventory level decreases, the probability of ordering more stock never decreases'
• Monotonic splits create interpretable thresholds: 'If temperature > 80°C, always reduce power'
• These guarantees are valuable in regulated domains where decisions must be consistent and non-erratic

Enforcing monotonicity during extraction acts as a form of domain-knowledge regularization, aligning the tree with physical or business constraints.

Each leaf node in the extracted tree can be annotated with the empirical action distribution observed from the teacher network for all training samples that reach that leaf.

• A leaf with 100% action A indicates a high-confidence decision region
• A leaf with a 60/40 split reveals policy ambiguity or a transition boundary
• Confidence scores enable risk-aware deployment: flag low-confidence leaves for human review

This transforms the tree from a simple classifier into a risk-stratified decision aid, combining the transparency of rule-based systems with the nuance of learned behavior.

The process of compressing a complex teacher policy (often a deep neural network) into a simpler, more interpretable student model. While decision tree extraction is one form, distillation can also target linear models or shallow networks. The goal is to preserve the teacher's input-output mapping while gaining structural transparency. Knowledge distillation typically uses the teacher's soft output probabilities as training targets for the student, capturing more information than hard labels alone.

The assignment of an importance score to each dimension of the state space to explain which features are critical for a specific action selection. Methods include:

• SHAP: Game-theoretic Shapley values adapted for RL credit assignment
• Integrated Gradients: Path integral of gradients from a baseline state
• Feature Ablation: Systematically removing inputs to measure policy impact These scores directly inform which state variables a decision tree should split on during extraction.

The application of causal inference tools to determine whether a policy relies on spurious correlations or true causal relationships. Techniques include:

• Intervention analysis: Modifying state variables and observing policy changes
• Counterfactual reasoning: Asking 'What would the agent have done if state X were different?' This analysis ensures that extracted decision trees encode genuine decision logic rather than dataset artifacts.

An explanation format that answers 'Why action A instead of action B?' by highlighting the minimal state differences that caused the policy to diverge. When extracting a decision tree, contrastive reasoning helps identify the decision boundaries between action classes. The tree's split conditions naturally encode these contrasts, making the extracted model a repository of contrastive explanations for every possible state.

A policy regularized via L1 penalty or discrete masks to use only a minimal subset of input features. Sparsity inherently explains which features are sufficient for control by forcing the agent to ignore irrelevant dimensions. When combined with decision tree extraction, a pre-trained sparse policy produces trees with fewer, more meaningful splits. This reduces tree depth and improves human interpretability without sacrificing fidelity.

A multi-agent reinforcement learning architecture that additively decomposes the joint Q-value into per-agent Q-values. This decomposition explains individual agent contributions to team rewards. In the context of decision tree extraction, VDN enables building separate interpretable trees for each agent, showing how local observations map to actions while the additive structure preserves the global coordination logic.