Rule Extraction

The core mechanism treats the trained neural network as an oracle teacher and queries it exhaustively to construct a student rule set. This involves systematically probing the model's decision boundary to identify hyper-rectangles in the input space where the model's classification is both consistent and describable by a conjunctive rule. Algorithms like TREPAN use a best-first search to grow a decision tree that mimics the network, generating splits based on the model's actual functional sensitivity rather than statistical impurity measures.

The quality of extracted rules is measured along two often-competing axes:

• Fidelity: The percentage of test inputs where the rule set's prediction matches the original neural network. High fidelity ensures the explanation is faithful to the model's actual logic.
• Comprehensibility: The syntactic complexity of the rule set, typically measured by the total number of rules and the average number of antecedents per rule. A compact set of 10-20 short rules is considered highly comprehensible for human audit.
• Accuracy: The rule set's performance against the ground-truth labels, which may differ from fidelity if the original model has learned spurious correlations.

Rule extraction algorithms fall into two distinct architectural categories:

• Decompositional: Analyzes the internal weights and activation patterns of individual neurons. It extracts rules at the level of a single hidden unit and then aggregates these local rules across the network's layers to form a global description. This approach is architecture-specific and struggles with non-monotonic activation functions.
• Pedagogical: Treats the network as a complete black box. It learns the extracted rules solely by observing the mapping between inputs and the network's final outputs. This is architecture-agnostic and can be applied to any differentiable model, but may require exponentially more queries to capture the full decision boundary.

The extracted knowledge is typically represented in one of three symbolic forms:

• M-of-N Rules: A rule is satisfied if at least M of its N specified conditions are true. This compactly represents the disjunctive logic often learned in a hidden layer's weight matrix.
• Oblique Decision Trees: Unlike standard axis-parallel splits, these rules test a linear combination of features against a threshold (e.g., 0.7*X1 - 0.3*X2 > 0.5). This directly mirrors the hyperplane computation of a single perceptron.
• DNF/CNF Logic: Extracting rules in Disjunctive Normal Form (OR of ANDs) or Conjunctive Normal Form (AND of ORs) provides a canonical, mathematically rigorous logical expression that can be formally verified using SAT solvers.

Neural networks operate on continuous real-valued inputs, but symbolic rules require discrete, logical predicates. A critical pre-processing or in-loop step is dynamic discretization:

• The algorithm identifies activation boundaries where the network's output changes sharply.
• It splits continuous features like temperature into meaningful intervals such as temperature > 100°C.
• Advanced methods like SUBSET use the network's own weight gradients to determine optimal split points, ensuring the discretization respects the model's learned function rather than arbitrary statistical binning.

The initial rule set extracted is often overly complex and overfit to the teacher's noise. A subsequent pruning phase is essential:

• Antecedent Pruning: Removes individual conditions from a rule if their removal does not significantly decrease the rule's fidelity on a validation set.
• Rule Pruning: Eliminates entire rules that cover only a few instances or are logically subsumed by a more general rule.
• Statistical Validation: Each extracted rule is tested against a held-out query set from the network to ensure it represents a genuine, stable region of the decision boundary and not an artifact of the querying process.

The classical approach where a global surrogate model—typically a decision tree or rule list—is trained on input-output pairs sampled from a trained neural network. The teacher is treated as an oracle, and the student learns to mimic its decision boundary using only the generated examples.

• Trepan algorithm: Uses recursive partitioning with oracle queries
• DecText: Extracts decision trees from neural nets with high fidelity
• G-REX: Employs genetic programming to evolve rule sets

Pedagogical methods are model-agnostic and treat the network as a black box, querying it to build a transparent approximation.

A white-box technique that analyzes the internal weights and activations of a trained neural network to derive rules at the level of individual hidden units. Each neuron's activation threshold is translated into a logical proposition.

• Subset algorithm: Extracts rules by searching for weight combinations
• MofN extraction: Identifies rules where M of N antecedents must be true
• KT algorithm: Generates rules directly from pruned network weights

This approach produces rules with higher fidelity to the original network but is architecture-dependent, requiring access to internal parameters.

A training-time technique that adds a penalty term to the loss function encouraging the neural network to learn decision boundaries that are inherently simple and rule-friendly. Rather than extracting rules post-hoc, the model is coerced into behaving like a logical system.

• L1 regularization: Promotes sparsity in weights, simplifying rule extraction
• Binary activation penalties: Encourage hidden units to saturate near 0 or 1
• Decision tree distillation loss: Penalizes divergence from a tree surrogate during training

The result is a network whose decision surface is piecewise linear and easily approximated by a compact set of if-then conditions.

A hybrid architecture that integrates differentiable logic directly into the neural network, allowing rules to be learned end-to-end via gradient descent. Instead of extracting rules from a trained black box, the model itself is a symbolic reasoner.

• Logic Tensor Networks: Ground first-order logic in neural computations
• Neural Theorem Provers: Learn to construct proofs using differentiable unification
• DeepProbLog: Combines probabilistic logic programming with neural predicates

Rules emerge as first-class citizens of the learning process, not as post-hoc approximations. This yields rules that are both faithful and executable.

Quantitative measures that assess how faithfully extracted rules replicate the original model's behavior. High fidelity is critical for regulatory compliance and auditability.

• Fidelity: Percentage of test instances where rule predictions match the teacher
• Rule set complexity: Number of rules and average antecedents per rule
• Coverage: Proportion of input space explicitly handled by the rule set
• Default rule accuracy: Performance of the fallback rule on uncovered instances

A rule set with 99% fidelity but 500 rules may be less useful than one with 95% fidelity and 12 rules. The interpretability-accuracy tradeoff governs extraction quality.

Post-extraction optimization that reduces rule set size while preserving predictive accuracy. Extracted rules often contain redundant antecedents or overly specific conditions that can be eliminated.

• Antecedent pruning: Removes individual conditions that don't affect classification
• Rule ordering optimization: Reorders rules to minimize evaluation cost
• Default rule optimization: Tunes the catch-all rule for uncovered cases
• Conflict resolution: Handles overlapping rules with contradictory predictions

Simplification transforms a raw extraction into a production-ready decision policy suitable for human review and deployment in high-stakes environments.