Network Dissection

The core mechanism of Network Dissection is the Intersection over Union (IoU) score, which quantifies how well a single convolutional unit's activation map aligns with a specific human-labeled visual concept (e.g., 'tree', 'window', 'texture'). A unit is considered a detector for a concept if its activation map, when thresholded, achieves an IoU exceeding a predefined value (typically 0.04) with the ground-truth segmentation of that concept. This process systematically evaluates every unit against a broad dictionary of concepts from densely annotated datasets like Broden.

Network Dissection relies on the Broadly and Densely Labeled Dataset (Broden), a composite dataset unifying several existing computer vision benchmarks. It combines:

• Object segmentation from ADE20k and Pascal-Context
• Scene classification from Places365
• Texture classification from the Describable Textures Dataset (DTD)
• Color classification from a custom color name dataset This multi-faceted labeling allows researchers to probe whether units respond to low-level features (color, texture), mid-level parts (object components), or high-level semantics (object classes, scenes).

The framework defines a network's overall interpretability as the fraction of its convolutional units that emerge as unique, consistent concept detectors. Key metrics include:

• Number of unique detectors: Units that align with at least one concept
• Concept coverage: The diversity of concepts the network learns to detect
• Layer-wise analysis: Lower layers typically detect textures and colors, while deeper layers specialize in objects and parts This quantification allows for direct, objective comparison of interpretability across different architectures (e.g., AlexNet vs. ResNet vs. GoogLeNet).

Network Dissection reveals a clear semantic hierarchy in CNNs. By applying the framework layer by layer, researchers observe:

• Early layers (conv1-conv2): Units are predominantly texture and color detectors. Interpretability is low.
• Middle layers (conv3-conv4): Detectors for simple material and part concepts (e.g., 'fabric', 'wheel') begin to emerge.
• Deep layers (conv5-fc): Units become highly selective for complex objects (e.g., 'dog', 'car') and scene-level concepts. This progression validates the intuition that CNNs learn compositional representations, moving from low-level primitives to high-level abstractions.

The framework serves as a diagnostic tool for comparing architectures. Key findings from original research include:

• Batch normalization significantly increases the number of interpretable units.
• Deeper networks (ResNet-152) do not necessarily produce more unique concept detectors than shallower ones (ResNet-50), but their detectors are more selective.
• Fully connected layers can also be dissected, revealing detectors for scene-level concepts.
• Dropout reduces the emergence of interpretable units, suggesting a trade-off between regularization and the formation of disentangled, human-aligned concepts.

Network Dissection has key limitations that must be understood for proper application:

• Dictionary constraint: A unit can only be matched to concepts present in the Broden dataset. A unit detecting an unlabeled concept will be classified as non-interpretable.
• Threshold sensitivity: The binary classification of a unit as a 'detector' depends on the IoU threshold, which is a somewhat arbitrary hyperparameter.
• Multi-concept units: A single unit may fire for multiple semantically related concepts (polysemy). The framework reports the top-matching concept but does not fully capture the distributed nature of neural representations.
• Correlation vs. causation: Alignment does not prove the unit is causally responsible for the network's classification decision.

Quantitative measures that assess how accurately an explanation method reflects the true reasoning process of the underlying machine learning model. A faithful explanation changes predictably when the model's internal logic is perturbed.

• Fidelity: Does the explanation predict the model's output on perturbed inputs?
• Infidelity: Measures expected error between explanation and model response to significant perturbations
• Sensitivity: Evaluates how explanations change under minimal input perturbations that do not alter the prediction

A property of machine learning models that are considered understandable by humans due to their simple structure. Unlike post-hoc methods applied to black-box models, intrinsically interpretable models—such as generalized additive models (GAMs), decision trees, and sparse linear models—allow direct inspection of their parameters.

• No need for separate explanation models that may be unfaithful
• Preferred in high-stakes regulatory contexts like FDA submissions
• Trade-off: often lower raw predictive performance than deep networks