Network Dissection

The core mechanism evaluates each convolutional unit against a broad set of visual concepts from a densely labeled dataset like Broden. For every unit, the method computes an Intersection over Union (IoU) score between the unit's thresholded activation map and the binary segmentation mask for each concept. A unit is labeled a detector for a concept if the IoU exceeds a threshold (typically 0.04), indicating that the unit's activation pattern consistently and specifically aligns with that semantic category across many images.

The framework defines a network's overall interpretability as the fraction of its units that are concept detectors. This provides a single, comparable metric across architectures and training regimes. Key findings include:

• Higher-level layers contain more semantically meaningful detectors than lower-level layers.
• Networks trained on scene classification (Places365) develop more object and part detectors than those trained on object classification (ImageNet).
• Batch normalization significantly increases the number of interpretable units compared to networks without it.

Broden (BROadly and DEnsely annotated Networks) is a unified dataset created specifically for Network Dissection. It combines images from ADE20K, Pascal VOC, Pascal Context, OpenSurfaces, and Describable Textures Database (DTD). Each image is densely annotated with pixel-level masks for multiple concept types:

• Objects: e.g., car, person, table
• Parts: e.g., car wheel, person's leg, table leg
• Scenes: e.g., kitchen, highway, forest
• Textures: e.g., striped, checkered, metallic
• Colors: e.g., red, blue, green

Network Dissection validates its unit-concept assignments through causal intervention. By directly manipulating the activations of identified concept detectors, researchers can observe predictable changes in the network's output. Two primary interventions are used:

• Ablation: Silencing a unit (setting its activation to zero) causes the network to selectively fail on images containing that concept.
• Activation: Forcing a unit to fire on unrelated inputs biases the network's classification toward the unit's associated concept. This moves beyond correlation to establish a functional, causal role for individual units.

The framework extends beyond discriminative CNNs to Generative Adversarial Networks (GANs). In GAN dissection, the goal is to identify which units in the generator's intermediate layers are responsible for producing specific visual concepts in the output image. By identifying units that control trees, windows, or sky, practitioners can:

• Edit generated images by turning specific units on or off.
• Remove artifacts by ablating units correlated with unrealistic textures.
• Understand failure modes by seeing which real-world concepts a GAN fails to represent internally.

Unlike post-hoc attribution methods such as Grad-CAM or Integrated Gradients, which explain a single prediction, Network Dissection provides a global, neuron-level explanation of the entire model's learned representations. Key distinctions:

• Saliency maps answer: 'Which input pixels mattered for this specific prediction?'
• Network Dissection answers: 'What semantic concept, if any, does this specific neuron detect across all inputs?' This makes it a foundational tool in mechanistic interpretability, aiming to reverse-engineer the model's internal feature space rather than just its output behavior.

A feature visualization technique that synthesizes the optimal input pattern to maximally excite a specific neuron, channel, or layer. Unlike Network Dissection—which aligns existing activations with human labels—Activation Maximization performs gradient ascent in the input space to generate a prototype image. This reveals the preferred stimulus of a unit, often producing interpretable, dream-like visualizations of textures, patterns, or object parts that complement the quantitative alignment scores from dissection.

A framework for testing model sensitivity to high-level, human-understandable concepts using directional derivatives. TCAV measures how much a concept (e.g., 'striped') influences a prediction by training a linear classifier to separate concept examples from random counterexamples in the activation space. This provides a quantitative sensitivity score that generalizes Network Dissection's unit-level analysis to arbitrary abstract concepts without requiring dense pixel-level annotations.

A localization technique that produces coarse heatmaps highlighting image regions most influential for a prediction. Grad-CAM uses the gradients of a target concept flowing into the final convolutional layer to weight activation maps. While Network Dissection identifies what individual units detect, Grad-CAM answers where the model is looking for a specific class, providing spatial explanations that complement the semantic labeling of hidden units.

The discipline of reverse-engineering the internal computations and learned algorithms encoded in a neural network's weights. Network Dissection operates at the unit level, labeling individual neurons with semantic concepts. Mechanistic interpretability goes further by identifying circuits—connected subgraphs of neurons that implement specific functions. This includes techniques like causal scrubbing to validate hypothesized circuits by resampling activations from corrupted datasets.

Quantitative evaluation criteria that measure how accurately an attribution map reflects true feature importance. Key protocols include:

• Deletion Metric: Remove pixels from most to least important and measure prediction decay
• Insertion Metric: Add pixels from most to least important into a blurred baseline and measure probability increase
• ROAR (RemOve And Retrain): Iteratively retrain a model after removing top-ranked features to measure degradation These metrics validate whether Network Dissection's unit-concept alignments genuinely capture causally relevant representations.

A decomposition technique that redistributes a model's prediction score backward through the network using local conservation rules. LRP assigns relevance scores to individual input features by propagating the output decision through each layer while preserving the total relevance. This provides a per-prediction explanation that complements Network Dissection's global, unit-level semantic analysis by showing which input features drove a specific decision.