Centered Kernel Alignment (CKA)

CKA is invariant to orthogonal transformations and isotropic scaling of representations. If you rotate or uniformly rescale the hidden activations of a layer, the CKA score remains unchanged. This is a critical property because many neural network operations—such as weight matrix rotations during training—produce representations that are functionally identical but numerically different. Earlier metrics like CCA (Canonical Correlation Analysis) are sensitive to these arbitrary rotations, leading to misleadingly low similarity scores. CKA correctly identifies that two layers learning the same features up to a rotation are essentially equivalent.

CKA is not dominated by dead or saturated neurons. In deep networks, many units may output near-constant values (e.g., ReLU neurons stuck at zero). Metrics based on dot-product similarity or standard CCA can be overwhelmed by these inactive dimensions, producing artificially high similarity scores because constant columns are trivially correlated. CKA's use of centered kernels removes the mean from each feature, effectively ignoring neurons that do not vary across examples. This ensures the similarity score reflects only the informative, varying structure of the representations.

CKA reliably compares layers from heterogeneous network architectures. You can compute the similarity between a convolutional layer in a ResNet and a transformer block in a ViT, or between layers of different widths. This cross-architectural comparison is essential for understanding whether two independently trained models converge to analogous internal computations. The kernel-based formulation operates on the Gram matrix of activations, which abstracts away the specific dimensionality of each layer, allowing meaningful comparisons even when the number of neurons differs by orders of magnitude.

CKA reveals that independently trained networks with different random initializations often converge to similar representations in their intermediate layers. This phenomenon, termed convergent learning, is a fundamental empirical finding in deep learning. By plotting CKA heatmaps between corresponding layers of separately trained models, researchers consistently observe a block-diagonal structure where early layers (low-level features) and late layers (task-specific features) show high similarity, while middle layers exhibit more variability. This property validates CKA as a tool for studying the universality of learned features.

CKA scores are not artificially inflated by increasing the width of a layer. Many similarity indices, including SVCCA (Singular Vector CCA), exhibit a bias where wider layers (more neurons) produce higher similarity scores simply due to the increased dimensionality of the space. CKA's normalization by the Hilbert-Schmidt norm of the centered Gram matrices corrects for this dimensionality bias. This allows for fair comparisons between narrow bottleneck layers and wide expansion layers within the same network, or between a compact student model and a large teacher model in knowledge distillation studies.

CKA produces a similarity matrix that maps the entire representational hierarchy between two networks. By computing CKA between every pair of layers—one from network A and one from network B—you generate a heatmap that reveals one-to-one, one-to-many, or many-to-one correspondences. A strong diagonal indicates a direct layer-to-layer match. Off-diagonal hotspots suggest that a single layer in one network may perform the computation spread across multiple layers in another. This mapping is invaluable for model stitching, layer pruning, and understanding the effective depth of a network's computation.

CKA quantifies the similarity between hidden layer activations of two different neural networks trained on the same data. Unlike correlation-based methods, CKA is invariant to orthogonal transformations and isotropic scaling, making it a reliable metric for determining whether independently trained diagnostic models have converged to functionally equivalent internal representations. This is critical for verifying reproducibility across research sites.

• Identifies layers where models diverge in feature extraction
• Robust to permutation symmetries in neural network weights
• Provides a scalar similarity score between 0 (dissimilar) and 1 (identical)

CKA enables direct comparison of representations between networks with different architectures—for example, comparing a ResNet-50 to a Vision Transformer trained on the same histopathology dataset. By computing the CKA matrix across all pairs of layers, researchers can identify which layers in disparate architectures learn analogous features, facilitating architecture selection for FDA submission without relying on task-specific metrics alone.

• Cross-architecture comparability (CNNs vs. Transformers)
• Identifies functional correspondences between dissimilar layer types
• Supports architecture-agnostic model validation

By computing CKA between model checkpoints at successive training epochs, developers can visualize how representational similarity evolves over time. A plateau in CKA values across later epochs indicates representational convergence, signaling that further training yields diminishing returns. This is particularly valuable for medical imaging models where overtraining can lead to learning spurious correlations tied to scanner artifacts rather than genuine pathology.

• Detects convergence of internal representations
• Identifies optimal early stopping points
• Reveals phases of rapid feature learning vs. fine-tuning

CKA reveals a consistent pattern in diagnostic neural networks: wider layers (more channels or hidden units) tend to learn more similar representations across random initializations than deeper layers. This insight guides architectural decisions for medical AI—when reproducibility is paramount, wider architectures may be preferred because their learned representations are less sensitive to initialization seed, leading to more consistent biomarker identification across training runs.

• Quantifies initialization sensitivity by layer
• Informs architectural trade-offs for regulated environments
• Supports design of reproducible diagnostic pipelines

CKA can evaluate whether a pre-trained model's representations align with a target medical domain before committing to full fine-tuning. By computing CKA between the source model's layers and a partially trained target model, teams can identify which layers transfer useful features and which require substantial retraining. This reduces computational waste and accelerates the development of diagnostic models for rare diseases where data is scarce.

• Pre-screens pre-trained models for domain relevance
• Identifies transferable vs. domain-specific layers
• Reduces compute costs for model selection

In federated learning scenarios across hospitals, CKA serves as an early warning system for representational collapse—a failure mode where models trained on heterogeneous data distributions converge to trivial, non-discriminative representations. By monitoring CKA between local and global model representations, system architects can detect when a federated round degrades feature quality before it impacts downstream diagnostic accuracy.

• Detects representational degradation in federated rounds
• Monitors cross-site model alignment
• Prevents silent failure modes in distributed training

The choice of kernel function fundamentally changes what CKA measures:

• Linear CKA: Uses a dot product kernel, measuring similarity up to an orthogonal rotation. Equivalent to comparing the covariance structures of representations. Computationally efficient for large networks.
• RBF (Gaussian) CKA: Uses a radial basis function kernel, capturing non-linear similarities between representations. More sensitive to local neighborhood structure but computationally heavier.

In practice, linear CKA often suffices for identifying corresponding layers, while RBF CKA is preferred when representations may differ by complex non-linear transformations.

A classical shape analysis metric adapted for comparing neural representations by finding the optimal orthogonal alignment between two sets of activations:

• Procedure: Finds the orthogonal transformation that minimizes the Frobenius norm between two representation matrices.
• Interpretation: A small Procrustes distance indicates that one representation can be rotated to closely match another.
• Relationship to CKA: Procrustes distance is sensitive to the specific dimensionality and scaling of representations. CKA normalizes these factors out, focusing purely on relative similarity structure rather than absolute distances.
• Use case: Preferred when you care about exact geometric alignment; CKA is preferred when you care about whether networks encode the same relational information.

The foundational data structure underlying CKA computation:

• Definition: An RSM is a matrix where entry (i, j) contains the similarity between the network's representation of input i and input j.
• Construction: For a layer with activations X, compute the Gram matrix K = XXᵀ (for linear kernel) or apply a kernel function.
• CKA's role: CKA is essentially a normalized similarity of similarities — it compares two RSMs to determine whether two networks organize their representational space in the same way.
• Visualization: RSMs are often displayed as heatmaps, revealing block-diagonal structures that indicate the network has learned to cluster similar inputs together.

The primary practical application of CKA in deep learning research:

• Goal: Identify which layers in different networks compute functionally similar representations, even when architectures differ.
• Typical findings:
  • Early layers in vision networks learn similar edge and texture detectors regardless of architecture.
  • Later layers diverge more significantly based on training objective and dataset.
  • Wide, shallow networks often converge to similar representations as deep, narrow ones at corresponding computational depths.
• Practical use: CKA heatmaps between all layer pairs of two models reveal a block-diagonal pattern when networks learn hierarchically aligned representations.

Beyond academic comparison, CKA serves as a practical tool for model development:

• Architecture search: Quantify how similar a candidate architecture's internal representations are to a proven baseline, helping identify promising designs without full training.
• Pruning validation: After pruning a network, compute CKA between pruned and original model layers to verify that representational structure is preserved despite parameter reduction.
• Transfer learning assessment: Measure how much a fine-tuned model's representations diverge from the pre-trained backbone, identifying which layers are most task-specific.
• Convergence monitoring: Track CKA between checkpoints during training to detect when representations stabilize, indicating that further training yields diminishing returns.