Knowledge Distillation CSI

The fundamental two-network paradigm where a high-capacity teacher model (e.g., a deep Transformer CSI) generates soft targets for a compact student model (e.g., a lightweight convolutional network). The student is trained to minimize the divergence between its own predictions and the teacher's output distribution, capturing inter-class correlations that one-hot labels miss. This transfers the teacher's generalization capability without transferring its computational cost.

Beyond matching final outputs, the student learns to replicate the teacher's intermediate feature representations. A regression loss aligns the student's hidden activations with the teacher's embedding space:

• Attention transfer: Matching spatial attention maps from self-attention layers
• Hint-based learning: Using a regressor to map student features to teacher dimensions This enforces structural similarity in how both networks internally represent the MIMO channel geometry.

Two operational modes for CSI model compression:

• Offline distillation: A pre-trained, frozen teacher guides student training. Simple but teacher errors propagate.
• Online distillation: Teacher and student train simultaneously with mutual knowledge exchange. The student benefits from the teacher's evolving representations while providing regularization feedback. Online methods often yield superior NMSE performance in non-stationary channel environments.

A joint optimization framework that combines knowledge distillation with post-training quantization. The student is trained to be robust to 8-bit or 4-bit weight precision while still matching the full-precision teacher's CSI reconstruction quality. This produces models deployable directly on neural processing units (NPUs) within distributed units (DUs) without floating-point hardware. Typical NMSE degradation is held below 0.5 dB compared to the full-precision teacher.

A single student model is trained to replicate a multi-task teacher that simultaneously predicts:

• Channel Impulse Response (CIR) for equalization
• Precoding Matrix Indicator (PMI) for beam selection
• Channel Quality Indicator (CQI) for link adaptation The student learns shared representations across these correlated tasks, achieving higher per-task accuracy than independently trained compact models through cross-task knowledge transfer.

The foundational two-network framework where a high-capacity teacher model generates soft targets for a compact student model. In CSI prediction, the teacher is often a deep Transformer CSI or CsiNet variant trained on high-resolution channel matrices. The student mimics not just the hard prediction but the teacher's output distribution, capturing inter-class relationships and uncertainty. The Kullback-Leibler divergence between softened logits drives the distillation loss.

A mechanism that smooths the teacher's output probability distribution using a temperature parameter (T). Higher T values reveal the dark knowledge of the teacher—relative similarities between incorrect classes that a hard label discards. For regression-based CSI tasks like Channel Impulse Response (CIR) Prediction, temperature scaling is adapted to soften the continuous output distribution, providing richer gradient signals to the student during backpropagation.

The objective function minimizing divergence between teacher and student outputs. Common formulations include:

• Kullback-Leibler Divergence: For classification-style CSI tasks like PMI selection.
• Mean Squared Error: Applied directly to the teacher's predicted channel matrix vs. the student's reconstruction.
• Hybrid Loss: Combines distillation loss with a task-specific NMSE loss against ground-truth CSI, preventing the student from inheriting teacher biases.

An extension beyond output mimicry where the student learns to replicate the teacher's intermediate feature representations. In CsiNet architectures, this involves matching the compressed latent vectors of the encoder. The student's hidden layer activations are regressed against the teacher's using an L2 loss, transferring the teacher's learned manifold of channel sparsity. This is critical for preserving CSI Compression fidelity in bandwidth-limited feedback links.

A dynamic variant where the teacher model continuously updates on fresh CSI-RS measurements while the student distills knowledge incrementally. This combats Channel Aging in high-mobility scenarios by ensuring the lightweight student deployed on the O-RAN Intelligent Controller adapts to changing Doppler Shift Estimation patterns without full retraining. The teacher serves as a moving target, regularizing the student against temporal distribution drift.

A technique where knowledge is transferred from an ensemble of diverse teacher models to a single student. Teachers may specialize in different Spatial Channel Model (SCM) scenarios—urban micro, rural macro, indoor hotspot. The student learns an averaged, robust posterior that generalizes across environments. This mitigates Pilot Contamination effects by distilling a consensus from teachers trained on varied interference patterns.