Akaike Information Criterion (AIC)

AIC estimates the relative information loss when using a model to represent the unknown true data-generating process. It is defined as:

• Formula: AIC = -2 · ln(L̂) + 2k
• L̂ is the maximized value of the likelihood function
• k is the number of estimated parameters
• Lower AIC values indicate a better model, trading off goodness-of-fit against complexity

The term 2k serves as a complexity penalty that discourages selecting models with unnecessary parameters:

• Prevents overfitting by adding a cost for each additional degree of freedom
• Unlike BIC, the penalty does not scale with sample size N
• For nested models, the difference in AIC values provides a measure of relative support
• A model with more parameters must improve the log-likelihood by at least 1 unit per extra parameter to be preferred

AIC is derived from the Kullback-Leibler (KL) divergence between the true distribution and the fitted model:

• Akaike showed that the expected log-likelihood is a biased estimator of the expected KL information
• The bias is approximately equal to k, the number of parameters
• Subtracting this bias yields an asymptotically unbiased estimator of relative KL distance
• This connects AIC directly to information theory and the principle of minimum information loss

In likelihood-based modulation classifiers, AIC is used to select the most probable modulation scheme:

• Compute the maximized log-likelihood under each candidate modulation hypothesis
• Apply the AIC formula with k equal to the number of unknown nuisance parameters (carrier phase, timing offset, etc.)
• Select the modulation with the minimum AIC value
• Particularly useful when the true number of active parameters is unknown, as in composite hypothesis testing

While both criteria balance fit and complexity, they differ fundamentally:

• AIC minimizes prediction error and is asymptotically efficient but not consistent
• BIC (Bayesian Information Criterion) uses penalty k · ln(N) and is consistent, selecting the true model with probability approaching 1 as N → ∞
• AIC tends to favor more complex models than BIC for large samples
• In modulation classification with moderate sample sizes, AIC often provides better classification accuracy because real signals rarely match idealized models exactly

When applying AIC in real-world signal classification:

• Requires accurate computation of the log-likelihood function under each hypothesis
• Sensitive to model misspecification—if no candidate model is close to the truth, AIC may still select the least bad option
• For small sample sizes, consider the corrected AIC (AICc): AICc = AIC + 2k(k+1)/(N-k-1)
• In non-coherent classification scenarios, the number of nuisance parameters k increases, strengthening the penalty's influence on model selection

A model selection criterion derived from Bayesian probability theory. Unlike AIC, the BIC penalty term scales with the logarithm of the sample size (k·ln(n)), making it asymptotically consistent—it will select the true model with probability approaching 1 as n → ∞.

• Penalty: k·ln(n) vs AIC's 2k
• Philosophy: Seeks the true model (assuming it exists in the candidate set)
• Behavior: Strongly favors simpler models on large datasets
• Derivation: Laplace approximation to the marginal likelihood

A principle equating model selection with data compression. The best model minimizes the total description length: the bits needed to encode the model parameters plus the bits needed to encode the data given the model.

• Two-part code: L(model) + L(data|model)
• Connection: Asymptotically equivalent to BIC under certain priors
• Stochastic complexity: Rissanen's refinement using normalized maximum likelihood
• Application: Used in signal detection for estimating the number of sources in array processing

The asymmetric information-theoretic measure that AIC was designed to estimate. KL divergence quantifies how much information is lost when approximating the true data-generating distribution f with a candidate model g.

• Definition: D_KL(f||g) = ∫ f(x) log(f(x)/g(x)) dx
• AIC's goal: Unbiased estimator of the expected relative KL divergence
• Non-symmetric: D_KL(f||g) ≠ D_KL(g||f)
• Cross-entropy connection: Minimizing KL is equivalent to minimizing cross-entropy

A fundamental matrix quantifying the curvature of the log-likelihood function. The FIM determines the Cramér-Rao lower bound on estimator variance and plays a role in understanding AIC's theoretical properties.

• Definition: I(θ) = -E[∂²/∂θ² log L(θ)]
• Role in AIC: The penalty term 2k emerges from the trace of I⁻¹J where J is the outer product gradient
• Takeuchi Information Criterion (TIC): A generalization of AIC using the FIM for misspecified models
• Singularity: When FIM is singular, standard AIC asymptotics break down

A robust generalization of AIC that does not assume the candidate model contains the true distribution. TIC replaces the simple 2k penalty with a trace-based correction involving the Fisher Information Matrix and the outer product of the score function.

• Penalty: tr(I(θ)⁻¹ J(θ)) where J is the covariance of the score
• AIC as special case: When the model is correctly specified, tr(I⁻¹J) = k, recovering AIC
• Misspecification: TIC remains valid when models are wrong
• Practical limitation: Requires estimating I and J, introducing additional variance

A resampling-based alternative to information criteria that directly estimates out-of-sample prediction error by partitioning data into training and validation sets.

• k-fold CV: Splits data into k partitions, trains on k-1, validates on the held-out fold
• Leave-one-out CV (LOOCV): n-fold cross-validation; asymptotically equivalent to AIC for linear models
• Advantage over AIC: No parametric assumptions about the model family
• Cost: Computationally expensive compared to single-fit criteria like AIC