Gaussian Process (GP)

A Gaussian Process defines a distribution over functions rather than a distribution over parameters. This means the model's capacity grows with the data, avoiding the underfitting constraints of fixed-size parametric models.

• Infinite-dimensional: Theoretically capable of representing any continuous function.
• Data-driven complexity: Model flexibility adapts naturally to the size and structure of the training set.
• No explicit feature engineering: The kernel function implicitly defines a high-dimensional feature space.

The behavior of a GP is entirely governed by its covariance function (kernel), which encodes prior assumptions about the function's smoothness, periodicity, and structure.

• Radial Basis Function (RBF): Assumes infinitely differentiable, smooth functions.
• Matérn kernel: A flexible class that models rougher, less smooth functions.
• Periodic kernel: Captures repeating patterns in the data.
• Composite kernels: Kernels can be added or multiplied to build custom structures (e.g., smooth trend + periodic seasonal effect).

A GP provides a full predictive distribution for every test point, outputting both a mean prediction and a variance estimate. This variance decomposes into two distinct sources of uncertainty.

• Epistemic uncertainty: High in regions far from training data; reflects model ignorance that can be reduced with more data.
• Aleatoric uncertainty: Captured by a noise term in the likelihood; reflects inherent, irreducible data noise.
• Credible intervals: Bayesian posterior intervals provide a direct probabilistic statement about the function's value.

GPs provide a principled framework for hyperparameter optimization without requiring a held-out validation set. The marginal likelihood automatically balances data fit against model complexity.

• Occam's razor: The Bayesian evidence penalizes overly complex models that do not significantly improve data fit.
• Automatic relevance determination (ARD): Using a separate lengthscale per input feature allows the model to learn which dimensions are most relevant, effectively performing embedded feature selection.
• Gradient-based optimization: The log marginal likelihood is typically differentiable, enabling efficient optimization of kernel hyperparameters.

Standard GP inference requires inverting an N x N covariance matrix, resulting in O(N³) computational complexity and O(N²) memory, which is prohibitive for large datasets.

• Sparse approximations: Methods like inducing points (e.g., Sparse Variational GP) reduce complexity to O(M²N), where M << N.
• Structured kernels: Exploiting Kronecker or Toeplitz structure enables fast matrix-vector multiplication.
• Local experts: Partitioning the input space and training independent local GPs scales linearly with data.

A single-layer, infinitely wide neural network with a Gaussian prior on its weights converges to a GP with a specific kernel. This Neural Network Gaussian Process (NNGP) correspondence provides a theoretical bridge between deep learning and kernel methods.

• Infinite-width limit: As the number of hidden units goes to infinity, a Bayesian neural network becomes a GP.
• Neural Tangent Kernel (NTK): Describes the training dynamics of infinitely wide networks under gradient descent.
• Insight: This connection explains why overparameterized networks generalize well and provides a tool for analyzing their behavior.

Model uncertainty arising from a lack of knowledge or insufficient data. In a GP, this is captured by the predictive variance that expands far from training points. This uncertainty is reducible—collecting more observations in sparse regions will shrink the credible intervals. It reflects the model's ignorance about the underlying function, not inherent randomness in the data.

The irreducible noise inherent in the data generation process. A GP models this explicitly through the noise variance parameter in the likelihood. Even with infinite data, this uncertainty floor remains. Examples include measurement error in sensors or genuine stochasticity in the phenomenon being modeled. Distinguishing this from epistemic uncertainty is critical for risk assessment.

A sequential design strategy that uses a GP as a surrogate model for an expensive black-box function. The GP's predictive mean and variance are combined into an acquisition function (e.g., Expected Improvement, Upper Confidence Bound) that balances exploration of uncertain regions against exploitation of promising areas. Widely used for hyperparameter tuning and experimental design.

A hierarchical composition of GPs where the output of one GP becomes the input to the next. This architecture introduces non-stationarity and the ability to model complex, multi-scale functions that a single GP with a stationary kernel cannot capture. Training involves approximate variational inference due to the intractability of exact Bayesian inference in these deep probabilistic models.

Approximation methods that reduce the O(N³) computational cost of exact GP inference to O(NM²) by introducing M inducing points. Techniques include:

• Fully Independent Training Conditional (FITC)
• Variational Free Energy (VFE) These pseudo-inputs summarize the training data, making GPs tractable for datasets with hundreds of thousands of points while preserving calibrated uncertainty.