System Identification

These methods analyze input-output sequences directly in the time domain to fit model parameters.

• Least Squares Estimation: The most common approach, minimizing the sum of squared errors between the model's predicted output and the measured data. It provides a closed-form solution for linear models.
• Recursive Least Squares (RLS): An online variant that updates parameter estimates with each new data point, enabling real-time model adaptation.
• Instrumental Variables: A technique used to obtain consistent parameter estimates when the measurement noise is correlated with the regressors, a common issue in closed-loop identification.

These methods are computationally efficient and form the backbone for identifying linear models like ARX (AutoRegressive with eXogenous inputs).

These techniques transform time-series data into the frequency domain to identify system characteristics like bandwidth and resonance.

• Spectral Analysis: Estimates the system's Frequency Response Function (FRF) by computing the ratio of the cross-spectral density of the output and input to the auto-spectral density of the input.
• Sine Sweep Testing: A direct method where the system is excited with a sinusoidal input of slowly increasing frequency, and the amplitude and phase of the output are measured to construct the FRF.
• Advantages: Excellent for identifying dominant oscillatory modes and separating noise. They are non-parametric, meaning they don't assume a specific model order upfront.

Frequency-domain methods are pivotal in fields like vibration analysis and classical control design.

A powerful, general framework that identifies models by minimizing a cost function based on prediction errors.

• Core Principle: A model parameterized by θ is used to predict the next output. The parameters are optimized to minimize the difference between these predictions and the actual observed values.
• Model Families: PEM can estimate a wide range of standard model structures, including:
  • ARMAX: AutoRegressive Moving Average with eXogenous inputs (handles process noise).
  • OE: Output-Error models (noise model is unity).
  • BJ: Box-Jenkins models (separate dynamics and noise models).
• Optimization: Typically solved using iterative numerical search algorithms like Gauss-Newton. PEM provides asymptotically efficient and consistent estimates under general conditions.

A state-space oriented technique that directly estimates state-space models without requiring nonlinear optimization.

• Process: It operates on block Hankel matrices constructed from input-output data. Using linear algebra tools like QR decomposition and Singular Value Decomposition (SVD), it extracts the system's observability matrix and state sequence.
• Key Algorithms: NASID (Numerical algorithms for Subspace State Space System IDentification) and MOESP (Multivariable Output-Error State sPace).
• Advantages:
  • Naturally handles multi-input, multi-output (MIMO) systems.
  • Provides a balanced state-space realization.
  • Avoids local minima issues common in PEM for high-order systems.

This method is favored for identifying complex, multivariable systems in industrial process control.

A set of advanced techniques for modeling systems where the superposition principle does not hold.

• Block-Oriented Models: Combine linear dynamic blocks with static nonlinearities (e.g., Hammerstein and Wiener models).
• Neural Network Models: Use Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, or Neural Ordinary Differential Equations (Neural ODEs) as universal function approximators for the dynamics.
• NARX Models: Nonlinear AutoRegressive with eXogenous inputs models, often implemented with basis function expansions or neural networks.
• Volterra Series: A functional power series representation that generalizes the convolution integral for linear systems, though it suffers from the curse of dimensionality.

These methods are essential for robotics, biomedical systems, and chemical processes where linear approximations fail.

A probabilistic framework that treats unknown model parameters as random variables with prior distributions, which are updated with data to form a posterior distribution.

• Core Output: Instead of a single "best" model, it provides a full posterior probability distribution over model parameters and structures, quantifying uncertainty.
• Techniques:
  • Maximum a Posteriori (MAP) Estimation: A point estimate that maximizes the posterior, incorporating prior knowledge.
  • Markov Chain Monte Carlo (MCMC): Used to sample from complex posterior distributions when closed-form solutions are unavailable.
• Applications: Critical for robust control and safe reinforcement learning, where understanding model uncertainty is necessary for risk-aware planning. It formally bridges system identification with Gaussian Processes and Bayesian Neural Networks.

A transition model, also known as a dynamics model, is a learned function that predicts the next state of an environment given the current state and an action. It is the core predictive component in system identification for model-based reinforcement learning (MBRL).

• Core Function: Maps (state, action) -> next_state.
• Types: Can be deterministic (single prediction) or stochastic (predicts a distribution).
• Role in MBRL: Enables agents to simulate or 'imagine' the consequences of actions without interacting with the real environment, which is critical for planning and improving sample efficiency.

A world model is an agent's internal, learned representation that simulates an environment's dynamics. It is a broader concept than a simple transition model, often incorporating predictions of future observations, rewards, and task completion.

• Key Feature: Provides a compressed, abstract representation of the environment for efficient long-horizon planning.
• Architectures: Often implemented as Recurrent State-Space Models (RSSM) or latent dynamics models.
• Application: Algorithms like Dreamer train policies entirely within the world model via 'latent imagination,' decoupling policy learning from expensive real-world interaction.

Uncertainty quantification is the process of estimating the confidence or error bounds of a learned model's predictions. In system identification, it is critical for robust planning and safe exploration.

• Two Main Types: Epistemic uncertainty (model uncertainty due to lack of data) and aleatoric uncertainty (inherent environmental stochasticity).
• Estimation Methods: Common techniques include Bayesian Neural Networks (BNNs), probabilistic ensembles, and dropout-based approximations.
• Practical Use: Agents can use uncertainty estimates for pessimistic exploration (avoiding uncertain states) or to trigger data collection where the model is least confident.

Model Predictive Control (MPC) is an online planning and control strategy that uses a learned dynamics model. At each time step, it plans an optimal sequence of actions over a finite horizon, executes only the first action, and then replans from the new state.

• Core Loop: Plan → Execute first action → Observe new state → Replan.
• Advantage: Naturally compensates for model inaccuracies and environmental disturbances through frequent re-planning.
• Optimization Backend: Often uses random shooting, cross-entropy method (CEM), or gradient-based trajectory optimization algorithms like iLQR.

Sample efficiency measures the number of interactions an agent requires with the real environment to learn a high-performing policy. It is the primary motivation for using model-based reinforcement learning over model-free methods.

• MBRL Advantage: By learning a model, an agent can generate vast amounts of simulated experience (imagined rollouts) from a limited set of real interactions.
• Trade-off: Increased algorithmic complexity and risk of compounding error from an imperfect model.
• Benchmark: Highly sample-efficient algorithms are essential for real-world applications like robotics, where physical interaction is slow, costly, or risky.

Model-based offline reinforcement learning is a paradigm where an agent learns solely from a fixed, pre-collected dataset without any online interaction. The core idea is to learn a dynamics model from this static data and then use it to train a policy.

• Key Challenge: Avoiding extrapolation error, where the policy queries the model on out-of-distribution state-action pairs.
• Solution Strategies: Employ pessimistic exploration or uncertainty-aware constraints to keep the policy within the support of the offline data.
• Use Case: Ideal for leveraging historical operational data (e.g., from industrial processes, clinical records) to train safe, high-performing agents.