Configuration Management (Hydra, YAML Config)

Configuration files are human-readable, structured documents (commonly YAML or JSON) that store all adjustable parameters for a machine learning experiment. This includes:

• Hyperparameters (e.g., learning rate, batch size, layer count)
• Data paths and preprocessing settings
• Model architecture definitions
• Training loop controls (e.g., number of epochs)

By externalizing these settings, the core training code remains static while configurations can be swapped easily, enabling parameter sweeps and A/B testing without code changes. This is the bedrock of reproducibility, as a run can be perfectly recreated from its config file and code commit.

Hierarchical configuration organizes settings into a tree-like structure, allowing for inheritance and overrides. A base config (e.g., model/base.yaml) can define common defaults, while experiment-specific configs (e.g., experiment/bert.yaml) extend and modify only the necessary parameters.

Composability allows configs to be assembled from multiple modular files. For instance, you can have separate files for dataset.yaml, optimizer.yaml, and model.yaml, which are stitched together at runtime. This promotes modularity, reduces duplication, and allows for easy mixing and matching of components (e.g., trying the same optimizer with five different model architectures).

Environment-specific configuration separates settings that vary between development, staging, and production environments (e.g., API endpoints, database URIs, log levels). This is often managed via separate config files (config_dev.yaml, config_prod.yaml) or environment variables.

Secret management is the secure handling of sensitive data like API keys, passwords, and tokens. Best practices dictate that these are never hardcoded in config files committed to version control. Instead, they are injected at runtime via environment variables or dedicated secret management services (e.g., HashiCorp Vault, AWS Secrets Manager), linking configuration management to security posture.

Config versioning treats configuration files as first-class artifacts, storing them in version control (e.g., Git) alongside the training code. Each experiment run is linked to the exact config file snapshot that produced it.

This establishes full lineage and provenance: given any trained model, you can trace back to the precise code commit, data version, and configuration that generated it. This is non-negotiable for auditability, debugging, and regulatory compliance. Experiment tracking platforms automatically log the config state as run metadata, creating an immutable record of the experimental setup.

Config validation ensures that configuration files are syntactically correct and semantically valid before a costly training run begins. This prevents runtime failures due to typos, incorrect types, or missing required fields.

Schema enforcement tools like Pydantic or OmegaConf's structured configs allow you to define a strict schema (types, value ranges, optional/mandatory fields) for your configuration. The framework then validates the loaded config against this schema, providing immediate error feedback. This acts as a contract between the configuration and the code, catching errors early and serving as machine-readable documentation for all allowable parameters.

Configuration management enforces a strict separation between a model's algorithmic logic (the code) and its tunable parameters (the config). This principle is foundational for reproducibility and scalable experimentation.

• Code defines the model architecture, training loop, and data loading logic.
• Config files (e.g., YAML, JSON) specify hyperparameters, file paths, dataset names, and environment flags.

This separation allows the same codebase to be executed with hundreds of different configurations without modification, enabling systematic hyperparameter sweeps and A/B testing. It also ensures that every experiment run is fully documented by its configuration file, a cornerstone of experiment tracking.

YAML (YAML Ain't Markup Language) is the de facto standard for ML configuration files due to its human-readable, hierarchical structure. It supports complex nested dictionaries, lists, and basic data types, making it ideal for organizing model parameters.

Example Structure:

yamlCopy
model:
  name: "resnet50"
  pretrained: true
training:
  batch_size: 32
  learning_rate: 0.001
  optimizer: "Adam"
data:
  path: "/datasets/cifar10"

Key advantages include readability for non-engineers and native support in most programming languages via libraries like PyYAML. The hierarchical nature allows logical grouping of parameters for data, model, training, and evaluation.

OmegaConf is the underlying library that powers Hydra, providing a unified interface for handling configuration data from YAML files, CLI arguments, and environment variables. It solves common issues with plain YAML dictionaries.

Key features include:

• Variable Interpolation: Reference other config values within the file: log_dir: "${data.path}/logs".
• Runtime Merging: Seamlessly merge configurations from defaults, YAML files, and command-line inputs.
• Structured Configs: Use Python dataclasses to define a strict schema for configurations, enabling type safety and validation.

While often used through Hydra, OmegaConf can be used independently as a robust configuration management solution, ensuring configs are resolved and validated before runtime.

Configuration management is intrinsically linked to experiment tracking. All major tracking platforms (MLflow, Weights & Biases, etc.) automatically log the complete configuration used for a training run as run parameters.

• Automatic Logging: Frameworks like Hydra can be configured to log the entire resolved config (including overrides) to the tracking server.
• Run Comparison: Trackers use logged configurations to enable run comparison, allowing engineers to filter and contrast experiments based on specific hyperparameter values.
• Reproducibility: The logged config, combined with the environment snapshot and code version, provides the complete recipe needed to reproduce any past experiment.

This creates a closed loop where configuration defines the experiment, and tracking validates and records its outcome.

Effective configuration management follows established engineering patterns to prevent complexity and errors.

• Hierarchical Organization: Structure configs to mirror your code's architecture (e.g., data, model, training, evaluation).
• Environment-Specific Configs: Use separate config files or overrides for local, staging, and production environments to manage different datasets or resource limits.
• Sensitive Data Handling: Never store secrets (API keys, passwords) in version-controlled config files. Use environment variables or secret management tools, which can be referenced via interpolation (e.g., api_key: ${oc.env:API_KEY} in OmegaConf).
• Schema Validation: Use Pydantic or Structured Configs in Hydra to validate data types and required fields at startup, catching errors before a costly training run begins.
• Defaults and Overrides: Maintain a base config.yaml with sensible defaults, allowing specific experiments to override only the necessary parameters.

YAML is a human-readable data serialization language and the de facto standard for configuration files in machine learning and DevOps. Its structure uses indentation and simple punctuation, making it more intuitive than JSON or XML for complex nested settings. In ML configuration, YAML files typically define:

• Hyperparameters (learning rate, batch size, epochs).
• Model architecture specifications (layer sizes, activation functions).
• Data pipeline paths and transformation parameters.
• Training environment settings (device, distributed strategy). Frameworks like Hydra, Kubernetes, and Ansible rely on YAML for declarative configuration management.

Configuration drift occurs when the actual runtime configuration of a system diverges from its intended, version-controlled specification. In ML, this is a critical failure mode for reproducibility. Causes include:

• Manual command-line overrides that are not logged.
• Environment variable changes not captured in the experiment track.
• Implicit defaults from libraries that differ between runs.
• Secret or local config files not committed to version control. Mitigation requires rigorous practices: using structured configs, logging all overrides as run metadata, and employing configuration hashing to detect changes.

Structured Configs refer to the practice of defining configuration schemas using native language constructs like Python dataclasses, Pydantic models, or TypedDicts. This approach moves beyond unstructured YAML dictionaries to provide:

• Type safety and validation at edit time and runtime.
• IDE autocompletion and inline documentation.
• Explicit defaults and optional/required field specification.
• Inheritance and composition through object-oriented principles. Frameworks like Hydra and Pydantic Settings enable structured configs, ensuring that configuration errors are caught early and the config itself becomes a source of documentation.

Environment variables and .env files are a standard method for managing configuration that varies between deployments (development, staging, production) or contains sensitive data. They are used to externalize:

• API keys, database passwords, and other secrets.
• Service endpoints and resource locations.
• Feature flags and operational modes. In ML, tools like python-dotenv load variables from a .env file into the process environment. Best practice is to treat the .env file as secret (excluded from Git) while committing a .env.example template. Configuration management systems like Hydra can merge these variables into the main config, keeping secrets out of YAML files.