Infrastructure as Code (IaC)

Terraform is an open-source, declarative IaC tool from HashiCorp that uses its own HashiCorp Configuration Language (HCL) to define and provision infrastructure across multiple cloud providers (AWS, Azure, GCP) and on-premises systems. Its core concepts include:

• State Management: A state file tracks the real-world resources, enabling Terraform to determine what changes are needed.
• Providers: Plugins that interact with APIs of cloud platforms and services.
• Modules: Reusable, encapsulated configurations for common infrastructure patterns. For LLM serving, Terraform is used to define GPU-enabled compute instances, load balancers, and autoscaling groups in a reproducible manner.

AWS CloudFormation is Amazon's native, declarative IaC service for modeling and provisioning AWS resources using JSON or YAML templates. It is tightly integrated with the AWS ecosystem. Key features include:

• Stacks: A collection of AWS resources created and managed as a single unit.
• Change Sets: A preview of the changes a template will make before execution.
• Drift Detection: Identifies configuration changes made outside of CloudFormation. For deploying LLM endpoints, CloudFormation templates can define Amazon SageMaker endpoints, EC2 instances with NVIDIA GPUs, and the necessary IAM roles and VPC configurations.

Pulumi is an IaC platform that allows developers to define infrastructure using general-purpose programming languages like Python, TypeScript, Go, and C#. This enables the use of loops, functions, and classes to create dynamic, reusable infrastructure code. Its advantages for ML/LLM teams include:

• Familiar Tooling: Leverage existing IDEs, testing frameworks, and package managers.
• Component Resources: Create high-level abstractions for complex patterns (e.g., a ModelServingCluster).
• State Management: Offers a managed service or self-hosted backend. Pulumi is particularly effective for teams building custom LLM serving platforms that require complex logic and integration with existing application code.

Crossplane is an open-source Kubernetes-native IaC tool that extends the Kubernetes API to manage both cloud services and on-prem infrastructure. It treats infrastructure as Kubernetes Custom Resources. Core concepts are:

• Providers: Packages that enable Crossplane to manage external systems (AWS, Azure, SQL databases).
• Managed Resources: The declarative representation of a cloud resource (e.g., GKECluster).
• Compositions: Bundles of managed resources into a higher-level, custom API (e.g., VectorDatabaseCluster). For LLM Ops, Crossplane allows platform teams to offer self-service, compliant infrastructure (like GPU node pools) through a familiar Kubernetes interface, governed by RBAC and policies.

Ansible is an open-source, agentless configuration management and automation tool that uses YAML playbooks written in a declarative style. It operates primarily in a procedural/mutable manner, making changes to bring existing systems into a desired state. Key components:

• Playbooks: Define a set of tasks and roles to execute on remote hosts.
• Modules: Reusable units of code (e.g., apt, copy, systemd) that perform specific actions.
• Idempotency: Modules are designed to produce the same result if run multiple times. In LLM deployment, Ansible is often used for post-provisioning configuration, such as installing NVIDIA drivers, Docker, and inference server software (like vLLM or TGI) on pre-existing virtual machines or bare-metal servers.

While not traditional IaC, Kubernetes manifests (YAML) and Helm are the de facto standard for defining and deploying containerized application infrastructure, including LLM inference services.

• Native Manifests: Declarative YAML files define Deployments, Services, and ConfigMaps for model serving pods.
• Helm: A package manager for Kubernetes that uses templated charts to manage complex applications. A chart can define an entire LLM serving stack with configurable values for model ID, resource requests, and autoscaling parameters. This approach is foundational for deploying inference endpoints using frameworks like KServe or Seldon Core, and is often provisioned onto a Kubernetes cluster that was itself created by a tool like Terraform.

A Kubernetes Operator is a method of packaging, deploying, and managing a Kubernetes application using custom resources and controllers to automate complex operational tasks. For LLM serving, an Operator can manage the full lifecycle of an inference service—handling deployment, scaling, updates, and recovery—by encoding operational knowledge into software.

• Key Function: Automates Day-2 operations like rolling updates and health checks for stateful applications.
• IaC Integration: The Operator's logic and the custom resources it manages are themselves defined and versioned as code, extending IaC principles into runtime orchestration.
• Example: An InferenceService Operator could automatically provision GPU nodes, deploy a model container, and configure a load balancer based on a declarative YAML file.

Terraform is an open-source Infrastructure as Code tool created by HashiCorp that uses a declarative configuration language (HCL) to provision and manage cloud and on-premises resources. It maintains a state file to map real-world resources to your configuration.

• Core Mechanism: Uses a plan-and-apply workflow. It generates an execution plan showing what actions will be taken to reach the desired state, then provisions the resources.
• Provider Ecosystem: Has providers for all major clouds (AWS, GCP, Azure) and Kubernetes, allowing you to define everything from virtual networks and GPU instances to Kubernetes namespaces in a single configuration.
• Use Case: Defining the foundational cloud infrastructure (VPCs, compute clusters, object storage) upon which LLM serving platforms like Kubernetes are deployed.

Pulumi is an Infrastructure as Code platform that allows engineers to define cloud infrastructure using general-purpose programming languages like Python, TypeScript, Go, and C#. It translates this code into commands that create and manage resources on cloud providers.

• Key Differentiator: Uses familiar programming languages instead of a domain-specific language (DSL). This enables the use of loops, functions, classes, and package management.
• Benefit for AI/ML: Complex LLM deployment logic—such as dynamically generating configuration based on model size or setting up canary deployment traffic splits—can be expressed directly in code with less abstraction.
• State Management: Like Terraform, it maintains a state backend to track resource relationships and dependencies.

Ansible is an open-source automation tool focused on configuration management, application deployment, and task automation. It uses YAML-based playbooks to describe the desired state of systems and is primarily agentless, using SSH or WinRM.

• Operational Model: Imperative and procedural. Playbooks define a series of tasks (e.g., install a package, start a service, copy a file) to be executed in order.
• IaC Role: Often used for configuration management within already-provisioned infrastructure (e.g., configuring OS settings on a VM, deploying and managing software on a Kubernetes cluster). It complements declarative provisioning tools like Terraform.
• Use Case: Bootstrapping a Kubernetes cluster node, installing NVIDIA drivers and container runtimes, or deploying and configuring monitoring agents across inference servers.

GitOps is an operational framework that applies IaC principles to the full application and infrastructure lifecycle. It uses Git as the single source of truth for declarative infrastructure and application definitions, with automated processes to reconcile the live state with the Git state.

• Core Principle: The desired state of the entire system (infrastructure, applications, configs) is versioned in Git. Automated operators (like Flux or ArgoCD) continuously monitor the repository and apply any changes.

• Key Components:

  • Declarative Descriptions: Everything is defined as code (Kubernetes manifests, Helm charts, Terraform modules).
  • Automated Reconciliation: A controller automatically pulls changes from Git and applies them to the cluster.
  • Observability & Rollback: Drift detection alerts when the live state diverges from Git. Failed deployments are reverted by rolling back a Git commit.

• LLM Application: Enables automated, auditable, and consistent deployment of model serving configurations, prompt version updates, and scaling policies.