Agent Affinity/Anti-Affinity Rules

Pod anti-affinity rules distribute agent replicas across failure domains to maximize availability. A common pattern is to prevent two pods of the same Agent StatefulSet or deployment from sharing a node or zone.

• Hard Anti-Affinity (requiredDuringSchedulingIgnoredDuringExecution): A strict rule. If no node satisfies the condition, the pod remains unscheduled.
• Soft Anti-Affinity (preferredDuringSchedulingIgnoredDuringExecution): A preference. The scheduler attempts to comply but will schedule the pod anyway if necessary, often used with a weighting system. This is essential for building fault-tolerant multi-agent systems.

These rules manage co-location based on agent labels, not node properties. They are used to:

• Reduce Communication Latency: An inference agent and its dedicated vector database sidecar can be forced onto the same node via pod affinity, minimizing network hops.
• Form Logical Units: Agents that are part of a tightly coupled orchestration workflow, such as a task decomposition agent and its first sub-task executor, can be scheduled together.
• Share Memory Volumes: Affinity enables pods to share an emptyDir volume for high-speed, in-memory data exchange.

A more advanced and granular form of anti-affinity, topology spread constraints control pod distribution across defined failure domains (nodes, zones, regions) based on skew. You can specify maxSkew (the maximum difference in pod count between any two domains) and whenUnsatisfiable (what to do if the constraint can't be met). This provides finer control than simple anti-affinity for ensuring high availability and even load distribution across a cluster's topology.

Affinity rules do not operate in isolation; the scheduler evaluates them alongside other constraints and policies:

• Resource Requests/Limits: An agent must be scheduled on a node with sufficient CPU and memory.
• Taints and Tolerations: A node can repel pods unless they explicitly tolerate its taint.
• Pod Priority and Preemption: Higher priority pods can evict others to satisfy their own scheduling requirements, including affinity rules.
• Pod Disruption Budgets: During voluntary disruptions, the scheduler respects PDBs, which may temporarily violate anti-affinity preferences to maintain minimum available replicas.

An orchestration capability where the system automatically detects agent failures and takes corrective action. This works by:

• Monitoring health check failures.
• Automatically restarting failed agents (often respecting restart policies).
• Rescheduling agents to healthy nodes if the original node is problematic. Self-healing complements anti-affinity rules; distributing agents across nodes ensures a node failure doesn't take down all instances of a critical service.

A Kubernetes policy that limits the number of agent pods in a voluntary disruption that can be down simultaneously. It ensures high availability during cluster maintenance operations like node drains or updates.

• A PDB defines minAvailable or maxUnavailable pods.
• The orchestrator respects this budget when evicting pods. This is a higher-level availability guarantee that works with anti-affinity rules; anti-affinity spreads pods across nodes, while a PDB ensures a minimum number stay running during controlled events.

A Kubernetes workload API object used to manage stateful agent applications. It provides guarantees essential for agents that manage persistent data:

• Stable, unique network identifiers (hostname).
• Stable, persistent storage tied to the pod identity.
• Ordered, graceful deployment and scaling. Affinity rules for a StatefulSet are crucial for ensuring its pods are scheduled onto nodes with the required local storage or to meet specific data-locality performance requirements.

A Kubernetes workload that ensures a copy of a specific agent pod runs on all (or some) nodes in the cluster. It's used for node-level infrastructure agents.

• Common uses: logging agents (Fluentd), monitoring agents (Prometheus Node Exporter), or networking agents.
• Affinity rules are inherent in its design (one per node), but node selectors or tolerations can refine which nodes get the DaemonSet pod. This pattern contrasts with deployment-based agents, where affinity/anti-affinity rules dynamically influence scheduling.