Agent Affinity/Anti-Affinity Rules

Affinity/anti-affinity rules express desired state rather than issuing direct commands. An orchestrator (e.g., Kubernetes) interprets these constraints during the agent scheduling phase to place pods on nodes that satisfy the declared rules. This is a core principle of declarative configuration, where the system continuously reconciles the actual state with the specified intent, allowing for dynamic rescheduling in response to node failures or scaling events.

Node affinity rules co-locate agents on specific nodes to leverage localized resources. This is critical for:

• Data Locality: Placing an agent on a node with a locally attached Persistent Volume containing its dataset to minimize I/O latency.
• Hardware Acceleration: Ensuring agents requiring a GPU or specialized Neural Processing Unit are scheduled onto nodes equipped with that hardware.
• Geographic/Zonal Placement: Using node labels for topology.kubernetes.io/zone to pin agents to a specific availability zone for compliance or low-latency access to regional services.

Pod anti-affinity rules distribute agent replicas across failure domains to maximize availability. A common pattern is to prevent two pods of the same Agent StatefulSet or deployment from sharing a node or zone.

• Hard Anti-Affinity (requiredDuringSchedulingIgnoredDuringExecution): A strict rule. If no node satisfies the condition, the pod remains unscheduled.
• Soft Anti-Affinity (preferredDuringSchedulingIgnoredDuringExecution): A preference. The scheduler attempts to comply but will schedule the pod anyway if necessary, often used with a weighting system. This is essential for building fault-tolerant multi-agent systems.

These rules manage co-location based on agent labels, not node properties. They are used to:

• Reduce Communication Latency: An inference agent and its dedicated vector database sidecar can be forced onto the same node via pod affinity, minimizing network hops.
• Form Logical Units: Agents that are part of a tightly coupled orchestration workflow, such as a task decomposition agent and its first sub-task executor, can be scheduled together.
• Share Memory Volumes: Affinity enables pods to share an emptyDir volume for high-speed, in-memory data exchange.

A more advanced and granular form of anti-affinity, topology spread constraints control pod distribution across defined failure domains (nodes, zones, regions) based on skew. You can specify maxSkew (the maximum difference in pod count between any two domains) and whenUnsatisfiable (what to do if the constraint can't be met). This provides finer control than simple anti-affinity for ensuring high availability and even load distribution across a cluster's topology.

Affinity rules do not operate in isolation; the scheduler evaluates them alongside other constraints and policies:

• Resource Requests/Limits: An agent must be scheduled on a node with sufficient CPU and memory.
• Taints and Tolerations: A node can repel pods unless they explicitly tolerate its taint.
• Pod Priority and Preemption: Higher priority pods can evict others to satisfy their own scheduling requirements, including affinity rules.
• Pod Disruption Budgets: During voluntary disruptions, the scheduler respects PDBs, which may temporarily violate anti-affinity preferences to maintain minimum available replicas.

The process by which an orchestration system decides which compute node or host machine should run a specific agent instance. This decision is based on:

• Resource requirements (CPU, memory, GPU)
• Constraints like node selectors and taints/tolerations
• Affinity/anti-affinity rules to influence co-location or distribution
• Overall cluster resource availability and load balancing goals.

A periodic diagnostic probe used by an orchestration system to determine if an agent is functioning correctly. There are two primary types:

• Liveness probes determine if an agent is running. Failure triggers a restart.
• Readiness probes determine if an agent is ready to accept work. Failure removes it from the service pool. These checks are critical for self-healing systems and work in tandem with affinity rules; an unhealthy agent on a node may influence where new agents are scheduled.

An orchestration capability where the system automatically detects agent failures and takes corrective action. This works by:

• Monitoring health check failures.
• Automatically restarting failed agents (often respecting restart policies).
• Rescheduling agents to healthy nodes if the original node is problematic. Self-healing complements anti-affinity rules; distributing agents across nodes ensures a node failure doesn't take down all instances of a critical service.

A Kubernetes policy that limits the number of agent pods in a voluntary disruption that can be down simultaneously. It ensures high availability during cluster maintenance operations like node drains or updates.

• A PDB defines minAvailable or maxUnavailable pods.
• The orchestrator respects this budget when evicting pods. This is a higher-level availability guarantee that works with anti-affinity rules; anti-affinity spreads pods across nodes, while a PDB ensures a minimum number stay running during controlled events.

A Kubernetes workload API object used to manage stateful agent applications. It provides guarantees essential for agents that manage persistent data:

• Stable, unique network identifiers (hostname).
• Stable, persistent storage tied to the pod identity.
• Ordered, graceful deployment and scaling. Affinity rules for a StatefulSet are crucial for ensuring its pods are scheduled onto nodes with the required local storage or to meet specific data-locality performance requirements.

A Kubernetes workload that ensures a copy of a specific agent pod runs on all (or some) nodes in the cluster. It's used for node-level infrastructure agents.

• Common uses: logging agents (Fluentd), monitoring agents (Prometheus Node Exporter), or networking agents.
• Affinity rules are inherent in its design (one per node), but node selectors or tolerations can refine which nodes get the DaemonSet pod. This pattern contrasts with deployment-based agents, where affinity/anti-affinity rules dynamically influence scheduling.