Agent Resource Quota

Agent resource quotas are applied at the namespace level, not the individual agent level. This creates logical resource boundaries for different teams, projects, or environments (e.g., development, staging, production). A quota defines the maximum sum of resources that all agents within that namespace can collectively use.

• Purpose: Isolates resource consumption to prevent a single team's agents from monopolizing cluster resources.
• Example: A 'research' namespace might have a quota of 32 CPU cores and 128GiB of memory, while a 'production' namespace has 200 CPU cores and 1TiB.
• Enforcement: The orchestration platform's scheduler (e.g., Kubernetes) rejects any agent creation request that would cause the namespace to exceed its quota.

These quotas limit the consumption of physical compute resources, primarily CPU and memory (RAM). They are the most critical for preventing node exhaustion and ensuring system performance.

• CPU: Measured in millicores (m). A quota of cpu: "2000m" allows agents in the namespace to collectively request or use up to 2 CPU cores.
• Memory: Measured in bytes, with common suffixes like Mi (mebibytes) and Gi (gibibytes). A quota of memory: "8Gi" limits total memory usage to 8 gibibytes.
• Ephemeral Storage: Can also be limited to control disk space used for agent caches, logs, and emptyDir volumes.
• Mechanism: Enforced via the agent's declared requests and limits in its pod specification.

These quotas limit the number of Kubernetes API objects that can be created within a namespace, preventing API server overload and managing cluster scale.

• Common Limited Objects:
  • pods: The total number of agent pods.
  • services: Network endpoints for agent communication.
  • persistentvolumeclaims: Requests for durable storage.
  • configmaps & secrets: Configuration and sensitive data used by agents.
• Purpose: Prevents a runaway agent deployment script from creating thousands of orphaned objects, which can degrade API performance and consume etcd storage.
• Example: A quota of pods: "50" ensures no more than 50 agent pods can exist concurrently in the namespace.

Quotas can be selectively applied based on scopes, which filter the set of agents to which the quota applies. This allows for fine-grained resource management policies.

• Terminating/NotTerminating: Applies only to pods with an activeDeadlineSeconds set (short-lived batch jobs) or to long-running pods.
• BestEffort/NotBestEffort: Applies to pods with no resource requests/limits (BestEffort QoS) or to pods that do have them (Burstable/Guaranteed QoS).
• PriorityClass: Applies to pods based on their assigned scheduling priority.
• Use Case: A quota with scope BestEffort can limit the number of low-priority, best-effort agent pods, ensuring they don't crowd out critical, resource-guaranteed agents.

Resource quotas interact directly with the Quality of Service (QoS) class assigned to each agent pod. The QoS class (Guaranteed, Burstable, BestEffort) determines how resource limits are enforced and influences eviction priority under node pressure.

• Guaranteed: Pods with equal requests and limits for CPU/memory. These are highest priority for quota fulfillment and are last to be evicted.
• Burstable: Pods with requests < limits. They can burst above their requested amount up to the limit, if resources are available.
• BestEffort: Pods with no requests or limits. They are first to be evicted and are only constrained by object count quotas, not compute quotas.
• Quota Accounting: The quota tracks the requests (for scheduling) and limits (for cgroup enforcement) of all pods, with QoS dictating the strictness of the limit.

Effective quota management requires monitoring, planning, and integration with other orchestration features to avoid agent deployment failures and resource deadlocks.

• Monitoring: Use tools like kubectl describe resourcequota and dashboards to track quota usage and avoid hitting limits unexpectedly.
• Requests vs. Limits: Set realistic requests on all agent pods. The sum of requests is counted against the quota and must be available for the pod to be scheduled. Limits control the hard ceiling on consumption.
• Combination with Auto-scaling: When using a HorizontalPodAutoscaler (HPA), ensure the namespace quota has sufficient headroom to allow the HPA to create new agent replicas during scale-out events.
• Default Quotas: Use LimitRanges in the namespace to set default requests and limits for agents, ensuring all pods are counted consistently against the quota.