Runaway Process Terminator

The terminator continuously samples system metrics—CPU utilization, memory allocation, disk I/O, and network bandwidth—comparing them against static or dynamic baselines. A moving average window prevents false positives from transient spikes while ensuring genuine runaway conditions are caught within milliseconds. Thresholds are typically configured as hard limits (e.g., 80% CPU for 5 seconds) or anomaly scores derived from historical usage patterns.

When a violation is detected, the terminator does not simply kill the parent agent process. It performs a recursive process tree walk to identify and terminate all child processes, threads, and subprocesses spawned by the agent. This prevents orphan process leakage, where background tasks continue consuming resources after the primary agent is stopped. The mechanism uses process group IDs and session IDs to ensure complete cleanup.

Rather than immediately force-killing the agent, the terminator follows an escalation ladder:

• SIGTERM: Requests graceful shutdown, allowing the agent to persist state
• SIGKILL: Force-terminates if the agent does not exit within a configurable grace period
• Quarantine: Isolates the agent's network namespace if termination fails
• Node Drain: Evicts all workloads from the affected compute node as a last resort

A critical design feature is the pre-termination snapshot. Before issuing the kill signal, the terminator triggers an immutable state dump—capturing the agent's memory, current task queue, and partial outputs to a forensic log. This enables post-mortem analysis and idempotent rollback to a known-good checkpoint, preventing data loss from incomplete transactions.

The terminator operates as a sidecar process or daemonset within container orchestration platforms like Kubernetes. It hooks into cgroup v2 controllers for resource accounting and uses eBPF probes for kernel-level observability without agent instrumentation. Integration with liveness probes ensures that terminated agents are automatically replaced by healthy instances if the workload is stateless.

Thresholds are defined as infrastructure-as-code policies, often expressed in Open Policy Agent Rego or custom DSLs. Policies can be agent-specific, tenant-specific, or environment-specific. Example rule: deny[msg] { input.cpu_percent > 85; input.duration_seconds > 10 }. This allows SRE teams to version-control safety limits and apply them consistently across fleets.

A software resilience pattern that prevents an agent from repeatedly attempting operations destined to fail. When failure thresholds are exceeded, the circuit breaker trips and immediately rejects calls for a configurable timeout, allowing downstream systems to recover. This is a preventative measure distinct from a terminator's reactive kill.

• States: Closed (normal), Open (failing fast), Half-Open (testing recovery)
• Use case: Preventing cascading API failures from overwhelming an agent
• Key metric: Failure rate over a sliding time window

A hardware or software timer that monitors agent liveness by requiring periodic heartbeat signals. If the agent hangs, deadlocks, or enters an infinite loop and fails to kick the watchdog, the timer expires and triggers a corrective action—typically a process restart or termination. This catches failures a resource monitor might miss.

• Hardware WDT: Independent clock, immune to OS crashes
• Software WDT: OS-level, monitors application threads
• Timeout: Typically configurable in milliseconds to seconds

A safety mechanism that automatically triggers a kill command or safe state if the human operator becomes incapacitated or fails to provide a periodic confirmation signal. Unlike automated monitors, this addresses human-in-the-loop failure modes where oversight is lost.

• Mechanism: Requires active human presence (e.g., pedal, periodic button press)
• Application: Physical robotics, industrial control systems
• Contrast: A runaway process terminator is fully automated; a dead man's switch requires human absence to activate

A non-lethal containment measure that dynamically strips an agent's access to specific tools, APIs, or data sources without terminating the process. This allows for graceful restriction—the agent remains alive but can no longer cause harm through compromised capabilities.

• Granularity: Per-tool, per-API-key, per-scope
• Use case: Revoking file system write access while preserving read-only diagnostic access
• Advantage: Enables forensic analysis of a live but contained agent

The process of reverting an agent's internal state, memory, and environment to a previously saved, stable checkpoint. After a runaway process is terminated, state rollback undoes the sequence of erroneous actions, restoring the system to a known-good configuration.

• Requirement: Immutable state snapshots or event-sourced logs
• Idempotency: Rollback operations must be safely repeatable
• Challenge: Rolling back external side effects (e.g., sent emails, database writes) may require compensating transactions