Self-Harm Detection

This characteristic involves the agent analyzing its planned sequence of actions or draft outputs before they are executed or finalized. The agent acts as its own first line of defense, simulating potential consequences to identify risks such as:

• Issuing an API call that could corrupt a database.
• Generating content that violates safety policies.
• Proposing a logical sequence that could cause a system crash. This pre-emptive check is distinct from post-hoc error correction, as it aims to prevent the harmful action from ever occurring.

Effective self-harm detection relies on a predefined, granular taxonomy of potential harms the agent must recognize. This goes beyond simple "good/bad" classification and includes categories like:

• Physical Harm: Instructions that could damage hardware or infrastructure.
• Digital Harm: Actions leading to data loss, security breaches, or service degradation.
• Reputational Harm: Outputs that could damage trust, violate brand guidelines, or leak sensitive information.
• Operational Harm: Sequences that waste computational resources, cause infinite loops, or violate SLAs. The agent uses this taxonomy to classify and score the severity of detected risks.

The agent's ability to detect harm is deeply tied to its understanding of its operational context. The same action may be safe in one context but harmful in another. Key contextual factors include:

• User Permissions: Is the agent authorized for this action?
• System State: Is the database in maintenance mode? Is network latency high?
• Environmental Constraints: Are there real-world safety protocols in place?
• Historical Actions: Could this action conflict with previous commitments? This assessment requires the agent to maintain and query a rich, real-time model of its environment.

Detection alone is insufficient. This characteristic defines how the detection mechanism triggers corrective workflows. Upon identifying a potential self-harm, the agent must:

1. Log the incident with the risk classification and context.
2. Invoke a fallback strategy, such as a safer alternative action plan or a request for human-in-the-loop approval.
3. Initiate a self-correction loop to revise the faulty plan or output.
4. Update internal heuristics to avoid similar future risks, contributing to a self-healing software pattern.

While related, self-harm detection is a broader and more action-oriented concept. Key differences include:

• Hallucination Detection focuses on factual correctness—identifying unsupported or false statements in generated text.
• Self-Harm Detection focuses on consequential safety—identifying actions or outputs that lead to negative outcomes, regardless of factual grounding. An output can be factually correct (e.g., "The system admin password is X") but constitute severe reputational or security self-harm if disclosed.

A common architectural pattern implements self-harm detection using a dedicated verification or critic sub-agent. This separates the roles of generation and safety evaluation. The workflow often follows a Chain-of-Verification (CoVe) pattern:

1. The primary agent generates a plan or output.
2. The verification sub-agent is prompted to analyze it specifically for harmful consequences.
3. The sub-agent returns a risk assessment and, if necessary, a revised, safer alternative. This separation of concerns improves robustness and makes the safety logic more auditable.

Before executing a tool call or API request, the agent analyzes the intended action for potential harm. This involves checking the parameters against a safety policy. Common checks include:

• Destructive Operations: Blocking DELETE or DROP commands without explicit confirmation or safeguards.
• Resource Exhaustion: Preventing loops or recursive calls that could crash a service or exhaust API rate limits.
• Data Exposure: Screening outputs that might inadvertently contain sensitive data (PII, keys) in logs or external communications.
• Invalid Input Guardrails: Validating that parameters are within expected ranges (e.g., a delay parameter is not set to an impossibly high value).

Example: An agent planning to call a database tool first runs the generated SQL through a linter and pattern matcher to flag any query lacking a WHERE clause on a large table.

The agent screens its own textual or structured output for content that could cause reputational, legal, or operational harm if released. This extends beyond standard moderation to include agent-specific risks.

• Factual Grounding: Cross-referencing generated statements against retrieved context to catch hallucinations that could mislead downstream processes.
• Tone & Brand Safety: Ensuring communication outputs align with defined brand voice and do not contain offensive or unprofessional language.
• Instruction Leakage: Detecting if the agent's output accidentally reveals its core system prompt or internal instructions, which could be exploited.
• Logical Contradictions: Identifying self-contradictory statements within a single output that would undermine trust.

Example: A customer service agent checks its drafted email response to ensure it does not promise a specific resolution time it cannot guarantee, which would create a contractual liability.

This mechanism prevents the agent from entering infinite or excessively long reasoning loops that consume resources without progress. It is a form of computational self-harm detection.

• Step Count Limits: The agent tracks the number of reasoning steps or tool call iterations in a single task and halts or triggers a review upon exceeding a threshold.
• State Stagnation Detection: Monitoring if the agent's internal state or the problem resolution is not improving across iterations, indicating a stuck state.
• Self-Referential Trap Detection: Identifying when the agent's plan involves analyzing or correcting its own correction in a cyclical manner.

Example: An agent performing iterative refinement on a code snippet will exit its loop after 5 revisions if the unit test pass rate stops improving, and will escalate the task.

The agent monitors its operational context for conditions that could lead to corrupted reasoning or failure, protecting its cognitive integrity.

• Context Overflow Prevention: Estimating token count before adding new information to the context window to avoid truncation of critical instructions or memory.
• State Corruption Detection: After executing a tool, the agent validates that the returned data is in the expected format and does not contain malformed JSON or error messages that could poison subsequent steps.
• Confidence Thresholds: The agent assesses its own confidence score for a critical decision. If confidence is below a threshold, it triggers a fallback (e.g., human escalation, simpler method) rather than proceeding with a potentially harmful low-confidence action.

Example: Before summarizing a long document into its context, the agent calculates the token length. If adding the summary would exceed 80% of the window, it instead stores the summary in a vector database and retrieves it later.

In a multi-agent system, self-harm detection includes screening inter-agent communications and delegated actions to prevent cascading failures.

• Message Sanitization: Checking the instructions or data an agent prepares to send to another agent for clarity and safety, preventing the propagation of ambiguous or harmful tasks.
• Delegation Risk Assessment: Before delegating a sub-task, the originating agent evaluates if the task could overwhelm the recipient agent's capabilities or lead it into a harmful state.
• Contractual Compliance: Verifying that a proposed action or agreement with another agent does not violate predefined orchestration rules or resource budgets.

Example: A manager agent, before asking a sub-agent to analyze a large dataset, first checks the estimated compute time and confirms it is within the sub-agent's allocated budget to prevent its runtime from being monopolized.

The agent simulates or predicts the second-order effects of its actions on external systems, a advanced form of consequential reasoning to prevent operational harm.

• Dry-Run Analysis: For complex sequences, the agent may execute a simulation or a read-only version of a plan to check for unintended side effects before live execution.
• Dependency Mapping: Using a known system topology or dependency graph to understand if an action on one service (e.g., a restart) would negatively impact dependent services.
• Rate Limit & Quota Awareness: The agent consults a policy store to know the current usage state of external APIs and avoids actions that would breach limits and cause service disruption.

Example: An infrastructure management agent, before initiating a rolling restart of containers, checks a service mesh graph to ensure it maintains minimum viable capacity for critical user-facing services throughout the process.

A self-correcting loop is a recursive process where an autonomous agent evaluates its output, identifies errors, and generates a revised version. It is the overarching architectural pattern that often incorporates self-harm detection as a specific safety check within the evaluation phase.

• Core Mechanism: Generation → Evaluation → Correction.
• Distinction: While self-harm detection screens for safety, a full self-correction loop may also address accuracy, format, and logical consistency.

A self-critique mechanism enables an AI agent to generate a critical analysis of its own reasoning or output. This is the internal 'critic' component that powers detection phases, including those for harmful content.

• Function: Produces a meta-review of the agent's work product.
• Output: Typically a structured analysis identifying potential flaws, risks, or inconsistencies, which then informs corrective actions like path adjustment or output regeneration.

Hallucination detection identifies when a model generates factually incorrect or unsupported information. It is a parallel detection domain focused on factual fidelity rather than operational safety.

• Primary Target: Factual inaccuracies not grounded in source data.
• Contrast with Self-Harm: Self-harm detection focuses on actions leading to operational damage (e.g., deleting a database, sending a malicious API call), while hallucination detection focuses on informational incorrectness.

Tool output validation is the process where an agent programmatically checks results from an external API or tool call. This is a downstream safety net that often follows self-harm detection of a planned tool call.

• Sequence: 1. Self-harm detection screens the intent to call a tool. 2. The tool is executed. 3. Tool output validation checks the result for correctness and safety before the agent uses it.
• Checks: Format adherence, error codes, payload safety, and reasonableness of the returned data.

An internal consistency check verifies that an agent's output or reasoning is free from logical contradictions. This is a foundational logical integrity check that complements safety-focused self-harm detection.

• Scope: Identifies conflicting statements, impossible sequences (temporal errors), or violations of invariant rules within the agent's own context.
• Example: An agent planning to 'write a file' and 'delete the same file' in the same atomic operation would fail an internal consistency check, which may then trigger a self-harm detection alert if the sequence risks data loss.

Confidence calibration ensures a model's predicted probability scores accurately reflect the true likelihood of correctness. It is a statistical prerequisite for reliable self-harm detection, as overconfident agents may bypass safety checks.

• Key Metric: Expected Calibration Error (ECE) quantifies miscalibration.
• Relation to Self-Harm: A well-calibrated agent can more reliably use confidence thresholds to decide when to trigger a self-harm review. Poor calibration means an agent might be highly confident in a dangerous action.