Data Sanitization

Whitelist validation (or allowlisting) is the most secure form of input validation, where data is accepted only if it matches a strictly defined set of permitted characters, patterns, or values. It operates on the principle of "deny by default."

• Character Set Whitelisting: Define a regex pattern (e.g., ^[a-zA-Z0-9\s-]*$) that only allows alphanumerics, spaces, and hyphens for a "name" field, rejecting any input containing symbols like ', ;, or <.
• Predefined Value Checking: For enumerated fields (e.g., status), validate that the input matches one of the exact allowed strings ("active", "pending", "closed").
• Structured Data Parsing: For complex inputs like dates or IDs, use a dedicated parser (e.g., a date library) to validate the structure; if parsing fails, the input is rejected.

This technique is superior to blacklisting (which tries to block known bad patterns) as it is inherently proactive and resilient against novel attack vectors.

Canonicalization is the process of converting data into its simplest, standard form before validation. Normalization is a related process of transforming data into a consistent format. This is critical because attackers often use alternative encodings to bypass validation.

• Unicode Normalization: Convert text to a standard form (NFC or NFD) to prevent confusion between visually similar characters from different code points. For example, ensure café is represented consistently.
• Path Canonicalization: Resolve relative path segments (. and ..) and symbolic links in file paths to prevent directory traversal attacks. An input like ../../../etc/passwd would be resolved to its absolute path and checked against a permitted directory root.
• URL Canonicalization: Decode percent-encoded characters and remove default ports to compare the URL against a security policy in its base form.

Failure to canonicalize before applying whitelist validation or output encoding is a common security flaw, as %3Cscript%3E (URL-encoded) might bypass a check for the literal characters <script>.

Context-aware output encoding applies the correct escaping or encoding routine at the point where data is rendered into a specific interpreter, based on the exact syntactic context. This is the definitive defense against injection attacks.

• HTML Context: Use libraries that differentiate between data placed in HTML text content (<div>{{ data }}</div>), HTML attributes (<input value="{{ data }}">), or JavaScript blocks (<script>var x = {{ data }};</script>), applying the appropriate encoder for each.
• SQL Query Context: Use parameterized queries (prepared statements) where user input is passed as data parameters, not string concatenation. The database driver handles the proper escaping for the SQL dialect.
• System Command Context: Avoid constructing commands via string concatenation. Use APIs that pass arguments as a list (e.g., Python's subprocess.run(['ls', '-la', user_input])) where the shell does not interpret metacharacters.

This technique complements input validation; you validate on input for correctness and encode on output for safety, creating a defense-in-depth strategy.

Data type casting forcibly converts input into a native, non-string data type, which inherently neutralizes any embedded code. Transformation alters the data's format to a safe subset while preserving its semantic meaning.

• String-to-Integer Casting: For numeric IDs, cast the input string to an integer. The operation int(user_input) will fail or convert a string like "123; DROP TABLE users" to the integer 123, stripping the malicious suffix.
• Boolean Conversion: Map string inputs ("true", "yes", "1") to a strict boolean primitive, eliminating any unexpected payload.
• Structured Format Parsing: Parse JSON or XML input into a native object/model using a secure parser (with XXE prevention enabled for XML). Subsequent code interacts with the object's properties, not the raw string.
• Content Sanitization (HTML/XML): Use a library like DOMPurify that parses HTML, removes all elements and attributes not on an allowlist, and returns clean HTML. This is a semantic validation step for rich content.

This technique is a powerful form of payload verification that moves data from an untrusted string into a trusted, structured domain object.

Length and size limitation is a fundamental sanitization control that restricts the amount of data an API or agent can accept, mitigating resource exhaustion attacks and limiting the potential impact of malicious payloads.

• String Length Validation: Enforce maximum character counts for all text inputs (e.g., 255 chars for a name, 10,000 chars for a description). This directly limits the size of injection payloads and log poisoning attempts.
• Upload Size Quotas: Restrict file uploads to specific byte limits and validate file types using whitelist validation of MIME types or magic numbers, not just file extensions.
• Array/Collection Bounds: Limit the number of items in an array input to prevent algorithms with O(n²) complexity from causing denial-of-service (e.g., a batch operation limited to 100 items).
• GraphQL Query Depth/Complexity: Implement query cost analysis to reject overly complex queries that could overwhelm the resolver.

These limits are a critical part of constraint checking and should be defined in the API contract (OpenAPI Specification) and enforced by validation middleware. They work in tandem with rate limit validation to protect system stability.

Input validation is the process of programmatically verifying that data provided to a system conforms to expected formats, types, and constraints before processing. It is a proactive guardrail that precedes sanitization.

• Primary Goal: Ensure data is structurally and semantically correct for the application's logic.
• Common Techniques: Type checking, range validation, regex pattern matching, and checking against an allowlist of permitted values.
• Key Difference from Sanitization: Validation typically rejects invalid input, while sanitization transforms or neutralizes it.

Output validation is the process of verifying that data generated by a system conforms to a defined schema and business rules before it is sent to a client or downstream system. It acts as a final integrity check.

• Primary Goal: Prevent malformed, incomplete, or sensitive data from leaking from the API.
• Context: Crucial for AI agents making tool calls, ensuring the data they send to external services is correct and safe.
• Implementation: Often enforced via JSON Schema or Pydantic models applied to an agent's structured outputs.

JSON Schema is a declarative language for validating the structure and content of JSON data. It is the foundational standard for defining API contracts and validation rules.

• Function: Defines allowed data types, required properties, value ranges, format patterns (e.g., email, URI), and nested object structures.
• Use in AI: Used to enforce structured output guarantees from LLMs, ensuring tool-calling parameters are correctly formatted.
• Standardization: Provides a machine-readable contract for both static validation during development and dynamic validation at runtime.

The OpenAPI Specification (OAS) is a standard, language-agnostic interface description for HTTP APIs. It defines the complete contract, including endpoints, operations, and request/response schemas.

• Role in Tool Calling: Serves as the primary source for API schema integration, allowing AI agents to discover and understand how to call external services.
• Validation Foundation: The requestBody and responses sections use JSON Schema to define precise validation rules, enabling automated payload verification.

Schema enforcement is the runtime application of validation rules, defined in a schema language like JSON Schema, to guarantee that data structures strictly conform to a predefined model.

• Mechanism: Typically implemented as validation middleware that intercepts requests and responses.
• Outcome: Ensures schema conformance, rejecting any payload that violates defined types, constraints, or required fields.
• Benefit: Creates a deterministic boundary, preventing malformed data from reaching core business logic or external services.

Semantic validation is the process of checking that data is not only syntactically correct but also meaningful and consistent within its business context. It operates at a higher logic layer than syntactic checks.

• Examples: Verifying that a start_date is before an end_date, that a country_code matches a currency, or that a discount percentage is applicable to the given product_category.
• Contrast with Syntactic Validation: Syntactic validation checks format (e.g., YYYY-MM-DD), while semantic validation checks logic (e.g., date is in the future).