CSV Generation Prompt for ETL Pipelines

This prompt is designed for data engineers and platform builders who need an LLM to produce parseable CSV output that loads directly into ETL pipelines, databases, or analytics tools without manual cleanup. The ideal user has a defined target schema—column names, types, and ordering—and needs the model to populate rows conforming to that schema from unstructured or semi-structured input such as logs, reports, emails, or scraped text. The prompt enforces delimiter consistency, correct quoting of special characters, aligned header rows, and dialect-specific formatting so downstream parsers never encounter malformed records.

Use this prompt when your ingestion pipeline has strict formatting requirements: a specific delimiter character, a known quoting convention, consistent line endings, and a header row that matches the expected column order exactly. It works well for batch extraction jobs where you feed batches of source documents and receive clean CSV rows in return. The prompt assumes you control the schema definition and can provide it as part of the instructions. It is not suitable for exploratory data work where the schema is unknown, for generating free-text narratives, or for conversational output. If your downstream system can accept JSON or structured objects natively, prefer a JSON Schema enforcement prompt instead—CSV adds quoting and escaping complexity that JSON avoids.

Avoid this prompt when the input contains deeply nested structures that don't flatten cleanly into rows, when column counts vary unpredictably across records, or when the target system requires real-time streaming with partial-row guarantees. For multi-line text fields embedded within CSV cells, the prompt includes quoting and escaping rules, but test thoroughly with your specific parser. If your pipeline requires CSV dialect detection from unknown sources, this prompt is not the right tool—it generates CSV, it does not parse or normalize existing CSV. Before deploying, validate outputs with your actual downstream parser, not just a visual scan, and build automated checks for delimiter count consistency, quote pairing, and header alignment.

The template below is designed to produce CSV output that conforms to a specified dialect and schema. It enforces delimiter consistency, quoting rules, escape character handling, and header row alignment. Copy the template, replace the square-bracket placeholders with your specific requirements, and integrate it into your generation step. The prompt is self-contained and can be used without the rest of this playbook, but you should pair it with a post-generation validation step that checks dialect conformance and schema compliance before the output enters your pipeline.

codeCopy
You are a CSV generation engine. Your entire response must be valid CSV conforming to the dialect and schema specified below. Do not include any text outside the CSV output—no explanations, no markdown fences, no commentary.

## CSV Dialect
- Delimiter: [DELIMITER]
- Quote character: [QUOTE_CHAR]
- Escape character: [ESCAPE_CHAR]
- Line terminator: [LINE_TERMINATOR]
- Header row: [INCLUDE_HEADER]
- Quoting rule: [QUOTING_RULE]

## Output Schema
[OUTPUT_SCHEMA]

## Constraints
[CONSTRAINTS]

## Input Data
[INPUT]

## Examples
[EXAMPLES]

Generate the CSV output now.

How to adapt this template: Replace [DELIMITER] with your field separator (e.g., , or \t). Set [QUOTE_CHAR] to " or ' depending on your parser. Define [ESCAPE_CHAR] as \ or another character your downstream system expects. [LINE_TERMINATOR] should be \n or \r\n. Set [INCLUDE_HEADER] to true or false. [QUOTING_RULE] should specify when fields are quoted: all, minimal, non-numeric, or none. [OUTPUT_SCHEMA] should list column names, types, and constraints (e.g., name: string, required; age: integer, min=0; email: string, nullable). [CONSTRAINTS] should capture cross-field rules, uniqueness requirements, and value ranges. [INPUT] is the source data the model should transform. [EXAMPLES] should include 2-3 correct input-output pairs showing edge cases like fields containing delimiters, quotes, or newlines. After generation, validate the output with a CSV parser configured to the same dialect. Reject outputs that fail to parse, have mismatched column counts, or violate schema constraints. For high-risk pipelines, add a human review step before ingestion.

The CSV generation prompt is not a standalone artifact—it is a component inside a larger data pipeline. In production, the prompt sits between a data extraction step and a downstream ingestion system (database, analytics engine, or file store). The harness must handle validation, retry logic, dialect enforcement, and graceful failure before any generated CSV reaches a parser. Treat the prompt output as untrusted input until it passes automated checks.

Validation layer. Before writing the model's output to a file or stream, validate it against the expected CSV dialect. Use a CSV parser library (Python's csv.Sniffer, Pandas read_csv, or a dedicated validator) to confirm: (a) the delimiter is consistent across all rows, (b) quoting rules match the specified dialect (e.g., RFC 4180), (c) the header row is present and column count matches every data row, (d) no unescaped delimiters or newlines appear inside unquoted fields, and (e) multi-line fields are properly quoted. If validation fails, log the failure with the raw output and the specific parser error for debugging.

Retry and repair loop. When validation fails, do not immediately discard the output. Feed the validation error message back to the model in a repair prompt: 'The CSV output failed validation with error: [ERROR]. The expected dialect is [DIALECT]. Please regenerate the CSV with corrected formatting.' Limit retries to 2-3 attempts. If all retries fail, route the record to a dead-letter queue for human review. For high-throughput pipelines, implement a circuit breaker that stops generation if the failure rate exceeds a threshold (e.g., >10% of records in a batch).

Model selection and temperature. For CSV generation, prefer models with strong instruction-following and low temperature settings (0.0–0.2). Higher temperatures introduce formatting inconsistencies, especially with delimiter handling and quoting. If using a model that supports structured output modes (e.g., JSON mode), consider generating JSON first and converting to CSV in application code—this eliminates dialect ambiguity entirely. Only use direct CSV generation when the model reliably produces parseable output or when latency constraints prevent a two-step process.

Logging and observability. Log every generation attempt with: prompt version, model identifier, input hash, output hash, validation result (pass/fail), retry count, and latency. Attach trace IDs that link the CSV generation step to upstream extraction and downstream ingestion. This makes it possible to diagnose whether a pipeline failure originated in the prompt, the input data, or the parser configuration. For regulated or audit-sensitive workflows, persist the raw model output alongside the validated CSV for later review.

Downstream handoff. Once the CSV passes validation, write it to the target destination with explicit encoding (UTF-8 with BOM if Excel compatibility is required). Include metadata headers or sidecar files if the downstream system needs dialect information (delimiter, quote character, escape character). Never assume the consumer will auto-detect these settings correctly. If the pipeline feeds a database, wrap the CSV parsing and INSERT in a transaction with rollback on parse failure. For streaming systems, emit each validated row as a structured event rather than waiting for the full CSV to accumulate.