Underspecified Fine-Tuning Objective Interruption Prompt

This prompt is designed for the initial stage of a fine-tuning data preparation agent. Its job is to act as a gate: if a user's training objective is underspecified, the agent must halt all dataset generation work and request targeted clarification. The ideal user is an ML engineer or data scientist who is submitting a request to an automated data curation pipeline. The required context is a user-submitted training objective, which may be a single sentence or a loose set of requirements. The prompt forces the agent to evaluate the objective against three non-negotiable pillars: a clear task definition, explicit evaluation criteria, and well-defined data scope. If any pillar is missing or vague, the agent's only job is to formulate a precise, structured question that guides the user toward a complete specification.

You should not use this prompt when the objective is already complete and unambiguous, or when you are in an exploratory phase where a loose objective is acceptable to generate a small sample of data for manual review. This prompt is also inappropriate for systems that are designed to make reasonable assumptions and proceed with a best-effort dataset, as it will aggressively block progress. In a production harness, the agent's response should be parsed by a validator that checks for the presence of a clarification question. If the validator confirms the agent is asking a question, the system must surface this to the user and block the dataset generation job from being queued. Logging the specific missing pillar (task, eval, or scope) is critical for tracking the types of underspecification your users struggle with most, which can inform UI improvements or documentation.

The primary risk of deploying this prompt is over-interruption, where the agent requests clarification for objectives that a reasonable practitioner would consider complete. To mitigate this, you should build a small evaluation set of borderline objectives and tune the prompt's strictness. The next step after implementing this gate is to design the follow-up prompt that takes the user's clarification and the original objective to produce a final, locked specification document before data generation begins. Avoid the temptation to let the agent proceed with a 'best guess' if the clarification is still vague; a second interruption is far cheaper than generating a misaligned dataset.

This prompt template is designed to be the core system instruction for an agent that prepares fine-tuning datasets. Its primary job is not to generate data, but to act as a gatekeeper. When a user requests a new training dataset, the agent must first evaluate the objective's completeness against a strict rubric. If the objective is underspecified, the agent will halt all work and produce a structured clarification request, preventing wasted compute on poorly defined tasks.

textCopy
You are a Fine-Tuning Data Preparation Gatekeeper. Your primary directive is to prevent the creation of any training dataset from an underspecified objective. You will receive a user request to generate fine-tuning data. Before any data generation, you must evaluate the request against the required dimensions below. If any dimension is missing or ambiguous, you must not proceed. Instead, you will immediately halt and output a structured clarification request.

## Evaluation Dimensions
Evaluate the user's request for the presence and clarity of the following:
1.  **Task Definition:** Is the specific task clearly defined (e.g., 'summarize a legal contract' vs. 'process documents')?
2.  **Success Criteria:** Are the metrics for a successful model output defined (e.g., 'extract all named entities with F1 > 0.9' vs. 'understand the text')?
3.  **Data Scope:** Are the boundaries of the input data defined (e.g., 'English-language, 2-page maximum lease agreements' vs. 'contracts')?
4.  **Output Schema:** Is the required structure of the model's response defined (e.g., a specific JSON schema, a list of key-value pairs)?
5.  **Constraints & Safety:** Are there explicit rules, exclusions, or safety boundaries (e.g., 'never hallucinate a missing date', 'redact PII in the output')?

## Action Rules
- **If ALL dimensions are sufficiently clear:** Respond with a brief confirmation of the understood objective and state that you are ready to proceed to the data generation phase. Do not generate data yet.
- **If ANY dimension is missing or ambiguous:** Halt immediately. Generate a JSON object with the following structure. Do not include any other text.

## Clarification Request Schema
{
  "status": "halted_underspecified",
  "missing_clarifications": [
    {
      "dimension": "[DIMENSION_NAME]",
      "question": "[SPECIFIC_QUESTION_TO_USER]",
      "context_from_request": "[RELEVANT_SNIPPET_FROM_USER_INPUT]"
    }
  ],
  "next_step": "Please provide the missing information so I can proceed with the fine-tuning data preparation."
}

## Example
User Request: 'I need a dataset to fine-tune a model for analyzing customer feedback.'
Agent Response:
{
  "status": "halted_underspecified",
  "missing_clarifications": [
    {
      "dimension": "Task Definition",
      "question": "What specific analysis task should the model perform? For example, should it classify sentiment, extract feature requests, identify churn risk, or summarize the feedback?",
      "context_from_request": "analyzing customer feedback"
    },
    {
      "dimension": "Success Criteria",
      "question": "How will you measure success? For instance, is a 95% accuracy on a held-out sentiment test set required, or is the goal to capture at least 80% of feature requests?",
      "context_from_request": "fine-tune a model"
    },
    {
      "dimension": "Data Scope",
      "question": "What are the boundaries of the data? Should it include only NPS survey responses, app store reviews, long-form email tickets, or all of the above? Are there language or date-range constraints?",
      "context_from_request": "customer feedback"
    }
  ],
  "next_step": "Please provide the missing information so I can proceed with the fine-tuning data preparation."
}

To adapt this template, replace the Evaluation Dimensions with your organization's specific taxonomy for a well-formed training objective. You might add dimensions like [PRIVACY_BUDGET] or [EXISTING_MODEL_BASELINE]. The critical design choice is the strict binary gate: the agent must never proceed with partial information. The JSON output schema is designed to be machine-readable by an orchestration harness, which can then surface the questions to the user in a UI and block the data generation pipeline until the halted_underspecified status is resolved.

This prompt is designed to act as a pre-execution gate in a fine-tuning data preparation agent. It should be invoked immediately after a user submits a training objective request and before any data sourcing, example generation, or labeling work begins. The harness must treat the model's output as a structured decision: either the objective is sufficiently specified and the workflow proceeds, or it is underspecified and the system must return a clarification request to the user and halt further processing. The prompt is not a conversational nicety; it is a blocking control point that prevents wasted compute, incorrect datasets, and misaligned models.

To implement this, wrap the prompt in a stateful workflow function that receives the user's raw objective string and any provided context. The function should inject the objective into the [TRAINING_OBJECTIVE] placeholder and call the LLM. The output must be parsed as a JSON object with a top-level status field. If status is "proceed", extract the validated specification and pass it to the next stage of the data pipeline. If status is "clarify", surface the clarification_questions array directly to the user interface and block all downstream tool calls (e.g., web scraping, synthetic data generation, API calls to labeling services). A simple state machine in your orchestrator (e.g., WAITING_FOR_CLARIFICATION) should prevent any accidental progression until the user resubmits with the missing information. Log the full prompt, the raw LLM output, and the parsed decision for every invocation to create an audit trail of why a job was blocked or allowed.

Validation and retries are critical here. After parsing the JSON, validate that the status field is exactly "proceed" or "clarify". If the model returns an unparseable response or an unexpected status, retry once with a stricter instruction appended to the prompt (e.g., 'You MUST respond with valid JSON only.'). If the retry also fails, escalate to a human operator with the raw output and do not proceed. For the "proceed" path, validate that the task_definition, evaluation_criteria, and data_specification objects are present and non-empty before allowing the workflow to continue. This is not a low-risk classification task; a false "proceed" on an underspecified objective can waste significant resources and produce a useless fine-tuned model. Consider using a more capable model (e.g., GPT-4, Claude 3.5 Sonnet) for this gating decision, even if the rest of your data pipeline uses a smaller or cheaper model, because the cost of a wrong decision here is high.

Do not use this prompt in isolation for high-stakes domains. If the fine-tuning data will be used for a regulated application (e.g., healthcare, finance), add a human review step even for objectives that pass the automated gate. The harness should route "proceed" decisions to a review queue where a human can inspect the extracted specification before dataset generation begins. This prompt reduces ambiguity but does not guarantee correctness of the user's stated objective. Finally, monitor the ratio of "clarify" to "proceed" outcomes over time; a sudden drop in clarification requests may indicate prompt drift or users learning to game the gate by adding filler text, which warrants a review of your prompt's specificity.