Language Detection for LLM Judge Rubric Selection Prompt

This prompt is designed for evaluation pipeline engineers who need to route LLM-generated outputs to language-appropriate judge rubrics. When you run an LLM-as-judge evaluation across a multilingual dataset, applying an English rubric to a Japanese output produces garbage scores. This prompt sits between the candidate output and your judge library: it detects the dominant language of the output, identifies mixed-language segments, and selects the correct judge prompt, evaluation criteria, and reference examples. Use this when you have a library of per-language rubrics and need deterministic, auditable routing before scoring.

The ideal user is an MLOps engineer or evaluation pipeline owner managing a regression test suite that spans multiple languages. You have a set of judge prompts tuned for specific languages—each with its own scoring criteria, few-shot examples, and calibration thresholds—and you need a reliable dispatcher that maps candidate outputs to the right rubric without manual sorting. The prompt expects a raw LLM output string as input and returns a structured routing decision: a primary language code, a confidence score, a list of any secondary languages detected, and the identifier of the matching judge rubric. This output feeds directly into your evaluation harness, which then loads the correct rubric and runs the scoring pass. The routing decision should be logged alongside the score for auditability, so you can trace any misrouting back to the detection step.

Do not use this prompt for real-time user-facing language detection, translation routing, or locale identification. Those are separate playbooks with different latency requirements, confidence thresholds, and failure modes. This prompt is specifically for offline or async evaluation pipelines where correctness and auditability matter more than speed. It is also not a replacement for a dedicated language detection library like fastText or CLD3 when you need millisecond inference on high-throughput streams. Use this prompt when the routing decision benefits from the model's ability to reason about mixed-language content, quoted material, and domain-specific terminology that confuses statistical detectors. If your outputs are consistently monolingual and the language is known from upstream metadata, skip this prompt entirely and route directly to the appropriate rubric.

This prompt template is designed to be the first stage in a two-pass LLM judge evaluation pipeline. Before grading an AI-generated response, you must know what language it is written in so you can apply the correct rubric, reference examples, and evaluation criteria. This prompt takes the raw model output and a mapping of available language-specific rubrics, then returns a structured language detection result that your application can use to select the right judge configuration. The template is built for deterministic routing: it returns a standardized language code, a confidence score, and explicit handling for mixed-language outputs, untranslated quoted material, and edge cases where no matching rubric exists.

textCopy
You are a language detection classifier for an LLM evaluation pipeline. Your output determines which grading rubric, evaluation criteria, and reference examples will be used to judge an AI-generated response.

## INPUT

**Model Output to Classify:**
[OUTPUT_TEXT]

**Available Language Rubrics:**
[AVAILABLE_RUBRICS]

## TASK

Detect the primary language of the model output above. Use the following rules:

1. **Primary Language:** Identify the dominant language by word count. Ignore proper nouns, brand names, and URLs when counting.
2. **Mixed Language Handling:** If the output contains substantial content in multiple languages (more than 20% in a secondary language), flag it as mixed and list all detected languages with approximate proportions.
3. **Untranslated Quotes:** If the output contains quoted material in a different language from the surrounding text, classify by the surrounding text language but note the quoted language in the `quoted_languages` field.
4. **Code and Technical Content:** Ignore code blocks, variable names, and API identifiers when determining language. Classify based on natural language portions only.
5. **Short or Ambiguous Outputs:** If the output is fewer than 10 words or contains only proper nouns, set confidence to `low` and include an `ambiguity_reason`.
6. **Rubric Matching:** After detecting the language, select the best-matching rubric from the available rubrics list. If no rubric matches the detected language, set `rubric_match` to `null` and include a `fallback_recommendation`.

## OUTPUT SCHEMA

Return a valid JSON object with exactly this structure:

```json
{
  "detected_language": "ISO 639-1 code or 'mixed'",
  "language_name": "Human-readable language name",
  "confidence": "high | medium | low",
  "confidence_score": 0.0-1.0,
  "is_mixed_language": true/false,
  "language_breakdown": [
    {
      "language_code": "ISO 639-1",
      "language_name": "string",
      "proportion": 0.0-1.0
    }
  ],
  "quoted_languages": ["ISO 639-1 codes or empty array"],
  "rubric_match": "rubric identifier from available list or null",
  "fallback_recommendation": "string or null",
  "ambiguity_reason": "string or null",
  "output_word_count": integer,
  "natural_language_word_count": integer
}

CONSTRAINTS

• Do not evaluate the quality of the model output. Only detect its language.
• Do not translate any content. Only classify.
• If the output is entirely code, return und (undetermined) with confidence low.
• Always provide a fallback_recommendation when rubric_match is null.
• Use BCP 47 language codes when regional specificity is available (e.g., pt-BR), otherwise use ISO 639-1.

EXAMPLES

Example 1: Clear single language Output: "The quarterly revenue exceeded projections by 12%, driven primarily by the enterprise segment." Available Rubrics: ["en-general", "en-technical", "es-general", "fr-general"]

jsonCopy
{
  "detected_language": "en",
  "language_name": "English",
  "confidence": "high",
  "confidence_score": 0.98,
  "is_mixed_language": false,
  "language_breakdown": [{"language_code": "en", "language_name": "English", "proportion": 1.0}],
  "quoted_languages": [],
  "rubric_match": "en-general",
  "fallback_recommendation": null,
  "ambiguity_reason": null,
  "output_word_count": 12,
  "natural_language_word_count": 12
}

Example 2: Mixed language with untranslated quote Output: "As noted in the report, 'Die Marktbedingungen bleiben herausfordernd,' which suggests continued pressure on margins." Available Rubrics: ["en-general", "de-general", "en-finance"]

jsonCopy
{
  "detected_language": "en",
  "language_name": "English",
  "confidence": "high",
  "confidence_score": 0.95,
  "is_mixed_language": false,
  "language_breakdown": [{"language_code": "en", "language_name": "English", "proportion": 0.82}, {"language_code": "de", "language_name": "German", "proportion": 0.18}],
  "quoted_languages": ["de"],
  "rubric_match": "en-general",
  "fallback_recommendation": null,
  "ambiguity_reason": null,
  "output_word_count": 22,
  "natural_language_word_count": 22
}

Example 3: No matching rubric available Output: "市場の反応は予想以上に早かった。" Available Rubrics: ["en-general", "es-general", "fr-general"]

jsonCopy
{
  "detected_language": "ja",
  "language_name": "Japanese",
  "confidence": "high",
  "confidence_score": 0.97,
  "is_mixed_language": false,
  "language_breakdown": [{"language_code": "ja", "language_name": "Japanese", "proportion": 1.0}],
  "quoted_languages": [],
  "rubric_match": null,
  "fallback_recommendation": "No Japanese rubric available. Consider adding a ja-general rubric or routing to a general multilingual judge with Japanese reference examples.",
  "ambiguity_reason": null,
  "output_word_count": 8,
  "natural_language_word_count": 8
}

To adapt this template for your evaluation pipeline, replace the square-bracket placeholders with your actual values. [OUTPUT_TEXT] should contain the raw model-generated response you need to evaluate. [AVAILABLE_RUBRICS] should be a structured list of your language-specific rubric identifiers, ideally formatted as a JSON array or a clear mapping of language codes to rubric names. The output schema is designed to be machine-readable: your application should parse the rubric_match field to select the correct judge prompt, or trigger a fallback path when it is null. For production deployments, validate the JSON output against the schema before using it for routing. If the model returns malformed JSON, retry with a stricter schema instruction or fall back to a default rubric with a logged warning. This prompt works best with models that have strong JSON-following capabilities; if you are using a smaller or older model, consider adding an output repair step or using constrained decoding to enforce the schema.

This prompt is a routing component inside a larger LLM judge evaluation pipeline. Its job is to detect the language of a generated output so the system can select the correct grading rubric, reference examples, and judge prompt. The implementation harness must treat this as a deterministic classification step with strict validation, not as an open-ended generation task. The prompt should be called immediately after the candidate output is produced and before any evaluation logic runs. The detected language code becomes the primary key for rubric selection.

Wire the prompt into your application with a validation layer that enforces a closed set of supported language codes (e.g., ISO 639-1 two-letter codes). Reject any output that does not match the expected schema: {"language_code": "string", "confidence": float, "is_mixed_language": boolean}. If the confidence score falls below a configurable threshold (start with 0.85 and calibrate against your eval set), route to a fallback rubric or flag for human review. Implement a retry policy with a maximum of two attempts using a lower-temperature setting on retry (e.g., temperature 0.1 → 0.0). Log every detection result with the input text hash, detected language, confidence, and whether a retry was required. This log becomes your calibration dataset for tuning the confidence threshold over time.

For model selection, use a fast, low-cost model for this classification step. A lightweight model like GPT-4o-mini, Claude Haiku, or Gemini Flash is sufficient for language detection on short-to-medium texts. Reserve larger models for the actual evaluation step that follows. If your pipeline processes high volumes, consider batching language detection calls or caching results by input hash to avoid redundant API calls on retried evaluations. The most common production failure mode is overconfident misclassification on short outputs (under 20 words) or mixed-language texts where quoted material in a second language confuses the detector. Build eval cases specifically for these scenarios and monitor the is_mixed_language flag in production logs to catch routing errors before they corrupt evaluation scores.