Scientific Reproducibility Claim Assessment Prompt Template

This prompt is designed for research integrity, meta-science, and peer review platform teams that need to systematically evaluate claims about the reproducibility of a scientific study. The core job-to-be-done is comparing a specific reproducibility claim against the study's methods section, data availability statement, and any provided replication attempt results. The output is a structured reproducibility confidence score and a detailed methodology gap analysis, enabling automated triage in platforms that screen papers for replication potential or audit published findings. Use this when you have a discrete, verifiable claim about a study's reproducibility and the source documents to verify it against.

Do not use this prompt for general literature review, initial study screening, or when the source documents are not available. It is not a replacement for a full replication attempt, nor is it designed to assess the novelty or impact of a study. The prompt assumes you have already extracted the relevant methods section, data availability statement, and any replication results into the [CONTEXT] placeholder. If you lack any of these components, the gap analysis will flag missing information, but the confidence score will degrade predictably. For high-stakes assessments, always route outputs to human review and require evidence grounding for every gap identified.

Before wiring this into a production pipeline, define your evaluation criteria: what constitutes an acceptable confidence threshold for auto-approval versus human review? The prompt outputs a structured JSON schema, so you can validate field completeness, score ranges, and evidence citations programmatically. If the model returns a confidence score above your threshold but the gap analysis is empty, that's a failure mode to test for. Start by running this against a golden set of papers with known reproducibility outcomes to calibrate your thresholds before scaling to untriaged content.

The template below is designed to be copied directly into your prompt library or AI harness. It uses square-bracket placeholders for all dynamic inputs, allowing you to adapt it to different studies, claims, and evidence sets without rewriting the core logic. The prompt instructs the model to act as a meta-science reviewer, comparing a claim about reproducibility against a methods section, data availability statement, and replication attempt results. The output is a structured reproducibility confidence score with a detailed methodology gap analysis.

codeCopy
You are a meta-science reviewer evaluating the reproducibility of a scientific claim. Your task is to compare the provided claim against the source materials and produce a structured assessment.

## INPUT
- **Claim to Assess:** [CLAIM]
- **Original Study Methods Section:** [METHODS_SECTION]
- **Data Availability Statement:** [DATA_AVAILABILITY_STATEMENT]
- **Replication Attempt Results:** [REPLICATION_RESULTS]

## OUTPUT_SCHEMA
Return a valid JSON object with the following structure:
{
  "reproducibility_score": "high|medium|low|unassessable",
  "confidence": 0.0-1.0,
  "methodology_gap_analysis": [
    {
      "gap_type": "missing_detail|protocol_deviation|data_unavailability|statistical_ambiguity|other",
      "description": "string",
      "severity": "critical|major|minor",
      "impact_on_reproducibility": "string"
    }
  ],
  "evidence_alignment_notes": "string",
  "requires_human_review": true|false
}

## CONSTRAINTS
- Base your assessment strictly on the provided source materials. Do not introduce external knowledge.
- If the methods section lacks sufficient detail to enable replication, flag this as a critical gap.
- If the data availability statement indicates restricted or unavailable data, note the impact on reproducibility.
- Compare the replication attempt results against the original claim. Explicitly state whether the replication supports, contradicts, or is inconclusive relative to the claim.
- Set `requires_human_review` to `true` if the `reproducibility_score` is `low` or `unassessable`, or if any gap has a `critical` severity.
- If the claim cannot be assessed due to missing information, set the score to `unassessable` and explain why in the `evidence_alignment_notes`.

To adapt this template, replace each square-bracket placeholder with your specific inputs. The [CLAIM] should be a single, atomic assertion about the study's reproducibility. The [METHODS_SECTION] and [DATA_AVAILABILITY_STATEMENT] should be the verbatim text from the original paper. The [REPLICATION_RESULTS] should contain the findings from any replication attempt, even if the result is a failure to replicate. If no replication attempt exists, set this field to 'No replication attempt provided' and expect the model to flag this as a major gap. After generating the output, validate the JSON structure before passing it to downstream systems. For high-stakes research integrity decisions, always route outputs with requires_human_review: true to a qualified reviewer.

This prompt is designed to be a single step in a larger research integrity pipeline. It should not be exposed directly to end users without a surrounding application harness. The harness is responsible for assembling the [PAPER_SECTION], [METHODS_SUMMARY], [DATA_AVAILABILITY_STATEMENT], and [REPLICATION_ATTEMPT] inputs from upstream extraction steps, enforcing the [OUTPUT_SCHEMA], and routing low-confidence outputs for human review. Treat the model's output as a structured recommendation, not a final adjudication.

Input Assembly and Validation: Before calling the model, validate that all four input placeholders contain non-empty strings. If [REPLICATION_ATTEMPT] is empty, the prompt should be aborted because the task is to assess a specific replication attempt, not to predict general reproducibility. Normalize the [OUTPUT_SCHEMA] to a strict JSON schema with required fields: reproducibility_confidence (enum: high, moderate, low, unable_to_assess), methodology_gap_analysis (array of strings), data_access_assessment (string), and overall_justification (string). Use a schema validation library like pydantic or ajv to reject malformed outputs before they enter downstream systems.

Retry and Self-Correction Logic: If the model's output fails schema validation, retry once by appending the validation error and the original prompt to a correction request: 'Your previous output failed validation with the following errors: [ERRORS]. Please correct your output to match the required schema.' If the second attempt also fails, log the failure and route the entire assessment to a human review queue. Do not loop indefinitely. For low or unable_to_assess confidence scores, automatically flag the assessment for human review regardless of schema validity.

Model Choice and Tool Use: This task requires strong reasoning over structured scientific text. Use a model with a large context window (at least 128k tokens) to accommodate lengthy methods sections and replication reports. GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are suitable. Do not use RAG for this prompt; all evidence must be provided in the prompt itself to ensure the model is comparing the exact text segments you intend. If the source documents exceed the context window, preprocess them with an extractive summarization step focused on the methods and data availability sections before calling this prompt.

Logging and Audit Trail: Log every assessment with a unique assessment_id, the model version, the full prompt input, the raw model output, the validated output, and the final routing decision (auto-accepted or escalated). This audit trail is critical for research integrity platforms because reproducibility assessments may be contested. Store the logs immutably and make them queryable by paper DOI and replication attempt ID. Never log or store the [PAPER_SECTION] or [REPLICATION_ATTEMPT] in plaintext if they contain unpublished or embargoed research; use your platform's access control layer to enforce permissions.

Human Review Integration: When an assessment is escalated, package the original inputs, the model's output, and the reason for escalation (e.g., low_confidence, schema_failure, data_access_gap) into a structured review task. The human reviewer should be able to accept, override, or annotate the model's assessment. Their final determination should be written back to the system of record and used to generate a completed reproducibility report. Do not use the model's output as the final public-facing assessment without this review step for any confidence level below high.