Duplicate Job Submission Detection Prompt

This prompt is for MLOps engineers, data-platform engineers, and pipeline operators who need to prevent duplicate batch job submissions before they waste compute, corrupt data, or trigger conflicting side effects. The job-to-be-done is fingerprinting a job's configuration—its code version, parameters, input paths, and environment—and comparing it against a history of recent submissions to produce a clear skip-or-run decision. The ideal user is someone embedding this check into a job orchestration harness (Airflow, Dagster, Prefect, Argo, or a custom scheduler) where duplicate detection must run before the job is queued, not after it has already consumed resources.

Use this prompt when you have a structured job specification and a known set of recently submitted jobs to compare against. It works best when the comparison logic requires semantic understanding of parameter changes—for example, distinguishing a meaningful hyperparameter change from a cosmetic log-level tweak. Do not use this prompt for real-time API request deduplication (use an idempotency-key system instead), for detecting duplicate records in a database (use SQL or a hash index), or when the job history is too large to fit in a single prompt context (use a retrieval step to pre-filter candidates first). This prompt is also inappropriate when the cost of a false negative—running a duplicate job—is catastrophic; in those cases, add a deterministic hash-based guard as a hard pre-check before invoking the model.

The prompt requires careful handling of near-duplicate detection. Two job submissions might differ only in a timestamp, a random seed, or a non-functional metadata field. The prompt must be instructed to ignore these fields when they don't affect the job's outcome. Conversely, a one-character change in a SQL query or a minor version bump in a dependency can represent a meaningful difference. Your harness should pre-process the job spec to strip known noise fields before passing them to the prompt, and the prompt itself should be tested against a golden set of known duplicates and known distinct pairs. Always log the model's reasoning alongside the decision so operators can audit borderline cases.

Before deploying this prompt, define your evaluation criteria: precision (did we skip jobs that should have run?), recall (did we run jobs that were duplicates?), and latency (does the detection step add unacceptable delay to job submission?). Build a regression test suite with at least 20 labeled pairs covering exact duplicates, near-duplicates with noise fields, semantically distinct parameter changes, and edge cases like empty job configs or malformed inputs. If the model's decision is used to automatically skip jobs, implement a human-review queue for low-confidence outputs and a circuit breaker that falls back to manual approval if the detection service becomes unavailable.

This template is the core instruction set for a duplicate job submission detector. It accepts a candidate job configuration and a set of previously submitted jobs, then determines whether the candidate is a duplicate. The prompt is designed to be dropped into an MLOps or data-platform harness that fingerprints job configurations and enforces idempotency before expensive compute is consumed. Adapt the placeholders to match your job schema, fingerprinting strategy, and risk tolerance.

textCopy
You are a duplicate job submission detector for an MLOps platform. Your task is to compare a candidate job submission against a set of previously submitted jobs and determine whether the candidate is a duplicate.

## INPUT

Candidate Job:
[JOB_CONFIG]

Previously Submitted Jobs (most recent first):
[PREVIOUS_JOBS]

## OUTPUT_SCHEMA

Return a JSON object with the following fields:
- "decision": "skip" | "run" | "review"
- "confidence": 0.0 to 1.0
- "matched_job_id": string | null
- "match_type": "exact" | "near_duplicate" | "none"
- "rationale": string (explain the match or why no match was found)
- "diff_summary": string | null (for near-duplicates, summarize the differences)

## CONSTRAINTS

1. An EXACT duplicate means all parameter values in [JOB_CONFIG] are identical to a previously submitted job. Decision must be "skip".
2. A NEAR-DUPLICATE means the job shares the same primary key fields ([PRIMARY_KEY_FIELDS]) but differs in non-critical parameters ([NON_CRITICAL_PARAMETERS]). Decision must be "review".
3. If no match is found, decision must be "run".
4. Ignore timestamp, submission ID, and run ID fields when comparing.
5. Parameter order does not matter; compare by key-value pairs.
6. For numeric parameters, treat values within [TOLERANCE_PERCENT]% as identical.
7. Do not hallucinate matches. If unsure, default to "run" with low confidence.

## EXAMPLES

[FEW_SHOT_EXAMPLES]

## RISK_LEVEL

[RISK_LEVEL]

To adapt this template, replace [JOB_CONFIG] with the serialized candidate job payload and [PREVIOUS_JOBS] with a list of recent submissions from your job store. Define [PRIMARY_KEY_FIELDS] as the set of parameters that constitute a unique job identity—typically the entrypoint, source dataset, and core hyperparameters. List [NON_CRITICAL_PARAMETERS] as the fields that can vary without changing the job's intent, such as logging verbosity or retry counts. Set [TOLERANCE_PERCENT] based on your domain's sensitivity; 0% enforces strict equality, while 1-5% catches parameter drift. Populate [FEW_SHOT_EXAMPLES] with 2-4 labeled examples covering exact match, near-duplicate, and no-match scenarios. Set [RISK_LEVEL] to "low" for read-only jobs, "medium" for jobs with side effects, or "high" for jobs that mutate production data—this influences the model's default conservatism. After copying the template, validate the output against the schema before acting on the decision. For high-risk workflows, route "review" decisions to a human operator and log all "skip" decisions with the matched job ID for auditability.

The duplicate detection prompt is not a standalone safety net—it must be embedded in a harness that enforces idempotency at the application layer before the prompt ever runs. The harness should generate a deterministic fingerprint from the job configuration (hashing parameter keys, values, resource targets, and scheduling metadata) and check an idempotency store (Redis, DynamoDB, or a database table with a unique constraint on the fingerprint) before calling the LLM. The prompt is the near-duplicate and semantic-conflict detector, not the primary deduplication mechanism. Treat the LLM as a second-pass filter that catches what exact-match fingerprinting misses: parameter reordering, equivalent resource identifiers, semantically identical but syntactically different configurations, and jobs submitted under different names that produce the same side effects.

Wire the prompt into a decision function with a clear contract: detect_duplicate(fingerprint, candidate_job, recent_jobs_context) -> {decision, confidence, matched_job_id, reason}. The function first checks the exact-match fingerprint store. On a cache hit, return skip immediately without calling the LLM. On a miss, retrieve the N most recent jobs for the same tenant, team, or resource scope (typically 10–50, depending on submission velocity) and pass them as [CONTEXT] alongside the candidate job as [INPUT]. The prompt returns a structured decision. Validate the output before acting on it: confirm the decision field is exactly skip or run, that confidence is a float between 0.0 and 1.0, and that a matched_job_id is present when the decision is skip. Reject and retry once if validation fails. Log every decision with the fingerprint, model output, and validation result for auditability.

Set a confidence threshold below which the harness defaults to run (or escalates to a human review queue if the job is high-risk). A threshold of 0.85 is a reasonable starting point: if the model is less than 85% confident that a duplicate exists, submit the job and let downstream idempotency guards handle any edge cases. This avoids false-positive skips that silently drop legitimate work. For high-risk batch jobs (data deletion, schema migrations, billing operations), flip the default: require confidence above 0.90 to skip, otherwise escalate to a human approval step. The harness should also enforce a time-bounded deduplication window—jobs older than a configurable TTL (e.g., 30 days) should be excluded from the context to prevent stale matches and keep the prompt's context window manageable.

Choose a model that balances latency, cost, and classification accuracy. For most pipelines, a fast classifier model (Claude Haiku, GPT-4o-mini, or a fine-tuned small model) is sufficient because the task is structured comparison, not open-ended generation. Set a strict timeout (2–5 seconds) and a single retry on timeout or validation failure. If the model is unavailable after retries, fail open (submit the job) and flag the failure for operational review—never block the submission pipeline on LLM availability. Instrument the harness with metrics: duplicate detection latency, skip vs. run ratio, validation failure rate, model timeout rate, and false-positive reports (users flagging skipped jobs that should have run). These metrics are essential for tuning the confidence threshold and detecting prompt drift over time.

Finally, treat the prompt template itself as a configuration artifact that evolves with your job schema. When job parameters change, update the [INPUT] schema description in the prompt and add new examples to the [EXAMPLES] block that cover the new fields. Run regression tests against a golden set of known-duplicate and known-unique job pairs before deploying prompt changes. The harness should support prompt versioning—store the prompt version alongside each decision log entry so you can correlate changes in skip/run behavior with prompt updates during incident review.