Broken Link Detection Prompt for Documentation Sites

This prompt is designed for doc site maintainers and platform engineers who need to audit internal and external links across documentation pages at scale. The core job-to-be-done is transforming raw HTTP validation results into a categorized, actionable remediation report. The ideal user is a technical writer or DevOps engineer integrating this into a CI/CD pipeline where link extraction and HTTP status checks are already handled by application code (e.g., a crawler or a script using wget or curl). The model's role is not to crawl the web, but to analyze a pre-compiled list of URLs and their statuses, identify patterns like redirect chains or broken anchors, and suggest specific replacement URLs or fixes.

You should use this prompt when you have a structured input of URLs with their HTTP status codes and any error messages. The prompt expects a specific [INPUT] format, typically a JSON array of objects containing the source page, the target link, the HTTP status, and any redirect chain or anchor validation result. It is most effective in a gated deployment where a human reviews the model's suggested replacements before they are committed to the documentation repository. This is not a real-time link checker for a live site; it is a batch analysis tool for pre-publication audits or scheduled integrity checks.

Do not use this prompt for crawling or discovering links. It will fail if asked to fetch URLs or if provided with unvalidated lists. It is also unsuitable for auditing authenticated endpoints unless the input data already includes the results of authenticated requests, as the model cannot perform login flows. Avoid using this prompt for pages with dynamic, JavaScript-rendered content unless the input data comes from a browser-based crawler that has already resolved the final DOM state. For high-risk documentation like security protocols or legal terms, always require human approval on any suggested link changes before publication.

This template is the core instruction set for a broken link detection workflow. It expects a list of documentation page URLs and a set of constraints that define what constitutes a broken link in your environment. The prompt instructs the model to produce a structured report categorizing each link by status code, redirect chain depth, anchor validity, and suggested replacements. Use this template as the foundation for a tool-calling agent or a batch processing pipeline that validates links against live HTTP responses.

textCopy
You are a documentation link auditor. Your task is to analyze a list of URLs from a documentation site and produce a categorized broken link report.

## INPUT
- Page URLs to audit: [PAGE_URLS]
- Optional: previously crawled link graph or sitemap: [LINK_GRAPH]

## CONSTRAINTS
- Follow redirects up to [MAX_REDIRECT_DEPTH] hops. Report any chain exceeding this depth as broken.
- Treat HTTP status codes [BROKEN_STATUS_CODES] as broken links.
- Ignore links matching these patterns (e.g., authenticated endpoints, dynamic routes): [IGNORE_PATTERNS]
- Check anchor targets (`#fragment`) against the destination page's heading IDs. Report anchors with no matching ID as broken.
- For external links, do not scrape the full page; rely on the HTTP status code and, if available, a HEAD request.
- If a link returns a status code in [WARNING_STATUS_CODES] (e.g., 301, 308), flag it as a warning with the redirect target.

## OUTPUT_SCHEMA
Return a JSON object with this exact structure:
{
  "audit_summary": {
    "total_links_checked": <int>,
    "broken_links": <int>,
    "warnings": <int>,
    "skipped_links": <int>
  },
  "broken_links": [
    {
      "source_page": "<URL of the page containing the link>",
      "link_text": "<visible link text>",
      "target_url": "<full href target>",
      "status_code": <int or null>,
      "error": "<connection error, timeout, or DNS failure description if no status code>",
      "redirect_chain": ["<url1>", "<url2>"],
      "anchor_valid": <true | false | null>,
      "category": "<404 | redirect_loop | anchor_missing | connection_error | timeout | blocked_by_robots | other>",
      "suggested_replacement": "<proposed fix or null>"
    }
  ],
  "warnings": [
    {
      "source_page": "<URL>",
      "target_url": "<URL>",
      "status_code": <int>,
      "redirect_target": "<final destination URL>",
      "category": "<permanent_redirect | temporary_redirect | soft_404 | other>"
    }
  ],
  "skipped_links": [
    {
      "source_page": "<URL>",
      "target_url": "<URL>",
      "reason": "<matched ignore pattern or authentication requirement>"
    }
  ]
}

## INSTRUCTIONS
1. For each source page in [PAGE_URLS], extract every `<a href="...">` link.
2. Filter out links matching [IGNORE_PATTERNS] and record them in `skipped_links`.
3. For each remaining link, perform an HTTP HEAD or GET request (respecting `Retry-After` headers).
4. Follow redirects up to [MAX_REDIRECT_DEPTH]. Record the full chain.
5. Classify the result into `broken_links` or `warnings` based on [BROKEN_STATUS_CODES] and [WARNING_STATUS_CODES].
6. For same-domain links with anchors, verify the anchor exists on the destination page.
7. Populate `suggested_replacement` only when a likely fix is obvious (e.g., a 404 where the correct page is known from the link graph).
8. If the link target is blocked by robots.txt, categorize it as `blocked_by_robots` and do not attempt to fetch.

## RISK_LEVEL
[RISK_LEVEL]

If RISK_LEVEL is "high", do not attempt to fetch any link that requires authentication, contains session tokens, or points to internal admin endpoints. Log these as skipped with the reason "high_risk_skip".

Adapting the template: Replace each square-bracket placeholder with concrete values before sending the prompt to a model or wiring it into an application. [PAGE_URLS] should be a JSON array of fully qualified URLs. [BROKEN_STATUS_CODES] is typically [404, 410] but may include [403, 500] if your team treats those as actionable failures. [IGNORE_PATTERNS] is critical for avoiding false positives from authenticated endpoints—use regex patterns like ["/api/admin/*", "*/login?*"]. [RISK_LEVEL] gates whether the system should attempt to resolve links that might trigger side effects. For documentation sites behind authentication, set this to "high" and combine the prompt with a pre-fetch step that strips session cookies.

Next steps: After adapting the placeholders, pair this prompt with a deterministic link extractor and an HTTP client that performs the actual requests. The model should receive the extracted links and HTTP response metadata, not raw HTML pages. This separation keeps the prompt focused on classification and reporting while the application layer handles network I/O, retries, and rate limiting. Before deploying, run the prompt against a known set of pages with intentionally broken links and verify that the output JSON matches the expected schema and correctly categorizes each failure mode.

The broken link detection prompt is designed to be called inside an automated documentation pipeline, not as a one-off chat interaction. You will typically invoke it once per documentation page or once per batch of pages, passing the page's rendered HTML, extracted anchor list, and a list of known-good internal routes as [PAGE_HTML], [ANCHOR_LIST], and [KNOWN_ROUTES] respectively. The model returns a structured JSON report that your harness must validate before merging into your link audit database or ticketing system.

Wire the prompt into a script or microservice that first extracts all <a> tags and their href attributes from each documentation page. For internal links, resolve them against [KNOWN_ROUTES] and your site's routing table before sending them to the model; the model should not guess whether /docs/v2/api is valid. For external links, perform a lightweight HEAD request to capture the HTTP status code, redirect chain length, and final destination before passing the result to the prompt as [EXTERNAL_STATUS_MAP]. This pre-processing reduces hallucination risk and lets the model focus on classification, anchor validity, and replacement suggestions rather than raw HTTP mechanics. After the model returns its JSON report, validate every url field against the original input set to catch fabricated links, and reject any report where confidence falls below your configured threshold (start with 0.7). Log every rejected report for later review.

For high-traffic documentation sites, batch pages into groups of 10-20 and run the prompt concurrently with a rate limiter. Use a model that supports structured output (JSON mode or function calling) to enforce the output schema defined in the prompt template. Implement a retry layer: if the model returns malformed JSON or missing required fields, retry once with the same input and an explicit repair instruction appended. If the second attempt fails, flag the page for human review and continue processing the rest of the batch. Store all results in a link audit table with columns for page_url, link_url, status, category, suggested_replacement, confidence, model_version, and reviewed_by. This schema lets you track false positive rates over time and feed corrections back into your evaluation dataset. Avoid wiring this prompt directly into a CI/CD block step on every commit; instead, run it as a scheduled audit (daily or weekly) or on-demand before a major release, since external link rot is slow-moving and full-site scans are expensive.