robots.txt Redirect Handling

Compliant crawlers must follow HTTP redirects when fetching robots.txt. The RFC 9309 standard specifies that a crawler should follow at least five consecutive redirects to resolve the final location of the file. This prevents broken configurations where a site migration or URL restructuring inadvertently blocks all crawling. A crawler encountering a 301 Moved Permanently or 302 Found status on /robots.txt must issue a new GET request to the Location header value.

A critical security and sovereignty boundary exists for cross-origin redirects. If a request to https://example.com/robots.txt redirects to a different origin (e.g., https://cdn-provider.net/robots.txt), the crawler must treat the file as applying only to the target origin. The rules fetched from the second domain do not govern crawling on the first. This prevents a compromised or misconfigured CDN from injecting permissive rules for an unrelated primary domain.

Crawlers should interpret the redirect type to optimize future requests. A 301 Moved Permanently signal instructs the bot to update its stored reference and request the new URL directly on subsequent fetches, bypassing the redirect chain. A 302 Found or 307 Temporary Redirect indicates the original URL should be retried on the next crawl cycle. This distinction preserves crawl budget and reduces latency by avoiding unnecessary round trips.

If a redirect chain exceeds the maximum hop limit or results in a non-2xx final status, the crawler must treat the robots.txt as unreachable. The standard behavior is to assume full disallow (Disallow: /) for the site until a valid file can be retrieved. This fail-closed posture protects the origin server from unbounded crawling when the access policy cannot be determined. Common failure triggers include redirect loops (301 pointing to itself) and 404 Not Found on the final target.

The Location header in a redirect response may contain a relative URI reference (e.g., /new-path/robots.txt) or an absolute URI (e.g., https://www.example.com/new-path/robots.txt). Crawlers must resolve relative references against the original request URI according to RFC 3986. A common misconfiguration involves a relative redirect that resolves to an unintended path, causing the crawler to fetch a non-existent file and default to a full disallow state.

The caching duration for a robots.txt file fetched via redirect is governed by the final response's Cache-Control header, not the intermediate redirect responses. A 301 redirect with a long max-age on the final 200 OK response instructs the crawler to cache the new URL mapping for the specified duration. During this period, the crawler will skip the redirect entirely and request the target URL directly, reducing server load and latency for subsequent crawl cycles.

The Robots Exclusion Protocol standard mandates that compliant crawlers must follow HTTP redirects when fetching robots.txt. A 301, 302, 307, or 308 status code triggers the crawler to request the file from the Location header URL. The standard limits redirect chains to 5 hops to prevent infinite loops. After 5 consecutive redirects, the crawler must treat the robots.txt as unavailable (500 error equivalent) and proceed with full access. Non-compliant bots may ignore redirects entirely, treating the original URL as disallowed.

Redirects occur in several production scenarios:

• HTTP to HTTPS migration: http://example.com/robots.txt → https://example.com/robots.txt
• Domain canonicalization: example.com → www.example.com or vice versa
• CDN or proxy layer: Origin server redirects through edge nodes
• Temporary maintenance: 302 redirect to a static fallback robots.txt
• Multi-region routing: Geo-IP based redirects to region-specific directives

Each redirect must preserve the User-Agent header so the target server can serve bot-specific rules.

RFC 9309 specifies a hard limit of 5 consecutive redirects during a single robots.txt fetch attempt. Key behaviors:

• Redirect counter resets only on a new fetch cycle
• Relative Location URLs must be resolved against the original request URI
• Cross-origin redirects are permitted; the crawler follows to any domain
• Redirect loops (A→B→A) exhaust the limit and result in a 500-equivalent failure
• Non-HTTP redirects (meta refresh, JavaScript) are not followed

Exceeding the limit triggers a transient error state; the crawler retries on the next scheduled fetch.

Crawlers cache robots.txt content based on the Cache-Control and Expires headers from the final response in the redirect chain. Critical considerations:

• Intermediate redirect responses do not influence cache duration
• A 301 Permanent Redirect may cause crawlers to cache the redirect target indefinitely
• 302/307 Temporary Redirects signal that the redirect is transient; crawlers should re-check the original URL on subsequent fetches
• Googlebot typically caches robots.txt for up to 24 hours unless headers specify otherwise
• A max-age=0 or no-cache directive forces revalidation on every fetch

When redirect handling fails, crawler behavior depends on the error type:

• Redirect chain exceeded (5 hops): Treated as 500 Internal Server Error → full crawl allowed
• Redirect to 4xx: The 4xx status on the final URL is honored; 404 means full allow, 403 means full disallow
• Redirect to unreachable host: DNS failure or timeout → treated as 500, full crawl allowed
• Redirect loop detected: Crawler may short-circuit before 5 hops if a cycle is identified
• Invalid Location header: Missing or malformed URL → treated as 500

Best practice: Monitor server logs for redirect chains to robots.txt and keep them to 1 hop maximum.

Validate your robots.txt redirect configuration using these methods:

• Google Search Console robots.txt Tester: Simulates Googlebot's redirect-following behavior
• curl with -L flag: curl -L -A "Googlebot" https://example.com/robots.txt traces the full redirect chain
• Server access logs: Verify that the final robots.txt endpoint receives requests with the correct User-Agent
• Staging environment testing: Use a temporary 302 redirect to test new rules before deploying permanent changes
• Redirect path auditing: Ensure no intermediate hop returns a 200 OK with HTML content instead of a proper redirect