Applebot-Extended

Apple employs a two-tier bot system to separate indexing from training. The standard Applebot crawls for Siri Suggestions, Spotlight Search, and Safari Reading List. Applebot-Extended is a secondary, distinct user-agent that specifically controls ingestion for foundation model training. This allows publishers to maintain search visibility while opting out of generative AI training by adding a specific disallow rule for 'Applebot-Extended' without affecting the standard Applebot.

Control is exercised purely through the robots.txt exclusion protocol. To block Apple's training crawler while allowing search functionality, you must target the specific token:

• User-agent: Applebot-Extended
• Disallow: / This syntax explicitly prevents Apple from using your site's data for its generative models. Unlike meta tags which operate on a per-page basis, this directive provides domain-wide coverage and is the only recognized mechanism for this opt-out.

Applebot-Extended is exclusively concerned with training data acquisition for Apple's foundation models. It does not affect:

• Siri Suggestions or Spotlight Search indexing
• Safari Browsing Data or Top Hit calculations
• Apple Maps evaluation Its sole purpose is to scrape publicly accessible web content to improve the generative capabilities of Apple's models, making it a pure opt-out mechanism for AI training rather than a general search crawler.

You can verify Applebot-Extended activity by analyzing server logs for its user-agent string. It will identify itself clearly in the HTTP request header. Legitimate traffic originates from Apple's published IP ranges (available in Apple's support documentation). To audit compliance:

• Monitor logs for the exact token 'Applebot-Extended'
• Cross-reference IPs against Apple's official CIDR blocks
• Implement crawl anomaly detection to identify spoofed bots mimicking the user-agent

This crawler directly supports Apple Intelligence, the company's personal intelligence system. By controlling Applebot-Extended, publishers decide if their content contributes to features like:

• Writing Tools and summarization
• Image generation capabilities
• On-device model fine-tuning Blocking this bot is a definitive signal that your proprietary content should not be used to enhance Apple's generative AI features, while still allowing your site to be surfaced in traditional search results.

Applebot-Extended follows the Google-Extended precedent, establishing an industry pattern of separate training-control tokens. This granular approach is now considered best practice for AI crawl consent management. Key comparisons:

• Google-Extended: Controls Bard/Vertex AI training
• GPTBot: Controls OpenAI model training
• Anthropic ClaudeBot: Controls Claude model training Publishers should implement a comprehensive content ingestion firewall that addresses all major AI crawler tokens simultaneously.