Meta-ExternalAgent

The full user-agent string is Meta-ExternalAgent/1.1 or similar versioned variants. It identifies itself in HTTP request headers, allowing web infrastructure engineers to write targeted directives in robots.txt to control its access. This token is distinct from Meta's social media scraper (facebookexternalhit), which is used for link previews on its platforms.

Meta-ExternalAgent crawls the open web to collect publicly accessible text, images, and other media. This data is used to:

• Pre-train and fine-tune Meta's Llama family of large language models.
• Ground responses in Meta AI, its generative assistant integrated across Facebook, Instagram, and WhatsApp.
• Improve multimodal models by ingesting diverse content types. It does not access content behind logins or paywalls.

You can block Meta-ExternalAgent from your entire site or specific paths using the standard Robots Exclusion Protocol. Add the following to your robots.txt:

codeCopy
User-agent: Meta-ExternalAgent
Disallow: /

For partial access, specify allowed or disallowed directories. Meta respects these directives and will cease crawling disallowed paths after re-processing your robots.txt file.

Meta-ExternalAgent crawls from a published set of IP ranges. You can verify legitimate Meta traffic by performing a reverse DNS lookup on the connecting IP, which should resolve to a *.fbsv.net hostname, and a corresponding forward DNS lookup that matches the original IP. Always verify against Meta's official documentation for the current IP list to prevent user-agent spoofing.

The crawler is designed to be polite and rate-limited. It respects the Crawl-Delay directive in robots.txt if specified. Meta-ExternalAgent fetches robots.txt periodically and caches it. Changes to your directives may not take effect immediately. It supports HTTP/1.1 and HTTP/2, and typically crawls with a moderate, distributed frequency to avoid overwhelming origin servers.

Blocking Meta-ExternalAgent via robots.txt is the primary technical mechanism for opting out of having your public content used for Meta's AI training. Meta has also introduced a data subject rights form for users in certain jurisdictions to object to their personal data being used for AI training. This is separate from the crawler-level block and addresses data already ingested.

An autonomous web crawler, identified by a specific user-agent token, deployed by an AI company to collect training data or ground generative responses from the public web. Meta-ExternalAgent is a specific instance of this broader category.

• Purpose: Training foundation models like Llama.
• Identification: Uses the Meta-ExternalAgent token in robots.txt.
• Function: Systematically browses the web to download publicly accessible content.

The standard protocol (robots.txt) used by websites to communicate with web crawlers and specify which parts of the site should not be accessed. This is the primary mechanism for controlling Meta-ExternalAgent.

• Directive: User-agent: Meta-ExternalAgent
• Action: Disallow: / to block completely.
• Granularity: Can block specific directories while allowing others.

The technical and policy mechanisms that allow website owners to signal their preference that their content not be used for training foundation models. Blocking Meta-ExternalAgent via robots.txt is the standard opt-out method.

• Mechanism: robots.txt disallow rule.
• Scope: Prevents future training data ingestion.
• Limitation: Does not remove data already scraped.

OpenAI's specific web crawler user-agent token, functionally equivalent to Meta-ExternalAgent but for OpenAI's ecosystem. Both are managed through identical robots.txt directives.

• Token: GPTBot
• Owner: OpenAI
• Purpose: Training models like GPT-4 and ChatGPT.

A standalone product token used in robots.txt to specifically control whether Google's crawlers can use a site's content for training its generative AI models, including Bard and Vertex AI.

• Token: Google-Extended
• Difference: Separate from the main Googlebot search indexer.
• Control: Allows search indexing while blocking AI training.

A public or private document detailing a website's interactions with AI crawlers like Meta-ExternalAgent, including access frequency, data ingested, and compliance with directives.

• Purpose: Auditing and governance.
• Contents: Crawl frequency, bytes downloaded, robots.txt compliance.
• Value: Verifies that opt-out directives are being respected.