Important: We do not block IP addresses or charge for clicks that are used by Meta's web crawlers, known as "Metabot."
Meta uses dedicated IP addresses for their many of their web crawlers. These crawlers are essential for getting your website indexed, ranked, and generally found online.
These crawlers can also append URLs with click trackers that give the appearance of a click from an ad.
Here's a brief summary of each of Meta's web crawlers:
Meta-ExternalFetcher: The Meta-ExternalFetcher crawler performs user-initiated fetches of individual links to support specific product functions. Because the fetch was initiated by a user, this crawler may bypass robots.txt rules.
Meta-ExternalAgent: The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly.
FacebookExternalHit: The primary purpose of FacebookExternalHit is to crawl the content of an app or website that was shared on Facebook, Instagram, or Messenger. This crawler gathers, caches, and displays information about the app or website such as its title, description, and thumbnail image.
How does Fraud Blocker handle IP addresses from Metabot?
Fraud Blocker does not block any IP address that is from Metabot.
Meta publishes the IP addresses they use for Metabot and Fraud Blocker imports those IP addresses every 2 weeks to ensure we have their latest list (you can download the list yourself with the CSV at the bottom of the link). If any visitor or ad click to your website comes from a Metabot we whitelabel that IP and prevent it from being blocked in your IP exclusions list.
We then display an IP address from Meta's web crawlers as "Metabot" in your reports with the status as "Unblocked."
Note: These crawlers can also append URLs with click trackers that give the appearance of a click from an ad, but we identify them and do not charge your account for these clicks.