Allow the facebookexternalhit Bot in the Robots.txt File

Link: https://support.brilliantdirectories.com/support/solutions/articles/12000100532

Facebook uses the facebookexternalhit crawler to retrieve website content when links are shared. To allow this crawler, add the following directives to the Robots.txt file.

User-agent: facebookexternalhit
Disallow: /api/
Allow: /


  • Navigate to Developer Hub.
  • Select Robots File.
  • Add the following directives above the default rules at the end of the Robots.txt file.



Bandwidth


IMPORTANT: Allowing the facebookexternalhit crawler can significantly increase a website's bandwidth usage.


Internal testing has shown that the facebookexternalhit crawler may consume substantially more bandwidth than other common crawlers, such as Googlebot and Bingbot. This increase in traffic may result in higher bandwidth usage and, in some cases, reduced website performance.

 

For this reason, the facebookexternalhit crawler is not allowed by default. It should only be enabled when Facebook link previews are required.


The following example illustrates the difference in bandwidth usage between the facebookexternalhit crawler and other common crawlers over the same time period:

 


For these reasons, we do not allow this bot by default, and only recommend allowing it when absolutely necessary.