Skip to content

robots.txt

robots.txt is a host-level file that tells crawlers which paths they may fetch; it controls crawling and not indexing.

A disallowed URL can still appear in search results, and blocking it prevents crawlers from ever reading a noindex directive on that page. Continue with SEO metadata that earns the click for the longer explanation.

Related tools

Related articles