What is Robots.txt?
A text file placed at a website's root that instructs search engine crawlers which pages or sections they are allowed or disallowed to crawl.
Robots.txt Explained
Robots.txt is a plain text file located at the root of a website (example.com/robots.txt) that communicates crawling instructions to search engine bots and other web crawlers. Using the Robots Exclusion Protocol, the file specifies which user agents (crawlers) are addressed, which URL paths are disallowed (should not be crawled), and which are explicitly allowed. The file can also reference the location of XML sitemaps. Important distinctions: robots.txt controls crawling, not indexing — a page blocked by robots.txt can still appear in search results if other pages link to it (Google will index the URL without crawling the content). To prevent indexing, use a noindex meta tag instead. Common robots.txt uses include blocking crawlers from admin areas, search result pages, staging environments, duplicate content paths, and resource-intensive pages that waste crawl budget. Misconfigured robots.txt is a frequent source of SEO disasters — accidentally blocking important content sections, CSS/JavaScript files needed for rendering, or entire site sections can devastate search visibility. Always test changes in Google Search Console's robots.txt tester before deploying.
Frequently Asked Questions
Does robots.txt prevent pages from appearing in Google?
Not necessarily. Robots.txt prevents crawling, not indexing. If other websites link to a page blocked by robots.txt, Google may still index the URL and show it in search results — but without being able to crawl the content, the listing will have a minimal or auto-generated description. To prevent a page from appearing in search results, use a noindex meta tag or X-Robots-Tag HTTP header instead.
What are common robots.txt mistakes?
Blocking CSS and JavaScript files (prevents Google from rendering the page properly), blocking entire directories unintentionally (a trailing slash matters), using Disallow: / without realizing it blocks the entire site, not testing changes before deployment, blocking important content pages without realizing it, and confusing crawl blocking with index blocking. Always validate robots.txt with Google's testing tool after any changes.
Should you block AI crawlers in robots.txt?
This depends on your content strategy. If you want to prevent AI companies from training on your content, you can block their crawlers (GPTBot for OpenAI, CCBot for Common Crawl, Google-Extended for Gemini). However, blocking these crawlers does not retroactively remove previously crawled content from training data. Also consider that blocking AI crawlers may eventually affect visibility in AI-powered search features. This is a rapidly evolving area requiring regular policy review.
Related Free Tools
Further Reading
Related Terms
Put robots.txt into practice
TeamBench helps content teams implement robots.txt with custom AI reviewers, scored feedback, and quality gates.
Try TeamBench Free