Where should the robots.txt file be placed on a website?

The robots.txt file must always be placed at the absolute root of your website domain (e.g., https://example.com/robots.txt) for search engine crawlers to discover and respect its directives.

🤖 Robots Exclusion Protocol (RFC 9309) Builder

Robots.txt Generator & Validator

Build and validate standardized robots.txt files to manage search engine crawlers, protect private directories, and link XML sitemaps.

Default Search Crawler Access (User-agent: *):

Sitemap XML URL:

Disallowed Directories (One per line):

Explicitly Allowed Paths (One per line):

Block Known AI Training Crawlers (GPTBot, CCBot, Google-Extended, Bytespider)

Common Configurations:

Standard Public Site Block AI Crawlers Staging / Development (Disallow All)

Generated robots.txt File Content

Robots Exclusion Protocol (RFC 9309) Directives

User-agent: * : Applies the subsequent rule block to all compliant search engine web crawlers.
Disallow: /admin/ : Instructs crawlers not to request URLs beginning with /admin/.
Sitemap: <URL> : Informs search engines where to find the indexable XML sitemap.

Robots.txt Generator & Validator

Robots Exclusion Protocol (RFC 9309) Directives

Related SEO Tools

🏷️ Meta Tag Generator

📄 HTML Minifier

📋 WHOIS Lookup