Free Robots.txt Generator
Build allow/disallow rules for crawlers, add your sitemap, and copy or download a ready-to-upload robots.txt.
robots.txt
robots.txt is a request, not a lock
The Robots Exclusion Protocol that robots.txt implements has been around since the mid-1990s, and its defining trait hasn't changed: it's entirely voluntary. A well-behaved crawler — Googlebot, Bingbot, and most legitimate SEO tools — checks robots.txt first and honors what it says. But nothing about the file technically prevents a browser, a script, or a non-compliant scraper from fetching a "disallowed" URL directly; the rules are a request for cooperation, not an access control mechanism. That distinction matters for anything genuinely sensitive — robots.txt is the wrong tool for keeping private content private, since the file itself is publicly readable and simply lists exactly which paths you'd rather crawlers skip, which can inadvertently advertise where sensitive areas of a site live.
Frequently Asked Questions
Where does robots.txt need to go?
In the root of your domain, exactly at https://yourdomain.com/robots.txt — not in a subfolder. Search engines only look for it there.
What's the difference between Allow and Disallow?
Disallow tells a crawler not to fetch matching paths. Allow carves out an exception within a disallowed path — for example, disallowing /private/ but allowing /private/preview/. Everything not disallowed is crawlable by default, so Allow rules are only needed for exceptions.
Does robots.txt stop pages from appearing in Google?
Not reliably. It stops crawling, but a disallowed URL can still get indexed (with no preview) if other sites link to it. To actually keep a page out of search results, use a noindex meta tag on the page instead, and make sure it isn't disallowed so Google can see that tag.
Should I list my sitemap in robots.txt?
Yes — it's the most common way crawlers discover your sitemap.xml automatically, and it's harmless to include even if you've also submitted the sitemap directly in Search Console.
Does robots.txt actually stop anyone from viewing a page?
No — it's a voluntary request that well-behaved crawlers choose to respect, not an access control. A browser, a script, or a non-compliant scraper can still fetch a "disallowed" URL directly; robots.txt was never designed to enforce privacy or security, only to guide cooperative crawlers.