Free Robots.txt Generator
Build allow/disallow rules for crawlers, add your sitemap, and copy or download a ready-to-upload robots.txt.
robots.txt
Frequently Asked Questions
Where does robots.txt need to go?
In the root of your domain, exactly at https://yourdomain.com/robots.txt — not in a subfolder. Search engines only look for it there.
What's the difference between Allow and Disallow?
Disallow tells a crawler not to fetch matching paths. Allow carves out an exception within a disallowed path — for example, disallowing /private/ but allowing /private/preview/. Everything not disallowed is crawlable by default, so Allow rules are only needed for exceptions.
Does robots.txt stop pages from appearing in Google?
Not reliably. It stops crawling, but a disallowed URL can still get indexed (with no preview) if other sites link to it. To actually keep a page out of search results, use a noindex meta tag on the page instead, and make sure it isn't disallowed so Google can see that tag.
Should I list my sitemap in robots.txt?
Yes — it's the most common way crawlers discover your sitemap.xml automatically, and it's harmless to include even if you've also submitted the sitemap directly in Search Console.