Robots.txt Generator
Build a valid robots.txt in a few clicks, with every directive explained so you know exactly what you are telling crawlers to do.
Build your robots.txt
The file updates as you choose.
Your robots.txt
What each directive means
User-agent
Which crawler the rules below apply to. * means all of them. Rules are grouped: a crawler obeys the most specific block that names it and ignores the rest, so a Googlebot block completely replaces the * block for Google rather than adding to it.
Disallow
A path prefix crawlers should not request. Disallow: / blocks everything; an empty Disallow: blocks nothing.
Disallow: /admin/
Allow
Carves an exception out of a Disallow rule. When both match, the longer, more specific rule wins. Disallow: /files/ · Allow: /files/public/
Sitemap
Tells crawlers where your sitemap is. It sits outside any user-agent group and applies to everyone. It must be an absolute URL. Sitemap: https://example.com/sitemap.xml
Crawl-delay
Seconds a crawler should wait between requests. Non-standard: Bing and Yandex respect it, Google does not. Crawl-delay: 10
Wildcards
* matches any run of characters and $ anchors the end of a URL. Both are supported by the major engines.
Disallow: /*?sort= · Disallow: /*.pdf$
How it works
- Choose the access rule: allow everything, block everything, or allow everything except specific paths.
- Add the paths you want kept out of crawling — one per line, each starting with a forward slash.
- Add your sitemap URL. This has to be the full address including
https://, not a relative path. - Copy or download the file and upload it to the root of your domain, so it is reachable at
yourdomain.com/robots.txt.
Why this matters
robots.txt is the first file a crawler asks for. Get it wrong and the consequences are immediate and total — a stray Disallow: / on a live site removes it from search results within days.
It is worth being clear about what the file does and does not do. It controls crawling, not indexing. A page blocked here can still appear in search results if other sites link to it; Google just shows it without a description. To keep a page out of the index you need a noindex robots meta tag — and for Google to see that tag, the page must not be blocked in robots.txt.
It is also public. Anyone can read yours. Listing /admin-secret/ in it advertises the path rather than protecting it. Use authentication for anything that genuinely needs protecting.
Questions
Where does robots.txt go?
The root of the domain, and nowhere else: https://example.com/robots.txt. Crawlers do not look anywhere else, and a file at /blog/robots.txt is ignored entirely. Each subdomain needs its own.
Does blocking a page in robots.txt remove it from Google?
No — it stops Google reading the page, which can leave a URL-only result in place with no description. To remove a page from the index, allow it to be crawled and add <meta name="robots" content="noindex">. Blocking it in robots.txt actually prevents that tag from being seen.
Do I need a robots.txt file at all?
Not strictly. A missing file returns a 404 and crawlers treat that as "crawl everything", which is usually what you want. It is still worth having one, if only to declare your sitemap.
Should I block my CSS and JavaScript?
No. Google renders pages to see them the way a visitor does. Blocking your stylesheets or scripts means it renders a broken page and judges it accordingly. This was common advice fifteen years ago and it is actively harmful now.
What is Crawl-delay?
A non-standard directive asking crawlers to slow down. Google ignores it entirely; Bing and Yandex honour it. If Googlebot is overloading your server, adjust the crawl rate in Search Console instead.