robots.txt Explained: What to Block, What to Never Block

Why this small file matters so much

robots.txt is the first thing a crawler requests when it arrives at your domain, before it looks at a single page. It sits at the root, it is plain text, and on most sites it is shorter than this paragraph.

It is also the file with the widest gap between how much attention it gets and how much damage it can do. A single stray Disallow: / on a live site — a line carried over from a staging environment, most often — removes that site from search results within days, and it will look completely normal to every human who visits.

No other four-character line on a website can cost you your entire organic traffic in a week.

Crawling is not indexing

This is the distinction that everything else depends on, and it is the one that catches people out.

robots.txt controls crawling. It tells a crawler which URLs it may request. That is all it does.

It does not control indexing. If other sites link to a page you have blocked, Google can still list that URL in search results — it just cannot describe it, so you get a bare URL with a note saying no information is available. Blocking made the listing worse, not absent.

The corollary is the part that surprises people. To keep a page out of the index you need <meta name="robots" content="noindex"> on the page. And for Google to read that tag, it must be allowed to crawl the page. Blocking a page in robots.txt actively prevents the noindex tag from ever being seen. Doing both does not double the protection; it cancels it.

Every directive, explained

User-agent

Which crawler the following rules apply to. User-agent: * means all of them. You can target a specific one — Googlebot, Bingbot — and a crawler obeys the most specific block that names it, ignoring the wildcard block entirely. That last detail causes a lot of accidental behaviour.

Disallow

A path prefix that must not be crawled. Disallow: /admin/ blocks everything beneath it. Disallow: / blocks the whole site. An empty Disallow: blocks nothing, which is a confusing but valid way of saying “crawl everything”.

Allow

Carves an exception out of a Disallow. The classic pattern is blocking a directory while permitting one file inside it. More specific rules win over less specific ones, regardless of order.

Sitemap

Declares where your XML sitemap lives. It must be a full absolute URL including https://, not a relative path. This is the single best reason to have a robots.txt at all even when you are blocking nothing — it is how crawlers that have never met you find the map of your site.

Crawl-delay

A non-standard request to slow down. Google ignores it completely; Bing and Yandex honour it. If Googlebot is genuinely overloading your server, adjust the crawl rate in Search Console instead — and consider whether the server is the actual problem.

What belongs in yours

A small business or brochure site

Almost nothing. Allow everything, declare your sitemap, done. Two lines. Sites with fifty pages do not have a crawl budget problem, and every Disallow you add is one more thing that can be wrong.

WordPress

Block /wp-admin/ while allowing /wp-admin/admin-ajax.php, which plenty of front-end functionality depends on. Do not block /wp-content/ — that is where your themes, images and stylesheets live, and blocking it is the single most common self-inflicted WordPress SEO wound.

Ecommerce

This is where the file earns its keep. Faceted navigation generates effectively unlimited URL combinations — colour, size, price, sort order, page number — and a crawler will happily spend your entire budget on them. Block the parameter patterns that produce no unique content, and handle near-duplicates with canonicals rather than blocks, since a blocked page's canonical can never be read either.

Sites with internal search

Block the search results path. Internal search pages are infinite, thin, and occasionally embarrassing when indexed — they can surface whatever anybody has ever typed into your search box.

Staging and development

Disallow: / is correct here, and it is exactly the line that must not travel to production. Better still, put staging behind HTTP authentication, which cannot be accidentally deployed and actually keeps people out.

Three things to never block

1. CSS and JavaScript

Google renders pages to see them as a visitor does. Blocking your stylesheets or scripts means it renders a broken page and evaluates that, including for mobile usability. This was standard advice fifteen years ago and it is actively harmful now — and it is still sitting in a great many inherited robots.txt files.

2. Anything you want removed from the index

Covered above, and worth repeating because it is counterintuitive. Blocking prevents the noindex tag from being read. If a page is already indexed and you want it gone, allow crawling, add noindex, wait for a recrawl, and only then consider blocking it.

3. Anything genuinely sensitive

The file is public at yourdomain.com/robots.txt and it is one of the first places anyone probing your site will look. Disallow: /internal-client-files/ is a signpost, not a lock. Use authentication for anything that needs protecting, and keep it out of robots.txt entirely.

Why generate rather than copy

Most robots.txt files on the internet were copied from another site, which means most of them contain rules for a platform their owner does not run, blocking directories that do not exist, and occasionally blocking one that does.

The generator here asks what you actually want — allow everything, block everything, or allow everything except specific paths — and produces a valid file with your sitemap declared. Every directive comes with a plain explanation of what it tells crawlers to do, so you finish with a file you can defend rather than one you inherited.

It runs entirely in your browser, needs no account, and stores nothing. Once you have the file, upload it to the root of your domain so it is reachable at yourdomain.com/robots.txt — nowhere else works, and each subdomain needs its own.

Then check the sitemap you declared in it actually loads, with the sitemap checker. Declaring a sitemap that returns a 404 is a surprisingly common way to make a correct robots.txt useless.

FAQ