The short version
Disallow: /deployed from staging is the most damaging single line in technical SEO, and it is invisible to everyone except crawlers.- Recovery is not instant. Crawling resumes quickly; re-indexing takes days to weeks, and rankings can take longer to return.
- A blocked
/wp-content/or/assets/means Google renders a broken page and evaluates it that way. - Blocking a page and adding
noindexto it cancels the noindex — the tag can never be read. - Rebuild the file deliberately instead of editing one you inherited.
Why this failure is invisible
Every other serious website problem announces itself. A broken deploy throws errors. A failed payment integration generates complaints. A slow page produces impatient customers.
A wrong robots.txt produces nothing. The site loads perfectly. Every page works. Nobody phones. The only symptom is a line on a graph in a tool most small businesses check monthly at best, and by the time somebody notices the drop, the cause is four weeks and three deploys behind them.
That invisibility is the whole problem. The fix takes thirty seconds; the detection takes a month.
Seven ways it goes wrong
1. Disallow: / shipped from staging
The catastrophic one. Staging environments block everything, correctly. Then the site is deployed and the file goes with it, because it lives in the repository alongside everything else.
Symptom: impressions falling toward zero over one to two weeks; pages
disappearing from search; Search Console showing “Blocked by robots.txt”
across the board. Check: open
yourdomain.com/robots.txt right now. It takes five seconds and I would
recommend doing it before finishing this article.
2. Blocking the folder your assets live in
Disallow: /wp-content/ or Disallow: /assets/, usually added to
“keep the uploads folder out of search”. It also blocks every stylesheet,
script and image the page needs to render.
Symptom: Search Console's URL Inspection rendering your page as unstyled text; mobile usability warnings on pages that are demonstrably responsive.
3. Blocking a page you also marked noindex
Belt and braces, and the braces cut the belt. The crawler is not permitted to fetch the
page, so it never sees the noindex tag, so the page stays in the index
— as a bare URL with no description.
Symptom: a page you removed months ago still listed, with no snippet under it.
4. A wildcard that matches more than intended
Disallow: *? to block query strings also blocks every legitimate URL carrying
a parameter — pagination, filters, language switches, and any page somebody shared
with a UTM string attached.
Symptom: paginated and filtered pages missing from the index while the unparameterised versions remain.
Free tool · Every directive explained
Generate the file instead of copying someone else’s
Pick the access rule, add the paths you want kept out of crawling, declare your sitemap, and copy a valid file — with a plain explanation of what each line is telling crawlers to do.
5. A specific user-agent block that overrides the wildcard
A crawler obeys the most specific group that names it, and ignores User-agent:
* entirely when such a group exists. So a User-agent: Googlebot block
added years ago for one rule silently exempts Google from every rule in your wildcard
group.
Symptom: Google crawling things you thought were blocked, while Bing respects them.
6. A sitemap line pointing nowhere
The sitemap moved, the domain changed, or the path was relative rather than absolute. The declaration remains, pointing at a 404.
Symptom: nothing dramatic — just slower discovery of new pages, which is exactly the sort of failure that persists for years. Verify with the sitemap checker.
7. The file returning the wrong status code
A 404 is fine — crawlers read it as “crawl everything”. A 5xx server error is not. Google treats a persistently failing robots.txt as a temporary instruction to stop crawling the site entirely, on the reasonable assumption that it should not proceed without knowing the rules.
Symptom: crawl rate collapsing with no change to the file's contents, because the contents were never the problem.
Use these together
- SEO Page Checker — Check any live page for title, description, headings, canonical, robots, alt text and internal links.
- Sitemap URL Checker — Fetch an XML sitemap and check it is valid, reachable and free of the usual structural mistakes.
- Page Performance Checker — Scan a live site for performance, SEO, conversion readiness and trust signals, and get a score out of 100.
- The full set of twelve — free, no signup, and for nine of them nothing you type leaves your browser.
What recovery actually looks like
The fix is thirty seconds. The recovery is not, and it is worth setting expectations honestly because the waiting period is when people panic and make things worse.
- Correct the file and deploy. Confirm by loading it in a private browser window — caches and CDNs sometimes serve the old version for hours.
- Test it in Search Console's robots.txt report, which shows you what Google currently holds. It refetches roughly daily.
- Request indexing for your most important pages via URL Inspection. This does not fix the site, but it accelerates the pages that matter most.
- Wait. Crawling resumes within days. Re-indexing takes days to weeks depending on site size and authority. Rankings usually return close to where they were, but not always immediately.
- Do not change anything else in the meantime. If you rewrite content and rebuild internal links during the recovery, you will never know what recovered and what you disturbed.
One reassurance: this is recoverable. A site blocked for two weeks generally comes back. The damage scales with duration, which is the argument for checking rather than assuming.
How to stop it happening again
- Do not use robots.txt to protect staging. Use HTTP authentication. It cannot be accidentally deployed and it actually works.
- Generate the production file separately from the staging one, and treat any difference as deliberate rather than accidental.
- Check it after every deploy that touches the root. Five seconds in a browser tab.
- Turn on Search Console email alerts. Indexing coverage warnings are the earliest signal you will get for free.
- Write down what each Disallow is for, as a comment in the file. A rule nobody can explain is a rule that will eventually be either deleted or trusted, and both are risks.
Why rebuild rather than patch
Most robots.txt files are archaeology: a line from a 2019 developer, a block copied from a tutorial, a directive for a plugin that was removed. Editing that file means reasoning about rules whose purpose nobody remembers, which is how a correct-looking edit produces an incorrect result.
Rebuilding takes two minutes and produces a file you can account for line by line. The generator here asks what access rule you want, takes the paths you genuinely need kept out of crawling, declares your sitemap properly with an absolute URL, and explains every directive as it writes it. Valid syntax is guaranteed, which removes an entire category of silent failure.
It runs in your browser, stores nothing, and needs no account. Then confirm the result on the live site with the SEO page checker, which reports whether a given page is actually crawlable and indexable — because the file being correct and the page being reachable are two different claims, and only one of them can be verified by reading the file.
FAQ
How quickly does a wrong robots.txt affect rankings?
Crawling stops almost immediately once the file is refetched, typically within a day. Pages then drop out of the index over the following days to weeks as Google revisits them and finds them unreachable. A large site can lose visibility noticeably within a week.
How long does recovery take after fixing it?
Crawling resumes within days. Full re-indexing takes days to weeks depending on how large the site is and how often it was crawled before. Requesting indexing for your most important pages in Search Console speeds up those specific pages but not the site as a whole.
Can a 404 on robots.txt hurt me?
No. Crawlers treat a missing robots.txt as permission to crawl everything, which is usually what you want anyway. A 5xx server error is the dangerous case — Google may stop crawling entirely rather than proceed without knowing the rules.
Why is my page still in Google after I blocked it?
Because blocking prevents crawling, not indexing. If other pages link to it, Google
can list the URL without a description. To actually remove it, allow crawling and add
a noindex meta tag so the instruction can be read, then wait for a
recrawl.
Should I check robots.txt after every deployment?
If the deployment touches the site root, yes — it takes five seconds in a browser. The failure mode is completely invisible otherwise, and the cost of missing it for a month is far larger than the cost of checking.
Thejas Acharya
Digital growth consultant based in Bangalore, working with founders and marketing teams across India and the US. I build websites, landing pages, Meta Ads campaigns and AI automation that are judged on one thing: whether they generate leads. The free tools on this site are the checks I run on client work, opened up.