robots.txt cannot be fetched (server error)
robots_txt_unavailableShort answer
When Glimana crawled your site, /robots.txt returned a 5xx error, a 429 or no response, and it still failed when we asked again during the audit. Google treats an unreachable robots.txt as “do not crawl”: it stops crawling the whole site, and after a while falls back to an old cached copy. Make the file return 200 (or 404 if you have no rules).
Why it matters
Google reads robots.txt before it crawls anything else. If the file answers with a server error, Google assumes the whole site is disallowed for the first 12 hours, then keeps using the last copy it fetched for up to 30 days, and retries. During that time new and updated pages are not crawled. A 404 is safe (it means “no rules”); a 5xx, 429 or timeout is not.
How Glimana detects it
At the start of each crawl Glimana requests /robots.txt from the site root and records the status code. If it was a 5xx, a 429 or a connection failure, the audit requests the file once more; the issue opens only when the second request fails too, so a momentary blip does not raise a critical alert. While robots.txt is unavailable, Glimana also stops reporting “page that gets clicks is blocked by robots.txt”, because that verdict would come from the temporary block, not from your rules.
How to fix it
- Find the affected pages in Glimana. Open Issues › robots.txt cannot be fetched (server error). Every affected page is a row; open one to see the evidence Glimana recorded for it (the measured value, the offending element or the target URL). The same list sits on the task card under Tasks, and the page's own detail view under Pages shows its full signals.
-
Apply the fix on your platform.
Open
https://your-domain/robots.txtin a browser and withcurl -I. It must answer200with plain text (or404if you want no rules). A 5xx usually means the file is generated by the application and the application is failing; a 403 or 429 for bots usually comes from a firewall or rate-limit rule.If an SEO plugin generates robots.txt (Yoast, Rank Math, All in One SEO), a PHP error in that plugin breaks the file. Check the PHP error log; as a quick fix, upload a static
robots.txtto the site root, which the web server serves before WordPress runs. Make sure security plugins (Wordfence, iThemes) and the host’s bot protection do not rate-limit/robots.txt.Shopify generates robots.txt from
robots.txt.liquid. If you customised that template, a Liquid error can break the file: revert to the default template under Online Store → Themes → Edit code and reapply your rules carefully. Server errors on Shopify’s own file are rare and temporary; contact Shopify support if they persist.Serve robots.txt as a static file from the web server (nginx
location = /robots.txt, Apache) rather than from the application, so an application error cannot take it down. Exclude/robots.txtfrom WAF challenges and rate limits (Cloudflare, Imunify, LiteSpeed). Return404, never5xx, when you have no rules. -
Publish and clear caches. Save and publish, then make sure the crawler will see the new version: WordPress — clear the page cache in your cache plugin and purge the CDN; Shopify — theme and content changes go live on save, but purge any CDN in front of the store; custom code — deploy and purge the edge cache. Check in a private window or with
curl -s https://your-domain/page | grep -i '<title\|canonical\|robots'that the live HTML has changed; Glimana reads what the server sends, not what your browser has cached. - Verify in Glimana. Open the task under Tasks and click Mark as done. The next crawl requests robots.txt again, and the task closes when it returns 200 or 404. Search Console’s robots.txt report (Settings › robots.txt) shows when Google last fetched the file successfully.
How the fix is verified
The next crawl requests robots.txt again; the issue closes when the file returns 200 or 404.