robots.txt is larger than Google reads (500 KiB)
robots_txt_too_largeShort answer
Your /robots.txt is larger than 500 KiB. Google enforces a maximum file size of 500 kibibytes and ignores everything after it, so any rule written beyond that point does not apply. Shorten the file until it is well under the limit.
Why it matters
Very large robots.txt files are almost always generated: a plugin or script lists individual URLs or parameters one by one. Because Google stops reading at 500 KiB, the rules that end up beyond the limit silently stop working — often the newest ones, which were appended at the end.
How Glimana detects it
At the start of each crawl Glimana downloads /robots.txt from the site root and records its size. The rule fires when the file that answered 200 is larger than 500 KiB (512,000 bytes). The evidence shows the size.
How to fix it
- Find the affected pages in Glimana. Open Issues › robots.txt is larger than Google reads (500 KiB). Every affected page is a row; open one to see the evidence Glimana recorded for it (the measured value, the offending element or the target URL). The same list sits on the task card under Tasks, and the page's own detail view under Pages shows its full signals.
-
Apply the fix on your platform.
Replace long lists of individual URLs with pattern rules (
*and$are supported), merge groups that repeat the same rules for several user agents, and delete rules for sections that no longer exist. Usenoindexrather than robots.txt for pages you want out of the index.Check which plugin writes the rules (SEO, security and cache plugins can all add to robots.txt). Turn off options that list every blocked URL, and replace them with one pattern per section.
Shopify generates a short default file; size problems come from a customised
robots.txt.liquid. Remove loops that print one rule per product or collection and use pattern rules instead.Generate robots.txt from a small set of patterns rather than from a URL list. Add a size check to your build so the file can never exceed a few kilobytes again.
-
Publish and clear caches. Save and publish, then make sure the crawler will see the new version: WordPress — clear the page cache in your cache plugin and purge the CDN; Shopify — theme and content changes go live on save, but purge any CDN in front of the store; custom code — deploy and purge the edge cache. Check in a private window or with
curl -s https://your-domain/page | grep -i '<title\|canonical\|robots'that the live HTML has changed; Glimana reads what the server sends, not what your browser has cached. - Verify in Glimana. Open the task under Tasks and click Mark as done. The next crawl measures robots.txt again, and the task closes when it is under 500 KiB.
How the fix is verified
The next crawl downloads robots.txt again; the issue closes when the file is under 500 KiB.