Page that gets clicks is blocked by robots.txt
robots_blocked_with_clicksShort answer
These pages received clicks from Google in the last 28 days, but robots.txt disallows crawling them. The page gets clicks but is blocked by robots.txt: Google can't read the content, can't build a title and description, and can't see updates. Since the page gets clicks from search, blocking it is probably accidental: narrow the matching Disallow pattern in robots.txt.
Why it matters
A robots.txt block does not remove a page from the index; Google can still list it from links alone, with a snippet like "No information is available for this page". It does stop Google from reading the content, so the ranking slowly decays and any update you make is never seen. A pattern such as Disallow: /blog meant for /blog-drafts/ also catches /blog/.
How Glimana detects it
Glimana takes pages with clicks in Search Console (web search, last 28 days), fetches robots.txt and evaluates it for Googlebot. The rule fires for pages with clicks that match a Disallow. The evidence shows the URL, the clicks and the matching rule.
How to fix it
- Find the affected pages in Glimana. Open Issues › Page that gets clicks is blocked by robots.txt. This is a site-level check, so the single row carries the full evidence: the URLs, values or days involved. The task under Tasks shows the same list and the expected gain.
-
Apply the fix on your platform.
- Find the matching line in robots.txt (the evidence names it).
- Narrow it.
Disallow: /blog→Disallow: /blog-drafts/;Disallow: /*?→Disallow: /*?sort=; add an explicitAllow:for the path if the pattern must stay. - Test with the robots.txt report in Search Console or Glimana's crawl.
- Request indexing of the affected pages so the snippet is rebuilt.
Edit robots.txt in the SEO plugin's file editor. Common accidental blocks:
Disallow: /wp-content/(blocks images and CSS),Disallow: /*.php$, and a leftoverDisallow: /from staging.Edit
templates/robots.txt.liquid; Shopify's defaults are safe, so a block on a page with clicks is a custom addition.Edit the file; use path-prefix rules with a trailing slash and test with a robots.txt parser (Google's open-source
robotstxtlibrary) before deploying. -
Publish and clear caches. Save the file, then purge
/robots.txt(and your sitemap URLs) at the CDN: Cloudflare and most CDNs cache them for up to a day. Confirm withcurl -s https://your-domain/robots.txtthat the live file shows the change. - Verify in Glimana. Open the task under Tasks and click Mark as done. The affected pages are queued for a verification crawl within a few hours, and the task closes when the page is no longer disallowed. If the check still fails, the task returns to New with a note; when it passes, the task is listed under Resolved technical issues on Impact reports. To check sooner, use Re-crawl affected pages on the issue.
How the fix is verified
Re-crawl; the task closes when the page is no longer disallowed.