Crawling, robots.txt, crawl controls and sitemaps
robots.txt and sitemap structure are sound. This cluster grew twice during testing, because verification proved the original list of 85 had holes: 1.8.1 (sitemap extension schemas) and 1.8.2 (a sitemap entry must name a live URL). Each new claim was written from the governing documentation, not from the defect that exposed the gap. Both carry the cluster's Gaps. 1.9 moved from Gap to Pass on re-measurement after a fix.
robots.txt sits at the host root: HTTP 200, text/plain, valid UTF-8. The http:// and non-www variants 301-redirect to the canonical host.
Four disallow rules, and none matches any of the 25 priority URLs. robots.txt is used for crawl control, not de-indexing.
No page serves noindex (meta or X-Robots-Tag), so the disallow-hides-noindex trap is not in play.
No allow rules and no overlapping disallows, so no rule-specificity conflict exists to get wrong.
All robots.txt rules and all 25 priority URL paths are lowercase, so there is no case-variance exposure.
robots.txt is under 1 KB, far below the 500 KiB parse limit.
2 sitemap indexes and 34 child sitemaps declare over 100,000 URLs. The largest file holds 15,000 URLs at 2.7 MB uncompressed, within the 50,000 URL and 50 MB limits. All files are UTF-8, and all 25 priority pages are present.
All locs are absolute, entity-escaping holds under strict XML parsing, and the urlset/url/loc structure is correct in all 36 files. The same run validated Google's extension namespaces and found an element-order defect. That became claim 1.8.1, added to close the hole in the list, and it carries the Gap.
Every entry in the video sitemap lists its children as title, description, thumbnail_loc, player_loc, while the governing XSD requires thumbnail_loc first. All required elements are present with valid values, so the failure is ordering only. Google's live parser tolerates it today, so impact is low. The exposure is strict consumers and forward compatibility. This claim was added during testing: the original list of 85 did not cover it.
Five URLs the sitemaps declare return 404 to Googlebot. The 404 depends on user agent, which matters for anyone re-checking it: a plain curl or a browser gets 200 with a JavaScript app shell, while a Googlebot user agent gets the 404. Search Console still reported one of them as 'Submitted and indexed', so the sitemap was holding a dead URL in the index. These 5 were the only non-200s in the probed set, which covered every URL outside the largest page class plus a deterministic sample of that class. This claim was added during testing, because none of the original 85 said anything about the state of the URL a sitemap entry names.
Re-verified after the fix. In two sweeps of the 25 priority pages, 33 minutes apart, none served a 'Loading...' placeholder in the crawl-stage Googlebot HTML, and all 25 passed the tool's rule: 200 status, at least 100 words in <main>, no placeholder. Two things are not claimed. Spinner placeholders elsewhere on the pages are reduced, not gone. And outside the 25, 3 of the 91 previously failing URLs still served a placeholder, in a variant our own per-page check could not detect, which was a blind spot on our side. The two sweeps also returned identical ETags, so they show stability within one cached snapshot, not two independent prerenders.
Corrected from Gap to Pass after the development team's fix shipped. Before the fix, 7 of 25 priority pages failed. The verdict moved on the same tool and the same deciding field that produced the Gap.
robots.txt carries a Sitemap: line pointing at the absolute URL of the root sitemap index.
Want this run on your site?
We test your site against the same claims, record a verdict and the evidence for each one, and tell you what to fix first.
Start a conversation