Dew Advisors
← SEO claims board Cluster 01 ยท 12 claims

Crawling, robots.txt, crawl controls and sitemaps

robots.txt and sitemap structure are sound. This cluster grew twice during testing, because verification proved the original list of 85 had holes: 1.8.1 (sitemap extension schemas) and 1.8.2 (a sitemap entry must name a live URL). Each new claim was written from the governing documentation, not from the defect that exposed the gap. Both carry the cluster's Gaps. 1.9 moved from Gap to Pass on re-measurement after a fix.

9
Pass
2
Gap
0
Partial
1
N/A
0
Pending
1.1Pass
A robots.txt file must be a single UTF-8 text file located at the root of the host (e.g. https://www.example.com/robots.txt); it cannot live in a subdirectory, and a given file applies only to the exact protocol, host, and port where it is posted.

robots.txt sits at the host root: HTTP 200, text/plain, valid UTF-8. The http:// and non-www variants 301-redirect to the canonical host.

checked 2026-08-27result site-leveltool curl-http-probe
1.2Pass
robots.txt is for managing crawler traffic, not for keeping a page out of the index: a URL disallowed in robots.txt can still be indexed URL-only (shown without a snippet) if it is linked elsewhere.

Four disallow rules, and none matches any of the 25 priority URLs. robots.txt is used for crawl control, not de-indexing.

checked 2026-08-27result 25 Passtool googlebot-page-check
1.3N/A
For a noindex directive (robots meta tag or X-Robots-Tag header) to work, the page must NOT be blocked by robots.txt; if it is disallowed, the crawler never sees the noindex and the page can still appear in results.

No page serves noindex (meta or X-Robots-Tag), so the disallow-hides-noindex trap is not in play.

checked 2026-08-27result 25 N/Atool googlebot-page-check
1.4Pass
When multiple allow/disallow rules match a URL, the most specific rule (the one with the longest matching path, measured in characters/octets) wins; Google breaks a genuine tie in favor of the least restrictive (allow) rule.

No allow rules and no overlapping disallows, so no rule-specificity conflict exists to get wrong.

checked 2026-08-27result site-leveltool curl-http-probe
1.5Pass
robots.txt path matching is case-sensitive: e.g. disallow: /file.asp blocks /file.asp but not /FILE.asp.

All robots.txt rules and all 25 priority URL paths are lowercase, so there is no case-variance exposure.

checked 2026-08-27result 25 Passtool googlebot-page-check
1.6Pass
Google parses at most the first 500 kibibytes (KiB) of a robots.txt file; content beyond that is ignored - and RFC 9309 sets 500 KiB as the minimum a conformant crawler must parse.

robots.txt is under 1 KB, far below the 500 KiB parse limit.

checked 2026-08-27result site-leveltool curl-http-probe
1.7Pass
A single sitemap file is limited to 50,000 URLs and 50MB uncompressed and must be UTF-8 encoded; the file may be gzip-compressed to save bandwidth as long as the UNCOMPRESSED size stays under 50MB.

2 sitemap indexes and 34 child sitemaps declare over 100,000 URLs. The largest file holds 15,000 URLs at 2.7 MB uncompressed, within the 50,000 URL and 50 MB limits. All files are UTF-8, and all 25 priority pages are present.

checked 2026-08-27result 36 filestool sitemap-conformance-check
1.8Pass
Sitemap entries must use fully-qualified absolute URLs (including protocol), and all data values must be entity-escaped, inside the required <urlset>/<url>/<loc> structure.

All locs are absolute, entity-escaping holds under strict XML parsing, and the urlset/url/loc structure is correct in all 36 files. The same run validated Google's extension namespaces and found an element-order defect. That became claim 1.8.1, added to close the hole in the list, and it carries the Gap.

checked 2026-08-27result 36 files, 1 notetool xmllint-sitemaps-xsd
1.8.1Gap
Sitemap extension namespaces carry their own governing schemas: inside <video:video>, Google requires video:thumbnail_loc, video:title, video:description and at least one of video:content_loc or video:player_loc, and the published sitemap-video/1.1 XSD declares these as an ordered sequence beginning with video:thumbnail_loc, so element order as well as presence determines schema validity.

Every entry in the video sitemap lists its children as title, description, thumbnail_loc, player_loc, while the governing XSD requires thumbnail_loc first. All required elements are present with valid values, so the failure is ordering only. Google's live parser tolerates it today, so impact is low. The exposure is strict consumers and forward compatibility. This claim was added during testing: the original list of 85 did not cover it.

checked 2026-08-27result 1 of 1 video sitemap failstool xmllint-sitemaps-xsd
1.8.2Gap
A sitemap is a list of URLs a crawler is being asked to fetch and index, so every <loc> must name a live, canonical URL: it should return 200 rather than an error status, and it should be the canonical form of the page rather than a redirecting or non-canonical variant. Entries that no longer resolve spend crawl budget on nothing and keep a removed URL in the index until the crawler independently rediscovers that it is gone.

Five URLs the sitemaps declare return 404 to Googlebot. The 404 depends on user agent, which matters for anyone re-checking it: a plain curl or a browser gets 200 with a JavaScript app shell, while a Googlebot user agent gets the 404. Search Console still reported one of them as 'Submitted and indexed', so the sitemap was holding a dead URL in the index. These 5 were the only non-200s in the probed set, which covered every URL outside the largest page class plus a deterministic sample of that class. This claim was added during testing, because none of the original 85 said anything about the state of the URL a sitemap entry names.

checked 2026-08-28result about 1,300 declared URLs probed, 5 Gaptool canonical-http-check
1.9PassCorrected
Googlebot processes JavaScript apps in three separate phases - crawling, rendering, indexing - using an evergreen Chromium; rendering is deferred in a queue and may lag crawling, and Google still recommends server-side or pre-rendering because not all bots run JavaScript.

Re-verified after the fix. In two sweeps of the 25 priority pages, 33 minutes apart, none served a 'Loading...' placeholder in the crawl-stage Googlebot HTML, and all 25 passed the tool's rule: 200 status, at least 100 words in <main>, no placeholder. Two things are not claimed. Spinner placeholders elsewhere on the pages are reduced, not gone. And outside the 25, 3 of the 91 previously failing URLs still served a placeholder, in a variant our own per-page check could not detect, which was a blind spot on our side. The two sweeps also returned identical ETags, so they show stability within one cached snapshot, not two independent prerenders.

Corrected from Gap to Pass after the development team's fix shipped. Before the fix, 7 of 25 priority pages failed. The verdict moved on the same tool and the same deciding field that produced the Gap.

checked 2026-09-07result 25 Pass, 0 Gap, twicetool googlebot-page-check
1.10Pass
Sitemaps can be submitted to Google via Search Console (Sitemaps report) or by adding a Sitemap: <absolute-url> line anywhere in robots.txt.

robots.txt carries a Sitemap: line pointing at the absolute URL of the root sitemap index.

checked 2026-08-27result site-leveltool curl-http-probe

Want this run on your site?

We test your site against the same claims, record a verdict and the evidence for each one, and tell you what to fix first.

Start a conversation