What Actually Blocks a Crawler From Reaching Your Pages

The frustrating version of this problem is not the site that is down. It is the site that works perfectly in your browser, returns a clean 200 to your own testing, looks fine in every dashboard — and still has pages that never reach the index. Nothing is red. The pages are simply not there.

The takeaway up front: a crawler's request is not one event, it is a chain of hops — DNS, TLS, the CDN edge, bot rules, the origin, redirects, then whatever the response actually contains. Every hop can answer differently for a bot in a datacenter than it does for you on your laptop, and a page disappears at whichever hop says no first. Debug it by walking the chain in order, because the symptom — "not indexed" — is identical no matter where the break is.

The chain a request passes through

When Googlebot decides to fetch https://example.com/services/roof-repair, roughly this happens:

1. DNS — resolve the host, following any CNAME chain.
2. TCP + TLS — connect and complete a handshake with a certificate valid for that hostname.
3. The edge — a CDN, reverse proxy, or WAF decides whether this client gets through at all.
4. The origin — your server builds a response.
5. Redirects — the crawler follows Location and repeats hops 1–4.
6. The payload — the HTML, its headers, and any client-side rendering.
7. Directives — robots.txt, X-Robots-Tag, meta robots, canonical.

Hops 1–5 decide whether a page is reachable; 6–7 decide whether a reachable page is indexable. Both produce the same empty result in search, which is why they get misdiagnosed.

Hop 1: DNS answers differently than you think

DNS problems rarely take a site down — they take parts of it down for some resolvers, which is far harder to notice.

• A long or dangling CNAME. Your apex points at a CDN hostname that points at another. Each extra lookup adds latency and a chance to fail, and a crawler on a fetch timeout gives up where a patient browser retries.
• A broken AAAA record. An IPv6 address that no longer serves traffic. Your IPv4 laptop never notices; crawlers on IPv6-preferring networks hit a dead address and log the page unreachable.
• Nameserver disagreement. Two of four nameservers carry a stale zone after a migration, so half of lookups get the old answer — intermittently, which reads as a flaky site rather than a DNS fault.

Check against several resolvers and both address families, not just your own:

dig +short A   example.com @1.1.1.1
dig +short AAAA example.com @8.8.8.8
dig +trace     example.com | tail -n 20

If the answers disagree, stop here — nothing downstream matters.

Hop 2: TLS that is valid for you and invalid for them

Certificate problems are binary for crawlers: a handshake failure is a hard fetch failure. The two that hide well are missing intermediates — browsers fetch or cache those themselves, so your site looks fine while a strict client fails the chain — and incomplete SAN coverage, where a certificate issued for example.com but not www.example.com breaks only the variant you never type. Test with a client that does no caching and no guessing:

curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://www.example.com/
openssl s_client -connect www.example.com:443 -servername www.example.com </dev/null 2>&1 | head -20

Hop 3: the edge decides who you are

Most silent crawl loss happens here. Edges classify every request before your application sees it, using signals such as:

• Source IP and ASN. Datacenter ranges score worse than residential by default — and every search crawler comes from a datacenter.
• User agent. Some rule sets challenge anything self-identifying as a bot, including the ones you want.
• Request rate. A crawler fetching 200 URLs a minute looks exactly like a scraper to a rate limiter.
• TLS and header fingerprints. Cipher and header ordering differ between browsers and HTTP libraries, and edges key on that.
• Geography. A country block added for fraud reasons also blocks crawler infrastructure in that region.

The result is rarely a 403 that shows up in a report. It is a 200 response containing a challenge page — a "checking your browser" interstitial or a CAPTCHA. To the crawler that is a valid, indexable document, so the URL is either indexed with the challenge as its content or judged thin and dropped. Your logs show 200s. Your dashboard is green. The page is gone.

Two fixes matter more than the rest. Allow verified search crawlers explicitly — most edge products ship a "verified bot" rule that confirms identity by reverse DNS on the source IP (googlebot.com, search.msn.com) plus a forward lookup back to the same address, which is why a user-agent allowlist alone is unsafe: anything can claim to be Googlebot. Then exempt those verified clients from rate limits, or your crawl budget quietly becomes your rate limit.

Hop 4: redirects that lose the destination

Redirects are where good intentions leak equity:

• Chains. http:// → https:// → www → trailing slash is four requests for one page. Engines follow a limited number of hops, and each link is another place to be blocked. Collapse them so any variant lands on the final URL in one step.
• Loops. Two systems disagreeing — a CDN forcing www, an application forcing the bare domain.
• Redirect-to-homepage. A retired page sent to / rather than a real replacement is treated as a soft 404 and its signals go nowhere. Redirect to the closest equivalent, or return a clean 410.
• IP or language redirects. If visitors from a country are bounced to a regional homepage and the crawler sits in that country, it never sees the page you wanted indexed. Use hreflang; do not force by IP.

Read the chain in one command:

curl -sIL -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
  http://example.com/services/roof-repair \
  | grep -Ei '^(HTTP/|location:|x-robots-tag:)'

HTTP/1.1 301 Moved Permanently
location: https://example.com/services/roof-repair
HTTP/2 301
location: https://www.example.com/services/roof-repair/
HTTP/2 200
x-robots-tag: noindex

That last header is the kind of thing you find only by looking: three hops, then a noindex nobody knew about — usually left over from a staging config copied to production.

Hop 5: what the crawler receives is not what you see

A 200 does not mean the crawler got your content.

Interstitials and consent walls. If your cookie banner, age gate, or region selector is rendered server-side as the whole body and the content loads only after a click, the crawler indexes the wall. Serve the content in the initial HTML and overlay the banner on top.

Client-side rendering. If the initial HTML is an empty <div id="root">, indexing depends on the engine executing your JavaScript — a second pass that is slower, is not guaranteed for every URL, and fails silently when a script errors. Server-render anything that must be indexed, and check that no robots.txt disallow on /assets/ or /api/ stops the renderer fetching the CSS and JSON the page needs.

Personalization. Content that varies by session, geography, or A/B bucket can reach the crawler as the least interesting variant. Make the default bucket the real page.

This is where the technical SEO guide picks up: once a page is reachable and its content really is in the response, the remaining question is why the engine chose not to keep it.

Testing the chain the way a crawler experiences it

You cannot debug this from your browser, because your browser is the one client guaranteed to be treated well. Reproduce the crawler's conditions: fetch from outside your network (office IPs get allowlisted and forgotten), with a bot user agent and again with a library default so you can see which rules key on identity, following every redirect and printing each hop's headers, then diff the raw HTML against a headless render. Do it on a schedule across your top templates, because edge rule sets change under you. The useful signal is not "site up" — it is did any hop answer a bot differently this week than last?

When your own monitor hits a challenge

Monitoring runs into the same walls it is meant to detect. A synthetic check that lands on a challenge page records a 200 and a healthy site while real crawlers are turned away — the monitor confirms the failure it exists to catch. To see behind the wall, on properties you own and public pages you may read, the check has to complete the challenge.

That is where a solving API earns a place in a monitoring stack. CaptchaAI is one, and its shape tells you whether it fits your tooling: it speaks the older 2Captcha-style protocol — submit, then poll — so a monitor that already talks that protocol needs a host change, not a rewrite.

curl -s "https://ocr.captchaai.com/in.php" \
  -d "key=YOUR_API_KEY" \
  -d "method=turnstile" \
  -d "sitekey=0x4AAAAAAA..." \
  -d "pageurl=https://www.example.com/services/roof-repair" \
  -d "json=1"

# 2. poll roughly every 5s until it stops returning CAPCHA_NOT_READY
curl -s "https://ocr.captchaai.com/res.php?key=YOUR_API_KEY&action=get&id=<taskId>&json=1"

Results come back in the same envelope — CAPCHA_NOT_READY while it works, then a token, or ERROR_UNSOLVABLE / ERROR_ZERO_BALANCE. For a Cloudflare Challenge it returns a clearance cookie plus the user agent it was issued for; send them together or the cookie is rejected. The per-type numbers are the vendor's own published claims — Turnstile 100% under 10s, Cloudflare Challenge above 99% under 15s, reCAPTCHA v2 above 99.5% under 60s — so verify them on your own URLs. Pricing is by concurrent thread rather than per solve (published tiers start at \$15/mo for 5 threads, \$90/mo for 50), which suits monitoring: a predictable number of parallel checks, not a meter running.

Scope matters. This is for verifying your own properties and public pages, under each site's terms, robots.txt, and rate limits — not a way around authentication, paywalls, or signup gates.

FAQ

Why do my pages return 200 but still never get indexed?

A 200 describes the transaction, not the content. A challenge page, a consent wall, and an empty JavaScript shell are all valid 200s containing nothing worth indexing. Fetch the URL from outside your network with a bot user agent and read the body, not the status code.

Can my firewall or CDN block Googlebot without me knowing?

Easily. Bot rules key on datacenter IP ranges, request rate, and header fingerprints — all of which describe legitimate crawlers. Enable your edge's verified-bot rule, which confirms identity by reverse-then-forward DNS on the source IP rather than trusting the user-agent string, and exempt those clients from rate limits.

Does DNS really affect indexing?

Indirectly but decisively: failed or slow resolution is a failed fetch, and repeated failures reduce how often a host is crawled. Broken AAAA records, stale nameservers, and long CNAME chains all cause failures a browser test never shows.

Is using a CAPTCHA solver for site monitoring legitimate?

For your own properties, and for public pages you may read within a site's terms and rate limits, yes — it is how a synthetic check sees past a challenge instead of scoring it as healthy. It is not appropriate for bypassing logins, paywalls, or signup gates.

Where to start

Take your ten highest-value URLs and walk the chain once: two public resolvers, the certificate on every hostname variant, a fetch from outside your network with a crawler user agent, every redirect hop with its headers, raw HTML diffed against rendered. Most sites turn up at least one surprise — a stray X-Robots-Tag, a third redirect nobody planned, an edge rule added during an incident and never removed.

Then make it a monitor rather than an audit, because the edge config that passes today is a different rule set next quarter. If your checks land on challenge pages and score them healthy, wire a solving step into the monitor so it can see past the wall — run a trial batch through CaptchaAI against your own challenged URLs, measure the success rate and latency you actually get, and keep it only if the numbers hold up.