Your next article cannot fix a locked door. In the last 30 days on kindredworkshop.com, Claude’s live-answer fetcher hit /robots.txt about 1,177 times: more than any article or the homepage. That file is the first gate an AI answer passes through, yet most teams have not opened theirs in years. I have seen ad accounts fail for equally boring reasons: a wrong conversion action, a swapped landing page, a negative list nobody maintained. AI visibility has its own version: a blocked bot, a JavaScript-only answer, a WAF rule. Run these 27 pass/fail checks before you pay for more content. Start at the top; a perfect answer is useless if the fetcher cannot reach it. This is a sequence, not a scorecard: passing 26 checks does not make up for a failed fetch on the page that sells your product. The bolded items are the ones teams tend to skip.

Gate 1: Can the bots get in?

  1. robots.txt allows OAI-SearchBot by name. Keep it separate from GPTBot: search access and training access are different controls, so allowing one does not allow the other (allow search bots but block training bots separately). Check the answer URL, not only the homepage.
  2. robots.txt allows Claude-User and Claude-SearchBot by name, separately from ClaudeBot. Check each directive rather than assuming one Claude rule covers every fetch; record which agent is denied before you change anything.
  3. robots.txt allows PerplexityBot and Perplexity-User by name. Test both; a pass for one is not a pass for the other.
  4. The server and WAF let ChatGPT-User and Perplexity-User fetch the page after you allow them. These user-triggered fetches may ignore robots.txt; access still depends on what your server and WAF return (user-triggered fetches may ignore robots.txt). A 200 on the homepage does not clear a 403 on the answer page.
  5. Google-Extended is not mistaken for an AI Overviews switch. If AI Overviews are the goal, changing this training control does not replace access for the regular Googlebot index (blocking Google-Extended does not opt you out).
  6. Cloudflare Super Bot Fight Mode, Block AI Bots, and generic WAF rules do not challenge the fetchers you intend to allow. A curl request with a bot user agent is a useful first test for a 403 or challenge, but confirm real fetches in edge logs; a user-agent string alone does not make your request a verified bot (CDN and WAF failures can be silent). Match the logged response to the URL you tested. For the command sequence, use Can AI Search Read Your Site? A 45-Minute curl and CDN Audit.

Gate 2: Can they read the answer they fetched?

  1. Core answer text appears in raw HTML, not only after JavaScript runs. Pass if View Source contains the answer paragraph; do not assume a page that renders for Google renders for every AI fetcher (AI crawler rendering limits). Look for the answer itself, not a description promising one.
  2. The page remains readable with JavaScript disabled. Reload it with JS off. An empty div and a script tag are not an answer, however polished the page looks in your usual browser.
  3. Core facts are present in source without a click, scroll, or consent action. Tabs loaded on click, infinite scroll, JS filters, and cookie walls can leave a fetcher with incomplete content (tabs and filters can hide content). If the answer depends on a filter, test what the unfiltered URL returns.
  4. View Source and Inspect Element contain the same answer section. Pass if the relevant paragraphs appear in both; fail if they exist only in the rendered DOM.
  5. ChatGPT and Claude can summarize the URL correctly when asked. Paste the URL and compare the summary with the page. Treat a refusal or invented summary as a prompt to inspect checks 7–10, not as proof of which gate failed. A fluent summary of the wrong page is not a pass.

Gate 3: Can they find the page you want cited?

  1. The XML sitemap lists canonical URLs that return 200 and is referenced in robots.txt. Check each target answer URL against its canonical rather than treating a submitted sitemap as a pass. The URL you want cited should not resolve to a different preferred version.
  2. lastmod reflects a real page change and uses an ISO 8601 date and time. Do not stamp every build date onto every page. Bing treats honest lastmod as a recrawl signal for AI indexing and ignores changefreq and priority (how Bing uses lastmod).
  3. The canonical tag points to the version you want cited. Pass if that URL is self-canonical and parameter variants point back to it; fail if your preferred answer points somewhere else.
  4. Each answer page has a link from a crawled hub. Navigation, a category page, or a sitemap-indexed hub will do if that hub returns 200 to AI bots. Open the hub as the bot and follow its link; do not leave the answer page orphaned and expect the sitemap to do all the work.

Gate 4: Can they extract an answer worth quoting?

  1. The first 100 words contain a direct, self-contained answer. Pass if someone can quote that paragraph alone and it answers the question in the page title.
  2. The first one or two sentences under each H2 answer that heading on their own. Put the answer before the setup or story. Q&A and list formats give fetchers clearer passages to extract, and Perplexity cites about five links per answer; make the passage earn its place.
  3. Organization plus Product or Service schema agrees with the visible page. Check sameAs and about, then compare name, price, and category with what a reader sees. Fail the check if the markup and page disagree; update both when those details change.
  4. Article schema links to the author; FAQPage or HowTo describes real Q&A or steps. Do not add markup as a citation switch. In WordLift tests, JSON-LD alone produced marginal lift until the graph was traversable and consistent with the page.
  5. Each buyer question has one canonical answer URL, not a cluster of keyword-variant pages saying the same thing. If the answer is clear but the page still loses, run The AI Overview Pre-Flight: 22 Landing Page Checks Before You Blame Google.

Gate 5: Can they tell which business answered?

  1. Business name, address, and category match across your site, Google Business Profile, and top directories. Pass if those details agree; fail if your site names one business and a major listing appears to name another. Do not make a fetcher reconcile your identity for you.
  2. The About page returns 200 to AI bots and is linked from the homepage. Its first paragraph says what you sell and who you sell to; any founder or entity details agree with your schema.
  3. Pricing and service-area information is crawlable without a login or form gate. Put price ranges, packages, or service areas in HTML on the relevant pages. Test the public URL, not a logged-in preview. A number available only after a sales call cannot be fetched from the page.

Gate 6: Can you measure access instead of guessing?

  1. Verified bot hits are separated from spoofed hits. A User-Agent is easy to copy. Check the source IP against the operators’ published feeds at openai.com/searchbot.json, gptbot.json, chatgpt-user.json, claude.com/crawling/bots.json, and perplexitybot.json; do not count an unverified hit as evidence of a real fetch (verify IPs against published feeds). Keep unknown hits out of the verified total.
  2. Live-answer fetches are tracked per page, not buried in a total bot-hit count. Pass if you can name the top five fetched URLs for ChatGPT-User, Claude-User, and Perplexity-User over the last 30 days. Report URLs and agents together; one aggregate curve hides which pages were reached. If an allowed bot shows zero hits, inspect edge and WAF logs before blaming copy.
  3. 403s and 429s on money pages are monitored by bot user agent every week. Check key URLs for OAI-SearchBot, ChatGPT-User, Claude-User, and PerplexityBot; use a request test to find obvious blocks, then verify what actual fetches received. Keep the key-page list explicit so monitoring does not quietly shrink to the homepage. A 403, 429, or challenge is a failed access check, whatever your rankings say.
  4. A clean markdown or text version of each key page returns the same answer, if you publish one. Test /page.md, ?output=1, or your text mirror for a 200 and readable content without JS. If you have no mirror, skip this optional check; do not build one just to pass an audit. Do not let a mirror disagree with the canonical page.

Pipes feeding a chatbot answer box, with one valve shut

Fix failures in the order they block the answer

  1. Clear WAF and CDN blocks first (check 6). A challenged fetcher cannot read the page. Confirm the rule change against real edge-log responses before touching copy; a 200 from your Chrome session proves little about bot access. Check the response for the page you need cited, not merely the root domain.
  2. Put the answer in source HTML (checks 7–10). If JavaScript builds the paragraph after the initial response, a fetcher reading raw HTML may get an empty page. Server-render the answer before revising its wording. Better copy inside a script-dependent tab leaves the same failure in place.
  3. Open the server path for live-fetch agents (check 4). An Allow directive in robots.txt does not undo a 403 at the origin. Test the exact pages buyers ask about, not just the homepage.
  4. Align canonicals and the sitemap (checks 12–14). If the sitemap lists one URL while its canonical selects another, clean up that route before producing another near-duplicate answer page.
  5. Separate verified, per-page fetches from total hits (checks 24–26). Without that split, you cannot tell whether a new article was fetched, whether a money page was blocked, or whether a rise in bot traffic came from an impostor.

Keep the pre-flight running after the fixes

  1. Retest the way the page failed: repeat the bot request, reload with JavaScript off, and check the answer in View Source. A fix is not a pass because somebody closed a ticket; it passes when the relevant page returns the answer through the path that failed.
  2. Watch verified live-answer fetches for seven days before rewriting copy. If the fetcher still cannot reach the page, a better opening paragraph will not help it. Keep the before-and-after URLs together so you are not comparing different pages.
  3. Pull verified fetches per page each week, confirm money pages still return 200 to live-answer agents, and flag a URL whose fetches fall to zero while impressions hold. Bot rules and client-side templates change; access can drift back out of spec.
  4. If you need someone to own the recurring work, groas handles bot access, rendering fixes, answer extraction rewrites, and per-page live-fetch tracking through Earned Search, with a named strategist accountable for direction. That is a different job from handing you another audit PDF to action.
  5. Do not skip check 26: weekly 403 and 429 monitoring on money pages. Your WAF can challenge Claude-User or PerplexityBot while Chrome and an allowlisted Googlebot still see 200. Rankings hold, the dashboard looks normal, and you buy more articles for a page the answer fetcher cannot read. If you are spending, say, $20k a month on content and ads, that budget keeps running while the door stays locked.