Technical checklist to get your site ready for AI crawlers and agents

The fastest way to find out if your site is ready for AI is to stop trusting what you see in the browser. The browser runs JavaScript, populates the content, and shows a complete page. Most AI crawlers do not do that: they read the raw HTML that arrives in the server's first response, exactly as view-source would show it, without waiting for anything to render afterward.

This mismatch is the most common cause of "my site looks great but AI can't find anything relevant on it." This checklist covers the technical points that show up most often as a real bottleneck, in the order worth testing them.

Open the page, right-click, and choose "View page source" (not "Inspect," which shows the DOM already processed by the browser). What appears there, with no scrolling, with nothing waiting to load, is close to what most AI crawlers actually receive.

A case reported in SEO forums showed a SaaS homepage with only 228 readable characters inside 19,000 bytes of HTML, because all the relevant content was injected via JavaScript after the initial load. To the user in the browser, the page looked complete. To a crawler that only reads the initial HTML, it was almost a blank page.

In practice: if view-source on your main page shows little more than empty tags and scripts, the problem is in rendering, not content. Fixing that (SSR, pre-rendering, or progressive hydration) comes before any text optimization.

Command-line tools like curl show exactly what a server delivers without executing anything:

curl -A "GPTBot" -s https://yoursite.com/important-page | grep -o "<title>.*</title>"
curl -A "GPTBot" -s https://yoursite.com/important-page | wc -c

Compare the size of the raw HTML with the size you see inspecting the loaded page in the browser. A large difference is a sign of heavy JavaScript dependency to display content.

Serving a Markdown version of the content (via content negotiation, checking the request's Accept header, or through a parallel route) is increasingly cited as a practice that helps AI crawlers extract clean text without dealing with decorative HTML. A real caveat, raised by Google and Bing themselves: carelessly generated Markdown (with no clear structure, mixing navigation content with main content) can increase crawl load instead of reducing it, because the crawler still has to parse the whole file to separate what matters.

Not all traditional SEO metadata is processed by AI crawlers. Informal tests reported by professionals in the field found that many of the metadata types tested (Open Graph, meta description, some schema fields) were not consistently captured by specific AI crawlers, while the title and the running body text were always read. When in doubt, do not rely on metadata alone to convey critical information: put the data in the visible text too.

A pattern observed in large-scale bot behavior analysis: LLM crawlers tend to prioritize question-and-answer pages (FAQ, help pages) instead of relying only on sitemap.xml to discover relevant content. This suggests two practical things: keeping a real FAQ section (not a generic one) helps you get found, and the sitemap alone is no guarantee the crawler will prioritize your most important pages.

It is worth implementing correct schema markup (Organization, Article, FAQPage, Product, depending on the case), but calibrate your expectations: an Ahrefs survey discussed in industry circles tracked schema additions across nearly 1,900 pages and saw AI citation barely move in direct response to that isolated change. Schema helps the machine understand the content with less ambiguity, but it does not replace content that answers the question clearly in the text.

Tools like Cloudflare's "Agent Readiness" scanner give a score based on categories such as discovery (robots.txt, sitemap, link headers), bot access control, content format, and agent capabilities (MCP, OAuth). A real-world usage report showed a site going from 29 to 71 points after applying selected recommendations, not all of them: the author deliberately left out items like OAuth authentication and Web Bot Auth because they did not make sense for that type of site (kevinbk.com). The goal is to apply what makes sense for your case, not to maximize the score.

A Cloudflare analysis of 200,000 domains found that 78% of sites have robots.txt, but only 4% declare any specific preference for AI bots, only 3.9% serve content in Markdown, and fewer than 15 sites exposed an MCP server (via dynamicasoft.com). Most of the web simply has not decided anything about this yet, which is, at the same time, a risk and a window of advantage for whoever decides first.

// real case: who hasn't seen content blocks that only appear after an interaction? They are invisible to most bots, and, guess what, to AI too.

1. Run view-source on the homepage and the 5 most important pages.
2. Compare the size of the raw HTML with the HTML rendered in the browser.
3. Test with curl using an AI bot user agent.
4. Check whether critical information exists in the visible text, not only in metadata or schema.
5. Have a real FAQ section, with questions people actually ask.
6. Confirm in robots.txt (see the AI bots guide) that search/citation bots are not accidentally blocked.
7. Run an Agent Readiness scanner just to get a baseline, without chasing a perfect score.

Running view-source on the homepage is the cheapest test on this list and usually the most revealing one. If the content is not there, no citation strategy works on top of it. This article is part of the pillar GEO in practice: how to get crawled, understood, and cited by AI search engines.