Module 3 · Make your site answerable · Lesson 6

What machines actually read

Many AI crawlers read the raw HTML your server sends and do not run JavaScript. Content that appears only after scripts run may be invisible to them.

5 min

Ce cours n'est pour l'instant disponible qu'en anglais. Le reste du site est en français.

In this lesson

  • Tell the raw HTML of a page apart from what a browser renders
  • Check what a non-rendering crawler receives from your key pages
  • Prioritise fixes that move important text into the server response

What you see in a browser is not always what a crawler receives. A browser downloads the page's HTML, runs its JavaScript and builds the final page. Many AI crawlers stop at the first step: they read the HTML the server sends and do not run scripts. Anything that only appears after JavaScript runs may be missing from what they read, and an assistant cannot quote what it never saw.

What you see, and what a crawler getsIn the browser, the pricing page shows three plans and an image of the opening hours. The raw HTML a non-rendering crawler receives contains only the heading, an empty app container, a script tag and an image tag: no plans, no prices, no hours.<h1>Pricing</h1><div id="app"></div><script src="app.js"></script><img src="hours.png">No plans, no prices, no hours

Raw HTML versus the rendered page

The raw HTML is the text your server returns for a URL, before any script runs. The rendered page is what exists after the browser has run every script, fetched data and filled in the template. On a server-rendered site the two are nearly identical. On a site that builds its content in the browser, the raw HTML can be an almost empty shell with a script tag.

Where content hides

Where content lives, and whether non-rendering crawlers see it
Content locationVisible to non-rendering crawlers?Action
Text in the server's HTML responseYesKeep it there; this is the target state
Text injected by JavaScript after loadNoRender it on the server or at build time
Text inside images, including charts of prices or hoursNoRepeat the facts in text near the image
Tabs and accordions present in the HTML but hiddenUsually yesFine, if the text is in the HTML; check it
Content behind a login, form or cookie wallNoPublish a public version of what buyers need to know
Text loaded on scroll or by clicking “load more”NoPut the first, most important part in the initial HTML

How to check a page

  1. Open the page and view its source (not the inspector, which shows the rendered page).
  2. Search the source for a sentence you care about: a price, your opening hours, the first line of your product description.
  3. If the sentence is not in the source, a non-rendering crawler does not get it.
  4. Repeat for your most important page types: home, product or service pages, pricing, locations, and your best answers to buyer questions.

Fixes, from cheapest to most thorough

The cheapest fix is often to repeat a few key facts in plain text: the hours under the image of the opening times, the price beside the interactive calculator. Next comes rendering the important sections on the server while keeping interactive parts in the browser. Most modern frameworks can do this page by page. The thorough fix is server-side or static rendering for every page that answers a buyer question.

Whatever you change, check the raw HTML again afterwards. It is easy to fix the visible page and leave the source exactly as it was.

Worked example: Lumen CRM

Lumen CRM's pricing page loads its plans from an API after the page opens, so the raw HTML contains a heading and an empty container. The comparison table on the features page is built the same way. In the team's scorecard, assistants describe Lumen's pricing vaguely or not at all, and never cite the pricing page.

The developers move the plan names, what each includes and the billing terms into the server response, and keep only the currency switcher in the browser. They do the same for the comparison table. Viewing the source now shows every plan in text. The team notes the date of the change in the worksheet so the next scorecard cycles can be read against it.

What not to do

Do not show crawlers different content from what people see. Serving a special text-only version to bots, with claims your visitors never read, is cloaking, and it undermines the trust you are trying to build. The goal is one page that people and machines can both read in full.

Key takeaways

  • Many AI crawlers read only the server's HTML and do not run JavaScript.
  • Check the page source, not the rendered page, for the facts that matter.
  • Move key text into the server response, template by template, and never show bots something people cannot see.

Practice task

Adds to your plan: Site fixes

  1. Pick one page of each important type on your site.
  2. View the source of each and search for its key facts: name, price or hours, the main claim.
  3. List every page type where a key fact is missing from the source, with the fix you will ask for.
Ouvrir la fiche de votre plan AEO →

Knowledge check

Three questions. Pick an answer to see whether it is right and why.

1. Where should you look to see what a non-rendering crawler receives?
2. Your opening hours are only in an image. What is the simplest fix?
3. Is it acceptable to serve bots a special version with extra claims?