Module 3 · Make your site answerable · Lesson 6
What machines actually read
Many AI crawlers read the raw HTML your server sends and do not run JavaScript. Content that appears only after scripts run may be invisible to them.
In this lesson
- Tell the raw HTML of a page apart from what a browser renders
- Check what a non-rendering crawler receives from your key pages
- Prioritise fixes that move important text into the server response
What you see in a browser is not always what a crawler receives. A browser downloads the page's HTML, runs its JavaScript and builds the final page. Many AI crawlers stop at the first step: they read the HTML the server sends and do not run scripts. Anything that only appears after JavaScript runs may be missing from what they read, and an assistant cannot quote what it never saw.
Raw HTML versus the rendered page
The raw HTML is the text your server returns for a URL, before any script runs. The rendered page is what exists after the browser has run every script, fetched data and filled in the template. On a server-rendered site the two are nearly identical. On a site that builds its content in the browser, the raw HTML can be an almost empty shell with a script tag.
Where content hides
| Content location | Visible to non-rendering crawlers? | Action |
|---|---|---|
| Text in the server's HTML response | Yes | Keep it there; this is the target state |
| Text injected by JavaScript after load | No | Render it on the server or at build time |
| Text inside images, including charts of prices or hours | No | Repeat the facts in text near the image |
| Tabs and accordions present in the HTML but hidden | Usually yes | Fine, if the text is in the HTML; check it |
| Content behind a login, form or cookie wall | No | Publish a public version of what buyers need to know |
| Text loaded on scroll or by clicking “load more” | No | Put the first, most important part in the initial HTML |
How to check a page
- Open the page and view its source (not the inspector, which shows the rendered page).
- Search the source for a sentence you care about: a price, your opening hours, the first line of your product description.
- If the sentence is not in the source, a non-rendering crawler does not get it.
- Repeat for your most important page types: home, product or service pages, pricing, locations, and your best answers to buyer questions.
Fixes, from cheapest to most thorough
The cheapest fix is often to repeat a few key facts in plain text: the hours under the image of the opening times, the price beside the interactive calculator. Next comes rendering the important sections on the server while keeping interactive parts in the browser. Most modern frameworks can do this page by page. The thorough fix is server-side or static rendering for every page that answers a buyer question.
Whatever you change, check the raw HTML again afterwards. It is easy to fix the visible page and leave the source exactly as it was.
Worked example: Lumen CRM
Lumen CRM's pricing page loads its plans from an API after the page opens, so the raw HTML contains a heading and an empty container. The comparison table on the features page is built the same way. In the team's scorecard, assistants describe Lumen's pricing vaguely or not at all, and never cite the pricing page.
The developers move the plan names, what each includes and the billing terms into the server response, and keep only the currency switcher in the browser. They do the same for the comparison table. Viewing the source now shows every plan in text. The team notes the date of the change in the worksheet so the next scorecard cycles can be read against it.
What not to do
Do not show crawlers different content from what people see. Serving a special text-only version to bots, with claims your visitors never read, is cloaking, and it undermines the trust you are trying to build. The goal is one page that people and machines can both read in full.
Key takeaways
- Many AI crawlers read only the server's HTML and do not run JavaScript.
- Check the page source, not the rendered page, for the facts that matter.
- Move key text into the server response, template by template, and never show bots something people cannot see.
Practice task
Adds to your plan: Site fixes
- Pick one page of each important type on your site.
- View the source of each and search for its key facts: name, price or hours, the main claim.
- List every page type where a key fact is missing from the source, with the fix you will ask for.
Knowledge check
Three questions. Pick an answer to see whether it is right and why.