/ GUIDE — 6 min read

Web scraping with custom extraction

Pull prices, SKUs, authors, dates or any value from every page of a site in one crawl — with CSS selectors, XPath or a regular expression.

What extraction is for

A crawl already collects the SEO facts about each page. Extraction collects your facts: the price on every product page, the author and date on every article, the schema type, the tracking tag, the text of a specific heading. The result is a table with one row per page and one column per extractor, exportable as CSV.

Typical uses: check that every product shows a price; list every post without an author; find pages whose breadcrumb is missing; confirm the analytics tag is on every page; collect all og:image values to find pages sharing one.

The three ways to point at a value

CSS selector — the simplest. h1, .price, meta[property="og:title"]. Name an Attribute (optional) such as content or href, or leave it empty for the text.
XPath — when you need structure: //div[@class="price"]/span[2], or "the paragraph after this heading".
Regex — run against the raw HTML, for things that aren't elements: an order ID pattern, a tracking ID, a date inside a script. It returns its first capture group, or the whole match.

Start from a preset prefills the common ones: Open Graph tags, Product price and SKU, Author and date, hreflang and Canonical.

The same $49 reached three ways: a CSS selector, an XPath, or a regular expression over the raw HTML.

1. Add extractors before the crawl

New crawl → Advanced options → Extraction & search. Under Custom extraction, choose Add extractor, give it a Name, pick its Type, paste the Expression, and tick All matches (up to 20) if a page can hold several values (every product on a category page). Up to 3, 5 or 10 extractors per crawl on Starter, Growth and Agency.

2. Add a custom search (optional)

The same section takes a Custom search: a string or pattern counted in every HTML page's source — 2 searches on Starter, 5 from Growth up. The count appears in the Custom search column on Pages, and the filter gtag: not found answers "which pages don't contain gtag(" in one click.

3. Read the Extraction tab

After the crawl, Data → Extraction shows the URL plus one column per extractor. Filter to Author: empty — or has a value — to find the pages that are missing something, and take the table away with Exports → Extraction (CSV).

One row per page, one column per extractor. Filtering to Author: empty leaves exactly the page that is missing a byline.

Turn an extracted value into a segment and every report filters to it — no spreadsheet needed. See plans →

4. Turn values into segments

In Advanced options → Segments, choose Add segment with the rule Extracted value equals — "pages where type = product" — and Save for this site. Segments apply from the site's next crawl; from then on Issues, Pages, Compare and every CSV can be filtered to the segment. This is how you check "do all product pages have structured data" without touching a spreadsheet.

An extractor named type returns product, article or page; a segment keeps the 312 product pages, and Issues can then be read for those pages alone.

5. Pages built by JavaScript

If the value only exists after JavaScript runs, set JavaScript rendering to Thin pages only or All pages and the extractors read the rendered page. On a verified site you can also add Custom JavaScript — Scroll to the bottom, Click “load more”, Read window.dataLayer, or your own function — and the value each snippet returns appears on the Extraction tab.

Limits and manners

Nosy Bat identifies itself as Echorank360Bot, obeys robots.txt and stays under two requests a second, on your site and on any other. Extraction is for sites you own or have permission to crawl; it is not a way around a site's terms. Custom JavaScript and the Googlebot or custom user agents are only available on sites you have verified as yours.