Skip to content

Why scraped retailer pages are a fragile base for shopping assistants

Scraping retailer pages looks like the quickest way to give a shopping assistant product data. In practice, layouts change, requests get blocked, identifiers go missing and prices go stale. This guide sets out where scraped data breaks, what a structured product source changes, and the honest trade-offs either way.

Cracked, tilted page and warning cards on the left, with clean light trails leading to a highlighted structured data card on the right.

The short answer

Scraping retailer pages is the obvious first move when you need product data for a shopping assistant. The pages are public, the prices are right there, and a prototype can be running by the end of the week. The trouble starts a few weeks later, when results quietly get worse and nobody can say exactly why.

This guide sets out where scraped data tends to break, what a structured product source changes, and the trade-offs that remain either way. It is written for teams deciding how to source product data, not to argue that scraping is never right.

Why does scraped product data break?

Layout changes

A scraper depends on the shape of the page: where the price sits, what the title element is called, how the spec table is laid out. Retailers redesign, run A/B tests and change templates by category. When that happens, the scraper rarely fails loudly. It returns an empty price, or picks up the "was" price instead of the current one, or reads a delivery charge as the product price. The assistant keeps answering, just wrongly.

Blocked and inconsistent requests

Retailers protect their sites from heavy automated traffic. Requests get rate-limited, challenged or served a different page. Coverage becomes patchy in ways that are hard to see: one retailer works on Monday and not on Tuesday, or works for some categories and not others. Site terms and robots rules also apply, and they are worth reading before you start rather than after.

Missing identifiers

Many product pages do not show a GTIN or MPN anywhere a scraper can reliably reach. Without identifiers you cannot be sure two retailers sell the same dishwasher, so price comparison falls back to matching titles, which is where wrong comparisons come from.

Stale or undated prices

A scraped price is only as fresh as the last successful run for that page. When a price arrives without the time it was captured, the assistant cannot tell the shopper how current it is, and you cannot tell which prices to distrust. We explain why this matters in why every product price needs a source and a checked time.

The maintenance burden

Each retailer needs its own parsing rules, and each set of rules needs watching. The work grows with every retailer and category you add, and it is the kind of work that rarely makes a roadmap until something breaks. For a small team, that time comes straight out of the product the shopper actually sees.

What a structured source changes

ConcernScraped pagesStructured product source
FormatDifferent per retailer, changes without noticeOne documented schema across retailers
IdentifiersOften missing or hard to extractSeparate GTIN, MPN and SKU fields where available
PricesOften undatedPrice with currency, retailer and snapshot date
FailuresSilent, discovered by usersExplicit: fewer or empty results
UpkeepYours, per retailerThe provider's, behind a stable interface

The biggest change is not any single field. It is that failures become visible. An empty result is easy to handle honestly; a wrong price that looks right is not. More on what good retailer data looks like is in our guides to retailer product data for shopping assistants and structured retailer data for AI search.

The honest trade-offs

A structured source is not free of problems, and it is worth being clear about them:

  • Coverage is the provider's, not yours. If you need one specific retailer that a provider does not include, scraping or a direct feed may be the only route.
  • Prices are still snapshots. A dated price is more honest than an undated one, but it can still be out of date by the time the shopper clicks.
  • You pay for it. Compare that cost against the engineering time scraping really takes, including the weeks when it breaks.
  • You depend on a supplier. Check that the interface is documented and that you can test it before committing.

Questions to ask any product data source

  • Does every price carry currency, retailer and a date?If not, you cannot tell shoppers how current it is.
  • Are identifiers separate fields?GTIN, MPN and SKU should not be buried in a description.
  • What happens when there is no match?Fewer or empty results are better than padding with weak matches.
  • Can you filter by market?Results should stay within the shopper's country unless asked otherwise.
  • Can you test it on your own requests?Run real shopper questions before deciding.

For a structured way to run that test, see how to evaluate product search quality.

Frequently asked questions

Is scraping always the wrong choice?
No. It can make sense for a narrow prototype or a single site you have permission to use. It becomes fragile as the number of retailers, categories and markets grows.
Does a structured source guarantee current prices?
No. It should tell you when each price was checked, so you can show that date and treat older prices with care.
How should I compare the two approaches?
Run the same set of real shopper requests against both, and check identifiers, price dates, empty-result handling and the time your team spends keeping each one working.
What usually breaks first when scraping retailer pages?
Layout changes are a common cause: a redesign or template change can make a scraper return an empty price, the old "was" price or a delivery charge, often without any error.

Sources

Jay

Co-founder, simlir

Jay is the co-founder of simlir