A pipeline that turns a public catalogue into a price history
Checking a competitor by hand gives you a snapshot and no way to compare it to last week. We built a pipeline that reads public catalogues page by page in a rendered browser, works out a site's category structure and pagination on its own, proves its extraction recipe returns real products before trusting it, and writes a price point on every pass. The report that follows is a difference rather than a dump, and it does not leave the building until a person approves it.
- IndustryRetail and e-commerce
- EngagementIn-house platform build
- StatusRunning the whole cycle, parts still unfinished
One cycle: collect, normalise, compare, package, approve
The pipeline runs on a schedule per client. It first crawls the competitor's own domain for pages worth reading, then harvests the product catalogue itself: category by category, page by page, inside a rendered browser, because a catalogue built in JavaScript is not in the HTML a plain request returns. Reviews and public posts feed the same store, so what comes out describes a competitor rather than only a price list.
Everything collected is normalised into the same shape. Amounts are parsed into a number and a currency across European, Turkish and United States formats. Products are updated by identity, so a fresh pass refreshes the product already stored, and the first seen and last seen dates are what make new and disappeared computable at all. Every pass with a price also writes a price point, which is what lets the report show what changed since the last pass.
What made this hard
The pipeline was designed against these three failures.
A snapshot cannot be compared to anything
Checking prices by hand produces a list that is already old as it is written and has nothing behind it. The question a category team actually asks is what moved since last time, and no amount of careful copying answers it, because the previous state was never kept.
The catalogue is not in the page source
Storefronts now build their product grid after the page loads, hang their images off lazy loading attributes, and split a category across pages in whichever way their platform happens to favour. Reading the source of a category page returns an empty shell.
A failed read looks exactly like an empty shelf
The dangerous failure is not the crash. It is the pass that finishes cleanly and reports nothing found, when what actually happened was a network flake. That result is indistinguishable from a real change, and it poisons the comparison that follows.
What we built
One scheduled cycle: reach the catalogue, work out its structure, make the values comparable, and turn the difference into something a person signs.
Collection that reaches the catalogue
The crawl stays on the competitor's own domain, respects robots.txt, rate limits itself, caps how far it goes, and visits pricing and product pages before generic ones. The catalogue harvest is a separate job: extraction runs inside the rendered page, because that is where a JavaScript grid exists at all.
- Same domain crawling that respects robots.txt, rate limits itself and reads commercial pages first.
- Three collection engines with automatic fallback, so a missing optional dependency degrades a crawl instead of failing it.
- Query string pagination, next link pagination, single page listings and infinite scroll, each identified by probing the site itself.
Structure the model proposes and code proves
A site's real categories are worked out from its own links and its sitemap, then classified. Nothing is accepted on the strength of that classification: a resolve and extract gate has to demonstrate that the proposed recipe returns actual products on the page it claims to work on.
- Category candidates come from the site's own navigation and its sitemap, and each one is validated before it is used.
- A proposed extraction recipe is trusted only once it is shown to yield products, with a hard no invention rule on the classification pass.
- Name and price extraction fall through a chain of fallbacks, preferring the shortest clean match so a whole product card is never mistaken for a name.
Values that can actually be compared
A price on a page is a string with a currency mark, a thousands separator that means different things in different countries, and often a discount badge sitting next to it. Everything is parsed into an amount and a currency before it is stored, and brands are read from a curated list that refuses to guess.
- Amounts are parsed across European, Turkish and United States number formats, with the currency read alongside.
- A percentage badge is stripped before parsing and the final currency anchored amount wins, so a discount is never stored as the price.
- Products are deduplicated by identity across passes, so re-reading a page updates the product already stored.
A report that is a difference, and a person who signs it
Every pass with a price writes a history point, so the next one is already a comparison. The change report is computed from what is already on disk, without a model call, a network request or a new table. Rendering leaves something in a pending review state, and approval is the only code path that sends.
- Movers up and down, new products, products that disappeared and new reviews, all computed deterministically.
- A spreadsheet data pack and a slide deck, rendered byte identically from identical inputs.
- Nothing reaches a client inbox until a person approves it, and with no mail provider configured the send refuses rather than recording one that never happened.
The decisions that make the report worth reading
Four choices that decide what the report can be trusted to say.
A failed read is not an empty shelf
A failed request, a page that loaded but held nothing, and a page of products are three distinct outcomes in the code. A pass marks itself complete, partial or failed on that evidence, so a transient network problem can never arrive as a clean result with nothing in it.
The stop condition belongs to code
Gap detection is rule based with a hard cap per pass, and the optional model critic sits on top of it and is off by default. A scan therefore costs nothing at the model until someone deliberately turns that on, and no language model decides when the work is finished.
Order that survives the database
Chronology lives in a microsecond timestamp captured at read time rather than in insertion order, with a random identifier only as a last resort tiebreak. A price history therefore reads the same whichever database is underneath it.
Isolation at a single choke point
Every read and write passes through one scoping choke point, backstopped by row security in the database itself, and the service refuses to start if its own runtime role could bypass that. The comparison engine inherits that boundary.
What changed
Qualitative, because what this project produced is a working pipeline rather than a measured outcome.
History
Comparison instead of a snapshot
The first pass produces a catalogue. Every pass after it produces a difference, because the prices behind it are already on disk with the dates they were read.
Honest
Passes that admit what they missed
A pass that could not read something says so and marks itself partial. Nothing downstream has to guess whether an empty section means a competitor changed or a request failed.
Reviewed
A person before the client
Rendering a report and sending it are two deliberately separate steps. Nothing has ever gone out because a schedule fired, and a health probe walks the whole cycle without touching the network or spending anything at the model.
Have a catalogue worth watching?
Tell us which sources matter and what a useful report would have to say. We will scope the collection honestly, including the parts we would decline.
Start a conversation