On a marketplace where the same product is sold by forty accounts, the product price is meaningless. What decides whether a sale happens is the specific offer — its delivery terms, its badge status, its position — and that is what the client's pricing team had never been able to see.
The source is named because it is public; the client is not. Every figure on this page is a measured production number the client agreed to publish. Have a source of your own? Send it over, whatever its size — you get an answer within 24 hours.
offer records since 2022
a year
measured coverage
records a day
Not more rows. Every one of these projects started with somebody who already had data and could not use it for the decision in front of them.
The problem
The client sells electronics on a marketplace where they compete with dozens of accounts on identical SKUs. Their pricing tool worked from product-level prices, which meant it could see that they were being undercut but never by whom, on what terms, or whether the cheaper offer was actually winning any traffic.
Fixed on day one
Every one of these is measured continuously and reported on the same dashboard the client watches. A commitment nobody measures is a sentence in a proposal.
None of these is solved by better headers or a bigger proxy pool. Each needs a different piece of engineering, and working out which one you are actually facing is most of the job.
The visible price is one offer out of many, and the competitive picture the client needs is the whole list — which costs an order of magnitude more to collect than the headline.
What we doOffer-level collection restricted to the products the client actually competes on, with the set maintained automatically as their assortment and their competitors' change.
Free delivery, delivery time and badge status shift which offer wins regardless of price. A price-only comparison explains nothing about why the client is losing.
What we doDelivery terms, badge status and handling time are captured per offer and delivered alongside price, so the pricing model compares landed propositions rather than numbers.
Competing accounts change display names, split into multiple accounts and relist their offers, which breaks any tracking keyed on what is shown.
What we doSeller identity is maintained on stable identifiers with a history, so a rename is a recorded event on one entity rather than the appearance of a new competitor.
A same-day repricing loop needs the competitive picture to be hours old, not a day, across millions of offers.
What we doRefresh priority follows competitive intensity: contested products are swept every six hours, uncontested ones daily, with promotion driven by observed movement.
The architecture is not interesting; every extraction system has a queue, a fetcher and a parser. These are the decisions that made this one work where the last one did not.
The unit is the offer. Product-level collection is cheap and cannot answer the only question the client has, which is who is beating them and how.
Price, delivery, badge and handling time are collected together, because the customer's decision uses all four and a pricing model that sees one of them will keep being surprised.
Stable identifiers with history mean a competitor renaming or splitting accounts is visible as an event rather than as a new arrival.
Products where offers move get six-hourly refresh; quiet ones get daily. The set is maintained automatically as competition shifts.
Data lands four times a day into the client's repricing engine, which is what makes a competitive picture actionable rather than historical.
Production figures, not a benchmark run. Coverage is recounted daily against an independent sample of the live source rather than asserted, which is why the numbers are not round.
Daily profile
One product page carries every competing offer — about twelve — so 6.8 million requests a day, 79 a second, yield 84 million offer records: 2.5 billion a month, 31 billion a year, 96 billion since 2022. Only 840 000 contested products get the six-hourly cadence; the rest of the twelve-million-product catalogue is swept once a day, which is the difference between a bill the client can pay and one they cannot.
Headline
These four sit in the support agreement. When one of them drifts outside its band, we are alerted within fifteen minutes and fixing it is routine work under the monthly arrangement, not a change request.
The columns are the client's own numbers from before the rebuild and the measured ones from production today. The left column is the part most vendors would rather not put on a page.
| Metric | Before | Today |
|---|---|---|
Competitive view |
Cheapest price per product | Every offer, with terms |
Repricing loop |
Next day | Same day, four cycles |
Competitor tracking |
Broke on rename | Persistent identities |
Losing without knowing why |
Routine | Explained by terms or badge |
Roughly a third of the cases where the client was losing turned out not to be about price at all. They were about delivery time, which nobody had been measuring.
One case file per source, each opening the same way: what the client needed, what the source put in the way, how the pipeline is built and what the numbers are today.
1 950 stores, promotions caught the day they turn
Read the case →
2 800 stores, loyalty and shelf priced separately
Read the case →
580k merchants, the fee stack per address
Read the case →
420k vehicles, availability by the hour
Read the case →
Half a million events on sale, priced continuously
Read the case →
Start here
Send the source, the fields you need and roughly how often. You get a straight answer within 24 hours: whether it can be done, what makes it hard, what coverage is achievable and roughly what it costs to build and to run. Whatever the size.
Direct
NDA before technical detail, as always. If it isn't our kind of work, we say so in the first reply.