UK grocery has two prices for a great many products: the shelf price and the loyalty price. Collapsing them into one number, as most feeds do, produces a price index that describes a shopper who does not exist.
The source is named because it is public; the client is not. Every figure on this page is a measured production number the client agreed to publish. Have a source of your own? Send it over, whatever its size — you get an answer within 24 hours.
SKU-store rows to date
a year
measured coverage
rows a day
Not more rows. Every one of these projects started with somebody who already had data and could not use it for the decision in front of them.
The problem
The client publishes a grocery price index used by analysts and the press, so a methodological flaw is a reputational problem rather than a data-quality one. Their previous supplier delivered a single price per product per store, with no statement of which tier it represented — and the answer turned out to vary by category.
Fixed on day one
Every one of these is measured continuously and reported on the same dashboard the client watches. A commitment nobody measures is a sentence in a proposal.
None of these is solved by better headers or a bigger proxy pool. Each needs a different piece of engineering, and working out which one you are actually facing is most of the job.
Loyalty and shelf prices coexist and the gap between them is often the entire promotion. A feed with one price field is making an undocumented editorial choice on every row.
What we doBoth tiers are captured as separate fields on every observation, and the index methodology chooses between them explicitly rather than inheriting a collector's default.
Three-for-two, meal deals and spend-and-save cannot be expressed as a unit price without assuming a basket, and that assumption belongs to the client, not to us.
What we doPromotional mechanics are captured structurally — type, threshold, reward — so the client applies their own basket assumptions and can defend them publicly.
Convenience, superstore and online carry different prices for the same product, and averaging across formats produces a number that describes no shopper.
What we doStore format is a first-class dimension and nothing is averaged across it inside the pipeline.
When the output is quoted in the press, the collection method has to survive being questioned by somebody hostile.
What we doMethodology, coverage measurement and known limitations are documented to a standard the client can publish, and the coverage recount is designed to be reproducible by a third party.
The architecture is not interesting; every extraction system has a queue, a fetcher and a parser. These are the decisions that made this one work where the last one did not.
Shelf and loyalty are separate fields on every row. Any collapse into one number is a decision the client makes and documents, not one the pipeline makes silently.
Promotion type, threshold and reward are recorded as structure. Converting a three-for-two into a unit price requires a basket assumption that belongs to the client's published methodology.
Convenience, superstore and online are distinct. Averaging across them is the fastest way to publish a number nobody recognises.
Coverage measurement, sampling and known gaps are written down in a form the client can put in front of a journalist. This was a requirement, not a courtesy.
The daily recount is specified so a third party could repeat it. An index that cannot be audited is an index waiting to be attacked.
Production figures, not a benchmark run. Coverage is recounted daily against an independent sample of the live source rather than asserted, which is why the numbers are not round.
Daily profile
Sixteen thousand items across 2 800 stores is 45 million rows a day — 1.35 billion a month, 16 billion a year, 48 billion since 2023 — from 2.6 million requests at 30 a second. Nine million of those rows a day carry a loyalty price as well as a shelf price: 3.3 billion a year on which a single-price feed silently picks one of the two tiers, and never says which.
Headline
These four sit in the support agreement. When one of them drifts outside its band, we are alerted within fifteen minutes and fixing it is routine work under the monthly arrangement, not a change request.
The columns are the client's own numbers from before the rebuild and the measured ones from production today. The left column is the part most vendors would rather not put on a page.
| Metric | Before | Today |
|---|---|---|
Price tiers |
One undocumented field | Shelf and loyalty, separate |
Promotions |
Flattened into a price | Structured mechanics |
Store format |
Averaged | A dimension |
Methodology |
Not publishable | Documented and reproducible |
The index was challenged in print the year after launch. The methodology document was the reason the challenge went nowhere.
One case file per source, each opening the same way: what the client needed, what the source put in the way, how the pipeline is built and what the numbers are today.
580k merchants, the fee stack per address
Read the case →
420k vehicles, availability by the hour
Read the case →
Half a million events on sale, priced continuously
Read the case →
135M properties, and the whole live market daily
Read the case →
1.6M properties, 120M rate quotes a day
Read the case →
Start here
Send the source, the fields you need and roughly how often. You get a straight answer within 24 hours: whether it can be done, what makes it hard, what coverage is achievable and roughly what it costs to build and to run. Whatever the size.
Direct
NDA before technical detail, as always. If it isn't our kind of work, we say so in the first reply.