The UK residential market is well served by one dominant portal, which is convenient until you try to collect all of it. A hard ceiling on results per search means the naive approach silently caps at a fraction of the market — and the fraction is not random, it is the cheap end.
The source is named because it is public; the client is not. Every figure on this page is a measured production number the client agreed to publish. Have a source of your own? Send it over, whatever its size — you get an answer within 24 hours.
property records to date
a year
measured coverage
UK properties in the graph
Not more rows. Every one of these projects started with somebody who already had data and could not use it for the decision in front of them.
The problem
The client models residential valuations for lenders, so a systematically missing segment is a systematically wrong model. Their previous supplier delivered a large file each night and had never measured what share of the live market it represented — and when we measured it, the answer was seventy-one per cent, concentrated in exactly the price bands the lender cared least about.
Fixed on day one
Every one of these is measured continuously and reported on the same dashboard the client watches. A commitment nobody measures is a sentence in a proposal.
None of these is solved by better headers or a bigger proxy pool. Each needs a different piece of engineering, and working out which one you are actually facing is most of the job.
Any search returns a bounded number of results however it is phrased, so a national sweep by area alone caps out in every city and quietly loses the busiest markets first.
What we doRecursive partitioning by geography, price band and property type, with each partition carrying a proof that it sits under the ceiling and splitting automatically when it approaches it.
A listing that disappears may have sold, been withdrawn, or been relisted at a new price by the same agent an hour later. Treating those as one event corrupts every time-on-market figure downstream.
What we doDisappearances are classified using relisting detection across agent, address and media fingerprints, so a relist is recognised as the same property rather than counted as a new one.
Price reductions are the strongest signal in the dataset and the source shows only the current asking price. If you were not observing yesterday, yesterday does not exist.
What we doEvery observed state is retained with its observation time, and price trajectories are reconstructed from our own record. Six years in, that archive is worth more than the daily file.
A national sweep that overruns the client's morning model run is a sweep that did not happen, and load shaping means we cannot simply add parallelism until it fits.
What we doPartitions are scheduled by expected cost and volatility so the sweep finishes inside the window with the shaping constraint respected, and the slow tail is started first rather than last.
The architecture is not interesting; every extraction system has a queue, a fetcher and a parser. These are the decisions that made this one work where the last one did not.
Every partition carries a proof of sitting under the ceiling, re-checked on every sweep. When a town gets busy, its partition splits before anything is lost rather than after a quarter of reporting.
Address normalisation, agent identity and media fingerprints are used to recognise the same property returning under a new listing, which is what makes time-on-market figures trustworthy.
Observations are never overwritten. The client's most valuable derived fields — price trajectory, days on market, reduction frequency — all come from our own history rather than from the source.
Cost and volatility drive the order of work so the sweep completes before the model run, with the expensive tail started early instead of left to the end.
An independent sample of the live market is collected each day by a different path, and the delivered file is measured against it. That measurement is the coverage figure in the client's dashboard.
Production figures, not a benchmark run. Coverage is recounted daily against an independent sample of the live source rather than asserted, which is why the numbers are not round.
Daily profile
The live market is swept completely inside four hours — 1.5 million requests at 104 a second, which is the number the whole partitioning strategy is designed around, not the listing count. The standing graph of 29 million properties turns over on a rolling thirty-day cycle, another million requests spread across the day. Together: 9.4 million records a day, 280 million a month, 3.4 billion a year, 12 billion since 2022.
Headline
These four sit in the support agreement. When one of them drifts outside its band, we are alerted within fifteen minutes and fixing it is routine work under the monthly arrangement, not a change request.
The columns are the client's own numbers from before the rebuild and the measured ones from production today. The left column is the part most vendors would rather not put on a page.
| Metric | Before | Today |
|---|---|---|
Share of live market delivered |
About 71%, unmeasured | 99.0%, recounted daily |
Time on market |
Wrong wherever relisting occurred | Correct, relists detected |
Price history |
None | Six years and growing |
Sweep completion |
Frequently after the model run | Inside the four-hour window |
The valuation model was rebuilt on the new history within a quarter. The lender's risk team asked for the coverage measurement in writing, which is not a question anybody had thought to ask the previous supplier.
One case file per source, each opening the same way: what the client needed, what the source put in the way, how the pipeline is built and what the numbers are today.
Three countries, 12M properties, one model
Read the case →
The market, and the 1.9M people selling it
Read the case →
7.4M listings, 210M calendar-nights a day
Read the case →
46M itineraries priced a day, decomposed
Read the case →
94k routes, 210M fare quotes a day
Read the case →
Start here
Send the source, the fields you need and roughly how often. You get a straight answer within 24 hours: whether it can be done, what makes it hard, what coverage is achievable and roughly what it costs to build and to run. Whatever the size.
Direct
NDA before technical detail, as always. If it isn't our kind of work, we say so in the first reply.