A secondary marketplace cannot price anything without knowing what the primary market is doing right now — but it only needs that for the events that are actually moving. We built the pipeline that decides which ones those are, every few minutes, across nine countries. Running continuously since 2019, and every new event that goes on sale is picked up the same day.
The source is named because it is public; the client is not. Every figure on this page is a measured production number the client agreed to publish. Have a source of your own? Send it over, whatever its size — you get an answer within 24 hours.
price records since 2019
a year
measured coverage
records a day
Not more rows. Every one of these projects started with somebody who already had data and could not use it for the decision in front of them.
The problem
The client prices secondary inventory against the primary market. Before us they bought a nightly feed, which meant every pricing decision on a hot event was made against data up to fourteen hours old. What they needed was not more rows — it was the same rows, minutes old, for the events that actually move, without paying minute-level prices for the long tail that does not.
Fixed on day one
Every one of these is measured continuously and reported on the same dashboard the client watches. A commitment nobody measures is a sentence in a proposal.
None of these is solved by better headers or a bigger proxy pool. Each needs a different piece of engineering, and working out which one you are actually facing is most of the job.
Queue systems put every client into a lobby before anything is served. A worker that simply retries burns its entire budget waiting, and the events worth watching are exactly the ones with the longest queues.
What we doA queue-aware scheduler with a budget per event: the pipeline decides which events deserve a slot in the next few minutes and which can wait for the slow sweep, instead of spending capacity uniformly.
The source does not score what you send, it scores how you move — pacing, navigation depth, the order in which things are touched. A technically perfect request from a session that behaves wrongly is still refused.
What we doSession behaviour modelled on how a real client of that source actually moves, refitted whenever the scoring changes. That model is the asset; everything else is plumbing around it.
The seat map does not exist in the page. It is assembled client-side from a private interface, which is why most vendors reach for a browser and end up with a cost per refresh that makes minute-level freshness impossible.
What we doWe read the interface directly and keep a browser in reserve for the small share of cases where nothing else works. That single decision is most of the two-orders-of-magnitude gap in cost per refresh.
A price two minutes old is worthless for a hot event and perfectly fine for a show in eight months. Treating both the same way is either ruinously expensive or uselessly stale.
What we doEvents are ranked continuously by volatility and given a refresh budget accordingly, so freshness is spent where it changes a pricing decision.
The architecture is not interesting; every extraction system has a queue, a fetcher and a parser. These are the decisions that made this one work where the last one did not.
The first thing the pipeline does each cycle is decide what is worth looking at. Volatility, time to event, remaining inventory and the client's own exposure feed a ranking that allocates the next few minutes of capacity. Most systems in this space fetch first and think later, which is why their freshness collapses on exactly the days that matter.
Identity rotation without a behaviour model is noise. We maintain a single model of how a legitimate client of this source behaves and instantiate it across identities, so every session is individually plausible rather than merely different from the last one.
Where the source exposes a private interface to its own front end, that is what we read. Browsers are expensive, fragile and slow, and reserving them for genuine exceptions is what keeps a minute-level refresh affordable at this catalogue size.
A second process samples the source independently — different entry points, different partitioning — and its result is compared against what the pipeline delivered. The difference is the coverage number. When it moves, we know within minutes rather than at the end of the quarter.
Ninety-second data in a nightly file is ninety-second data nobody can use. Records land on a stream the moment they are reconciled, with a versioned schema, and the client's pricing engine consumes them directly.
Production figures, not a benchmark run. Coverage is recounted daily against an independent sample of the live source rather than asserted, which is why the numbers are not round.
Daily profile
The arithmetic is the whole design. 500 000 events are on sale at any moment and 2.1 million distinct events pass through the catalogue in a year, because events go on sale and then happen. The refresh is tiered: 12 000 volatile events every three minutes over an eighteen-hour window is 4.3 million refreshes, 90 000 warm events twelve times a day adds 1.1 million, the remaining 400 000 are swept twice. That is 6.2 million refreshes a day — 72 a second, sustained. Each one returns the event's whole price map, about 25 rows, so 155 million price records a day, 4.7 billion a month, 57 billion a year and roughly 210 billion since 2019. Refreshing all 500 000 events every three minutes instead would be 180 million requests a day, and no client on earth could pay for it.
Headline
These four sit in the support agreement. When one of them drifts outside its band, we are alerted within fifteen minutes and fixing it is routine work under the monthly arrangement, not a change request.
The columns are the client's own numbers from before the rebuild and the measured ones from production today. The left column is the part most vendors would rather not put on a page.
| Metric | Before | Today |
|---|---|---|
Freshness on volatile events |
Nightly batch, up to 14 h old | Three minutes |
Catalogue coverage |
62% of events, tail missing | 99.2%, recounted daily |
Decisions made on stale data |
31% of repricing events | Under 2% |
Engineering spent on collection |
3 FTE, permanently firefighting | None on the client's side |
The pipeline has run through two changes of bot vendor on the source, one full redesign of its front end, and a migration of the client's warehouse. It has not been rebuilt.
One case file per source, each opening the same way: what the client needed, what the source put in the way, how the pipeline is built and what the numbers are today.
135M properties, and the whole live market daily
Read the case →
1.6M properties, 120M rate quotes a day
Read the case →
A 180M-SKU competitive catalogue, swept weekly
Read the case →
Every store, every day: 64M rows
Read the case →
640k merchants found by a 46k-point grid
Read the case →
Start here
Send the source, the fields you need and roughly how often. You get a straight answer within 24 hours: whether it can be done, what makes it hard, what coverage is achievable and roughly what it costs to build and to run. Whatever the size.
Direct
NDA before technical detail, as always. If it isn't our kind of work, we say so in the first reply.