Full inventory, not a niche
Every body type the dealer's own search offers — adventure, naked, sport, cruiser, classic, scooter and the rest. 1208 listings captured across 13 categories.
A Playwright scraping pipeline that reads every used bike on bigmoto.co.uk — 13 body types and counting — models what each one should cost from real Auto Trader and MCN classifieds listings of the same model and era, and ranks the outliers. Built to answer one question: is this bike a deal, or is it just old?
A cheap-looking 2018 adventure bike is usually cheap for a reason: high mileage, an old era, or a model nobody wants. This project replaces the guesswork with a per-bike expected price, derived from the live market for that exact model, and then measures the gap.
Every body type the dealer's own search offers — adventure, naked, sport, cruiser, classic, scooter and the rest. 1208 listings captured across 13 categories.
Expected price is fitted on price vs mileage within the bike's own model and era, not a flat category average. Low mileage is rewarded, but only up to what the market actually pays.
A deal-finder that cries wolf is worthless. Every scoring stage has a bail-out: refuse to score rather than guess, and label the confidence when it does.
No framework, no database, no scheduler to learn. Every stage reads a file and writes a file, so any
step can be re-run on its own and the whole pipeline is debuggable by opening a spreadsheet.
run_pipeline.py chains them end to end, and is what the host's
scheduler calls — it logs every stage, snapshots the result and emails when something notable changed.
scrape_bigmoto_pw.pyPlaywright walk of every body-type search on the dealer's site, with Cloudflare-challenge handling and gap refetching.
fetch_autotrader.pyShallow Auto Trader comparables — one keyword search per model found in the stock list.
deep_fetch.pyEra-ranged queries (year-from/year-to) so old bikes get old comparables. Idempotent by URL.
fetch_mcn.pyThe second marketplace: MCN's own classifieds via the public search API their pages use, deduplicated against Auto Trader.
score_deals.pyBuilds the per-model market model and emits expected price, deal %, comp count, method and confidence.
scrape_autotrader_deals.pyThe census: a whole-of-market walk of Auto Trader's adventure, sports-tourer and naked ads, split into year bands to beat the 100-page cap. It refreshes and stamps last_seen on every advert it serves, and harvests each advert's photo from the same page, so the market and its photographs arrive in one pass.
retire_stale_autotrader.pyMoves out what the crawls can no longer vouch for — adverts Auto Trader no longer lists, measured at roughly a quarter of the corpus, and comparables whose price has not been re-read in weeks. Guarded: it will not act on a census that did not finish, and it refuses a prune far larger than the measured dead share.
score_autotrader.pyScores every Auto Trader listing against its own model's market — deals that never touch BigMoto.
distance.pyAnnotates each Auto Trader listing with miles from the dealer's postcode. Distance is reported, never used to fit the market.
enrich_details.pyOpens detail pages for the shortlist to pull the admin fee, owner count, service history and MOT.
fetch_mot_history.pyReads the registration enrich_details.py captured off each advert and fetches that bike's real MOT record from the DVSA MOT History API: expiry, defect and advisory counts, and the odometer at every test, normalised to miles. It replaced a hand-off to the gov.uk checker with the actual record, which is what makes the mileage check possible at all.
harvest_images.pyOne photo URL per listing, read from the search pages rather than 1,215 detail pages. Resumable, and optional: without it every card draws a placeholder.
harvest_autotrader_images.pyBackfill for Auto Trader photos the census missed, over the same search pages. It is no longer part of the routine scan — the census writes both outputs from one walk — but it is what fills a gap by hand.
classify_images.pyMeasures each photo's pixels to spot the listings whose first image is a finance or delivery promo banner, then substitutes a clean gallery photo where one exists.
build_dashboard.pyBuilds dashboard.html: the page skeleton, the payload, and the server-rendered chrome. The front end itself lives in mh_ui.py (design tokens, the utility layer, the component styles, one icon set) and mh_dash_js.py (the client script), so the builder stays about building.
build_info_page.pyRenders this page from the same CSVs, so the numbers here are regenerated rather than maintained by hand.
A deal-finder that cries wolf is worthless, so these run after every producer and gate the run.
data_quality.pyChecks every crawl against invariants and a stored baseline. A parser that quietly stops matching fails the run instead of shrinking the corpus.
check_readme.pyOwns every figure the README publishes and generates its summary tables from the data, so prose cannot drift from the CSVs.
snapshot_deals.pyDiffs today's scored data against yesterday's and reports new deals, price drops, rises and withdrawals.
banding.pyThe one place a deal band is named, ranked and styled, so the page, the dashboard and the alerts cannot disagree about what a deal is.
Shared browser layer: pw_client.py — one Playwright client that waits out the JS challenge on the
same page, rotates browser contexts, and relaunches the whole browser process when a context goes stale.
Comparables are every Auto Trader listing for the same make and model, matched by compact-substring (so R 1250 GS pairs with R 1250 GS Adventure, but V-Strom 650 never pairs with V-Strom 1050). The scorer then walks down a chain of methods and stops at the first one that is statistically defensible. 69,511 comparables back the BigMoto scores; the Auto Trader tab uses 748,422 more.
Deal distribution across all 1208 BigMoto bikes, where deal % = (expected − asking) / expected. 44 bikes have no score at all: the scorer refused a valuation rather than guess one.
The scorer values a bike from comparables of its own registration year first, widening to
±1, ±2 then ±4 years only when the own-year pool is too thin. That order was chosen by benchmark,
and the honest reading of the benchmark is narrower than it first looks: a same-year fit scores
5.7% median error, the ±1-year chain it replaced scores 5.8%, and the shipped
own-year-first estimator scores 5.9% — a tie inside the per-fold spread, not a win.
What the numbers do support is that the year restriction carries the accuracy: ±3-year
medians degrade to 9% and a year-blind median to 12.3%. (tools/benchmark_valuation.py
re-runs it.) If no pool qualifies the scorer returns no data instead of a number —
44 bikes in this run are in that bucket, shown as such rather than quietly dropped.
BigMoto advertises on Auto Trader too. Any comparable matching the bike's own price, year and mileage is dropped, so a listing can never be compared against itself.
Every score carries an 80% interval — a range the market level for that model plausibly sits in, given how many comparable listings went into it and how tightly they agree. It is not the spread of individual asking prices, which is much wider; it is how well the model knows the average. A bike is only called a deal when its price falls below the bottom of that range, so 109 bikes here are cheap but inside the model's own margin of error — labelled ? rather than sold as bargains. Where a pool is too thin to narrow the range at all, the model is saying "I don't know", and a wide interval must not read as a big discount.
same-year724 bikes · 60%OLS price↔mileage fit on comparables of the bike's own registration year
same-year-median249 bikes · 21%Own-year median (used when the own-year pool is too small to fit a slope)
year1117 bikes · 10%±1-year regression fit, used only when the own-year pool is too thin
year1-median39 bikes · 3%±1-year median
none38 bikes · 3%No usable market data for this model at all
year211 bikes · 1%±2-year regression fit (wider, so less precise)
same-year-mileage-median7 bikes · 1%—
era-gap6 bikes · 0%Refused to score — comparables do not cover the bike's era
year2-median5 bikes · 0%±2-year median
all-years4 bikes · 0%Every comparable for the model, regardless of year — last resort
year1-mileage-median3 bikes · 0%—
all-years-median2 bikes · 0%—
year42 bikes · 0%±4-year regression fit (wide, low confidence)
year2-mileage-median1 bikes · 0%—
A pool of comparables skews newer than the bike being scored, so a pool-wide median makes every
older bike look like a bargain. Fixed with the era-gap bail-out and era-proximate windows.
A mileage fit evaluated far outside the window that produced it can return fantasy prices. Fix: no extrapolation beyond ±5k miles of the window, and a hard clamp to the window's own min/max price.
Below the minimum comparable count the scorer falls back to a window median and labels the result low-confidence rather than fitting a line to three points.
The dashboard flags low mileage relative to age (≤40% of typical = very low), so a genuinely fresh bike is distinguishable from a leggy one that is merely cheap.
Every row here is a bike priced below the bottom of the interval the model puts around its own market estimate — a gap its own uncertainty cannot explain. Every one of them cleared the model's own margin of error. Click a model to open the actual listing.
| # | Bike | Year | Miles | Ask | Expected | Deal | Conf. | Compare |
|---|---|---|---|---|---|---|---|---|
| 1 | BMW R 1200 GS
Adventure · 1170 cc · White |
2018 | 17,854 | £7,188£6,989 + £199 fee | £9,999 | +28.1% | high | Auto Trader · MOT |
| 2 | HARLEY-DAVIDSON SPORTSTER
Cruiser · 883 cc · Silver |
2020 | 1,109 | £6,288£6,089 + £199 fee | £8,634 | +27.2% | medium | Auto Trader · MOT |
| 3 | BMW F 850 GS
Adventure · 853 cc · Red |
2021 | 13,299 | £5,688£5,489 + £199 fee | £7,515 | +24.3% | high | Auto Trader · MOT |
| 4 | KTM 890 Duke
Naked · 889 cc · Orange |
2021 | 4,502 | £4,688£4,489 + £199 fee | £6,163 | +23.9% | high | Auto Trader · MOT |
| 5 | ROYAL ENFIELD SUPER METEOR 650
Classic / Retro · 648 cc · Other |
2025 | 5,805 | £4,188£3,989 + £199 fee | £5,339 | +21.6% | high | Auto Trader · MOT |
| 6 | TRIUMPH STREET TRIPLE 765
Naked · 765 cc · Silver |
2024 | 460 | £7,788£7,589 + £199 fee | £9,899 | +21.3% | high | Auto Trader · MOT |
| 7 | YAMAHA R1
Sport · 998 cc · Red |
2000 | 21,644 | £3,188£2,989 + £199 fee | £3,979 | +19.9% | medium | Auto Trader · MOT |
| 8 | INDIAN FTR
Naked · 1203 cc · Black |
2021 | 5,774 | £6,688£6,489 + £199 fee | £8,190 | +18.3% | high | Auto Trader · MOT |
| 9 | HONDA CBR1000RR FIREBLADE Sport · 1000 cc · Black | 2019 | 2,707 | £9,488£9,289 + £199 fee | £11,554 | +17.9% | medium | Auto Trader · MOT |
| 10 | KTM 125 DUKE
Naked · 125 cc · Blue |
2013 | 18,313 | £2,488£2,289 + £199 fee | £3,017 | +17.5% | medium | Auto Trader · MOT |
573 of 1208 scored bikes carry high confidence, and 90 sit at 6% or more below their modelled market value. Ask is the drive-away total — the advertised price plus BigMoto's admin fee, which 1213 of the 1213 enriched detail pages carry — and the deal % is measured against that same total, so the two reconcile.
The same scoring engine runs across Auto Trader's own inventory, which catches under-priced bikes anywhere in the country — including ones BigMoto will never list.
Auto Trader serves at most ~2,000 listings per query, so the crawl splits each body type into year bands that each stay under the ceiling. The only remaining gap is the busiest 2026 adventure band, which loses roughly the last 100 pages' worth of ads.
This run scored 14,116 trade and 932 private ads. Private asking prices sit well below dealer ones for the same model, so each listing is valued against a pool of its own seller type wherever that pool is large enough — otherwise the comparison measures the seller, not the bike. The dashboard shows which is which on every row.
dashboard.html is a single self-contained file — no server, no
CDN, no build step. Open it with file:// and it works offline,
with the data embedded as JSON.
BigMoto stock scored against the market, and Auto Trader deals scored against their own model — 1208 + 15,048 rows. The source switch sits with the scan's numbers, so it never competes with the motorcycles for the top of the page.
Deals to browse, Shortlist to keep, Compare to decide and Market to understand the scan. The card view is the default; the dense sortable table is one click away and remembers the choice.
The seven questions asked on every visit — make, model, category, max price, min deal %, mileage and year — stay in the open. Confidence, evidence, first seen and the four toggles fold behind More filters. On anything narrower than a laptop the whole panel becomes a sheet. Filter state lives in the URL, so a view can be bookmarked or sent to someone.
The crawler tags twelve categories — naked, adventure, sport, classic, scooter, cruiser, sports tourer, tourer, motocross, supermoto, enduro and three-wheeler — and the page reads that list out of the data. The filter carries a count per category, the summary strip names the two largest, and the Market view has a category board showing each one's median discount and best recorded deal. Add a category to the crawler and it appears here by itself.
Type Triumph under £8,000 or adventure under 30,000 miles and the price and mileage clauses are lifted onto the real filters and shown as chips you can remove. Nothing is invented: a clause can only ever narrow the list, and free text still behaves exactly as it always did.
Star any bike to keep it, then filter to the shortlist. Pick up to four — explicitly, or by falling back to the shortlist — and Compare puts them on one shared price axis and then in a table that marks the row that differs and the value that wins.
Every bike has its own page: listed at, estimated market value, and the difference — underneath, the comparable count that produced it, the model's 80% range drawn as a distribution, the other examples of that make and model in the scan, and the price history if the snapshots can prove one.
Rows that moved since an earlier run are flagged with the change, using the dated snapshots the pipeline already writes — and saying nothing at all where the snapshots cannot prove a claim.
Download whatever you have filtered to as CSV, or print the current view as a clean paper shortlist with no chrome.
The DVSA MOT History API now supplies each bike's real test record — expiry, defect and advisory counts, and the odometer at every test — shown as a badge on the row. The highest recorded odometer is compared with the advertised mileage, but it is evidence, not a verdict: an odometer can be replaced and a private plate moves between bikes, so a contradiction is flagged for a human and never asserted as a clocking claim.
Python 3.13 with requests,
bs4, lxml and
playwright (Chromium installed). Node is only needed to
syntax-check the dashboard's JavaScript.
# from the project root — scrapes, scores, enriches, builds python -u run_pipeline.py # or step by step (each stage is independent) python -u scrape_bigmoto_pw.py # -> data/bigmoto.csv python -u fetch_autotrader.py # -> data/autotrader.csv python -u deep_fetch.py # era-ranged comps (idempotent) python -u score_deals.py # -> data/scored.csv python -u scrape_autotrader_deals.py # -> data/autotrader_deals.csv python -u score_autotrader.py # -> data/autotrader_scored.csv python -u enrich_details.py # -> data/details.csv python -u build_dashboard.py # -> dashboard.html
# no scraping needed if the CSVs are already populated python -u score_deals.py python -u score_autotrader.py python -u enrich_details.py python -u build_dashboard.py python -u build_info_page.py # -> about.html (this page) # serve the dashboard locally python -m http.server 8000 # http://127.0.0.1:8000/dashboard.html
# dashboard integrity + the JS it embeds python validate_dashboard.py node --check _check.js # must print: JS syntax OK # is the scoring healthy? want mostly window1*, few medians Import-Csv data/scored.csv | Group-Object method
23,239 lines of pipeline Python (plus 23,876 lines of
probes/, tests/ and
tools/ scripts — the experiments that reverse-engineered the two
sites, the tests that keep them honest, and the audits and one-off repairs run by hand).
Both sources are protected and JavaScript-rendered. These are the findings that survived contact, kept in the repo so nobody has to rediscover them.
Plain requests gets a 403 challenge; headless Playwright Chromium clears it reliably
on a fresh browser process. Re-navigating during a challenge resets it — wait on the same page
instead. A page titled "Just a moment…" is a challenge; an empty page still says "Bigmoto".
Legacy REST endpoints 404, the GraphQL endpoint is Cloudflare-protected. The winning approach is
parsing the rendered DOM by data-testid. A postcode parameter is mandatory or the page
only shows a prompt, and make must accompany keywords or the search returns
unrelated bikes.
If a navigation fails, the previous query's cards stay in the DOM and get parsed as results. Guard: assert the URL-encoded keyword appears in the final URL before trusting the cards.
body-type=Adventure returns thousands of bikes; adventure returns zero.
The full facet list is saved in html/facets_resp_0.json.
sold_prices.csv has to be filled in by handInvestigated properly, no route exists. Recorded here so nobody spends another afternoon on them.