last run 2026-11-01 reported no failures · page built 01 Nov 2026 18:15

Find the motorcycle priced below its market.

A Playwright scraping pipeline that reads every used bike on bigmoto.co.uk — 13 body types and counting — models what each one should cost from real Auto Trader and MCN classifieds listings of the same model and era, and ranks the outliers. Built to answer one question: is this bike a deal, or is it just old?

1,208
bikes scraped
BigMoto stock, 13 body types
15,048
market listings
Auto Trader ads scored — 14,116 trade, 932 private
817,933
comparables used
same-model, same-era comps matched
54
mega deals
≥10% below modelled market value
01 — The problem

Deal sites list the price. They don't tell you the market.

A cheap-looking 2018 adventure bike is usually cheap for a reason: high mileage, an old era, or a model nobody wants. This project replaces the guesswork with a per-bike expected price, derived from the live market for that exact model, and then measures the gap.

🎯

Full inventory, not a niche

Every body type the dealer's own search offers — adventure, naked, sport, cruiser, classic, scooter and the rest. 1208 listings captured across 13 categories.

📉

Mileage-aware valuation

Expected price is fitted on price vs mileage within the bike's own model and era, not a flat category average. Low mileage is rewarded, but only up to what the market actually pays.

🛡️

False positives are the enemy

A deal-finder that cries wolf is worthless. Every scoring stage has a bail-out: refuse to score rather than guess, and label the confidence when it does.

02 — What's in the dataset

13 body types, 1208 bikes, one market model

⬡

Naked

343 bikes
Median ask
£4,389
Price band
£1,589 – £14,989
2026 pre-regs (0 mi)
70 zero-mile + 143 under 5k
▲

Adventure

233 bikes
Median ask
£5,989
Price band
£1,789 – £19,489
2026 pre-regs (0 mi)
27 zero-mile + 74 under 5k
◈

Classic / Retro

156 bikes
Median ask
£5,389
Price band
£1,689 – £12,589
Low mileage
95 under 5,000 miles
⚡

Sport

149 bikes
Median ask
£5,889
Price band
£1,889 – £17,989
2026 pre-regs (0 mi)
12 zero-mile + 48 under 5k
●

Scooter

110 bikes
Median ask
£2,999
Price band
£1,489 – £13,289
2026 pre-regs (0 mi)
71 zero-mile + 32 under 5k
◇

Cruiser

106 bikes
Median ask
£5,689
Price band
£1,789 – £17,489
2026 pre-regs (0 mi)
19 zero-mile + 48 under 5k
◆

Sports Tourer

56 bikes
Median ask
£6,589
Price band
£3,189 – £17,289
Low mileage
12 under 5,000 miles
▵

Motocross

15 bikes
Median ask
£3,789
Price band
£3,189 – £5,789
2026 pre-regs (0 mi)
15 zero-mile + 0 under 5k
▣

Tourer

14 bikes
Median ask
£7,539
Price band
£2,989 – £19,789
Low mileage
1 under 5,000 miles
◐

Supermoto

12 bikes
Median ask
£5,439
Price band
£2,799 – £7,889
2026 pre-regs (0 mi)
1 zero-mile + 6 under 5k
△

Enduro

7 bikes
Median ask
£6,289
Price band
£1,889 – £8,289
Low mileage
3 under 5,000 miles
◬

Three-wheeler

6 bikes
Median ask
£5,389
Price band
£2,789 – £10,789
Low mileage
4 under 5,000 miles
·

Other

1 bikes
Median ask
£3,789
Price band
£3,789 – £3,789
Low mileage
0 under 5,000 miles
Asking prices span £1,489–£19,789 (median £5,289). 215 are 2026 pre-registration bikes with delivery mileage, and 466 more are under 5,000 miles.
03 — Architecture

16 standalone stages, one CSV handoff each

No framework, no database, no scheduler to learn. Every stage reads a file and writes a file, so any step can be re-run on its own and the whole pipeline is debuggable by opening a spreadsheet. run_pipeline.py chains them end to end, and is what the host's scheduler calls — it logs every stage, snapshots the result and emails when something notable changed.

1
scrape_bigmoto_pw.py

Playwright walk of every body-type search on the dealer's site, with Cloudflare-challenge handling and gap refetching.

→ data/bigmoto.csv
2
fetch_autotrader.py

Shallow Auto Trader comparables — one keyword search per model found in the stock list.

→ data/autotrader.csv
3
deep_fetch.py

Era-ranged queries (year-from/year-to) so old bikes get old comparables. Idempotent by URL.

→ data/autotrader.csv
4
fetch_mcn.py

The second marketplace: MCN's own classifieds via the public search API their pages use, deduplicated against Auto Trader.

→ data/mcn.csv
5
score_deals.py

Builds the per-model market model and emits expected price, deal %, comp count, method and confidence.

→ data/scored.csv
6
scrape_autotrader_deals.py

The census: a whole-of-market walk of Auto Trader's adventure, sports-tourer and naked ads, split into year bands to beat the 100-page cap. It refreshes and stamps last_seen on every advert it serves, and harvests each advert's photo from the same page, so the market and its photographs arrive in one pass.

→ data/autotrader_deals.csv · data/autotrader_images.csv
7
retire_stale_autotrader.py

Moves out what the crawls can no longer vouch for — adverts Auto Trader no longer lists, measured at roughly a quarter of the corpus, and comparables whose price has not been re-read in weeks. Guarded: it will not act on a census that did not finish, and it refuses a prune far larger than the measured dead share.

→ data/autotrader_retired.csv · data/autotrader_comparables_retired.csv
8
score_autotrader.py

Scores every Auto Trader listing against its own model's market — deals that never touch BigMoto.

→ data/autotrader_scored.csv
9
distance.py

Annotates each Auto Trader listing with miles from the dealer's postcode. Distance is reported, never used to fit the market.

→ data/autotrader_scored.csv
10
enrich_details.py

Opens detail pages for the shortlist to pull the admin fee, owner count, service history and MOT.

→ data/details.csv
11
fetch_mot_history.py

Reads the registration enrich_details.py captured off each advert and fetches that bike's real MOT record from the DVSA MOT History API: expiry, defect and advisory counts, and the odometer at every test, normalised to miles. It replaced a hand-off to the gov.uk checker with the actual record, which is what makes the mileage check possible at all.

→ data/mot_history.csv
12
harvest_images.py

One photo URL per listing, read from the search pages rather than 1,215 detail pages. Resumable, and optional: without it every card draws a placeholder.

→ data/bigmoto_images.csv
13
harvest_autotrader_images.py

Backfill for Auto Trader photos the census missed, over the same search pages. It is no longer part of the routine scan — the census writes both outputs from one walk — but it is what fills a gap by hand.

→ data/autotrader_images.csv
14
classify_images.py

Measures each photo's pixels to spot the listings whose first image is a finance or delivery promo banner, then substitutes a clean gallery photo where one exists.

→ data/bigmoto_images.csv
15
build_dashboard.py

Builds dashboard.html: the page skeleton, the payload, and the server-rendered chrome. The front end itself lives in mh_ui.py (design tokens, the utility layer, the component styles, one icon set) and mh_dash_js.py (the client script), so the builder stays about building.

→ dashboard.html
16
build_info_page.py

Renders this page from the same CSVs, so the numbers here are regenerated rather than maintained by hand.

→ about.html

Verification is a stage, not an afterthought

A deal-finder that cries wolf is worthless, so these run after every producer and gate the run.

17
data_quality.py

Checks every crawl against invariants and a stored baseline. A parser that quietly stops matching fails the run instead of shrinking the corpus.

→ exit 1 on violation
18
check_readme.py

Owns every figure the README publishes and generates its summary tables from the data, so prose cannot drift from the CSVs.

→ README.md
19
snapshot_deals.py

Diffs today's scored data against yesterday's and reports new deals, price drops, rises and withdrawals.

→ data/alerts.md
20
banding.py

The one place a deal band is named, ranked and styled, so the page, the dashboard and the alerts cannot disagree about what a deal is.

→ shared by all

Shared browser layer: pw_client.py — one Playwright client that waits out the JS challenge on the same page, rotates browser contexts, and relaunches the whole browser process when a context goes stale.

04 — Methodology

How an "expected price" is earned

Comparables are every Auto Trader listing for the same make and model, matched by compact-substring (so R 1250 GS pairs with R 1250 GS Adventure, but V-Strom 650 never pairs with V-Strom 1050). The scorer then walks down a chain of methods and stops at the first one that is statistically defensible. 69,511 comparables back the BigMoto scores; the Auto Trader tab uses 748,422 more.

Deal distribution across all 1208 BigMoto bikes, where deal % = (expected − asking) / expected. 44 bikes have no score at all: the scorer refused a valuation rather than guess one.

  • 54 10%+ under market (4%)
  • 36 6–10% under market (3%)
  • 594 within ±6% of market (49%)
  • 371 more than 6% over market (31%)
  • 109 cheap, but inside the model's own error bar (9%)
  • 44 no market data — not scored (4%)

Fit chain — measured, not assumed

The scorer values a bike from comparables of its own registration year first, widening to ±1, ±2 then ±4 years only when the own-year pool is too thin. That order was chosen by benchmark, and the honest reading of the benchmark is narrower than it first looks: a same-year fit scores 5.7% median error, the ±1-year chain it replaced scores 5.8%, and the shipped own-year-first estimator scores 5.9% — a tie inside the per-fold spread, not a win. What the numbers do support is that the year restriction carries the accuracy: ±3-year medians degrade to 9% and a year-blind median to 12.3%. (tools/benchmark_valuation.py re-runs it.) If no pool qualifies the scorer returns no data instead of a number — 44 bikes in this run are in that bucket, shown as such rather than quietly dropped.

Self-ad exclusion

BigMoto advertises on Auto Trader too. Any comparable matching the bike's own price, year and mileage is dropped, so a listing can never be compared against itself.

Cheap is not automatically a deal

Every score carries an 80% interval — a range the market level for that model plausibly sits in, given how many comparable listings went into it and how tightly they agree. It is not the spread of individual asking prices, which is much wider; it is how well the model knows the average. A bike is only called a deal when its price falls below the bottom of that range, so 109 bikes here are cheap but inside the model's own margin of error — labelled ? rather than sold as bargains. Where a pool is too thin to narrow the range at all, the model is saying "I don't know", and a wide interval must not read as a big discount.

Which method produced which score

same-year724 bikes · 60%

OLS price↔mileage fit on comparables of the bike's own registration year

same-year-median249 bikes · 21%

Own-year median (used when the own-year pool is too small to fit a slope)

year1117 bikes · 10%

±1-year regression fit, used only when the own-year pool is too thin

year1-median39 bikes · 3%

±1-year median

none38 bikes · 3%

No usable market data for this model at all

year211 bikes · 1%

±2-year regression fit (wider, so less precise)

same-year-mileage-median7 bikes · 1%

—

era-gap6 bikes · 0%

Refused to score — comparables do not cover the bike's era

year2-median5 bikes · 0%

±2-year median

all-years4 bikes · 0%

Every comparable for the model, regardless of year — last resort

year1-mileage-median3 bikes · 0%

—

all-years-median2 bikes · 0%

—

year42 bikes · 0%

±4-year regression fit (wide, low confidence)

year2-mileage-median1 bikes · 0%

—

Guardrails built from real mistakes

1 · All-years medians overprice old bikes

A pool of comparables skews newer than the bike being scored, so a pool-wide median makes every older bike look like a bargain. Fixed with the era-gap bail-out and era-proximate windows.

2 · Regression extrapolation invents value

A mileage fit evaluated far outside the window that produced it can return fantasy prices. Fix: no extrapolation beyond ±5k miles of the window, and a hard clamp to the window's own min/max price.

3 · Thin windows produce confident nonsense

Below the minimum comparable count the scorer falls back to a window median and labels the result low-confidence rather than fitting a line to three points.

4 · Cheap is not the same as good

The dashboard flags low mileage relative to age (≤40% of typical = very low), so a genuinely fresh bike is distinguishable from a leggy one that is merely cheap.

05 — Output

The ten biggest gaps in the current run

Every row here is a bike priced below the bottom of the interval the model puts around its own market estimate — a gap its own uncertainty cannot explain. Every one of them cleared the model's own margin of error. Click a model to open the actual listing.

#BikeYearMilesAskExpectedDealConf.Compare
1 BMW R 1200 GS Adventure · 1170 cc · White 2018 17,854 £7,188£6,989 + £199 fee £9,999 +28.1% high Auto Trader · MOT
2 HARLEY-DAVIDSON SPORTSTER Cruiser · 883 cc · Silver 2020 1,109 £6,288£6,089 + £199 fee £8,634 +27.2% medium Auto Trader · MOT
3 BMW F 850 GS Adventure · 853 cc · Red 2021 13,299 £5,688£5,489 + £199 fee £7,515 +24.3% high Auto Trader · MOT
4 KTM 890 Duke Naked · 889 cc · Orange 2021 4,502 £4,688£4,489 + £199 fee £6,163 +23.9% high Auto Trader · MOT
5 ROYAL ENFIELD SUPER METEOR 650 Classic / Retro · 648 cc · Other 2025 5,805 £4,188£3,989 + £199 fee £5,339 +21.6% high Auto Trader · MOT
6 TRIUMPH STREET TRIPLE 765 Naked · 765 cc · Silver 2024 460 £7,788£7,589 + £199 fee £9,899 +21.3% high Auto Trader · MOT
7 YAMAHA R1 Sport · 998 cc · Red 2000 21,644 £3,188£2,989 + £199 fee £3,979 +19.9% medium Auto Trader · MOT
8 INDIAN FTR Naked · 1203 cc · Black 2021 5,774 £6,688£6,489 + £199 fee £8,190 +18.3% high Auto Trader · MOT
9 HONDA CBR1000RR FIREBLADE Sport · 1000 cc · Black 2019 2,707 £9,488£9,289 + £199 fee £11,554 +17.9% medium Auto Trader · MOT
10 KTM 125 DUKE Naked · 125 cc · Blue 2013 18,313 £2,488£2,289 + £199 fee £3,017 +17.5% medium Auto Trader · MOT

573 of 1208 scored bikes carry high confidence, and 90 sit at 6% or more below their modelled market value. Ask is the drive-away total — the advertised price plus BigMoto's admin fee, which 1213 of the 1213 enriched detail pages carry — and the deal % is measured against that same total, so the two reconcile.

06 — Second horizon

The whole market, not just one dealer

The same scoring engine runs across Auto Trader's own inventory, which catches under-priced bikes anywhere in the country — including ones BigMoto will never list.

15,048listings scored
946≥10% under market
1,641≥6% under market
748,422comparables matched

Coverage by category

  • Naked6,592
  • Adventure6,548
  • Sports Tourer1,908

Beating the 100-page cap

Auto Trader serves at most ~2,000 listings per query, so the crawl splits each body type into year bands that each stay under the ceiling. The only remaining gap is the busiest 2026 adventure band, which loses roughly the last 100 pages' worth of ads.

Dealer and private, matched like for like

This run scored 14,116 trade and 932 private ads. Private asking prices sit well below dealer ones for the same model, so each listing is valued against a pool of its own seller type wherever that pool is large enough — otherwise the comparison measures the seller, not the bike. The dashboard shows which is which on every row.

07 — Deliverable

The dashboard

dashboard.html is a single self-contained file — no server, no CDN, no build step. Open it with file:// and it works offline, with the data embedded as JSON.

🗂️

Two sources, one shell

BigMoto stock scored against the market, and Auto Trader deals scored against their own model — 1208 + 15,048 rows. The source switch sits with the scan's numbers, so it never competes with the motorcycles for the top of the page.

🧭

Four destinations

Deals to browse, Shortlist to keep, Compare to decide and Market to understand the scan. The card view is the default; the dense sortable table is one click away and remembers the choice.

🎚️

Filters in two levels

The seven questions asked on every visit — make, model, category, max price, min deal %, mileage and year — stay in the open. Confidence, evidence, first seen and the four toggles fold behind More filters. On anything narrower than a laptop the whole panel becomes a sheet. Filter state lives in the URL, so a view can be bookmarked or sent to someone.

🏷️

Every category, not two of them

The crawler tags twelve categories — naked, adventure, sport, classic, scooter, cruiser, sports tourer, tourer, motocross, supermoto, enduro and three-wheeler — and the page reads that list out of the data. The filter carries a count per category, the summary strip names the two largest, and the Market view has a category board showing each one's median discount and best recorded deal. Add a category to the crawler and it appears here by itself.

💬

Search that reads a sentence

Type Triumph under £8,000 or adventure under 30,000 miles and the price and mileage clauses are lifted onto the real filters and shown as chips you can remove. Nothing is invented: a clause can only ever narrow the list, and free text still behaves exactly as it always did.

⭐

A shortlist and a comparison

Star any bike to keep it, then filter to the shortlist. Pick up to four — explicitly, or by falling back to the shortlist — and Compare puts them on one shared price axis and then in a table that marks the row that differs and the value that wins.

🔎

The deal, explained

Every bike has its own page: listed at, estimated market value, and the difference — underneath, the comparable count that produced it, the model's 80% range drawn as a distribution, the other examples of that make and model in the scan, and the price history if the snapshots can prove one.

📉

Price history

Rows that moved since an earlier run are flagged with the change, using the dated snapshots the pipeline already writes — and saying nothing at all where the snapshots cannot prove a claim.

📤

Export and print

Download whatever you have filtered to as CSV, or print the current view as a clean paper shortlist with no chrome.

🔧

MOT, from the record

The DVSA MOT History API now supplies each bike's real test record — expiry, defect and advisory counts, and the odometer at every test — shown as a badge on the row. The highest recorded odometer is compared with the advertised mileage, but it is evidence, not a verdict: an odometer can be replaced and a private plate moves between bikes, so a contradiction is flagged for a human and never asserted as a clocking claim.

The dashboard: a search bar and market summary above a grid of motorcycle cards, each showing its photograph, deal verdict, asking price, saving against the modelled market and its confidence. The same dashboard on a phone: one card per row with a large photograph, sticky search, and the filters behind a single button.
The dashboard — the motorcycles lead, the numbers support them, and every verdict can be traced back to the evidence behind it. One card per bike on a phone, up to five across on a wide screen, and nothing scrolls sideways at any width.
A single motorcycle's detail page: its photograph, the deal verdict, listed price against estimated market value and the difference, then the modelled market range drawn as a distribution with the asking price marked.
One bike's own page — the three figures that decide it, the comparable count they rest on, and the model's 80% range drawn so the size of the claim is visible rather than implied.
08 — Running it

Quick start

Python 3.13 with requests, bs4, lxml and playwright (Chromium installed). Node is only needed to syntax-check the dashboard's JavaScript.

# from the project root — scrapes, scores, enriches, builds
python -u run_pipeline.py

# or step by step (each stage is independent)
python -u scrape_bigmoto_pw.py        # -> data/bigmoto.csv
python -u fetch_autotrader.py         # -> data/autotrader.csv
python -u deep_fetch.py               # era-ranged comps (idempotent)
python -u score_deals.py              # -> data/scored.csv
python -u scrape_autotrader_deals.py  # -> data/autotrader_deals.csv
python -u score_autotrader.py         # -> data/autotrader_scored.csv
python -u enrich_details.py           # -> data/details.csv
python -u build_dashboard.py          # -> dashboard.html
# no scraping needed if the CSVs are already populated
python -u score_deals.py
python -u score_autotrader.py
python -u enrich_details.py
python -u build_dashboard.py
python -u build_info_page.py          # -> about.html (this page)

# serve the dashboard locally
python -m http.server 8000            # http://127.0.0.1:8000/dashboard.html
# dashboard integrity + the JS it embeds
python validate_dashboard.py
node --check _check.js                # must print: JS syntax OK

# is the scoring healthy? want mostly window1*, few medians
Import-Csv data/scored.csv | Group-Object method
09 — Orientation

Where everything lives

23,239 lines of pipeline Python (plus 23,876 lines of probes/, tests/ and tools/ scripts — the experiments that reverse-engineered the two sites, the tests that keep them honest, and the audits and one-off repairs run by hand).

pipeline pw_client.py shared Playwright client, Cloudflare-challenge handling scrape_bigmoto_pw.py BigMoto category crawl -> bigmoto.csv fetch_autotrader.py shallow Auto Trader comps; also holds make/model parsing deep_fetch.py era-ranged comps via year-from / year-to score_deals.py the valuation model -> scored.csv scrape_autotrader_deals.py year-banded crawl of the whole market, dealers and private sellers score_autotrader.py same model, applied to Auto Trader inventory enrich_details.py admin fee / owners / FSH / MOT for shortlisted bikes build_dashboard.py -> dashboard.html build_info_page.py -> about.html (this page) data bigmoto.csv 1208 bikes with price, year, mileage, cc autotrader.csv comparable listings for scoring scored.csv 1208 bikes scored + method + confidence autotrader_deals.csv full Auto Trader crawl, dealers and private sellers autotrader_scored.csv 15,048 listings scored details.csv enriched detail-page facts output dashboard.html the site root; built on the host, not committed about.html this project page; built on the host, not committed HANDOVER.md scraping knowledge earned the hard way reference probes/, tests/ the experiments behind the scraping rules, and the tests tools/ audits, benchmarks and one-off repairs, run by hand html/ saved DOM snapshots used to write the parsers
10 — Field notes

Scraping two hostile sites, what actually worked

Both sources are protected and JavaScript-rendered. These are the findings that survived contact, kept in the repo so nobody has to rediscover them.

BigMoto · ClickDealer

Plain requests gets a 403 challenge; headless Playwright Chromium clears it reliably on a fresh browser process. Re-navigating during a challenge resets it — wait on the same page instead. A page titled "Just a moment…" is a challenge; an empty page still says "Bigmoto".

Auto Trader · Vite SPA

Legacy REST endpoints 404, the GraphQL endpoint is Cloudflare-protected. The winning approach is parsing the rendered DOM by data-testid. A postcode parameter is mandatory or the page only shows a prompt, and make must accompany keywords or the search returns unrelated bikes.

The stale-DOM trap

If a navigation fails, the previous query's cards stay in the DOM and get parsed as results. Guard: assert the URL-encoded keyword appears in the final URL before trusting the cards.

Facets are case-sensitive

body-type=Adventure returns thousands of bikes; adventure returns zero. The full facet list is saved in html/facets_resp_0.json.

11 — Next

Roadmap

Shipped

  • ✓Dual-source ingestion with challenge handling
  • ✓Mileage-and-era-aware valuation with confidence tiers
  • ✓Self-ad exclusion and false-positive guardrails
  • ✓Uncertainty intervals and abstention — a cheap bike is not automatically a deal
  • ✓Admin-fee-aware pricing, shown as a drive-away total
  • ✓Distance to the dealer, reported but never used to filter the market
  • ✓Self-contained dashboard with detail enrichment
  • ✓Scheduled pipeline with price-history alerts
  • ✓Data-quality gates — every crawl checked against a stored baseline
  • ✓One shared band rule, so the page, dashboard and alerts cannot disagree
  • ✓Published figures checked or generated rather than maintained by hand
  • ✓Real MOT history per bike, through the DVSA MOT History API — the credentials were approved, retiring the gov.uk hand-off

Next

  • Calibrate against real sold prices — the tooling and an external MCN band check exist, but every automated source for achieved prices is now shut, so sold_prices.csv has to be filled in by hand
  • Publish the dashboard to a host so it is one click away

Closed

Investigated properly, no route exists. Recorded here so nobody spends another afternoon on them.

  • ✕eBay UK sold-price signal — eBay walls its sold/completed pages behind a sign-in and captcha challenge: 15 attempts across four strategies returned zero rows while ordinary eBay pages rendered fine from the same browser. The only API with sold data is the restricted Marketplace Insights one, and automating a signed-in session past the wall would breach eBay's terms and risk the account