v2 plan: the data leap — recon-verified sources, model and feature phases
Six-source reconnaissance from both vantage points found the v3-era blockers gone: FotMob serves full xG/shotmaps/player-ratings from the VPS with no signed header, and FIFA's official v3 API (lineups with player IDs + headshots, XY timelines, attendance) is open too. The plan: capture-first ingestion (FotMob/FIFA/ESPN-deep/weather), four CV-gated model experiments (xG-informed Team-DC, lineup-aware refresh, bias-corrected in-play, a scorer model benchmarked against DraftKings props), and a feature tier built on it — with explicit design-handoff slots for the visual overhaul arriving from Claude Design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+134
@@ -0,0 +1,134 @@
|
||||
# Cup26 "v2" — the data leap (internally v7)
|
||||
|
||||
Drafted June 12, 2026 (matchday 2) from a six-source reconnaissance, probed
|
||||
from both the residential IP and the production VPS. The visual overhaul is
|
||||
handled separately in Claude Design; this plan reserves explicit handoff
|
||||
slots for it and keeps all new features design-ready (data + logic first,
|
||||
plain markup that a design pass can reskin).
|
||||
|
||||
## Reconnaissance results (verified June 12)
|
||||
|
||||
| Source | Local | VPS | Verdict |
|
||||
|---|---|---|---|
|
||||
| **FotMob** `/api/data/*` | ✅ 200 | ✅ 200 — **no signed header anymore** | **INGEST NOW** |
|
||||
| **FIFA official** `api.fifa.com/api/v3` | ✅ | ✅ — no auth, no block | **INGEST NOW** |
|
||||
| **ESPN deep fields** (already cached!) + propBets | ✅ | ✅ | **INGEST NOW** |
|
||||
| **Open-Meteo** weather+elevation | ✅ | ✅ — keyless | **INGEST NOW** |
|
||||
| SofaScore | browser-only (403 to scripts) | ❌ hard IP block | defer (FotMob supersedes) |
|
||||
| FBref | ❌ Cloudflare wall | ❌ | skip (no 2026 xG yet anyway) |
|
||||
|
||||
What the three rich sources carry (all verified against real June 2026 matches):
|
||||
|
||||
- **FotMob `matchDetails`** (~244KB/match): per-shot xG + xGOT with pitch
|
||||
x/y, team xG split (open play/set play), **player ratings** (e.g. 7.5) for
|
||||
all 52 players, player-of-the-match, momentum curve, attacking zones, match
|
||||
facts/insights, lineups with formations. Works for live matches too.
|
||||
Key ids: league 77; matches by `?date=YYYYMMDD` (e.g. 4667751 = MEX–RSA).
|
||||
- **FIFA v3**: IdCompetition=17, IdSeason=285023. Official **confirmed
|
||||
lineups with stable player IDs + headshot URLs**, typed event timeline with
|
||||
pitch X/Y, official attendance (80,824 at the opener), team tactics,
|
||||
officials, standings with QualificationStatus, live match clock.
|
||||
- **ESPN, unused but already stored**: per-player match stat lines (14 stats:
|
||||
shots, on-target, saves, fouls, cards, goals conceded…), formation +
|
||||
formationPlace (pitch positions), team leaders, ~109-entry commentary
|
||||
ticker, broadcasters, match recap articles. Plus one cheap new fetch:
|
||||
**propBets — DraftKings any/first-goalscorer odds, ~962 props per match**.
|
||||
- **Open-Meteo**: one batched call covers all 16 venues; returns elevation
|
||||
(Azteca 2,245m!), hourly temp/precip/wind in venue-local time; archive API
|
||||
for played matches.
|
||||
|
||||
## Phase D — capture first (urgent: live data is unrecoverable)
|
||||
|
||||
FotMob is unofficial — it could re-lock any day. Capture beats features.
|
||||
|
||||
- `server/src/ingest/fotmob.ts`: daily `matches?date=` sweep → id map
|
||||
(fixture ↔ matchId via team-pair+date, the ESPN pattern); `matchDetails`
|
||||
per finished match (once, re-fetch at FT+2h for settled ratings) and per
|
||||
live match every ~90s. Store raw-ish normalized JSON in new `match_fotmob`
|
||||
table (whole payload is fine — ~25MB for all 104 matches). Respectful
|
||||
cadence: ≥2s between calls, circuit breaker, health-tracked like ESPN.
|
||||
- `server/src/ingest/fifa.ts`: calendar sweep (id map), `live/football/...`
|
||||
per live match (clock/lineups), `timelines` at FT. Tables: `match_fifa`
|
||||
(lineups/officials/attendance/tactics), reuse health.
|
||||
- `server/src/ingest/espnExtra`: widen `espnNormalize` to keep player stat
|
||||
lines, formation+formationPlace, leaders, commentary, attendance when
|
||||
present (normalizer v3 + the existing boot backfill re-stores everything).
|
||||
New `scorer_props` table fed from propBets on the odds cadence (3h) —
|
||||
line history is unrecoverable, same urgency as 1X2 odds was in v3.
|
||||
- `server/src/ingest/weather.ts`: forecast at enrich cadence for fixtures
|
||||
<48h away; archive once at FT. `match_weather` table. Venue lat/lon/alt
|
||||
as a committed static (`scripts/buildVenues.ts` → `public/data/venues.json`).
|
||||
- Storage: everything also lands in the nightly storage-box snapshot.
|
||||
|
||||
## Phase M — model upgrades (CV-gated, same discipline as v4/v5)
|
||||
|
||||
Every experiment runs through `scripts/cvSearch.ts` (5-fold rolling CV,
|
||||
pre-registered ship bar ≥0.0005 mean-fold RPS) — ship only what validates.
|
||||
|
||||
- **M1 xG-informed Team-DC**: backfill FotMob `matchDetails` for recent
|
||||
*international* matches (last ~4 years; local script into the lake, weeks
|
||||
of polite fetching — start immediately, run in background). Experiment:
|
||||
fit attack/defence on a goals/xG blend (xG is less noisy than goals).
|
||||
This is the single largest model upside available.
|
||||
- **M2 lineup-aware kickoff refresh**: when FIFA confirmed lineups land
|
||||
(~75min before kickoff), adjust the match forecast for missing key players
|
||||
(availability share from minutes×ratings, NOT market values). Snapshot for
|
||||
the scoreboard stays frozen at T-15min as today — honesty unchanged.
|
||||
- **M3 in-play v2**: blend the live model with score-state rates from the
|
||||
StatsBomb lake, stratified by pre-match strength gap (fixes the v3
|
||||
selection-bias concern), validated on lake matches; FotMob momentum as a
|
||||
display layer, not a model input.
|
||||
- **M4 scorer model (new headline)**: per-player goal-rate model (minutes ×
|
||||
team λ share from historical scorer data + tournament form) → predicted
|
||||
Golden Boot table, benchmarked against DraftKings scorer props on a
|
||||
**Scorers: model vs market** scoreboard — the existing honest-benchmark
|
||||
franchise, extended to players.
|
||||
|
||||
## Phase F — features (each lands design-ready; final skin = design handoff)
|
||||
|
||||
Tier 1 (data already flowing after D):
|
||||
1. **Match center, full**: xG race chart (reuse the Story page viz), shotmap
|
||||
(per-shot xG, x/y), attack-momentum strip, lineup pitch with formations +
|
||||
headshots (FIFA), per-player stat lines + ratings table, POTM card,
|
||||
officials + attendance + broadcasters, weather/altitude badge,
|
||||
commentary ticker. This page becomes the product's core.
|
||||
2. **Player pages + tournament leaderboards**: G+A (assists verified
|
||||
structured in ESPN participants), ratings-based Best XI, saves/cards
|
||||
boards from ESPN stat lines; FIFA headshots; links everywhere.
|
||||
3. **Luck-adjusted table**: group standings re-computed on xG ("the table
|
||||
if chances decided games") next to the real one — very shareable.
|
||||
4. **Golden Boot: model vs market** (M4 + scorer_props).
|
||||
5. **Weather strip** on Live/match pages; altitude notes (Azteca 2,245m).
|
||||
|
||||
Tier 2 (after M experiments settle):
|
||||
6. xG-aware form chips and team pages (xG trend last 5).
|
||||
7. In-play v2 probability + momentum fusion on live pages.
|
||||
8. Lineup-drop push notification ("XI is in — model moved to 54/24/22").
|
||||
|
||||
## Design handoff slots (Claude Design → me for implementation)
|
||||
|
||||
- Match center layout (shotmap + xG race + momentum + lineup pitch + ticker)
|
||||
- Player page / player card / Best XI grid
|
||||
- Leaderboards (boot race, G+A, ratings) and the luck-adjusted table
|
||||
- Weather/altitude badges and the match-info panel
|
||||
- Any reskin of existing pages — all current components are plain Tailwind
|
||||
with tokenized colors (`--app-*`), so a token+component redesign drops in.
|
||||
|
||||
## Order & verification
|
||||
|
||||
D (capture-first, deploy same day) → M1 backfill kicked off in background →
|
||||
F1–F5 in batches → M experiments as data accrues → Tier 2. Per batch: tests +
|
||||
exit-code build gate + preview matrix (EN/DE × mobile/desktop × themes) +
|
||||
prod verification against the live tournament. Honest framing everywhere:
|
||||
xG and ratings are third-party estimates; scorer odds remain benchmark-only.
|
||||
|
||||
## Risks
|
||||
|
||||
- FotMob is unofficial: could re-add signed headers or block datacenter IPs
|
||||
any day (SofaScore precedent). Mitigations: capture-first, raw payload
|
||||
storage, circuit breaker + health, app never depends on it being live.
|
||||
- FIFA API is unauthenticated but official-internal; same mitigations.
|
||||
- Identity mapping (fixture ↔ fotmobId ↔ fifaId ↔ espnId) is the foundation —
|
||||
one `source_map` per provider, team-pair+date matching, tested.
|
||||
- Player identity across sources (names vs FIFA IdPlayer): normalize via
|
||||
name+team with a manual alias map fallback; keep per-source ids.
|
||||
Reference in New Issue
Block a user