diff --git a/docs/V2-PLAN.md b/docs/V2-PLAN.md new file mode 100644 index 0000000..553f4a2 --- /dev/null +++ b/docs/V2-PLAN.md @@ -0,0 +1,134 @@ +# Cup26 "v2" — the data leap (internally v7) + +Drafted June 12, 2026 (matchday 2) from a six-source reconnaissance, probed +from both the residential IP and the production VPS. The visual overhaul is +handled separately in Claude Design; this plan reserves explicit handoff +slots for it and keeps all new features design-ready (data + logic first, +plain markup that a design pass can reskin). + +## Reconnaissance results (verified June 12) + +| Source | Local | VPS | Verdict | +|---|---|---|---| +| **FotMob** `/api/data/*` | ✅ 200 | ✅ 200 — **no signed header anymore** | **INGEST NOW** | +| **FIFA official** `api.fifa.com/api/v3` | ✅ | ✅ — no auth, no block | **INGEST NOW** | +| **ESPN deep fields** (already cached!) + propBets | ✅ | ✅ | **INGEST NOW** | +| **Open-Meteo** weather+elevation | ✅ | ✅ — keyless | **INGEST NOW** | +| SofaScore | browser-only (403 to scripts) | ❌ hard IP block | defer (FotMob supersedes) | +| FBref | ❌ Cloudflare wall | ❌ | skip (no 2026 xG yet anyway) | + +What the three rich sources carry (all verified against real June 2026 matches): + +- **FotMob `matchDetails`** (~244KB/match): per-shot xG + xGOT with pitch + x/y, team xG split (open play/set play), **player ratings** (e.g. 7.5) for + all 52 players, player-of-the-match, momentum curve, attacking zones, match + facts/insights, lineups with formations. Works for live matches too. + Key ids: league 77; matches by `?date=YYYYMMDD` (e.g. 4667751 = MEX–RSA). +- **FIFA v3**: IdCompetition=17, IdSeason=285023. Official **confirmed + lineups with stable player IDs + headshot URLs**, typed event timeline with + pitch X/Y, official attendance (80,824 at the opener), team tactics, + officials, standings with QualificationStatus, live match clock. +- **ESPN, unused but already stored**: per-player match stat lines (14 stats: + shots, on-target, saves, fouls, cards, goals conceded…), formation + + formationPlace (pitch positions), team leaders, ~109-entry commentary + ticker, broadcasters, match recap articles. Plus one cheap new fetch: + **propBets — DraftKings any/first-goalscorer odds, ~962 props per match**. +- **Open-Meteo**: one batched call covers all 16 venues; returns elevation + (Azteca 2,245m!), hourly temp/precip/wind in venue-local time; archive API + for played matches. + +## Phase D — capture first (urgent: live data is unrecoverable) + +FotMob is unofficial — it could re-lock any day. Capture beats features. + +- `server/src/ingest/fotmob.ts`: daily `matches?date=` sweep → id map + (fixture ↔ matchId via team-pair+date, the ESPN pattern); `matchDetails` + per finished match (once, re-fetch at FT+2h for settled ratings) and per + live match every ~90s. Store raw-ish normalized JSON in new `match_fotmob` + table (whole payload is fine — ~25MB for all 104 matches). Respectful + cadence: ≥2s between calls, circuit breaker, health-tracked like ESPN. +- `server/src/ingest/fifa.ts`: calendar sweep (id map), `live/football/...` + per live match (clock/lineups), `timelines` at FT. Tables: `match_fifa` + (lineups/officials/attendance/tactics), reuse health. +- `server/src/ingest/espnExtra`: widen `espnNormalize` to keep player stat + lines, formation+formationPlace, leaders, commentary, attendance when + present (normalizer v3 + the existing boot backfill re-stores everything). + New `scorer_props` table fed from propBets on the odds cadence (3h) — + line history is unrecoverable, same urgency as 1X2 odds was in v3. +- `server/src/ingest/weather.ts`: forecast at enrich cadence for fixtures + <48h away; archive once at FT. `match_weather` table. Venue lat/lon/alt + as a committed static (`scripts/buildVenues.ts` → `public/data/venues.json`). +- Storage: everything also lands in the nightly storage-box snapshot. + +## Phase M — model upgrades (CV-gated, same discipline as v4/v5) + +Every experiment runs through `scripts/cvSearch.ts` (5-fold rolling CV, +pre-registered ship bar ≥0.0005 mean-fold RPS) — ship only what validates. + +- **M1 xG-informed Team-DC**: backfill FotMob `matchDetails` for recent + *international* matches (last ~4 years; local script into the lake, weeks + of polite fetching — start immediately, run in background). Experiment: + fit attack/defence on a goals/xG blend (xG is less noisy than goals). + This is the single largest model upside available. +- **M2 lineup-aware kickoff refresh**: when FIFA confirmed lineups land + (~75min before kickoff), adjust the match forecast for missing key players + (availability share from minutes×ratings, NOT market values). Snapshot for + the scoreboard stays frozen at T-15min as today — honesty unchanged. +- **M3 in-play v2**: blend the live model with score-state rates from the + StatsBomb lake, stratified by pre-match strength gap (fixes the v3 + selection-bias concern), validated on lake matches; FotMob momentum as a + display layer, not a model input. +- **M4 scorer model (new headline)**: per-player goal-rate model (minutes × + team λ share from historical scorer data + tournament form) → predicted + Golden Boot table, benchmarked against DraftKings scorer props on a + **Scorers: model vs market** scoreboard — the existing honest-benchmark + franchise, extended to players. + +## Phase F — features (each lands design-ready; final skin = design handoff) + +Tier 1 (data already flowing after D): +1. **Match center, full**: xG race chart (reuse the Story page viz), shotmap + (per-shot xG, x/y), attack-momentum strip, lineup pitch with formations + + headshots (FIFA), per-player stat lines + ratings table, POTM card, + officials + attendance + broadcasters, weather/altitude badge, + commentary ticker. This page becomes the product's core. +2. **Player pages + tournament leaderboards**: G+A (assists verified + structured in ESPN participants), ratings-based Best XI, saves/cards + boards from ESPN stat lines; FIFA headshots; links everywhere. +3. **Luck-adjusted table**: group standings re-computed on xG ("the table + if chances decided games") next to the real one — very shareable. +4. **Golden Boot: model vs market** (M4 + scorer_props). +5. **Weather strip** on Live/match pages; altitude notes (Azteca 2,245m). + +Tier 2 (after M experiments settle): +6. xG-aware form chips and team pages (xG trend last 5). +7. In-play v2 probability + momentum fusion on live pages. +8. Lineup-drop push notification ("XI is in — model moved to 54/24/22"). + +## Design handoff slots (Claude Design → me for implementation) + +- Match center layout (shotmap + xG race + momentum + lineup pitch + ticker) +- Player page / player card / Best XI grid +- Leaderboards (boot race, G+A, ratings) and the luck-adjusted table +- Weather/altitude badges and the match-info panel +- Any reskin of existing pages — all current components are plain Tailwind + with tokenized colors (`--app-*`), so a token+component redesign drops in. + +## Order & verification + +D (capture-first, deploy same day) → M1 backfill kicked off in background → +F1–F5 in batches → M experiments as data accrues → Tier 2. Per batch: tests + +exit-code build gate + preview matrix (EN/DE × mobile/desktop × themes) + +prod verification against the live tournament. Honest framing everywhere: +xG and ratings are third-party estimates; scorer odds remain benchmark-only. + +## Risks + +- FotMob is unofficial: could re-add signed headers or block datacenter IPs + any day (SofaScore precedent). Mitigations: capture-first, raw payload + storage, circuit breaker + health, app never depends on it being live. +- FIFA API is unauthenticated but official-internal; same mitigations. +- Identity mapping (fixture ↔ fotmobId ↔ fifaId ↔ espnId) is the foundation — + one `source_map` per provider, team-pair+date matching, tested. +- Player identity across sources (names vs FIFA IdPlayer): normalize via + name+team with a manual alias map fallback; keep per-source ids.