Case study · MLB projection system
My model kept getting one team wrong in the same direction, every single year. Chasing that error is how I found the thing it was blind to — and how the model eventually told me the fix had worked.
For most of this project my player-based forecast lost to an embarrassingly simple baseline: last season's run differential, regressed toward the mean. I had built player projections, aging curves, playing-time models, a Monte Carlo season simulator. A one-line calculation beat all of it.
What follows is how I found out why. I am writing it up because the reasoning matters more than the result — and because two of the steps were wrong before they were right.
Is the error random, or is it the same teams every year?
A model that is merely noisy misses different teams each season. A model that is missing something misses the same teams in the same direction. That is a testable difference, so I tested it: take the residual (actual wins minus projected wins) and check whether it persists.
| Residual correlation | r |
|---|---|
| 2023 → 2024 | +0.338 |
| 2024 → 2025 | +0.372 |
They do not sit near zero. About a third of what my model missed was a stable property of the organisation, not luck. And when I looked at who:
| Team | Mean wins under-projected |
|---|---|
| Milwaukee Brewers | +10.2 |
| Baltimore Orioles | +7.1 |
| Philadelphia Phillies | +6.2 |
Ten wins is not a rounding error. Something real was happening in Milwaukee that my model could not see.
What does the baseline know that I don't?
Here is the question that unlocked it. I had been treating run differential as a rival forecast to beat. It is not. It is a measurement — and it measures things my model never looked at.
My run prevention came entirely from pitcher FIP. But FIP is defined to strip out defense; it counts only strikeouts, walks and home runs, the outcomes a pitcher controls without his fielders. So my model was structurally blind to team defense, baserunning and catcher framing.
Run differential contains all three by construction. I had been competing against a measure that included them while leaving them out.
That is a hypothesis, and it makes a prediction: if the residual is unmodelled defense, the two should correlate.
| Test | r | n |
|---|---|---|
| Residual vs. unmodelled defense + baserunning | +0.464 | 60 |
Build it — and check whether the patch becomes unnecessary.
Before finding the cause I had papered over the problem with a hierarchical correction: a per-team adjustment fit on prior residuals. It helped, which is exactly why it was dangerous. A correction that works can hide the thing it is correcting for indefinitely.
So the real test was not just "does adding defense improve accuracy." It was sharper, and it could fail:
If the residual really was defense, then once defense is modelled explicitly, the correction term should stop helping — and start hurting.
| Model | MAE | RMSE |
|---|---|---|
| Players only (no defense, park, durability) | 7.925 | 9.914 |
| Regressed run differential — the baseline | 7.752 | 10.023 |
| All components integrated | 7.422 | 9.346 |
| … plus the hierarchical correction | 7.743 | — |
Adding defense, baserunning, park and durability moved the forecast from 2.2% worse than the baseline to 4.2% better. And the correction term I had been leaning on went from helpful to actively harmful — 7.743 against 7.422. That residual was defense. Once the real thing was in the model, the patch became redundant noise.
That last line is the part I would want a reader to notice. I did not just improve a number. I made a prediction about what would happen to a different part of the system if my explanation was right, and it happened.
The +4.2% above rests on three seasons. Extending the backtest to five — 150 team-seasons — drops it to +1.8% MAE: two seasons won, two lost, one tied. 2023 was the model's best year by a wide margin and was carrying the average. The RMSE gap survives better, +5.3%, which says large misses are still avoided even where average error is close.
I am leaving both numbers on the page. A three-season edge that halves on five seasons is exactly the kind of result that gets quietly re-scoped, and the diagnosis above still holds — the defense finding is why the model beats the baseline at all. But the honest headline is the smaller number.
Two of these steps failed first.
The roster bug. Partway through I realised the model was projecting each season using the previous season's rosters — so it could not see free agency at all, which is most of the point of a player-based model. Fixing it produced a jump to +6.00% and I nearly wrote it up.
Then I checked which roster the API had actually given me. rosterType=fullSeason includes every player who appeared at any point in the year — the Yankees list 54 players for 2024 against 26 on opening day. My model was being credited with deadline acquisitions the team had not made yet. With true opening-day rosters, that +6.00% became −2.62%.
The calibration bug. I fit the conversion from projected wOBA to runs and got a correlation of .84 — suspiciously good. It was: I had regressed season-N actual runs against season-(N+1) projections. Off by one year. Applying that "improvement" made the forecast substantially worse before I found it.
Every one of them made a result look better than it was, and every one was found by asking the same question: could the model have seen its own answer? The full list, with the before and after numbers, is in the repository. I would rather a reader see those than a clean set of wins.
Every number above comes from one command.
git clone <repo> && cd mlb-projection-system python3 src/statsapi.py --start 2019-03-15 --end 2025-10-05 # build the database python3 scripts/verify.py --network # 27 end-to-end checks python3 src/unified.py --compare # the headline table
The verification harness runs before and after every change and exits non-zero on a regression, including a dedicated check that the model still beats the baseline. The no-leakage rule is enforced by an automated test, not by discipline.