Short answer
We took a validated model — 3.87% error on unseen months — and split it in two to get finer detail. One half came back at 832% error. The other at 32%. Splitting 14 markets into two groups starved each model of what it needed. The pooled model was the only trustworthy one, and only the holdout showed the difference.
The first model was good. Three months hidden, 3.87% error. Then we tried to make it more useful, and broke it.
Log scale — the real gap is far wider than it looks
All three looked fine in training. Sensible charts, reasonable fits, curves the right shape. Only hiding data exposed the difference — and without it we would have shipped two confident, wrong models.
What we tried to do
The client split their markets into two groups for planning. Two models would give each group its own curves — more detail, better decisions. Obviously a good idea.
Same data, same setup, same validation. The only change was that each model now saw about half the markets.
Why did splitting break it?
- Pooling
- Building one model across many markets so each market’s estimate borrows strength from the others. Markets with thin data get pulled toward the pattern the whole set supports, instead of chasing their own noise.
Fourteen markets gave us fourteen partly independent views of the same media plan. Cut it to seven and three things happen at once.
- 01Far fewer data points, so every estimate gets shakier.
- 02Channels that were separable across 14 markets move together within 7. The model cannot tell them apart.
- 03The pooling that was steadying the noisy markets is gone.
What we did instead
We reported the pooled model as the real result and told the client their data could not support the split. A worse deliverable than the one we planned. Also the honest one.
“More detail is not free. You pay for it in what the data can actually tell apart.”
What changed after
- Every version of every model gets tested on hidden data before anyone sees it.
- When a client asks for more detail, we check whether the data can support it before agreeing to the scope.
- Failures get reported with the numbers, not quietly replaced with the version that worked.
How can error be over 100%?
Percentage error compares the miss to the real number. Predict several times too high and you pass 100%. At 832%, the predictions mean nothing.
Was the pooled model still trustworthy?
Yes. The failure was in the split, and the pooled model was validated on its own hidden window.
Could a different technique have saved the segments?
A model where segments share structure instead of standing alone can work. Fitting separate models on chunks that small cannot.
How often do models fail like this?
Often enough that anyone claiming it never happens is either new or not checking. Judge people on what they do when it does.
Find your wasted third
Want this run on your own numbers?
The two-week audit shows you where each of your channels sits on its own curve — what to cut, what to grow, and how much budget is sitting in the wrong place.
Book the two-week auditKeep reading
- What a 3.87% error on unseen data actually provesOne of our models predicted unseen months with 3.87% error. What that proves, what it does not, and why almost nobody publishes the number.
- Google Meridian vs Robyn vs a custom modelTwo free open-source modeling frameworks and the custom alternative. What each assumes, what each needs from your team, and the failure nobody warns you about.