Short answer
Hiding recent months from a model and making it predict them tells you whether it learned your business or just memorized your history. One of our client models scored 3.87% on a three-month hidden window, against 4.85% in training. That proves it generalizes. It does not prove every channel number inside it is right — only a test does that.
Ask a vendor how accurate their model is and you usually get an R-squared, a fit chart, or a case study. All three answer a different question than the one you asked.
Predicting months it never saw came in below the fit on months it did. That is the signature of a model that learned the business instead of memorizing its history.
What is a holdout test?
- Holdout test
- Remove the most recent months and build the model without them. Then make it predict those months blind. That error is the only accuracy number that says anything about the future.
Fitting the past is not hard. A flexible enough model traces any history perfectly and predicts nothing. In-sample fit measures memorizing. Holdout error measures learning.
What does it not prove?
A strong holdout proves the model predicts the total well. It does not prove the split between channels is right.
- Two channels that moved together in your history can trade credit without changing the total at all.
- The hidden window is one stretch of one business. Evidence, not a guarantee about a different spend level.
- A model can predict well and still be wrong about cause.
A model with nothing to check it against. The line looks confident. The shaded area is how wrong it could be — and at high spend, it could be very wrong. This is what you are moving budget on when nobody tests.
What should you ask a vendor for?
- 01The error on a recent client model, and how long the hidden window was.
- 02The training error next to it, so you can see the gap.
- 03Whether the hidden window was chosen before or after the model was built.
- 04How many versions they tried before the one they are showing you.
- 05What happened the last time a model failed.
“If a vendor will not tell you their error on unseen data, that silence is your answer.”
What counts as a good error?
Depends on the KPI and the grain. For weekly retail traffic, low single digits is strong. What matters is the comparison to training error. We will not ship above 10%.
How long should the hidden window be?
Three to six months of weekly data is our standard. Too short and you are testing against noise.
Can a model be useful without this?
It can be interesting. It should not move budget.
Does a good holdout mean the channel numbers are right?
No. Total accuracy and channel accuracy are different claims. Match-market tests check the second.
Find your wasted third
Want this run on your own numbers?
The two-week audit shows you where each of your channels sits on its own curve — what to cut, what to grow, and how much budget is sitting in the wrong place.
Book the two-week auditKeep reading
- The model that failed, and what it cost to find outWe split a working model in two. One came back with 832% error on unseen data. Why it happened, and why it would have been invisible without a holdout.
- How to find the point where your channel stops paying offWhat data you need, which method fits your budget, and how to check the answer against the real world before you move money.