What “a good model” means in demand forecasting — and how it's measured

In most companies demand forecasting rests on the past and on instinct: “last year it was this much, this year we'll nudge it up.” The moment a model is built, the first question should be: how will we know it is a good one?

“A good model” is not one that prints a single correct number. A good model is one whose error is known, and whose error is small enough for the decision. To see this, the model's forecasts are compared with values that were unknown at the time but later became real — and that comparison is made backwards, on historical data.

The method: split the history in two. Build the model on the older part, test it on the newer part it has never seen. The gap between what the model said for that period and what actually happened is the model's real performance. Testing happens where the model has to generalise, not memorise — otherwise you get a model that looks flawless on paper and misses in production.

How much a forecast changes the decision matters too. Stock, capacity and pricing decisions work with thresholds; whether the forecast keeps those thresholds on the right side is more meaningful than a decimal error rate. So measurement is tied to the decision: “how many times did this forecast tip it to the wrong side?”

At Miletus, when a forecasting engine is delivered, the layer that measures its accuracy ships with it: back-testing on historical data, continuous monitoring in production, and every forecast recorded with its basis. Output below the threshold does not reach the user. A good model is good not because it looks good — but because it is measured to be.