DEMDemandStop 6 of 12

Forecast accuracy & bias

MAPE, WMAPE, MASE, bias and tracking signal, what each one really measures and when each one misleads.

3 min gist11 min in full Draftreview pending

Read atChange any time

Gist3 min How it works+4 min Deep dive+4 min

How far off you are, which way you are off, and why the metric quietly shapes the forecast.

Two different questions hide behind “how good is the forecast?”

Accuracy: how far off are we, ignoring direction?
Bias: are we consistently too high or too low?

Bias is the more dangerous one. Random errors partly cancel out over time; a steady 5% over-forecast doesn’t, and it quietly fills warehouses month after month. Report both, always.

The common accuracy measures each have a catch:

MAE (average absolute error) is easy to read but in units, so you can’t compare a fast mover with a slow one.
RMSE squares errors before averaging, so it punishes big misses hardest. That is useful for inventory, because safety stock is driven by the size of the errors.
MAPE (average percentage error) is the corporate favourite, and a trap. It can’t handle zero sales, it over-weights tiny items, and it treats over- and under-forecasts unequally, which pushes models toward forecasting low.
WMAPE (also called WAPE) divides total absolute error by total demand. Big items dominate, zeros are fine, and it’s the practical default in many planning teams.
MASE compares your error with the error of a naive “repeat last period” forecast. Below 1 means you beat naive; above 1 means you would do better doing nothing clever.

To track bias in production, use the tracking signal: the running sum of errors divided by the mean absolute deviation. If it drifts outside about ±4, errors are piling up on one side and the model needs attention.

And test properly. Never judge a model on the data it was fitted to; backtest on rolling periods it hasn’t seen, at the horizon the decision actually uses.

Words to use

WMAPE: Sum of absolute errors ÷ sum of actual demand; volume-weighted and safe with zeros.
MASE: Your mean absolute error ÷ that of a naive forecast; below 1 beats naive.
Tracking signal: Cumulative error ÷ mean absolute deviation; outside about ±4 signals bias.

Every common measure computed on the same five weeks, plus bias and the tracking signal.

One example, every measure

Define each week’s error as forecast minus actual, so a positive error means over-forecasting.

Worked example: five weeks of one item

Week	1	2	3	4	5	Total
Actual	100	120	90	110	130	550
Forecast	110	110	105	100	120	545
Error	+10	−10	+15	−10	−10	−5
Absolute error	10	10	15	10	10	55

Bias (mean error): −5 / 5 = −1 a week, or −0.9% of demand: almost unbiased.
MAE: 55 / 5 = 11 units.
RMSE: $\sqrt{(100 + 100 + 225 + 100 + 100)/5} = \sqrt{125} \approx 11.2$ , a little above MAE because the 15-unit miss counts extra.
MAPE: the average of 10/100, 10/120, 15/90, 10/110 and 10/130, which is 10.4%.
WMAPE: 55 / 550 = 10.0%.
MASE: if the naive “same as last week” forecast had an average error of 19 units, MASE = 11 / 19 = 0.58, so this forecast beats naive comfortably.
Tracking signal: cumulative error ÷ MAE = −5 / 11 = −0.45, well inside ±4.

The formulas

\text{MAE} = \frac{1}{n}\sum \lvert e_t \rvert \qquad \text{RMSE} = \sqrt{\frac{1}{n}\sum e_t^{2}} \qquad \text{WMAPE} = \frac{\sum \lvert e_t \rvert}{\sum A_t}

\text{MAPE} = \frac{1}{n}\sum \frac{\lvert e_t \rvert}{A_t} \qquad \text{MASE} = \frac{\text{MAE}}{\text{MAE of the naive forecast}}

Strictly, MASE scales by the naive forecast’s average error on the training data, which makes it comparable across items and periods.

Why MAPE leans low

MAPE divides by the actual. Forecast 150 when 100 sell and the error is 50%; forecast 100 when 150 sell and it is 33%, for the same 50-unit miss. A model or a planner judged on MAPE learns that forecasting low is safer, and the supply chain pays for it in stockouts. And a single week with zero sales makes MAPE undefined.

Bias needs its own number

Accuracy measures ignore direction, so a forecast can look accurate while leaning one way. Track the mean error or the percentage bias, Σ(forecast − actual) ÷ Σ actual, by item group and by whoever adjusts the forecast, and watch the tracking signal for drift.

Matching the measure to the decision, why the median can starve your shelves, checking intervals, and how metrics get gamed.

Each measure rewards a different forecast

The forecast that minimizes MAE is the median of possible outcomes; the one that minimizes RMSE is the mean. For symmetric demand they coincide. For skewed demand, which covers most slow movers and anything with occasional spikes, the median sits below the mean, so a model tuned to MAE forecasts low. If replenishment uses that number as the average rate, shelves run short. Choose the measure for the decision: mean-based measures for replenishment volumes, quantile measures for service-driven stock.

Measure at the level and lag that matter

Level: accuracy at brand level always looks better than at SKU-location level, because errors cancel when you add up. Report the level where stock is decided.
Lag: use the forecast made one lead time before the period.
Weighting: a plain average of item MAPEs treats a ₹100 item like a ₹1 crore item. Volume- or value-weighted measures reflect the money at stake.

Checking prediction intervals

If you publish P10 and P90 forecasts, check them. Over a year, actual demand should land above the P90 in about 10% of periods. If it happens 25% of the time, the intervals are too narrow, and stock set from them will run short. Two properties matter: calibration (stated probabilities come true) and sharpness (intervals as narrow as calibration allows). Pinball loss scores both.

How good is good?

There is no universal benchmark. A stable brand forecast monthly can be far more accurate than a fashion item forecast by store and week, and both can be well managed. The meaningful comparisons are against a naive forecast for the same items (MASE, forecast value add) and against your own history over time.

Metrics get gamed

Accuracy = 1 − MAPE can go negative, so it gets capped at zero, which hides the worst items.
Rewarding planners on accuracy alone invites the safe, low number. Reward accuracy and bias together.
Excluding “abnormal” periods from the metric is legitimate only if the rule was fixed in advance.

Interview kit

Practise it out loud

Answer each question yourself before opening the model answer.

Say it in 30 seconds

I’d measure accuracy and bias separately. For accuracy I prefer WMAPE, which is volume-weighted and handles zeros, or MASE, which is scale-free and benchmarked against naive, over plain MAPE, which breaks on zeros and over-weights small items. For bias I track the mean error and a tracking signal, and I evaluate models with rolling-origin backtests at the horizon the replenishment decision uses.

What's wrong with MAPE?

It divides by actual demand, so it’s undefined when demand is zero and explodes for small items. It also weights small SKUs as much as big ones, and it penalizes over-forecasting without limit while under-forecasting is capped at 100%, which nudges models toward low forecasts and stockouts.

A forecast is accurate on average but the warehouse keeps overstocking. What's going on?

Probably bias: small errors that are always in the same direction. Accuracy metrics hide it because they ignore the sign. Check the mean error and the tracking signal; a persistent positive bias means the forecast, or an override process, is systematically too high.

Connected stops

Where this idea feeds in, or gets its inputs from.

DEMForecasting fundamentals DEMConsensus forecasting & FVA INVSafety stock & reorder points DEMIntermittent demand: Croston, SBA, TSB DEMProbabilistic forecasting

Sources and further reading

Hyndman, R. J. & Koehler, A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22(4), 679–688.
Hyndman, R. J. & Athanasopoulos, G. Forecasting: Principles and Practice (3rd ed.). OTexts. Chapter on evaluating forecast accuracy.

1 Level 1 The gist 3 min

2 Level 2 How it works 4 min

One example, every measure

The formulas

Why MAPE leans low

Bias needs its own number

3 Level 3 Deep dive 4 min

Each measure rewards a different forecast

Measure at the level and lag that matter

Checking prediction intervals

How good is good?

Metrics get gamed

Connected stops

Sources and further reading