How accurate is your forecast?

This is probably the most common question asked during our interaction with businesses. The expected answer is something in the percentage form! This expectation originates from the fact that predictions are typically evaluated with respect to actual numbers, in which 100% means a perfect fit.

Whilst this can be a reasonable question, the criteria used to evaluate the answer is more reminiscent of a strict academic framework where 90+% is deemed excellent while anything below is deemed unsatisfactory!

The main drawback of this evaluation method is the absence of any business KPIs. After all, the accuracy of a forecasting model is an abstract mathematical concept which does not have a direct business interpretation. We have seen many instances where 90% of accurate forecasts fail to drive business impact whilst 70% accurate forecasts create significant value.

Since wastage and inadequate stock are two competing business KPI’s, it’s important to combine them and derive a single KPI. The monetary loss of wastage primarily determined by the ordering cost of wasted goods. One may also add logistics and storage costs, but incremental costs that can be directly attributed to wasted items are generally insignificant. Hence, calculating monetary loss due to wastage is fairly straight forward.

Monetary loss due to inadequate stock can also be derived with some simple assumptions and a few additional data about the time point at which the out of stock event occurred. Unfortunately, though, the negative impact on customer satisfaction due to out of stock cannot be easily quantified. Nevertheless, assessing the overall monetary loss is the most appropriate way of combining the above two business KPIs.

The forecasting framework 

The objective was to forecast the daily demand of a particular variety of meat item at outlet level. For this particular evaluation exercise, we used data approximately 2 years prior to 2019 as training data. Apart from daily sales at outlet level, we included some predictor variables in the model which can potentially impact demand.

 The predictors considered were;

• Retail price
• Presence of outlet promotion (1/0)
• Wet market price
• Rainfall in the region

However, only price and outlet level promotions improved modeling accuracy significantly. We obtained the following forecasting accuracies for the 5 outlets, which can be generated from FORECAST Squared system.

As we can see, the average daily accuracies were about 75% for four of the five outlets. One outlet had an accuracy level of above 80%

Combined monetary impact as the evaluation metric

As previously discussed, we did not use model accuracy as the evaluation criteria. Instead we calculated the monetary impact due to wastage and out of stock situations to evaluate the model.

The basic steps of this calculation are summarized below;

The following table details corresponding calculation for Outlet: SEA001 

Forecasting models with less than 90% accuracy levels proved to be driving significant business outcomes. Hence the forecasting models should preferably be evaluated in the context of business outcomes, as opposed to abstract mathematical concepts.

The outcome

The financial gain would have amounted USD1,546 per SKU per outlet for period of 7 days if orders were placed based on the forecast.

The total financial gain across all five outlets amounted to approximately  USD 5,950 per SKU, for period of 7 day