The usual way of evaluating forecasting performance employs MAPE, or mean absolute percent measure, to determine the magnitude of forecast error, but this measure is not very helpful, says Gilliland, product marketing manager at SAS. "To fully and properly evaluate forecasting performance, companies need to know how efficient they were in reaching that level of accuracy and how much better they should be able to get," he says.

A number of simple tools can help properly evaluate forecasting performance, he says. All start out by measuring volatility, which is the coefficient of variation and is found by taking the standard deviation of the demand pattern and dividing it by the mean, Gilliland explains. "For example, if a product is selling 2,000 units per week and it has a standard deviation of 500, the coefficient of variation is 25 percent," he says. "This measure, while imperfect, is useful in determining the forecastability of demand - if the coefficient of variation is low, the pattern is smooth and stable and can be forecast quite accurately. If the measure is high with lots of ups and downs, it generally means the product can't be forecast as well."

An exception occurs with a highly seasonal pattern that repeats every year, he adds. This would create a high coefficient of variation, but the pattern could still be accurately predicted.

A coefficient of variation needs to be determined for every item being forecast, he says. Then each of these can be represented as a point on a simple scatter plot. "When you do this, you typically get a comet pattern," says Gilliland. "Very low volatility items cluster toward the head of the comet and very high volatility items spread out like a comet's tail.

If there is a very large tail, a high percentage of inventory is highly volatile and difficult to forecast, Gilliland says. "This raises the question of what can be done to reduce volatility and make these products more forecastable," he says. "Often several things can be done. For example, pricing practices and promotions drive volatility that is not necessarily natural in the product; if you can reduce some of those you will get a more stable pattern and a better forecast."

Another tools is a forecast value added analysis or FVA, Gilliland says. The FVA method begins with a simple forecast that requires no effort or cost. In a typical example this would be the last observed number: if you sold 100 last week, your forecast for next week would be 100. The idea behind FVA is to compare whatever forecasting method you actually are employing against this last-observation number. If what you are doing is not significantly better than the naïve model, then why bother, Gilliland asks.

"What we encourage our customers to do with an FVA analysis is to find things they are doing that may actually be making the forecast worse," he says. "Forecasting is typically a politicized process within an organization. Some may want to drive the forecast higher and others have reasons to want to drive it lower, so you can't always trust what people are contributing to a collaborative process." Doing an FVA analysis enables them to "stop shooting themselves in the foot," he says. "That is the first step to getting a better forecasting process. Then you can take the next step to automate the process with statistical modeling, which should get the forecasts about as good as they can ever expect be."

How good is that? It depends on the nature of what you are forecasting, Gilliland says, but it is an important question to ask. "Once a company reaches that level, it is wasting resources to try and get better. We know that no forecast is right 100 percent; if 80 percent is the best you can get, stop working on it and and focus those resources elsewhere."

To view the video in its entirety, click here

Keywords: Supply chain, it supply chain, supply chain management, it supply chain management, supply chain management it, supply chain management scm, inventory management, inventory management it, value chain, value chain it, inventory control, logistics & supply chain, supply chain solutions, supply chain planning, supply chain systems