A forecaster is only as good as their track record. Many people make forecasts, and they speak confidently about the future. However, not many people take the time and effort to test their accuracy and look for ways to improve. It might surprise people to learn that the people they watch on television, or see in front of boards and governments, are really bad at forecasting. Yet, they keep doing it, and people keep listening. Not because of their track record, but because of their assertion as an expert in the area. It’s almost funny, but its true. “Experts” are rarely very good at forecasting.

On this page we track and examine our forecasts. This page shows how we’ve been doing, and where we need to focus more attention. There are a few basic ways that we evaluate the success of a forecast. First, we want to make sure that our forecasts are well calibrated. That is to say, when we say something has a 10% chance of occurring, it should occur 10% of the time. If we find out that our 10% assigned probabilities actually happen 40% of the time, we need to adjust our methods so that people can rely on our assessments. Likewise if our 10% probability assignment don’t seem to ever actually happen. Calibration is important, as the assigned probabilities often feed into a decision model. If the assigned distribution is not reliable, then neither is the output.

The second test is what is commonly called a “Brier Score.” This is a way to test how good of a forecaster you are. This measurement awards you for forecasting correctly, and weights your score for how confidently you made your forecast. So, a perfect score would be someone who is 100% confident and 100% accurate. Likewise, a perfect bad score is someone who is 100% confident and 100% wrong (it turns out that this forecaster is actually very valuable. Just do the opposite of whatever he says). We are always looking for ways to improve our Brier score, as this generate even more value in the forecast. A well calibrated forecast is good, but a forecast that can eliminate uncertainty and stay calibrated is even better.

Finally, we want to make sure that our forecasts are unbiased and as close to accurate as possible. This is most important when forecasting continuous values, rather than binary outcomes. This test checks that our forecast errors balance to zero. If they don’t, we need to make an adjustment. We also want to reduce the range of errors as much as possible, although that is not always possible. Some uncertainty is due to pure randomness, which cannot be reduced by more study or better models. We need to know the difference.

Below are our previous and active forecasts. As the forecast events become known, we can begin testing our forecasts for bias and for accuracy. From there, we can find the ways we need to improve our skills and get better. Think of these scores as batting averages. Nobody expects someone to get a hit every at-bat, but we do want to understand how good a batter is before we put them in the line-up. Beware of any forecaster that isn’t willing to look at their stats.

 

Category Event Probability of Occurrence Forecast Date Event Date Forecast  Advance (days) Method
Oil The FY18 average daily ANS oil price will be above $61.62 90% 4/1/2018 6/30/2018 90 AR1
Oil The FY18 average daily ANS oil price will be above $62.60 50% 4/1/2018 6/30/2018 90 AR1
Oil The FY18 average daily ANS oil price will be above $63.59 10% 4/1/2018 6/30/2018 90 AR1
Oil The April 2018 average daily ANS oil price will be above $66.66 90% 4/1/2018 4/30/2018 29 AR1
Oil The April 2018 average daily ANS oil price will be above $69.45 50% 4/1/2018 4/30/2018 29 AR1
Oil The April 2018 average daily ANS oil price will be above $72.23 10% 4/1/2018 4/30/2018 29 AR1
Oil The May 2018 average daily ANS oil price will be above $65.22 90% 4/1/2018 5/31/2018 60 AR1
Oil The May 2018 average daily ANS oil price will be above $69.89 50% 4/1/2018 5/31/2018 60 AR1
Oil The May 2018 average daily ANS oil price will be above $74.63 10% 4/1/2018 5/31/2018 60 AR1
Oil The June 2018 average daily ANS oil price will be above $64.50 90% 4/1/2018 6/30/2018 90 AR1
Oil The June 2018 average daily ANS oil price will be above $70.251 50% 4/1/2018 6/30/2018 90 AR1
Oil The June 2018 average daily ANS oil price will be above $75.94 10% 4/1/2018 6/30/2018 90 AR1