Each October, the Alaska Department of Revenue (DOR) holds an oil price forecasting session. They gather about 30 people in a room in Anchorage, talk to them about what’s going on in the oil markets, and eventually ask them to write down what they think is going to happen to oil prices for the next 10 years. After some internal debate, some massaging of the numbers, and a nod from the commissioner, they produce a forecast of what they expect oil prices to be.
This forecast becomes an input into their revenue forecast, which they give to the legislature in December. It is this number that the legislature relies upon as they decide how much money the state should spend on government services, how much they should distribute as dividends, how much they can afford on capital projects, and whether or not they need to generate additional revenues through taxes.
If you would like to double-check our numbers or run your own analysis, you can find DOR’s historic publications here.
Fiscal Year 2018
As FY18 closes out in about 3 weeks, it is interesting to see how the DOR forecast for this year has changed over time. In the chart below, the blue line plots DOR’s forecasts of FY18 over time. The orange line is what we now estimate will be the actual price.
The first forecast that DOR provided for FY18 was in 2008 where they approached the fast escalating prices of the day with some conservatism. As oil prices of the day kept climbing, DOR kept chasing it with their projection of the future. Finally, when oil prices cratered in 2015, DOR chased the drop with their forecast. Once the reality of the over-correction was obvious to everyone, they allowed their forecast to track back up.
Things to Note
There are a couple of things to note here. First, it is clear that DOR’s forecasting technique tilts toward tracking the price they are observing in the present. While this may be valid for near term projections, it does not perform well in the long-term. What this does is tell the legislature that the future is bright whenever the present is rosy. And they end up playing “Chicken Little” every time the commodity cycle hits a low point. This is not a good way to budget.
Second, note that the legislature budgets for the fiscal year that begins after the session ends. So, it was the 2017 legislative session in which they budgeted for FY18. However, the Revenue Sources Book (RSB) is published in December of the calendar year prior to the session. That means that the FY18 budget was made based on the Fall 2016 RSB. Take another look at that graphic above. By the time DOR started catching up to what the market was doing, the budget had already passed a year prior.
The legislature made their FY18 plans based on a $54 oil price, while the actual price ended up around $64. This error represents somewhere between $300 – $600 million.
Fiscal Year 2019
The legislature recently ended the 2018 session with a FY19 budget in place. That budget relied on an oil price of $57 per barrel. In March, DOR revised the forecast up to $63 per barrel before the budget was finally passed.
The price of oil today is $72.04 per barrel and our current projection of FY19 is $74.36. If the actual price ends up in this range, the legislature will have budgeted on $500 million to $1 billion less than the state will actually collect.
By the way, our model gives the likelihood of the FY19 oil price being $57 or lower at 7%. For the revised $63 value provided in the spring, the model says there’s a 16% chance of being that low or lower.
Historic Errors
As you can see from the chart above, DOR has historically provided price forecasts that are lower than what ends up happening. That pattern changed in 2015 when oil price crashed, but resumed when prices began to recover.
While some call this “conservative forecasting for budgeting purposes,” this is the wrong way to approach the issue. Artificially lowering one variable in your forecast is a terrible way to reduce your budget target.
On net, DOR budget year forecasts have been off by an average of $20 (31%) over the last 15 years. The size of these errors raises two questions. First, is the effort of forecasting providing better results than a simple heuristic? And second, are there opportunities to reduce those errors?
Simple Heuristic
We tested a couple easy options to see how they perform, and how much better DOR’s method does. First, we simply used the price of the previous year to forecast the next year (so the 2017 RSB would use the FY17 actual price as the forecast for FY19). The average error of this approach is $22 (35%).
We also tested several moving average techniques. None of them performed very well. The reason is that there is so much volatility in the numbers that any moving average does not adjust quickly enough to new information. And oil price does not revert to an average value quickly enough for averaging to be effective in the short-term.
Given this need to adjust quickly to new information, we tested whether just simply using the most recent data might be more effective. To do this, will simply used the average price in October (when DOR does their forecast) as a predictor for the upcoming budget year (so October 2016 would predict FY18). This too failed to provide a good prediction, with an error averaging $26 (42%).
So what if we just didn’t try to forecast at all? What if we just used a $70 oil price every year? Well, doing that results in an average error of just $18 (25%). Maybe trying to forecast a random variable is just chasing your tail.
DOR Forecast Results
DOR has a history of lagging the trend in its forecasts. This is typical in any type of forecasting approach that puts more weight on what is happening at the time of the forecast. This type of approach reacts quickly to new information, but is also very volatile.
DOR’s forecast does seem to have some value in the near term (it is statistically significant, although has little explanatory power). But, it has a timing problem. Their forecast is produced in October of one year, and is forecasting nearly two full years later. It does little good to have a forecast that relies on current data if the forecast is out of date before it is used.
The short shelf-life of their forecast is reason for concern. And beyond the budget year, the forecast yields no value at all. The legislature should not rely on the 6 month old forecast as they make budget decisions and should not rely on the long-term forecast whatsoever.
How DOR Forecasts Oil Prices
I’ve participated in their price forecasting sessions, so I can tell you how it works (side note: when I worked for DOR, I only worked on the production forecasts. There was another team in charge of the price forecast).
At various point in time, DOR has used different forecasting methods. But they generally rely on what they call a “modified Delphi method” (aka “expert opinion”). Participants are presented with various information over the course of a day, and then are asked their opinion of what future prices will be.
Those opinions are averaged, blended with other forecasts, and ultimately taken to the Commissioner for approval. The result is a very generic view of the direction that price is headed. But, all this averaging and emphasis on current events ends up putting a lot of weight on the present. While this is generally ok for the near term, it creates a pretty useless view of the future.
And then, when the price forecast is updated in the Spring, they don’t bother to go through this exercise again. They don’t even ask the participants to update their own forecasts. Instead, they simply plug in the actual values that have occurred since October into the corresponding months. They then “adjust” the future months accordingly.
Opportunities for Improvement
If DOR wants to continue to use the opinions of individuals to forecast price, I recommend they do some research into how to harness the “Wisdom of Crowds” and how to properly manage small team meetings. Having been through the process and having studied the topic, I can tell you that they are not following best practices.
Not only do they set anchors before eliciting responses, they do not allow adequate independence and diversity of ideas to be explored. Therefore, they end up telling participants how they should answer the question before asking them to do so.
But in reality, I’m not so sure the “subjective assessment” of a room full of quasi experts is even the best way to approach this issue. If they insist on trying to tell the future, I would recommend an algorithmic approach to setting a base rate (maybe like the one we use). They could then test whether subjective adjustments improve accuracy. This is the new age approach that is proving successful in things like the “Good Judgement Open” and other attempts to leverage technology to get better forecasts in the digital age.
Fix the Timing Issue
The biggest improvement would be to decrease the lag between when the forecast is produced and when it is used. There is simply no justification for a six month lag in this day and age. They need to move into the modern era. Make an adaptive model that can update in real-time, and move the whole thing to an online platform.
Improve Transparency
The next biggest improvement would be to remove any internal control from the forecast results. Even if every past forecast has been made with absolute integrity, the current process lends itself to manipulation. Because the Commissioner has the ability to arbitrarily adjust the forecast, the credibility of the output is compromised.
An administration that wants to justify spending more money could theoretically increase the forecast to show that the increased spending is affordable. One that wants to cut government spending could do the opposite. Therefore, the lack of transparency and the ability to skew results should raise questions by anyone that uses the forecast.
This could be corrected by using an outside source (like a publicly published value). Or by using an average of historical data that is fully transparent (although you may trade some accuracy for transparency). Or by hiring an independent and impartial third-party that is judged by testing their forecasts against the results over time.
Of course, another option is to stop trying to forecast at all (which appears may be an improvement).
Takeaways
First, a user of DOR’s forecast should be aware of the range around that forecast. The legislature should not budget based on the forecast value without fully appreciating that the actual value could easily be $20 higher or lower than the forecast. Be aware that the revenue forecast could easily be off by $600 million. Any explanation that justifies how the future will unfold should be dismissed out of hand. Instead, budgeting should be approached honoring how much we don’t know about the future rather than pretending like we do.
Second, think about what we are getting in exchange for the time and effort that goes into creating the price forecast. My guess is that there is a better use for that money and a better approach to forecasting in general.
A New Approach
Finally, the legislature should rethink how it approaches budgeting. Given the volatility of oil prices (and investment returns), budgeting to a revenue projection is a fool’s errand to begin with. Let’s say I had a crystal ball that was 100% accurate. From it, I tell you that you are going to have $10 billion one year and $2 billion the next. Should you spend $10 billion one year and then adjust down in the next? Or would it be smarter to spend $6 billion in each?
I recommend the entire budgeting process be reevaluated on a sustainable budget basis with embedded contingency plans and risk mitigation strategies. Doing so requires valid and dependable long-term forecasts that should ignore day-to-day price movements altogether. These forecasts should not be adjusted without verifiable evidence that the underlying market conditions have fundamentally changed.
If this approach was taken in the past, the legislature would not have increased spending when oil prices reached historic highs. Then, they would have had money in the bank to balance the budget when oil prices returned to Earth.
Somehow I doubt this will happen.