The Promise and Perils of Forecasting the Future

You’d be excused for thinking that the growth of big data combined with the increased sophistication of computer algorithms would make it easier to forecast the future. But that isn’t true.

This was made clear on June 23, the day a majority of voters in the United Kingdom cast ballots in favor of leaving the European Union. “I don’t think anybody saw that coming,” says Philip Tetlock, a professor at the University of Pennsylvania’s The Wharton School, and one of the world’s leading experts on forecasting.

Prediction markets had priced in an 85 percent probability that the U.K. would remain in the EU. A poll of polls by the Financial Times implied that 48 percent of people would vote to stay in the union compared to 46 percent who would cast their vote to leave. And on the day of the referendum, before results were in, equity markets in both the United States and Europe closed higher.

This reminder about the frailty of forecasts comes at an opportune time for the bank industry. Stress testing and the desire to avoid the mistakes that led to the financial crisis has made peering into the future an indispensable part of a bank’s planning and risk management processes. [See also the article in this issue on the new CECL accounting standard emphasizing future forecasts.] But as a conversation with bankers at two regional banks shows, the forecasting process isn’t the only thing that matters. Equally significant is what one does with those forecasts. “Plans are useless,” Dwight Eisenhower once said, “but planning is indispensable.”

Forecasting is nothing new to banks, of course. “We used it for credit. It was for asset/liability management. Banks with trading operations used it to calculate value at risk,” a way to gauge potential trading losses, says Darren King, chief financial officer of M&T Bank, a $124 billion bank based in Buffalo, New York. But what the stress tests have done is to tie these pieces together. “All the different parts of a bank are now included in the process, which means we have more minds around what could go wrong and what that would mean for loss rates and thus the capital position of the bank,” King continued.

From a strategic perspective, however, forecasting remains largely a defensive tactic, used to test the resiliency of a bank’s balance sheet rather than to make bets on interest rates or asset prices. “We try to run the bank as much as we can so that we don’t have a position in the direction of interest rates, commodity prices or anything else,” explains King. “We’re not oblivious to them, but we’re not making speculative bets on their direction.”

It’s easy to understand why when you consider the volatility of energy prices, which weigh heavily on the broader economy. “In 2014, nobody in the oil and gas industry was expecting prices to fall as much as they did,” says Mark Cranmer, head of energy finance at Cullen/Frost Bankers, a San Antonio, Texas-based financial holding company with $28 billion in assets. “We were forecasting for around $75 a barrel and the futures market was around $90 a barrel.” The actual price of West Texas Intermediate crude fell from a peak of nearly $110 a barrel in June 2014 to below $30 a barrel less than two years later.

Where a bank must use forecasts to set loan policy and make other strategic decisions, in turn, as is the case with oil and gas prices at Cullen/Frost and other energy lenders, the forecasts are only the starting point. Layered on top is a substantial margin for safety.

In Cullen/Frost’s case, it incorporates several layers of “discounting” to build a buffer between its forecasts and what might actually come to fruition, explains Cranmer. With respect to oil and gas production loans, which make up the large majority of the bank’s energy exposure, Cullen/Frost limits loan commitments to 65 percent of an estimated value of the underlying reserves, based on a conservative price estimate. Additionally, borrowers must be able to service their loans fully even if energy prices come in 25 percent below the assumed price.

M&T frames the strategic value of forecasting in a similar light. “Where the models can be really helpful is in understanding tail risk and how bad things could go,” says King. This is the point of the annual stress tests, but it also has positive business implications. King explained that it helps M&T determine if it’s too heavily concentrated in a particular industry and whether the bank is appropriately pricing in its risk.

Another point M&T and Cullen/Frost agree on is the role that subjective judgment-what Cranmer refers to as a banker’s “intangibles”-continues to play in the forecasting process, despite the growth of big data and the sophistication of computer algorithms. “Running the bank only on models would be dangerous, but running the bank only on subjective judgment would also be dangerous,” says King. “The power is putting the two together.”

There is nothing mystical or powerful about accurate intuition, Tetlock writes in his book “Superforecasting: The Art and Science of Prediction.” It’s pattern recognition. It’s about using one’s experience to tie together what may otherwise seem to be unrelated observations.

A story that Cranmer shared bears this out. “In 2014, the average wait on a Saturday night for a restaurant in Midland, Texas, was one hour, and only half of the restaurant was open because they didn’t have enough wait staff because everyone had gone out to the oil fields,” recounts Cranmer. “It was at that point that I first suspected oil prices were too high.”

Thus, while forecasting has gained prominence in response to new regulations and the growth of big data and computing power, it’s still a function of both art and science. David Ruffin, co-founder and chief strategy officer of Credit Risk Management Analytics, says this explains why the use of forecasting and predictive analytics to set strategy, as opposed to stress test a portfolio of loans or other assets, remains in its infancy.

What Does It Take to Be a Superforecaster?

Philip Tetlock has spent his professional life studying the accuracy of forecasts. His latest book, “Superforecasting: The Art and Science of Prediction,” delves into his latest research, revealing the traits that make some people better forecasters than others. Here are five of the most important traits that so-called superforcasters share.

1. Be pragmatic, not dogmatic: “For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded.”
2. Break it down: “[S]uperforecasters often tackle questions in a roughly similar wayu2013one that any of us can follow: unpack the question into components.”
3. Practice: “To get better at a certain type of forecasting, that is the type of forecasting you must dou2013over and over again…”
4. Synthesize others’ views: “Superforecasters constantly look for other views they can synthesize into their own.”
5. Perpetual beta: “The strongest predictor of rising into the ranks of superforecasters is perpetual beta, the degree to which one is committed to belief updating and self-improvement.”