Forecast Accuracy Quotes

Pyscho-history dealt not with man, but with man-masses. It was the science of mobs; mobs in their billions. It could forecast reactions to stimuli with something of the accuracy that a lesser science could bring to the forecast of a rebound of a billiard ball. The reaction of one man could be forecast by no known mathematics; the reaction of a billion is something else again.

If you ever do have to heed a forecast, keep in mind that its accuracy degrades rapidly as you extend it through time.

History cannot be explained deterministically and it cannot be predicted because it is chaotic. So many forces are at work and their interactions are so complex that extremely small variations in the strength of the forces and the way they interact produce huge differences in outcomes. Not only that, but history is what is called a ‘level two’ chaotic system. Chaotic systems come in two shapes. Level one chaos is chaos that does not react to predictions about it. The weather, for example, is a level one chaotic system. Though it is influenced by myriad factors, we can build computer models that take more and more of them into consideration, and produce better and better weather forecasts. Level two chaos is chaos that reacts to predictions about it, and therefore can never be predicted accurately. Markets, for example, are a level two chaotic system. What will happen if we develop a computer program that forecasts with 100 per cent accuracy the price of oil tomorrow? The price of oil will immediately react to the forecast, which would consequently fail to materialise. If the current price of oil is $90 a barrel, and the infallible computer program predicts that tomorrow it will be $100, traders will rush to buy oil so that they can profit from the predicted price rise. As a result, the price will shoot up to $100 a barrel today rather than tomorrow. Then what will happen tomorrow? Nobody knows.

For a number of years, professors at Duke University conducted a survey in which the chief financial officers of large corporations estimated the returns of the Standard & Poor’s index over the following year. The Duke scholars collected 11,600 such forecasts and examined their accuracy. The conclusion was straightforward: financial officers of large corporations had no clue about the short-term future of the stock market; the correlation between their estimates and the true value was slightly less than zero!

More often forecasts are made and then…nothing. Accuracy is seldom determined after the fact and is almost never done with sufficient regularity and rigor that conclusions can be drawn. The reason? Mostly it’s a demand-side problem: The consumers of forecasting—governments, business, and the public—don’t demand evidence of accuracy. So there is no measurement. Which means no revision. And without revision, there can be no improvement.

The Weather Service was initially organized under the Department of War by President Ulysses S. Grant, who authorized it in 1870. This was partly because President Grant was convinced that only a culture of military discipline could produce the requisite accuracy in forecasting25 and partly because the whole enterprise was so hopeless that it was only worth bothering with during wartime when you would try almost anything to get an edge.

Couples who regularly practice empathy see stunning results. It is the independent variable that predicts a successful marriage, according to behaviorist John Gottman, who, post hoc criticisms notwithstanding, forecasts divorce probabilities with accuracy rates approaching 90 percent. In Gottman’s studies, if the wife felt she was being heard by her husband—to the point that he accepted her good influence on his behavior—the marriage was essentially divorce-proof. (Interestingly, whether the husband felt heard was not a factor in divorce rates.) If that empathy trafficking was absent, the marriage foundered. Research

Larry Kudlow hosted a business talk show on CNBC and is a widely published pundit, but he got his start as an economist in the Reagan administration and later worked with Art Laffer, the economist whose theories were the cornerstone of Ronald Reagan’s economic policies. Kudlow’s one Big Idea is supply-side economics. When President George W. Bush followed the supply-side prescription by enacting substantial tax cuts, Kudlow was certain an economic boom of equal magnitude would follow. He dubbed it “the Bush boom.” Reality fell short: growth and job creation were positive but somewhat disappointing relative to the long-term average and particularly in comparison to that of the Clinton era, which began with a substantial tax hike. But Kudlow stuck to his guns and insisted, year after year, that the “Bush boom” was happening as forecast, even if commentators hadn’t noticed. He called it “the biggest story never told.” In December 2007, months after the first rumblings of the financial crisis had been felt, the economy looked shaky, and many observers worried a recession was coming, or had even arrived, Kudlow was optimistic. “There is no recession,” he wrote. “In fact, we are about to enter the seventh consecutive year of the Bush boom.”19 The National Bureau of Economic Research later designated December 2007 as the official start of the Great Recession of 2007–9. As the months passed, the economy weakened and worries grew, but Kudlow did not budge. There is no recession and there will be no recession, he insisted. When the White House said the same in April 2008, Kudlow wrote, “President George W. Bush may turn out to be the top economic forecaster in the country.”20 Through the spring and into summer, the economy worsened but Kudlow denied it. “We are in a mental recession, not an actual recession,”21 he wrote, a theme he kept repeating until September 15, when Lehman Brothers filed for bankruptcy, Wall Street was thrown into chaos, the global financial system froze, and people the world over felt like passengers in a plunging jet, eyes wide, fingers digging into armrests. How could Kudlow be so consistently wrong? Like all of us, hedgehog forecasters first see things from the tip-of-your-nose perspective. That’s natural enough. But the hedgehog also “knows one big thing,” the Big Idea he uses over and over when trying to figure out what will happen next. Think of that Big Idea like a pair of glasses that the hedgehog never takes off. The hedgehog sees everything through those glasses. And they aren’t ordinary glasses. They’re green-tinted glasses—like the glasses that visitors to the Emerald City were required to wear in L. Frank Baum’s The Wonderful Wizard of Oz. Now, wearing green-tinted glasses may sometimes be helpful, in that they accentuate something real that might otherwise be overlooked. Maybe there is just a trace of green in a tablecloth that a naked eye might miss, or a subtle shade of green in running water. But far more often, green-tinted glasses distort reality. Everywhere you look, you see green, whether it’s there or not. And very often, it’s not. The Emerald City wasn’t even emerald in the fable. People only thought it was because they were forced to wear green-tinted glasses! So the hedgehog’s one Big Idea doesn’t improve his foresight. It distorts it. And more information doesn’t help because it’s all seen through the same tinted glasses. It may increase the hedgehog’s confidence, but not his accuracy. That’s a bad combination.

The only thing that we know about financial predictions of startups is that 100 percent of them are wrong. If you can predict the future accurately, we have a few suggestions for other things you could be doing besides starting a risky early stage company. Furthermore, the earlier stage the startup, the less accurate any predications will be. While we know you can't predict your revenue with any degree of accuracy (although we are always very pleased in that rare case where revenue starts earlier and grows faster than expected), the expense side of your financial plan is very instructive as to how you think about the business. You can't predict your revenue with any level of precision, but you should be able to manage your expenses exactly to plan. Your financials will mean different things to different investors. In our case, we focus on two things: (1) the assumptions underlying the revenue forecast (which we don't need a spreadsheet for—we'd rather just talk about them) and (2) the monthly burn rate or cash consumption of the business. Since your revenue forecast will be wrong, your cash flow forecast will be wrong. However, if you are an effective manager, you'll know how to budget for this by focusing on lagging your increase in cash spend behind your expected growth in revenue.

Taleb, Kahneman, and I agree there is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious—“there will be conflicts”—and the odd lucky hits that are inevitable whenever lots of forecasters make lots of forecasts. These limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems. In my EPJ research, the accuracy of expert predictions declined toward chance five years out. And yet, this sort of forecasting is common, even within institutions that should know better. Every four years, Congress requires the Department of Defense to produce a twenty-year forecast of the national security environment.

The first 20 percent often begins with having the right data, the right technology, and the right incentives. You need to have some information—more of it rather than less, ideally—and you need to make sure that it is quality-controlled. You need to have some familiarity with the tools of your trade—having top-shelf technology is nice, but it’s more important that you know how to use what you have. You need to care about accuracy—about getting at the objective truth—rather than about making the most pleasing or convenient prediction, or the one that might get you on television. Then you might progress to a few intermediate steps, developing some rules of thumb (heuristics) that are grounded in experience and common sense and some systematic process to make a forecast rather than doing so on an ad hoc basis.

The complexity was again staggering. There seemed to be this endless list of challenges in building the models: better observations, more observations, better use of observations, more efficient use, better calibration, higher resolution, higher accuracy, faster computers, or more frequent outputs. There was never one thing to tweak. Every time I thought I might have a handle on how things worked, I would hear about another layer.

Similarly, your anxiety, worry, and fears try to predict the future, but they do so with even less accuracy than weather forecasters.

Every day, experts bombard us with predictions, but how reliable are they? Until a few years ago, no one bothered to check. Then along came Philip Tetlock. Over a period of ten years, he evaluated 28,361 predictions from 284 self-appointed professionals. The result: In terms of accuracy, the experts fared only marginally better than a random forecast generator. Ironically, the media darlings were among the poorest performers; and of those, the worst were the prophets of doom and disintegration.

We were now receiving daily very accurate weather reports from the Bracknell Weather Centre in the UK. These gave us the most advanced precision forecast available anywhere in the world. The meteorologists were able to determine wind strengths to within five knots accuracy at every thousand feet of altitude. Our lives would depend on these forecasts back up the mountain. Each morning, the entire team would crowd eagerly around the laptop to see what the skies were bringing--but it did not look good. Those early signs of the monsoon arriving in the Himalayas, the time when the strong winds over Everest’s summit begin to rise, didn’t seem to be coming. All we could do was wait. Our tents were very much now home to us at base camp. We had all our letters and little reminders from our families. I had a seashell I had taken from a beach on the Isle of Wight, in which Shara had written my favorite verse--one I had depended on so much through the military. “Be sure of this, that I am with you always, even unto the end of the earth.” Matthew 28:20. I reread it every night at base camp before I went to sleep. There was no shame in needing any help up here.

Level two chaos is chaos that reacts to predictions about it, and therefore can never be predicted accurately. Markets, for example, are a level two chaotic system. What will happen if we develop a computer program that forecasts with 100 per cent accuracy the price of oil tomorrow? The price of oil will immediately react to the forecast, which would consequently fail to materialise.

Researchers have found that merely asking people to assume their initial judgment is wrong, to seriously consider why that might be, and then make another judgment, produces a second estimate which, when combined with the first, improves accuracy almost as much as getting a second estimate from another person.

The gap between a conventional forecast and one that uses RCF varies by project type, but for over half the projects for which we have data, RCF is better by 30 percentage points or more. That’s on average. A 50 percent increase in accuracy is common. Improvements of more than 100 percent are not uncommon. Most gratifyingly, given the method’s intellectual roots, Daniel Kahneman wrote in Thinking, Fast and Slow that using reference-class forecasting is “the single most important piece of advice regarding how to increase accuracy in forecasting through improved methods.

Chaotic systems come in two shapes. Level one chaos is chaos that does not react to predictions about it. The weather, for example, is a level one chaotic system. Though it is influenced by myriad factors, we can build computer models that take more and more of them into consideration, and product better and better weather forecasts. Level two chaos is chaos that reacts to predictions about it and therefore can never be predicted accurately. Markets, for example, are a level two chaotic system. What will happen if we develop a computer program that forecasts with 100 per cent accuracy the price of oil tomorrow? The price of oil will immediately react to the forecast, which would consequently fail to materialize.

point of making forecasts is not to tick all the boxes on the “how to make forecasts” checklist. It is to foresee what’s coming. To have accountability for process but not accuracy is like ensuring that physicians wash their hands, examine the patient, and consider all the symptoms, but never checking to see whether the treatment works.

in the presence of nonlinearities, the longer the forecast the worse its accuracy.

Level two chaos is chaos that reacts to predictions about it, and therefore can never be predicted accurately. Markets, for example, are a level two chaotic system. What will happen if we develop a computer program that forecasts with 100 per cent accuracy the price of oil tomorrow? The price of oil will immediately react to the forecast, which would consequently fail to materialise. If the current price of oil is $90 a barrel, and the infallible computer program predicts that tomorrow it will be $100, traders will rush to buy oil so that they can profit from the predicted price rise. As a result, the price will shoot up to $100 a barrel today rather than tomorrow. Then what will happen tomorrow? Nobody knows.

Accurate forecasts may help do that sometimes, and when they do accuracy is welcome, but it is pushed aside if that’s what the pursuit of power requires.

History cannot be explained deterministically and it cannot be predicted because it is chaotic. So many forces are at work and their interactions are so complex that extremely small variations in the strength of the forces and the way they interact produce huge differences in outcomes. Not only that, but history is what is called a ‘level two’ chaotic system. Chaotic systems come in two shapes. Level one chaos is chaos that does not react to predictions about it. The weather, for example, is a level one chaotic system. Though it is influenced by myriad factors, we can build computer models that take more and more of them into consideration, and produce better and better weather forecasts. Level two chaos is chaos that reacts to predictions about it, and therefore can never be predicted accurately. Markets, for example, are a level two chaotic system. What will happen if we develop a computer program that forecasts with 100 per cent accuracy the price of oil tomorrow? The price of oil will immediately react to the forecast, which would consequently fail to materialise. If the current price of oil is $90 a barrel, and the infallible computer program predicts that tomorrow it will be $100, traders will rush to buy oil so that they can profit from the predicted price rise. As a result, the price will shoot up to $100 a barrel today rather than tomorrow. Then what will happen tomorrow? Nobody knows. Politics, too, is a second-order chaotic system. Many people criticise Sovietologists for failing to predict the 1989 revolutions and castigate Middle East experts for not anticipating the Arab Spring revolutions of 2011. This is unfair. Revolutions are, by definition, unpredictable. A predictable revolution never erupts.

If you can predict with any level of accuracy for a startup, you belong in the forecaster’s hall of fame. You will be the first inductee. John Kenneth Galbraith once said that there are two kinds of forecasters, “those who know they don’t know and those who don’t know they don’t know.

In 1997, money manager David Leinweber wondered which statistics would have best predicted the performance of the U.S. stock market from 1981 through 1993. He sifted through thousands of publicly available numbers until he found one that had forecast U.S. stock returns with 75% accuracy: the total volume of butter produced each year in Bangladesh. Leinweber was able to improve the accuracy of his forecasting “model” by adding a couple of other variables, including the number of sheep in the United States. Abracadabra! He could now predict past stock returns with 99% accuracy. Leinweber meant his exercise as satire, but his point was serious: Financial marketers have such an immense volume of data to slice and dice that they can “prove” anything.