Noise Daniel Kahneman Quotes

To understand error in judgment, we must understand both bias and noise.

wherever there is judgment, there is noise—and more of it than you think.

There is at least one source of occasion noise that we have all noticed: mood.

Causally, noise is nowhere; statistically, it is everywhere.

Bias and noise—systematic deviation and random scatter—are different components of error.

It is more useful to pay attention to people who disagree with you than to pay attention to those who agree.

Life is often more complex than the stories we like to tell about it.

Most organizations prefer consensus and harmony over dissent and conflict. The procedures in place often seem expressly designed to minimize the frequency of exposure to actual disagreements and, when such disagreements happen, to explain them away.

people are rarely aware of their own biases when they are being misled by them. This lack of awareness is itself a known bias, the bias blind spot. People often recognize biases more easily in others than they do in themselves

In a negotiation situation, for instance, good mood helps. People in a good mood are more cooperative and elicit reciprocation. They tend to end up with better results than do unhappy negotiators.

When physicians are under time pressure, they are apparently more inclined to choose a quick-fix solution, despite its serious downsides.

good judgments depend on what you know, how well you think, and how you think.

People cannot be faulted for failing to predict the unpredictable, but they can be blamed for a lack of predictive humility.

it is hard to agree with reality if you cannot agree with yourself.

In terms of noise, psychiatry is an extreme case.

Judgment is not a synonym for thinking, and making accurate judgments is not a synonym for having good judgment.

Judgment can therefore be described as measurement in which the instrument is a human mind.

people who make judgments behave as if a true value exists, regardless of whether it does.

Noise is mostly a by-product of our uniqueness, of our “judgment personality.

When Vul and Pashler let three weeks pass before asking their subjects the same question again, the benefit rose to one-third the value of a second opinion.

If people are not making their own judgements and are relying instead on what other people think, crowds might not be so wise after all.

On the other hand, a good mood makes us more likely to accept our first impressions as true without challenging them.

When the facts change, I change my mind. What do you do?”)

Scientists in diverse disciplines were quick to adopt the least squares method. Over two centuries later, it remains the standard way to evaluate errors wherever achieving accuracy is the goal.

Averaging two guesses by the same person does not improve judgments as much as does seeking out an independent second opinion. As Vul and Pashler put it, “You can gain about 1/10th as much from asking yourself the same question twice as you can from getting a second opinion from someone else.” This is not a large improvement. But you can make the effect much larger by waiting to make a second guess.

Words that you have seen before become easier to see again—you can identify them better than other words when they are shown very briefly or masked by noise, and you will be quicker (by a few hundredths of a second) to read them than to read other words. In short, you experience greater cognitive ease in perceiving a word you have seen earlier, and it is this sense of ease that gives you the impression of familiarity.

There is good reason to believe that general intelligence is likely to be associated with better judgment. Intelligence is correlated with good performance in virtually all domains. All other things being equal, it is associated not only with higher academic achievement but also with higher job performance.

Groups can go in all sorts of directions, depending in part on factors that should be irrelevant. Who speaks first, who speaks last, who speaks with confidence, who is wearing black, who is seated next to whom, who smiles or frowns or gestures at the right moment—all these factors, and many more, affect outcomes.

Causal thinking helps us make sense of a world that is far less predictable than we think. It also explains why we view the world as far more predictable than it really is. In the valley of the normal, there are no surprises and no inconsistencies. The future seems as predictable as the past. And noise is neither heard nor seen.

We can live comfortably with colleagues without ever noticing that they actually do not see the world as we do.

When we substitute an easier question for the one we should be answering, errors are bound to occur.

Most of us, most of the time, live with the unquestioned belief that the world looks as it does because that’s the way it is.

objective ignorance accumulates steadily the further you look into the future.

There is essentially no evidence of situations in which people do very poorly and models do very well with the same information.

there is a large amount of objective ignorance in the prediction of human behavior.

why do we all—seem to underestimate our objective ignorance of the world?

the preference for causal thinking also contributes to the neglect of noise as a source of error, because noise is a fundamentally statistical notion.

Errors are bound to occur when a judgment of similarity is substituted for a judgment of probability,

Venn diagrams apply only to probability, not to similarity. Hence the predictable logical error that many people make.

however much we would like to believe that our judgment about a candidate is based on facts, our interpretation of facts is colored by prior attitudes.

the process of understanding reality is backward-looking

In short, doctors are significantly more likely to order cancer screenings early in the morning than late in the afternoon.

The reliance on flawed explanations is perhaps inevitable, if the alternative is to give up on understanding our world.

The invisibility of noise is a direct consequence of causal thinking.

Judgments are both less noisy and less biased when those who make them are well trained, are more intelligent, and have the right cognitive style.

the strong conclusion that simple mechanical rules were generally superior to human judgment

Meehl discovered that clinicians and other professionals are distressingly weak in what they often see as their unique strength: the ability to integrate information.

You may believe that you are subtler, more insightful, and more nuanced than the linear caricature of your thinking. But in fact, you are mostly noisier.

From the perspective of noise reduction, a singular decision is a recurrent decision that happens only once.

System noise is inconsistency, and inconsistency damages the credibility of the system.

causal thinking and the illusion of understanding the past contribute to overconfident predictions of the future.

Level noise is variability in the average level of judgments by different judges. Pattern noise is variability in judges’ responses to particular cases.

For embezzlement actions that were similar to one another, one man was sentenced to 117 days in prison, while another was sentenced to 20 years.

Relying on causal thinking about a single case is a source of predictable errors. Taking the statistical view, which we will also call the outside view, is a way to avoid these errors.

Moral values are constantly evolving. If we lock everything down, we won’t make space for changing values. Some efforts to reduce noise are just too rigid; they would prevent moral change.

As humans, we are keenly aware that we make mistakes, but that is a privilege we are not prepared to share. We expect machines to be perfect. If this expectation is violated, we discard them.

Two men, neither of whom had a criminal record, were convicted for cashing counterfeit checks in the amounts of $58.40 and $35.20, respectively. The first man was sentenced to fifteen years, the second to 30 days.

People often tend to trust and like leaders who are firm and clear and who seem to know, immediately and deep in their bones, what is right. Such leaders inspire confidence. But the evidence suggests that if the goal is to reduce error, it is better for leaders (and others) to remain open to counterarguments and to know that they might be wrong. If they end up being decisive, it is at the end of a process, not at the start.

Personnel decisions are noisy. Interviewers of job candidates make widely different assessments of the same people. Performance ratings of the same employees are also highly variable and depend more on the person doing

MSE has that property—and it is the only definition of overall error that has it. In figure 6, we have computed the value of MSE in the set of five measurements for ten possible integer values of the line’s true length.

if your goal is to bring out the best in people, you can reasonably ask whether measuring individual performance and using that measurement to motivate people through fear and greed is the best approach (or even an effective one).

Matters of judgment, including professional judgments, occupy a space between questions of fact or computation on the one hand and matters of taste or opinion on the other. They are defined by the expectation of bounded disagreement.

Diversity of opinions is essential for generating ideas and options. Contrarian thinking is essential to innovation. A plurality of opinions among movie critics is a feature, not a bug. Disagreements among traders make markets. Strategy differences among competing start-ups enable markets to select the fittest. In what we call matters of judgment, however, system noise is always a problem. If two doctors give you different diagnoses, at least one of them is wrong.

Creative people need space. People aren’t robots. Whatever your job, you deserve some room to maneuver. If you’re hemmed in, you might not be noisy, but you won’t have much fun and you won’t be able to bring your original ideas to bear.

Every large branch of the company has several qualified underwriters. When a quote is requested, anyone who happens to be available may be assigned to prepare it. In effect, the particular underwriter who will determine a quote is selected by a lottery.

Enforcing decision hygiene can be thankless. Noise is an invisible enemy, and a victory against an invisible enemy can only be an invisible victory. But like physical health hygiene, decision hygiene is vital. After a successful operation, you like to believe that it is the surgeon’s skill that saved your life—and it did, of course—but if the surgeon and all the personnel in the operating room had not washed their hands, you might be dead. There may not be much glory to be gained in hygiene, but there are results.

To be actively open-minded is to actively search for information that contradicts your preexisting hypotheses. Such information includes the dissenting opinions of others and the careful weighing of new evidence against old beliefs. Actively openminded people agree with statements like this: “Allowing oneself to be convinced by an opposing argument is a sign of good character.” They disagree with the proposition that “changing your mind is a sign of weakness” or that “intuition is the best guide in making decisions.

Level noise is when judges show different levels of severity. Pattern noise is when they disagree with one another on which defendants deserve more severe or more lenient treatment. And part of pattern noise is occasion noise—when judges disagree with themselves.

Speaking of Singular Decisions “The way you approach this unusual opportunity exposes you to noise.” “Remember: a singular decision is a recurrent decision that is made only once.” “The personal experiences that made you who you are are not truly relevant to this decision.

There is reason to believe that some people make better judgments than others do. Task-specific skill, intelligence, and a certain cognitive style—best described as being actively open-minded—characterize the best judges. Unsurprisingly, good judges will make few egregious mistakes. Given the multiple sources of individual differences, however, we should not expect even the best judges to be in perfect agreement on complex judgment problems. The infinite variety of backgrounds, personalities, and experiences that make each of us unique is also what makes noise inevitable.

In theory, a judgment of risk should be based on a long-term average. In reality, recent incidents are given more weight because they come more easily to mind. Substituting a judgment of how easily examples come to mind for an assessment of frequency is known as the availability heuristic.

We have described these studies of mood in some detail because we need to emphasize an important truth: you are not the same person at all times. As your mood varies (something you are, of course, aware of), some features of your cognitive machinery vary with it (something you are not fully aware of).

You may have noticed that the decomposition of system noise into level noise and pattern noise follows the same logic as the error equation in the previous chapter, which decomposed error into bias and noise. This time, the equation can be written as follows: System Noise2 = Level Noise2 + Pattern Noise2

More often than not, in fact, outstanding performance will become less outstanding. Conversely, very poor performance will improve. It is easy to imagine social, psychological, or even political reasons for this observation, but reasons are not required. The phenomenon is purely statistical. Extreme observations in one direction or the other will tend to become less extreme, simply because past performance is not perfectly correlated with future performance. This tendency is called regression to the mean (hence the technical term nonregressive for matching predictions, which fail to take it into account).

Another formal process for aggregating diverse views is known as the Delphi method. In its classic form, this method involves multiple rounds during which the participants submit estimates (or votes) to a moderator and remain anonymous to one another. At each new round, the participants provide reasons for their estimates and respond to the reasons given by others, still anonymously. The process encourages estimates to converge (and sometimes forces them to do so by requiring new judgments to fall within a specific range of the distribution of previous-round judgments). The method benefits both from aggregation and social learning.

Despite all the evidence in favor of mechanical and algorithmic prediction methods, and despite the rational calculus that clearly shows the value of incremental improvements in predictive accuracy, many decision makers will reject decision-making approaches that deprive them of the ability to exercise their intuition.

adding complexity to the decision-making processes of an organization that is already bureaucratic will not make things better. But decision hygiene need not be slow and certainly doesn’t need to be bureaucratic. On the contrary, it promotes challenge and debate, not the stifling consensus that characterizes bureaucracies.

In the words of one observer, “the reliance on the patient’s subjective symptoms, the clinician’s interpretation of the symptoms, and the absence of objective measure (such as a blood test) implant the seeds of diagnostic unreliability of psychiatric disorders.” In this sense, psychiatry may prove especially resistant to attempts at noise reduction.

As we noted in chapter 12, our normal way of thinking is causal. We naturally attend to the particular, following and creating causally coherent stories about individual cases, in which failures are often attributed to errors, and errors to biases. The ease with which bad judgments can be explained leaves no space for noise in our accounts of errors.

Both intractable uncertainty (what cannot possibly be known) and imperfect information (what could be known but isn’t) make perfect prediction impossible. These unknowns are not problems of bias or noise in your judgment; they are objective characteristics of the task. This objective ignorance of important unknowns severely limits achievable accuracy.

predictive judgments that provide input—for instance, how a candidate will perform in her first year, how the stock market will respond to a given strategic move, or how quickly the epidemic will spread if left unchecked. But the final decisions entail trade-offs between the pros and cons of various options, and these trade-offs are resolved by evaluative judgments.

Content is specific; process is generic. Using intuition and judgment is fun; following process is not. Conventional wisdom holds that good decisions—especially the very best ones—emerge from the insight and creativity of great leaders. (We especially like to believe this when we are the leader in question.) And to many, the word process evokes bureaucracy, red tape, and delays.

mood has a measurable influence on what you think: what you notice in your environment, what you retrieve from your memory, how you make sense of these signals. But mood has another, more surprising effect: it also changes how you think. And here, the effects are not those you might imagine. Being in a good mood is a mixed blessing, and bad moods have a silver lining. The costs and benefits

There is a limit to the accuracy of our predictions, and this limit is often quite low. Nevertheless, we are generally comfortable with our judgments. What gives us this satisfying confidence is an internal signal, a self-generated reward for fitting the facts and the judgment into a coherent story. Our subjective confidence in our judgments is not necessarily related to their objective accuracy.

group deliberation often adds more error in bias than it removes in noise. Organizations that want to harness the power of diversity must welcome the disagreements that will arise when team members reach their judgments independently. Eliciting and aggregating judgments that are both independent and diverse will often be the easiest, cheapest, and most broadly applicable decision hygiene strategy.

If you are unfamiliar with research on the employment interview, what follows may surprise you. In essence, if your goal is to determine which candidates will succeed in a job and which will fail, standard interviews (also called unstructured interviews to distinguish them from structured interviews, to which we will turn shortly) are not very informative. To put it more starkly, they are often useless.

you can safely expect that people who engage in predictive tasks will underestimate their objective ignorance. Overconfidence is one of the best-documented cognitive biases. In particular, judgments of one’s ability to make precise predictions, even from limited information, are notoriously overconfident. What we said of noise in predictive judgments can also be said of objective ignorance: wherever there is prediction, there is ignorance, and more of it than you think.

But noisy systems do not make multiple judgments of the same case. They make noisy judgments of different cases. If one insurance policy is overpriced and another is underpriced, pricing may on average look right, but the insurance company has made two costly errors. If two felons who both should be sentenced to five years in prison receive sentences of three years and seven years, justice has not, on average, been done. In noisy systems, errors do not cancel out. They add up.

The Delphi method has worked well in many situations, but it can be challenging to implement. A simpler version, mini-Delphi, can be deployed within a single meeting. Also called estimate-talk-estimate, it requires participants first to produce separate (and silent) estimates, then to explain and justify them, and finally to make a new estimate in response to the estimates and explanations of others. The consensus judgment is the average of the individual estimates obtained in that second round.

This request required the subjects to think actively of information they had not considered the first time. The instructions to participants read as follows: First, assume that your first estimate is off the mark. Second, think about a few reasons why that could be. Which assumptions and considerations could have been wrong? Third, what do these new considerations imply? Was the first estimate rather too high or too low? Fourth, based on this new perspective, make a second, alternative estimate.

personal values, individuality, and creativity are needed, even essential, in many phases of thinking and decision making, including the choice of goals, the formulation of novel ways to approach a problem, and the generation of options. But when it comes to making a judgment about these options, expressions of individuality are a source of noise. When the goal is accuracy and you expect others to agree with you, you should also consider what other competent judges would think if they were in your place.

A study of thousands of juvenile court decisions found that when the local football team loses a game on the weekend, the judges make harsher decisions on the Monday (and, to a lesser extent, for the rest of the week). Black defendants disproportionately bear the brunt of that increased harshness. A different study looked at 1.5 million judicial decisions over three decades and similarly found that judges are more severe on days that follow a loss by the local city’s football team than they are on days that follow a win.

In an ideal world, evaluating people’s performance would not be a judgment task; objective facts would be sufficient to determine how well people are doing. But most modern organizations have little in common with Adam Smith’s pin factory, in which every worker had a measurable output. What would that output be for a chief financial officer or for a head of research? Today’s knowledge workers balance multiple, sometimes contradictory objectives. Focusing on only one of them might produce erroneous evaluations and have harmful incentive effects.

The large subject of performance evaluation raises many questions, both practical and philosophical. Some people ask, for instance, to what extent the notion of individual performance is meaningful in today’s organizations, where outcomes often depend on how people interact with one another. If we believe the notion is indeed meaningful, we must wonder how levels of individual performance are distributed among people in a given organization—for instance, whether performance follows a normal distribution or whether there exists “star talent” making a hugely disproportionate contribution.

The exact value of the quote has significant consequences for the company. A high premium is advantageous if the quote is accepted, but such a premium risks losing the business to a competitor. A low premium is more likely to be accepted, but it is less advantageous to the company. For any risk, there is a Goldilocks price that is just right—neither too high nor too low—and there is a good chance that the average judgment of a large group of professionals is not too far from this Goldilocks number. Prices that are higher or lower than this number are costly—this is how the variability of noisy judgments hurts the bottom line.

Herzog and Hertwig then averaged the two estimates thus produced. Their technique, which they named dialectical bootstrapping, produced larger improvements in accuracy than did a simple request for a second estimate immediately following the first. Because the participants forced themselves to consider the question in a new light, they sampled another, more different version of themselves—two “members” of the “crowd within” who were further apart. As a result, their average produced a more accurate estimate of the truth. The gain in accuracy with two immediately consecutive “dialectical” estimates was about half the value of a second opinion.

Most of us, most of the time, live with the unquestioned belief that the world looks as it does because that’s the way it is. There is one small step from this belief to another: “Other people view the world much the way I do.” These beliefs, which have been called naive realism, are essential to the sense of a reality we share with other people. We rarely question these beliefs. We hold a single interpretation of the world around us at any one time, and we normally invest little effort in generating plausible alternatives to it. One interpretation is enough, and we experience it as true. We do not go through life imagining alternative ways of seeing what we see.

You can reasonably assume that overconfidence affects project teams in general, but you cannot be sure that it is the only bias (or even the main one) affecting a particular project team. Maybe the team leader has had a bad experience with a similar project and so has learned to be especially conservative when making estimates. The team thus exhibits the opposite error from the one you thought you should correct. Or perhaps the team developed its forecast by analogy with another similar project and was anchored on the time it took to complete that project. Or maybe the project team, anticipating that you would add a buffer to its estimate, has preempted your adjustment by making its recommendation even more bullish than its true belief.

To experience how noise and bias contribute to error, we invite you to play a game that will take you less than one minute. If you have a smartphone with a stopwatch, it probably has a lap function, which enables you to measure consecutive time intervals without stopping the stopwatch or even looking at the display. Your goal is to produce five consecutive laps of exactly ten seconds without looking at the phone. You may want to observe a ten-second interval a few times before you begin. Go. Now look at the lap durations recorded on your phone. (The phone itself was not free from noise, but there was very little of it.) You will see that the laps are not all exactly ten seconds and that they vary over a substantial range. You tried to reproduce the same timing exactly, but you were unable to do so. The variability you could not control is an instance of noise.

Fifty judges from various districts were asked to set sentences for defendants in hypothetical cases summarized in identical pre-sentence reports. The basic finding was that “absence of consensus was the norm” and that the variations across punishments were “astounding.” A heroin dealer could be incarcerated for one to ten years, depending on the judge. Punishments for a bank robber ranged from five to eighteen years in prison. The study found that in an extortion case, sentences varied from a whopping twenty years imprisonment and a $65,000 fine to a mere three years imprisonment and no fine. Most startling of all, in sixteen of twenty cases, there was no unanimity on whether any incarceration was appropriate. This study was followed by a series of others, all of which found similarly shocking levels of noise. In 1977, for example, William Austin and Thomas Williams conducted a survey of forty-seven judges, asking them to respond to the same five cases, each involving low-level offenses. All the descriptions of the cases included summaries of the information used by judges in actual sentencing, such as the charge, the testimony, the previous criminal record (if any), social background, and evidence relating to character. The key finding was “substantial disparity.” In a case involving burglary, for example, the recommended sentences ranged from five years in prison to a mere thirty days (alongside a fine of $100). In a case involving possession of marijuana, some judges recommended prison terms; others recommended probation.