Bayes Theorem Quotes

Under Bayes' theorem, no theory is perfect. Rather, it is a work in progress, always subject to further refinement and testing.

A learner that uses Bayes’ theorem and assumes the effects are independent given the cause is called a Naïve Bayes classifier. That’s because, well, that’s such a naïve assumption.

In science, progress is possible. In fact, if one believes in Bayes' theorem, scientific progress is inevitable as predictions are made and as beliefs are tested and refined.

Bayes’s theorem and that looks like this: People who understand Bayes’s theorem can use it to work out complex problems involving probability distributions—or inverse probabilities, as they are sometimes called.

For a Bayesian, in fact, there is no such thing as the truth; you have a prior distribution over hypotheses, after seeing the data it becomes the posterior distribution, as given by Bayes’ theorem, and that’s all.

If you hold there is a 100 percent probability that God exists, or a 0 percent probability, then under Bayes’s theorem, no amount of evidence could persuade you otherwise.

Bayes’s theorem requires us to state—explicitly—how likely we believe an event is to occur before we begin to weigh the evidence. It calls this estimate a prior belief.

Bayes’s theorem deals with epistemological uncertainty—the limits of our knowledge.

Bayes’s theorem predicts that the Bayesians will win.

Try not to get overwhelmed or lost in the question. Always begin by writing down what you want to discover.

The combination of Bayes and Markov Chain Monte Carlo has been called "arguably the most powerful mechanism ever created for processing data and knowledge." Almost instantaneously MCMC and Gibbs sampling changed statisticians' entire method of attacking problems. In the words of Thomas Kuhn, it was a paradigm shift. MCMC solved real problems, used computer algorithms instead of theorems, and led statisticians and scientists into a worked where "exact" meant "simulated" and repetitive computer operations replaced mathematical equations. It was a quantum leap in statistics.

What isn’t acceptable under Bayes’s theorem is to pretend that you don’t have any prior beliefs. You should work to reduce your biases, but to say you have none is a sign that you have many. To state your beliefs up front—to say “Here’s where I’m coming from”12—is a way to operate in good faith and to recognize that you perceive reality through a subjective filter.

prior probability that the sun will rise, since it’s prior to seeing any evidence. It’s not based on counting the number of times the sun has risen on this planet in the past, because you weren’t there to see it; rather, it reflects your a priori beliefs about what will happen, based on your general knowledge of the universe. But now the stars start to fade, so your confidence that the sun does rise on this planet goes up, based on your experience on Earth. Your confidence is now a posterior probability, since it’s after seeing some evidence. The sky begins to lighten, and the posterior probability takes another leap. Finally, a sliver of the sun’s bright disk appears above the horizon and perhaps catches “the Sultan’s turret in a noose of light,” as in the opening verse of the Rubaiyat. Unless you’re hallucinating, it is now certain that the sun will rise. The crucial question is exactly how the posterior probability should evolve as you see more evidence. The answer is Bayes’ theorem. We can think of it in terms of cause and effect. Sunrise causes the stars to fade and the sky to lighten, but the latter is stronger evidence of daybreak, since the stars could fade in the middle of the night due to, say, fog rolling in. So the probability of sunrise should increase more after seeing the sky lighten than after seeing the stars fade. In mathematical notation, we say that P(sunrise | lightening-sky), the conditional probability of sunrise given that the sky is lightening, is greater than P(sunrise | fading-stars), its conditional probability given that the stars are fading. According to Bayes’ theorem, the more likely the effect is given the cause, the more likely the cause is given the effect: if P(lightening-sky | sunrise) is higher than P(fading-stars | sunrise), perhaps because some planets are far enough from their sun that the stars still shine after sunrise, then P(sunrise | lightening sky) is also higher than P(sunrise | fading-stars).

Imagine you're sitting having dinner in a restaurant. At some point during the meal, your companion leans over and whispers that they've spotted Lady Gaga eating at the table opposite. Before having a look for yourself, you'll no doubt have some sense of how much you believe your friends theory. You'll take into account all of your prior knowledge: perhaps the quality of the establishment, the distance you are from Gaga's home in Malibu, your friend's eyesight. That sort of thing. If pushed, it's a belief that you could put a number on. A probability of sorts. As you turn to look at the woman, you'll automatically use each piece of evidence in front of you to update your belief in your friend's hypothesis Perhaps the platinum-blonde hair is consistent with what you would expect from Gaga, so your belief goes up. But the fact that she's sitting on her own with no bodyguards isn't, so your belief goes down. The point is, each new observations adds to your overall assessment. This is all Bayes' theorem does: offers a systematic way to update your belief in a hypothesis on the basis of the evidence. It accepts that you can't ever be completely certain about the theory you are considering, but allows you to make a best guess from the information available. So, once you realize the woman at the table opposite is wearing a dress made of meat -- a fashion choice that you're unlikely to chance up on in the non-Gaga population -- that might be enough to tip your belief over the threshold and lead you to conclude that it is indeed Lady Gaga in the restaurant. But Bayes' theorem isn't just an equation for the way humans already make decisions. It's much more important that that. To quote Sharon Bertsch McGrayne, author of The Theory That Would Not Die: 'Bayes runs counter to the deeply held conviction that modern science requires objectivity and precision. By providing a mechanism to measure your belief in something, Bayes allows you to draw sensible conclusions from sketchy observations, from messy, incomplete and approximate data -- even from ignorance.

Perhaps the most striking illustration of Bayes’s theorem comes from a riddle that a mathematics teacher that I knew would pose to his students on the first day of their class. Suppose, he would ask, you go to a roadside fair and meet a man tossing coins. The first toss lands “heads.” So does the second. And the third, fourth . . . and so forth, for twelve straight tosses. What are the chances that the next toss will land “heads” ? Most of the students in the class, trained in standard statistics and probability, would nod knowingly and say: 50 percent. But even a child knows the real answer: it’s the coin that is rigged. Pure statistical reasoning cannot tell you the answer to the question—but common sense does. The fact that the coin has landed “heads” twelve times tells you more about its future chances of landing “heads” than any abstract formula. If you fail to use prior information, you will inevitably make foolish judgments about the future. This is the way we intuit the world, Bayes argued. There is no absolute knowledge; there is only conditional knowledge. History repeats itself—and so do statistical patterns. The past is the best guide to the future.

The main ones are the symbolists, connectionists, evolutionaries, Bayesians, and analogizers. Each tribe has a set of core beliefs, and a particular problem that it cares most about. It has found a solution to that problem, based on ideas from its allied fields of science, and it has a master algorithm that embodies it. For symbolists, all intelligence can be reduced to manipulating symbols, in the same way that a mathematician solves equations by replacing expressions by other expressions. Symbolists understand that you can’t learn from scratch: you need some initial knowledge to go with the data. They’ve figured out how to incorporate preexisting knowledge into learning, and how to combine different pieces of knowledge on the fly in order to solve new problems. Their master algorithm is inverse deduction, which figures out what knowledge is missing in order to make a deduction go through, and then makes it as general as possible. For connectionists, learning is what the brain does, and so what we need to do is reverse engineer it. The brain learns by adjusting the strengths of connections between neurons, and the crucial problem is figuring out which connections are to blame for which errors and changing them accordingly. The connectionists’ master algorithm is backpropagation, which compares a system’s output with the desired one and then successively changes the connections in layer after layer of neurons so as to bring the output closer to what it should be. Evolutionaries believe that the mother of all learning is natural selection. If it made us, it can make anything, and all we need to do is simulate it on the computer. The key problem that evolutionaries solve is learning structure: not just adjusting parameters, like backpropagation does, but creating the brain that those adjustments can then fine-tune. The evolutionaries’ master algorithm is genetic programming, which mates and evolves computer programs in the same way that nature mates and evolves organisms. Bayesians are concerned above all with uncertainty. All learned knowledge is uncertain, and learning itself is a form of uncertain inference. The problem then becomes how to deal with noisy, incomplete, and even contradictory information without falling apart. The solution is probabilistic inference, and the master algorithm is Bayes’ theorem and its derivates. Bayes’ theorem tells us how to incorporate new evidence into our beliefs, and probabilistic inference algorithms do that as efficiently as possible. For analogizers, the key to learning is recognizing similarities between situations and thereby inferring other similarities. If two patients have similar symptoms, perhaps they have the same disease. The key problem is judging how similar two things are. The analogizers’ master algorithm is the support vector machine, which figures out which experiences to remember and how to combine them to make new predictions.

It is easy to appreciate the theological import of this line of reasoning. Standard probability theory asks us to predict consequences from abstract knowledge: Knowing God’s vision, what can you predict about Man? But Bayes’s theorem takes the more pragmatic and humble approach to inference. It is based on real, observable knowledge: Knowing Man’s world, Bayes asks, what can you guess about the mind of God?

Bayes’s Theorem is one of those insights that can change the way we go through life. Each of us comes equipped with a rich variety of beliefs, for or against all sorts of propositions. Bayes teaches us (1) never to assign perfect certainty to any such belief; (2) always to be prepared to update our credences when new evidence comes along; and (3) how exactly such evidence alters the credences we assign. It’s a road map for coming closer and closer to the truth.

This is all Bayes' theorem does: offers a systematic way to update your belief in a hypothesis on the basis of the evidence. It accepts that you can't ever be completely certain about the theory you're considering, but allows you to make a best guess from the information available

This is all Bayes’ theorem does: offers a systematic way to update your belief in a hypothesis on the basis of the evidence.30 It accepts that you can’t ever be completely certain about the theory you’re considering, but allows you to make a best guess from the information available.

They point out that we never know for sure which hypothesis is the true one, and so we shouldn’t just pick one hypothesis, like a value of 0.7 for the probability of heads; rather, we should compute the posterior probability of every possible hypothesis and entertain all of them when making predictions. The sum of the probabilities of all the hypotheses must be one, so if one becomes more likely, the others become less. For a Bayesian, in fact, there is no such thing as the truth; you have a prior distribution over hypotheses, after seeing the data it becomes the posterior distribution, as given by Bayes’ theorem, and that’s all.

The probability that Monty opens door B if the prize is behind door B is: P(Monty opens B|B) = 0. The probability that Monty opens door B if the prize is behind door C is: P(Monty opens B|C) = 1. We can now calculate the probability that Monty opens door B: P(A) × P(Monty opens B|A) + P(B) × P(Monty opens B|B) + P(C) × P(Monty opens B|C) = . Finally, apply Bayes theorem: P(A|Monty opens B) = . P(C|Monty opens B) = . In plain English: if you happen to choose door A and Monty opens door B to reveal a lemon then the probability of the Bugatti being behind door C is . If you ever find yourself in this

The Bayesian Invisible Hand … free-market capitalism and Bayes’ theorem come out of something of the same intellectual tradition. Adam Smith and Thomas Bayes were contemporaries, and both were educated in Scotland and were heavily influenced by the philosopher David Hume. Smith’s 'Invisible hand' might be thought of as a Bayesian process, in which prices are gradually updated in response to changes in supply and demand, eventually reaching some equilibrium. Or, Bayesian reasoning might be thought of as an 'invisible hand' wherein we gradually update and improve our beliefs as we debate our ideas, sometimes placing bets on them when we can’t agree. Both are consensus-seeking processes that take advantage of the wisdom of crowds. It might follow, then, that markets are an especially good way to make predictions. That’s really what the stock market is: a series of predictions about the future earnings and dividends of a company. My view is that this notion is 'mostly' right 'most' of the time. I advocate the use of betting markets for forecasting economic variables like GDP, for instance. One might expect these markets to improve predictions for the simple reason that they force us to put our money where our mouth is, and create an incentive for our forecasts to be accurate. Another viewpoint, the efficient-market hypothesis, makes this point much more forcefully: it holds that it is 'impossible' under certain conditions to outpredict markets. This view, which was the orthodoxy in economics departments for several decades, has become unpopular given the recent bubbles and busts in the market, some of which seemed predictable after the fact. But, the theory is more robust than you might think. And yet, a central premise of this book is that we must accept the fallibility of our judgment if we want to come to more accurate predictions. To the extent that markets are reflections of our collective judgment, they are fallible too. In fact, a market that makes perfect predictions is a logical impossibility.

The main objective of Bayes’ Theorem is to convert test probabilities into real probabilities. The theorem offers a holistic approach in weighing probabilities; that means it considers true positive, true negative, false positive and false negative results in a test.  With such approach, you can find out and interpret the real probabilities for an event.

Remember to take your time. There is no need to rush.

At heart, Bayes’ theorem is just a simple rule for updating your degree of belief in a hypothesis when you receive new evidence: if the evidence is consistent with the hypothesis, the probability of the hypothesis goes up; if not, it goes down. For example, if you test positive for AIDS, your probability of having it goes up. Things get more interesting when you have many pieces of evidence, such as the results of multiple tests. To combine them all without suffering a combinatorial explosion, we need to make simplifying assumptions. Things get even more interesting when we consider many hypotheses at once, such as all the different possible diagnoses for a patient. Computing the probability of each disease from the patient’s symptoms in a reasonable amount of time can take a lot of smarts. Once we know how to do all these things, we’ll be ready to learn the Bayesian way. For Bayesians, learning is “just” another application of Bayes’ theorem, with whole models as the hypotheses and the data as the evidence: as you see more data, some models become more likely and some less, until ideally one model stands out as the clear winner.

Bayesianism as we know it was invented by Pierre-Simon de Laplace, a Frenchman who was born five decades after Bayes. Bayes was the preacher who first described a new way to think about chance, but it was Laplace who codified those insights into the theorem that bears Bayes’s name.

Bayes’ theorem says that P(cause | effect) = P(cause) × P(effect | cause) / P(effect). Replace cause by A and effect by B and omit the multiplication sign for brevity, and you get the ten-foot formula in the cathedral.

Bayes’ theorem is useful because what we usually know is the probability of the effects given the causes, but what we want to know is the probability of the causes given the effects. For example, we know what percentage of flu patients have a fever, but what we really want to know is how likely a patient with a fever is to have the flu. Bayes’ theorem lets us go from one to the other. Its significance extends far beyond that, however. For Bayesians, this innocent-looking formula is the F = ma of machine learning, the foundation from which a vast number of results and applications flow. And whatever the Master Algorithm is, it must be “just” a computational implementation of Bayes’ theorem. I put just in quotes because implementing Bayes’ theorem on a computer turns out to be fiendishly hard for all but the simplest problems, for reasons that we’re about to see.

In reality, a doctor doesn’t diagnose the flu just based on whether you have a fever; she takes a whole bunch of symptoms into account, including whether you have a cough, a sore throat, a runny nose, a headache, chills, and so on. So what we really need to compute is P(flu | fever, cough, sore throat, runny nose, headache, chills, … ). By Bayes’ theorem, we know that this is proportional to P(fever, cough, sore throat, runny nose, headache, chills, …| flu). But now we run into a problem. How are we supposed to estimate this probability? If each symptom is a Boolean variable (you either have it or you don’t) and the doctor takes n symptoms into account, a patient could have 2n possible combinations of symptoms.

This is where Bayes' Theorem comes in and helps us have a clearer picture. By using the theorem, we are forced to look at all data and update our hypothesis with new evidence.

Now, remember what Bayes' Theorem does: it helps us update a hypothesis based on new evidence.

Bayes' Theorem does: it helps us update a hypothesis based on new evidence.

To begin solving this problem, we always need to determine what we are wanting to find.

An Intuitive Explanation of Bayes’s Theorem