Null Hypothesis Quotes

we don’t prove a hypothesis is true; we search for evidence to disprove a null hypothesis.

The power of a study typically should be set at 80% or greater, so if there is truly a difference in treatments, the chances are 80% or greater that your research project will identify this fact. Power = 1 - beta. Beta is the probability of making a Type II error (accepting the null hypothesis when in fact it is incorrect).

When the CERN teams reported a 'five-sigma' result for the Higgs boson, corresponding to a P-value of around 1 in 3.5 million, the BBC reported the conclusion correctly, saying this meant 'about a one-on-3.5 million chance that the signal they see would appear if there were no Higgs particle.' But nearly every other outlet got the meaning of this P-value wrong. For example, Forbes Magazine reported, 'The chances are less than 1 in a million that it is not the Higgs boson,' a clear example of the prosecutor's fallacy. The Independent was typical in claiming that 'there is less than a one in a million chance that their results are a statistical fluke.' This may not be blatantly mistaken as Forbes, but it is still assigning the small probability to 'their results are a statistical fluke', which is logically the same as saying this is the probability of the null hypothesis being tested.

First, because, in the first case, the right of conquest being in fact no right at all, it could not serve as a foundation for any other right, the conqueror and the conquered ever remaining with respect to each other in a state of war, unless the conquered, restored to the full possession of their liberty, should freely choose their conqueror for their chief. Till then, whatever capitulations might have been made between them, as these capitulations were founded upon violence, and of course de facto null and void, there could not have existed in this hypothesis either a true society, or a political body, or any other law but that of the strongest. Second, because these words strong and weak, are ambiguous in the second case; for during the interval between the establishment of the right of property or prior occupation and that of political government, the meaning of these terms is better expressed by the words poor and rich, as before the establishment of laws men in reality had no other means of reducing their equals, but by invading the property of these equals, or by parting with some of their own property to them. Third, because the poor having nothing but their liberty to lose, it would have been the height of madness in them to give up willingly the only blessing they had left without obtaining some consideration for it: whereas the rich being sensible, if I may say so, in every part of their possessions, it was much easier to do them mischief, and therefore more incumbent upon them to guard against it; and because, in fine, it is but reasonable to suppose, that a thing has been invented by him to whom it could be of service rather than by him to whom it must prove detrimental.

Almost all official statistics and policy documents on wages, income, gross domestic product (GDP), crime, unemployment rates, innovation rates, cost of living indices, morbidity and mortality rates, and poverty rates are compiled by governmental agencies and international bodies worldwide in terms of both total aggregate and per capita metrics. Furthermore, well-known composite indices of urban performance and the quality of life, such as those assembled by the World Economic Forum and magazines like Fortune, Forbes, and The Economist, primarily rely on naive linear combinations of such measures.6 Because we have quantitative scaling curves for many of these urban characteristics and a theoretical framework for their underlying dynamics we can do much better in devising a scientific basis for assessing performance and ranking cities. The ubiquitous use of per capita indicators for ranking and comparing cities is particularly egregious because it implicitly assumes that the baseline, or null hypothesis, for any urban characteristic is that it scales linearly with population size. In other words, it presumes that an idealized city is just the linear sum of the activities of all of its citizens, thereby ignoring its most essential feature and the very point of its existence, namely, that it is a collective emergent agglomeration resulting from nonlinear social and organizational interactions. Cities are quintessentially complex adaptive systems and, as such, are significantly more than just the simple linear sum of their individual components and constituents, whether buildings, roads, people, or money. This is expressed by the superlinear scaling laws whose exponents are 1.15 rather than 1.00. This approximately 15 percent increase in all socioeconomic activity with every doubling of the population size happens almost independently of administrators, politicians, planners, history, geographical location, and culture.

My hypothesis is mimetic: because humans imitate one another more than animals, they have had to find a means of dealing with contagious similarity, which could lead to the pure and simple disappearance of their society. The mechanism that reintroduces difference into a situation in which everyone has come to resemble everyone else is sacrifice. Humanity results from sacrifice; we are thus the children of religion. What I call after Freud the founding murder, in other words, the immolation of a sacrificial victim that is both guilty of disorder and able to restore order, is constantly re-enacted in the rituals at the origin of our institutions. Since the dawn of humanity, millions of innocent victims have been killed in this way in order to enable their fellow humans to live together, or at least not to destroy one another. This is the implacable logic of the sacred, which myths dissimulate less and less as humans become increasingly self-aware. The decisive point in this evolution is Christian revelation, a kind of divine expiation in which God through his Son could be seen as asking for forgiveness from humans for having revealed the mechanisms of their violence so late. Rituals had slowly educated them; from then on, humans had to do without. Christianity demystifies religion. Demystification, which is good in the absolute, has proven bad in the relative, for we were not prepared to shoulder its consequences. We are not Christian enough. The paradox can be put a different way. Christianity is the only religion that has foreseen its own failure. This prescience is known as the apocalypse. Indeed, it is in the apocalyptic texts that the word of God is most forceful, repudiating mistakes that are entirely the fault of humans, who are less and less inclined to acknowledge the mechanisms of their violence. The longer we persist in our error, the stronger God’s voice will emerge from the devastation. […] The Passion unveiled the sacrificial origin of humanity once and for all. It dismantled the sacred and revealed its violence. […] By accepting crucifixion, Christ brought to light what had been ‘hidden since the foundation of the world,’ in other words, the foundation itself, the unanimous murder that appeared in broad daylight for the first time on the cross. In order to function, archaic religions need to hide their founding murder, which was being repeated continually in ritual sacrifices, thereby protecting human societies from their own violence. By revealing the founding murder, Christianity destroyed the ignorance and superstition that are indispensable to such religions. It thus made possible an advance in knowledge that was until then unimaginable. […] A scapegoat remains effective as long as we believe in its guilt. Having a scapegoat means not knowing that we have one. Learning that we have a scapegoat is to lose it forever and to expose ourselves to mimetic conflicts with no possible resolution. This is the implacable law of the escalation to extremes. The protective system of scapegoats is finally destroyed by the Crucifixion narratives as they reveal Jesus’ innocence, and, little by little, that of all analogous victims. The process of education away from violent sacrifice is thus underway, but it is going very slowly, making advances that are almost always unconscious. […] Mimetic theory does not seek to demonstrate that myth is null, but to shed light on the fundamental discontinuity and continuity between the passion and archaic religion. Christ’s divinity which precedes the Crucifixion introduces a radical rupture with the archaic, but Christ’s resurrection is in complete continuity with all forms of religion that preceded it. The way out of archaic religion comes at this price. A good theory about humanity must be based on a good theory about God. […] We can all participate in the divinity of Christ so long as we renounce our own violence.

understanding of this formula, it is useful to relate it to the discussion of hypothesis testing in Chapter 10. First, note that the difference of means, appears in the numerator: the larger the difference of means, the larger the t-test test statistic, and the more likely we might reject the null hypothesis. Second, sp is the pooled variance of the two groups, that is, the weighted average of the variances of each group.3 Increases in the standard deviation decrease the test statistic. Thus, it is easier to reject the null hypotheses when two populations are clustered narrowly around their means than when they are spread widely around them. Finally, more observations (that is, increased information or larger n1 and n2) increase the size of the test statistic, making it easier to reject the null hypothesis. Figure 12.1 The T-Test: Mean Incomes by Gender

The null hypothesis of normality is that the variable is normally distributed: thus, we do not want to reject the null hypothesis. A problem with statistical tests of normality is that they are very sensitive to small samples and minor deviations from normality. The extreme sensitivity of these tests implies the following: whereas failure to reject the null hypo thesis indicates normal distribution of a variable, rejecting the null hypothesis does not indicate that the variable is not normally distributed. It is acceptable to consider variables as being normally distributed when they visually appear to be so, even when the null hypothesis of normality is rejected by normality tests. Of course, variables are preferred that are supported by both visual inspection and normality tests. In Greater Depth … Box 12.1 Why Normality? The reasons for the normality assumption are twofold: First, the features of the normal distribution are well-established and are used in many parametric tests for making inferences and hypothesis testing. Second, probability theory suggests that random samples will often be normally distributed, and that the means of these samples can be used as estimates of population means. The latter reason is informed by the central limit theorem, which states that an infinite number of relatively large samples will be normally distributed, regardless of the distribution of the population. An infinite number of samples is also called a sampling distribution. The central limit theorem is usually illustrated as follows. Assume that we know the population distribution, which has only six data elements with the following values: 1, 2, 3, 4, 5, or 6. Next, we write each of these six numbers on a separate sheet of paper, and draw repeated samples of three numbers each (that is, n = 3). We

T-TESTS FOR INDEPENDENT SAMPLES T-tests are used to test whether the means of a continuous variable differ across two different groups. For example, do men and women differ in their levels of income, when measured as a continuous variable? Does crime vary between two parts of town? Do rich people live longer than poor people? Do high-performing students commit fewer acts of violence than do low-performing students? The t-test approach is shown graphically in Figure 12.1, which illustrates the incomes of men and women as boxplots (the lines in the middle of the boxes indicate the means rather than the medians).2 When the two groups are independent samples, the t-test is called the independent-samples t-test. Sometimes the continuous variable is called a “test variable” and the dichotomous variable is called a “grouping variable.” The t-test tests whether the difference of the means is significantly different from zero, that is, whether men and women have different incomes. The following hypotheses are posited: Key Point The independent-samples t-test is used when one variable is dichotomous and the other is continuous. H0: Men and women do not have different mean incomes (in the population). HA: Men and women do have different mean incomes (in the population). Alternatively, using the Greek letter m to refer to differences in the population, H0: μm = μf, and HA: μm ≠ μf. The formula for calculating the t-test test statistic (a tongue twister?) is As always, the computer calculates the test statistic and reports at what level it is significant. Such calculations are seldom done by hand. To further conceptual understanding of this formula, it is useful to relate it to the discussion of hypothesis testing in Chapter 10. First, note that the difference of means, appears in the numerator: the larger the difference of means, the larger the t-test test statistic, and the more likely we might reject the null hypothesis. Second, sp is the pooled variance of the two groups, that is, the weighted average of the variances of each group.3 Increases in the standard deviation decrease the test statistic. Thus, it is easier to reject the null hypotheses when two populations are clustered narrowly around their means than when they are spread widely around them. Finally, more observations (that is, increased information or larger n1 and n2) increase the size of the test statistic, making it easier to reject the null hypothesis. Figure 12.1 The T-Test: Mean Incomes by Gender

safety at the beginning of the program was 4.40 (standard deviation, SD = 1.00), and one year later, 4.80 (SD = 0.94). The mean safety score increased among 10th graders, but is the increase statistically significant? Among other concerns is that the standard deviations are considerable for both samples. As part of the analysis, we conduct a t-test to answer the question of whether the means of these two distributions are significantly different. First, we examine whether test assumptions are met. The samples are independent, and the variables meet the requirement that one is continuous (the index variable) and the other dichotomous. The assumption of equality of variances is answered as part of conducting the t-test, and so the remaining question is whether the variables are normally distributed. The distributions are shown in the histograms in Figure 12.3.12 Are these normal distributions? Visually, they are not the textbook ideal—real-life data seldom are. The Kolmogorov-Smirnov tests for both distributions are insignificant (both p > .05). Hence, we conclude that the two distributions can be considered normal. Having satisfied these t-test assumptions, we next conduct the t-test for two independent samples. Table 12.1 shows the t-test results. The top part of Table 12.1 shows the descriptive statistics, and the bottom part reports the test statistics. Recall that the t-test is a two-step test. We first test whether variances are equal. This is shown as the “Levene’s test for equality of variances.” The null hypothesis of the Levene’s test is that variances are equal; this is rejected when the p-value of this Levene’s test statistic is less than .05. The Levene’s test uses an F-test statistic (discussed in Chapters 13 and 15), which, other than its p-value, need not concern us here. In Table 12.1, the level of significance is .675, which exceeds .05. Hence, we accept the null hypothesis—the variances of the two distributions shown in Figure 12.3 are equal. Figure 12.3 Perception of High School Safety among 10th Graders Table 12.1 Independent-Samples T-Test: Output Note: SD = standard deviation. Now we go to the second step, the main purpose. Are the two means (4.40 and 4.80)

12.2. The transformed variable has equal variances across the two groups (Levene’s test, p = .119), and the t-test statistic is –1.308 (df = 85, p = .194). Thus, the differences in pollution between watersheds in the East and Midwest are not significant. (The negative sign of the t-test statistic, –1.308, merely reflects the order of the groups for calculating the difference: the testing variable has a larger value in the Midwest than in the East. Reversing the order of the groups results in a positive sign.) Table 12.2 Independent-Samples T-Test: Output For comparison, results for the untransformed variable are shown as well. The untransformed variable has unequal variances across the two groups (Levene’s test, p = .036), and the t-test statistic is –1.801 (df = 80.6, p =.075). Although this result also shows that differences are insignificant, the level of significance is higher; there are instances in which using nonnormal variables could lead to rejecting the null hypothesis. While our finding of insignificant differences is indeed robust, analysts cannot know this in advance. Thus, analysts will need to deal with nonnormality. Variable transformation is one approach to the problem of nonnormality, but transforming variables can be a time-intensive and somewhat artful activity. The search for alternatives has led many analysts to consider nonparametric methods. TWO T-TEST VARIATIONS Paired-Samples T-Test Analysts often use the paired t-test when applying before and after tests to assess student or client progress. Paired t-tests are used when analysts have a dependent rather than an independent sample (see the third t-test assumption, described earlier in this chapter). The paired-samples t-test tests the null hypothesis that the mean difference between the before and after test scores is zero. Consider the following data from Table 12.3. Table 12.3 Paired-Samples Data The mean “before” score is 3.39, and the mean “after” score is 3.87; the mean difference is 0.54. The paired t-test tests the null hypothesis by testing whether the mean of the difference variable (“difference”) is zero. The paired t-test test statistic is calculated as where D is the difference between before and after measurements, and sD is the standard deviation of these differences. Regarding t-test assumptions, the variables are continuous, and the issue of heterogeneity (unequal variances) is moot because this test involves only one variable, D; no Levene’s test statistics are produced. We do test the normality of D and find that it is normally distributed (Shapiro-Wilk = .925, p = .402). Thus, the assumptions are satisfied. We proceed with testing whether the difference between before and after scores is statistically significant. We find that the paired t-test yields a t-test statistic of 2.43, which is significant at the 5 percent level (df = 9, p = .038 < .05).17 Hence, we conclude that the increase between the before and after scores is significant at the 5 percent level.18 One-Sample T-Test Finally, the one-sample t-test tests whether the mean of a single variable is different from a prespecified value (norm). For example, suppose we want to know whether the mean of the before group in Table 12.3 is different from the value of, say, 3.5? Testing against a norm is akin to the purpose of the chi-square goodness-of-fit test described in Chapter 11, but here we are dealing with a continuous variable rather than a categorical one, and we are testing the mean rather than its distribution. The one-sample t-test assumes that the single variable is continuous and normally distributed. As with the paired t-test, the issue of heterogeneity is moot because there is only one variable. The Shapiro-Wilk test shows that the variable “before” is normal (.917, p = .336). The one-sample t-test statistic for testing against the test value of 3.5 is –0.515 (df = 9, p = .619 > .05). Hence, the mean of 3.39 is not significantly

different from 3.5. However, it is different from larger values, such as 4.0 (t = 2.89, df = 9, p = .019). Another example of this is provided in the Box 12.2. Finally, note that the one-sample t-test is identical to the paired-samples t-test for testing whether the mean D = 0. Indeed, the one-sample t-test for D = 0 produces the same results (t = 2.43, df = 9, p = .038). In Greater Depth … Box 12.2 Use of the T-Test in Performance Management: An Example Performance benchmarking is an increasingly popular tool in performance management. Public and nonprofit officials compare the performance of their agencies with performance benchmarks and draw lessons from the comparison. Let us say that a city government requires its fire and medical response unit to maintain an average response time of 360 seconds (6 minutes) to emergency requests. The city manager has suspected that the growth in population and demands for the services have slowed down the responses recently. He draws a sample of 10 response times in the most recent month: 230, 450, 378, 430, 270, 470, 390, 300, 470, and 530 seconds, for a sample mean of 392 seconds. He performs a one-sample t-test to compare the mean of this sample with the performance benchmark of 360 seconds. The null hypothesis of this test is that the sample mean is equal to 360 seconds, and the alternate hypothesis is that they are different. The result (t = 1.030, df = 9, p = .330) shows a failure to reject the null hypothesis at the 5 percent level, which means that we don’t have sufficient evidence to say that the average response time is different from the benchmark 360 seconds. We cannot say that current performance of 392 seconds is significantly different from the 360-second benchmark. Perhaps more data (samples) are needed to reach such a conclusion, or perhaps too much variability exists for such a conclusion to be reached. NONPARAMETRIC ALTERNATIVES TO T-TESTS The tests described in the preceding sections have nonparametric alternatives. The chief advantage of these tests is that they do not require continuous variables to be normally distributed. The chief disadvantage is that they are less likely to reject the null hypothesis. A further, minor disadvantage is that these tests do not provide descriptive information about variable means; separate analysis is required for that. Nonparametric alternatives to the independent-samples test are the Mann-Whitney and Wilcoxon tests. The Mann-Whitney and Wilcoxon tests are equivalent and are thus discussed jointly. Both are simplifications of the more general Kruskal-Wallis’ H test, discussed in Chapter 11.19 The Mann-Whitney and Wilcoxon tests assign ranks to the testing variable in the exact manner shown in Table 12.4. The sum of the ranks of each group is computed, shown in the table. Then a test is performed to determine the statistical significance of the difference between the sums, 22.5 and 32.5. Although the Mann-Whitney U and Wilcoxon W test statistics are calculated differently, they both have the same level of statistical significance: p = .295. Technically, this is not a test of different means but of different distributions; the lack of significance implies that groups 1 and 2 can be regarded as coming from the same population.20 Table 12.4 Rankings of

second variable, we find that Z = 2.103, p = .035. This value is larger than that obtained by the parametric test, p = .019.21 SUMMARY When analysts need to determine whether two groups have different means of a continuous variable, the t-test is the tool of choice. This situation arises, for example, when analysts compare measurements at two points in time or the responses of two different groups. There are three common t-tests, involving independent samples, dependent (paired) samples, and the one-sample t-test. T-tests are parametric tests, which means that variables in these tests must meet certain assumptions, notably that they are normally distributed. The requirement of normally distributed variables follows from how parametric tests make inferences. Specifically, t-tests have four assumptions: One variable is continuous, and the other variable is dichotomous. The two distributions have equal variances. The observations are independent. The two distributions are normally distributed. The assumption of homogeneous variances does not apply to dependent-samples and one-sample t-tests because both are based on only a single variable for testing significance. When assumptions of normality are not met, variable transformation may be used. The search for alternative ways for dealing with normality problems may lead analysts to consider nonparametric alternatives. The chief advantage of nonparametric tests is that they do not require continuous variables to be normally distributed. The chief disadvantage is that they yield higher levels of statistical significance, making it less likely that the null hypothesis may be rejected. A nonparametric alternative for the independent-samples t-test is the Mann-Whitney test, and the nonparametric alternative for the dependent-samples t-test is the Wilcoxon

usually does not present much of a problem. Some analysts use t-tests with ordinal rather than continuous data for the testing variable. This approach is theoretically controversial because the distances among ordinal categories are undefined. This situation is avoided easily by using nonparametric alternatives (discussed later in this chapter). Also, when the grouping variable is not dichotomous, analysts need to make it so in order to perform a t-test. Many statistical software packages allow dichotomous variables to be created from other types of variables, such as by grouping or recoding ordinal or continuous variables. The second assumption is that the variances of the two distributions are equal. This is called homogeneity of variances. The use of pooled variances in the earlier formula is justified only when the variances of the two groups are equal. When variances are unequal (called heterogeneity of variances), revised formulas are used to calculate t-test test statistics and degrees of freedom.7 The difference between homogeneity and heterogeneity is shown graphically in Figure 12.2. Although we needn’t be concerned with the precise differences in these calculation methods, all t-tests first test whether variances are equal in order to know which t-test test statistic is to be used for subsequent hypothesis testing. Thus, every t-test involves a (somewhat tricky) two-step procedure. A common test for the equality of variances is the Levene’s test. The null hypothesis of this test is that variances are equal. Many statistical software programs provide the Levene’s test along with the t-test, so that users know which t-test to use—the t-test for equal variances or that for unequal variances. The Levene’s test is performed first, so that the correct t-test can be chosen. Figure 12.2 Equal and Unequal Variances The term robust is used, generally, to describe the extent to which test conclusions are unaffected by departures from test assumptions. T-tests are relatively robust for (hence, unaffected by) departures from assumptions of homogeneity and normality (see below) when groups are of approximately equal size. When groups are of about equal size, test conclusions about any difference between their means will be unaffected by heterogeneity. The third assumption is that observations are independent. (Quasi-) experimental research designs violate this assumption, as discussed in Chapter 11. The formula for the t-test test statistic, then, is modified to test whether the difference between before and after measurements is zero. This is called a paired t-test, which is discussed later in this chapter. The fourth assumption is that the distributions are normally distributed. Although normality is an important test assumption, a key reason for the popularity of the t-test is that t-test conclusions often are robust against considerable violations of normality assumptions that are not caused by highly skewed distributions. We provide some detail about tests for normality and how to address departures thereof. Remember, when nonnormality cannot be resolved adequately, analysts consider nonparametric alternatives to the t-test, discussed at the end of this chapter. Box 12.1 provides a bit more discussion about the reason for this assumption. A combination of visual inspection and statistical tests is always used to determine the normality of variables. Two tests of normality are the Kolmogorov-Smirnov test (also known as the K-S test) for samples with more than 50 observations and the Shapiro-Wilk test for samples with up to 50 observations. The null hypothesis of

The test statistics of a t-test can be positive or negative, although this depends merely on which group has the larger mean; the sign of the test statistic has no substantive interpretation. Critical values (see Chapter 10) of the t-test are shown in Appendix C as (Student’s) t-distribution.4 For this test, the degrees of freedom are defined as n – 1, where n is the total number of observations for both groups. The table is easy to use. As mentioned below, most tests are two-tailed tests, and analysts find critical values in the columns for the .05 (5 percent) and .01 (1 percent) levels of significance. For example, the critical value at the 1 percent level of significance for a test based on 25 observations (df = 25 – 1 = 24) is 2.797 (and 1.11 at the 5 percent level of significance). Though the table also shows critical values at other levels of significance, these are seldom if ever used. The table shows that the critical value decreases as the number of observations increases, making it easier to reject the null hypothesis. The t-distribution shows one- and two-tailed tests. Two-tailed t-tests should be used when analysts do not have prior knowledge about which group has a larger mean; one-tailed t-tests are used when analysts do have such prior knowledge. This choice is dictated by the research situation, not by any statistical criterion. In practice, two-tailed tests are used most often, unless compelling a priori knowledge exists or it is known that one group cannot have a larger mean than the other. Two-tailed testing is more conservative than one-tailed testing because the critical values of two-tailed tests are larger, thus requiring larger t-test test statistics in order to reject the null hypothesis.5 Many statistical software packages provide only two-tailed testing. The above null hypothesis (men and women do not have different mean incomes in the population) requires a two-tailed test because we do not know, a priori, which gender has the larger income.6 Finally, note that the t-test distribution approximates the normal distribution for large samples: the critical values of 1.96 (5 percent significance) and 2.58 (1 percent significance), for large degrees of freedom (∞), are identical to those of the normal distribution. Getting Started Find examples of t-tests in the research literature. T-Test Assumptions Like other tests, the t-test has test assumptions that must be met to ensure test validity. Statistical testing always begins by determining whether test assumptions are met before examining the main research hypotheses. Although t-test assumptions are a bit involved, the popularity of the t-test rests partly on the robustness of t-test conclusions in the face of modest violations. This section provides an in-depth treatment of t-test assumptions, methods for testing the assumptions, and ways to address assumption violations. Of course, t-test statistics are calculated by the computer; thus, we focus on interpreting concepts (rather than their calculation). Key Point The t-test is fairly robust against assumption violations. Four t-test test assumptions must be met to ensure test validity: One variable is continuous, and the other variable is dichotomous. The two distributions have equal variances. The observations are independent. The two distributions are normally distributed. The first assumption, that one variable is continuous and the other dichotomous,

Galef and Tomasello come from a null-hypothesis-testing, experimental psychology background. The null hypothesis is something like “chimpanzees do not possess culture,” with culture being defined by something like “traditional behavior transmitted by imitation or teaching.” They could not show in their own or others’ experimental studies that captive chimpanzees could imitate or teach, so did not reject the null hypothesis. No culture.

If this experimental drug has no effect on heart disease (our null hypothesis), how likely is it that 91 out of 100 patients getting the drug would show improvement compared with only 49 out of 100 patients getting a placebo?

Always reject the null hypothesis

I think that the alternative views that have been presented in the literature are substantially worse—including the default view, or “null hypothesis,” according to which we can for the time being safely or reasonably ignore the prospect of superintelligence.

In theory, the scientific method should guard against the risk of confirmation bias. If, for example, we are testing a new drug, our experiment should not aim to confirm the hypothesis that the treatment works. Instead, we should test the “null hypothesis” that the drug has no effect. When the results allow for rejecting this null hypothesis with sufficient probability, the alternative hypothesis—that the drug has an effect—is plausible, and the conclusion of the study is positive. On paper, the process of scientific discovery goes against our natural instincts: it seeks to disprove an initial hypothesis.

So, if it’s not actually probable that the true value of a parameter is contained within a given confidence interval, why report it? If it’s not actually highly probable that the null hypothesis is false, why reject it?

Null hypothesis

The p-value is the probability of obtaining a result equal to or more extreme than what was observed, assuming that the Null hypothesis is true. The conditioning on the Null hypothesis is critical.

duality between p-values and confidence intervals. For the Null hypothesis of no-difference commonly used in controlled experiments, a 95% confidence interval of the Treatment effect that does not cross zero implies that the p-value is < 0.05.

Correlations have a hypothesis test. As with any hypothesis test, this test takes sample data and evaluates two mutually exclusive statements about the population from which the sample was drawn. For Pearson correlations, the two hypotheses are the following: Null hypothesis: There is no linear relationship between the two variables. ρ = 0. Alternative hypothesis: There is a linear relationship between the two variables. ρ ≠ 0. A correlation of zero indicates that no linear relationship exists. If your p-value is less than your significance level, the sample contains sufficient evidence to reject the null hypothesis and conclude that the correlation does not equal zero. In other words, the sample data support the notion that the relationship exists in the population.

The p-value for each independent variable tests the null hypothesis that the variable has no relationship with the dependent variable.

P-values indicate the strength of the sample evidence against the null hypothesis. If it is less than the significance level, your results are statistically significant.

P-values are the probability that you would obtain the effect observed in your sample, or larger, if the null hypothesis is correct.

p-values tell you how strongly your sample data contradict the null.

If the p-value is less than or equal to the significance level, you reject the null hypothesis and your results are statistically significant.

When the p-value is low, the null must go. If the p-value is high, the null will fly.

You can think of the null as the default theory that requires sufficiently strong evidence in your sample to be able to reject it.

The null and alternative hypotheses are always mutually exclusive.

The effect is the difference between the population value and the null hypothesis value. The effect is also known as population effect or the difference.

It specifies how strongly the sample evidence must contradict the null hypothesis before you can reject the null for the entire population.

A p-value describes the probability of getting data at least as extreme as those observed, if the null hypothesis were true.

If your sample contains sufficient evidence, you can reject the null and favor the alternative hypothesis.

To make this point, I often do the same coin flipping exercise that I explained during the probability discussion. In a class of forty students or so, I’ll have each student flip a coin. Any student who flips tails is eliminated; the rest flip again. In the second round, those who flip tails are once again eliminated. I continue the rounds of flipping until one student has flipped five or six heads in a row. You may recall some of the silly follow-up questions: “What’s your secret? Is it in the wrist? Can you teach us to flip heads all the time? Maybe it’s that Harvard sweatshirt you’re wearing.” Obviously the string of heads is just luck; the students have all watched it happen. However, that is not necessarily how the result could or would be interpreted in a scientific context. The probability of flipping five heads in a row is 1/32, or .03. This is comfortably below the .05 threshold we typically use to reject a null hypothesis. Our null hypothesis in this case is that the student has no special talent for flipping heads; the lucky string of heads (which is bound to happen for at least one student when I start with a large group) allows us to reject the null hypothesis and adopt the alternative hypothesis: This student has a special ability to flip heads. After he has achieved this impressive feat, we can study him for clues about his flipping success—his flipping form, his athletic training, his extraordinary concentration while the coin is in the air, and so on. And it is all nonsense. This phenomenon can plague even legitimate research. The accepted convention is to reject a null hypothesis when we observe something that would happen by chance only 1 in 20 times or less if the null hypothesis were true. Of course, if we conduct 20 studies, or if we include 20 junk variables in a single regression equation, then on average we will get 1 bogus statistically significant finding.

Per your mission instructions, you can reject the null hypothesis that this bus contains a random sample of 60 Changing Lives study participants at the .05 significance level. This means (1) the mean weight on the bus falls into a range that we would expect to observe only 5 times in 100 if the null hypothesis were true and this were really a bus full of Changing Lives passengers; (2) you can reject the null hypothesis at the .05 significance level; and (3) on average, 95 times out of 100 you will have correctly rejected the null hypothesis, and 5 times out of 100 you will be wrong, meaning that you have concluded that this is not a bus of Changing Lives participants, when in fact it is. This sample of Changing Lives folks just happens to have a mean weight that is particularly high or low relative to the mean for the study participants overall.

Here is a quick intuitive example. Suppose your null hypothesis is that male professional basketball players have the same mean height as the rest of the adult male population. You randomly select a sample of 50 professional basketball players and a sample of 50 men who do not play professional basketball. Suppose the mean height of your basketball sample is 6 feet 7 inches, and the mean height of the non–basketball players is 5 feet 10 inches (a 9-inch difference). What is the probability of observing such a large difference in mean height between the two samples if in fact there is no difference in average height between professional basketball players and all other men in the overall population? The nontechnical answer: very, very, very low.* The autism research paper has the same basic methodology

Once we get the regression results, we would calculate a t-statistic, which is the ratio of the observed coefficient to the standard error for that coefficient.* This t-statistic is then evaluated against whatever t-distribution is appropriate for the size of the data sample (since this is largely what determines the number of degrees of freedom). When the t-statistic is sufficiently large, meaning that our observed coefficient is far from what the null hypothesis would predict, we can reject the null hypothesis at some level of statistical significance. Again, this is the same basic process of statistical inference that we have been employing throughout the book. The fewer the degrees of freedom (and therefore the “fatter” the tails of the relevant t-distribution), the higher the t-statistic will have to be in order for us to reject the null hypothesis at some given level of significance. In the hypothetical regression example described above, if we had four degrees of freedom, we would need a t-statistic of at least 2.13 to reject the null hypothesis at the .05 level (in a one-tailed test). However, if we have 20,000 degrees of freedom (which essentially allows us to use the normal distribution), we would need only a t-statistic of 1.65 to reject the null hypothesis at the .05 level in the same one-tailed test.

In either case, we now have a good reason to believe that the difference we observed in the experiment was probably not due to chance. This is referred to in technical jargon with a confusing double negative: “rejection of the null hypothesis.” To determine exactly what caused the difference we observed requires much more research, but at the initial stage when we just want to know if there are any differences at all, this outcome provides ample motivation to keep on investigating. In particular, even if an experiment produces extremely high odds against chance, this doesn’t mean that the effect we’re interested in is proven. All the annoying cautions and qualifications commonly used in scientific lingo—it might be this, it could possibly be that, the purported results may perhaps be such and such—sound like a curious lack of enthusiasm, or an unwillingness to take a firm stand. But the prudence is intentional. It prevents existing knowledge from coagulating into unshakable dogma, which is the forte of religious faith. Also, just because a statistical test ends up with huge odds against chance doesn’t necessarily mean that the effect we were measuring is what we imagined it to be. To gain that sort of confidence it takes many independent scientists repeatedly examining the same effect in different ways, and for the results to be consistent on average.

Before we proceed further, be wary that neither the null hypothesis nor the alternative hypothesis can be unequivocally proven correct within hypothesis testing. Analyzing a sample extracted from a larger population is a subset of the data, and thus, any conclusions formed about the larger population based on analyzing the sample data are considered probabilistic rather than absolute.

For a long period of human history, most of the world thought swans were white and black swans didn’t exist inside the confines of mother nature. The null hypothesis that swans are white was later dispelled when Dutch explorers discovered black swans in Western Australia in 1697. Prior to this discovery, “black swan” was a euphemism for “impossible” or “non-existent,” but after this finding, it morphed into a term to express a perceived impossibility that might become an eventuality and therefore disproven. In recent times, the term “black swan” has been popularized by the literary work of Nassim Taleb to explain unforeseen events such as the invention of the Internet, World War I, and the breakup of the Soviet Union.

categorical and the dependent variable is continuous. The logic of this approach is shown graphically in Figure 13.1. The overall group mean is (the mean of means). The boxplots represent the scores of observations within each group. (As before, the horizontal lines indicate means, rather than medians.) Recall that variance is a measure of dispersion. In both parts of the figure, w is the within-group variance, and b is the between-group variance. Each graph has three within-group variances and three between-group variances, although only one of each is shown. Note in part A that the between-group variances are larger than the within-group variances, which results in a large F-test statistic using the above formula, making it easier to reject the null hypothesis. Conversely, in part B the within-group variances are larger than the between-group variances, causing a smaller F-test statistic and making it more difficult to reject the null hypothesis. The hypotheses are written as follows: H0: No differences between any of the group means exist in the population. HA: At least one difference between group means exists in the population. Note how the alternate hypothesis is phrased, because the logical opposite of “no differences between any of the group means” is that at least one pair of means differs. H0 is also called the global F-test because it tests for differences among any means. The formulas for calculating the between-group variances and within-group variances are quite cumbersome for all but the simplest of designs.1 In any event, statistical software calculates the F-test statistic and reports the level at which it is significant.2 When the preceding null hypothesis is rejected, analysts will also want to know which differences are significant. For example, analysts will want to know which pairs of differences in watershed pollution are significant across regions. Although one approach might be to use the t-test to sequentially test each pair of differences, this should not be done. It would not only be a most tedious undertaking but would also inadvertently and adversely affect the level of significance: the chance of finding a significant pair by chance alone increases as more pairs are examined. Specifically, the probability of rejecting the null hypothesis in one of two tests is [1 – 0.952 =] .098, the probability of rejecting it in one of three tests is [1 – 0.953 =] .143, and so forth. Thus, sequential testing of differences does not reflect the true level of significance for such tests and should not be used. Post-hoc tests test all possible group differences and yet maintain the true level of significance. Post-hoc tests vary in their methods of calculating test statistics and holding experiment-wide error rates constant. Three popular post-hoc tests are the Tukey, Bonferroni, and Scheffe tests.

regression results. Standardized Coefficients The question arises as to which independent variable has the greatest impact on explaining the dependent variable. The slope of the coefficients (b) does not answer this question because each slope is measured in different units (recall from Chapter 14 that b = ∆y/∆x). Comparing different slope coefficients is tantamount to comparing apples and oranges. However, based on the regression coefficient (or slope), it is possible to calculate the standardized coefficient, β (beta). Beta is defined as the change produced in the dependent variable by a unit of change in the independent variable when both variables are measured in terms of standard deviation units. Beta is unit-less and thus allows for comparison of the impact of different independent variables on explaining the dependent variable. Analysts compare the relative values of beta coefficients; beta has no inherent meaning. It is appropriate to compare betas across independent variables in the same regression, not across different regressions. Based on Table 15.1, we conclude that the impact of having adequate authority on explaining productivity is [(0.288 – 0.202)/0.202 =] 42.6 percent greater than teamwork, and about equal to that of knowledge. The impact of having adequate authority is two-and-a-half times greater than that of perceptions of fair rewards and recognition.4 F-Test Table 15.1 also features an analysis of variance (ANOVA) table. The global F-test examines the overall effect of all independent variables jointly on the dependent variable. The null hypothesis is that the overall effect of all independent variables jointly on the dependent variables is statistically insignificant. The alternate hypothesis is that this overall effect is statistically significant. The null hypothesis implies that none of the regression coefficients is statistically significant; the alternate hypothesis implies that at least one of the regression coefficients is statistically significant. The

It is clear that Bhu Mandala, as described in the Bhagvatam, can be interpreted as a geocentric map of the solar system out ot Saturn. But an obvious and important question is: Did some real knowledge of planetary distances enter into the construction of the Bhu Mandala system, or are the correlations between Bhu Mandala features and planetary orbits simply coincidental? Being a mathematician interested in probability theory, Thompson is better equipped than most to answer this question and does so through computer modelling of a proposed 'null hypothesis' -- i.e., 'that the author of the Bhagvatam had no access to correct planetary distances and therefore all apparent correlations between Bhu Mandala features and planetary distances are simply coincidental.' However, the Bhu Mandala/solar system correlations proved resilient enough to survive the null hypothesis. 'Analysis shows that the observed correlations are in fact highly improbable.' Thompson concludes: 'If the dimensions given in the Bhagvatam do, in fact, represent realistic planetary distances based on human observation, then we must postulate that Bhagvata astronomy preserves material from an earlier and presently unknown period of scientific development ... [and that] some people in the past must have had accurate values for the dimensions of the planetary orbits. In modern history, this information has only become available since the development of high-quality telescopes in the last 200 years. Accurate values of planetary distances were not known by Hellenistic astronomers such as Claudius Ptolemy, nor are they found in the medieval Jyotisa Sutras of India. If this information was known it must have been acquired by some unknown civilization that flourished in the distant past.

In hypothesis testing, the null hypothesis (H0) is assumed to be the commonly accepted fact but that is simultaneously open to contrary arguments. If there is substantial evidence to the contrary and the null hypothesis is disproved or rejected, the alternative hypothesis is accepted to explain a given phenomenon. The alternative hypothesis is expressed as Ha or H1. Intuitively, “A” represents “alternative.” The alternative hypothesis covers all possible outcomes excluding the null hypothesis.

the failure to reject a null hypothesis that is actually false.