A researcher desperate to find a statistically significant difference between tw

Question

A researcher desperate to find a statistically significant difference between two groups has data on 20 variables from each group. He tests 20 null hypotheses, each null hypothesis being that the two group means of a given variable are equal, hoping to find a significant difference on at least one variable.

(a) Suppose that the 20 tests are mutually independent and are conducted individually at the 5% significance level. What is the probability that at least one null hypothesis will be incorrectly rejected even if all null hypotheses are true?

(b) What does the above calculation tell you about the perils of searching for significance by multiple testing?

Explanation / Answer

a. All 20 tests are mutually independent. Each test has 5% probability of incorrectly rejecting the null hpothesis if it is true.

Let X denote the number of tests where null hypothesis will be incorrectly rejected even if all null hypotheses are true.

X follows a binomial distribution with n=20 and p = 0.05

P(X >=1 ) = 1 - P(X=0) = 1 - 20C0 * 0.050 *(1-0.05)20-0 = 0.6415

b. In multiple testing, level of type 1 error increases in the sense that, it will become very rare that all null hypothesis will be correctly accepted, given they are all true. In the above example, probabiity of committing type 1 error on at least 1 tests was 64.15%, which is significantly larger than 5%.

Related Questions

Navigate

• Browse (All)
• Browse A
• Subjects

Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.