4.5.3 Power and Significance Level

4.5.3  Power and Significance Level

In our example, we sought to limit to .05 the probability, conditional on  being valid, of rejecting . Rejecting a valid null hypothesis is known as a type I error. If we design a test so the conditional probability of a type I error does not exceed ε, we say the test has significance level ε.

A second form of error, called type II error, occurs when a test fails to reject an invalid null hypothesis. In our coin tossing example, our test design focused on avoiding type I error, but it could have focused on avoiding type II error.

The power of a test is the probability of it rejecting . That probability depends, of course, on the unknown parameter θ—significance level is a constant, whereas power is a function of θ. Despite this difference, significance level and power are somewhat competing priorities. At θ = θ₀, we want low power, because we don’t want to reject a valid null hypothesis. Because significance level is a cap on power at θ = θ₀, we desire a low significance level. However, we want power to be high when θ ≠ θ₀, but low power when θ = θ₀ tends to produce low power when θ ≠ θ₀. Hence, the desirable property of a low significance level tends to be incompatible with the desirable property of high power when θ ≠ θ₀. The priorities compete.