Significance Level of each Individual Test in a Sequential Testing Procedure

The Joinpoint program uses a sequence of "permutation" tests to select the final model. Each one tests the null hypothesis H₀: k = k_a against the alternative hypothesis H₁: k = k_b. The procedure begins with k_a= K_min and k_b= K_max. If the null is rejected, then increase k_a by 1; otherwise, decrease k_b by 1. Let \(\hat{k}k = k_a = k_b\) be the final selected number of Joinpoints.

Because multiple tests are performed, Bonferroni adjustment is used to ensure that the approximate overall type I error is less than the specified significance level (significance level is also called the α-level, default α=.05). Each of these permutation test are carried out a significance level of α₁=α/(K_max-K_min), i.e., if the p-value < α₁, then it rejects the null.

The Bonferroni adjustment is conservative because the actual overall significance level is usually less than the nominal level α. The new adjustment procedure controls the overall over-fitting probabilities.

\(P \left( \hat{k}k > k_a \| k = k_a \right), \ \ \ \ \ \ \ k_a = K_{MIN}, \ldots, K_{MAX} - 1.\)

Let α(k_a,k_b) be the significance level of each individual test H₀: k=k_a vs. H₁: k=k_b. The new procedure set α(k_a,k_b)=α/(K_max-k_a). Notice that the individual significance level depends on the number of joinpoints k_a under the null. Consider an example where K_min= 0 and K_max= 4. The new procedure has the following properties:

\(P \left( \hat{k}k > 0 \| k = 0 \right) \leq \alpha \left( 0, 4 \right) + \alpha \left( 0, 3 \right) + \alpha \left( 0, 2 \right) + \alpha \left( 0, 1 \right)\)

\(P \left( \hat{k}k > 1 \| k = 1 \right) \leq \alpha \left( 1, 4 \right) + \alpha \left( 1, 3 \right) + \alpha \left( 1, 2 \right)\)

\(P \left( \hat{k}k > 2 \| k = 2 \right) \leq \alpha \left( 2, 4 \right) + \alpha \left( 2, 3 \right)\)

\(P \left( \hat{k}k > 3 \| k = 3 \right) \leq \alpha \left( 3, 4 \right).\)

If we like to bound these over-fitting probabilities by α, then we can assign different values for each α(k_a,k_b). That means, we can achieve a better power by setting

α(0,4)= α(0,3)= α(0,2)= α(0,1)= α/4

α(1,4)= α(1,3)= α(1,2)= α/3

α(2,4)= α(2,3)= α/2

α(3,4)= α