New Significance Levels

Answer:

The Joinpoint software uses a series of permutation tests to determine the number of joinpoints. Prior to Version 3.0, the software used the Bonferroni adjustment to control the error probability of each of the multiple tests. The Bonferroni adjustment has been shown to be conservative and the procedure tends to select fewer joinpoints than it should. The procedure with new significance levels controls the over-fitting probability and it is superior to the traditional Bonferroni adjustment.

Details: The Bonferroni adjustment is conservative because the actual overall significance level is usually less than the nominal level α. Starting with Version 3.0, the new adjustment procedure controls the overall over-fitting error probabilities, P(k̂ >k_a | k = k_a), k_a = MIN,...,MAX-1 , under α.

Let k denote the number of joinpoints and α(k_a; k_b) be the significance level of each individual test H₀:k = k_a vs. H_a:k = k_b.

The new procedure set α(k_a; k_b) = α/(MAX - k_a).

Notice that the individual significance level depends on the number of joinpoints k_a under the null. Consider an example where MIN = 0 and MAX= 4. The new procedure has the following properties:

P(k̂ > 0 | k = 0) ≤ α(0,4) + α(0,3) + α(0,2) + α(0,1);

P(k̂ > 1 | k = 1) ≤ α(1,4) + α(1,3) + α(1,2);

P(k̂ > 2 | k = 2) ≤ α(2,4) + α(2,3);

P(k̂ > 3 | k = 3) ≤ α(3,4).

If we like to bound these over-fitting probabilities by α, then we can assign different values for each α(k_a; k_b) . That means, we can achieve a better power by setting

α(0,4) = α(0,3) = α(0,2) = α(0,1) = α / 4;

α(1,4) = α(1,3) = α(1,2) = α / 3;

α(2,4) = α(2,3) = α / 2;

α(3,4) = α