Permutation Test Details

Describe the permutation test used here.


In this program, the permutation test is used repeatedly for testing between two different joinpoint models, a simpler model with fewer joinpoints called the null model and a more complicated model called the alternative model. The alternative model fits better because it is more complicated.

The question for the test is: does it fit much better than would be expected by chance. To test this statistically, we calculate a ratio, T=SSEN/SSEA, where SSEN is the sum of squared errors (SSE) from the null model and SSEA is the SSE from the alternative model. Values of the ratio (T) close to 1 mean that the alternative is not much better than the null model, while larger values mean that the alternative is much better.

In order to decide how much larger a ratio needs to be to be statistically significant, we use the permutation method, since it is not possible to drive this analytically. In this method, we randomly permute (that is, shuffle) the errors (also called the residuals) from the null model and add them back onto the modeled values from the null model to create a permutation data set. Then we calculate the ratio T for the permutation data set and measure how much evidence the data provide against the null hypothesis by estimating the proportion of the permutation data sets whose T values are at least as extreme as the one we observed with the original data set. This proportion is called the p-value in statistical testing.

If the true model was the null model (i.e. the truth being the simpler model), we would expect that on average, about half of the T ratios computed from the permutation data sets would be greater than the one derived from the original data (i.e. the distribution of the p-value under the null is uniform over the interval from 0 to 1).

If the true model was the alternative model, we would expect that after permuting the errors most of the new T ratios would be less than the original T ratio and thus the p-value would be small. In other words, the permuted data set would look less like the alternative model than the original data.

So we reject the null model (or null hypothesis) if less than a certain proportion of the T ratios are greater than or equal to the original T ratio. That is, we reject the null hypothesis if the p-value is small.

For more specific details, see Kim HJ, Fay MP, Feuer EJ, Midthune DN. Permutation Tests for Joinpoint Regression with Applications to Cancer Rates. Stat Med 2000;19:335-351. To request a reprint, email Mr. Reggie Taborn for a copy at: