An official website of the United States government

How Joinpoint Selects the Final Model

Selecting the model, that is, the number of joinpoints, can be done by using one of the following options in Joinpoint:

Data Driven BIC Methods

  • Weighted BIC (WBIC)
  • Data Dependent Selection (DDS)

Traditional BIC Methods

  • Bayesian Information Criterion (BIC)
  • BIC3
  • Modified BIC

Permutation Test

There are five methods based on the information criteria and the Permutation test. The information-based criteria methods are much more computationally efficient than the Permutation test. These information-based criteria methods can be grouped into two categories, Traditional Bayesian Information Criteria (BIC) methods and Data Driven BIC methods.  The Traditional BIC methods select the model for which the object function, which is either the sum of the model fit error and the penalty term (BIC and BIC3) or an asymptotic approximation of the Bayes factor (Modified BIC), is minimized.  The Data Driven BIC methods include the DDS method introduced in Joinpoint v4.6.0.0 and the Weighted BIC introduced in v4.7.0.0.

The motivation of the Data Driven BIC methods is to internally determine the model selection method, BIC or BIC3, based on the characteristics of data. The basic idea is to use BIC if change sizes are relatively small, and to use BIC3 otherwise. Through simulations, the permutation test has been shown to produce reasonable results, (i.e. predicts the correct number of joinpoints), when the effect size, a function of the size of slope changes adjusted for the variability in data, is large. The BIC3 method has been shown to produce results similar to the permutation test in situations when the permutation test is performing well relative to the traditional BIC method. For situations with smaller effect sizes, the BIC performs better. However, unlike in simulations, the analyst does not know the true effect sizes in practice, and the data driven methods were developed with an aim to use the data to approximate the effect sizes. The data driven methods either choose between BIC and BIC3 (Data Driven Selection-DDS) or use a weighted average of the BIC and BIC3 (Weighted BIC). Simulations have shown that among the Data Driven BIC methods, the DDS and Weighted BIC methods perform similarly, and the Weighted BIC is preferred because it has a stronger conceptual justification.

 

The Permutation test, that was proposed in Kim et al. (2000) and implemented in the original version of Joinpoint, is computationally intensive.

Therefore, our overall recommendations are:

  • Use the Weighted BIC method if the user prefers a method that on average performs best across a wide range of situations. While the permutation test, BIC, and BIC3 might perform better in specific situations, the Weighted BIC is the most flexible in adapting to different situations. As of version 5.0, the Weighted BIC is the default for use in the Joinpoint software. The DDS method is a worthy competitor to the Weighted BIC, and the Weighted BIC was selected as preferred based on conceptual reasons, rather than on the results of our simulations.
  • Use the BIC3 method if the user would like to produce results similar to the permutation test, but computation time is an issue.
  • The Modified BIC is generally not recommended for most users and are included for record of methodology development.
  • Use the permutation test if the user prefers the method that has the longest track record and generally produces parsimonious results (i.e., will detect fewer joinpoints than other approaches, especially if the slope changes are small).

 

Details of each method are given below: 

Bayesian Information Criteria (BIC)

The value of BIC is the loglikelihood value with penalizing the cost of extra parameters. The model with the minimum value of BIC is selected as the optimal model.

Weighted BIC (WBIC)   

While the Data Dependent Selection (DDS) internally uses BIC or BIC3 based on the empirically determined cut-off values for the selection statistics, the weighted BIC combines BIC and BIC3 using a weighted penalty term based on the data characteristic. 

Data Dependent Selection (DDS)   

This procedure internally determines the model selection method, BIC or BIC3, based on the characteristics of data, and its basic idea is to use BIC if change sizes are relatively small and BIC3 otherwise. 

BIC3

A modification of traditional BIC with a harsher penalty.

Modified BIC

A modification of traditional BIC proposed to improve its performance.

Permutation Test

This method uses the sequence of permutation tests to ensure that the approximate probability of overall Type I error is less than the specified significance level (also called the alpha level, default = .05). Assuming that the default value of the minimum number of joinpoints is 0, "the overall Type I error" is the probability of incorrectly concluding that the underlying model has one or more joinpoints when, in fact, the true underlying model has no joinpoints.