Keywords

1 Introduction

Fluid intelligence is a major factor in measuring general intelligence [1]. Therefore, the ABCD, the largest long-term study of brain development and child health in the United States, has captured MRI data as well as genetics, neuropsychological, behavioral, and other health assessments to “determine how childhood experiences (such as sports, videogames, social media, unhealthy sleep patterns, and smoking) interact with each other and with a child’s changing biology to affect brain development and social, behavioral, academic, health, and other outcomes”. Hence, one specific challenge is determining if machine learning prediction methods can be used to predict fluid intelligence from T1-weighted MRI [2].

Regarding prediction methods, there are hundreds of methods that can be used to predict a continuous variable from a set of features. Random Forest [3], Support Vector Machines [4], Recursive partitioning for classification [5], and Least Absolute Shrinkage and Selection Operator (LASSO) [6] are among the main machine learning approaches that can learn a linear structure from a set of predictors. Furthermore, there are several ways to improve prediction by first selecting the features that are associated with the data. Once selected, they’re used in a regression equation. Bootstrap Stage-Wise Model Selection (BSWiMS) used this strategy, plus bootstrap samples, to extract a set of regression models that predict the desired outcome [7]. The aim of this work is to explore the behavior of these approaches on the ABCD challenge data, report the features used to predict the fluid intelligence, build a final prediction based on top three predictors, and finally predict validation and testing data sets.

2 Materials and Methods

Data was provided by the ABCD challenge team: 3739, 415, and 4515 children were used for training, validation, and testing, respectively. All the children were measured by standardized protocols, and MRI features were extracted from T1-weighted images using a standard procedure described by the ABCD challenge organizers [2]. In this paper, we describe the methods used to predict the fluid intelligence score from the provided data sets.

2.1 Data Processing

We transformed all the volume scores provided via the challenge organizers by computing the cubic root on all measurements. This operation transformed the inherent distribution of volumetric errors into a Gaussian distribution and consequently mitigating the influence of measurement errors on each one of the volumes’ distributions. Furthermore, brain asymmetry has been associated with several chronic neurological diseases such as Schizophrenia and Alzheimer [8, 9]; therefore, here we explore the hypothesis that structural asymmetry may also be associated with fluid intelligence. To explore this hypothesis, we computed the mean and absolute difference of the left and right volumes of each the brain structures. Finally, we augmented the set of transformed features by adding the descriptive features described in Table 1. All transformations and descriptive computations were done in Microsoft Excel (2013).

Table 1. Descriptive features derived from volumetric data. Transformed volumes were computed using the cubic root transformation.
Full size table

Once we computed the descriptive features, we adjusted all the features (transformed and descriptive) to any residual associations to volume or age, and we did this independently on males and females. The adjustment was done by computing the residuals of the following linear model:

$$ f_{i} = \beta_{o} + \beta_{1} Age + \beta_{2} AVGVolume + \beta_{3} Age*AVGVolume, $$
(1)

on all features using the FRESA.CAD package [7].

2.2 Benchmarking Machine Learning Methods

The adjusted training data was explored by a Benchmarking function provided by the FRESA.CAD R package. The FRESA.CAD benchmarking function does a repeated hold-out cross-validation (RHOCV) [10] of the following machine learning methods: Recursive partitioning (RPART), Random Forest (RF), Support Vector Machine (SVM), LASSO, and Bootstrapped Stage-Wise Model Selection (BSWiMS) [3,4,5,6, 11]. Furthermore, the benchmarking evaluates the efficiency of different feature selection methods like minimum redundancy maximum relevance mRMR [11] and univariate selection adjusted for false discovery rate [12]. Features selected by these methods were used to train robust regression, ridge regression, and linear regression models. Trained models were evaluated in the holdout testing data. Finally, internal-test performance results were reported. Besides exploring the regression models using the augmented data, we also explored the capability of the original ABCD challenge volume scores to model fluid intelligence.

All the benchmarks were run separately on males’ and females’ data set using 90% of the data for training and holding 10% of the data for test evaluation. The hold-out procedure was repeated 100 times. Figures 1 and 2 show the results of benchmarking the regression methods on the ABCD challenge data set. The Pearson correlation between fluid intelligence and benchmarking test results indicated that BSWiMS had an equivalent performance on both males’ and females’ data sets on the augmented training data. On the other hand, the CV results indicate that SVM was able to predict males’ and females’ datasets based only on features provided by the original training set.

Fig. 1.
figure 1

Benchmarking Machine Learning Regression Algorithms for male subjects. Pearson correlation results of the predicted fluid intelligence to fluid intelligence. The top plots show the Pearson correlation of several; ML methods using the training dataset provided by the ABCD challenge. The bottom plots show the Pearson correlation of the same methods using the augmented data set.

Full size image
Fig. 2.
figure 2

Benchmarking Machine Learning Regression Algorithms for female subjects. Pearson correlation results of the predicted fluid intelligence to fluid intelligence. The top plots show the Pearson correlation of several; ML methods using the training dataset provided by the ABCD challenge. The bottom plots show the Pearson correlation of the same methods using the augmented data set.

Full size image

2.3 Method Ensemble

The results of the Benchmarking function on the training data set were used to determine successful machine learning strategies that may be able to predict the fluid intelligence on the validation and testing data. We selected three methods according to the benchmark results. The first method was SVM. SVM showed a statistically significant association using original features. The second method was the BSWiMS that selected linear models composed on a small number of features (n < 8), and it had an equivalent performance on both males and females. Finally, we selected the Random Forest method because RF uses an independent learning methodology than SVM and BSWiMS, and the benchmarking results indicated that it was able to predict fluid intelligence. Regarding the selected modeling methods, SVM used the radial kernel, RF is based on decision trees, and BSWiMS built simple linear models. Hence, they approached the regression problem using three different strategies. Once we selected the methods, we refit the SVM, BSWiMS, and RF using the training data sets. SVM was fitted to male and female by training original data using the radial basis function, and we did not attempt to optimize any of the SVM parameters. Once trained, we predicted the validation and testing sets. For BSWiMS and RF we refit the models using the augmented training data set. First, we fitted the BSWiMS model using default parameters and all the sex-stratified training data. The BSWiMS procedure was repeated 10 times, and the final model coefficients were bagged. We also fitted the data utilizing the RF method, by using default parameters. Hence, no parameter optimization was attempted. To minimize prediction noise, we repeated the sex-stratified RF fitting 15 times. Once BSWiMS and RF were fitted on males and females, we proceeded to predict the validation and to test datasets. First by cubic-root transformation, then we augmented the features using the described features. Finally, we adjusted the data using the coefficients of Eq. 1. The final prediction was the average of the three methods.

2.4 Prediction Evaluation

The evaluation of the results was done using the ABCD challenge validation data set using the R code provided by challenge organizers. The code evaluated the mean square error (MSE) and the R2 between predicted scores and the actual scores.

3 Results

3.1 Benchmarking Results

Figures 1 and 2 shows the Pearson correlation results of the repeated hold-out validation strategy using only validation data. The Pearson correlation varied from not being statistically significant, for BSWiMS method on the original data set, to a maximum correlation of 0.16 with 95% confidence interval of 0.12 to 0.21. Figure 3 shows the most commonly selected features by the different filtering strategies using the transformed data sets. It is clear that the features that are useful to predict fluid intelligence are different between males and females. Only five features: |frontalsup_gm|, |wm400_wm|, insula_gm, parahippocampal_gm, and parietalinf_gm were common between males and females. Among these five features, two were related to differences between left and right volumes, and three were related to volume size.

Fig. 3.
figure 3

Features associated with Fluid Intelligence Scores dived by Filter Method. The heat-map shows the relative frequency of the feature selection, where yellow represents the most common discovery after 100 repetitions of the hold-out cross-validation. (Color figure online)

Full size image

3.2 Feature Relevance

The BSWiMS procedure returns a summary analysis of the relevant features required to model the fluid intelligence score using the sex-stratified transformed data sets. Table 2 shows the required features needed to model the male intelligence scores. Table 3 shows the features involved in predicting female scores. The tables show that the volume of the parahippocampal gyrus and the volume of the pons are an important feature associated with fluid intelligence scores in both males and females. Moreover, the other features are unique to males and females. Among them, four features are associated with differences between the left and right volumes in males while only two features associated with differences between and right volumes affect females significantly.

Table 2. Male Model. |°| are features related to the left-right absolute difference
Full size table
Table 3. Female Model. |°| are features related to the left-right absolute difference
Full size table

3.3 Models Performance

The training results of the BSWiMS procedure indicated that the linear models associated with fluid intelligence can explain 3.6% and 2.2% of the male and the female variance, respectively. Table 4 shows the MSE of the internal cross-validation of the models based. The validation results of the ensemble procedure showed that structural MRI predicted 0.7% (MSE = 71.86) of the fluid intelligence variance and that this prediction was marginally significant (p-value = 0.06). Finally, the test results indicated that the ensemble method had an MSE of 100.89.

Table 4. MSE of the internal validation of the three models
Full size table

4 Discussion

Machine learning strategies can be used to explore systematically different methodologies to model and predict abstracts outcomes like fluid intelligence. The exploration of volumetric data and transformed data showed that data transformation can be useful in finding anatomical structures that may affect the expression or development of child fluid intelligence. Several brain structures were found to explain the difference in the intelligence score, and the aggregation of them in linear models increased the power to describe the difference between children. We also found that differences between the left and right structures in the brain are associated with the fluid intelligence scores. Clear gender differences were also observed. Males were modeled by simple linear models while explaining the female fluid intelligence required pattern matching methods. In other words, there are patterns in female fluid intelligence that are difficult to model by simple linear models. Even with this difficulty, we found that there are significant associations between changes in the pons white mater, parahippocampal gyrus, and fluid intelligence in both males in females and that this association can only be explained 0.7% of the variance. Hence, the reported findings are indicating that there are factors beyond differences in brain structures that play a more important role in the development of child fluid intelligence.

5 Conclusion

Machine learning methods can be used to discover the main factors associated with fluid intelligence. Here, we found that brain morphology plays a small role in explaining the difference in fluid intelligence in children between ages 9 and 10.