An evaluation of intelligent prognostic systems for colorectal cancer

https://doi.org/10.1016/S0933-3657(98)00052-9Get rights and content

Abstract

In this paper we describe attempts at building a robust model for predicting the length of survival of patients with colorectal cancer. The aim of the research, reported in this paper, is to study the effective utilisation of artificial intelligence techniques in the medical domain. We suggest that an important research objective of proponents of intelligent prognostic systems must be to evaluate the additionality that AI techniques can bring to an already well-established field of medical prognosis. Towards this end, we compare a number of different AI techniques that lend themselves to the task of predicting survival in colorectal cancer patients. We describe the pros and cons of each of these methods using the usual metrics of accuracy and perspicuity. We then present the notion of intelligent hybrid systems and evaluate the role that they may potentially play in developing robust prognostic models. In particular we evaluate a hybrid system that utilises the k Nearest Neighbour technique in conjunction with Genetic Algorithms. We describe a number of innovations used within this hybrid paradigm used to build the prognostic model. We discuss the issue of censored patients and how this issue can be tackled within the various models used. In keeping with our objective of studying the additionality that AI techniques bring to building prognostic models, we use Cox’s regression as a standard and compare each AI technique with it, attempting to discover their capabilities in enhancing prognostic methods in medicine. In doing so we address two main questions—which model fits the data best?, and are the results obtained by the various AI techniques significantly different from those of Cox’s regression? We conclude this paper by discussing future enhancements to the work presented and lessons learned from the study to date.

Introduction

In Western countries, cancer of the large bowel is the second most common cause of death from malignancy [14] and recent reports indicate that the incidence of colorectal cancer is increasing. In spite of much research into new diagnostic and treatment methodologies, mortality from colorectal cancer has not been significantly reduced. Likewise, the prognostic classification of colorectal cancer has changed little since Dukes introduced a scheme for staging rectal cancer almost 60 years ago [10]. Modifications to this staging method have been proposed although the additional benefit is thought to be slight. Recent research into outcome prediction for colorectal cancer showed that artificial neural networks (ANN) are more accurate than current clinico-pathological methods as they permit recognition of patterns in complex biological data sets that clinicians are unable to detect [4].

Improving upon current methods of prognosis remains important for two reasons. Firstly, it provides information with respect to survival, allowing appropriate patient counselling and management to be planned. Secondly, it allows patients to be stratified into groups for the study of different primary and adjuvant treatment modalities.

Current methods of analysing patient prognosis and the influence of potential predictor variables rely on statistical survival analyses. These techniques are based on the distribution of survival times in one or more patient groups or alternatively on the construction of a regression model which permits the simultaneous analysis of a number of predictor variables. While these techniques provide an informative insight into the survival distributions of populations and the influence of patient features on survival within these populations, they are less useful at providing prognostic information for an individual patient. If the results of such analyses are to be used to provide a prognostic estimate for an individual patient then other approaches need to be explored. This was the purpose of the current study.

The suitability of three AI techniques, namely Neural Networks, Regression Trees and a k Nearest Neighbour (kNN)/Genetic Algorithm (GA) hybrid, as prognostic tools is investigated. The aim is to bring AI methods into mainstream prognostic analysis by enhancing them with additional facilities provided by traditional statistical techniques and in the process overcoming some of the shortcomings of statistical techniques. However, AI techniques for performing such a modelling task are limited based on the fact that few applications of these techniques have attempted to address characteristics that arise within the medical domain. One such feature of medical survival data is censored patient data. Few AI-based prognostic models address this issue while the majority of them ignore it, which we believe, seriously limits the applicability of AI within medicine. In this paper, we discuss some of the problems that censored patient data causes and attempt to provide partial solutions that we feel are promising and need to be considered as important future research directions.

This paper follows the following format: In Section 2we discuss some related research in the domain of building medical prognostic systems. In particular we discuss recent work in building intelligent prognostic systems for colorectal cancer as well as research that the authors believe is one of the few serious attempts to address the issue of censored data within the realm of AI-based prognosis. In Section 3we discuss the data that we used in our study and compare the performance of three different AI techniques, namely, Neural Networks, Regression Trees and kNN algorithms in terms of their accuracy and perspicuity. Section 4discusses some of the enhanced distance metrics developed to remove biases in traditional distance metrics that limit the use of the kNN algorithms. In Section 5we further enhance the capabilities of the kNN algorithm by hybridising it with a genetic algorithm. In Section 6we provide an empirical evaluation of the models and analyse the results. We discuss the issue of censored patients, in Section 7, proposing a possible starting point for handling them within the kNN paradigm. In Section 8we discuss problems in comparing the accuracy of models for censored data. The paper concludes in Section 9, with a discussion on further research that the authors intend to carry out in the near future to further the integration of AI techniques into mainstream prognosis.

Section snippets

Related work

Bottaci et al. [4] used ANNs to build a model for predicting outcome in colorectal cancer patients. They claimed that ANNs are capable of recognising patterns in complex biological data sets that conventional statistical analysis as well as clinicians themselves are unable to recognise. While they provided some evidence to suggest that ANNs perform better than clinicians who tend to use pathological staging such as Dukes [10] and Jass [13] to arrive at an outcome prediction, little empirical

Comparison of different AI paradigms

All patients in this study presented with colorectal cancer between 1973 and 1983 in the Royal Victoria Hospital and the Belfast City Hospital, Belfast, Northern Ireland. Complete clinical and pathological data were collected on a total of 312 patients. Exclusion of operative deaths (deaths which occurred within 30 days of operation), patients who died from causes other than tumour recurrence or metastasis, and cases with missing data resulted in a final number of 216 cases.

For each case,

Using enhanced distance metrics

The Euclidean distance metric and its variant known as the overlap distance metric are the most commonly used distance metrics in kNN algorithms. The Euclidean distance measure is known to bias retrieval of cases to those that match on the categorical variables within the data set [2], especially in the case when all variables have equal weights associated with them. Attribute weightings may reduce this bias, however, weights must be chosen with this purpose in mind, and their intuitive meaning

Attribute weight discovery

To overcome the problem of allocating weights to attributes the authors developed a hybrid system that utilised a genetic algorithm to conduct a search of the attribute weight space, discovering the optimal attribute weightings. The genetic algorithm used is also part of the Mining Kernel System [3]. Ten-fold cross-validation using the kNN algorithm was used as the fitness evaluation function. The population size was set to 50 and each chromosome in the population consisted of 105 bits (7 bits

Empirical evaluation and analysis of models

As discussed in Section 3, to produce accurate generalisations from the data, Regression Tree induction techniques require the data to not be sparse. In the case of our data set, this criterion is not satisfied and so it is not surprising that the predictive accuracy was not good. In fact, as can be seen from Table 9, its accuracy was the same as that of the Base Line Prediction method used which was using the average survival value in the training data set as the prediction for the test data.

Incorporating censored patients

The data set used in this study consisted of a number of censored patients. In this section we discuss preliminary steps taken by the authors towards dealing with censored data within the kNN approach. Future enhancements to these methods proposed by the authors are discussed in Section 9.

The first step was to analyse the uncensored patient data separately. The purpose of this was to see whether the nature of the uncensored patient data was any different from that of the censored patient data.

Analysis of models for censored data

In the case of the censored patients the observed survival values are in fact a minimal estimate as opposed to the actual survival. Therefore, over estimates of these survival values are valid, however, it is not possible to get an exact measure of the error. In cases where the predicted survival values are less than the (minimal) observed survival value a minimal error can be calculated. For example, if a censored patient has a recorded survival length of 85 months and the predicted survival

Conclusions and future work

In this paper we have described research undertaken, by the authors, into the building of a model for prognosis of colorectal cancer patients. A number of AI modelling techniques were used to build prognostic models. The paper also proposes a method for arriving at point estimates of survival from Cox’s regression which allowed comparisons of its output with AI modelling output. The models developed were found, in general, to achieve comparable accuracies. Statistical tests used to decide on

References (18)

  • M Buyse et al.

    Artificial neural networks, letter to the editor

    Lancet

    (1997)
  • Anand SS, Patterson D, Hughes JG. Knowledge Intensive Exception Spaces, Proceedings of the AAAI-98 Conference. 1998;...
  • Anand SS, Hughes JG. Hybrid data mining systems. In: Proceedings of the Second Pacific Asia Conference on Knowledge...
  • Anand SS, Scotney BW, Tan MG, McClean SI, Bell DA, Hughes JG, Magill IC. Designing a Kernel for data mining. IEEE...
  • L Bottaci et al.

    Artificial neural networks applied to outcome prediction for colorectal cancer patients in separate institutions

    Lancet

    (1997)
  • CLEMENTINE User Manual. Integral Solutions Limited,...
  • S Cost et al.

    A weighted nearest neighbour algorithm for learning with symbolic features

    Mach Learn

    (1993)
  • DR Cox

    Regression models and life-tables

    J R Stat Soc

    (1972)
  • Craighurst R, Martin W. Enhancing GA performance through crossover prohibitions based on ancestry. In: Proceedings of...
There are more references available in the full text version of this article.

Cited by (25)

  • Hybrid genetic algorithms

    2003, Data Handling in Science and Technology
  • Evaluation of inherent performance of intelligent medical decision support systems: Utilising neural networks as an example

    2003, Artificial Intelligence in Medicine
    Citation Excerpt :

    Most continuous classifications have been in survival or time-to-event analysis. For example, Anand et al. [4] compared several NN approaches with Cox’s regression, where there were censored cases in a colorectal cancer cohort of patients. Kaplan–Meier graphs and NNs were the subject of Biganzoli et al. [13], who produced graphs for both techniques for survival, again with censored data.

  • Machine learning for survival analysis: A case study on recurrence of prostate cancer

    2000, Artificial Intelligence in Medicine
    Citation Excerpt :

    A comprehensive overview and classification of existing neural network-based techniques for survival analysis is provided by Ripley and Ripley [22], while D’Amico et al. [7] provide a list of artificial neural networks techniques that specifically target the prostate cancer recurrence prediction. Anand et al. [2] stress the need to develop the prognostic models that would, instead of hazard or survival functions, explicitly provide a prognostic estimate for an individual patient. They compare regression trees, k-nearest neighbors (k-NN) and a regression variant of an artificial neural network to model the patient’s survival time after being diagnosed with colorectal cancer.

  • Evolutionary computation in medicine: An overview

    2000, Artificial Intelligence in Medicine
View all citing articles on Scopus
View full text