mROC: a computer program for combining tumour markers in predicting disease states

https://doi.org/10.1016/S0169-2607(00)00129-2Get rights and content

Abstract

Receiver operating characteristic (ROC) curves are limited when several diagnostic tests are available, mainly due to the problems of multiplicity and inter-relationships between the different tests. The program presented in this paper uses the generalised ROC criteria, as well as its confidence interval, obtained from the non-central F distribution, as a possible solution to this problem. This criterion corresponds to the best linear combination of the test for which the area under the ROC curve is maximal. Quantified marker values are assumed to follow a multivariate normal distribution but not necessarily with equal variances for two populations. Other options include Box–Cox variable transformations, QQ-plots, interactive graphics associated with changes in sensitivity and specificity as a function of the cut-off. We provide an example to illustrate the usefulness of data transformation and of how linear combination of markers can significantly improve discriminative power. This finding highlights potential difficulties with methods that reject individual markers based on univariate analyses.

Introduction

Tumour markers are increasingly used for the diagnosis of a pathological condition, for treatment surveillance, for disease evolution and for prognosis. Although the initial expert and methodological evaluation phases of a new marker are generally well elaborated, the subsequent phases of marker evaluation still need to be developed. New markers are often introduced into routine clinical practice without a rigorous analysis of their utility. However, the use of a marker in any situation on a poorly defined population can lead to divergent interpretations. Inversely, the misunderstanding of the performances of a marker will either lead to its non-use or to its poorly adapted use.

Before a marker can be routinely used as a diagnostic tool, it is evaluated for its sensitivity and its specificity in distinguishing between “diseased” and “non-diseased” individuals. This methodology relies on a strict unambiguous definition of two well-differentiated populations, summarised by a dichotomous outcome such as “yes/no” or “presence/absence”. This is considered the “gold standard”.

The most common statistical tool used for evaluating sensitivity and specificity is the receiver operating characteristic curve [1]. These curves measure the utility of tumour markers by quantifying their sensitivity and specificity for diagnosis, for treatment surveillance, for disease evolution and for prognosis of certain pathological conditions. These methods are often used in evaluating the discriminating value of each marker taken one at a time. However, these methods have their limits in the presence of several markers simultaneously and methodology is needed for evaluating their relative importance.

One solution consists in regrouping the ROC curves by the best linear combination, which maximises the area under the ROC curve under the hypothesis of a multivariate normal distribution [2]. Methods for estimating confidence intervals for the area under these curves are also provided [3]. From this best linear combination, parametric and/or non-parameteric methods can be applied interactively. This paper presents a computer program called mROC, which implements this approach.

In Section 2 we present the methods. In Section 3 mROC is described and in Section 4, an application is demonstrated.

Section snippets

Definitions and formulae

This section presents a brief description of the methodology used. A more detailed presentation can be found in previous works [2], [3], [4].

Computer program

mROC is a stand alone computer program written in C++ in a PC environment running under Windows 95/NT in 32 bit mode. It does not function with computers running with Windows 3.x and uses a minimum of 5 Mbytes of hard disk space.

Application

To illustrate the use of mROC, we consider an example evaluating the serial carcinoembryonic antigen marker (ACE) measured before and after treatment in evaluating the response to chemotherapy, in patients presenting with advanced digestive cancers [9]. Two populations were defined according to clinical response to chemotherapy. Base 10 logarithmic transformations were used for each variable since the Box–Cox estimates for λ contained 0 in the 95% confidence interval. Each marker taken

Availability

Information about program availability can be obtained from the first author.

Acknowledgements

This project was partially financed by the French Ligue Nationale Contre le Cancer.

References (10)

  • D Faraggi et al.

    Some methodological issues associated with tumour marker development: biostatistical aspects,

    Urol. Oncol.

    (2000)
  • J.A Hanley et al.

    The meaning and use of the area under the receiver operating characteristic (ROC) curve

    Radiology

    (1982)
  • J.Q Su et al.

    Linear combinations of multiple diagnostic markers

    J. Am. Stat. Ass.

    (1993)
  • B Reiser et al.

    Confidence intervals for the generalized ROC criterion

    Biometrics

    (1997)
  • A Kramar et al.

    Critères ROC généralisés pour l'évaluation de plusieurs marqueurs tumoraux

    Rev. Epidem. Santé Publ.

    (1999)
There are more references available in the full text version of this article.

Cited by (46)

  • Changes in markers associated with dendritic cells driving the differentiation of either T<inf>H</inf>2 cells or regulatory T cells correlate with clinical benefit during allergen immunotherapy

    2016, Journal of Allergy and Clinical Immunology
    Citation Excerpt :

    Statistical and graphic analyses were performed with Prism 6 software (GraphPad Software, La Jolla, Calif). ROC analyses of combinations of markers were performed with the mROC program.25 We first defined optimal culture conditions to polarize immature MoDCs toward a DC2 profile capable of promoting the differentiation of naive CD4+ T lymphocytes into TH2 cells.

  • Radiation-induced CD8 T-lymphocyte Apoptosis as a Predictor of Breast Fibrosis After Radiotherapy: Results of the Prospective Multicenter French Trial

    2015, EBioMedicine
    Citation Excerpt :

    Stata was used for all statistical analyses (version 13.0) and the SAS macro %cif was used for Gray's test. To complement analysis, receiver–operator characteristic (ROC) curve analyses for RILA were performed to identify patients who experienced at least a grade 2 bf + within three years (Kramar et al., 2001). The empirical areas under the ROC curves (AUC) and the respective 95%CI were used for RILA to determine the sensitivity, specificity, positive (PPV), and negative predictive value (NPV).

  • The β5/focal adhesion kinase/glycogen synthase kinase 3β integrin pathway in high-grade osteosarcoma: A protein expression profile predictive of response to neoadjuvant chemotherapy

    2013, Human Pathology
    Citation Excerpt :

    Multivariate analysis was based on receiver operating characteristic curves, which allow characterization of the discrimination between 2 well-defined populations. As previously described [24,25], the generalized receiver operating characteristic criterion finds the best linear combination of the tumor markers such that the area under the curve (AUC) is maximized. Sensitivity, which represents its ability to detect the diseased population, and specificity, which represents its ability to detect the responder population, for individual and combined marker performance, were evaluated with the use of the optimal threshold value calculated to maximize the Youden index.

View all citing articles on Scopus
View full text