Skip to main content
SpringerLink
Log in

A mixed integer programming algorithm for minimizing the training sample misclassification cost in two-group classification

  • Published:
Annals of Operations Research Aims and scope Submit manuscript

Abstract

In this paper, we introduce the Divide and Conquer (D&C) algorithm, a computationally attractive algorithm for determining classification rules which minimize the training sample misclassification cost in two-group classification. This classification rule can be derived using mixed integer programming (MIP) techniques. However, it is well-documented that the complexity of MIP-based classification problems grows exponentially as a function of the size of the training sample and the number of attributes describing the observations, requiring special-purpose algorithms to solve even small size problems within a reasonable computational time. The D&C algorithm derives its name from the fact that it relies, a.o., on partitioning the problem in smaller, more easily handled sub-problems, rendering it substantially faster than previously proposed algorithms. The D&C algorithm solves the problem to the exact optimal solution (i.e., it is not a heuristic that approximates the solution), and allows for the analysis of much larger training samples than previous methods. For instance, our computational experiments indicate that, on average, the D&C algorithm solves problems with 2 attributes and 500 observations more than 3 times faster, and problems with 5 attributes and 100 observations over 50 times faster than Soltysik and Yarnold's software, which may be the fastest existing algorithm. We believe that the D&C algorithm contributes significantly to the field of classification analysis, because it substantially widens the array of data sets that can be analyzed meaningfully using methods which require MIP techniques, in particular methods which seek to minimize the misclassification cost in the training sample. The programs implementing the D&C algorithm are available from the authors upon request.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. P.L. Abad and W.J. Banks, New LP based heuristics for the classification problem, European Journal of Operational Research 27(1993)88–100.

    Article  Google Scholar 

  2. J.A. Anderson, Separate sample logistic discrimination, Biometrika 59(1972)19–35.

    Article  Google Scholar 

  3. T.W. Anderson, An Introduction to Multivariate Statistical Analysis, 2nd ed., Wiley, New York, 1984.

    Google Scholar 

  4. O.K. Asparoukhov, Microprocessor system for investigation of thromboembolic complications, Unpublished Ph.D. Dissertation, Technical University of Sofia, Bulgaria, 1985 (in Bulgarian).

  5. S.M. Bajgier and A. Hill, An experimental comparison of statistical and linear programming approaches to the discriminant problem, Decision Sciences 13(1982)604–618.

    Google Scholar 

  6. W.J. Banks and P.L. Abad, An efficient optimal solution algorithm for the classification problem, Decision Sciences 22(1991)1008–1023.

    Google Scholar 

  7. W.J. Banks and P.L. Abad, On the performance of linear programming heuristics applied on a quadratic transformation in the classification problem, European Journal of Operational Research 74(1994)23–28.

    Article  Google Scholar 

  8. T.S. Campbell and J.K. Dietrich, The determinants of default on insured conventional residential mortgage loans, Journal of Finance 38(1983)1569–1581.

    Article  Google Scholar 

  9. N. Capon, Credit scoring systems: A critical analysis, Journal of Marketing 46(1982)82–91.

    Article  Google Scholar 

  10. C. Chen, Hybrid misclassification minimisation, Paper presented at the National INFORMS Meeting, Washington, DC, May 1996.

  11. A.P. Duarte Silva, Minimizing misclassification costs in two-group classification analysis, Unpublished Ph.D. Dissertation, The University of Georgia, 1995.

  12. A.P. Duarte Silva and A. Stam, Second order mathematical programming formulations for discriminant analysis, European Journal of Operational Research 74(1994)4–22.

    Article  Google Scholar 

  13. R.A. Eisenbeis, Pitfalls in the application of discriminant analysis, Journal of Finance 32(1977) 875–900.

    Article  Google Scholar 

  14. L.P. Fatti, D.M. Hawkins and E.L. Raath, Discriminant analysis, in: Topics in Applied Multivariate Analysis, D.M. Hawkins, ed., Cambridge University Press, Cambridge, England, 1982, pp. 1–77.

    Google Scholar 

  15. R.A. Fisher, The use of multiple measurements in taxonomy problems, Annals of Eugenics 7(1936) 179–188.

    Google Scholar 

  16. N. Freed and F. Glover, A linear programming approach to the discriminant problem, Decision Sciences 12(1981)68–74.

    Google Scholar 

  17. N. Freed and F. Glover, Simple but powerful goal programming formulations for the discriminant problem, European Journal of Operational Research 7(1981)44–60.

    Article  Google Scholar 

  18. N. Freed and F. Glover, Evaluating alternative linear programming models to solve the two-group discriminant problem, Decision Sciences 17(1986)151–162.

    Google Scholar 

  19. W.V. Gehrlein, General mathematical programming formulations for the statistical classification problem, Operations Research Letters 5(1986)299–304.

    Article  Google Scholar 

  20. L.W. Glorfeld and M.W. Kattan, A comparison of the performance of three classification procedures when applied to contaminated data, in: Proceedings of the 21st Annual Meeting of the Decision Sciences Institute, 1989, pp. 1153–1155.

  21. F. Glover, Improved linear programming models for discriminant analysis, Decision Sciences 21 (1990)771–785.

    Google Scholar 

  22. F. Glover, S. Keene and B. Duea, A new class of models for the discriminant problem, Decision Sciences 19(1988)269–280.

    Google Scholar 

  23. W. Gochet, A. Stam, V. Srinivasan and S. Chen, Multigroup discriminant analysis using linear programming, Operations Research 45(1997)213–225.

    Google Scholar 

  24. D.J. Hand, Discrimination and Classification, Wiley, New York, 1981.

    Google Scholar 

  25. F.S. Hillier and G.J. Lieberman, Introduction to Operations Research, 5th ed., McGraw-Hill, New York, 1990.

    Google Scholar 

  26. C.J. Huberty, Issues in the use and interpretation of discriminant analysis, Psychological Bulletin 95(1984)156–171.

    Article  Google Scholar 

  27. T. Ibaraki and S. Muroga, Adaptive linear classifier by linear programming, IEEE Transactions on Systems, Science and Cybernetics SSC-6(1970)53–62.

    Article  Google Scholar 

  28. International Business Machines, IBM Mathematical Programming Systems Extended/370 (MPSX/370), Mixed Integer Programming/370 (MIP/370), White Plains, New York, 1975.

    Google Scholar 

  29. E.A. Joachimsthaler and A. Stam, Four approaches to the classification problem in discriminant analysis: An experimental study, Decision Sciences 19(1988)322–333.

    Google Scholar 

  30. E.A. Joachimsthaler and A. Stam, Mathematical programming approaches for the classification problem in two-group discriminant analysis, Multivariate Behavioral Research 25(1990)427–454.

    Article  Google Scholar 

  31. L. Kim and Y. Kim, Innovation in a newly industrializing country: A multiple discriminant analysis, Management Science 31(1985)312–322.

    Article  Google Scholar 

  32. G.J. Koehler, Discriminant functions determined by genetic search, ORSA Journal on Computing 3(1991)345–357.

    Google Scholar 

  33. G.J. Koehler and S.S. Erenguc, Minimizing misclassifications in linear discriminant analysis, Decision Sciences 21(1990)63–85.

    Google Scholar 

  34. J.S. Koford and G.F. Groner, The use of an adaptive threshold element to design a linear optimal pattern classifier, IEEE Transactions on Information Theory IT-12(1966)42–50.

    Article  Google Scholar 

  35. P.A. Lachenbruch, C. Sneeringer and L.T. Revo, Robustness of the linear and quadratic discriminant function to certain types of non-normality, Communications in Statistics 1(1973)39–57.

    Article  Google Scholar 

  36. J.M. Liitschwager and C. Wang, Integer programming solution of a classification problem, Management Science 24(1978)1515–1525.

    Google Scholar 

  37. M.A. Mahmood and E.C. Lawrence, A performance analysis of parametric and nonparametric discriminant approaches to business decision making, Decision Sciences 18(1987)308–326.

    Google Scholar 

  38. O.L. Mangasarian, Linear and nonlinear separation of patterns by linear programming, Operations Research 13(1965)444–452.

    Google Scholar 

  39. C.A. Markowski and E.P. Markowski, Some difficulties and improvements in applying linear programming formulations to the discriminant problem, Decision Sciences 16(1985)237–247.

    Google Scholar 

  40. C.A. Markowski and E.P. Markowski, An experimental comparison of several approaches to the discriminant problem with both qualitative and quantitative variables, European Journal of Operational Research 28(1987)74–78.

    Article  Google Scholar 

  41. G.J. McLachlan, Discriminant Analysis and Statistical Pattern Recognition, Wiley, New York, 1992.

    Book  Google Scholar 

  42. D.F. Morrison, Multivariate Statistical Methods, 3rd ed., McGraw-Hill, New York, 1990.

    Google Scholar 

  43. G.E. Pinches and K.A. Mingo, A multivariate analysis of industrial bond ratings, Journal of Finance 28(1973)1–18.

    Article  Google Scholar 

  44. C.T. Ragsdale and A. Stam, An efficient heuristic method for minimizing the number of misclassified observations in discriminant analysis, Working Paper, Terry College of Business, The University of Georgia, 1991.

  45. V. Ramanujan, N. Venkatraman and J.C. Camillus, Multi-objective assessment of effectiveness of strategic planning: A discriminant analysis approach, Academy of Management Journal 29(1986) 347–372.

    Article  Google Scholar 

  46. P.A. Rubin, Heuristic solution procedures for a mixed-integer programming discriminant model, Managerial and Decision Economics 11(1990)255–266.

    Google Scholar 

  47. P.A. Rubin, A comment regarding polynomial discriminant analysis, European Journal of Operational Research 72(1994)29–31.

    Article  Google Scholar 

  48. L. Schrage, LINDO: User's Manual, Release 5.0, The Scientific Press, South San Francisco, CA, 1991.

    Google Scholar 

  49. C.A.B. Smith, Some examples of discrimination, Annals of Eugenics 13(1947)272–282.

    Google Scholar 

  50. F.W. Smith, Pattern classifier design by linear programming, IEEE Transactions on Computers C-17(1968)367–372.

    Google Scholar 

  51. R.C. Soltysik and P.R. Yarnold, Fast solutions to optimal discriminant analysis problems, presented at the TIMS/ORSA National Meeting (Invited), Orlando, FL, April 1992.

  52. R.C. Soltysik and P.R. Yarnold, ODA 1.0: Optimal Discriminant Analysis for DOS, Optimal Data Analysis, Chicago, IL, 1993.

    Google Scholar 

  53. R.C. Soltysik and P.R. Yarnold, The Warmack-Gonzalez algorithm for linear two-category multivariate optimal discriminant analysis, Computers and Operations Research 21(1994)735–745.

    Article  Google Scholar 

  54. D.J. Spiegelhalter and R.P. Knill-Jones, Statistical and knowledge-based approaches to clinical decision-support systems, with an application to gastroenterology, Journal of the Royal Statistical Society, Series A 147, Part 1 (1984)35–77.

    Article  Google Scholar 

  55. V. Srinivasan and Y.H. Kim, Credit granting: A comparative analysis of classification procedures, Journal of Finance 42(1987)665–683.

    Article  Google Scholar 

  56. A. Stam and E.A. Joachimsthaler, Solving the classification problem in discriminant analysis via linear and nonlinear programming methods, Decision Sciences 20(1989)285–293.

    Google Scholar 

  57. A. Stam and E.A. Joachimsthaler, A comparison of a robust mixed-integer approach to existing methods for establishing classification rules for the discriminant problem, European Journal of Operational Research 46(1990)113–120.

    Article  Google Scholar 

  58. A. Stam and D.G. Jones, Classification performance of mathematical programming techniques in discriminant analysis: Results for small and medium sample sizes, Managerial and Decision Economics 11(1990)243–253.

    Google Scholar 

  59. A. Stam and C.T. Ragsdale, On the classification gap in MP-based approaches to the discriminant problem, Naval Research Logistics 39(1992)545–559.

    Google Scholar 

  60. R.E. Warmack and R.C. Gonzalez, An algorithm for the optimal solution of linear inequalities and its application to pattern recognition, IEEE Transactions on Computers C-22(1973)1065–1075.

    Google Scholar 

  61. P.R. Yarnold, R.C. Soltysik and G.J. Martin, Heart rate variability and susceptibility for sudden cardiac death: An example of multivariable optimal discriminant analysis, Statistics in Medicine 13(1994)1015–1021.

    Google Scholar 

Download references

Authors
  1. A. Pedro Duarte Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonie Stam
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Duarte Silva, A.P., Stam, A. A mixed integer programming algorithm for minimizing the training sample misclassification cost in two-group classification. Annals of Operations Research 74, 129–157 (1997). https://doi.org/10.1023/A:1018962102794

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1018962102794

Search

Navigation