Skip to main content
Springer Nature Link
Log in

Evolutionary response surfaces for classification: an interpretable model

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Response surfaces are powerful tools for both classification and regression because they are able to model many different phenomena and construct complex boundaries between classes. With very simple expressions, response surfaces are able to accurately solve difficult problems. Thus, the interpretability of the results is very interesting from the point of view of the expert, which is provided by a classification model from which useful information may be inferred.

However, response surfaces suffer from a major problem that limits their applicability. Even with a low degree and a moderate number of features, the number of terms in the surfaces is extremely large. Thus, standard learning algorithms find many problems to efficiently obtain the coefficients of the terms, and the risk of overfitting is high.

To overcome this problem we present evolutionary response surfaces for the classification of two-class problems. The use of a fitness function that combines accuracy and interpretability obtains accurate classifiers that are simple and interpretable by the expert. The results obtained for 20 problems from the UCI Machine Learning Repository are comparable with well-known classification algorithms with a more interpretable polynomial function.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. Furthermore, the use of multi-objective evolutionary algorithms [57] is questionable because not all Pareto front members would be useful for our model. For instance, a non-dominated solution with high interpretability but very low accuracy cannot be considered a good solution. In such a case, we would need to resort to a multi-objective algorithm with goals and priority specifications [45]. The use of that kind of algorithm is an interesting future research line for our proposal.

  2. Because we used ten-fold cross-validation, in the following examples all of results are referred to the first partition.

References

  1. Adami H-O, Malker B, Holmberg L, Persson I, Stone B (1986) The relation between survival and age at diagnosis in breast cancer. N Engl J Med 315:559–563

    Article  Google Scholar 

  2. An S-Y, Kang J-G, mChoi W-S, Oh S-Y (2011) A neural network based retrainable framework for robust object recognition with application to mobile robotics. Appl Intell 35:190–210

    Article  Google Scholar 

  3. Assaleh K, Shanableh T (2010) Robust polynomial classifier using l 1-norm minimization. Appl Intell 33:330–339

    Article  Google Scholar 

  4. Bambha NK, Bhattacharyya SS, Teich J, Zitzler E (2004) Systematic integration of parameterized local search into evolutionary algorithms. IEEE Trans Evol Comput 8:137–155

    Article  Google Scholar 

  5. Brent M (2008) Steady progress and recent breakthroughs in the accuracy of automated gene annotation. Nat Rev Genet 9:62–73

    Article  Google Scholar 

  6. Campbell WT, Assaleh KT, Broun CC (2002) Speaker recognition with polynomial classifiers. IEEE Trans Speech Audio Process 10:205–212

    Article  Google Scholar 

  7. Cheng J, Li QS (2009) Application of the response surface methods to solve inverse reliability problems with implicit response functions. Comput Mech 43(4):451–459

    Article  Google Scholar 

  8. del Castillo-Gomariz R, García-Pedrajas N (2006) Classification by means of evolutionary response surfaces. In: Proceedings of the 14th European symposium on artificial neural networks, Bruges, Belgium, April

    Google Scholar 

  9. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30

    MathSciNet  MATH  Google Scholar 

  10. Doi K (2007) Computer-aided diagnosis in medical imaging: Historical review, current status and future potential. Comput Med Imaging Graph 31:198–211

    Article  Google Scholar 

  11. Eshelman LJ, Schaffer JD (1993) Real-coded genetic algorithms and interval-schemata. In: Whitley LD (ed) Foundations of genetic algorithms, vol 2. Morgan Kaufmann, San Mateo, pp 187–202

    Google Scholar 

  12. Ferreira SLC, Bruns RE, da Silva EGP, dos Santos WNL, Quintella CM, David JM, de Andrade JB, Breitkreitz MC, Jardim ICSF, Neto BB (2007) Statistical designs and response surface techniques for the optimization of chromatographic systems. J Chromatogr A 1158(1–2):2–14

    Google Scholar 

  13. Francis F, Sabu A, Nampoothiri KM, Ramachandran S, Ghosh S, Szakacs G, Pandey A (2003) Use of response surface methodology for optimizing process parameters for the production of α-amylase by aspergillus oryzae. Biochem Eng J 15(2):107–115

    Article  Google Scholar 

  14. Frank A, Asuncion A (2010) UCI machine learning repository

  15. García-Pedrajas N, Ortiz-Boyer D (2011) An empirical study of binary classifier fusion methods for multiclass classification. Inf Fusion 12:111–130

    Article  Google Scholar 

  16. Goethals PL, Cho BR (2011) Solving the optimal process target problem using response surface designs in heteroscedastic conditions. Int J Prod Res 49(12):3455–3478

    Article  MATH  Google Scholar 

  17. Guyon I, Weston J, Barnhill S, Vapnik V (2002) Gene selection for cancer classification using support vector machines. Mach Learn 46:389–422

    Article  MATH  Google Scholar 

  18. Gönen M, Alpaydin E (2011) Regularizing multiple kernel learning using response surface methodology. Pattern Recognit 44:159–171

    Article  MATH  Google Scholar 

  19. Haykin S (1999) Neural networks—a comprehensive foundation, 2nd edn. Prentice-Hall, Upper Saddle River

    MATH  Google Scholar 

  20. Herrera F, Lozano M, Verdegay JL (1998) Tackling real-coded genetic algorithms: operators and tools for behavioural analysis. Artif Intell Rev 12:265–319

    Article  MATH  Google Scholar 

  21. Janev M, Pekar D, Jakovljevic N, Delic V (2010) Eigenvalues driven Gaussian selection in continuous speech recognition using hmms with full covariance matrices. Appl Intell 33:107–116

    Article  Google Scholar 

  22. Jones DR (2001) A taxonomy of global optimization methods based on response surfaces. J Glob Optim 21:345–383

    Article  MATH  Google Scholar 

  23. Kalil SJ, Maugeri F, Rodrigues MI (2000) Response surface analysis and simulation as a tool for bioprocess design and optimization. Process Biochem 35(6):539–550

    Article  Google Scholar 

  24. Khuri AI, Mukhopadhyay S (2010) Response surface methodology. Comput Stat 2:128–149

    Google Scholar 

  25. Klein EJ, Rivera SL (2000) A review of criteria functions and response surface methodology for the optimization of analytical scale hplc separations. J Liq Chromatogr Relat Technol 23(14):2097–2121

    Article  Google Scholar 

  26. Kuncheva L, Jain LC (1999) Nearest neighbor classifier: simultaneous editing and descriptor selection. Pattern Recognit Lett 20:1149–1156

    Article  Google Scholar 

  27. Lew TL, Spencera AB, Scarpaa F, Wordena K, Rutherford A, Hemez F (2006) Identification of response surface models using genetic programming. Mech Syst Signal Process 20:1819–1831

    Article  Google Scholar 

  28. Liu Ch-L, Sako H (2006) Class-specific feature polynomial classifier for pattern classification and its application to handwritten numeral recognition. Pattern Recognit 39:669–681

    Article  MATH  Google Scholar 

  29. Marquardt DW (1963) An algorithm for least-squares estimation of non-linear parameters. J Soc Ind Appl Math 11(2):431–441

    Article  MathSciNet  MATH  Google Scholar 

  30. Miller GF, Todd PM, Hedge SU (1991) Designing neural networks. Neural Netw 4:53–60

    Article  Google Scholar 

  31. Minto CF, Schnider TW, Short TG, Gregg KM, Gentilini A, Shafer SL (2000) Response surface model for anesthetic drug interactions. Anesthesiology 92(6):1603–1616

    Article  Google Scholar 

  32. Myers RH, Montgomery DC (2002) Response surface methodology: process and product optimization using designed experiments, 2nd edn. Wiley, New York

    MATH  Google Scholar 

  33. Myers RH, Montgomery DC, Vining G, Borror CM, Kowalski SM (2004) Response surface methodology: a retrospective and literature survey. J Qual Technol 36(1):53–78

    Google Scholar 

  34. Nock R (2002) Inducing interpretable voting classifiers without trading accuracy for simplicity: theoretical results, approximation algorithms, and experiments. J Artif Intell Res 17:137–170

    MathSciNet  MATH  Google Scholar 

  35. Ong YS, Keane AJ (2004) Meta-Lamarckian learning in memetic algorithms. IEEE Trans Evol Comput 8:99–110

    Article  Google Scholar 

  36. Ortiz-Boyer D, Hervás-Martínez C, García-Pedrajas N (2007) Improving crossover operator for real-coded genetic algorithms using virtual parents. J Heuristics 13:265–314

    Article  Google Scholar 

  37. Park B-J, Pedrycz W, Oh S-K (2010) Polynomial-based radial basis function neural networks (p-rbf nns) and their application to pattern classification. Appl Intell 32:27–46

    Article  Google Scholar 

  38. Puri S, Beg QK, Gupta R (2002) Optimization of alkaline protease production from bacillus sp. by response surface methodology. Curr Microbiol 44(4):286–290

    Article  Google Scholar 

  39. Quinlan JR (1993) C4.5: programs for machine learning. Morgan Kaufmann, San Mateo

    Google Scholar 

  40. Rawlings JO, Pantula SG, Dickey D (1998) Applied regression analysis: a research tool. Springer, New York

    Book  MATH  Google Scholar 

  41. Satapathy SCh, Chittinemi S, Krishna SM, Murthy JVR, Reddy PVGDP (2012) Kalman particle swarm optimized polynomials for data classification. Appl Math Model 36:115–126

    Article  MathSciNet  MATH  Google Scholar 

  42. Shaffer RE, Small GW (1996) Genetic algorithms for the optimization of piecewise linear discriminants. Chemom Intell Lab Syst 35:87–104

    Article  Google Scholar 

  43. Shanableh T, Assaleh K (2010) Feature modeling using polynomial classifiers and stepwise regression. Neurocomputing 73:1752–1759

    Article  Google Scholar 

  44. Sutton GC (1989) Computer-aided diagnosis: a review. Br J Surg 76:82–85

    Article  Google Scholar 

  45. Tan KCh, Khor EF, Lee TH (2003) An evolutionary algorithm with advanced goal and priority specification for multi-objective optimization. J Artif Intell Res 18:183–215

    MathSciNet  MATH  Google Scholar 

  46. Ting Ch-Y, Phon-Amnuaisuk S (2010) Optimal dynamic decision network model for scientific inquiry learning environment. Appl Intell 33:387–406

    Article  Google Scholar 

  47. Toh K-A, Tran Q-L, Srinivasan D (2004) Benchmarking a reduced multivariate polynomial pattern classifier. IEEE Trans Pattern Anal Mach Intell 26:740–755

    Article  Google Scholar 

  48. Tong L-I, Chang Y-Ch, Lin Sh-H (2011) Determining the optimal re-sampling strategy for a classification model with imbalanced data using design of experiments and response surface methodologies. Expert Syst Appl 28:4222–4227

    Article  Google Scholar 

  49. Toropov VV, Alvarez LF (1998) Application of genetic programming and response surface methodology to optimization and inverse problems. In: Tanaka M, Dulikravich GS (eds) Inverse problems in engineering mechanics. Elsevier, Oxford, pp 551–560

    Chapter  Google Scholar 

  50. Tran Q-L, Toh K-A, Srinivasan D, Wong K-L, Low ShQ-C (2005) An empirical comparison of nine pattern classifiers. IEEE Trans Syst Man Cybern, Part B, Cybern 35:1079–1091

    Article  Google Scholar 

  51. Valova I, Milano G, Bowen K, Gueorguieva N (2011) Bridging the fuzzy, neural and evolutionary paradigms for automatic target recognition. Appl Intell 35:211–225

    Article  Google Scholar 

  52. Vapnik V (1999) The nature of statistical learning theory. Springer, New York

    Google Scholar 

  53. Vladislavleva EJ, Smits GF, den Hertog D (2009) Order of nonlinearity as a complexity measure for models generated by symbolic regression via Pareto genetic programming. IEEE Trans Evol Comput 13:333–349

    Article  Google Scholar 

  54. Yeun YS, Yang YS, Ruy WS, Kim BJ (2005) Polynomial genetic programming for response surface modeling part 1: a methodology. Struct Multidiscip Optim 29:19–34

    Article  MathSciNet  MATH  Google Scholar 

  55. Yeun YS, Kim BJ, Yang YS, Ruy WS (2005) Polynomial genetic programming for response surface modeling part 2: adaptive approximate models with probabilistic optimization problems. Struct Multidiscip Optim 29:35–494

    Article  MathSciNet  MATH  Google Scholar 

  56. Zhao X, Gao X-Sh, Hu Z-Ch (2007) Evolutionary programming based on non-uniform mutation. Appl Math Comput 192:1–11

    Article  MathSciNet  MATH  Google Scholar 

  57. Zitzler E, Thiele L, Laumanns M, Fonseca CM, Grunert V (2003) Performance assessment of multiobjective optimizers: an analysis and review. IEEE Trans Evol Comput 7(2):117–132

    Article  Google Scholar 

  58. Özcan E, Başaran C (2009) A case study of memetic algorithms for constraint optimization. Soft Comput 13(8–9):871–882

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computing and Numerical Analysis, University of Cordoba, Campus Universitario de Rabanales, 14071, Córdoba, Spain

    Rafael del Castillo-Gomariz & Nicolás García-Pedrajas

Authors
  1. Rafael del Castillo-Gomariz
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Nicolás García-Pedrajas
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Nicolás García-Pedrajas.

Additional information

This work was supported in part by the Project TIN2008-03151 of the Spanish Ministry of Science and Innovation and the project P09-TIC-4623 of the Junta de Andalucía.

Rights and permissions

Reprints and permissions

About this article

Cite this article

del Castillo-Gomariz, R., García-Pedrajas, N. Evolutionary response surfaces for classification: an interpretable model. Appl Intell 37, 463–474 (2012). https://doi.org/10.1007/s10489-012-0340-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-012-0340-5

Keywords