Skip to main content
SpringerLink
Log in

Neural Data Analysis: Ensemble Neural Network Rule Extraction Approach and Its Theoretical and Historical Backgrounds

(Invited Paper)

  • Conference paper
Artificial Intelligence and Soft Computing (ICAISC 2013)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 7894))

Included in the following conference series:

Abstract

In this paper, we first survey the theoretical and historical backgrounds related to ensemble neural network rule extraction. Then we propose a new rule extraction method for ensemble neural networks. We also demonstrate that the use of ensemble neural networks produces higher recognition accuracy than do individual neural networks. Because the extracted rules are more comprehensible. The rule extraction method we use is the Ensemble-Recursive-Rule eX traction (E-Re-RX) algorithm. The E-Re-RX algorithm is an effective rule extraction algorithm for dealing with data sets that mix discrete and continuous attributes. In this algorithm, primary rules are generated, followed by secondary rules to handle only those instances that do not satisfy the primary rules, and then these rules are integrated. We show that this reduces the complexity of using multiple neural networks. This method achieves extremely high recognition rates, even with multiclass problems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Breiman, L.: Bagging predictors. Mach. Learn. 24, 123–140 (1996)

    MathSciNet  MATH  Google Scholar 

  2. Freund, Y., Schapire, R.: Experiments with a new boosting algorithm. In: Proc. of the Thirteenth International Conference on Machine Learning, Bari, Italy, pp. 148–156 (1996)

    Google Scholar 

  3. Zhang, G.P.: Neural networks for classification: A Survey. IEEE Trans. Systems, Man and Cybernetics–Part C: Applications and Reviews 30(4), 451–462 (2000)

    Article  Google Scholar 

  4. Rokach, L.: Ensemble-based classifiers. Artificial Intelligence Review 33, 1–39 (2010)

    Article  Google Scholar 

  5. Yao, X., Islam, M.: Evolving artificial neural network ensembles. IEEE Computational Intelligence Magazine 3(1), 31–42 (2008)

    Article  Google Scholar 

  6. Cybenko, G.: Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems 2, 303–314 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  7. Setiono, R.: A penalty-function approach for pruning feedforward neural networks. Neural Comp. 9(1), 185–204 (1997)

    Article  MATH  Google Scholar 

  8. Bishop, C.: Neural Networks for Pattern Recognition. Oxford University Press (1995)

    Google Scholar 

  9. Setiono, R., Baesens, B., Mues, C.: Recursive neural network rule extraction for data with mixed attributes. IEEE Trans. Neural Netw. 19, 299–307 (2008)

    Article  Google Scholar 

  10. Akhand, M.A.H., Murase, K.: Neural Network Ensembles. Lambert Academic Publishing (LAP) (2010)

    Google Scholar 

  11. Alpaydin, E.: Multiple Neural Networks and Weighted Voting. IEEE Trans. on Pattern Recognition 2, 29–32 (1992)

    Google Scholar 

  12. University of California, Irvine Machine Learning Repository, http://archive.ics.uci.edu/ml/

  13. Hara, A., Hayashi, Y.: Ensemble neural network rule extraction using Re-RX algorithm. In: Proc. of WCCI (IJCNN) 2012, Brisbane, Australia, June 10-15, pp. 604–609 (2012)

    Google Scholar 

  14. Mitra, S., Hayashi, Y.: Neuro-Fuzzy Rule Generation: Survey in Soft Computing Framework. IEEE Trans. Neural Netw. 11(3), 748–768 (2000)

    Article  Google Scholar 

  15. Gallant, S.I.: Connectionist expert systems. Commun. ACM 31, 152–169 (1988)

    Article  Google Scholar 

  16. Saito, K., Nakano, R.: Medical diagnosis expert systems based on PDP model. In: Proc. IEEE Int. Conf. Neural Netw, San Diego, CA, pp. I.255–I.262 (1988)

    Google Scholar 

  17. Hayashi, Y.: Neural expert system using teaching fuzzy input and its application to medical diagnosis. Inform. Sci.: Applicat. 1, 47–58 (1994)

    MATH  Google Scholar 

  18. Hayashi, Y.: A neural expert system with automated extraction of fuzzy if-then rules and its application to medical diagnosis. In: Lippmann, R.P., Moody, J.E., Touretzky, D.S. (eds.) Advances in Neural Information Processing Systems, pp. 578–584. Morgan Kaufmann, Los Altos (1991)

    Google Scholar 

  19. Hudson, D.L., Cohen, M.E., Anderson, M.F.: Use of neural network techniques in a medical expert system. Int. J. Intell. Syst. 6, 213–223 (1991)

    Article  Google Scholar 

  20. Takagi, H.: Fusion technology of fuzzy theory and neural network—Survey and future directions. In: Proc. Int. Conf. Fuzzy Logic and Neural Networks, Iizuka, Japan, pp. 13–26 (1990)

    Google Scholar 

  21. Buckley, J.J., Hayashi, Y., Czogala, E.: On the equivalence of neural nets and fuzzy expert systems. Fuzzy Sets Syst. 53(2), 129–134 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  22. Hayashi, Y., Buckley, J.J.: Approximation between fuzzy expert systems and neural networks. Int. J. Approx. Res. 10, 63–73 (1994)

    Article  MATH  Google Scholar 

  23. Buckley, J.J., Hayashi, Y.: Numerical relationship between neural networks, continuous function and fuzzy systems. Fuzzy Sets Syst. 60(1), 1–8 (1993)

    Article  MathSciNet  Google Scholar 

  24. Buckley, J.J., Hayashi, Y.: Hybrid neural nets can be fuzzy controllers and fuzzy expert systems. Fuzzy Sets and Syst. 60, 135–142 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  25. Golea, M.: On the complexity of rule extraction from neural networks and network querying. In: Proc. the AIBS 1996 Workshop on the Rule Extraction from Trained Neural Networks, Brighton, UK, pp. 51–59 (1996)

    Google Scholar 

  26. Roy, A.: On connectionism, rule extraction, and brain-like learning. IEEE Trans. Fuzzy Systems 8(2), 222–227 (2000)

    Article  Google Scholar 

  27. Zhou, Z.-H.: Rule Extraction: Using Neural Networks or for Neural Networks? J. Comput. Sci. & Technol. 19(2), 249–253 (2004)

    Article  Google Scholar 

  28. Buckley, J.J., Hayashi, Y.: Fuzzy neural networks: A survey. Fuzzy Sets Syst. 66, 1–13 (1994)

    Article  MathSciNet  Google Scholar 

  29. Hayashi, Y., Buckley, J.J., Czogala, E.: Fuzzy neural network with fuzzy signals and weights. Int. J. Intell. Syst. 8(4), 527–573 (1993)

    Article  MATH  Google Scholar 

  30. Ishibuchi, H., Kwon, K., Tanaka, H.: A learning algorithm of fuzzy neural networks with triangular fuzzy weights. Fuzzy Sets Syst. 71, 277–293 (1995)

    Article  Google Scholar 

  31. Feuring, T., Buckley, J.J., Hayashi, Y.: A gradient descent learning algorithm for fuzzy neural networks. In: Proc. IEEE Int. Conf. Fuzzy Syst. FUZZ-IEEE 1998, Anchorage, AK, pp. 1136–1141 (1998)

    Google Scholar 

  32. Bologna, G.: Is it worth generating rules from neural network ensembles? J. of Applied Logic 2, 325–348 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  33. Bologna, G.: A study on rule extraction from several combined neural networks. Int. J. Neural Syst. 11(3), 247–255 (2001)

    Google Scholar 

  34. Bologna, G.: A model for single and multiple knowledge based networks. Artificial Intelligence in Medicine 28, 141–163 (2003)

    Article  Google Scholar 

  35. Zhou, Z.-H.: Extracting symbolic rules from trained neural network ensembles. AI Communications 16, 3–15 (2003)

    Google Scholar 

  36. Zhou, Z.-H.: Ensemble neural networks: Many could be better than all. Artificial Intelligence 137, 239–263 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  37. Hara, A., Hayashi, Y.: A new neural data analysis approach using ensemble neural network rule extraction. In: Villa, A.E.P., Duch, W., Érdi, P., Masulli, F., Palm, G. (eds.) ICANN 2012, Part I. LNCS, vol. 7552, pp. 515–522. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  38. Hansen, L.K., Salamon, P.: Neural network ensembles. IEEE Trans. Pattern Analysis and Machine Intelligence 12, 993–1001 (1990)

    Article  Google Scholar 

  39. Adeodato, P.J.L., et al.: MLP ensembles improve long term prediction accuracy over single networks. Int. J. Forecasting 27, 661–671 (2011)

    Article  Google Scholar 

  40. Hornik, K., et al.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989)

    Article  Google Scholar 

  41. Chorowski, J., Zurada, J.M.: Extracting Rules from Neural Networks as Decision Diagrams. IEEE Trans. Neural Netw. 22(12), 2435–2446 (2011)

    Article  Google Scholar 

  42. Augasta, M.G., Kathirvalavakumar, T.: Reverse engineering the neural networks for rule extraction in classification problems. Neural Processing Letters 35, 131–150 (2012)

    Article  Google Scholar 

  43. Barakat, N., Bradley, A.P.: Rule extraction from support vector machines: A review. Neurocomputing 74, 178–190 (2010)

    Article  Google Scholar 

  44. Liu, S., et al.: Combined rule extraction and feature elimination in supervised classification. IEEE Trans. Nanobioscience 11(3), 228–236 (2012)

    Article  Google Scholar 

  45. Zhou, Z.H.: Medical diagnosis with C4.5 rule preceded by artificial neural network ensemble. IEEE Trans. Information Technology in Biomedicine 7(1), 37–42 (2003)

    Article  Google Scholar 

  46. Zhou, Z.H., et al.: A statistics based approach for extracting priority rules from trained neural networks. In: Proc. IEEE-INNS-ENNS Int. Conf. Neural Netw., Como, Italy, vol. 3, pp. 401–406 (2000)

    Google Scholar 

  47. We, X., et al.: Top 10 algorithms in data mining. Knowledge and Information Systems 14, 1–17 (2008)

    Article  Google Scholar 

  48. Robert, B., et al.: Boosting of the margin: A new explanation for the effectiveness of voting methods. The Annals of Statistics 26(5), 1651–1686 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  49. Duch, W., Setiono, R., Zurada, J.M.: Computational intelligence methods for rule-based data understanding. Proceedings of the IEEE 92(5), 771–805 (2004)

    Article  Google Scholar 

  50. Tickle, A.B., et al.: The truth will come to light: Directions and challenges in extracting the knowledge embedded within trained artificial neural networks. IEEE Trans. Neural Netw. 9, 1057–1068 (1998)

    Article  Google Scholar 

  51. Lin, C.T., Lee, C.S.G.: Neural fuzzy systems—A neuro–fuzzy synergism to intelligent systems. Prentice-Hall, Englewood Cliff (1996)

    Google Scholar 

  52. Khosravi, A., et al.: Comprehensive review of neural network-based prediction intervals and new advances. IEEE Trans. Neural Netw. 22(9), 1341–1356 (2011)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Meiji University, Tama-ku, Kawasaki, 214-8571, Japan

    Yoichi Hayashi

Authors
  1. Yoichi Hayashi
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Częstochowa University of Technology, Armii Krajowej 36, 42-200, Częstochowa, Poland

    Leszek Rutkowski , Marcin Korytkowski  & Rafał Scherer ,  & 

  2. AGH University of Science and Technology, Michiewicza 30, 30-059, Kraków, Poland

    Ryszard Tadeusiewicz

  3. Department of Electrical Engineering and Computer Sciences, University of California, 94720-1776, Berkeley, CA, USA

    Lotfi A. Zadeh

  4. Electrical and Computer Engineering, University of Louisville, 405 Lutz Hall, 40292, Louisville, KY, USA

    Jacek M. Zurada

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Hayashi, Y. (2013). Neural Data Analysis: Ensemble Neural Network Rule Extraction Approach and Its Theoretical and Historical Backgrounds. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2013. Lecture Notes in Computer Science(), vol 7894. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-38658-9_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-38658-9_1

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-38657-2

  • Online ISBN: 978-3-642-38658-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation