Skip to main content
SpringerLink
Log in

Lessons from Past, Current Issues, and Future Research Directions in Extracting the Knowledge Embedded in Artificial Neural Networks

  • Conference paper
Hybrid Neural Systems (Hybrid Neural Systems 1998)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 1778))

Included in the following conference series:

Abstract

Active research into processes and techniques for extracting the knowledge embedded within trained artificial neural networks has continued unabated for almost ten years. Given the considerable effort invested to date, what progress has been made? What lessons have been learned? What direction should the field take from here? This paper seeks to answer these questions. The focus is primarily on techniques for extracting rule-based explanations from feed-forward ANNs since, to date, the preponderance of the effort has been expended in this arena. However the paper also briefly reviews the broadening overall agenda for ANN knowledge-elicitation. Finally the paper identifies some of the key research questions including the search for criteria for deciding in which problem domains these techniques are likely to out-perform techniques such as Inductive Decision Trees.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alexander, J.A., Mozer, M.C.: Template-Based Procedures for Neural Network Interpretation. Neural Networks 12, 479–498 (1999)

    Article  Google Scholar 

  2. Andrews, R., Diederich, J., Tickle, A.B.: A Survey and Critique of Techniques for Extracting Rules from Trained Artificial Neural Networks. Knowledge-Based Systems 8(6), 373–389 (1995)

    Article  Google Scholar 

  3. Andrews, R., Cable, R., Diederich, J., Geva, S., Golea, M., Hayward, R., Ho-Stuart, C., Tickle, A.B.: An Evaluation and Comparison of Techniques for Extracting and Refining Rules from Artificial Neural Networks. Technical report, Queensland University of Technology, Australia (1996)

    Google Scholar 

  4. Banerjee, M., Mitra, S., Pal, S.K.: Rough Fuzzy MLP: Knowledge Encoding and Classification. IEEE Transactions on Neural Networks 9(6), 1203–1216 (1998)

    Article  Google Scholar 

  5. Bologna, G., Pellegrini, C.: Symbolic Rule Extraction from Modular Transparent Boxes. In: Proceedings of the Conference of Neural Networks and their Applications (NEURAP), pp. 393–398 (1998)

    Google Scholar 

  6. Bologna, G., Pellegrini, C.: Rule-Extraction from the Oblique Multi-layer Perceptron. In: Proceedings of the Australian Conference on Neural Networks (ACNN 1998), University of Queensland, Australia, pp. 260–264 (1998)

    Google Scholar 

  7. Bologna, G.: Symbolic Rule Extraction from the DIMLP Neural Network. In: Sun, R., Wermter, S. (eds.) Hybrid Neural Systems 1998. LNCS, vol. 1778, pp. 240–254. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  8. Carpenter, G.A., Tan, A.W.: Rule Extraction: From Neural Architecture to Symbolic Representation. Connection Science 7(1), 3–27 (1995)

    Article  Google Scholar 

  9. Craven, M., Shavlik, J.W.: Using Sampling and Queries to Extract Rules From Trained Neural Networks. In: Machine Learning: Proceedings of the Eleventh International Conference, pp. 73–80. Morgan-Kaufmann, Amherst MA (1994)

    Google Scholar 

  10. Craven, M.: Extracting Comprehensible Models from Trained Neural Networks. PhD Thesis, University of Wisconsin, Madison Wisconsin (1996)

    Google Scholar 

  11. Fu, L.M.: Rule Generation from Neural Networks. IEEE Transactions on Systems, Man and Cybernetics 28(8), 1114–1124 (1994)

    Google Scholar 

  12. Fu, L.M.: A Neural Network Model for Learning Domain Rules Based on its Activation Function Characteristics. IEEE Transactions on Neural Networks 9(5), 787–795 (1998)

    Article  Google Scholar 

  13. Fu, L.M.: A Neural Network for Learning Domain Rules with Precision. In: Proceedings of the International Joint Conference on Neural Networks, IJCNN 1999 (1999) (to appear)

    Google Scholar 

  14. Giles, C.L., Miller, C.B., Chen, D., Chen, H., Sun, Z., Lee, Y.C.: Learning and Extracting Finite State Automata with Second-order Recurrent Neural Networks. Neural Computation 4, 393–405 (1992)

    Article  Google Scholar 

  15. Giles, C.L., Omlin, C.W.: Rule Refinement with Recurrent Neural Networks. In: Proceedings of the IEEE International Conference on Neural Networks, San Francisco, CA, pp. 801–806 (1993)

    Google Scholar 

  16. Giles, C.L., Omlin, C.W.: Extraction, Insertion, and Refinement of Symbolic Rules in Dynamically Driven Recurrent Networks. Connection Science 5(3-4), 307–328 (1993)

    Article  Google Scholar 

  17. Giles, C.L., Omlin, C.W.: Rule Revision with Recurrent Networks. IEEE Transactions on Knowledge and Data Engineering 8(1), 183 (1996)

    Article  Google Scholar 

  18. Giles, C.L., Lawrence, S., Tsoi, A.C.: Rule Inference for Financial Prediction using Recurrent Neural Networks. In: Proceedings of the IEEE/IAFE Conference on Computational Intelligence for Financial Engineering (CIFEr), pp. 253–259. IEEE, Piscataway NJ (1997)

    Chapter  Google Scholar 

  19. Golea, M.: On the Complexity of Rule Extraction from Neural Networks and Network Querying. In: Proceedings of the Rule Extraction From Trained Artificial Neural Networks Workshop, Society For the Study of Artificial Intelligence and Simulation of Behavior Workshop Series (AISB 1996), University of Sussex, Brighton, UK, pp. 51–59 (1996)

    Google Scholar 

  20. Healy, M.J.: A Topological Semantics for Rule Extraction with Neural Networks. Connection Science 11(1), 91–113 (1999)

    Article  Google Scholar 

  21. Humphrey, M., Cunningham, S.J., Witten, I.H.: Knowledge Visualization Techniques for Machine Learning. Intelligent Data Analysis 2(4) (1998)

    Google Scholar 

  22. Ivanova, I., Kubat, M.: Initialisation of Neural Networks by Means of Decision Trees. Knowledge Based Systems 8(6), 333–344 (1995)

    Article  Google Scholar 

  23. Krishnan, R.: A Systematic Method for Decompositional Rule Extraction From Neural Networks. In: Proceedings of the NIPS 1996 Rule Extraction From Trained Artificial Neural Networks Workshop, Queensland University of Technology, pp. 38–45 (1996)

    Google Scholar 

  24. Krishnan, R., Sivakumar, G., Battacharya, P.: A Search Technique for Rule Extraction from Trained Neural Networks. Pattern Recognition Letters 20, 273–280 (1999)

    Article  MATH  Google Scholar 

  25. Kubat, M.: Decision Trees Can Initialize Radial-Basis Function Networks. IEEE Transactions on Neural Networks 9(5), 813–821 (1998)

    Article  MathSciNet  Google Scholar 

  26. Lin, C.-T., Lee, C.S.G.: Neural Fuzzy Systems: a Neuro-Fuzzy Synergism to Intelligent Systems. Prentice-Hall, Upper Saddle River NJ (1996)

    Google Scholar 

  27. Maire, F.: A Partial Order for the M-of-N Rule-Extraction Algorithm. IEEE Transactions on Neural Networks 8(6), 1542–1544 (1997)

    Article  Google Scholar 

  28. Maire, F.: Rule-Extraction by Backpropagation of Polyhedra. Journées Francophones sur l’Apprentissage Automatique (JFA 1998), Arras France (May1998).

    Google Scholar 

  29. McGarry, K., Wermter, S., MacIntyre, J.: Knowledge Extraction from Radial Basis Function Networks. In: Proceedings of the International Joint Conference on Neural Networks, IJCNN 1999 (1999) (to appear)

    Google Scholar 

  30. Meneganti, M., Saviello, S., Tagliaferri, R.: Fuzzy Neural Networks for Classification and Detection of Anomalies. IEEE Transactions on Neural Networks 9(5), 848–861 (1998)

    Article  Google Scholar 

  31. Michie, D., Spiegelhalter, D.L., Taylor, C.C.: Machine Learning. Neural and Statistical Classification. Hertfordshire Ellis, Horwood (1994)

    Google Scholar 

  32. Neumann, J.: Classification and Evaluation of Algorithms for Rule Extraction From Artificial Neural Networks A Review. Centre for Cognitive Science, University of Edinburgh (1998) (unpublished)

    Google Scholar 

  33. Omlin, C.W., Giles, C.L.: Extraction of Rules from Discrete-Time Recurrent Neural Networks. Connection Science 5(3-4), 307–336 (1993)

    Google Scholar 

  34. Omlin, C.W., Thornber, K., Giles, C.L.: Fuzzy Finite State Automata Can be Deterministically Encoded Into Recurrent Neural Networks. IEEE Transactions on Fuzzy Systems (to appear)

    Google Scholar 

  35. Optiz, D.W., Shavlik, J.W.: Dynamically Adding Symbolically Meaningful Nodes to Knowledge-Based Neural Networks. Knowledge-Based Systems 8(6), 301–311 (1995)

    Article  Google Scholar 

  36. Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, San Mateo (1993)

    Google Scholar 

  37. Quinlan, J.R.: Comparing Connectionist and Symbolic Learning Methods. In: Rivest, R. (ed.) Computational Learning Theory and Natural Learning Systems: Constraints and Prospects, pp. 445–456. MIT Press, Cambridge (1994)

    Google Scholar 

  38. Saito, K., Nakano, R.: Law Discovery Using Neural Networks. In: Proceedings of the NIPS 1996 Rule Extraction From Trained Artificial Neural Networks Workshop, pp. 62–69. Queensland University of Technology (1996)

    Google Scholar 

  39. Schellhammer, I., Diederich, J., Towsey, M., Brugman, C.: Knowledge Extraction and Recurrent Neural Networks: an analysis of an Elman network trained on a natural language learning task. Queensland University of Technology, NRC Technical Report, 97–151 (1997)

    Google Scholar 

  40. Setiono, R., Huan, L.: NeuroLinear: From Neural Networks to Oblique Decision Rules. Neurocomputing 17, 1–24 (1997)

    Article  Google Scholar 

  41. Setiono, R.: Extracting Rules from Neural Networks by Pruning and Hidden Unit Splitting. Neural Computation 9, 205–225 (1997)

    Article  MATH  Google Scholar 

  42. Setiono, R., Thong, J.Y.L., Yap, C.S.: Symbolic Rule Extraction from Neural Networks: An Application to Identifying Organisations Adopting IT. Information and Management 34(2), 91–101 (1998)

    Article  Google Scholar 

  43. Sun, R., Sessions, C.: Extracting Plans from Reinforcement Learners. In: Proceedings of the International Symposium on Intelligent Data Engineering and Learning (IDEAL 1998), Springer, Heidelberg (1998)

    Google Scholar 

  44. Sun, R., Peterson, T.: Autonomous Learning of Sequential Tasks: Experiments and Analyses. IEEE Transactions on Neural Networks 9(6), 1217–1234 (1998)

    Article  Google Scholar 

  45. Sun, R.: Knowledge Extraction from Reinforcement Learning. In: Proceedings of the International Joint Conference on Neural Networks, IJCNN 1999 (1999) (to appear)

    Google Scholar 

  46. Thrun, S.B., Bala, J., Bloedorn, E., Bratko, I., Cestnik, B., Cheng, J., De Jong, K., Dzeroski, S., Fahlman, S.E., Fisher, D., Hamann, R., Kaufman, K., Keller, S., Kononenko, I., Kreuziger, J., Michalski, R.S., Mitchell, T., Pachowicz, P., Reich, Y., Vafaie, H., Van de Welde, K., Wenzel, W., Wnek, J., Zhang, J.: The MONK’s Problems: a Performance Comparison of Different Learning Algorithms. Carnegie Mellon University, Technical report CMU-CS-91-197 (1991)

    Google Scholar 

  47. Thrun, S.B.: Extracting Provably Correct Rules From Artificial Neural Networks. Technical Report IAI-TR-93-5 Institut fur Informatik III Universitat Bonn (1994)

    Google Scholar 

  48. Tickle, A.B., Hayward, R., Diederich, J.: Recent Developments in Techniques for Extracting Rules from Trained Artificial Neural Networks. Herbstschule Konnektionismus (HeKonn 1996), Munster (October 1996)

    Google Scholar 

  49. Tickle, A.B., Andrews, R., Golea, M., Diederich, J.: Rule Extraction from Trained Artificial Neural Networks. In: Browne, A. (ed.) Neural Network Analysis, pp. 61–99. Architectures and Applications Institute of Physics Publishing, Bristol (1997)

    Google Scholar 

  50. Tickle, A.B., Golea, M., Hayward, R., Diederich, J.: The Truth is in There: Current Issues in Extracting Rules from Trained Feed-Forward Artificial Neural Networks. In: Proceedings of the 1997 IEEE International Conference on Neural Networks (ICNN 1997), vol. 4, pp. 2530–2534 (1997)

    Google Scholar 

  51. Tickle, A.B.: Machine Learning, Neural Networks and Information Security: Techniques for Extracting Rules from Trained Feed-Forward Artificial Neural Networks and their Application in an Information Security Problem Domain. PhD Dissertation, Queensland University of Technology (1997)

    Google Scholar 

  52. Tickle, A.B., Andrews, R., Golea, M., Diederich, J.: The Truth will come to Light: Directions and Challenges in Extracting the Knowledge Embedded within Trained Artificial Neural Networks. IEEE Transactions on Neural Networks 9(6), 1057–1068 (1998)

    Article  Google Scholar 

  53. Tickle, A.B., Maire, F., Bologna, G., Diederich, J.: Extracting the Knowledge Embedded within Trained Artificial Neural Networks: Defining the Agenda. In: Proceedings of the Third International ICSC Symposia on Intelligent Industrial Automation (IIA 1999), and Soft Computing (SOCO 1999), pp. 732–738 (1999)

    Google Scholar 

  54. Towell, G., Shavlik, J.: The Extraction of Refined Rules From Knowledge Based Neural Networks. Machine Learning 131, 71–101 (1993)

    Google Scholar 

  55. Valiant, L.G.: A theory of the learnable. Communications of the ACM 27(11), 1134–1142 (1984)

    Article  MATH  Google Scholar 

  56. Williams, R.J., Zipser, D.: A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computation 1(2), 270–280 (1989)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Machine Learning Research Centre, Queensland University of Technology, Box 2434 GPO, Brisbane, Queensland, 4001, Australia

    Alan B. Tickle, Frederic Maire, Guido Bologna, Robert Andrews & Joachim Diederich

Authors
  1. Alan B. Tickle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frederic Maire
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guido Bologna
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Robert Andrews
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Joachim Diederich
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of Sunderland, SR6 0DD, Sunderland, United Kingdom

    Stefan Wermter

  2. CECS Department, University of Missouri-Colombia , 201 Engineering Building West, MO 65211-2060, Columbia, USA

    Ron Sun

Rights and permissions

Reprints and permissions

Copyright information

© 2000 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Tickle, A.B., Maire, F., Bologna, G., Andrews, R., Diederich, J. (2000). Lessons from Past, Current Issues, and Future Research Directions in Extracting the Knowledge Embedded in Artificial Neural Networks. In: Wermter, S., Sun, R. (eds) Hybrid Neural Systems. Hybrid Neural Systems 1998. Lecture Notes in Computer Science(), vol 1778. Springer, Berlin, Heidelberg. https://doi.org/10.1007/10719871_16

Download citation

  • DOI: https://doi.org/10.1007/10719871_16

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-67305-7

  • Online ISBN: 978-3-540-46417-4

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation