Abstract
This paper presents a pruning algorithm based on optimal brain surgeon (OBS) for general dynamic neural networks (GDNN). The pruning algorithm uses Hessian information and considers the order of time delay for saliency calculation. In GDNNs all layers have feedback connections with time delays to the same and to all other layers. The parameters are trained with the Levenberg-Marquardt (LM) algorithm. Therefore the Jacobian matrix is required. The Jacobian is calculated by real time recurrent learning (RTRL). As both LM and OBS need Hessian information, a rational implementation is suggested.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Le Cun, Y., Denker, J.S., Solla, S.A.: Optimal Brain Damage. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems, pp. 598–605. Morgan Kaufmann, San Francisco (1990)
Hassibi, B., Stork, D.G., Wolff, G.J.: Optimal Brain Surgeon and General Network Pruning. In: IEEE International Conference on Neural Networks, vol. 1, pp. 293–299 (April 1993)
Reed, R.: Pruning Algorithms – A Survey. IEEE Transactions on Neural Networks 4(5), 740–747 (1993)
Attik, M., Bougrain, L., Alexandre, F.: Optimal Brain Surgeon Variants For Feature Selection. In: Proceedings of the International Joint Conference on Neural Networks, pp. 1371–1374. IEEE, Los Alamitos (2004)
De Jesús, O., Hagan, M.: Backpropagation Algorithms Through Time for a General Class of Recurrent Network. In: IEEE Int. Joint Conf. Neural Network, pp. 2638–2643. IEEE Computer Society Press, Washington (2001)
De Jesús, O., Hagan, M.: Forward Perturbation Algorithm For a General Class of Recurrent Network. In: IEEE Int. Joint Conf. Neural Network, pp. 2626–2631. IEEE Computer Society Press, Washington (2001)
De Jesús, O.: Training General Dynamic Neural Networks. Ph.D. dissertation, Oklahoma State University, Stillwater, OK (2002)
De Jesús, O., Hagan, M.: Backpropagation Algorithms for a Broad Class of Dynamic Networks. IEEE Transactions on Neural Networks 18(1), 14–27 (2007)
Hagan, M., Mohammed, B.M.: Training Feedforward Networks with the Marquardt Algorithm. IEEE Transactions on Neural Networks 5(6), 989–993 (1994)
Ngia, L.S.H., Sjöberg, J.: Efficient Training of Neural Nets for Nonlinear Adaptive Filtering Using a Recursive Levenberg-Marquardt Algorithm. IEEE Transactions on Signal Processing 48(7), 1915–1927 (2000)
Werbos, P.J.: Backpropagation Through Time: What it is and how to it. Proc. IEEE 78(10), 1550–1560 (1990)
Williams, R.J., Zipser, D.: A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computing 1, 270–280 (1989)
Nelles, O.: Nonlinear System Identification. Springer, New York (2001)
Schröder, D.: Elektrische Antriebe - Regelung von Antriebssystemen, 2nd edn. Springer, Heidelberg (2001)
Narendra, K.S., Parthasarathy, K.: Identification and Control of Dynamical Systems Using Neural Networks. IEEE Transactions on Neural Networks 1(1), 4–27 (1990)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Endisch, C., Hackl, C., Schröder, D. (2007). Optimal Brain Surgeon for General Dynamic Neural Networks. In: Neves, J., Santos, M.F., Machado, J.M. (eds) Progress in Artificial Intelligence. EPIA 2007. Lecture Notes in Computer Science(), vol 4874. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-77002-2_2
Download citation
DOI: https://doi.org/10.1007/978-3-540-77002-2_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-77000-8
Online ISBN: 978-3-540-77002-2
eBook Packages: Computer ScienceComputer Science (R0)