Abstract
We focus on the recently introduced nearest neighbor based entropy estimator from Kraskov, Stögbauer and Grassberger (KSG) [10], the nearest neighbor search of which is performed by the so called box assisted algorithm [7]. We compare the performance of KSG with respect to three spatial indexing methods: box-assisted, k-D trie and projection method, on a problem of mutual information estimation of a variety of pdfs and dimensionalities. We conclude that the k-D trie method is significantly faster then box-assisted search in fixed-mass and fixed-radius neighborhood searches in higher dimensions. The projection method is much slower than both alternatives and not recommended for practical use.
The first author was supported by the 6RP EU project BRACCIA (Contract No 517133 NEST). The second author was supported by the grant of Austrian Research Fonds FWF-H-226 (2005) under Charlotte Bühler Program and by ASCR 1ET 100 750 401, Project Baddyr.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Beirlant, J., Dudewitz, E.J., Györfi, L., van der Meulen, E.C.: Nonparametric entropy estimation: An overview. Int. J. Math. And Statistical Sciences 6, 17–39 (1997)
Bentley, J.L.: Multidimensional binary search trees used in associative searching. Communications of the ACM 18(9), 509–517 (1975)
Bingham, S., Kot, M.: Multidimensional trees, range searching, and a correlation dimension algorithm of reduced complexity. Phys. Lett. A 140, 327 (1989)
Friedman, J.H., Baskett, F., Shustek, L.J.: An Algorithm for finding nearest neighbor. IEEE Transactions on Computers, 1000–1006 (1975)
Freidman, J.H., Bentley, J.L., Finkel, R.A.: An algorithm for finding best matches in logarithmic expected time. ACM Trans. Math. Softw. 3(3), 209–226 (1977)
Goria, M.N., Leonenko, N.N., Mergel, V.V., Novi Inverardi, P.L.: A new class of random vector entropy estimators and its applications in testing statistical hypotheses. Nonparametric Statistics 17(3), 277–297 (2005)
Grassberger, P.: An optimized box-assisted algorithm for fractal dimensions. Phys. Lett. A 148, 63–68 (1990)
Hyvaerinnen, A., Karhunen, J., Oja, E.: Independent Component Analysis. Wiley, Chichester (2001)
Kozachenko, L.F., Leonenko, N.N.: Sample estimate of the entropy of a random vector. Problems of Information Transmission 23(2), 95–101 (1987)
Kraskov, A., Stögbauer, H., Grassberger, P.: Estimating mutual information. Physical Review E 69, 066138 (2004)
Kraskov, A.: Synchronization and interdependence measures and their applications to the electroencephalogram of epilepsy patients and clustering of data. PhD thesis, John von Neumann Institute for Computing (2004)
Kwak, N., Choi, C.: Input feature selection by mutual information based on Parzen window. IEEE Trans. On Pattern Analysis and Machine Intellingence 24(12), 1667–1671 (2002)
Paninski, L.: Estimation of entropy and mutual information. Neural Computation 15, 1191–1253 (2003)
Schreiber, T.: Efficient neighbor searching in nonlinear time series analysis. Int. Journal of Bifurc. and Chaos 5(2), 349–358 (1995)
Shannon, C.: A mathematical theory of communication. Bell System Tech. J. 27, 379–423 (1948)
Theiler, J.: Efficient algorithm for estimation the correlation dimension from a set of discrete points. Physical Review A 36, 4456–4462 (1987)
Van Hulle, M.M.: Edgeworth approximation of multivariate differential entropy. Neural Computation 17, 1903–1910 (2005)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Vejmelka, M., Hlaváčková-Schindler, K. (2007). Mutual Information Estimation in Higher Dimensions: A Speed-Up of a k-Nearest Neighbor Based Estimator. In: Beliczynski, B., Dzielinski, A., Iwanowski, M., Ribeiro, B. (eds) Adaptive and Natural Computing Algorithms. ICANNGA 2007. Lecture Notes in Computer Science, vol 4431. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-71618-1_88
Download citation
DOI: https://doi.org/10.1007/978-3-540-71618-1_88
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-71589-4
Online ISBN: 978-3-540-71618-1
eBook Packages: Computer ScienceComputer Science (R0)