Abstract
Random projection is a simple technique that has had a number of applications in algorithm design. In the context of machine learning, it can provide insight into questions such as “why is a learning problem easier if data is separable by a large margin?” and “in what sense is choosing a kernel much like choosing a set of features?” This talk is intended to provide an introduction to random projection and to survey some simple learning algorithms and other applications to learning based on it. I will also discuss how, given a kernel as a black-box function, we can use various forms of random projection to extract an explicit small feature space that captures much of what the kernel is doing. This talk is based in large part on work in [BB05, BBV04] joint with Nina Balcan and Santosh Vempala.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Achlioptas, D.: Database-friendly random projections. Journal of Computer and System Sciences 66(4), 671–687 (2003)
Arriaga, R.I., Vempala, S.: An algorithmic theory of learning, robust concepts and random projection. In: Proceedings of the 40th Annual IEEE Symposium on Foundation of Computer Science, pp. 616–623 (1999)
Balcan, M.-F., Blum, A.: A PAC-style model for learning from labeled and unlabeled data. In: Proceedings of the 18th Annual Conference on Computational Learning Theory (COLT), pp. 111–126 (2005)
Balcan, M.-F., Blum, A.: On a theory of kernels as similarity functions (mansucript, 2006)
Balcan, M.F., Blum, A., Vempala, S.: Kernels as features: On kernels, margins, and low-dimensional mappings. In: Ben-David, S., Case, J., Maruoka, A. (eds.) ALT 2004. LNCS, vol. 3244, pp. 194–205. Springer, Heidelberg (2004), An extended version is available at: http://www.cs.cmu.edu/~avrim/Papers/
Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: Proceedings of the Fifth Annual Workshop on Computational Learning Theory (1992)
Ben-Israel, A., Greville, T.N.E.: Generalized Inverses: Theory and Applications. Wiley, New York (1974)
Block, H.D.: The perceptron: A model for brain functioning. Reviews of Modern Physics 34, 123–135 (1962), Reprinted in Neurocomputing, Anderson and Rosenfeld
Bartlett, P., Shawe-Taylor, J.: Generalization performance of support vector machines and other pattern classifiers. In: Advances in Kernel Methods: Support Vector Learning. MIT Press, Cambridge (1999)
Cortes, C., Vapnik, V.: Support-vector networks. Machine Learning 20(3), 273–297 (1995)
Dasgupta, S.: Experiments with random projection. In: Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 143–151 (2000)
Dasgupta, S., Gupta, A.: An elementary proof of the Johnson-Lindenstrauss Lemma. Random Structures & Algorithms 22(1), 60–65 (2002)
Rabani, Y., Kushilevitz, E., Ostrovsky, R.: Efficient search for approximate nearest neighbor in high dimensional spaces. SIAM J. Computing 30(2), 457–474 (2000)
Fradkin, D., Madigan, D.: Experiments with random projections for machine learning. In: KDD 2003: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 517–522 (2003)
Freund, Y., Schapire, R.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55(1), 119–139 (1997)
Freund, Y., Schapire, R.E.: Large margin classification using the Perceptron algorithm. Machine Learning 37(3), 277–296 (1999)
Goal, N., Bebis, G., Nefian, A.: Face recognition experiments with random projection. In: Proceedings SPIE, vol. 5779, pp. 426–437 (2005)
Goemans, M.X., Williamson, D.P.: Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM, 1115–1145 (1995)
Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: Proceedings of the 30th Annual ACM Symposium on Theory of Computing, pp. 604–613 (1998)
Johnson, W.B., Lindenstrauss, J.: Extensions of Lipschitz mappings into a Hilbert space. In: Conference in Modern Analysis and Probability, pp. 189–206 (1984)
Littlestone, N.: From on-line to batch learning. In: COLT 1989: Proceedings of the 2nd Annual Workshop on Computational Learning Theory, pp. 269–284 (1989)
Muller, K.R., Mika, S., Ratsch, G., Tsuda, K., Scholkopf, B.: An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks 12, 181–201 (2001)
Minsky, M., Papert, S.: Perceptrons: An Introduction to Computational Geometry. The MIT Press, Cambridge (1969)
Novikoff, A.B.J.: On convergence proofs on perceptrons. In: Proceedings of the Symposium on the Mathematical Theory of Automata, vol. XII, pp. 615–622 (1962)
Schapire, R.E.: The strength of weak learnability. Machine Learning 5(2), 197–227 (1990)
Schulman, L.: Clustering for edge-cost minimization. In: Proceedings of the 32nd Annual ACM Symposium on Theory of Computing, pp. 547–555 (2000)
Shawe-Taylor, J., Bartlett, P.L., Williamson, R.C., Anthony, M.: Structural risk minimization over data-dependent hierarchies. IEEE Trans. on Information Theory 44(5), 1926–1940 (1998)
Vapnik, V.N.: Statistical Learning Theory. John Wiley and Sons Inc., New York (1998)
Vempala, S.: Random projection: A new approach to VLSI layout. In: Proceedings of the 39th Annual IEEE Symposium on Foundation of Computer Science, pp. 389–395 (1998)
Vempala, S.: The Random Projection Method. In: American Mathematical Society. DIMACS: Series in Discrete Mathematics and Theoretical Computer Science (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Blum, A. (2006). Random Projection, Margins, Kernels, and Feature-Selection. In: Saunders, C., Grobelnik, M., Gunn, S., Shawe-Taylor, J. (eds) Subspace, Latent Structure and Feature Selection. SLSFS 2005. Lecture Notes in Computer Science, vol 3940. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11752790_3
Download citation
DOI: https://doi.org/10.1007/11752790_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-34137-6
Online ISBN: 978-3-540-34138-3
eBook Packages: Computer ScienceComputer Science (R0)