Abstract
Rare category detection (RCD) aims at finding out at least one data example of each rare category in an unlabeled data set with the help of a labeling oracle to prove the existence of such a rare category. Various approaches have been proposed for RCD with quadratic or even cubic time complexity. In this paper, by using histogram density estimation and wavelet analysis, we propose FRED algorithm and its prior-free version iFRED algorithm for RCD, both of which achieve linear time complexity w.r.t. either the data set size N or the data dimension d. Theoretical analysis guarantees its effectiveness, and comprehensive experiments on both synthetic and real data sets verify the effectiveness and efficiency of our algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Asuncion, A., Newman, D.: UCI Machine Learning Repository (2007)
Bay, S., Kumaraswamy, K., Anderle, M., Kumar, R., Steier, D.: Large scale detection of irregularities in accounting data. In: ICDM, Hong Kong, China, pp. 75–86 (2006)
Garcia-Pedrajas, N., Garcia-Osorio, C.: Boosting for class-imbalanced datasets using genetically evolved supervised non-linear projections. Progress in AI 2(1), 29–44 (2013)
He, J., Carbonell, J.: Nearest-neighbor-based active learning for rare category detection. In: NIPS 2007, Vancouver, British Columbia, Canada, December 3-6, pp. 633–640 (2007)
He, J., Carbonell, J.: Prior-free rare category detection. In: SDM 2009, Sparks, Nevada, USA, April 30-May 2, pp. 155–163 (2009)
He, J., Carbonell, J.: Coselection of features and instances for unsupervised rare category analysis. Statistical Analysis and Data Mining 3(6), 417–430 (2010)
He, J., Liu, Y., Lawrence, R.: Graph-based rare category detection. In: ICDM 2008, Pisa, Italy, December 15-19, pp. 833–838 (2008)
He, J., Tong, H., Carbonell, J.: An effective framework for characterizing rare categories. Frontiers of Computer Science 6(2), 154–165 (2012)
Hospedales, T.M., Gong, S., Xiang, T.: Finding rare classes: Active learning with generative and discriminative models. TKDE 25(2), 374–386 (2013)
Huang, H., Chiew, K., Gao, Y., He, Q., Li, Q.: Rare category exploration. Expert Systems with Applications (2014)
Huang, H., He, Q., Chiew, K., Qian, F., Ma, L.: Clover: A faster prior-free approach to rare category detection. Knowledge and Information Systems 35(3), 713–736 (2013)
Huang, H., He, Q., He, J., Ma, L.: Radar: Rare category detection via computation of boundary degree. In: Huang, J.Z., Cao, L., Srivastava, J. (eds.) PAKDD 2011, Part II. LNCS (LNAI), vol. 6635, pp. 258–269. Springer, Heidelberg (2011)
Khor, K., Ting, C., Phon-Amnuaisuk, S.: A cascaded classifier approach for improving detection rates on rare attack categories in network intrusion detection. Applied Intelligence 36(2), 320–329 (2012)
Nenadic, Z., Burdick, J.: Spike detection using the continuous wavelet transform. IEEE Transactions on Biomedical Engineering 52(1), 74–87 (2005)
Pelleg, D., Moore, A.: Active learning for anomaly and rare-category detection. In: NIPS 2004, Vancouver, British Columbia, Canada, December 13-18, pp. 1073–1080 (2004)
Scott, D.W.: Multivariate Density Estimation: Theory, Practice, and Visualization. Wiley, New York (1992)
Vatturi, P., Wong, W.: Category detection using hierarchical mean shift. In: KDD 2009, Paris, France, June 28-July 1, pp. 847–856 (2009)
Wand, M.P.: Data-based choice of histogram bin width. The American Statistician 51(1), 59–64 (1997)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Liu, Z., Huang, H., He, Q., Chiew, K., Ma, L. (2014). Rare Category Detection on O(dN) Time Complexity. In: Tseng, V.S., Ho, T.B., Zhou, ZH., Chen, A.L.P., Kao, HY. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2014. Lecture Notes in Computer Science(), vol 8444. Springer, Cham. https://doi.org/10.1007/978-3-319-06605-9_41
Download citation
DOI: https://doi.org/10.1007/978-3-319-06605-9_41
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-06604-2
Online ISBN: 978-3-319-06605-9
eBook Packages: Computer ScienceComputer Science (R0)