Peiqing Chen
ABSTRACT
Item batch denotes a consecutive sequence of identical items that are close in time in a data stream. It is a useful data stream pattern in cache, burst detection, APT detection, \etc Basic item batch measurement tasks include membership, cardinality, time span and size. Currently, there is no algorithm tailored for item batch measurement. The greatest challenge lies in accurately estimating the time gap between two consecutive identical items. In this paper, we propose Clock-sketch, a framework that introduces the well-known CLOCK algorithm into item batch measurement. The methodology of Clock-sketch is to clean outdated information as much as possible, while guaranteeing that the information of all items visited within the time window $\mathcalT $ is preserved. We conduct experiments on three real-world datasets that feature in item batch pattern. We compare the accuracy and throughput performance of our Clock-sketch against the state-of-the-art and two naive approaches without using Clock-sketch technique. Results of item batch activeness show that Clock-sketch outperforms the state-of-the-art SWAMP in generating 50 times less false positive rate when memory is small. All source codes are open-sourced and released at Github.
Supplemental Material
- Eran Assaf, Ran Ben Basat, Gil Einziger, and Roy Friedman. 2018. Pay for a sliding bloom filter and get counting, distinct elements, and entropy for free. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 2204--2212.Google Scholar
Digital Library
- Bryan Ball, Mark Flood, Hosagrahar Visvesvaraya Jagadish, Joe Langsam, Louiqa Raschid, and Peratham Wiriyathammabhum. 2014. A flexible and extensible contract aggregation framework (caf) for financial data stream analytics. In Proceedings of the International Workshop on Data Science for Macro-Modeling. 1--6.Google ScholarDigital Library
- Sorav Bansal and Dharmendra S Modha. 2004. CAR: Clock with Adaptive Replacement.. In FAST, Vol. 4. 187--200.Google ScholarDigital Library
- Burton H Bloom. 1970. Space/time trade-offs in hash coding with allowable errors. Commun. ACM , Vol. 13, 7 (1970), 422--426.Google ScholarDigital Library
- Oscar Boykin, Sam Ritchie, and Jimmy Lin. 2014. Summingbird: A Framework for Integrating Batch and Online MapReduce Computations. Proc. VLDB Endow. , Vol. 7, 13 (Aug. 2014), 1441-451. https://doi.org/10.14778/2733004.2733016Google ScholarDigital Library
- Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering , Vol. 36, 4 (2015).Google Scholar
- Fernando J Corbato. 1968 a. A paging experiment with the multics system . Technical Report. MASSACHUSETTS INST OF TECH CAMBRIDGE PROJECT MAC.Google Scholar
- Graham Cormode and Shan Muthukrishnan. 2005. An improved data stream summary: the count-min sketch and its applications. Journal of Algorithms , Vol. 55, 1 (2005), 58--75.Google ScholarDigital Library
- Michael K Daly. 2009. Advanced persistent threat. Usenix, Nov , Vol. 4, 4 (2009), 2013--2016.Google Scholar
- Gil Einziger and Roy Friedman. 2015. Counting with TinyTable: Every bit counts!. In 2015 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). IEEE, 77--78.Google ScholarCross Ref
- Lajos Gergely Gyurkó , Terry Lyons, Mark Kontkowski, and Jonathan Field. 2013. Extracting information from the signature of a financial data stream. arXiv preprint arXiv:1307.7244 (2013).Google Scholar
- Laura M. Haas, Walter Chang, Guy M. Lohman, John McPherson, Paul F. Wilms, George Lapis, Bruce Lindsay, Hamid Pirahesh, Michael J. Carey, and Eugene Shekita. 1990. Starburst mid-flight: As the dust clears. IEEE Transactions on Knowledge & Data Engineering 1 (1990), 143--160.Google ScholarDigital Library
- Thoufique Haq, Jinjian Zhai, and Vinay K Pidathala. 2017. Advanced persistent threat (APT) detection center. US Patent 9,628,507.Google Scholar
- Hyang-Ah Kim and David R O'Hallaron. 2003. Counting network flows in real time. In GLOBECOM'03. IEEE Global Telecommunications Conference (IEEE Cat. No. 03CH37489), Vol. 7. IEEE, 3888--3893.Google Scholar
- Jon Kleinberg. 2003. Bursty and hierarchical structure in streams. Data Mining and Knowledge Discovery , Vol. 7, 4 (2003), 373--397.Google ScholarDigital Library
- Shijin Kong, Tao He, Xiaoxin Shao, Changqing An, and Xing Li. 2006. Time-out bloom filter: A new sampling method for recording more flows. In International Conference on Information Networking. Springer, 590--599.Google ScholarDigital Library
- Marshall Kirk McKusick, Keith Bostic, Michael J Karels, and John S Quarterman. 1996. The design and implementation of the 4.4 BSD operating system. Vol. 2. Addison-Wesley Reading, MA.Google Scholar
- Prashant Pandey, Michael A. Bender, Rob Johnson, and Rob Patro. 2017. A General-Purpose Counting Filter: Making Every Bit Count. In Proceedings of the 2017 ACM International Conference on Management of Data (Chicago, Illinois, USA) (SIGMOD 7). Association for Computing Machinery, New York, NY, USA, 775-87. https://doi.org/10.1145/3035918.3035963Google ScholarDigital Library
- Yanqing Peng, Jinwei Guo, Feifei Li, Weining Qian, and Aoying Zhou. 2018. Persistent bloom filter: Membership testing for the entire history. In Proceedings of the 2018 International Conference on Management of Data . 1037--1052.Google ScholarDigital Library
- Pratanu Roy, Arijit Khan, and Gustavo Alonso. 2016. Augmented sketch: Faster and more accurate stream processing. In Proceedings of the 2016 International Conference on Management of Data . 1449--1463.Google ScholarDigital Library
- Jingsong Shan, Jianxin Luo, Guiqiang Ni, Zhaofeng Wu, and Weiwei Duan. 2016. CVS: fast cardinality estimation for large-scale data streams over sliding windows. Neurocomputing , Vol. 194 (2016), 107--116.Google ScholarDigital Library
- Anshumali Shrivastava, Arnd Christian Konig, and Mikhail Bilenko. 2016. Time adaptive sketches (ada-sketches) for summarizing data streams. In Proceedings of the 2016 International Conference on Management of Data . 1417--1432.Google ScholarDigital Library
- Kai Sheng Tai, Vatsal Sharan, Peter Bailis, and Gregory Valiant. 2018. Sketching linear classifiers over data streams. In Proceedings of the 2018 International Conference on Management of Data. 757--772.Google ScholarDigital Library
- Nan Tang, Qing Chen, and Prasenjit Mitra. 2016. Graph stream summarization: From big bang to big crunch. In Proceedings of the 2016 International Conference on Management of Data. 1481--1496.Google ScholarDigital Library
- Daniel Ting. 2018. Count-Min: Optimal Estimation and Tight Error Bounds using Empirical Error Distributions. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 2319--2328.Google ScholarDigital Library
- Tingting Chen, Yi Wang, Binxing Fang, and Jun Zheng. 2006. Detecting Lasting and Abrupt Bursts in Data Streams Using Two-Layered Wavelet Tree. In Advanced Int'l Conference on Telecommunications and Int'l Conference on Internet and Web Applications and Services (AICT-ICIW'06). 30--30.Google Scholar
- Nikos Virvilis and Dimitris Gritzalis. 2013. The big four-what we did wrong in advanced persistent threat detection?. In 2013 international conference on availability, reliability and security. IEEE, 248--254.Google ScholarDigital Library
- Zhewei Wei, Ge Luo, Ke Yi, Xiaoyong Du, and Ji-Rong Wen. 2015. Persistent data sketching. In Proceedings of the 2015 ACM SIGMOD international conference on Management of Data. 795--810.Google ScholarDigital Library
- Kyu-Young Whang, Brad T Vander-Zanden, and Howard M Taylor. 1990. A linear-time probabilistic counting algorithm for database applications. ACM Transactions on Database Systems (TODS) , Vol. 15, 2 (1990), 208--229.Google ScholarDigital Library
- Wei Xie, Feida Zhu, Jing Jiang, Ee-Peng Lim, and Ke Wang. 2016. Topicsketch: Real-time bursty topic detection from twitter. IEEE Transactions on Knowledge and Data Engineering , Vol. 28, 8 (2016), 2216--2229.Google ScholarDigital Library
- Linfeng Zhang and Yong Guan. 2008. Detecting click fraud in pay-per-click streams of online advertising networks. In 2008 The 28th International Conference on Distributed Computing Systems. IEEE, 77--84.Google ScholarDigital Library
- Yang Zhou, Tong Yang, Jie Jiang, Bin Cui, Minlan Yu, Xiaoming Li, and Steve Uhlig. 2018. Cold filter: A meta-framework for faster and more accurate stream processing. In Proceedings of the 2018 International Conference on Management of Data . 741--756.Google ScholarDigital Library
- Yunyue Zhu and Dennis Shasha. 2003. Efficient elastic burst detection in data streams. In Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining . 336--345.Google ScholarDigital Library
Index Terms
- Out of Many We are One: Measuring Item Batch with Clock-Sketch
Recommendations
Two Birds One Stone: On both Cold-Start and Long-Tail Recommendation
MM '17: Proceedings of the 25th ACM international conference on MultimediaThe number of "hits" has been widely regarded as the lifeblood of many web systems, e.g., e-commerce systems, advertising systems and multimedia consumption systems. However, users would not hit an item if they cannot see it, or they are not interested ...
Forgetting techniques for stream-based matrix factorization in recommender systems
Forgetting is often considered a malfunction of intelligent agents; however, in a changing world forgetting has an essential advantage. It provides means of adaptation to changes by removing effects of obsolete (not necessarily old) information from ...
Improving one-class collaborative filtering by incorporating rich user information
CIKM '10: Proceedings of the 19th ACM international conference on Information and knowledge managementOne-Class Collaborative Filtering (OCCF) is an emerging setup in collaborative filtering in which only positive examples or implicit feedback can be observed. Compared with the traditional collaborative filtering setting where the data has ratings, OCCF ...
Comments