Pratyush Agnihotri
ABSTRACT
Resource management in Distributed Stream Processing Systems (DSPS) defines the way queries are deployed on in-network resources to deliver query results while fulfilling the Quality of Service (QoS) requirements of the end-users. Various resource management mechanisms have been proposed in DSPS; however, they become inefficient in challenging conditions imposed by the dynamic environment and heterogeneous resources. This is because they focus on pre-configuration of both single and static QoS requirements. In addition, they lack cooperation between heterogeneous resources which amplify the problem of coordination between resources. This could lead to severe performance degradation such as inconsistent and incorrect query results in comparison to homogeneous resources. To solve the above challenges, in this research work, we will propose mechanisms: (i) to forecast the performance of network and heterogeneous resources, (ii) to select an efficient resource management approach, and (iii) for cooperation between resources in a dynamic environment.
- Hadeer Ahmed, Issa Traore, and Sherif Saad. 2018. Detecting opinion spams and fake news using text classification. Security and Privacy 1 (2018), 1--9.Google Scholar
Cross Ref
- Nathan Backman, Rodrigo Fonseca, and Uǧur Çetintemel. 2012. Managing parallelism for stream processing in the cloud. In Proceedings of the 1st International Workshop on Hot Topics in Cloud Data Processing (HotCDP). 1--5. Google ScholarDigital Library
- Valeria Cardellini, Vincenzo Grassi, Francesco Lo Presti, and Matteo Nardelli. 2017. Optimal operator replication and placement for distributed stream processing systems. ACM SIGMETRICS Performance Evaluation Review 44 (2017), 11--22. Google ScholarDigital Library
- Valeria Cardellini, Francesco Lo Presti, Matteo Nardelli, and Gabriele Russo Russo. 2018. Decentralized self-adaptation for elastic data stream processing. Future Generation Computer Systems 87 (2018), 171--185.Google ScholarDigital Library
- Gianpaolo Cugola and Alessandro Margara. 2012. Low latency complex event processing on parallel hardware. J. Parallel and Distrib. Comput. 72 (2012), 205--218. Google ScholarDigital Library
- Kareem Darwish, Walid Magdy, and Tahar Zanouda. 2017. Trump vs. Hillary: What went viral during the 2016 US presidential election. In International Conference on Social Informatics (ICSI). 143--161.Google ScholarCross Ref
- Miyuru Dayarathna and Srinath Perera. 2018. Recent advancements in event processing. ACM Computing Surveys (CSUR) 51 (2018), 1--36. Google ScholarDigital Library
- Marcos Dias de Assuncao, Alexandre da Silva Veith, and Rajkumar Buyya. 2018. Distributed data stream processing and edge computing: A survey on resource elasticity and future directions. Journal of Network and Computer Applications 103 (2018), 1--17. Google ScholarDigital Library
- Benjamin Hilprecht, Carsten Binnig, and Uwe Röhm. 2019. Towards learning a partitioning advisor with deep reinforcement learning. In Proceedings of the Second International Workshop on Exploiting Artificial Intelligence Techniques for Data Management (aiDM). 1--4. Google ScholarDigital Library
- Gabriela Jacques-Silva, Buğra Gedik, Rohit Wagle, Kun-Lung Wu, and Vibhore Kumar. 2012. Building user-defined runtime adaptation routines for stream processing applications. arXiv preprint arXiv:1208.4176 (2012). Google ScholarDigital Library
- Sabihe Kabirzadeh, Dadmehr Rahbari, and Mohsen Nickray. 2017. A hyper heuristic algorithm for scheduling of fog networks. In Proceedings of the 21st Conference of Open Innovations Association (FRUCT). 148--155. Google ScholarDigital Library
- Junaid Ahmed Khan, Cedric Westphal, and Yacine Ghamri-Doudane. 2017. Offloading content with self-organizing mobile fogs. In 29th International Teletraffic Congress (ITC). 223--231.Google ScholarCross Ref
- Rohit Khandekar, Kirsten Hildrum, Sujay Parekh, Deepak Rajan, Joel Wolf, KunLung Wu, Henrique Andrade, and Buğra Gedik. 2009. COLA: Optimizing stream processing applications via graph partitioning. In ACM/IFIP/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing. 308--327. Google ScholarDigital Library
- Roland Kotto Kombi, Nicolas Lumineau, and Philippe Lamarre. 2017. A preventive auto-parallelization approach for elastic stream processing. In Proceedings of the 37th IEEE International Conference on Distributed Computing Systems (ICDCS). 1532--1542.Google ScholarCross Ref
- Changlong Li, Hang Zhuang, Qingfeng Wang, and Xuehai Zhou. 2018. SSLB: Self-Similarity-Based Load Balancing for Large-Scale Fog Computing. Arabian Journal for Science & Engineering (Springer Science & Business Media BV) 43, 12 (2018).Google Scholar
- Liqing Liu, Zheng Chang, and Xijuan Guo. 2018. Socially aware dynamic computation offloading scheme for fog computing system with energy harvesting devices. IEEE Internet of Things Journal 5 (2018), 1869--1879.Google ScholarCross Ref
- Xunyun Liu and Rajkumar Buyya. 2017. Performance-oriented deployment of streaming applications on cloud. IEEE Transactions on Big Data 5 (2017), 46--59.Google ScholarCross Ref
- Xunyun Liu and Rajkumar Buyya. 2020. Resource management and scheduling in distributed stream processing systems: a taxonomy, review, and future directions. ACM Computing Surveys (CSUR) 53 (2020), 1--41. Google ScholarDigital Library
- Xunyun Liu, Amir Vahid Dastjerdi, Rodrigo N Calheiros, Chenhao Qu, and Rajkumar Buyya. 2017. A stepwise auto-profiling method for performance optimization of streaming applications. ACM Transactions on Autonomous and Adaptive Systems (TAAS) 12 (2017), 1--33. Google ScholarDigital Library
- Manisha Luthra, Sebastian Hennig, Kamran Razavi, Lin Wang, and Boris Koldehofe. 2020. Operator as a Service: Stateful Serverless Complex Event Processing. In Proceedings of the IEEE International Conference on Big Data (BigData). 1964--1973.Google ScholarCross Ref
- Manisha Luthra, Boris Koldehofe, Niels Danger, Pascal Weisenberger, Guido Salvaneschi, and Ioannis Stavrakakis. 2021. TCEP: Transitions in operator placement to adapt to dynamic network environments. J. Comput. System Sci. 122 (2021), 94--125.Google ScholarCross Ref
- Manisha Luthra, Boris Koldehofe, Pascal Weisenburger, Guido Salvaneschi, and Raheel Arif. 2018. TCEP: Adapting to dynamic user environments by enabling transitions between operator placement mechanisms. In Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems (DEBS). 136--147. Google ScholarDigital Library
- Ruben Mayer and Hans-Arno Jacobsen. 2020. Scalable deep learning on distributed infrastructures: Challenges, techniques, and tools. ACM Computing Surveys (CSUR) 53 (2020), 1--37. Google ScholarDigital Library
- Ruben Mayer, Boris Koldehofe, and Kurt Rothermel. 2014. Meeting predictable buffer limits in the parallel execution of event processing operators. In Proceedings of the IEEE International Conference on Big Data (BigData). 402--411.Google Scholar
- Ruben Mayer, Boris Koldehofe, and Kurt Rothermel. 2015. Predictable low-latency event detection with parallel complex event processing. IEEE Internet of Things Journal 2 (2015), 274--286.Google ScholarCross Ref
- Priyanka Meel and Dinesh Kumar Vishwakarma. 2020. Fake news, rumor, information pollution in social media and web: A contemporary survey of state-of-the-arts, challenges and opportunities. Expert Systems with Applications 153 (2020), 112986.Google ScholarCross Ref
- Gabriele Mencagli. 2016. A game-theoretic approach for elastic distributed data stream processing. ACM Transactions on Autonomous and Adaptive Systems (TAAS) 11 (2016), 1--34. Google ScholarDigital Library
- Yugo Nakamura, Hirohiko Suwa, Yutaka Arakawa, Hirozumi Yamaguchi, and Keiichi Yasumoto. 2016. Design and implementation of middleware for iot devices toward real-time flow processing. In Proceedings of the 36th IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW). 162--167.Google ScholarCross Ref
- Nicoló Rivetti, Leonardo Querzoni, Emmanuelle Anceaume, Yann Busnel, and Bruno Sericola. 2015. Efficient key grouping for near-optimal load balancing in stream processing systems. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems (DEBS). 80--91. Google ScholarDigital Library
- Henriette Röger and Ruben Mayer. 2019. A comprehensive survey on parallelization and elasticity in stream processing. ACM Computing Surveys (CSUR) 52 (2019), 1--37. Google ScholarDigital Library
- Gabriele Russo Russo, Matteo Nardelli, Valeria Cardellini, and Francesco Lo Presti. 2018. Multi-level elasticity for wide-area data streaming systems: A reinforcement learning approach. Algorithms 11 (2018), 134.Google ScholarCross Ref
- Saad Sadiq, Nicolas Wagner, Mei-Ling Shyu, and Daniel Feaster. 2019. High dimensional latent space variational autoencoders for fake news detection. In Proceedings of the IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). 437--442.Google ScholarCross Ref
- Scott Schneider, Martin Hirzel, Bugra Gedik, and Kun-Lung Wu. 2012. Auto-parallelizing stateful distributed streaming applications. In Proceedings of the 21st international conference on Parallel architectures and compilation techniques (PACT). 53--64. Google ScholarDigital Library
- Mennan Selimi, Llorenç Cerdà Alabern, Felix Freitag, Luís Veiga, Arjuna Sathiaseelan, and Jon Crowcroft. 2019. A lightweight service placement approach for community network micro-clouds. Journal of Grid Computing 17 (2019), 169--189. Google ScholarDigital Library
- Olena Skarlat, Matteo Nardelli, Stefan Schulte, Michael Borkowski, and Philipp Leitner. 2017. Optimized IoT service placement in the fog. Service Oriented Computing and Applications 11 (2017), 427--443. Google ScholarDigital Library
- William Thies, Michal Karczmarek, and Saman Amarasinghe. 2002. StreamIt: A language for streaming applications. In Proceedings of the International Conference on Compiler Construction (CC). 179--196. Google ScholarDigital Library
- Chunkai Wang, Xiaofeng Meng, Qi Guo, Zujian Weng, and Chen Yang. 2017. Automating characterization deployment in distributed data stream management systems. IEEE Transactions on Knowledge and Data Engineering 29 (2017), 2669--2681.Google ScholarCross Ref
- Daniel Warneke and Odej Kao. 2011. Exploiting dynamic resource allocation for efficient parallel data processing in the cloud. IEEE transactions on parallel and distributed systems 22 (2011), 985--997. Google ScholarDigital Library
- World Health Organization (WHO). 2021. Infodemic. https://www.who.int/health-topics/infodemic. [Online; accessed 12-09-2021].Google Scholar
- Nikos Zacheilas, Vana Kalogeraki, Nikolas Zygouras, Nikolaos Panagiotou, and Dimitrios Gunopulos. 2015. Elastic complex event processing exploiting prediction. In IEEE International Conference on Big Data (BigData). 213--222. Google ScholarDigital Library
Index Terms
- Autonomous resource management in distributed stream processing systems
Recommendations
Resource Management and Scheduling in Distributed Stream Processing Systems: A Taxonomy, Review, and Future Directions
Stream processing is an emerging paradigm to handle data streams upon arrival, powering latency-critical application such as fraud detection, algorithmic trading, and health surveillance. Though there are a variety of Distributed Stream Processing ...
Distributed resource allocation for stream data processing
HPCC'06: Proceedings of the Second international conference on High Performance Computing and CommunicationsData streaming applications are becoming more and more common due to the rapid development in the areas such as sensor networks, multimedia streaming, and on-line data mining, etc. These applications are often running in a decentralized, distributed ...
Resource management in large distributed systems
In this paper, we propose that a resource management system for large distributed systems should have two levels --- a lower one, responsible for export and allocation of resources in local distributed systems, and an upper one, which manages special ...
Comments