Abstract
Graph data is becoming dominant for many applications such as social networks, targeted advertising, and web indexing. As a result of that, advances in machine learning and data mining techniques depend tightly on the ability to process this data structure efficiently and reliably. Despite the importance of processing dynamic graphs in real-time, it remains a challenge to maintain such graphs and process them over data streams. We propose Sprouter, an end-to-end framework which enable storing enormous graph data, allows updates in real-time, and supports efficient complex analytics in addition to OLTP queries. We demonstrate that our framework can ingest and process streaming data efficiently using a scalable multi-cluster distributed architecture, apply incremental graph updates, and store the dynamic graph for fast query performance. Experiments showed the system is able to update graphs having up to 100 million edges in under 50 s in a moderate underlying cluster. As we use all open source tools, the framework can be easily extended in the future with other equivalent software.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abughofa, T., Zulkernine, F.: Towards online graph processing with spark streaming. In: IEEE International Conference on Big Data, pp. 2787–2794. IEEE (2017)
Choudhury, S., et al.: NOUS: Construction and querying of dynamic knowledge graphs. In: 2017 IEEE 33rd International Conference on Data Engineering (ICDE), pp. 1563–1565. IEEE (2017)
Dave, A., Jindal, A., Li, L.E., Xin, R., Gonzalez, J., Zaharia, M.: Graphframes: an integrated API for mixing graph and relational queries. In: Proceedings of the Fourth International Workshop on Graph Data Management Experiences and Systems, p. 2. ACM (2016)
Gonzalez, J.E., Xin, R.S., Dave, A., Crankshaw, D., Franklin, M.J., Stoica, I.: Graphx: Graph processing in a distributed dataflow framework. In: OSDI, vol. 14, pp. 599–613 (2014)
Iyer, A.P., Li, L.E., Das, T., Stoica, I.: Time-evolving graph processing at scale. In: Proceedings of the Fourth International Workshop on Graph Data Management Experiences and Systems, p. 5. ACM (2016)
Malewicz, G., et al.: Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, pp. 135–146. ACM (2010)
Sharma, A., Jiang, J., Bommannavar, P., Larson, B., Lin, J.: Graphjet: real-time content recommendations at twitter. Proc. VLDB Endow. 9(13), 1281–1292 (2016)
Yin, S., et al.: Node-grained incremental community detection for streaming networks. In: IEEE 28th International Conference on Tools with Artificial Intelligence, pp. 585–592. IEEE (2016)
Zaharia, M., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, p. 2. USENIX Association (2012)
Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., Stoica, I.: Discretized streams: fault-tolerant streaming computation at scale. In: Proceedings of the 24th ACM Symposium on Operating Systems Principles, pp. 423–438. ACM (2013)
Zaharia, M., et al.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Abughofa, T., Zulkernine, F. (2018). Sprouter: Dynamic Graph Processing over Data Streams at Scale. In: Hartmann, S., Ma, H., Hameurlain, A., Pernul, G., Wagner, R. (eds) Database and Expert Systems Applications. DEXA 2018. Lecture Notes in Computer Science(), vol 11030. Springer, Cham. https://doi.org/10.1007/978-3-319-98812-2_28
Download citation
DOI: https://doi.org/10.1007/978-3-319-98812-2_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-98811-5
Online ISBN: 978-3-319-98812-2
eBook Packages: Computer ScienceComputer Science (R0)