Abstract
Apache Kafka is a widely used distributed publish-subscribe messaging system, offering high throughput and low latency. It is significant to guarantee the reliability and correctness of message transmissions in Kafka via formal modeling and verification. One of Kafka’s core functions is the producer-consumer communication. In this paper, the message delivery between producers and consumers in Kafka is formally modeled and verified via Mediator, a hierarchical component-based modeling language for distributed and concurrent systems. Moreover, we translate the Mediator model into the model checker PRISM to verify its properties, such as Deadlock Freedom, Acknowledgement Mechanism, Parallelism, Sequentiality and Fault Tolerance. Results demonstrate the reliability of the producer-consumer communication in Kafka.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bano, S., Carlini, E., Cassarà, P., Coppola, M., Dazzi, P., Gotta, A.: A novel approach to distributed model aggregation using apache kafka. In: Ferrucci, L., Coppola, M., Kavalionak, H., Kontopoulos, I. (eds.) FRAME@HPDC 2022: Proceedings of the 2nd Workshop on Flexible Resource and Application Management on the Edge, Minneapolis, MN, USA, 1 July 2022, pp. 33–36. ACM (2022)
Hoare, C.A.R.: Communicating Sequential Processes. Prentice-Hall (1985)
Kwiatkowska, M., Norman, G., Parker, D.: PRISM: probabilistic symbolic model checker. In: Field, T., Harrison, P.G., Bradley, J., Harder, U. (eds.) TOOLS 2002. LNCS, vol. 2324, pp. 200–204. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-46029-2_13
Lakkad, A.K., Bhadaniya, R.D., Shah, V.N., Lavanya, K.: Complex events processing on live news events using apache Kafka and clustering techniques. Int. J. Intell. Inf. Technol. 17(1), 39–52 (2021)
Li, Y., Sun, M.: Component-based modeling in mediator. In: Proença, J., Lumpe, M. (eds.) FACS 2017. LNCS, vol. 10487, pp. 1–19. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68034-7_1
Li, Y., Sun, W., Sun, M.: Mediator: a component-based modeling language for concurrent and distributed systems. Sci. Comput. Program. 192, 102438 (2020)
Liu, Y., Sun, J., Dong, J.S.: Developing model checkers using PAT. In: Bouajjani, A., Chin, W.-N. (eds.) ATVA 2010. LNCS, vol. 6252, pp. 371–377. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15643-4_30
Peddireddy, K.: Streamlining enterprise data processing, reporting and realtime alerting using apache Kafka. In: 11th International Symposium on Digital Forensics and Security, ISDFS 2023, Chattanooga, TN, USA, May 11-12, 2023, pp. 1–4. IEEE (2023)
Raptis, T.P., Cicconetti, C., Falelakis, M., Kalogiannis, G., Kanellos, T., Lobo, T.P.: Engineering resource-efficient data management for smart cities with apache Kafka. Future Internet 15(2), 43 (2023)
Sax, M..J.: Apache kafka. In: Sakr S., Zomaya, A.Y. (eds.) Encyclopedia of Big Data Technologies. Springer (2019). https://doi.org/10.1007/978-3-319-63962-8_196-2
Sun, W., Sun, M.: PRISM code generation for verification of mediator models. In: Perkusich, A. (ed.) The 31st International Conference on Software Engineering and Knowledge Engineering, SEKE 2019, Hotel Tivoli, Lisbon, Portugal, July 10-12, 2019, pp. 271–354. KSI Research Inc. and Knowledge Systems Institute Graduate School (2019)
Wu, H., Shang, Z., Peng, G., Wolter, K.: A reactive batching strategy of apache Kafka for reliable stream processing in real-time. In: Vieira, M., Madeira, H., Antunes, N., Zheng, Z. (eds.) 31st IEEE International Symposium on Software Reliability Engineering, ISSRE 2020, Coimbra, Portugal, October 12-15, 2020, pp. 207–217. IEEE (2020)
Wu, H., Shang, Z., Wolter, K.: Learning to reliably deliver streaming data with apache Kafka. In: 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2020, Valencia, Spain, June 29 - July 2, 2020, pp. 564–571. IEEE (2020)
Xu, J., Yin, J., Zhu, H., Xiao, L.: Modeling and verifying producer-consumer communication in Kafka using CSP. In: ECBS 2021: 7th Conference on the Engineering of Computer Based Systems, Novi Sad, Serbia, pp. 9:1–9:10. ACM (2021)
Acknowledgment
This research was sponsored by the National Key R &D Program of China (under Grant No. 2022YFB2702200), and the National Natural Science Foundation of China under Grant No. 62172019.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix
Appendix
The Leader automaton with \(n_p = 2\), \(n_f = 2\), \(n_c = 2\):
The follower automaton with fixed leader L1 where \(n_c = 2\):
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sun, M., Chen, Z. (2024). Formal Modeling and Verification of Kafka Producer-Consumer Communication in Mediator. In: Arai, K. (eds) Intelligent Computing. SAI 2024. Lecture Notes in Networks and Systems, vol 1016. Springer, Cham. https://doi.org/10.1007/978-3-031-62281-6_41
Download citation
DOI: https://doi.org/10.1007/978-3-031-62281-6_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-62280-9
Online ISBN: 978-3-031-62281-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)