Skip to main content

Advertisement

Springer Nature Link
Log in

Multi-core accelerated CRDT for large-scale and dynamic collaboration

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

With the advancement of networking technologies and large-scale social computing, multi-user collaboration is becoming popular. In large-scale and dynamic collaborative environments, users may arbitrarily join or quit a collaboration group and work on common tasks for a long time. When users synchronize their work with the other collaborators, a large number of concurrent updates from the collaborators must be reconciled or merged to produce a consistent document state, which could challenge existing consistency maintenance approaches. Our idea is to accelerate the commutative replicated data type (CRDT)-based consistency maintenance algorithms with multi-core processors. We theoretically prove a general commutative condition for CRDT operations. One typical CRDT algorithm, replicated growable array (RGA), is transformed into Parallel RGA (PRGA) under the guidance of the general commutative condition. PRGA allows a batch of RGA operations to be executed in parallel. Experimental results showed that when processing millions of concurrent updates, PRGA achieves approximately 4\(\times\) performance gains over RGA on an ordinary four-core processor.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://www.google.com/docs/about/.

  2. https://teletype.atom.io/.

  3. https://www.wikipedia.org/.

  4. https://github.com/.

  5. In the literature, conflict-free replicated data types have two basic ways to propagate changes on replicas: state-based and operation-based, which are called convergent replicated data types and commutative replicated data types, respectively [5]. In this paper, we focus on the operation-based conflict-free data types, i.e., commutative replicated data types.

References

  1. Ho LY, Wu JJ, Liu PF (2012) Distributed graph database for large-scale social computing, In: Proc of IEEE Conference on Cloud Computing, IEEE, pp. 455-462

  2. Grevet C, Gilbert E (2015) Piggyback prototyping: using existing, large-scale social computing systems to prototype new ones, In: Proc of ACM Conference on Human Factors in Computing Systems, ACM, pp 4047-4056

  3. Li T (2019) Solving mysteries with the wisdom of crowds: a modularized pipeline and context slices, In: Proc of ACM Conference on Computer Supported Cooperative Work and Social Computing, ACM, pp. 68–71

  4. Sun CZ, Xia S, Sun D, Chen D, Shen HF, Cai WT (2006) Transparent adaptation of single-user applications for multi-user real-time collaboration. ACM Trans Comput Hum Interact 13(4):531–582

    Article  Google Scholar 

  5. Shapiro M, Preguiça N, Baquero C, Zawirski M (2011) A comprehensive study of convergent and commutative replicated data types, Research Report RR-7506, inria-00555588

  6. Bartel JW, Dewan P (2012) Towards multi-domain collaborative toolkits, In: Proc of ACM Conference on Computer Supported Cooperative Work, ACM, pp 1297–1306

  7. Xu Y, Sun CZ (2016) Conditions and patterns for achieving convergence in OT-based co-editors. IEEE Trans Parallel Distrib Syst 27(3):695–709

    Article  Google Scholar 

  8. Weiss S, Urso P, Molli P (2010) Logoot-undo: distributed collaborative editing system on P2P networks. IEEE Trans Parallel Distrib Syst 21(8):1162–1174

    Article  Google Scholar 

  9. Yu WH (2014) Supporting string-wise operations and selective undo for peer-to-peer group editing, In: Proc of ACM Conference on Supporting Group Work, ACM, pp 226–237

  10. Lv X, He FZ, Cai WW, Cheng Y (2018) Supporting selective undo of string-wise operations for collaborative editing systems. Futur Gener Comput Syst 82:41–62

    Article  Google Scholar 

  11. Lv X, He FZ, Cai WW, Cheng Y (2019) An optimized RGA supporting selective undo for collaborative text editing systems. J Parallel Distrib Comput 132:310–330

    Article  Google Scholar 

  12. Zheng Y, Shen HF, Sun CZ (2010) Inspiring innovative design integration by collaborative exploration of boolean operations. IEEE Trans Ind Inform 6(2):243–250

    Article  Google Scholar 

  13. Liang YQ, He FZ, Zeng XT, Luo JK (2022) An improved Loop subdivision to coordinate the smoothness and the number of faces via multi-objective optimization. Integr Comput-aid E 29(1):23–41

    Article  Google Scholar 

  14. Liang YQ, He FZ, Zeng XT (2020) 3D mesh simplification with feature preservation based on whale optimization algorithm and differential evolution. Integr Comput-aid E 27(4):417–435

    Article  Google Scholar 

  15. Ng AC, Sun Z (2016) Operational transformation for real-time synchronization of shared workspace in cloud storage, In: Proc of ACM Conference on Supporting Group Work, ACM pp 61–70

  16. Cai WW, Ng AC, Sun A (2018) Some discoveries from a concurrency benchmark study of major cloud storage systems, In: Proc of International Conference on Cooperative Design, Visualization and Engineering, Springer, pp 44–48

  17. Preguica NM, Marques JM, Shapiro M, Letia M (2009) A commutative replicated data type for cooperative editing, In: Proc of IEEE Conference on Distributed Computing Systems, IEEE, pp 395–403

  18. Roh H-G, Jeon M, Kim J, Lee J (2011) Replicated abstract data types: building blocks for collaborative applications. J Parallel Distrib Comput 71(3):354–368

    Article  Google Scholar 

  19. Ahmed-Nacer M, Ignat CL, Oster G, Roh HG, Urso P (2011) Evaluating CRDTs for real-time document editing, In: Proc of ACM Symposium on Document Engineering, ACM, pp 103–112

  20. Andrè L, Martin S, Oster G, Ignat C-L (2013) Supporting adaptable granularity of changes for massive-scale collaborative editing, In: Proc of IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing, IEEE, pp 50-59

  21. Briot L,Urso P, Shapiro M (2016) High responsiveness for group editing CRDTs, In: Proc of ACM Conference on Supporting group work, ACM, pp 1391–1400

  22. Attiya H, Burckhardt S, Gotsman A, Morrison A, Yang H, Zawirski M (2021) Specification and space complexity of collaborative text editing. Theor Comput Sci 855:141–160

    Article  MathSciNet  Google Scholar 

  23. Oster G, Urso P,Molli P, Imine A (2006) Data consistency for P2P collaborative editing, In: Proc of ACM Conference on Computer Supported Cooperative Work, ACM, pp 259–268

  24. Stèphane W, Urso P, Molli P (2007) Wooki: a P2P Wiki-based collaborative writing tool, In: Proc of International Conference on Web Information Systems Engineering, pp 503–512

  25. Biuk-Aghai RP, Pang C-I, Si Y-W (2014) Visualizing large-scale human collaboration in Wikipedia. Futur Gener Comput Syst 31:120–133

    Article  Google Scholar 

  26. Dang QV, Ignat CL (2015) Performance of real-time collaborative editors at large scale: user perspective, In: Proc of IEEE IFIP Networking Conference, Networking and Workshops, IEEE, pp 548–553

  27. Lv X, He FZ, Cai WW, Cheng Y (2017) A string-wise CRDT algorithm for smart and large-scale collaborative editing systems. Adv Eng Inf 33:397–409

    Article  Google Scholar 

  28. Yu WH (2018) Managing multi-synchronous sessions for collaborative editing, In: Proc of International Conference on Cooperative Design, Visualization, and Engineering, pp 69-77

  29. Ginny, Kumar C, Naik K (2021) Smartphone processor architecture, operations, and functions: current state-of-the-art and future outlook: energy performance trade-off. J Super Comput 77(2):1377–1454

    Article  Google Scholar 

  30. Vu L, Alaghband G (2014) Novel parallel method for association rule mining on multi-core shared memory systems. Parallel Comput 40(10):768–785

    Article  Google Scholar 

  31. Zhang DL, Dechev D (2016) A lock-free priority queue design based on multi-dimensional linked lists. IEEE Trans Parallel Distrib Syst 27(3):613–626

    Article  Google Scholar 

  32. Booth JD, Ellingwood ND, Thornquist HK, Rajamanickam S (2017) Basker: parallel sparse lu factorization utilizing hierarchical parallelism and data layouts. Parallel Comput 68:17–31

    Article  MathSciNet  Google Scholar 

  33. Li HR, He FZ, Pan YT (2021) MLFS-CCDE: multi-objective large-scale feature selection by cooperative coevolutionary differential evolution. Memet Comput 13(1):1–18

    Article  Google Scholar 

  34. Atashpendar A, Dorronsoro B, Danoy G, Bouvry P (2018) A scalable parallel cooperative coevolutionary pso algorithm for multi-objective optimization. J Parallel Distrib Comput 112:111–125

    Article  Google Scholar 

  35. Kenneth P, Mittal N, Venkatesan S (2020) Practical concurrent unrolled linked lists using lazy synchronization. J Parallel Distrib Comput 139:110–134

    Article  Google Scholar 

  36. Aravind N, Ramachandran A, Mittal N (2020) FEAST a lightweight lock-free concurrent binary search tree. ACM Trans Parallel Comput 7(2):10:1-10:64

    Google Scholar 

  37. Maier T, Sanders P, Dementiev R (2019) Concurrent hash tables: fast and general (?)! ACM Trans Parallel Comput 5(4):16:1-16:32

    Article  Google Scholar 

  38. Gu N, Yang JM, Zhang QW (2005)Consistency maintenance based on the mark & retrace technique in groupware systems, In: Proc of ACM Conference on Supporting Group Work, ACM, pp 264–273

  39. Lamport L (1978) Time, clocks, and the ordering of events in a distributed system. Commun ACM 21(7):558–565

    Article  Google Scholar 

  40. Luo JK, He FZ, Li HR, Zeng XT, Liang YQ (2022) novel whale optimization algorithm with filtering disturbance and non-linear step. Int J Bio-Inspired Comput. https://doi.org/10.1504/IJBIC.2020.10036562

    Article  Google Scholar 

  41. Gates M, Tomov S, Dongarra JJ (2018) Accelerating the SVD two stage bidiagonal reduction and divide and conquer using GPUs. Parallel Comput 74:3–18

    Article  MathSciNet  Google Scholar 

  42. Wu HR, He FZ, Duan YS, Yan XH (2022) Perceptual metric-guided human image generation. Integr Comput-aid E. https://doi.org/10.3233/ICA-210672

    Article  Google Scholar 

  43. Saadatmand FS, Rohbani N, Baharvand F, Farbeh H (2021) TAMER: an adaptive task allocation method for aging reduction in multi-core embedded real-time systems. J Super Comput 77(2):1939–1957

    Article  Google Scholar 

  44. Moreno JJ, Miroforidis J, Filatovas E, Kaliszewski I, Garzón EM (2021) Parallel radiation dose computations with GENOCOP III on GPUs. J Super Comput 77(1):66–76

    Article  Google Scholar 

  45. Gao LP, Yu FY, Chen QK, Xiong NX (2016) Consistency maintenance of Do and Undo/Redo operations in real-time collaborative bitmap editing systems. Clust Comput 19(1):255–267

    Article  Google Scholar 

  46. Gao LP, Gao DF, Xiong NX, Lee C (2018) Cowebdraw: a real-time collaborative graphical editing system supporting multi-clients based on HTML5. Multimed Tools Appl 77(4):5067–5082

    Article  Google Scholar 

  47. Pan YT, He FZ, Yu HP (2020) Learning social representations with deep autoencoder for recommender system. World Wide Web 23(4):2259–2279

    Article  Google Scholar 

  48. Wu CX, Li LF, Peng CW, W Y, Xiong NX, Lee C (2019) Design and analysis of an effective graphics collaborative editing system, EURASIP J Image Video Process 2019, 50

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (Grant No. 62072348), the Science and Technology Major Project of Hubei Province (Next-Generation AI Technologies) (Grant No. 2019AEA170), and the Natural Science Foundation of Hubei Province (No. 2019CFB627).

Author information

Authors and Affiliations

  1. School of Computer Science, Wuhan University, Wuhan, Hubei, 430072, China

    Weiwei Cai & Fazhi He

  2. Department of Computer Engineering, Naval University of Engineering, Wuhan, Hubei, 430072, China

    Xiao Lv

Authors
  1. Weiwei Cai
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Fazhi He
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Xiao Lv
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Fazhi He.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cai, W., He, F. & Lv, X. Multi-core accelerated CRDT for large-scale and dynamic collaboration. J Supercomput 78, 10799–10828 (2022). https://doi.org/10.1007/s11227-022-04308-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04308-7

Keywords