Design of Highly Scalable Graph Database Systems without Exponential Performance Degradation

The main challenge faced by today's graph database systems is sacrificing performance (computation) for scalability (storage). Such systems probably can store a large amount of data across many instances but can't offer adequate graph-computing power to deeply penetrate dynamic graph dataset in real time. A seemingly simple and intuitive graph query like K-hop traversal or finding all shortest paths may lead to deep traversal of large amount of graph data, which tends to cause a typical BSP (Bulky Synchronous Processing) system to exchange heavily amongst its distributed instances, therefore causing significant latencies. This paper proposes three schools of architectural designs for distributed and horizontally scalable graph database while achieving highly performant graph data processing capabilities. The first school, coined HTAP, augments distributed consensus algorithm RAFT paired with vector-based computing acceleration to achieve fast online data ingestion and real-time deep-data traversal in a TP and AP hybrid mode. The second school, named as GRID, leverages human-intelligence for data partitioning, and preserving the HTAP data processing capabilities across all partitioned clusters. The last school incorporates SHARD and advanced GQL optimization techniques to allow data partitioning to be done fully automated yet strive to achieve lower latency via minimum I/O cost data migration model when queries spread across multiple clusters.