Skip to main content
Log in

RailwayDB: adaptive storage of interaction graphs

  • Regular Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

We are living in an ever more connected world, where data recording the interactions between people, software systems, and the physical world is becoming increasingly prevalent. These data often take the form of a temporally evolving graph, where entities are the vertices and the interactions between them are the edges. We call such graphs interaction graphs. Various domains, including telecommunications, transportation, and social media, depend on analytics performed on interaction graphs. The ability to efficiently support historical analysis over interaction graphs requires effective solutions for the problem of data layout on disk. This paper presents an adaptive disk layout called the railway layout for optimizing disk block storage for interaction graphs. The key idea is to divide blocks into one or more sub-blocks. Each sub-block contains the entire graph structure, but only a subset of the attributes. This improves query I/O, at the cost of increased storage overhead. We introduce optimal integer linear program (ILP) formulations for partitioning disk blocks into sub-blocks with overlapping and nonoverlapping attributes. Additionally, we present greedy heuristics that can scale better compared to the ILP alternatives, yet achieve close to optimal query I/O. We provide an implementation of the railway layout as part of RailwayDB—an open-source graph database we have developed. To demonstrate the benefits of the railway layout, we provide an extensive experimental evaluation, including model-based as well as empirical results comparing our approach to baseline alternatives.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. Since time is an implicit part of an interaction, we do not show it as part of the schema.

  2. https://github.com/usi-systems/railwaydb.git

References

  1. Hoque, I., Gupta, I.: Disk layout techniques for online social network data. Internet Comput. 16(3), 24–36 (2012)

    Article  Google Scholar 

  2. Steinhaus, R.: G-Store: a storage manager for graph data. Master’s thesis, University of Oxford (2011)

  3. Gedik, B., Bordawekar, R.: Disk-based management of interaction graphs. IEEE Trans. Knowl. Data. Eng. (TKDE) 26(11), 650–665 (2014)

    Article  Google Scholar 

  4. Alagiannis, I., Idreos, S., Ailamaki, A.: H2O: a hands-free adaptive store. In: ACM International Conference on Management of Data (SIGMOD), pp. 1103–1114 (2014)

  5. Grund, M., Zeier, A., Krüger, J., Plattner, H., Madden, S.: HYRISE: a main memory hybrid storage engine. VLDB J. (VLDBJ) 4(2), 105–116 (2010)

    Google Scholar 

  6. Stonebraker, M., Abadi, D.J., Batkin, A., Chen, X., Cherniack, M., Ferreira, M., Lau, E., Lin, A., Madden, S., O’Neil, E., O’Neil, P., Rasin, A., Tran, N., Zdonik, S.: C-store: a column-oriented DBMS. In: International Conference on Very Large Data Bases (VLDB), pp. 553–564 (2005)

  7. Hellerstein, J.M., Stonebraker, M., Hamilton, J.: Architecture of a database system. Found. Trends Databases 1(2), 141–259 (2007). doi:10.1561/1900000002

    Article  MATH  Google Scholar 

  8. Abadi, D., Boncz, P., Harizopoulos, S., Idreos, S., Madden, S.: The design and implementation of modern column-oriented database systems. Found. Trends Databases 5(3), 197–280 (2013)

    Article  Google Scholar 

  9. Wilson, C., Boe, B., Sala, A., Puttaswamy, K.P., Zhao, B.Y.: User interactions in social networks and their implications. In: European Conference on Computer Systems (EUROSYS), pp. 205–218 (2009)

  10. Wilson, C., Sala, A., Puttaswamy, K.P.N., Zhao, B.Y.: Beyond social graphs: user interactions in online social networks and their implications. ACM Trans. Web (TWEB) 6(4), 17:1–17:31 (2012)

    Google Scholar 

  11. Ghemawat, S.: LevelDB - a fast and lightweight key/value database library by Google. Available at https://github.com/google/leveldb, retrieved March, 2015

  12. Hadjieleftheriou, M., Hoel, E.G., Tsotras, V.J.: SaIL: a spatial index library for efficient application integration. GeoInformatica 9(4), 367–389 (2005)

    Article  Google Scholar 

  13. Gurobi Optimization Inc.: The Gurobi Optimizer. Available at http://www.gurobi.com

  14. Turkey Protests Spread from Istanbul to Ankara, Euronews. http://www.euronews.com/2013/05/31/turkey-protests-spread-from-istanbul-to-ankara/

  15. Malewicz, G., Austern, M.H., Bik, A.J., Dehnert, J.C., Horn, I., Leiser, N., Czajkowski, G.: Pregel: a system for large-scale graph processing. In: ACM International Conference on Management of Data (SIGMOD), pp. 135–146 (2010)

  16. Apache giraph. http://giraph.apache.org

  17. Gonzalez, J.E., Low, Y., Gu, H., Bickson, D., Guestrin, C.: PowerGraph: distributed graph-parallel computation on natural graphs. In: USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 17–30 (2012)

  18. Kyrola, A., Blelloch, G., Guestrin, C.: GraphChi: large-scale graph computation on just a PC. In: USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 31–46 (2012)

  19. Kang, U., Tsourakakis, C.E., Faloutsos, C.: PEGASUS: A peta-scale graph mining system implementation and observations. In: IEEE International Conference on Data Mining (ICDM), pp. 229–238 (2009)

  20. Idreos, S., Kersten, M.L., Manegold, S.: Self-organizing tuple reconstruction in column-stores. In: ACM International Conference on Management of Data (SIGMOD), pp. 297–308 (2009)

  21. Idreos, S., Kersten, M.L., Manegold, S.: Database cracking. In: Conference on Innovative Data Systems Research (CIDR), pp. 68–78 (2007)

  22. Graefe, G., Kuno, H.: Self-selecting, self-tuning, incrementally optimized indexes. In: International Conference on Extending Database Technology (EDBT), pp. 371–381 (2010)

  23. Graefe, G., Kuno, H.A.: Adaptive indexing for relational keys. In: Workshop on Self-Managing Database Systems (SMDB), pp. 69–74 (2010)

  24. Idreos, S., Manegold, S., Kuno, H., Graefe, G.: Merging what’s cracked, cracking what’s merged: adaptive indexing in main-memory column-stores. In: International Conference on Very Large Data Bases (VLDB), pp. 586–597 (2014)

  25. Wu, E., Madden, S.: Partitioning techniques for fine-grained indexing. In: IEEE International Conference on Data Engineering (ICDE), pp. 1127–1138 (2011)

  26. Neo4j Graph Database. http://www.neo4j.org

  27. Kang, U., Tong, H., Sun, J., Lin, C.Y., Faloutsos, C.: GBASE: A scalable and general graph management system. In: ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD), pp. 1091–1099 (2011)

  28. Khurana, U., Deshpande, A.: Efficient snapshot retrieval over historical graph data. In: IEEE International Conference on Data Engineering (ICDE), pp. 997–1008 (2013)

  29. Bornea, M.A., Dolby, J., Kementsietsidis, A., Srinivas, K., Dantressangle, P., Udrea, O., Bhattacharjee, B.: Building an efficient rdf store over a relational database. In: ACM International Conference on Management of Data (SIGMOD), pp. 121–132 (2013)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Robert Soulé.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Soulé, R., Gedik, B. RailwayDB: adaptive storage of interaction graphs. The VLDB Journal 25, 151–169 (2016). https://doi.org/10.1007/s00778-015-0407-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-015-0407-0

Keywords

Navigation