Skip to main content
SpringerLink
Log in

Evaluating Docker storage performance: from workloads to graph drivers

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Containers are a widely successful technology today popularized by Docker. They improve system utilization by increasing workload density and enable seamless deployment of workloads across development, test, and production environments. Docker’s unique approach to data management, which involves frequent snapshot creation and removal, presents a new set of exciting challenges for storage systems. At the same time, storage management for Docker containers has remained largely unexplored with a dizzying array of solution choices and configuration options. In this paper we unravel the multi-faceted nature of Docker storage and demonstrate its impact on system and workload performance. As we uncover new properties of the popular Docker storage drivers, this is a sobering reminder that widespread use of new technologies can often precede their careful evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. Storage drivers are sometimes also called graphdrivers because they maintain the graph (tree) of Docker layers and images.

References

  1. 451 Research: Application Containers Will Be a \$2.7Bn Market by 2020. https://451research.com/images/Marketing/press_releases/Application-container-market-will-reach-2-7bn-in-2020_final_graphic.pdf. Accessed March 2018

  2. Control Group v2. https://www.kernel.org/doc/Documentation/cgroup-v2.txt. Accessed March 2018

  3. Biederman, E.: Multiple instances of the global Linux namespaces. In: Linux Symposium (2006)

  4. Docker. https://www.docker.com/. Accessed March 2018

  5. AUFS—Another Union Filesystem. http://aufs.sourceforge.net. Accessed March 2018

  6. Overlay Filesystem. https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt. Accessed March 2018

  7. Rodeh, O., Bacik, J., Mason, C.: BTRFS: The Linux B-Tree Filesystem. ACM Trans. Storage 9(3), 9 (2013)

    Article  Google Scholar 

  8. Bonwick, J., Ahrens, M., Henson, V., Maybee, M., Shellenbaum, M.: The Zettabyte File System. In: Proceedings of the 2nd Usenix Conference on File and Storage Technologies (FAST) (2003)

  9. Device Mapper Thin Provisioning Targets. https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt. Accessed March 2018

  10. Harter, T., Salmon, B., Liu, R., Arpaci-Dusseau, A., Arpaci-Dusseau, R.: Slacker: Fast Distribution with Lazy Docker Containers. In: Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST) (2016)

  11. Lamourine, M.: Storage Options for Software Containers. ;login: The USENIX Magazine 40(1) (2015)

  12. Docker Hub. https://hub.docker.com/. Accessed March 2018

  13. Docker Swarm Mode. https://docs.docker.com/engine/swarm/. Accessed March 2018

  14. Kubernetes. https://www.kubernetes.io/. Accessed March 2018

  15. Juju. https://jujucharms.com/. Accessed March 2018

  16. fabric8. https://fabric8.io/. Accessed March 2018

  17. ZFS on Linux. http://zfsonlinux.org/. Accessed March 2018

  18. Solaris Porting Layer. https://github.com/zfsonlinux/spl. Accessed March 2018

  19. Select a Storage Driver. https://docs.docker.com/engine/userguide/storagedriver/selectadriver/. Accessed March 2018

  20. Spillane, R.P., Wang, W., Lu, L., Austruy, M., Rivera, R., Karamanolis, C.: Exo-clones: Better Container Runtime Image Management Across the Clouds. In: Proceedings of the 8th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage) (2016)

  21. Amazon EC2 Instance Types. https://aws.amazon.com/ec2/instance-types/. Accessed March 2018

  22. Docker and OverlayFS in Practice. https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/. Accessed March 2018

  23. Aufs Returns ENOSUP when Writing in the Middle of a File on XFS. https://sourceforge.net/p/aufs/mailman/aufs-users/thread/27641.1496877655%40jrobl/. Accessed March 2018

  24. ZFS on Linux for RHEL and CentOS. https://github.com/zfsonlinux/zfs/wiki/RHEL-and-CentOS. Accessed March 2018

  25. Tarasov, V., Zadok, E., Shepler, S.: Filebench: A Flexible Framework for File System Benchmarking. ;login: The USENIX Magazine 41(1) (2016)

  26. Campello, D., Lopez, H., Useche, L., Koller, R., Rangaswami, R.: Non-blocking Writes to Files. In: Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST) (2015)

  27. Dockerfile Reference. https://docs.docker.com/engine/reference/builder/. Accessed March 2018

  28. Santana, R., Rangaswami, R., Tarasov, V., Hildebrand, D.: A fast and slippery slope for file systems. ACM SIGOPS Oper. Syst. Rev. 49(2), 27–34 (2016)

    Article  Google Scholar 

  29. Dm-delay. https://www.kernel.org/doc/Documentation/device-mapper/delay.txt. Accessed March 2018

  30. Ubuntu Thinks Btrfs Disk Is Full but It Is Not. https://askubuntu.com/questions/464074/ubuntu-thinks-btrfs-disk-is-full-but-its-not. Accessed March 2018

  31. Btrfs Out of Space, Even Though There Should Be 10% Left. https://superuser.com/questions/1096658/btrfs-out-of-space-even-though-there-should-be-10-left. Accessed March 2018

  32. Btrfs: No Space Left on Device. https://nathantypanski.com/blog/2014-07-14-no-space-left.html. Accessed March 2018

  33. Eder, J., Murphy, C.: Performance Analysis of Docker on Red Hat Enterprise Linux 7 (2014). https://developers.redhat.com/blog/2014/08/19/performance-analysis-docker-red-hat-enterprise-linux-7/. Accessed March 2018

  34. Eder, J.: Comprehensive overview of storage scalability in Docker (2014). https://developers.redhat.com/blog/2014/09/30/overview-storage-scalability-docker/. Accessed March 2018

  35. Estes, P.: Storage Drivers in Docker: A Deep Dive (2016). https://integratedcode.us/2016/08/30/storage-drivers-in-docker-a-deep-dive/. Accessed March 2018

  36. Simon, R., von Eicken, T.: Perspectives on Docker (2014). https://www.slideshare.net/rightscale/docker-meetup-40826948. Accessed March 2018

  37. Tyczynski, W.: 1000 Nodes and Beyond: Updates to Kubernetes Performance and Scalability in 1.2 (2016). http://blog.kubernetes.io/2016/03/1000-nodes-and-beyond-updates-to-Kubernetes-performance-and-scalability-in-12.html. Accessed March 2018

  38. Sharma, P., Chaufournier, L., Shenoy, P.J., Tay, Y.: Containers and Virtual Machines at Scale: A Comparative Study. In: Proceedings of the 17th International Middleware Conference (Middleware) (2016)

  39. Hegde, A., Ghosh, R., Mukherjee, T., Sharma, V.: SCoPe: A Decision System for Large Scale Container Provisioning Management. In: Proceedings of the 9th IEEE International Conference on Cloud Computing (CLOUD) (2016)

  40. Jannen, W., Yuan, J., Zhan, Y., Akshintala, A., Esmet, J., Jiao, Y., Mittal, A., Pandey, P., Reddy, P., Walsh, L., Bender, M., Farach-Colton, M., Johnson, R., Kuszmaul, B.C., Porter, D.E.: BetrFS: A Right-Optimized Write-Optimized File System. In: Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST) (2015)

  41. Zhao, F., Xu, K., Shain, R.: Improving Copy-on-Write Performance in Container Storage Drivers. In: Storage Developers Conference (2016)

  42. Wu, X., Wang, W., Jiang, S.: TotalCOW: Unleash the Power of Copy-on-Write for Thin-Provisioned Containers. In: Proceedings of the 6th Asia-Pacific Workshop on Systems (APsys) (2015)

  43. Canon, R.S., Jacobsen, D.: Shifter: Containers for HPC. In: Cray User Group (2016)

  44. Nathan, S., Ghosh, R., Mukherjee, T., Narayanan, K.: CoMICon: A Co-Operative Management System for Docker Container Images. In: Proceedings of the 5th IEEE International Conference on Cloud Engineering (IC2E) (2017)

Download references

Acknowledgements

This manuscript is an extended version of the paper titled “In Search of the Ideal Storage Configuration for Docker Containers” published in Proceedings of the Workshop on Autonomic Management of Large Scale Container-based Systems (AMLCS) in 2017. We thank the co-authors of the workshop paper Amit Warke, Dean Hildebrand, Mohamed Mohamed, and Nagapramod Mandagere for the contributions to the project. We also thank the AMLCS reviewers for their valuable comments. This work was supported in part by the NSF via Grants CNS-1563883, CNS-1320426, CNS-1562837, and CNS-1619653.

Author information

Authors and Affiliations

  1. IBM Research, San Jose, USA

    Vasily Tarasov, Lukas Rupprecht & Dimitris Skourtis

  2. Arizona State University, Tempe, USA

    Wenji Li & Ming Zhao

  3. Florida International University, Miami, USA

    Raju Rangaswami

Authors
  1. Vasily Tarasov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lukas Rupprecht
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dimitris Skourtis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenji Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Raju Rangaswami
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ming Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vasily Tarasov.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tarasov, V., Rupprecht, L., Skourtis, D. et al. Evaluating Docker storage performance: from workloads to graph drivers. Cluster Comput 22, 1159–1172 (2019). https://doi.org/10.1007/s10586-018-02893-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-018-02893-y

Keywords

Search

Navigation