Fan Guo
John C. S. Lui
ABSTRACT
Docker containers are widely deployed to provide lightweight virtualization, and they have many desirable features such as ease of deployment and near bare-metal performance. However, both the performance and cache efficiency of containers are still limited by their storage drivers due to the coarse-grained copy-on-write operations, and the large amount of redundancy in both I/O requests and page cache. To improve I/O performance and cache efficiency of containers, we develop HP-Mapper, a high performance storage driver for Docker containers. HP-Mapper provides a two-level mapping strategy to support fine-grained copy-on-write with low overhead, and an efficient interception method to reduce redundant I/Os. Furthermore, it uses a novel cache management mechanism to reduce duplicate cached data. Experiment results with our prototype system show that HP-Mapper significantly reduces copy-on-write latency due to its finer-grained copy-on-write scheme. Moreover, HP-Mapper can also reduce 65.4% cache usage on average due to elimination of duplicated data. As a result, HP-Mapper improves the throughput of real-world workloads by up to 39.4%, and improves the startup speed of containers by 2.0x.
- 2019. AUFS: Another Union Filesystem. http://aufs.sourceforge.net.Google Scholar
- 2019. AWS containers. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html.Google Scholar
- 2019. Azure containers. https://azure.microsoft.com/en-us/overview/containers.Google Scholar
- 2019. CaaS, the foundation of next generation PaaS. https://kubernetes.io/blog/2017/02/caas-the-foundation-for-next-gen-paas/.Google Scholar
- 2019. Cgroups. https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt.Google Scholar
- 2019. Container services on google cloud. https://cloud.google.com/containers/.Google Scholar
- 2019. Device Mapper. https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt.Google Scholar
- 2019. Docker Hub. https://hub.docker.com/explore/.Google Scholar
- 2019. hugetlbfs. https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt.Google Scholar
- 2019. Images and storage drivers of Docker containers. https://docs.docker.com/storage/storagedriver/.Google Scholar
- 2019. iostat. https://linux.die.net/man/1/iostat.Google Scholar
- 2019. Namespaces. http://man7.org/linux/man-pages/man7/namespaces.7.html.Google Scholar
- 2019. Overlay filesystem. https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt.Google Scholar
- 2019. Select a storage driver. https://docs.docker.com/storage/storagedriver/select-storage-driver/.Google Scholar
- 2019. ZFS. https://en.wikipedia.org/wiki/ZFS.Google Scholar
- Abutalib Aghayev, Theodore Ts'o, Garth Gibson, and Peter Desnoyers. 2017. Evolving Ext4 for Shingled Disks. In 15th USENIX Conference on File and Storage Technologies (FAST 17). USENIX Association, 105--120.Google ScholarDigital Library
- Andrea Arcangeli, Izik Eidus, and Chris Wright. 2009. Increasing memory density by using KSM. In Proceedings of the linux symposium. Citeseer, 19--28.Google Scholar
- Sergei Arnautov, Bohdan Trach, Franz Gregor, Thomas Knauth, Andre Martin, Christian Priebe, Joshua Lind, Divya Muthukumaran, Dan O'Keeffe, Mark L. Stillwell, David Goltzsche, Dave Eyers, Rüdiger Kapitza, Peter Pietzuch, and Christof Fetzer. 2016. SCONE: Secure Linux Containers with Intel SGX. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, 689--703.Google ScholarDigital Library
- Uchechukwu Awada and Adam Barker. 2017. Improving resource efficiency of container-instance clusters on clouds. In Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing. IEEE Press, 929--934.Google ScholarDigital Library
- Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. 2003. Xen and the art of virtualization. In ACM SIGOPS operating systems review, Vol. 37. ACM, 164--177.Google Scholar
- Janki Bhimani, Zhengyu Yang, Ningfang Mi, Jingpei Yang, Qiumin Xu, Manu Awasthi, Rajinikanth Pandurangan, and Vijay Balakrishnan. [n.d.]. Docker Container Scheduler for I/O Intensive Applications running on NVMe SSDs. IEEE Transactions on Multi-Scale Computing Systems 1 ([n.d.]), 1--1.Google Scholar
- Zhen Cao, Vasily Tarasov, Hari Prasath Raman, Dean Hildebrand, and Erez Zadok. 2017. On the Performance Variation in Modern Storage Stacks. In 15th USENIX Conference on File and Storage Technologies (FAST 17). USENIX Association, 329--344.Google Scholar
- Chaoquan Chen, Zhengzheng Zhang, and Xiaolan Xie. 2018. Container Cloud Resource Allocation Based on Combinatorial Double Auction. In Proceedings of the 3rd International Conference on Intelligent Information Processing. ACM, 146--151.Google ScholarDigital Library
- Douglas Comer. 1979. Ubiquitous B-tree. ACM Computing Surveys (CSUR) 11, 2 (1979), 121--137.Google ScholarDigital Library
- Benoit des Ligneris. 2005. Virtualization of Linux based computers: the Linux-VServer project. In High Performance Computing Systems and Applications, 2005. HPCS 2005. 19th International Symposium on. IEEE, 340--346.Google ScholarDigital Library
- Lian Du, Tianyu Wo, Renyu Yang, and Chunming Hu. 2017. Cider: A Rapid Docker Container Deployment System through Sharing Network Storage. In High Performance Computing and Communications; IEEE 15th International Conference on Smart City; IEEE 3rd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), 2017 IEEE 19th International Conference on. IEEE, 332--339.Google Scholar
- Rajdeep Dua, Vaibhav Kohli, Sriram Patil, and Swapnil Patil. 2016. Performance analysis of Union and CoW File Systems with Docker. In Computing, Analytics and Security Trends (CAST), International Conference on. IEEE, 550--555.Google Scholar
- Tyler Harter, Brandon Salmon, Rose Liu, Andrea C Arpaci-Dusseau, and Remzi H Arpaci-Dusseau. 2016. Slacker: Fast Distribution with Lazy Docker Containers.. In FAST, Vol. 16. 181--195.Google ScholarDigital Library
- Matt Helsley. 2009. LXC: Linux container tools. IBM devloperWorks Technical Library 11 (2009).Google Scholar
- Luis Herrera-Izquierdo and Marc Grob. 2017. A performance evaluation between Docker container and Virtual Machines in cloud computing architectures. Maskana 8 (2017), 127--133.Google Scholar
- Hyukjoong Kim, Dongkun Shin, Yun Ho Jeong, and Kyung Ho Kim. 2017. SHRD: Improving Spatial Locality in Flash Storage Accesses by Sequentializing in Host and Randomizing in Device. In 15th USENIX Conference on File and Storage Technologies (FAST 17). USENIX Association, 271--284.Google Scholar
- Avi Kivity, Yaniv Kamay, Dor Laor, Uri Lublin, and Anthony Liguori. 2007. kvm: the Linux virtual machine monitor. In Proceedings of the Linux symposium, Vol. 1. Dttawa, Dntorio, Canada, 225--230.Google Scholar
- Zhanibek Kozhirbayev and Richard O Sinnott. 2017. A performance comparison of container-based technologies for the cloud. Future Generation Computer Systems 68 (2017), 175--182.Google ScholarCross Ref
- Ying Mao, Jenna Oak, Anthony Pompili, Daniel Beer, Tao Han, and Peizhao Hu. 2017. Draps: Dynamic and resource-aware placement scheme for docker containers in a heterogeneous cluster. In Performance Computing and Communications Conference (IPCCC), 2017 IEEE 36th International. IEEE, 1--8.Google ScholarCross Ref
- Dirk Merkel. 2014. Docker: lightweight linux containers for consistent development and deployment. Linux Journal 2014, 239 (2014), 2.Google ScholarDigital Library
- Edward Oakes, Leon Yang, Dennis Zhou, Kevin Houck, Tyler Harter, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. 2018. SOCK: Rapid Task Provisioning with Serverless-Optimized Containers. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC' 18). USENIX Association.Google Scholar
- Ohad Rodeh, Josef Bacik, and Chris Mason. 2013. BTRFS: The Linux B-tree filesystem. ACM Transactions on Storage (TOS) 9, 3 (2013), 9.Google ScholarDigital Library
- James E Smith and Ravi Nair. 2005. The architecture of virtual machines. Computer 38, 5 (2005), 32--38.Google ScholarDigital Library
- Yuqiong Sun, David Safford, Mimi Zohar, Dimitrios Pendarakis, Zhongshu Gu, and Trent Jaeger. 2018. Security Namespace: Making Linux Security Frameworks Available to Containers. In 27th USENIX Security Symposium (USENIX Security 18). USENIX Association, 1423--1439.Google ScholarDigital Library
- Vasily Tarasov, Lukas Rupprecht, Dimitris Skourtis, Amit Warke, Dean Hildebrand, Mohamed Mohamed, Nagapramod Mandagere, Wenji Li, Raju Rangaswami, and Ming Zhao. 2017. In search of the ideal storage configuration for Docker containers. In Foundations and Applications of Self* Systems (FAS* W), 2017 IEEE 2nd International Workshops on. IEEE, 199--206.Google Scholar
- Vasily Tarasov, Erez Zadok, and Spencer Shepler. 2016. Filebench: A flexible framework for file system benchmarking.; login: The USENIX Magazine, 41 (1): 6--12.Google Scholar
- Jörg Thalheim, Pramod Bhatotia, Pedro Fonseca, and Baris Kasikci. 2018. Cntr: Lightweight OS Containers. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC' 18). USENIX Association.Google Scholar
- Bharath Kumar Reddy Vangoor, Vasily Tarasov, and Erez Zadok. 2017. To FUSE or Not to FUSE: Performance of User-Space File Systems. In 15th USENIX Conference on File and Storage Technologies (FAST 17). USENIX Association, 59--72.Google Scholar
- Qiumin Xu, Manu Awasthi, Krishna T Malladi, Janki Bhimani, Jingpei Yang, Murali Annavaram, and Hsieh Ming. 2017. Performance Analysis of Containerized Applications on Local and Remote Storage. In International Conference on Massive Storage Systems and Technology.Google Scholar
- Qi Zhang, Lu Cheng, and Raouf Boutaba. 2010. Cloud computing: state-of-the-art and research challenges. Journal of internet services and applications 1, 1 (2010), 7--18.Google ScholarCross Ref
Index Terms
- HP-Mapper: A High Performance Storage Driver for Docker Containers
Recommendations
Hint-K: An Efficient Multilevel Cache Using K-Step Hints
I/O performance has been critical for large-scale distributed systems. Many approaches, including hint-based multilevel cache, have been proposed to smooth the gap between different levels. These solutions demote or promote cache blocks based on the ...
ELI: bare-metal performance for I/O virtualization
ASPLOS '12Direct device assignment enhances the performance of guest virtual machines by allowing them to communicate with I/O devices without host involvement. But even with device assignment, guests are still unable to approach bare-metal performance, because ...
ELI: bare-metal performance for I/O virtualization
ASPLOS XVII: Proceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating SystemsDirect device assignment enhances the performance of guest virtual machines by allowing them to communicate with I/O devices without host involvement. But even with device assignment, guests are still unable to approach bare-metal performance, because ...
Comments