research-article

Baoverlay: a block-accessible overlay file system for fast and efficient container storage

Authors:

Hui LuAuthors Info & Claims

SoCC '20: Proceedings of the 11th ACM Symposium on Cloud Computing

Pages 90 - 104

https://doi.org/10.1145/3419111.3421291

Published: 12 October 2020 Publication History

Abstract

Container storage commonly relies on overlay file systems to interpose read-only container images upon backing file systems. While being transparent to and compatible with most existing backing file systems, the overlay file-system approach imposes nontrivial I/O overhead to containerized applications, especially for writes: To write a file originating from a read-only container image, the whole file will be copied to a separate, writable storage layer, resulting in long write latency and inefficient use of container storage. In this paper, we present BAOverlay, a lightweight, block-accessible overlay file system: Equipped with a new block-accessibility attribute, BAOverlay not only exploits the benefit of using an asynchronous copy-on-write mechanism for fast file updates but also enables a new file format for efficient use of container storage space. We have developed a prototype of BAOverlay upon Linux Ext4. Our evaluation with both micro-benchmarks and real-world applications demonstrates the effectiveness of BAOverlay with improved write performance and on-demand container storage usage.

Supplementary Material

MP4 File (p90-sun-presentation.mp4)

Download
222.81 MB

References

[1]

Alexandra Agache, Marc Brooker, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and Diana-Maria Popa. 2020. Firecracker: Lightweight Virtualization for Serverless Applications. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). 419--434.

[2]

David G Andersen, Jason Franklin, Michael Kaminsky, Amar Phanishayee, Lawrence Tan, and Vijay Vasudevan. 2009. FAWN: A fast array of wimpy nodes. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. 1--14.

Digital Library

[3]

AWS. 2020. AWS Lambda: Run code without thinking about servers. Pay only for the compute time you consume. https://aws.amazon.com/lambda/.

[4]

Jens Axboe. 2020. fio(1) - Linux man page. https://hub.docker.com/.

[5]

Jon CR Bennett and Hui Zhang. 1996. WF/sup 2/Q: worst-case fair weighted fair queueing. In INFOCOM'96. Fifteenth Annual Joint Conference of the IEEE Computer Societies. Networking the Next Generation. Proceedings IEEE.

[6]

Jon CR Bennett and Hui Zhang. 1997. Hierarchical packet fair queueing algorithms. IEEE/ACM Transactions on Networking (TON) 5, 5 (1997), 675--689.

Digital Library

[7]

David Bernstein. 2014. Containers and cloud: From lxc to docker to kubernetes. IEEE Cloud Computing 1, 3 (2014), 81--84.

[8]

Anton Burtsev, Kiran Srinivasan, Prashanth Radhakrishnan, Kaladhar Voruganti, and Garth R Goodson. 2009. Fido: Fast Inter-Virtual-Machine Communication for Enterprise Appliances. In USENIX Annual technical conference.

[9]

Google Cloud. 2020. Containers at Google. https://cloud.google.com/containers/.

[10]

Kata Containers community. 2020. The speed of containers, the security of VMs. https://katacontainers.io/.

[11]

Brian F Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking cloud serving systems with YCSB. In Proceedings of the 1st ACM symposium on Cloud computing. 143--154.

Digital Library

[12]

Christina Delimitrou and Christos Kozyrakis. 2013. Paragon: QoS-aware scheduling for heterogeneous datacenters. In ACM SIGPLAN Notices, Vol. 48. 77--88.

Digital Library

[13]

Alan Demers, Srinivasan Keshav, and Scott Shenker. 1989. Analysis and simulation of a fair queueing algorithm. In ACM SIGCOMM Computer Communication Review, Vol. 19. 1--12.

Digital Library

[14]

Docker Docs. 2020. About storage drivers. https://docs.docker.com/storage/storagedriver/.

[15]

Docker Docs. 2020. Use the OverlayFS storage driver. https://docs.docker.com/storage/storagedriver/overlayfs-driver/.

[16]

Docker Docs. 2020. Use volumes. https://docs.docker.com/storage/volumes/.

[17]

VMWare Docs. 2020. Single Root I/O Virtualization (SRIOV). https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-CC021803-30EA-444D-BCBE-618E0D836B9F.html.

[18]

Garth Gibson and Greg Ganger. 2011. Principles of operation for shingled disk devices. (2011).

[19]

GitHub. 2020. Aufs5. https://github.com/sfjro/aufs5-linux.

[20]

Mel Gorman. 2004. Understanding the Linux virtual memory manager. Prentice Hall Upper Saddle River.

[21]

Ajay Gulati, Irfan Ahmad, and Carl A. Waldspurger. 2009. PARDA: Proportional Allocation of Resources for Distributed Storage Access. In Proceedings of the USENIX Conference on File and Storage Technologies.

[22]

Ajay Gulati, Arif Merchant, and Peter J. Varman. 2007. pClock: an arrival curve based approach for QoS guarantees in shared storage systems. In SIGMETRICS. ACM.

[23]

Tyler Harter, Brandon Salmon, Rose Liu, Andrea C Arpaci-Dusseau, and Remzi H Arpaci-Dusseau. 2016. Slacker: Fast distribution with lazy docker containers. In 14th USENIX Conference on File and Storage Technologies (FAST 16).

[24]

Docker Inc. 2020. Get Started with Docker. https://www.docker.com.

[25]

William Jannen, Jun Yuan, Yang Zhan, Amogh Akshintala, John Esmet, Yizheng Jiao, Ankur Mittal, Prashant Pandey, Phaneendra Reddy, Leif Walsh, et al. 2015. BetrFS: A right-optimized write-optimized file system. In 13th USENIX Conference on File and Storage Technologies (Fast 15). 301--315.

Digital Library

[26]

Wei Jin, Jeffrey S Chase, and Jasleen Kaur. 2004. Interposed proportional sharing for a storage service utility. ACM SIGMETRICS Performance Evaluation Review 32, 1 (2004), 37--48.

Digital Library

[27]

Meaza Taye Kebede. 2012. Performance comparison of btrfs and ext4 filesystems. Master's thesis.

[28]

Terence Kelly, Ira Cohen, Moises Goldszmidt, and Kimberly Keeton. 2004. Inducing models of black-box storage arrays. HP Laboratories, Palo Alto, CA, Technical Report HPL-2004--108 (2004).

[29]

Linux Kernel. 2020. CFQ. https://www.kernel.org/doc/Documentation/block/cfq-iosched.txt.

[30]

Alexey Kopytov. 2004. SysBench manual. (2004).

[31]

Harendra Kumar, Yuvraj Patel, Ram Kesavan, and Sumith Makam. 2017. High Performance Metadata Integrity Protection in the {WAFL} Copy-on-Write File System. In 15th USENIX Conference on File and Storage Technologies (FAST 17). 197--212.

Digital Library

[32]

Changman Lee, Dongho Sim, Jooyoung Hwang, and Sangyeun Cho. 2015. F2FS: A new file system for flash storage. In 13th USENIX Conference on File and Storage Technologies (FAST 15). 273--286.

Digital Library

[33]

Jiuxing Liu, Wei Huang, Bulent Abali, and Dhabaleswar K. Panda. 2006. High Performance VMM-bypass I/O in Virtual Machines. In Proceedings of the Annual Conference on USENIX '06 Annual Technical Conference.

[34]

David Lomet and Mark Tuttle. 1999. Logical logging to extend recovery to new domains. ACM SIGMOD Record 28, 2 (1999), 73--84.

Digital Library

[35]

Hui Lu, Brendan Saltaformaggio, Ramana Kompella, and Dongyan Xu. 2015. vFair: Latency-aware Fair Storage Scheduling via per-IO Cost-based Differentiation. In Proceedings of the 6th ACM Symposium on Cloud Computing.

Digital Library

[36]

Hui Lu, Brendan Saltaformaggio, Cong Xu, Umesh Bellur, and Dongyan Xu. 2016. Bass: Improving i/o performance for cloud block storage via byte-addressable storage stack. In Proceedings of the Seventh ACM Symposium on Cloud Computing.

Digital Library

[37]

Garrett McGrath and Paul R Brenner. 2017. Serverless computing: Design, implementation, and performance. In 2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW). IEEE, 405--410.

[38]

Athicha Muthitacharoen, Robert Morris, Thomer M Gil, and Benjie Chen. 2002. Ivy: A read/write peer-to-peer file system. ACM SIGOPS Operating Systems Review 36, SI (2002), 31--44.

[39]

Jun Nakajima. 2007. Intel virtualization technology roadmap and VT-d support in Xen. http://www-archive.xenproject.org/files/xensummit_4/VT_roadmap_d_Nakajima.pdf. (2007).

[40]

Ripal Nathuji, Aman Kansal, and Alireza Ghaffarkhah. 2010. Q-clouds: Managing performance interference effects for qos-aware clouds. In Proceedings of the 5th European Conference on Computer Systems.

Digital Library

[41]

Edward Oakes, Leon Yang, Dennis Zhou, Kevin Houck, Tyler Harter, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. 2018. SOCK: Rapid task provisioning with serverless-optimized containers. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). 57--70.

[42]

Junjiro R. Okajima. 2020. Linux AuFS Examples: Another Union File System Tutorial. http://aufs.sourceforge.net/aufs2/.

[43]

Openzfs. 2020. zfs-0.8.3. https://github.com/openzfs/zfs/releases.

[44]

Abhay K Parekh and Robert G Gallager. 1993. A generalized processor sharing approach to flow control in integrated services networks: the single-node case. IEEE/ACM transactions on networking 1, 3 (1993), 344--357.

[45]

Abhishek Prakash. 2020. Don't Use ZFS on Linux. https://itsfoss.com/linus-torvalds-zfs/.

[46]

Ohad Rodeh, Josef Bacik, and Chris Mason. 2013. BTRFS: The Linux B-tree filesystem. ACM Transactions on Storage (TOS) 9, 3 (2013), 1--32.

Digital Library

[47]

Yannis Sfakianakis, Stelios Mavridis, Anastasios Papagiannis, Spyridon Papageorgiou, Markos Fountoulakis, Manolis Marazakis, and Angelos Bilas. 2014. Vanguard: Increasing Server Efficiency via Workload Isolation in the Storage I/O Path. In Proceedings of the ACM Symposium on Cloud Computing.

Digital Library

[48]

Amit Shah, Allen M. Kay, Muli Ben-Yehuda, and Ben-Ami Yassour. 2008. PCI Device Passthrough for KVM. https://www.linux-kvm.org/images/d/d0/KvmForum2008%24kdf2008_14.pdf. (2008).

[49]

Simon Shillaker and Peter Pietzuch. 2020. Faasm: Lightweight Isolation for Efficient Stateful Serverless Computing. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). USENIX Association, 419--433. https://www.usenix.org/conference/atc20/presentation/shillaker

[50]

David Shue, Michael J. Freedman, and Anees Shaikh. 2012. Performance Isolation and Fairness for Multi-Tenant Cloud Storage. In Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation.

[51]

Vasily Tarasov, Lukas Rupprecht, Dimitris Skourtis, Wenji Li, Raju Rangaswami, and Ming Zhao. 2019. Evaluating Docker storage performance: from workloads to graph drivers. Cluster Computing 22, 4 (2019), 1159--1172.

Digital Library

[52]

Steven J. Vaughan-Nichols. 2020. Docker makes ready-to-run container apps available. https://www.zdnet.com/article/docker-makes-ready-to-run-container-apps-available/.

[53]

Zhikun Wang, Dan Feng, Ke Zhou, and Fang Wang. 2008. PCOW: Pipelining-based COW snapshot method to decrease first write penalty. In International Conference on Grid and Pervasive Computing. Springer, 266--274.

[54]

Brent Welch, Marc Unangst, Zainul Abbasi, Garth A Gibson, Brian Mueller, Jason Small, Jim Zelenka, and Bin Zhou. 2008. Scalable Performance of the Panasas Parallel File System. In Proceedings of the USENIX Conference on File and Storage Technologies.

[55]

Charles P. Wright and Erez Zadok. 2004. Kernel Korner: Unionfs: Bringing Filesystems Together. Linux J. 2004, 128 (Dec. 2004), 8.

[56]

Cong Xu, Sahan Gamage, Hui Lu, Ramana Rao Kompella, and Dongyan Xu. 2013. vTurbo: Accelerating Virtual Machine I/O Processing Using Designated Turbo-Sliced Core. In USENIX Annual Technical Conference.

[57]

Ethan G Young, Pengfei Zhu, Tyler Caraza-Harter, Andrea C Arpaci-Dusseau, and Remzi H Arpaci-Dusseau. 2019. The true cost of containing: A gVisor case study. In 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19).

[58]

Jun Yuan, Yang Zhan, William Jannen, Prashant Pandey, Amogh Akshintala, Kanchan Chandnani, Pooja Deo, Zardosht Kasheff, Leif Walsh, Michael Bender, et al. 2016. Optimizing every operation in a write-optimized file system. In 14th USENIX Conference on File and Storage Technologies (FAST 16). 1--14.

Digital Library

[59]

Erez Zadok, Rakesh Iyer, Nikolai Joukov, Gopalan Sivathanu, and Charles P Wright. 2006. On incremental file system development. ACM Transactions on Storage (TOS) 2, 2 (2006), 161--196.

Digital Library

[60]

Lixia Zhang. 1991. VirtualClock: a new traffic control algorithm for packet-switched networks. ACM Transactions on Computer Systems (TOCS) 9, 2 (1991), 101--124.

Digital Library

[61]

Shuanglong Zhang, Helen Catanese, and Andy An-I Wang. 2016. The composite-file file system: Decoupling the one-to-one mapping of files and metadata for better performance. In 14th USENIX Conference on File and Storage Technologies (FAST 16). 15--22.

Digital Library

[62]

Xiaolan Zhang, Suzanne McIntosh, Pankaj Rohatgi, and John Linwood Griffin. 2007. XenSocket: A High-throughput Interdomain Transport for Virtual Machines. In Proceedings of the ACM/IFIP/USENIX 2007 International Conference on Middleware.

[63]

Nannan Zhao, Vasily Tarasov, Hadeel Albahar, Ali Anwar, Lukas Rupprecht, Dimitrios Skourtis, Amit S Warke, Mohamed Mohamed, and Ali R Butt. 2019. Large-Scale Analysis of the Docker Hub Dataset. In 2019 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 1--10.

Cited By

Zhang DXie YXu MCheng EKui XHe BLi Y(2024)Troy: Efficient Service Deployment for Windows SystemsChinese Journal of Electronics10.23919/cje.2022.00.40533:1(313-322)Online publication date: Jan-2024
https://doi.org/10.23919/cje.2022.00.405
Han H(2023)High-Performance OverlayFS for ContainersJournal of Digital Contents Society10.9728/dcs.2023.24.11.284124:11(2841-2847)Online publication date: 30-Nov-2023
https://doi.org/10.9728/dcs.2023.24.11.2841
Choi KSeo HHan HRyu MKang S(2023)CredsCache: Making OverlayFS scalable for containerized servicesFuture Generation Computer Systems10.1016/j.future.2023.04.027147(44-58)Online publication date: Oct-2023
https://doi.org/10.1016/j.future.2023.04.027
Show More Cited By

Index Terms

Baoverlay: a block-accessible overlay file system for fast and efficient container storage
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        File systems management

Recommendations

My VM is Lighter (and Safer) than your Container
SOSP '17: Proceedings of the 26th Symposium on Operating Systems Principles

Containers are in great demand because they are lightweight when compared to virtual machines. On the downside, containers offer weaker isolation than VMs, to the point where people run containers in virtual machines to achieve proper isolation. In this ...
Libservices: dynamic storage provisioning for multitenant I/O isolation
APSys '20: Proceedings of the 11th ACM SIGOPS Asia-Pacific Workshop on Systems

Containers are commonly used to run the data-intensive applications of different tenants in cloud infrastructures. The storage I/O of the colocated tenants is typically handled by the shared system kernel of the container host. When a data-intensive ...
Transfer-sycro-control Mechanism of iSCISI Virtualization Switch to Speed up Its Read/Write Requests
ICICSE '10: Proceedings of the 2010 Fifth International Conference on Internet Computing for Science and Engineering

We present a method of accelerating the execution process of read/write requests for the implementation of the iSCSI virtualization switch used in the SANs. Improving the read/write speed of iSCSI network protocol is important in its storage systems. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SoCC '20: Proceedings of the 11th ACM Symposium on Cloud Computing

October 2020

535 pages

ISBN:9781450381376

DOI:10.1145/3419111

General Chair:
Rodrigo Fonseca
Microsoft and Brown University
,
Program Chairs:
Christina Delimitrou
Cornell University
,
Beng Chin Ooi
National University of Singapore

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SoCC '20

Sponsor:

SoCC '20: ACM Symposium on Cloud Computing

October 19 - 21, 2020

Virtual Event, USA

Acceptance Rates

SoCC '20 Paper Acceptance Rate 35 of 143 submissions, 24%;

Overall Acceptance Rate 169 of 722 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
475
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)5

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang DXie YXu MCheng EKui XHe BLi Y(2024)Troy: Efficient Service Deployment for Windows SystemsChinese Journal of Electronics10.23919/cje.2022.00.40533:1(313-322)Online publication date: Jan-2024
https://doi.org/10.23919/cje.2022.00.405
Han H(2023)High-Performance OverlayFS for ContainersJournal of Digital Contents Society10.9728/dcs.2023.24.11.284124:11(2841-2847)Online publication date: 30-Nov-2023
https://doi.org/10.9728/dcs.2023.24.11.2841
Choi KSeo HHan HRyu MKang S(2023)CredsCache: Making OverlayFS scalable for containerized servicesFuture Generation Computer Systems10.1016/j.future.2023.04.027147(44-58)Online publication date: Oct-2023
https://doi.org/10.1016/j.future.2023.04.027
Lee HHan SLee D(2022)Kernel-Based Container File Access Control Architecture to Protect Important Application InformationElectronics10.3390/electronics1201005212:1(52)Online publication date: 23-Dec-2022
https://doi.org/10.3390/electronics12010052
Wu SHuang ZChen PFan HIbrahim SJin HCriswell JWilliams DXia Y(2022)Container-aware I/O stack: bridging the gap between container storage drivers and solid state devicesProceedings of the 18th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3516807.3516818(18-30)Online publication date: 25-Feb-2022
https://dl.acm.org/doi/10.1145/3516807.3516818
Fan HBian SWu SJiang SIbrahim SJin H(2021)Gear: Enable Efficient Container Storage and Deployment with a New Image Format2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS51616.2021.00020(115-125)Online publication date: Jul-2021
https://doi.org/10.1109/ICDCS51616.2021.00020

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten