skip to main content
10.1145/3655038.3665942acmconferencesArticle/Chapter ViewAbstractPublication PageshotstorageConference Proceedingsconference-collections
research-article
Open access

Shadow Filesystems: Recovering from Filesystem Runtime Errors via Robust Alternative Execution

Published: 08 July 2024 Publication History

Abstract

We present Robust Alternative Execution (RAE), an approach to transparently mask runtime errors in performance-oriented filesystems via temporarily executing an alternative shadow filesystem. A shadow filesystem has the primary goal of robustness, achieved through a simple implementation without performance optimizations and concurrency while adhering to the same API and on-disk formats as the base filesystem it enhances. While the base performance-oriented filesystem may contain bugs, the shadow implementation is formally verified, leveraging advancements in the verification of low-level systems code. In the common case, the base filesystem executes and delivers high performance to applications; however, when a bug is triggered, the slow-but-correct shadow takes over, updates state correctly, and then resumes the base, thus providing high availability.

References

[1]
Atul Adya, Daniel Myers, Jon Howell, Jeremy Elson, Colin Meek, Vishesh Khemani, Stefan Fulger, Pan Gu, Lakshminath Bhuvanagiri, Jason Hunter, Roberto Peon, Larry Kai, Alexander Shraer, Arif Merchant, and Kfir Lev-Ari. 2016. Slicer: Auto-Sharding for Datacenter Applications. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI '16). Savannah, GA.
[2]
Abutalib Aghayev, Sage Weil, Michael Kuchnik, Mark Nelson, Gregory Ganger, and George Amvrosiadis. 2019. File Systems Unfit as Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP '19). Ontario, Canada.
[3]
Todd M. Austin. 1999. DIVA: A Reliable Substrate for Deep Submicron Microarchitecture Design. In Proceedings of the 32nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO'99). Haifa, Israel.
[4]
Algirdas A. Avižienis. 1995. The Methodology of N-Version Programming. In Software Fault Tolerance, Michael R. Lyu (Ed.). John Wiley & Sons Ltd., Chapter 2.
[5]
Algirdas A. Avižienis and Liming Chen. 1977. On the Implementation of N-Version Programming for Software Fault Tolerance During Execution. In Proceedings of 1st Annual International Computer Software and Applications Conference (COMPSAC'77). Chicago, USA.
[6]
Algirdas A. Avižienis and John P. J. Kelly. 1984. Fault Tolerance by Design Diversity: Concepts and Experiments. IEEE Computer 17, 8 (August 1984).
[7]
Kevin Boos, Namitha Liyanage, Ramla Ijaz, and Lin Zhong. 2020. Theseus: an Experiment in Operating System Structure and State Management. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI '20). Virtual Conference.
[8]
James Bornholt, Rajeev Joshi, Vytautas Astrauskas, Brendan Cully, Bernhard Kragl, Seth Markle, Kyle Sauri, Drew Schleit, Grant Slatton, Serdar Tasiran, Jacob Van Geffen, and Andrew Warfield. 2021. Using Lightweight Formal Methods to Validate a Key-Value Storage Node in Amazon S3. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP '21). Virtual Event, Germany.
[9]
Matthias Brun, Reto Achermann, Tej Chajed, Jon Howell, Gerd Zellweger, and Andrea Lattuada. 2023. Beyond isolation: OS verification as a foundation for correct applications. In Proceedings of the Workshop on Hot Topics in Operating Systems (HOTOS '23). Providence, Rhode Island.
[10]
George Candea, Shinichi Kawamoto, Yuichi Fujiki, Greg Friedman, and Armando Fox. 2004. Microreboot - A Technique for Cheap Recovery. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI '04). San Francisco, CA.
[11]
Haogang Chen, Tej Chajed, Alex Konradi, Stephanie Wang, Atalay undefinedleri, Adam Chlipala, M. Frans Kaashoek, and Nickolai Zeldovich. 2017. Verifying a high-performance crash-safe file system using a tree specification. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP '17). Shangai, China.
[12]
Haogang Chen, Daniel Ziegler, Tej Chajed, Adam Chlipala, M. Frans Kaashoek, and Nickolai Zeldovich. 2015. Using Crash Hoare logic for certifying the FSCQ file system. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP '15). Monterey, California.
[13]
Zhihao Cheng. 2018. ext4_handle_inode_extension: i_size_read(inode) < EXT4_I(inode)->i_disksize. https://bugzilla.kernel.org/show_bug.cgi?id=217159.
[14]
Jonathan Corbet. 2013. The multiqueue block layer. https://lwn.net/Articles/552904/.
[15]
Jonathan Corbet. 2021. Clarifying memory management with page folios. https://lwn.net/Articles/849538/.
[16]
Jonathan Corbet. 2021. Multi-generational LRU: the next generation. https://lwn.net/Articles/856931/.
[17]
Alex Depoutovitch and Michael Stumm. 2010. Otherworld: Giving Applications a Chance to Survive OS Kernel Crashes. In Proceedings of the 5th European Conference on Computer Systems (EuroSys '10). Paris, France.
[18]
Harish Dattatraya Dixit, Sneha Pendharkar, Matt Beadon, Chris Mason, Tejasvi Chakravarthy, Bharath Muthiah, and Sriram Sankar. 2021. Silent Data Corruptions at Scale. CoRR abs/2102.11245 (2021). https://arxiv.org/abs/2102.11245
[19]
Jake Edge. 2023. Converting filesystems to iomap. https://lwn.net/Articles/935934/.
[20]
Daniel Ford, François Labelle, Florentina I. Popovici, Murray Stokely, Van-Anh Truong, Luiz Barroso, Carrie Grimes, and Sean Quinlan. 2010. Availability in Globally Distributed Storage Systems. In Proceedings of the 9th Symposium on Operating Systems Design and Implementation (OSDI '10). Vancouver, Canada.
[21]
Daniel Fryer, Kuei Sun, Rahat Mahmood, Tinghao Cheng, Shaun Benjamin, Ashvin Goel, and Angela Demke Brown. 2012. Recon: Verifying File System Consistency at Runtime. In Proceedings of the 10th USENIX Symposium on File and Storage Technologies (FAST '12). San Jose, CA.
[22]
Haryadi S. Gunawi, Cindy Rubio-Gonzalez, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Ben Liblit. 2008. EIO: Error Handling is Occasionally Correct. In Proceedings of the 6th USENIX Symposium on File and Storage Technologies (FAST '08). San Jose, CA.
[23]
Jorrit N. Herder, Herbert Bos, Ben Gras, Philip Homburg, and Andrew S. Tanenbaum. 2006. Construction of a Highly Dependable Operating System. In Proceedings of the 6th European Dependable Computing Conference.
[24]
Peter H. Hochschild, Paul Turner, Jeffrey C. Mogul, Rama Govindaraju, Parthasarathy Ranganathan, David E. Culler, and Amin Vahdat. 2021. Cores That Don't Count. In Proceedings of the Workshop on Hot Topics in Operating Systems (HOTOS '21). Ann Arbor, Michigan.
[25]
Lexiang Huang, Matthew Magnusson, Abishek Bangalore Muralikrishna, Salman Estyak, Rebecca Isaacs, Abutalib Aghayev, Timothy Zhu, and Aleksey Charapko. 2022. Metastable Failures in the Wild. In Proceedings of the 16th USENIX Conference on Operating Systems Design and Implementation (OSDI '22). Carlsbad, CA.
[26]
Jonathan Corbet. 2020. The ABI status of filesystem formats. https://lwn.net/Articles/833696/.
[27]
The kernel development community. 2024. Linux kernel coding style: Do not crash the kernel. https://www.kernel.org/doc/html/latest/process/coding-style.html#do-not-crash-the-kernel.
[28]
Seulbae Kim, Meng Xu, Sanidhya Kashyap, Jungyeon Yoon, Wen Xu, and Taesoo Kim. 2019. Finding Semantic Bugs in File Systems with an Extensible Fuzzing Framework. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP '19). Ontario, Canada.
[29]
Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Michael Norrish, Rafal Kolanski, Thomas Sewell, Harvey Tuch, and Simon Winwood. 2009. seL4: Formal Verification of an OS Kernel. In Proceedings of the 22nd ACM Symposium on Operating Systems Principles (SOSP '09). Big Sky, Montana.
[30]
John C. Knight and Nancy G. Leveson. 1986. An experimental evaluation of the assumption of independence in multiversion programming. IEEE Transactions on Software Engineering SE-12, 1 (1986), 96--109. https://doi.org/10.1109/TSE.1986.6312924
[31]
Michalis Kokologiannakis, Ilya Kaysin, Azalea Raad, and Viktor Vafeiadis. 2021. PerSeVerE: Persistency semantics for verification under ext4. Proceedings of the ACM on Programming Languages 5, POPL (2021), 1--29.
[32]
Harendra Kumar, Yuvraj Patel, Ram Kesavan, and Sumith Makam. 2017. High Performance Metadata Integrity Protection in the WAFL Copy-on-Write File System. In Proceedings of the 15th USENIX Conference on File and Storage Technologies (FAST '17). Santa Clara, CA.
[33]
Andrea Lattuada, Travis Hance, Chanhee Cho, Matthias Brun, Isitha Subasinghe, Yi Zhou, Jon Howell, Bryan Parno, and Chris Hawblitzel. 2023. Verus: Verifying rust programs using linear ghost types. Proceedings of the ACM on Programming Languages OOPSLA1 (2023), 286--315.
[34]
Jialin Li, Samantha Miller, Danyang Zhuo, Ang Chen, Jon Howell, and Thomas Anderson. 2021. An incremental path towards a safer OS kernel. In Proceedings of the Workshop on Hot Topics in Operating Systems (HOTOS '21). Ann Arbor, Michigan.
[35]
Linus Torvalds. 2002. [BK PATCH] USB changes for 2.5.34. https://yarchive.net/comp/linux/BUG.html.
[36]
Inc. Linux Kernel Organization. [n.d.]. Linux Page Replacement Policy. https://www.kernel.org/doc/gorman/html/understand/understand013.html.
[37]
Jing Liu, Anthony Rebello, Yifan Dai, Chenhao Ye, Sudarsun Kannan, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2021. Scale and Performance in a Filesystem Semi-Microkernel. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP '21). Virtual Event, Germany.
[38]
Wenqing Liu. 2018. array-index-out-of-bounds in fs/f2fs/segment.c. https://bugzilla.kernel.org/show_bug.cgi?id=215657.
[39]
Yifei Liu, Manish Adkar, Gerard Holzmann, Geoff Kuenning, Pei Liu, Scott A. Smolka, Wei Su, and Erez Zadok. 2024. Metis: File System Model Checking via Versatile Input and State Exploration. In Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST '24). Santa Clara, CA.
[40]
David E Lowell, Subhachandra Chandra, and Peter Chen. 2000. Exploring Failure Transparency and the Limits of Generic Recovery. In Proceedings of the 4th Symposium on Operating Systems Design and Implementation (OSDI '00). San Diego, CA.
[41]
Lanyue Lu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Shan Lu. 2013. A Study of Linux File System Evolution. In Proceedings of the 11th USENIX Symposium on File and Storage Technologies (FAST '13). San Jose, CA.
[42]
Michael Marty, Marc de Kruijf, Jacob Adriaens, Christopher Alfeld, Sean Bauer, Carlo Contavalli, Mike Dalton, Nandita Dukkipati, William C. Evans, Steve Gribble, Nicholas Kidd, Roman Kononov, Gautam Kumar, Carl Mauer, Emily Musick, Lena Olson, Mike Ryan, Erik Rubow, Kevin Springborn, Paul Turner, Valas Valancius, Xi Wang, and Amin Vahdat. 2019. Snap: a Microkernel Approach to Host Networking. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP '19). Ontario, Canada.
[43]
Daejun Park and Dongkun Shin. 2017. iJournaling: Fine-Grained Journaling for Improving the Latency of Fsync System Call. In Proceedings of the USENIX Annual Technical Conference (USENIX '17). Santa Clara, CA.
[44]
Cilium Project (post in Hacker News). 2020. EBPF is turning the Linux kernel into a microkernel. https://news.ycombinator.com/item?id=22953730.
[45]
Feng Qin, Joseph Tucek, Jagadeesan Sundaresan, and Yuanyuan Zhou. 2005. Rx: Treating Bugs As Allergies. In Proceedings of the 20th ACM Symposium on Operating Systems Principles (SOSP '05). Brighton, UK.
[46]
Bianca Schroeder, Eduardo Pinheiro, and Wolf-Dietrich Weber. 2009. DRAM Errors in the Wild: A Large-scale Field Study. In Proceedings of the Eleventh International Joint Conference on Measurement and Modeling of Computer Systems (SIGMETRICS '09). Seattle, WA, USA.
[47]
SPDK Open-source Team. 2021. The Storage Performance Development Kit. https://spdk.io/doc.
[48]
Swaminathan Sundararaman, Sriram Subramanian, Abhishek Rajimwale, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Michael M. Swift. 2010. Membrane: Operating System Support for Restartable File Systems. In Proceedings of the 8th USENIX Symposium on File and Storage Technologies (FAST '10). San Jose, CA.
[49]
Michael M. Swift, Brian N. Bershad, and Henry M. Levy. 2003. Improving the Reliability of Commodity Operating Systems. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP '03). Bolton Landing, New York.
[50]
Michael M. Swift, Brian N. Bershad, and Henry M. Levy. 2004. Recovering Device Drivers. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI '04). San Francisco, CA, 1--16.
[51]
Shaobu Wang, Guangyan Zhang, Junyu Wei, Yang Wang, Jiesheng Wu, and Qingchao Luo. 2023. Understanding Silent Data Corruptions in a Large Production CPU Population. In Proceedings of the 28th ACM Symposium on Operating Systems Principles (SOSP '23). Koblenz, Germany.
[52]
Wen Xu. 2018. use-after-free in ext4_put_super(). https://bugzilla.kernel.org/show_bug.cgi?id=200931.
[53]
Junfeng Yang, Can Sar, and Dawson Engler. 2006. EXPLODE: A Lightweight, General System for Finding Serious Storage System Errors. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI '06). Seattle, WA.
[54]
Mo Zou, Haoran Ding, Dong Du, Ming Fu, Ronghui Gu, and Haibo Chen. 2019. Using concurrent relational logic with helpers for verifying the AtomFS file system. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP '19). Ontario, Canada.

Index Terms

  1. Shadow Filesystems: Recovering from Filesystem Runtime Errors via Robust Alternative Execution

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    HotStorage '24: Proceedings of the 16th ACM Workshop on Hot Topics in Storage and File Systems
    July 2024
    141 pages
    ISBN:9798400706301
    DOI:10.1145/3655038
    This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 July 2024

    Check for updates

    Author Tags

    1. Filesystems
    2. Reliability
    3. Verification

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    • NSF

    Conference

    HOTSTORAGE '24
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 34 of 87 submissions, 39%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 392
      Total Downloads
    • Downloads (Last 12 months)392
    • Downloads (Last 6 weeks)59
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media