DEFUSE: An Interface for Fast and Correct User Space File System Access

As explained in Section 2.2, when an LD_PRELOAD library is used, the mapping table between FDs and user space file metadata may be stored in memory. However, while open FDs and the internal state of the FS maintained by the LD_PRELOAD library are copied from parent to child upon a fork, the FS state is destroyed when the child invokes exec if it is in memory.

Illustration. Consider the processes in Figure 3: (a) User process p, in the LD_PRELOAD environment, executes library function wrapped system calls that are intercepted by the LD_PRELOAD library. (b) Both the LD_PRELOAD and FS libraries are stored in shared memory and the internal state of the FS resides within the address space of p. (c) When p performs an open system call, the LD_PRELOAD library makes the corresponding call to the FS library. The FS returns an internal representation of the file as F. Process p expects an FD number to be returned from open, not the internal representation of F. (d) To avoid passing the open call to the kernel (to not increase latency), the LD_PRELOAD library manufactures FD 4, updates its internal file mapping 4 to F, and returns the manufactured FD to p. This manufactured FD is only valid in the context of the LD_PRELOAD library and only if the mapping is in memory. Future system calls using FD 4 will be intercepted by the LD_PRELOAD library, mapped to F and passed to the FS library. (e) Process p creates child process c that inherits, among other things, the open FDs and the memory of the LD_PRELOAD library. After performing an exec, the child’s address space, including the internal state for manufactured FDs, is destroyed by the executable of c. (f) Process c later writes to the inherited FD 4. (g) As the LD_PRELOAD library memory was cleared during exec, the library cannot map FD 4 to F, so it passes this I/O to the kernel. (h) The kernel does not know what FD 4 references, since the LD_PRELOAD library of parent process p manufactured it. Ultimately, the operation fails in the kernel with an invalid FD and is returned to c via the LD_PRELOAD library.

Prominent examples. The FD heritage dilemma occurs commonly in shell file redirection. Consider a shell running under the context of an LD_PRELOAD library, where the user executes a cloud application running in Spark [99] by running spark-submit … > out.txt to capture the output to a file located in a user space FS. The shell opens out.txt, which is intercepted by the LD_PRELOAD library prior to creating the spark-submit process. Due to the FD heritage dilemma, the FD for the output file out.txt inherited by the child is no longer valid after the exec call to invoke spark-submit. The FD heritage dilemma is also common in widely used tools such as gcc that create child processes for performing subtasks, and in resource manager frameworks such as the PySpark workload daemon [76], which creates child processes for worker tasks at runtime, with each child task sending and receiving data to and from the daemon through FDs.

Impact in other interfaces. The user space FS library interface suffers from a related problem if a parent process needs to pass file metadata to a child. The application must be aware of the FS library API and does not access the FS using FDs, it instead uses the user space file metadata directly. As with open FDs, child processes cannot inherit file metadata.

FUSE is not affected by the FD heritage dilemma, since the FUSE server is a single process, separated from the application processes. Hence the FUSE identifier table is unique. When a parent opens a file, the corresponding (inheritable) FD matches a unique inode within kernel space.

Similarly to FUSE, in-kernel FSs are not affected by FD heritage dilemma, since all file structures are managed within the kernel address space. References to files are synchronized by the FS driver within the kernel to ensure consistency when accessed by applications via FDs. The FDs maintained by the kernel and inherited by child processes map to a unique inode within kernel space.

Mapping files into memory using mmap is a common method to share memory between processes. The LD_PRELOAD interface cannot provide consistent memory-mapping of files hosted by user space FSs for the two following reasons.

First, while an LD_PRELOAD library can intercept explicitly invoked system calls such as mmap, it cannot intercept implicit I/O operations. For instance, accesses to a memory-mapped file such as reads and writes are performed using load and store paging instructions to the address region of the mapped memory. These instructions, which may result in an I/O operation to the underlying file, are not explicit system calls and therefore cannot be intercepted.

Second, as a direct consequence of the FD heritage dilemma, a child process cannot access a user space file mapped in memory if the mapping has been done by its parent process. For example, consider the following set of actions taken by a process: (a) A process opens a user space file, the open call is intercepted and redirected to the user space FS library. (b) The process then maps this file into its address space using mmap for shared access; however, since the FD is manufactured by the intercepting FS library, the call to mmap fails, because the FD is invalid.

Impact in other interfaces. Directly linked user space FS libraries equally suffer from the same problem for mmap-ed files as LD_PRELOAD libraries do. FUSE does not, however, since the FUSE kernel driver translates paging operations to read or write sent to the user space FUSE server.

First we describe the setup required for our experiments.

Hardware. Benchmarks were performed on a single machine running Ubuntu 18.04.1 LTS with kernel version 4.15.0-43, two Intel Xeon E5-2420 processors with 2.2 GHz, 24 GB of main memory with SATA-attached 256 GB SSD and 1 TB HDD running at 7,200 RPM. With the exception of tmpfs, which uses system memory, the backing storage for all evaluated FSs use the attached HDD.

Caching policy. FS caches were cleared between each I/O operation of the benchmarks to remove any effect of caching. Files were flushed to disk (fsync) after each write operation.

FSs. We used ext4 [27], JFS [42], FAT [55], and tmpfs [90]. Both ext4 and JFS were chosen due to their large install base in consumer Linux as well as industry servers. While FAT is no longer a commonplace FS for PCs or industry servers, it was chosen for evaluation due to its prevalence in external media. Finally, tmpfs was chosen for its low I/O overhead, since it resides in memory, allowing us to isolate the FS logic overhead. Note that for FSs built on actual media, the time spent for I/O operations on the media itself may overshadow the time performing FS logic.

Baselines. We compared DEFUSE: to (1) an LD_PRELOAD library, (b) Direct-FUSE [100], (3) FUSE (v2.9.7) with and without direct I/O enabled, and (4) an FS mounted using a kernel driver.

The LD_PRELOAD library maintains its own FD table (cf. Section 2.2) that is thus invalid when inherited. Direct-FUSE is a user space FS interface built as an extension to the libsysio library [89]. Like DEFUSE:, Direct-FUSE intercepts I/O system calls using hooks in libsysio. However, Direct-FUSE has similar drawbacks as the LD_PRELOAD approach, i.e., it does not address the FD heritage dilemma nor user space paging. LD_PRELOAD and Direct-FUSE are inconsistent solutions, they are included for comparison only. The user space FS library for DEFUSE: and FUSE is a simple wrapper over POSIX system calls, as expressed in Section 5.2. Using direct I/O with FUSE makes I/O operations skip kernel FS caches.

Benchmarking tool. We used the widely adopted IOzone [39] FS benchmarking tool—a user space application that creates, writes, and reads files of varying sizes using POSIX system calls. IOzone generated three workloads described further alongside the corresponding results in Section 6.1.2.

I/O throughputs were measured in terms of data read/written per second. The depicted results were normalized as a ratio of the direct mount to simplify comparisons (otherwise the greatly varying access performance for different FSs would render some figures’ y axis hardly readable).

We evaluated file access throughput separately for small and large file as well as for a mixture of files with different sizes.

Small files. We evaluated the FSs for small file accesses where a large quantity of system calls must be serviced. IOzone created 4,096 files of 128 B in size. This workload is prevalent in many parallel computing applications [17, 21] where large quantities (sometimes in the millions) of files smaller than 64 kB are used to store checkpoint data. Due to the large quantity of system calls needed to create, write, and read that many files, we expected FUSE’s performance to be greatly affected as a large quantity of system calls requires an equally large quantity of waits.

Overall, as Figures 7(a) and 7(b) show, DEFUSE: always achieved within 10% of the best-performer LD_PRELOAD while the inconsistent Direct-FUSE was within 1%. DEFUSE: reached at best equivalent throughput to direct mount (write with FAT) yet at worse 50% (read with tmpfs). The lower performance is primarily caused by the additional time spent resolving FS metadata from the FD mapping table; however, it is only minor when compared to the overhead of FUSE. Direct I/O for FUSE has negligible effect on performance as the benefits are lost due to the small buffer sizes and the large number of system calls to be serviced (cf. discussion on Figure 8).

The direct mount actual throughput varied across FSs: (a) ext4 had 2.6 MB/s write and 6.8 MB/s read throughputs, (b) JFS was slower with 1.1 MB/s write and 4.8 MB/s read throughputs, (c) FAT was also slower than ext4 with 130 kB/s write and 6 MB/s read throughputs, (d) tmpfs I/O operations being completely in memory, achieved the best throughput with 20.4 MB/s for both write and read.

However, given the large number of system calls needed to process all the files, achievable throughput is far below the capability of the storage media.

Large files. We evaluated the FSs for large file accesses where the fewest quantity of system calls were needed. A single 128-MB file was created. This benchmark highlights the amortization of wait context switch cost for FUSE’s system calls.

The results in Figures 7(c) and 7(d) show that DEFUSE: reached close to optimal throughputs: between 85% and 93% that of direct mount for reads, and between 92% and 100% for writes. LD_PRELOAD’s throughput was almost always above 98% (except for reads on FAT) and, as with small files, Direct-FUSE closely followed. While FUSE performance improved with large files, it remained largely below that of the other interfaces and at best only reached 60% for tmpfs. Write performance; however, almost reached that of direct mount when used with direct I/O for FAT. Since only one file is accessed, the number of system calls serviced by the kernel and FUSE server is small and the total operation time is more dependent on the I/O time to the storage media.

The actual I/O throughput for direct mount is very similar for most FSs, achieving \(\approx\)200 MB/s write and \(\approx\)900 MB/s read throughput. The exception is tmpfs, which achieved 2.3-GB/s write and read throughput due to I/O operations being completely in memory.

Mixed sized files. We evaluated the FSs using mixed sized files to simulate a more diverse workload. The benchmark created 2,048 files linearly distributed in sizes from 64 B to 128 kB. Figures 7(e) and 7(f) show close results in all cases for all interfaces except FUSE that performs significantly worse. As with previous benchmarks, direct I/O slightly improves FUSE but is still 15% slower than DEFUSE: at best (writes to JFS).

Our evaluation with small, large, and mixed file workloads show that the throughput of DEFUSE: was significantly higher than FUSE and was often close to that of LD_PRELOAD. We then evaluated the total number of waits needed to perform each of these benchmarks to confirm that the reduced throughput is directly related to context switching and not to any other bottleneck in FUSE. We thus used the Linux time command that has the ability to retrieve statistics including the number of context switches performed for both time slice expiration and the purpose of waiting for I/O operations to complete (using time %w [52]). Here, we are only concerned about the subset of context switches made by the process for the latter case, i.e., waits. We ran the benchmarks on tmpfs to focus on the context switches caused by the FS interface, since I/O requests to access a physical device induce extra context switches.

The results shown in Figure 8 (y axes use a log scale) confirm that the overhead of FUSE is caused in a large part by waits. Direct mount, the LD_PRELOAD library, Direct-FUSE, and DEFUSE: require \(\approx\)9 waits for all workloads while FUSE requires orders of magnitude more. For the small file workload (cf. Figure 8(a)), FUSE requires \(\approx\)16,000 waits (i.e., permission lookup + open + read/write + close for each of the 4,096 files) and using direct I/O does not reduce this number. For the large file workload (cf. Figure 8(b)), we note that the FUSE kernel driver splits large writes into multiple requests sent to the FUSE server in user space, each causing a wait. Even with a single (large) file, FUSE requires \(\approx\)1,000 waits for a read and \(\approx\)33,000 for a write. Direct I/O reduces these considerably, from \(\approx\)33,000 down to \(\approx\)1,000 for writes, which is still 100× higher than DEFUSE:. The mixed file workload (cf. Figure 8(c)) shows an even larger difference in the required waits: approximately 9 for DEFUSE: but \(\approx\)41,000 for FUSE. While there are half as many files created in this workload compared to the small file one (2,048 vs. 4,096), files are larger on average thus forcing the FUSE server to split write requests into smaller ones. As with the large file workload, the number of waits is reduced using direct I/O, down to \(\approx\)10,000, which is still 1,000× more than DEFUSE:.

In addition to synthetic workloads generated by IOzone, we also evaluated DEFUSE:’s impact on three practical (throughput-intensive) applications using the sources of the Linux kernel v4.15 as dataset. First, we decompressed the 160-MB archive comprised of 67,000 files into an ext4 backed FS using the tar command.

We then backed up the decompressed small files from the ext4 backed FS to a JFS backed FS using rsync. We finally compiled the source files using gcc on the ext4 backed FS.

Figure 9(a) shows the decompression time, Figure 9(b) the backup time, and Figure 9(c) the kernel compilation time. For decompression, DEFUSE: achieved almost identical runtime to the direct mount while FUSE required as much as 4× the amount of time to finish. Using direct I/O only marginally improved FUSE performance. Similarly, DEFUSE: ran at near direct kernel mount speeds when backing up, while FUSE and FUSE with direct I/O were significantly slower, taking 3× longer. For compilation, where DEFUSE: and FUSE perform the closest, DEFUSE: took 1.23× longer than direct mount and FUSE yet 1.41× longer than DEFUSE:, on average. This evaluation showed the practical advantages of DEFUSE: over FUSE as a user space FS interface for everyday tasks.

To complete our single-machine evaluation, we ran benchmarks using various system calls that read, write, and/or modify the metadata of 10,000 files. Unlike the previous benchmarks, these ones are not throughput-intensive but do perform a lot of small I/O operations.

Figure 10(a) shows the time to create 10,000 files, Figure 10(b) shows the time to run the access system call on all 10,000 files, Figure 10(c) the time to change the mode of all 10,000 files using the chmod system call, and Figure 10(d) the time to remove (unlink) the 10,000 files. Due to the relatively short amount of time to perform the metadata operation compared to the overhead of the FS access method, FUSE performs significantly worse than direct mount and DEFUSE:. Even at its best, for unlink, FUSE requires 4.3× longer than direct mount compared to DEFUSE:, which takes 1.3× longer than direct mount; while at its worst FUSE requires 12× longer than direct mount for the file access benchmark where DEFUSE: requires 1.6× longer.