Collaborative GPU Preemption via Spatial Multitasking for Efficient GPU Sharing

Abstract

GPUs have been widely used in data centers and are often over-provisioned to satisfy the stringent latency targets of latency-sensitive (LS) jobs. The GPU under-utilization provides a strong incentive to share GPUs among LS jobs and batch jobs. Preemptive GPU prioritization is costly due to the large contexts. Many novel GPU preemption techniques have been proposed, exhibiting different trade-offs between preemption latency and overhead. Prior works also propose collaborative methods, which intelligently select the preemption techniques at preemption time. However, GPU kernels usually adopt code transformation to improve performance, which also impacts the preemption costs. As kernel transformation is performed before launching, the preemption technique choices are also determined then. It is impractical to select a preemption technique arbitrarily at preemption time if code transformation is adopted. This paper presents CPSpatial, which combines GPU preemption techniques via GPU spatial multitasking. CPSpatial proposes preemption hierarchy and SM-prefetching, achieving both low latency and high throughput. Evaluations show that CPSpatial also has zero preemption latency like the traditional instant preemption techniques, and at the time, achieves up to 1.43$\times$ throughput. When dealing with sudden LS job workload increasing, CPSpatial reduces the preemption latency by 87.3% compared with the state-of-the-art GPU context switching method.