Exploring AMD GPU scheduling details by experimenting with “worst practices”

Abstract

Graphics processing units (GPUs) have been the target of a significant body of recent real-time research, but research is often hampered by the “black box” nature of GPU hardware and software. Now that one GPU manufacturer, AMD, has embraced an open-source software stack, one may expect an increased amount of real-time research to use AMD GPUs. Reality, however, is more complicated. Without understanding where internal details may differ, researchers have no basis for assuming that observations made using NVIDIA GPUs will continue to hold for AMD GPUs. Additionally, the openness of AMD’s software does not mean that their scheduling behavior is obvious, especially due to sparse, scattered documentation. In this paper, we gather the disparate pieces of documentation into a single coherent source that provides an end-to-end description of how compute work is scheduled on AMD GPUs. In doing so, we start with a concrete demonstration of how incorrect management triggers extreme worst-case behavior in shared AMD GPUs. Subsequently, we explain the internal scheduling rules for AMD GPUs, how they led to the “worst practices,” and how to correctly manage some of the most performance-critical factors in AMD GPU sharing.

Appendices

Appendix 1: Detailed kernel-launch behavior

Section 4.1 attempts to describe ROCm’s kernel-launch behavior in human-readable terms, but researchers attempting to build modifications on top of existing ROCm code may benefit from a more detailed description, including more specific source-code references. This appendix assumes that the reader is already familiar with the content discussed in Sect. 4. While this is not necessary to understand the main body of our paper, we have elected to include it as a reference for researchers hoping to modify or work with AMD’s ROCm software stack.

Figure 10 summarizes the kernel-launch process in greater detail. Each component of the flowchart in Fig. 10 contains four pieces of information: the name of a function in ROCm’s source code, a (brief) comment describing the purpose of the function, the ROCm component in which the function is defined, and the specific source-code file within the component containing the function’s definition.

As shown in Fig. 2, there are several ROCm components, all of which are open source. The source code for each relevant component is available online: HIP,²⁰ ROCclr,²¹ the HSA runtime²² and low-level driver interface (ROCT-Thunk-Interface).²³ This paper was based on ROCm version 4.2, but continues to apply to ROCm 4.3 (current at the time of submission), and has remained relatively stable since ROCm version 3.7.

Figure 10 follows the process outlined in Fig. 3, but with a greater level of detail. As discussed in Sect. 4.1, kernel-launch requests are first enqueued in a userspace C++ HostQueue object, and eventually converted into an AQL packet and inserted into an HSA queue. One new detail shown in Fig. 10 is that the conversion from a HostQueue entry into an AQL packet is carried out by an asynchronous thread, i.e., a thread other than the one that called hipLaunchKernelGGL. While unsurprising, given the asynchronous behavior expected when launching kernels, this may be an important detail when designing real-time systems, as such a thread may block any other thread waiting for kernels to complete.

Curious readers may notice that Fig. 10 does not cover queue creation in detail. Instead, we provide a deeper explanation of queue creation, including the code responsible for assigning queues to GPU hardware, in Appendix 2. Note that the Stream::Create block occurs in both flowcharts, giving an indication of where, if necessary, queue creation will take place in the overall kernel-launch process.

Fig. 10

Overview of ROCm source code involved in HIP kernel launches

Appendix 2: Detailed queue-handling behavior

As with Appendix 1, this appendix is intended for readers already familiar with the material in Sects. 4.1 and 4.2. Figure 11 presents a more detailed view of the ROCm source code required to create queues and assign them to GPU hardware. Once again, this information is intended for researchers hoping to work with AMD’s code, especially those hoping to modify AMD’s GPU queue management at either the user level or within the driver.

As discussed in the main body of the paper, kernel launches on AMD GPUs do not require driver intervention, so Fig. 10, in Appendix 1, did not include any driver code. This is unfortunately not the case with the more complicated logic behind creating queues in the first place, meaning that Fig. 11 also must include portions of AMD’s driver code, located within the Linux kernel. In addition to labeling every individual component in Fig. 11 with their respective ROCm components, driver portions of the flowchart are also enclosed in the dashed rectangle. Unfortunately, the full “File” path for the driver components of the flowchart is too long to cleanly fit in the flowchart, so we instead note here that all of the paths given in the “amdgpu Driver” boxes are located under the drivers/gpu/drm/amd directory in the Linux 5.14 source tree.

When casually observing ROCm’s code, it may initially be difficult to discern where HSA queues are created. The key point is the createVirtualDevice function, called when ROCm starts a new thread to process a HIP stream’s kernel launches. The “virtual device” is, in reality, a C++ interface granting access to a GPU; many virtual devices may be associated with a single underlying GPU.

An interesting characteristic of Fig. 11 is the presence of the function names with the _cpsch postfix in the driver code. The “cpsch” postfix stands for Command Processor Scheduling, referring to the use of HWS (discussed in Sect. 4.2). Alternative versions of these functions (postfixed _nocpsch) can also be found in kernel code, and are used when HWS is disabled. Internally, the driver is able to alternate between different versions of such functions by calling them indirectly, using a list of function pointers.

Similarly to how some functions in Fig. 11 depend on whether or not HWS is enabled, some functions may change depending on GPU architecture. These functions are likewise called indirectly, via lists of function pointers. We only include one such architecture-specific function in Fig. 11 (the init_mqd function), but several more are used for still-lower-level details, such as populating the contents of the runlist packet or the unmap-queues request.

Fig. 11

Overview of ROCm source code involved in creating GPU queues