Abstract
The two largest barriers to adoption of FPGA platforms for HPC applications are the difficulty of programming FPGAs and the performance gap when compared to GPUs. To address the first barrier, new ecosystems like Intel oneAPI, and Xilinx Vitis HLS aim to improve programmability for FPGA platforms. From a performance aspect, FPGAs trade off lower compute frequencies for more customized hardware acceleration and power efficiency when compared to GPUs. The performance for memory-bound applications on recent GPU platforms like NVIDIA’s H100 and AMD’s MI210 has also improved due to the inclusion of high-bandwidth memories (HBM), and newer FPGA platforms are also starting to include HBM in addition to traditional DRAM.
To understand the current state-of-the-art and performance differences between FPGAs and GPUs, we consider realized memory bandwidth for recent FPGA and GPU platforms. We utilize a custom STREAM benchmark to evaluate two Intel FPGA platforms, the Stratix 10 SX PAC and Bittware 520N-MX, two AMD/Xilinx FPGA platforms, the Alveo U250 and Alveo U280, as well as GPU platforms from NVIDIA and AMD. We also extract power measurements and estimate memory bandwidth per Watt ((GB/s)/W) on these platforms to evaluate how FPGAs compare against GPU execution. While the GPUs far exceed the FPGAs in raw performance, the HBM equipped FPGAs demonstrate a competitive performance-power balance for larger data sizes that can be easily implemented with oneAPI and Vitis HLS kernels. These findings suggest a potential sweet spot for this emerging FPGA ecosystem to serve bandwidth limited applications in an energy-efficient fashion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alveo U250 data center accelerator card. https://www.xilinx.com/products/boards-and-kits/alveo/u250.html. Accessed 13 June 2023
Alveo U280 data center accelerator card. https://www.xilinx.com/products/boards-and-kits/alveo/u280.html. Accessed 13 June 2023
AMD Instinct MI210 accelerator. https://www.amd.com/en/products/server-accelerators/amd-instinct-mi210. Accessed 15 Mar 2023
Bittware 520N-MX PCIE card with Intel Stratix 10 MX FPGA. https://www.bittware.com/fpga/520n-mx/. Accessed 15 Mar 2023
NVIDIA A100 tensor core GPU: Unprecedented acceleration at every scale. https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/a100-80gb-datasheet-update-nvidia-us-1521051-r2-web.pdf. Accessed 15 Mar 2023
NVIDIA H100 tensor core GPU. https://www.nvidia.com/en-us/data-center/h100/. Accessed 15 Mar 2023
Open programmable acceleration engine. https://opae.github.io/latest/index.html. Accessed 15 Mar 2023
Bittware 520N-MX gen 3x16 BSP release notes (2021). release 1.8.1
Alt, C., et al.: Shallow water DG simulations on FPGAs: design and comparison of a novel code generation pipeline. In: Bhatele, A., Hammond, J., Baboulin, M., Kruse, C. (eds.) ISC High Performance 2023. LNCS, vol. 13948, pp. 86–105. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-32041-5_5
Calore, E., Schifano, S.F.: Performance assessment of FPGAs as HPC accelerators using the FPGA empirical roofline. In: 2021 31st International Conference on Field-Programmable Logic and Applications (FPL), pp. 83–90 (2021). https://doi.org/10.1109/FPL53798.2021.00022
Deakin, T., Price, J., Martineau, M., McIntosh-Smith, S.: Evaluating attainable memory bandwidth of parallel programming models via BabelStream. Int. J. Comput. Sci. Eng. 17(3), 247–262 (2018). https://doi.org/10.1504/IJCSE.2017.10011352
Huang, H., et al.: Shuhai: a tool for benchmarking high bandwidth memory on FPGAs. IEEE Trans. Comput. 71(5), 1133–1144 (2022). https://doi.org/10.1109/TC.2021.3075765
Kara, K., Hagleitner, C., Diamantopoulos, D., Syrivelis, D., Alonso, G.: High bandwidth memory on FPGAs: a data analytics perspective. In: 2020 30th International Conference on Field-Programmable Logic and Applications (FPL), pp. 1–8 (2020). https://doi.org/10.1109/FPL50879.2020.00013
McCalpin, J.D.: Memory bandwidth and machine balance in current high performance computers. IEEE Comput. Soc. Tech. Committee Comput. Architect. (TCCA) Newsl. 2, 19–25 (1995)
Meyer, M., Kenter, T., Plessl, C.: Evaluating FPGA accelerator performance with a parameterized OpenCL adaptation of selected benchmarks of the HPC Challenge benchmark suite. In: 2020 IEEE/ACM International Workshop on Heterogeneous High-Performance Reconfigurable Computing (H2RC), pp. 10–18 (2020). https://doi.org/10.1109/H2RC51942.2020.00007
Nguyen, T., Williams, S., Siracusa, M., MacLean, C., Doerfler, D., Wright, N.J.: The performance and energy efficiency potential of FPGAs in scientific computing. In: 2020 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pp. 8–19 (2020). https://doi.org/10.1109/PMBS51919.2020.00007
Parravicini, A., Cellamare, L.G., Siracusa, M., Santambrogio, M.D.: Scaling up HBM efficiency of Top-K SpMV for approximate embedding similarity on FPGAs. In: 2021 58th ACM/IEEE Design Automation Conference (DAC), pp. 799–804 (2021). https://doi.org/10.1109/DAC18074.2021.9586203
Siefert, C., Olivier, S., Voskuilen, G., Young, J.: MultiGrid on FPGA using data parallel C++. In: 23rd IEEE International Workshop on Parallel and Distributed Scientific and Engineering Computing (2022). https://doi.org/10.1109/IPDPSW55747.2022.00147
Siracusa, M., et al.: A comprehensive methodology to optimize FPGA designs via the roofline model. IEEE Trans. Comput. 71(8), 1903–1915 (2022). https://doi.org/10.1109/TC.2021.3111761
Zeni, A., O’Brien, K., Blott, M., Santambrogio, M.D.: Optimized implementation of the HPCG benchmark on reconfigurable hardware. In: Sousa, L., Roma, N., Tomás, P. (eds.) Euro-Par 2021. LNCS, vol. 12820, pp. 616–630. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85665-6_38
Acknowledgment
Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA-0003525. This written work is authored by an employee of NTESS. The employee, not NTESS, owns the right, title and interest in and to the written work and is responsible for its contents. Any subjective views or opinions that might be expressed in the written work do not necessarily represent the views of the U.S. Government. The publisher acknowledges that the U.S. Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this written work or allow others to do so, for U.S. Government purposes. The DOE will provide public access to results of federally sponsored research in accordance with the DOE Public Access Plan.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Siefert, C.M., Olivier, S.L., Voskuilen, G.R., Young, J.S. (2023). Observed Memory Bandwidth and Power Usage on FPGA Platforms with OneAPI and Vitis HLS: A Comparison with GPUs. In: Bienz, A., Weiland, M., Baboulin, M., Kruse, C. (eds) High Performance Computing. ISC High Performance 2023. Lecture Notes in Computer Science, vol 13999. Springer, Cham. https://doi.org/10.1007/978-3-031-40843-4_46
Download citation
DOI: https://doi.org/10.1007/978-3-031-40843-4_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-40842-7
Online ISBN: 978-3-031-40843-4
eBook Packages: Computer ScienceComputer Science (R0)