Abstract
We introduce a MATLAB-Simulink software able to generate customizable hardware IP cores for the Reinforcement Learning algorithm called Q-Learning. The tool automatically produces the VHDL code and runs both synthesis and implementation for any AMD-Xilinx FPGA using the Vivado software chain. Our automatic generator relies on the “HDL coder” from Mathworks to produce an efficient hardware accelerator based on the state of the art. The model can be customized by the user according to the desired Q-Matrix size and bit-depth for all the algorithm parameters.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. A Bradford Book, Cambridge (2018)
Rothmann, M., Porrmann, M.: A survey of domain-specific architectures for reinforcement learning. IEEE Access 10, 13753–13767 (2022)
Waseem, S.M., Roy, S.K.: Hardware realization of reinforcement learning algorithms for edge devices. In: VLSI and Hardware Implementations Using Modern Machine Learning Methods, pp. 233–254. CRC Press (2021)
Su, J.D., Tsai, P.Y.: Processing element architecture design for deep reinforcement learning with flexible block floating point exploiting signal statistics. In: 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 82–87. IEEE (2020)
Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3), 279–292 (1992)
Cardarilli, G.C., et al.: An FPGA-based multi-agent reinforcement learning timing synchronizer. Comput. Electr. Eng. 99, 107749 (2022)
Liu, X., Diao, J., Li, N.: A FPGA-based accelerator implementation for path planning using q_learning algorithm. J. Phys. Conf. Ser. 2245, 012014 (2022). IOP Publishing
Cardarilli, G.C., et al.: “MR Q-Learning” algorithm for efficient hardware implementations. In: 2021 55th Asilomar Conference on Signals, Systems, and Computers, pp. 1186–1190. IEEE (2021)
Sahoo, S.S., Baranwal, A.R., Ullah, S., Kumar, A.: MemOReL: a memory-oriented optimization approach to reinforcement learning on FPGA-based embedded systems. In: Proceedings of the 2021 on Great Lakes Symposium on VLSI, pp. 339–346 (2021)
Baranwal, A.R., Ullah, S., Sahoo, S.S., Kumar, A.: ReLAccS: a multilevel approach to accelerator design for reinforcement learning on FPGA-based systems. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 40(9), 1754–1767 (2020)
Meng, Y., Kuppannagari, S., Rajat, R., Srivastava, A., Kannan, R., Prasanna, V.: QTAccel: a generic FPGA based design for Q-table based reinforcement learning accelerators. In: 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 107–114. IEEE (2020)
Spanó, S., et al.: An efficient hardware implementation of reinforcement learning: the Q-learning algorithm. IEEE Access 7, 186340–186351 (2019)
Da Silva, L.M., Torquato, M.F., Fernandes, M.A.: Parallel implementation of reinforcement learning Q-learning technique for FPGA. IEEE Access 7, 2782–2798 (2018)
Sun, J., Sharma, N., Chakareski, J., Mastronarde, N., Lao, Y.: Hardware acceleration for post-decision state reinforcement learning in IoT systems. IEEE Internet Things J. 9(12), 9889–9903 (2022)
Cardarilli, G.C., et al.: An action-selection policy generator for reinforcement learning hardware accelerators. In: Saponara, S., De Gloria, A. (eds.) ApplePies 2020. LNEE, vol. 738, pp. 267–272. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-66729-0_32
Acknowledgments
The authors would like to thank Advanced Micro Devices, Inc. (AMD) for providing the FPGA software tools with the AMD-Xilinx University Program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Canese, L., Cardarilli, G.C., Di Nunzio, L., Fazzolari, R., Re, M., Spanó, S. (2023). Automatic IP Core Generator for FPGA-Based Q-Learning Hardware Accelerators. In: Berta, R., De Gloria, A. (eds) Applications in Electronics Pervading Industry, Environment and Society. ApplePies 2022. Lecture Notes in Electrical Engineering, vol 1036. Springer, Cham. https://doi.org/10.1007/978-3-031-30333-3_32
Download citation
DOI: https://doi.org/10.1007/978-3-031-30333-3_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30332-6
Online ISBN: 978-3-031-30333-3
eBook Packages: EngineeringEngineering (R0)