Abstract
We empirically find the optimal batch size for training the Deep Q-network on the cart-pole system. The efficiency of the training is evaluated by the performance of the network on task after training, and total time and steps required to train. The neural network is trained for 10 different sizes of batch with other hyper parameter values fixed. The network is able to carry out the cart-pole task with the probability of 0.99 or more with the batch sizes from 8 to 2048. The training time per step for training tends to increase linearly, and the total steps for training decreases more than exponentially as the batch size increases. Due to these tendencies, we could empirically observe the quadratic relationship between the total time for training and the logarithm of batch size, which is convex, and the optimal batch size that minimizes training time could also be found. The total steps and time for training are minimum at the batch size 64. This result can be expanded to other learning algorithm or tasks, and further, theoretical analysis on the relationship between the size of batch or other hyper-parameters and the efficiency of training from the optimization point of view.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2012)
Redmon, J., et al.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)
Girshick, R., et al.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)
Bojarski, M., et al.: End to end learning for self-driving cars (2016). arXiv:1604.07316
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning (2015). arXiv:1509.02971
Mnih, V., et al.: Playing Atari with deep reinforcement learning (2013). arXiv:1312.5602
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Sprague, N.: Parameter selection for the deep q-learning algorithm. In: Proceedings of the Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM) (2015)
Bengio, Y.: Practical recommendations for gradient-based training of deep architectures. In: Neural Networks: Tricks of the Trade, pp. 437–478. Springer, Berlin, Heidelberg (2012)
Loshchilov, I., Hutter, F.: Online batch selection for faster training of neural networks (2015). arXiv:1511.06343
Barto, A.G., Sutton, R.S., Anderson, C.W.: Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. 5, 834–846 (1983)
Acknowledgements
This work was supported by the ICT R&D program of MSIP/IITP [2016-0-00563, Research on Adaptive Machine Learning Technology Development for Intelligent Autonomous Digital Companion].
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Choi, M. (2019). An Empirical Study on the Optimal Batch Size for the Deep Q-Network. In: Kim, JH., et al. Robot Intelligence Technology and Applications 5. RiTA 2017. Advances in Intelligent Systems and Computing, vol 751. Springer, Cham. https://doi.org/10.1007/978-3-319-78452-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-78452-6_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-78451-9
Online ISBN: 978-3-319-78452-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)