Abstract
In our previous papers [12, 13], we proposed the parallel realization of the Deep Belief Network (DBN). This research confirmed the potential usefulness of the first generation of the Intel MIC architecture for implementing DBN and similar algorithms. In this work, we investigate how the Intel MIC and CPU platforms can be applied to implement efficiently the complete learning process using DBNs with layers corresponding to the Restricted Boltzman Machines. The focus is on the new generation of Intel MIC devices known as Knights Landing. Unlike the previous generation, called Knights Corner, they are delivered not as coprocessors, but as standalone processors.
The learning procedure is based on the matrix approach, where learning samples are grouped into packages, and represented as matrices. We study the possible ways of improving the performance taking into account features of the Knights Landing architecture, and parameters of the learning algorithm. In particular, the influence of the package size on the accuracy of learning, as well as on the performance of computations are investigated using conventional CPU and Intel Xeon Phi. The performance advantages of Knights Landing over Knights Corner are presented and discussed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dongarra, J., Tomov, S., Luszczek, P., Kurzak, J., Gates, M., Yamazaki, I., Anzt, H., Haidar, A., Abdelfattah, A.: With extreme computing, the rules have changed. Comput. Sci. Eng. 19(3), 52–62 (2017)
Fang, J., Varbanescu, A.L., Sips, H.: Benchmarking Intel Xeon Phi to guide kernel design. Delft University of Technology Parallel and Distributed Systems Report Series. No. PDS-2013-005, pp. 1–22 (2013)
Hinton, G., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)
Intel Math Kernel Library: Reference Manual. Intel Corporation, Santa Clara (2009)
Jeffers, J., Reinders, J.: Intel Xeon Phi Coprocessor High Performance Programming, 1st edn. Morgan Kaufmann Publishers Inc., San Francisco (2013)
Jeffers, J., Reinders, J., Sodani, A.: Intel Xeon Phi Processor High Performance Programming: Knights Landing Edition. Elsevier Science, Cambridge (2016)
Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 664–676 (2017)
Karpathy, A., Joulin, A., Li, F.F.F.: Deep fragment embeddings for bidirectional image sentence mapping. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, pp. 1889–1897 (2014)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, Department of Computer Science, University of Toronto (2009)
Levesque, J., Vose, A.: Programming for Hybrid Multi/Manycore MPP Systems, 1st edn. Taylor & Francis Ltd, London (2017)
Lastovetsky, A., Szustak, L., Wyrzykowski, R.: Model-based optimization of EULAG kernel on Intel Xeon Phi through load imbalancing. IEEE Trans. Parallel Distrib. Syst. 28(3), 787–797 (2017). https://doi.org/10.1109/TPDS.2016.2599527
Olas, T., Mleczko, W.K., Nowicki, R.K., Wyrzykowski, R.: Adaptation of deep belief networks to modern multicore architectures. In: Wyrzykowski, R., Deelman, E., Dongarra, J., Karczewski, K., Kitowski, J., Wiatr, K. (eds.) PPAM 2015. LNCS, vol. 9573, pp. 459–472. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-32149-3_43
Olas, T., Mleczko, W.K., Nowicki, R.K., Wyrzykowski, R., Krzyzak, A.: Adaptation of RBM learning for Intel MIC architecture. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2015. LNCS (LNAI), vol. 9119, pp. 90–101. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19324-3_9
Raina, R., Madhavan, A., Ng, A.Y.: Large-scale deep unsupervised learning using graphics processors. In: Proceeding of 26th Annual International Conference on Machine Learning, ICML 2009, pp. 873–880. ACM (2009). https://doi.org/10.1145/1553374.1553486
Reinders, J.: An overview of programming for Intel Xeon processors and Intel Xeon Phi coprocessors. Technical report, Intel Corporation (2012)
Sarikaya, R., Hinton, G.E., Deoras, A.: Application of deep belief networks for natural language understanding. IEEE/ACM Trans. Audio Speech Lang. Process. 22(4), 778–784 (2014). https://doi.org/10.1109/TASLP.2014.2303296
Saule, E., Kaya, K., Çatalyürek, Ü.V.: Performance evaluation of sparse matrix multiplication kernels on Intel Xeon Phi. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Waśniewski, J. (eds.) PPAM 2013. LNCS, vol. 8384, pp. 559–570. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-55224-3_52
Song, K., Liu, Y., Wang, R., Zhao, M., Hao, Z., Qian, D.: Restricted Boltzmann machines and deep belief networks on sunway cluster. In: 2016 IEEE 18th International Conference on High Performance Computing and Communications, pp. 245–252 (2016). https://doi.org/10.1109/HPCC-SmartCity-DSS.2016.0044
Szustak, L., Rojek, K., Gepner, P.: Using Intel Xeon Phi coprocessor to accelerate computations in MPDATA algorithm. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Waśniewski, J. (eds.) PPAM 2013. LNCS, vol. 8384, pp. 582–592. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-55224-3_54
Szustak, L., Rojek, K., Olas, T., Kuczynski, L., Halbiniak, K., Gepner, P.: Adaptation of MPDATA heterogeneous stencil computation to Intel Xeon Phi coprocessor. Sci. Program. 2015, 14 (2015). https://doi.org/10.1155/2015/642705
Ueyoshi, K., Marukame, T., Asai, T., Motomura, M., Schmid, A.: FPGA implementation of a scalable and highly parallel architecture for restricted Boltzmann machines, 07, 2132–2141 (2016)
Wolf, M.E., Lam, M.S.: A data locality optimizing algorithm. SIGPLAN Not. 26(6), 30–44 (1991). https://doi.org/10.1145/113446.113449
Wolfe, M.: More iteration space tiling. In: Proceedings of the 1989 ACM/IEEE Conference on Supercomputing, Supercomputing 1989, pp. 655–664. ACM, New York (1989). https://doi.org/10.1145/76263.76337
Zhang, X.L., Wu, J.: Deep belief networks based voice activity detection. IEEE Trans. Audio Speech Lang. Process. 21(4), 697–710 (2013). https://doi.org/10.1109/TASL.2012.2229986
Acknowledgements
This project was supported by the Polish Ministry of Science and Education under Grant No. BS/PB-1-112-3030/17/P. The authors are grateful to the Czestochowa University of Technology for granting access to Intel CPU and Xeon Phi platforms provided by the MICLAB project No. POIG.02.03.00.24-093/13.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Olas, T., Mleczko, W.K., Wozniak, M., Nowicki, R.K., Gepner, P. (2018). Performance Evaluation of DBN Learning on Intel Multi- and Manycore Architectures. In: Wyrzykowski, R., Dongarra, J., Deelman, E., Karczewski, K. (eds) Parallel Processing and Applied Mathematics. PPAM 2017. Lecture Notes in Computer Science(), vol 10777. Springer, Cham. https://doi.org/10.1007/978-3-319-78024-5_49
Download citation
DOI: https://doi.org/10.1007/978-3-319-78024-5_49
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-78023-8
Online ISBN: 978-3-319-78024-5
eBook Packages: Computer ScienceComputer Science (R0)