Abstract
Modern technical products have access to a huge amount of data and by utilizing machine learning algorithms this data can be used to improve usability and performance of the products. However, the data is likely to be large in quantity and privacy sensitive, which excludes the possibility of sending and storing all the data centrally. This in turn makes it difficult to train global machine learning models on the combined data of different devices. A decentralized approach known as federated learning solves this problem by letting devices, or clients, update a global model using their own data and only sending changes of the global model, which means that they do not need to communicate privacy sensitive data.
Deep neural decision forests (DNDF), inspired by the versatile algorithm random forests, combine the divide-and-conquer principle together with the property representation learning. In this paper we further develop the concept of DNDF to be more suited for the framework of federated learning. By parameterizing the probability distributions in the prediction nodes of the forest, and include all trees of the forest in the loss function, a gradient of the whole forest can be computed which some/several federated learning algorithms utilize. We demonstrate the inclusion of DNDF in federated learning by an empirical experiment with both homogeneous and heterogeneous data and baseline it against a convolutional neural network with the same architecture as the DNDF. Experimental results show that the modified DNDF, consisting of three to five decision trees, outperform the baseline convolutional neural network.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bojarski, M., et al.: End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 (2016)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Caruana, R., Karampatziakis, N., Yessenalina, A.: An empirical evaluation of supervised learning in high dimensions. In: Proceedings of the 25th International Conference on Machine Learning, pp. 96–103. ACM (2008)
Criminisi, A., Shotton, J.: Decision Forests for Computer Vision and Medical Image Analysis. Springer, London (2013). https://doi.org/10.1007/978-1-4471-4929-3
Díaz-Uriarte, R., De Andres, S.A.: Gene selection and classification of microarray data using random forest. BMC Bioinform. 7(1), 3 (2006)
Diederik, P., Kingma, J.B.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D.: Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res. 15(1), 3133–3181 (2014)
Harari, G.M., Lane, N.D., Wang, R., Crosier, B.S., Campbell, A.T., Gosling, S.D.: Using smartphones to collect behavioral data in psychological science: opportunities, practical considerations, and challenges. Perspect. Psychol. Sci. 11(6), 838–854 (2016)
Konečnỳ, J., McMahan, H.B., Ramage, D., Richtárik, P.: Federated optimization: distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527 (2016)
Konečnỳ, J., McMahan, H.B., Yu, F.X., Richtárik, P., Suresh, A.T., Bacon, D.: Federated learning: strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016)
Kontschieder, P., Fiterau, M., Criminisi, A., Rota Bulo, S.: Deep neural decision forests. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1467–1475 (2015)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
McMahan, B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282 (2017)
Nilsson, A., Smith, S., Ulm, G., Gustavsson, E., Jirstrand, M.: A performance evaluation of federated learning algorithms. In: Proceedings of the Second Workshop on Distributed Infrastructures for Deep Learning (DIDL 2018), New York, NY, USA, vol. 18, pp. 1–8 (2018)
Poushter, J., et al.: Smartphone ownership and internet usage continues to climb in emerging economies. Pew Res. Cent. 22, 1–44 (2016)
Shotton, J., et al.: Real-time human pose recognition in parts from single depth images. In: CVPR, vol. 2, p. 3 (2011)
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, vol. 2, p. 745. Springer, New York (2009). https://doi.org/10.1007/978-0-387-84858-7
Ulm, G., Gustavsson, E., Jirstrand, M.: Functional federated learning in erlang (ffl-erl). In: Silva, J. (ed.) WFLP 2018. LNCS, vol. 11285, pp. 162–178. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-16202-3_10
Acknowledgements
This research was supported by the project Onboard/Offboard Distributed Data Analytics (OODIDA) in the funding program FFI: Strategic Vehicle Research and Innovation (DNR 2016-04260), which is administered by VINNOVA, the Swedish Government Agency for Innovation Systems.
This work was developed in Fraunhofer Cluster of Excellence Cognitive Internet Technologies.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Sjöberg, A., Gustavsson, E., Koppisetty, A.C., Jirstrand, M. (2019). Federated Learning of Deep Neural Decision Forests. In: Nicosia, G., Pardalos, P., Umeton, R., Giuffrida, G., Sciacca, V. (eds) Machine Learning, Optimization, and Data Science. LOD 2019. Lecture Notes in Computer Science(), vol 11943. Springer, Cham. https://doi.org/10.1007/978-3-030-37599-7_58
Download citation
DOI: https://doi.org/10.1007/978-3-030-37599-7_58
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37598-0
Online ISBN: 978-3-030-37599-7
eBook Packages: Computer ScienceComputer Science (R0)