Abstract
Accurate uncertainty predictions are crucial to assess the reliability of a model, especially for neural networks. Part of this uncertainty is the observation noise, which is dynamic in our marine virtual sensor task. Typically, dynamic noise is not trained directly, but approximated through terms in the loss function. Unfortunately, this noise loss function needs to be scaled by a trade-off-parameter to achieve accurate uncertainties. In this paper we propose an upgrade to the existing architecture, which increases interpretability and introduces a novel direct training procedure for dynamic noise modelling. To that end, we train the point prediction model and the noise model separately. We present a new loss function that requires Monte Carlo runs of the model to directly train for the uncertainty prediction accuracy. In an experimental evaluation, we show that in most tested cases the uncertainty prediction is more accurate than the manually tuned trade-off-parameter. Because of the architectural changes we are able to analyze the importance of individual parts of the time series of our prediction.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Badewien, T.H., Zimmer, E., Bartholomä, A., Reuter, R.: Towards continuous long-term measurements of suspended particulate matter (SPM) in turbid coastal waters. Ocean Dyn. 59(2), 227–238 (2009)
Balke, T., et al.: Experimental salt marsh islands: a model system for novel metacommunity experiments. Estuar. Coast. Shelf Sci. 198(Part A), 288–298 (2017)
Gal, Y.: Uncertainty in deep learning. Ph.D. thesis, University of Cambridge (2016)
Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: International Conference on Machine Learning (ICML), pp. 1050–1059 (2016)
Gal, Y., Ghahramani, Z.: A theoretically grounded application of dropout in recurrent NNs. In: Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NIPS), pp. 1019–1027 (2016)
Graves, A., Fernández, S., Schmidhuber, J.: Bidirectional LSTM networks for improved phoneme classification and recognition. In: Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S. (eds.) ICANN 2005. LNCS, vol. 3697, pp. 799–804. Springer, Heidelberg (2005). https://doi.org/10.1007/11550907_126
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE (2016)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269. IEEE (2017)
Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 0.5mb model size. CoRR abs/1602.07360 (2016)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning (ICML), pp. 448–456. International Machine Learning Society (2015)
Lin, M., Chen, Q., Yan, S.: Network in network. CoRR abs/1312.4400 (2013)
Murata, S., Masuda, W., Tomioka, S., Ogata, T., Sugano, S.: Mixing actual and predicted sensory states based on uncertainty estimation for flexible and robust robot behavior. In: International Conference on Artificial NNs - (ICANN), pp. 11–18 (2017)
Oehmcke, S., Zielinski, O., Kramer, O.: Input quality aware convolutional LSTM networks for virtual marine sensors. Neurocomputing 275, 2603–2615 (2017)
Oehmcke, S., Zielinski, O., Kramer, O.: Rnns and exponential PAA for virtual marine sensors. In: International Joint Conference on NNs (IJCNN), pp. 4459–4466. IEEE (2017)
Romeu, P., Zamora-Martínez, F., Botella-Rocamora, P., Pardo, J.: Time-series forecasting of indoor temperature using pre-trained deep neural networks. In: Mladenov, V., Koprinkova-Hristova, P., Palm, G., Villa, A.E.P., Appollini, B., Kasabov, N. (eds.) ICANN 2013. LNCS, vol. 8131, pp. 451–458. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40728-4_57
Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: A simple way to prevent nns from overfitting. J. Mach. Learn. Res. (JLMR) 15(1), 1929–1958 (2014)
Zhang, K., Sun, M., Han, T.X., Yuan, X., Guo, L., Liu, T.: Residual networks of residual networks: multilevel residual networks. IEEE Trans. Circuits Syst. Video Technol. 28(6), 1303–1314 (2018). https://doi.org/10.1109/TCSVT.2017.2654543
Acknowledgments
The BEFmate project was funded by the Ministry for Science and Culture of Lower Saxony, Germany under project number ZN2930. Our gratitude goes to the experts of the TSS and the BEFmate project for their support, i.e., Thomas Badewien, Axel Braun, and Daniela Meier.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Oehmcke, S., Zielinski, O., Kramer, O. (2018). Direct Training of Dynamic Observation Noise with UMarineNet. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds) Artificial Neural Networks and Machine Learning – ICANN 2018. ICANN 2018. Lecture Notes in Computer Science(), vol 11139. Springer, Cham. https://doi.org/10.1007/978-3-030-01418-6_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-01418-6_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01417-9
Online ISBN: 978-3-030-01418-6
eBook Packages: Computer ScienceComputer Science (R0)