Abstract
Industry 5.0 focuses on collaboration between humans and machines, demanding robustness and efficiency and the accuracy of intelligent and innovative components used. The use of sensors and the fusion of data obtained from various sensors/modes increased with the rise of the internet of things. The multimodal sensor fusion provides better accuracies than a single source/mode system. Typically, in multimodal fusion, data from all sources is assumed to be available all the time, aligned, and noiseless. Contrary to this highly unrealistic assumption, data from one or more sources is generally unavailable or noisy in most real-world applications. Hence, there is a need for robust sensor fusion, which is a more realistic scenario and critical for Industry 5.0 implementations. Multimodal co-learning is one such approach to study the robustness of sensor fusion for missing and noisy modalities. In this work, to demonstrate the effectiveness of multimodal co-learning for robustness study, gas detection systems are considered a case study. Such gas detection systems are prevalent and crucial to avoid accidents due to gas leaks in many industries and form a part of the industry 5.0 setup. This work considers the primary dataset of gas sensors data and thermal images for robustness experimentation. The results demonstrate that multi-task fusion is robust to missing and noisy modalities than intermediate fusion. Having an additional low-resolution thermal modality supports co-learning and makes it robust to 20% missing sensor data, 90% missing thermal image data, and Gaussian and Normal noise. The end-to-end system architecture proposed can be easily extended to other multimodal applications in Industry 5.0. This study is a step towards creating standard practices for a multimodal co-learning charter for various industrial applications.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and material
The multimodal gas detection dataset presented in (Narkhede et al. 2021) is used for this study, which was made available upon request for non-commercial use. The readers can contact the corresponding authors of this research paper.
Code availability
Code can be made available upon request post-publication of the paper for non-commercial use.
References
Arslan M, Guzel M, Demirci M, Ozdemir S (2019) SMOTE and Gaussian Noise Based Sensor Data Augmentation. In: 2019 4th International Conference on Computer Science and Engineering (UBMK). IEEE, pp 1–5
Avila LF (2005) Leak detection with thermal imaging
Baltrusaitis T, Ahuja C, Morency LP (2019) Multimodal machine learning: a survey and taxonomy. IEEE Trans Pattern Anal Mach Intell 41:423–443. https://doi.org/10.1109/TPAMI.2018.2798607
Bijelic M, Muench C, Ritter W, et al (2018) Robustness against unknown noise for raw data fusing neural networks. IEEE Conf Intell Transp Syst Proceedings, ITSC 2018-Novem:2177–2184. https://doi.org/10.1109/ITSC.2018.8569911
Boonprong S, Cao C, Chen W et al (2018) The classification of noise-afflicted remotely sensed data using three machine-learning techniques: Effect of different levels and types of noise on accuracy. ISPRS Int J Geo-Inform. https://doi.org/10.3390/ijgi7070274
Caruana R (1997) Multitask Learning. Mach Learn 28:41–75
Chen X, Ma H, Wan J et al. (2017) Multi-view 3D object detection network for autonomous driving. Proc - 30th IEEE Conf Comput Vis Pattern Recognition, CVPR 2017 2017-Janua:6526–6534. https://doi.org/10.1109/CVPR.2017.691
Choi JH, Lee JS (2019) EmbraceNet: A robust deep learning architecture for multimodal classification. Inf Fusion 51:259–270. https://doi.org/10.1016/j.inffus.2019.02.010
Cotta J, Breque M, Nul L De, Petridis A (2021) Industry 5.0 Towards a sustainable. Humancentric and resilient European industry
Demetgul M (2013) Fault diagnosis on production systems with support vector machine and decision trees algorithms. Int J Adv Manuf Technol 67:2183–2194. https://doi.org/10.1007/s00170-012-4639-5
Diez-Olivan A, Del Ser J, Galar D, Sierra B (2019) Data fusion and machine learning for industrial prognosis: Trends and perspectives towards Industry 4.0. Inf Fusion 50:92–111. https://doi.org/10.1016/j.inffus.2018.10.005
Dong Y, Gao S, Tao K et al (2014) Performance evaluation of early and late fusion methods for generic semantics indexing. Pattern Anal Appl 17:37–50. https://doi.org/10.1007/s10044-013-0336-8
Doyle-Kent M, Kopacek P (2020) Industry 5.0: Is the manufacturing industry on the cusp of a new revolution? Proceed Int Symp Prod Res 2019:432–441
Elmenreich W (2007) A review on system architectures for sensor fusion applications. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 4761 LNCS:547–559. https://doi.org/10.1007/978-3-540-75664-4_57
Fortin MP, Chaib-Draa B (2019) Multimodal sentiment analysis: A multitask learning approach. ICPRAM 2019 - Proc 8th Int Conf Pattern Recognit Appl Methods 368–376. https://doi.org/10.5220/0007313503680376
Garcia NC, Morerio P, Murino V (2018) Modality distillation with multiple stream networks for action recognition. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). pp 106–121
Glodek M, Tschechne S, Layher G, et al. (2011) Multiple classifier systems for the classification of audio-visual emotional states. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). pp 359–368
Gomri S, Contaret T, Seguin JL (2018) A new gases identifying method with MOX gas sensors using noise spectroscopy. IEEE Sens J 18:6489–6496. https://doi.org/10.1109/JSEN.2018.2850817
Guo W, Wang J, Wang S (2019) Deep multimodal representation learning: a survey. IEEE Access 7:63373–63394. https://doi.org/10.1109/ACCESS.2019.2916887
Han L, Yu C, Xiao K, Zhao X (2019) A new method of mixed gas identification based on a convolutional neural network for time series classification. Sensors (swiTzerland). https://doi.org/10.3390/s19091960
Hanga KM, Kovalchuk Y (2019) Machine learning and multi-agent systems in oil and gas industry applications: A survey. Comput Sci Rev 34:100191. https://doi.org/10.1016/j.cosrev.2019.08.002
Havens KJ, Sharp EJ (2015) Thermal imaging techniques to survey and monitor animals in the wild: a methodology. Academic Press, Cambridge, MA, USA
Hendrycks D, Dietterich T (2019) Benchmarking neural network robustness to common corruptions and perturbations. arXiv
Jadin MS, Ghazali KH (2014) Gas leakage detection using thermal imaging technique. Proc - UKSim-AMSS 16th Int Conf Comput Model Simulation Uksim 2014:302–306. https://doi.org/10.1109/UKSim.2014.95
Javaid M, Haleem A (2020) Critical components of industry 5.0 towards a successful adoption in the field of manufacturing. J Ind Integr Manag 05:327–348. https://doi.org/10.1142/S2424862220500141
Khalaf WM (2012) Electronic nose system for safety monitoring at refineries. J Eng Dev Vol. 16, N:220–228
Kim T, Ghosh J (2016) Robust detection of non-motorized road users using deep learning on optical and LIDAR data. In: 2016 IEEE 19th international conference on intelligent transportation systems (ITSC). IEEE, pp 271–276
Kim T, Ghosh J (2019) On single source robustness in deep fusion models. Adv Neural Inf Process Syst 32:
Kim J, Koh J, Kim Y, et al. (2019) Robust deep multi-modal learning based on gated information fusion network. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 11364 LNCS:90–106. https://doi.org/10.1007/978-3-030-20870-7_6
Kiros R, Popuri K, Cobzas D, Jagersand M (2014) Stacked Multiscale Feature Learning for Domain Independent Medical Image Segmentation. In: In International Workshop on Machine Learning in Medical Imaging. pp 25–32
Kittler J, Hatef M, Duin RPW, Matas J (1998) On combining classifiers. IEEE Trans Pattern Anal Mach Intell 20:226–239. https://doi.org/10.1109/34.667881
Lahat D, Adali T, Jutten C (2015) Multimodal data fusion: an overview of methods, challenges, and prospects. Proc IEEE 103:1449–1477. https://doi.org/10.1109/JPROC.2015.2460697
Li JB, Ma K, Qu S, et al (2020) Audio-visual event recognition through the lens of adversary. arXiv 2–6
Liang Y, He TP (2020) Survey on soft computing. Soft Comput 24:761–770. https://doi.org/10.1007/s00500-019-04508-z
Lichtenwalter D, Burggräf P, Wagner J, Weißer T (2021) Deep multimodal learning for manufacturing problem solving. Procedia CIRP 99:615–620. https://doi.org/10.1016/j.procir.2021.03.083
Liu Q, Hu X, Ye M et al (2015a) Gas recognition under sensor drift by using deep learning. Int J Intell Syst 30:907–922. https://doi.org/10.1002/int.21731
Liu S, Liu S, Cai W et al (2015b) Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer’s disease. IEEE Trans Biomed Eng 62:1132–1140. https://doi.org/10.1109/TBME.2014.2372011
Liu K, Li Y, Xu N, Natarajan P (2018) Learn to combine modalities in multimodal deep learning. arXiv:1805.11730
Lu Y, Da XuL (2019) Internet of Things (IoT) cybersecurity research: a review of current research topics. IEEE Internet Things J 6:2103–2115. https://doi.org/10.1109/JIOT.2018.2869847
Luo Y, Ye W, Zhao X et al (2017) Classification of data from electronic nose using gradient tree boosting algorithm. Sensors (switzerland) 17:1–10. https://doi.org/10.3390/s17102376
Marathe S (2019) Leveraging Drone Based Imaging Technology for Pipeline and RoU Monitoring Survey. In: Day 2 Wed, April 24, 2019. SPE
MDC Systems Inc. Detection Methods. In: MDC Syst. Inc. https://mdcsystemsinc.com/detection-methods/. Accessed 8 Aug 2021
Narkhede P, Walambe R, Mandaokar S et al (2021) Gas detection and identification using multimodal artificial intelligence based sensor fusion. Appl Syst Innov 4:1–14. https://doi.org/10.3390/asi4010003
Ngiam J, Khosla A, Kim M, et al. (2011) Multimodal deep learning. Proc 28th Int Conf Mach Learn ICML 2011 689–696
Pan X, Zhang H, Ye W et al (2019) A fast and robust gas recognition algorithm based on hybrid convolutional and recurrent neural network. IEEE Access 7:100954–100963. https://doi.org/10.1109/ACCESS.2019.2930804
Pashami S, Lilienthal AJ, Trincavelli M (2012) Detecting changes of a distant gas source with an array of MOX gas sensors. Sensors (switzerland) 12:16404–16419. https://doi.org/10.3390/s121216404
Peng P, Zhao X, Pan X, Ye W (2018) Gas classification using deep convolutional neural networks. Sensors (switzerland) 18:1–11. https://doi.org/10.3390/s18010157
Petrović VS, Xydeas CS (2003) Sensor noise effects on signal-level image fusion performance. Inf Fusion 4:167–183. https://doi.org/10.1016/S1566-2535(03)00035-6
Pham H, Liang PP, Manzini T, et al. (2018) Found in translation: Learning robust joint representations by cyclic translations between modalities. arXiv
Poria S, Cambria E, Bajpai R, Hussain A (2017) A review of affective computing: From unimodal analysis to multimodal fusion. Inf Fusion 37:98–125. https://doi.org/10.1016/j.inffus.2017.02.003
Qin J, Liu Y, Grosvenor R (2016) A categorical framework of manufacturing for industry 4.0 and beyond. Procedia CIRP 52:173–178. https://doi.org/10.1016/j.procir.2016.08.005
Rahate A, Walambe R, Ramanna S, Kotecha K (2022) Multimodal co-learning: challenges, applications with datasets, recent advances and future directions. Inf Fusion 81:203–239. https://doi.org/10.1016/j.inffus.2021.12.003
Ramachandram D, Taylor GW (2017) Deep multimodal learning - A survey on recent advances and trends. IEEE Signal Process Mag 96–108
Rusak E, Schott L, Zimmermann RS, et al. (2020) A simple way to make neural networks robust against diverse image corruptions. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 12348 LNCS:53–69. https://doi.org/10.1007/978-3-030-58580-8_4
Sayyad S, Kumar S, Bongale A et al (2021) Data-driven remaining useful life estimation for milling process: sensors, algorithms, datasets, and future directions. IEEE Access 9:110255–110286. https://doi.org/10.1109/ACCESS.2021.3101284
Seo S, Na S, Kim J (2020) HMTL: Heterogeneous modality transfer learning for audio-visual sentiment analysis. IEEE Access 8:140426–140437. https://doi.org/10.1109/ACCESS.2020.3006563
Sharp M, Ak R, Hedberg T (2018) A survey of the advancing use and development of machine learning in smart manufacturing. J Manuf Syst 48:170–179. https://doi.org/10.1016/j.jmsy.2018.02.004
Stauffer E, Dolan JA, Newman R (2008) Gas Chromatography and Gas Chromatography—Mass Spectrometry. In: Fire Debris Analysis. Elsevier, pp 235–293
Steffens CR, Messias LRV, Drews-Jr PJL, da Botelho SSC (2021) On robustness of robotic and autonomous systems perception. J Intell Robot Syst 101:61. https://doi.org/10.1007/s10846-021-01334-0
Trivedi et al. (2014) Major industrial disasters in India. 9:7
Tsao Y, Lai YH, Wang HM, et al. (2017) Audio-visual speech enhancement based on multimodal deep convolutional neural network. arXiv
Walambe RA, Joshi VA, Apte AA, et al. (2015) Study of sensorless control algorithms for a permanent magnet synchronous motor vector control drive. In: 2015 International Conference on Industrial Instrumentation and Control (ICIC). IEEE, pp 423–428
Woodward JR, Gindy N (2010) A hyper-heuristic multi-criteria decision support system for eco-efficient product life cycle. In: 5th International Conference on Responsive Manufacturing - Green Manufacturing (ICRM 2010). IET, pp 201–205
Wuest T, Weimer D, Irgens C, Thoben K-D (2016) Machine learning in manufacturing: advantages, challenges, and applications. Prod Manuf Res 4:23–45. https://doi.org/10.1080/21693277.2016.1192517
Yin X, Zhang L, Tian F, Zhang D (2016) Temperature modulated gas sensing e-nose system for low-cost and fast detection. IEEE Sens J 16:464–474. https://doi.org/10.1109/JSEN.2015.2483901
Yunusa Z (2014) Gas sensors: a review. Sens Transducers 168(4):61–75
Zadeh A, Liang PP, Morency LP (2020) Foundations of Multimodal Co-learning: Multimodal Co-learning. Inf Fusion 64:188–193. https://doi.org/10.1016/j.inffus.2020.06.001
Zhang J, Yin Z, Chen P, Nichele S (2020) Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review. Inf Fusion 59:103–126. https://doi.org/10.1016/j.inffus.2020.01.011
Zhang R, Candra SA, Vetter K, Zakhor A (2015) Sensor fusion for semantic segmentation of urban scenes. In: 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp 1850–1857
Zheng Z, Ma A, Zhang L, Zhong Y (2021) Deep multisensor learning for missing-modality all-weather mapping. ISPRS J Photogramm Remote Sens 174:254–264. https://doi.org/10.1016/j.isprsjprs.2020.12.009
Zhou Y, Zhao X, Zhao J, Chen D (2016) Research on fire and explosion accidents of oil depots. Chem Eng Trans 51:163–168. https://doi.org/10.3303/CET1651028
Acknowledgments
This research uses a primary gas detection dataset created as a part of a minor research project funded by Symbiosis International (Deemed University), Pune 412115, India.
Funding
Not applicable.
Author information
Authors and Affiliations
Contributions
Conceptualization: Anil Rahate, Rahee Walambe; Methodology: Anil Rahate, Ketan Kotecha, Rahee Walambe; Software, Data Curation, Experimentation, Validation: Anil Rahate, Shruti Mandaokar, Pulkit Chandel; Writing- original draft & editing, Visualization, Investigation: Anil Rahate; Writing—review & editing: Rahee Walambe, Sheela Ramanna, Ketan Kotecha; Project Administration, Supervision, Approval: Ketan Kotecha, Rahee Walambe.
Corresponding authors
Ethics declarations
Conflict of interest
The authors have no relevant financial or non-financial interests to declare relevant to this article’s content.
Ethical approval
Not applicable.
Consent to participate
Not applicable
Consent for publication
The authors provide consent for publication in this journal
Additional information
Communicated by Deepak kumar Jain.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Rahate, A., Mandaokar, S., Chandel, P. et al. Employing multimodal co-learning to evaluate the robustness of sensor fusion for industry 5.0 tasks. Soft Comput 27, 4139–4155 (2023). https://doi.org/10.1007/s00500-022-06802-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-022-06802-9