Abstract
Natural surfaces offer the opportunity to provide augmented reality interactions in everyday environments without the use of cumbersome body-mounted equipment. One of the key techniques of detecting user interactions with natural surfaces is the use of thermal imaging that captures the transmitted body heat onto the surface. A major challenge of these systems is detecting user swipe pressure on different material surfaces with high accuracy. This is because the amount of transferred heat from the user body to a natural surface depends on the thermal property of the material. If the surface material type is known, these systems can use a material-specific pressure classifier to improve the detection accuracy. In this work, we address to solve this problem as we propose a novel approach that can detect material type from a user’s thermal finger impression on a surface. Our technique requires the user to touch a surface with a finger for 2 s. The recorded heat dissipation time series of the thermal finger impression is then analyzed in a classification framework for material identification. We studied the interaction of 15 users on 7 different material types, and our algorithm is able to achieve 74.65% material classification accuracy on the test data in a user-independent manner.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Kurz, D.: Thermal touch: thermography-enabled everywhere touch interfaces for mobile augmented reality applications. In: 2014 IEEE International Symposium on Mixed and Augmented Reality, pp. 9–16. IEEE, Munich (2014). https://doi.org/10.1109/ISMAR.2014.6948403
Wilson, A.D.: Playanywhere: a compact interactive tabletop projection-vision system. In: Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, pp. 83–92. ACM, New York (2005)
Mistry, P., Maes, P.: Sixthsense: a wearable gestural interface. In: ACM SIGGRAPH ASIA 2009 Sketches, pp. 11. ACM, New York (2009). https://doi.org/10.1145/1667146.1667160
Oka, K., Sato, Y., Koike, H.: Real-time tracking of multiple fingertips and gesture recognition for augmented desk interface systems. In: Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 429–434. IEEE Computer Society, Washington, DC (2002). http://dl.acm.org/citation.cfm?id=874061.875437
Iwai, D., Sato, K.: Heat sensation in image creation with thermal vision. In: Proceedings of the 2005 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, pp. 213–216. ACM, New York (2005). https://doi.org/10.1145/1178477.1178510
Saba, E.N., Larson, E.C., Patel, S.N.: Dante vision: in-air and touch gesture sensing for natural surface interaction with combined depth and thermal cameras. In: 2012 IEEE International Conference on Emerging Signal Processing Applications. pp. 167–170, IEEE, Las Vegas (2012). https://doi.org/10.1109/ESPA.2012.6152472
Larson, E., et al.: HeatWave: thermal imaging for surface user interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2565–2574. ACM, New York (2011). https://doi.org/10.1145/1978942.1979317
Dunn, T., Banerjee, S., Banerjee, N.K.: User-independent detection of swipe pressure using a thermal camera for natural surface interaction. In: 2018 IEEE 20th International Workshop on Multimedia Signal Processing, pp. 1–6. IEEE, Vancouver (2018)
Gundupalli, S.P., Hait, S., Thakur, A.: Multi-material classification of dry recyclables from municipal solid waste based on thermal imaging. J. Waste Manag. 70, 13–21 (2017)
Cho, Y., Bianchi-Berthouze, N., Marquardt, N., Julier, S.J.: Deep thermal imaging: proximate material type recognition in the wild through deep learning of spatial surface temperature patterns. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–13. ACM, Montreal (2018). https://doi.org/10.1145/3173574.3173576
Aujeszky, T., Korres, G., Eid, M.: Thermography-based material classification using machine learning. In: 2017 IEEE International Symposium on Haptic, Audio and Visual Environments and Games, pp. 1–6. IEEE, Abu Dhabi (2017). https://doi.org/10.1109/HAVE.2017.8240344
Bai, H., Bhattacharjee, T., Chen, H., Kapusta, A., Kemp, C.: Towards Material Classification of Scenes Using Active Thermography. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4262–4269, IEEE, Madrid (2018). https://doi.org/10.1109/IROS.2018.8594469
Frank, E., Hall, M.A., Witten, I.H.: The WEKA Workbench. Online Appendix for “Data Mining: Practical Machine Learning Tools and Techniques”. Morgan Kaufmann, Burlington (2016)
Acknowledgements
This work was partially supported by the National Science Foundation (NSF) grant #1730183.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Gately, J., Liang, Y., Wright, M.K., Banerjee, N.K., Banerjee, S., Dey, S. (2020). Automatic Material Classification Using Thermal Finger Impression. In: Ro, Y., et al. MultiMedia Modeling. MMM 2020. Lecture Notes in Computer Science(), vol 11961. Springer, Cham. https://doi.org/10.1007/978-3-030-37731-1_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-37731-1_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37730-4
Online ISBN: 978-3-030-37731-1
eBook Packages: Computer ScienceComputer Science (R0)