research-article

HeatWave: thermal imaging for surface user interaction

Authors:

Beverly Harrison,

Shwetak PatelAuthors Info & Claims

CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 2565 - 2574

https://doi.org/10.1145/1978942.1979317

Published: 07 May 2011 Publication History

Abstract

We present HeatWave, a system that uses digital thermal imaging cameras to detect, track, and support user interaction on arbitrary surfaces. Thermal sensing has had limited examination in the HCI research community and is generally under-explored outside of law enforcement and energy auditing applications. We examine the role of thermal imaging as a new sensing solution for enhancing user surface interaction. In particular, we demonstrate how thermal imaging in combination with existing computer vision techniques can make segmentation and detection of routine interaction techniques possible in real-time, and can be used to complement or simplify algorithms for traditional RGB and depth cameras. Example interactions include (1) distinguishing hovering above a surface from touch events, (2) shape-based gestures similar to ink strokes, (3) pressure based gestures, and (4) multi-finger gestures. We close by discussing the practicality of thermal sensing for naturalistic user interaction and opportunities for future work.

References

[1]

Arai, T., Machii, K. and Kuzunuki, S. Retrieving electronic documents with real-world objects on InteractiveDESK. In Proc. of UIST '95. New York: ACM Press, 1995, pp. 37--38.

Digital Library

[2]

Barras, C. Microsoft's body-sensing, button-busting controller. New Scientist: Technology. 7 Jan 2010.

[3]

Bradski, G.; Kaehler, A. (2008), Learning OpenCV: Computer Vision with the OpenCV Library.

Digital Library

[4]

Cohen, C. J., Beach, G. J. and Foulk, G. A. Basic hand gesture control system for PC applications. In Proc. of AIPR '01. IEEE Computer Society, 2001, pp. 74.

Digital Library

[5]

Dietz, P. and Leigh, D. DiamondTouch: A multi-user \ touch technology. In Proc. of UIST '01. New York: ACM Press, 2001, pp. 219--226.

Digital Library

[6]

Duda, R. and Hart, P "Use of the Hough Transforma-tion to Detect Lines and Curves in Pictures," Comm. ACM, Vol. 15, pp. 11--15, Jan., 1972.

Digital Library

[7]

Felzenszwalb, P., McAllester, D., and Ramanan, D. A Discriminatively Trained, Multiscale, Deformable Part Model. In Proc. of IEEE CVPR '08, 2008.

[8]

Hilliges, O., Izadi, S., Wilson, A. D., Hodges, S., Garcia-Mendoza, A., Butz, A. Interactions in the air: adding further depth to interactive tabletops. In Proc. of UIST '09. New York: ACM Press, 2009, pp. 139--148.

Digital Library

[9]

Iwai, D. and Sato, K. "Heat Sensation in Image Creation with Thermal Vision", ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE2005), pp.213--216, Jun. 2005.

Digital Library

[10]

Iwai, D. and Sato K, "Limpid Desk: theoretical papers of transparent projection of Mixed Reality", IPSJ, vol.48, no.3, pp.1294--1306, 2007.

[11]

1Izadi, S., Hodges, S. Taylor, S., Rosenfeld, D., Villar, N., Butler, A., Westhues, J. Going beyond the display: a surface technology with an electronically switchable diffuser. In Proc. of UIST '09. New York: ACM Press, 2009, pp. 269--278.

Digital Library

[12]

Kane, S., Avrahami, D., Wobbrock, J. O., Harrison, B., Rea, A. D., Philipose, M., LaMarca, A. Bonfire: a nomadic systems for hybrid laptop-tabletop interaction. In Proc. of UIST '09. New York: ACM Press, 2009.

Digital Library

[13]

Khan, M. M., Ingleby, M., and Ward, R. D. Automated Facial Expression Classification and affect interpretation using infrared measurement of facial skin temperature variations. ACM Trans. on Autonomous and Adaptive Systems, 2006.

Digital Library

[14]

Khan, M. M., Ward, R. D., and Ingleby, M. Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature. ACM Trans. on Applied Perception, 2009.

Digital Library

[15]

Kittel, C. and Kroemer, H. Thermal Physics. 2nd ed. New York: W. H. Freeman, 1980.

[16]

Kong, S. G., Heo, J., Boughorbel, F., Zheng, Y., Abidi, B. R., Koschan, A., Yi, M., and Abidi, M. A. Multiscale Fusion of Visible and Thermal IR Images for Illumination-Invariant Face Recognition. International Journal of Computer Vision, 2007.

Digital Library

[17]

Kurtenbach, G. The Design and Evaluation of Marking Menus, Ph.D. Thesis, Department of Computer Science, University of Toronto. May 1993.

Digital Library

[18]

Lee, J., Hudson, S., Summet, J., and Dietz, P. Moveable interactive projected displays using projector based tracking. In Proc. of UIST '05. New York: ACM Press, 2005, pp. 63--72.

Digital Library

[19]

Letessier, J. and B&#233;rard, F. Visual tracking of bare fingers for interactive surfaces. In Proc. of UIST '04. New York: ACM Press, 2004, pp. 119--122.

Digital Library

[20]

Manresa, C., Varona, J., Mas, R. and Perales, F. Hand tracking and gesture recognition for humancomputer interaction. Electronic Letters on Computer Vision and Image Analysis, 5 (3), 2005, pp. 96--104.

[21]

Marshall, J., Pridmore, T., Pound, M., Benford, S., Koleva, B. Pressing the Flesh: Sincing Multiple Touch and Finger Pressure on Arbitrary Surfaces. In Proc. of Pervasive '08. Berlin: Springer-Verlag, 2008, pp. 38--55.

Digital Library

[22]

Microsoft Surface. http://www.microsoft.com/surface.

[23]

Oka, K., Sato, Y. and Koike, H. "Real-time tracking of multiple fingertips and gesture recognition for augmented desk interface systems." IEEE Computer Graphics and Applications. Vol. 22, No. 6, pp. 64--71, 2002.

Digital Library

[24]

Otsu, N, "A threshold selection method from gray-level histograms". IEEE Trans. Systems, Man, and Cybernet-ics. 9: 62--66, 1972 .

[25]

Pavlidis, I., Dowdall, J., Sun, N., Puri, C., Fei, J., and Garbey, M. Interacting with human physiology. Computer Vision and Image Understanding, 2007.

Digital Library

[26]

Puri, C., Olson, L., Pavlidis, I., Levine, J., and Starren, J. StressCam: non-contact measurement of users' emotional states through thermal imaging. In Proc. of CHI '05, 2005.

Digital Library

[27]

Quinlan, J. R. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, 1993.

Digital Library

[28]

RazIR. NANO Thermal Camera, http://raz-ir.com/raz-ir-nano-thermal-infrared-camera.html.

[29]

Rekimoto, J. SmartSkin: an infrastructure for freehand manipulation on interactive surfaces. In Proc.of CHI '02. New York: ACM Press, 2002, pp. 113--120.

Digital Library

[30]

Saletan, W. Heat Check: Swine flu, body heat, and airport scanners. http://www.slate.com/id/2217148/, 28 April 2009.

[31]

Sato, Y., Kobayashi, Y., and Koike, H. "Fast tracking of hands and fingertips in infrared images for augmented desk interface," Proc. 2000 IEEE International Conference on Automatic Face and Gesture Recognition (FGR 2000), pp. 462--467, March 2000.

Digital Library

[32]

Tanriverdi, V. and Jacob R. J. K. Interacting with eye movements in virtual environments. In Proc. of CHI '00. New York: ACM Press, 2000, pp. 265--272.

Digital Library

[33]

Wellner, P. Interacting with paper on the DigitalDesk. Communications of the ACM, 36 (7), 1993, pp. 87--96.

Digital Library

[34]

Wilson, A. D. PlayAnywhere: a compact interactive tabletop projection-vision system. In Proc. of UIST '05. New York: ACM Press, 2005, pp. 83--92.

Digital Library

[35]

Witten, I. and Frank, E.: "Data Mining: Practical machine learning tools and techniques", 2nd Edition, Morgan Kaufmann, San Francisco (2005).

Digital Library

[36]

Wong, W. K., Tan, P. N., Loo, C. K., and Lim, W. S. An Effective Surveillance System Using Thermal Camera. In Proc. of International Conference on Signal Acquisition and Processing, 2009.

Digital Library

[37]

Wu, M. and Balakrishnan, R. Multi-finger and whole hand gestural interaction techniques for multiuser tabletop displays. In Proc. of UIST '03. New York: ACM Press, 2003, pp. 193--202.

Digital Library

[38]

Yun, C., Shastri, D., Pavlidis, I., and Deng, Z. O' game, can you feel my frustration?: improving user's gaming experience via stresscam. In Proc. of CHI '09, 2009.

Digital Library

Cited By

Dupré CAppert CRey SSaidi HPietriga E(2024)TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking OnlyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642323(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642323
Pokharel ALuitel NKhatri AKhadka SShrestha R(2024)Review on the evolving role of infrared thermography in oncological applicationsInfrared Physics & Technology10.1016/j.infrared.2024.105399140(105399)Online publication date: Aug-2024
https://doi.org/10.1016/j.infrared.2024.105399
Lee KPark J(2023)Contactless Interface Using Exhaled Breath and Thermal ImagingSensors10.3390/s2307360123:7(3601)Online publication date: 30-Mar-2023
https://doi.org/10.3390/s23073601
Show More Cited By

Index Terms

HeatWave: thermal imaging for surface user interaction
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
2. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

Investigation of Material Properties for Thermal Imaging-Based Interaction
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Recent work demonstrated the exciting opportunities that thermal imaging offers for the development of interactive systems. It was shown that a thermal camera can sense when a user touches a surface, performs gestures in the camera's direct field of ...
Visual touchpad: a two-handed gestural input device
ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces

This paper presents the Visual Touchpad, a low-cost vision-based input device that allows for fluid two-handed interactions with desktop PCs, laptops, public kiosks, or large wall displays. Two downward-pointing cameras are attached above a planar ...
Thermal cameras and applications: a survey

Thermal cameras are passive sensors that capture the infrared radiation emitted by all objects with a temperature above absolute zero. This type of camera was originally developed as a surveillance and night vision tool for the military, but recently ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

May 2011

3530 pages

ISBN:9781450302289

DOI:10.1145/1978942

General Chair:
Desney Tan
Microsoft Research
,
Program Chairs:
Geraldine Fitzpatrick
Vienna University of Technology
,
Carl Gutwin
University of Saskatchewan
,
Bo Begole
PARC
,
Wendy A. Kellogg
IBM Research

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '11

Sponsor:

SIGCHI

CHI '11: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2011

BC, Vancouver, Canada

Acceptance Rates

CHI '11 Paper Acceptance Rate 410 of 1,532 submissions, 27%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
1,094
Total Downloads

Downloads (Last 12 months)57
Downloads (Last 6 weeks)11

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Dupré CAppert CRey SSaidi HPietriga E(2024)TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking OnlyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642323(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642323
Pokharel ALuitel NKhatri AKhadka SShrestha R(2024)Review on the evolving role of infrared thermography in oncological applicationsInfrared Physics & Technology10.1016/j.infrared.2024.105399140(105399)Online publication date: Aug-2024
https://doi.org/10.1016/j.infrared.2024.105399
Lee KPark J(2023)Contactless Interface Using Exhaled Breath and Thermal ImagingSensors10.3390/s2307360123:7(3601)Online publication date: 30-Mar-2023
https://doi.org/10.3390/s23073601
Liang CWang XLi ZHsia CFan MYu CShi Y(2023)ShadowTouch: Enabling Free-Form Touch-Based Hand-to-Surface Interaction with Wrist-Mounted Illuminant by Shadow ProjectionProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606785(1-14)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606785
Alharbi RShahi SCruz SLi LSen SPedram MRomano CHester JKatsaggelos AAlshurafa N(2023)SmokeMonProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694606:4(1-25)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3569460
Perovich LRogowitz BCrabb VVogelsang JHartleben SOffenhuber D(2023)The tactile dimension: a method for physicalizing touch behaviorsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581137(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581137
Iravantchi YZhao YKin KSample A(2023)SAWSense: Using Surface Acoustic Waves for Surface-bound Event RecognitionProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580991(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580991
Saad AIzadi KAhmad Khan AKnierim PSchneegass SAlt FAbdelrahman Y(2023)HotFoot: Foot-Based User Identification Using Thermal ImagingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580924(1-13)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580924
Faltaous SWittpoth MAbdelrahman YSchneegass S(2022)HeatGoggles: Enabling Ubiquitous Touch Input through Head-Mounted Devices using Thermal ImagingProceedings of the 21st International Conference on Mobile and Ubiquitous Multimedia10.1145/3568444.3570597(254-256)Online publication date: 27-Nov-2022
https://dl.acm.org/doi/10.1145/3568444.3570597
Dar FEmenike HYin ZLiyanage MSharma RZuniga AHoque MRadeta MNurmi PFlores H(2022)The MIDAS touch: Thermal dissipation resulting from everyday interactions as a sensing modalityPervasive and Mobile Computing10.1016/j.pmcj.2022.10162584(101625)Online publication date: Aug-2022
https://doi.org/10.1016/j.pmcj.2022.101625
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten