research-article

Privacy-preserving Image Processing with Binocular Thermal Cameras

Authors:

Erin Griffiths,

Kamin WhitehouseAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 1, Issue 4

Article No.: 133, Pages 1 - 25

https://doi.org/10.1145/3161198

Published: 08 January 2018 Publication History

Abstract

Today, cameras and digital image processing are transforming industries and the human environment with rich, informative sensing. However, image processing is not utilized nearly as much in homes where concerns about image privacy dominate. In a preliminary study with 200 participants, we found 21% would reject a camera based system even if the system was designed to not report images as they could still be collected if the camera system was hacked. In this paper, we demonstrate a hardware-based approach for privacy-preserving image processing: the ability to automatically extract information from imaging sensors without the risk of compromising image privacy, even if the system is hacked. The basic idea is to limit both the memory available on board the camera and the data rate of camera communication to prevent a full image from ever being extracted. As a proof of concept, we prototype a system, called Lethe, that tracks and identifies individuals by height with a thermal camera as they move from room to room. Our results show that Lethe can detect the presence of individuals with 96.9% accuracy and determine their direction of travel with 99.7% accuracy. Additionally, Lethe can identify individuals 96.0% of the time with a 5cm (~2in) or greater difference in walking height and 92.9% with a 2.5cm (~1in) or greater difference. Finally, Lethe performs this processing with only 33 bytes of memory (or 0.69% of the full thermal image).

Supplementary Material

griffiths (griffiths.zip)

Supplemental movie, appendix, image and software files for, Privacy-preserving Image Processing with Binocular Thermal Cameras

Download
3.69 MB

References

[1]

M.D. Addlesee, A. Jones, F. Livesey, and F. Samaria. 1997. The ORL active floor [sensor system]. Personal Communications, IEEE 4, 5 (1997), 35--41.

[2]

Derek Anderson, James M Keller, Marjorie Skubic, Xi Chen, and Zhihai He. 2006. Recognizing falls from silhouettes. In Engineering in Medicine and Biology Society, 2006. EMBS‘06. 28th Annual International Conference of the IEEE. IEEE, 6388--6391.

[3]

Jörg Appenrodt, Ayoub Al-Hamadi, and Bernd Michaelis. 2010. Data gathering for gesture recognition systems based on single color-, stereo color-and thermal cameras. International Journal of Signal Processing, Image Processing and Pattern Recognition 3, 1 (2010), 37--50.

[4]

Alex Beltran, Varick L Erickson, and Alberto E Cerpa. 2013. Thermosense: Occupancy thermal based sensing for hvac control. In Proceedings of the 5th ACM Workshop on Embedded Systems For Energy-Efficient Buildings. ACM, 1--8.

Digital Library

[5]

Horesh Ben Shitrit, Jérôme Berclaz, François Fleuret, and Pascal Fua. 2014. Multi-commodity network flow for tracking multiple people. Pattern Analysis and Machine Intelligence, IEEE Transactions on 36, 8 (2014), 1614--1627.

Digital Library

[6]

Chiraz BenAbdelkader, Ross Cutler, and Larry Davis. 2002. Person identification using automatic height and stride estimation. In Pattern Recognition, 2002. Proceedings. 16th International Conference on, Vol. 4. IEEE, 377--380.

[7]

Anjan Chakravorty, Tomasz Wlodarczyk, and Chunming Rong. 2013. Privacy Preserving Data Analytics for Smart Homes. In Security and Privacy Workshops (SPW), 2013 IEEE. IEEE, 23--27.

Digital Library

[8]

Eun Kyoung Choe, Sunny Consolvo, Jaeyeon Jung, Beverly Harrison, and Julie A Kientz. 2011. Living in a glass house: a survey of private moments in the home. In Proceedings of the 13th international conference on Ubiquitous computing. ACM, 41--44.

Digital Library

[9]

Tim Coughlan, Kerstin Leder Mackley, Michael Brown, Sarah Martindale, Stephan Schlögl, Becky Mallaband, John Arnott, Jettie Hoonhout, Dalila Szostak, Robin Brewer, et al. 2013. Current issues and future directions in methods for studying technology in the home. PsychNology Journal 11, 2 (2013), 159--184.

[10]

Current Population Reports. 2014. An Aging Nation: The Older Population in the United States. https://www.census.gov/prod/2014pubs/p25-1140.pdf. (2014).

[11]

Joshua D. Neuheisel Diego A. Socolinsky, Andrea Selinger. 2003. Face recognition with visible and thermal infrared imagery. Computer Vision and Image Understanding Volume 91 (2003).

[12]

Alex Edgcomb and Frank Vahid. 2012. Privacy perception and fall detection accuracy for in-home video assistive monitoring with privacy enhancements. ACM SIGHIT Record 2, 2 (2012), 6--15.

Digital Library

[13]

FiveThrityEight. 2014. How Common Is It For A Man To Be Shorter Than His Partner? https://fivethirtyeight.com/datalab/how-common-is-it-for-a-man-to-be-shorter-than-his-partner/. (2014).

[14]

Francois Fleuret, Jerome Berclaz, and Richard Lengagne. 2008. Multicamera People Tracking with a Probabilistic Occupancy Map. IEEE Transactions on Pattern Analysis and Machine Intelligence 30 (2008), 267--282. Issue 2. https://doi.org/document/4359319/

Digital Library

[15]

Rikke Gade and Thomas B Moeslund. 2014. Thermal cameras and applications: a survey. Machine vision and applications 25, 1 (2014), 245--262.

Digital Library

[16]

D. Gafurov and E. Snekkenes. 2009. Gait recognition using wearable motion recording sensors. EURASIP Journal on Advances in Signal Processing 2009 (2009), 7.

Digital Library

[17]

Adam Gudyś, Jakub Rosner, Jakub Segen, Konrad Wojciechowski, and Marek Kulbacki. 2014. Tracking people in video sequences by clustering feature motion paths. In Computer Vision and Graphics. Springer, 236--245.

[18]

Timothy W Hnat, Erin Griffiths, Ray Dawson, and Kamin Whitehouse. 2012. Doorjamb: unobtrusive room-level tracking of people in homes using doorway sensors. In Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems. ACM, 309--322.

Digital Library

[19]

Roberto Hoyle, Robert Templeman, Steven Armes, Denise Anthony, David Crandall, and Apu Kapadia. 2014. Privacy behaviors of lifeloggers using wearable cameras. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 571--582.

Digital Library

[20]

Avinash Kalyanaraman, Erin Griffiths, and Kamin Whitehouse. 2016. TransTrack: Tracking Multiple Targets by Sensing Their Zone Transitions. In Distributed Computing in Sensor Systems (DCOSS), 2016 International Conference on. IEEE, 59--66.

[21]

Nacer Khalil, Driss Benhaddou, Omprakash Gnawali, and Jaspal Subhlok. 2016. Nonintrusive Occupant Identification by Sensing Body Shape and Movement. In Proceedings of the 3rd ACM International Conference on Systems for Energy-Efficient Built Environments. ACM, 1--10.

Digital Library

[22]

Sa-Mun Kim, Dae-Jong Lee, Ho-Hyun Lee, and Myung-Geun Chun. 2016. Person Recognition Using Gait and Face Features on Thermal Images. The Transactions of the Korean Institute of Electrical Engineers P 65, 2 (2016), 130--135.

[23]

Tiffany Hyun-Jin Kim, Lujo Bauer, James Newsome, Adrian Perrig, and Jesse Walker. 2010. Challenges in Access Right Assignment for Secure Home Networks. In HotSec.

[24]

Alexandra König, Carlos Fernando Crispim Junior, Alexandre Derreumaux, Gregory Bensadoun, Pierre-David Petit, François Bremond, Renaud David, Frans Verhey, Pauline Aalten, and Philippe Robert. 2015. Validation of an automatic video monitoring system for the detection of instrumental activities of daily living in dementia patients. Journal of Alzheimer's Disease 44, 2 (2015), 675--685.

[25]

Pavel Korshunov, Claudia Araimo, Francesca De Simone, Carmelo Velardo, J-L Dugelay, and Touradj Ebrahimi. 2012. Subjective study of privacy filters in video surveillance. In Multimedia Signal Processing (MMSP), 2012 IEEE 14th International Workshop on. Ieee, 378--382.

[26]

Shihong Lao and Masato Kawade. 2004. Vision-based face understanding technologies and their applications. In Advances in Biometric Person Authentication. Springer, 339--348.

Digital Library

[27]

Eric Larson, Gabe Cohn, Sidhant Gupta, Xiaofeng Ren, Beverly Harrison, Dieter Fox, and Shwetak Patel. 2011. HeatWave: thermal imaging for surface user interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2565--2574.

Digital Library

[28]

Alex Leykin, Yang Ran, and Riad Hammoud. 2007. Thermal-visible video fusion for moving target tracking and pedestrian classification. In Computer Vision and Pattern Recognition, 2007. CVPR‘07. IEEE Conference on. IEEE, 1--8.

[29]

Chenglong Li, Shiyi Hu, Sihan Gao, and Jin Tang. 2016. Real-time grayscale-thermal tracking via laplacian sparse representation. In International Conference on Multimedia Modeling. Springer, 54--65.

[30]

Z. Liu and S. Sarkar. 2007. Outdoor recognition at a distance by fusing gait and face. Image and Vision Computing 25, 6 (2007), 817--832.

Digital Library

[31]

Zhan Lu, Yichen Xu, Zuoxiao Dai, and Bei Ma. 2016. A gait recognition based on link model of infrared thermal imaging. In Control Science and Systems Engineering (ICCSSE), 2016 2nd International Conference on. IEEE, 165--168.

[32]

Hessam Mohammadmoradi, Sirajum Munir, Omprakash Gnawali, and Charles Shelton. [n. d.]. Measuring People-Flow Through Doorways using Easy-to-Install IR Array Sensors. ([n. d.]).

[33]

Eslam Mostafa, Riad Hammoud, Asem Ali, and Aly Farag. 2013. Face recognition in low resolution thermal images. Computer Vision and Image Understanding 117, 12 (2013), 1689--1694.

Digital Library

[34]

Matteo Munaro, Filippo Basso, and Emanuele Menegatti. 2012. Tracking people within groups with RGB-D data. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, 2101--2107.

[35]

Sirajum Munir, Ripudaman Singh Arora, Craig Hesling, Juncheng Li, Jonathan Francis, Charles Shelton, Christopher Martin, Anthony Rowe, and Mario Berges. 2017. Real-Time Fine Grained Occupancy Estimation using Depth Sensors on ARM Embedded Platforms. In Real-Time and Embedded Technology and Applications Symposium (RTAS), 2017 IEEE. IEEE, 295--306.

[36]

Carman Neustaedter, Saul Greenberg, and Michael Boyle. 2006. Blur filtration fails to preserve privacy for home-based video conferencing. ACM Transactions on Computer-Human Interaction (TOCHI) 13, 1 (2006), 1--36.

Digital Library

[37]

Savvas Papaioannou, Andrew Markham, and Niki Trigoni. 2016. Tracking People in Highly Dynamic Industrial Environments. IEEE Transactions on Mobile Computing (2016).

[38]

Fei Peng, Xiao-wen Zhu, and Min Long. 2013. An ROI privacy protection scheme for H. 264 video based on FMO and Chaos. Information Forensics and Security, IEEE Transactions on 8, 10 (2013), 1688--1699.

Digital Library

[39]

R Rajeev and SS Sridhar. 2014. Tracking People Motion Based on LloydâĂ&Zacute;s Clustering Algorithm. Int. J. Comput. Sci. Inf. Technol 5, 2 (2014), 2289--2292.

[40]

Elliot N Saba, Eric C Larson, and Shwetak N Patel. 2012. Dante vision: In-air and touch gesture sensing for natural surface interaction with combined depth and thermal cameras. In Emerging Signal Processing Applications (ESPA), 2012 IEEE International Conference on. IEEE, 167--170.

[41]

Florian Schaub, Bastian Konings, and Michael Weber. 2015. Context-adaptive privacy: Leveraging context awareness to support privacy decision making. Pervasive Computing, IEEE 14, 1 (2015), 34--43.

[42]

Y.L. Shen and C.S. Shin. 2009. Distributed Sensing Floor for an Intelligent Environment. Sensors Journal, IEEE 9, 12 (2009), 1673--1678.

[43]

Andrew Sixsmith and Neil Johnson. 2004. A smart sensor to detect the falls of the elderly. IEEE Pervasive computing 3, 2 (2004), 42--47.

Digital Library

[44]

Paul Smith, Mubarak Shah, and Niels da Vitoria Lobo. 2000. Monitoring head/eye motion for driver alertness with one camera. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, Vol. 4. IEEE, 636--642.

Digital Library

[45]

Fang Tang and Amine Bermak. 2012. An 84 pW/Frame per pixel current-mode CMOS image sensor with energy harvesting capability. IEEE Sensors Journal 12, 4 (2012), 720--726.

[46]

D Sabuncuoglu Tezcan, Selim Eminoglu, and Tayfun Akin. 2003. A low-cost uncooled infrared microbolometer detector in standard CMOS technology. IEEE Transactions on electron devices 50, 2 (2003), 494--502.

[47]

The Atlantic. 2013. Why It's So Rare for a Wife to Be Taller Than Her Husband. https://www.theatlantic.com/sexes/archive/2013/01/why-its-so-rare-for-a-wife-to-be-taller-than-her-husband/272585/. (2013).

[48]

The Physics Factbook. 2014. Temperature Of A Healthy Human (Skin Temperature). http://hypertextbook.com/facts/2001/AbantyFarzana.shtml. (2014).

[49]

Andreas Savvides Thiago Teixeira. 2007. Lightweight People Counting and Localizing in Indoor Spaces Using Camera Sensor Nodes. First ACM/IEEE International Conference on Distributed Smart Cameras (2007). https://doi.org/10.1109/ICDSC.2007.4357503

[50]

Atousa Torabi, Guillaume Massé, and Guillaume-Alexandre Bilodeau. 2012. An iterative integrated framework for thermal--visible image registration, sensor fusion, and people tracking for video surveillance applications. Computer Vision and Image Understanding 116, 2 (2012), 210--221.

Digital Library

[51]

U.S. Energy Information Administration (EIA). 2017. How much energy is consumed in U.S. residential and commercial buildings? https://www.eia.gov/tools/faqs/faq.php?id=86&t=1. (2017).

[52]

Shreyas Venugopalan, Unni Prasad, Khalid Harun, Kyle Neblett, Douglas Toomey, Joseph Heyman, and Marios Savvides. 2011. Long range iris acquisition system for stationary and mobile subjects. In Biometrics (IJCB), 2011 International Joint Conference on. IEEE, 1--8.

Digital Library

[53]

John Vines, Stephen Lindsay, Gary W Pritchard, Mabel Lie, David Greathead, Patrick Olivier, and Katie Brittain. 2013. Making family care work: dependence, privacy and remote home monitoring telecare systems. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, 607--616.

Digital Library

[54]

Wei Wang, Alex X Liu, and Muhammad Shahzad. 2016. Gait recognition using wifi signals. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 363--373.

Digital Library

[55]

Hongyang Xue, Yao Liu, Deng Cai, and Xiaofei He. 2016. Tracking people in RGBD videos using deep learning and motion clues. Neurocomputing 204 (2016), 70--76.

Digital Library

[56]

Yunze Zeng, Parth H Pathak, and Prasant Mohapatra. 2016. WiWho: WiFi-based Person Identification in Smart Spaces. In 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 1--12.

Digital Library

[57]

Tianzhu Zhang, Kui Jia, Changsheng Xu, Yi Ma, and Narendra Ahuja. 2014. Partial occlusion handling for visual tracking via robust part matching. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1258--1265.

Digital Library

[58]

Zhengyou Zhang. 2012. Microsoft kinect sensor and its effect. IEEE multimedia 19, 2 (2012), 4--10.

Digital Library

[59]

Zui Zhang, Oscar Perez Concha, and Massimo Piccardi. 2015. Tracking people under heavy occlusions by layered data association. Multimedia Tools and Applications 74, 17 (2015), 7239--7259.

Digital Library

Cited By

Khatoon SMahmood AAkhtar THu B(2024)Environmental Sustainability Through Optimal Energy Consumption Using IoT-Based Edge-Computing and Image ProcessingExploring Pillars of Sustainability for Modern Age Improvements10.4018/979-8-3693-5748-4.ch013(289-316)Online publication date: 4-Oct-2024
https://doi.org/10.4018/979-8-3693-5748-4.ch013
Xiao YSharma HZhang ZBergen-Cico DRahman TSalekin A(2024)Reading Between the HeatProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314417:4(1-30)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631441
Venugopalan HDin ZCarpenter TLowe-Power JKing SShafiq Z(2024)AragornProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314067:4(1-31)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631406
Show More Cited By

Index Terms

Privacy-preserving Image Processing with Binocular Thermal Cameras
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Sensors and actuators
2. Security and privacy
  1. Human and societal aspects of security and privacy
    1. Privacy protections

Recommendations

Multi-level privacy preserving data publishing

Policedata is an important source of social media data and can be regarded as a technical assistance to increase government accountability and transparency. Notably, it contains large amounts of personal private information that should be preserved ...
Privacy-Preserving Data Publishing: An Overview
Privacy preserving data publishing

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 4

December 2017

1298 pages

EISSN:2474-9567

DOI:10.1145/3178157

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 January 2018

Accepted: 01 October 2017

Revised: 01 August 2017

Received: 01 May 2017

Published in IMWUT Volume 1, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
628
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)5

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Khatoon SMahmood AAkhtar THu B(2024)Environmental Sustainability Through Optimal Energy Consumption Using IoT-Based Edge-Computing and Image ProcessingExploring Pillars of Sustainability for Modern Age Improvements10.4018/979-8-3693-5748-4.ch013(289-316)Online publication date: 4-Oct-2024
https://doi.org/10.4018/979-8-3693-5748-4.ch013
Xiao YSharma HZhang ZBergen-Cico DRahman TSalekin A(2024)Reading Between the HeatProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314417:4(1-30)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631441
Venugopalan HDin ZCarpenter TLowe-Power JKing SShafiq Z(2024)AragornProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314067:4(1-31)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631406
Lupión MPolo-Rodríguez AMedina-Quero JSanjuan JOrtigosa P(2024)3D Human Pose Estimation from multi-view thermal vision sensorsInformation Fusion10.1016/j.inffus.2023.102154104(102154)Online publication date: Apr-2024
https://doi.org/10.1016/j.inffus.2023.102154
Al-Ghaili AGunasekaran SJamil NAlyasseri ZAl-Hada NIbrahim ZBakar AKasim HHosseini EOmar RKasmani RRazali R(2023)A Review on Role of Image Processing Techniques to Enhancing Security of IoT ApplicationsIEEE Access10.1109/ACCESS.2023.331268211(101924-101948)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3312682
Burns MNugent CMcClean SQuero JPolo-Rodríguez A(2023)A Deep Learning and Probabilistic Approach to Recognising Activities of Daily Living with Privacy Preserving Thermal SensorsProceedings of the 15th International Conference on Ubiquitous Computing & Ambient Intelligence (UCAmI 2023)10.1007/978-3-031-48642-5_15(155-166)Online publication date: 26-Nov-2023
https://doi.org/10.1007/978-3-031-48642-5_15
Du WLi AZhou PNiu BWu D(2022)PrivacyEye: A Privacy-Preserving and Computationally Efficient Deep Learning-Based Mobile Video Analytics SystemIEEE Transactions on Mobile Computing10.1109/TMC.2021.305045821:9(3263-3279)Online publication date: 1-Sep-2022
https://doi.org/10.1109/TMC.2021.3050458
Zhang SWang XLiu M(2022)Construction of Intelligent Sensing Environment for Equestrian Sports Venues based on Binocular Motion Detection Cameras2022 International Conference on Edge Computing and Applications (ICECAA)10.1109/ICECAA55415.2022.9936201(1513-1516)Online publication date: 13-Oct-2022
https://doi.org/10.1109/ICECAA55415.2022.9936201
Lupión MPolo-Rodríguez AMedina-Quero JSanjuan JOrtigosa P(2022)On the limits of Conditional Generative Adversarial Neural Networks to reconstruct the identification of inhabitants from IoT low-resolution thermal sensorsExpert Systems with Applications10.1016/j.eswa.2022.117356203(117356)Online publication date: Oct-2022
https://doi.org/10.1016/j.eswa.2022.117356
Azimi SO'Brien W(2022)Fit-for-purpose: Measuring occupancy to support commercial building operations: A reviewBuilding and Environment10.1016/j.buildenv.2022.108767212(108767)Online publication date: Mar-2022
https://doi.org/10.1016/j.buildenv.2022.108767
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents