Getting it Just Right: Towards Balanced Utility, Privacy, and Equity in Shared Space Sensing
Authors: 
Andrew Xu, 
Jacob Biehl, Adam LeeAuthors Info & Claims
Andrew Xu, Jacob Biehl, Adam LeeAuthors Info & ClaimsArticle No.: 14, Pages 1 - 26
Abstract
Low-cost sensors have enabled a wide array of data-driven applications and insights. As a result, encountering spaces with pervasive sensing has become all but unavoidable. This creates a fundamental tension: the success of smart environments will become increasingly dependent on equity of access to data-driven insights and consideration of the privacy expectations of sensed individuals. These concerns highlight the need to bring equity to all stakeholders of smart environments, which in turn would preserve public trust in these smart spaces. In this work, we explored several approaches to identity-obscuring visual representations through a progressive series of experiments. We designed and validated a series of visual representations through stakeholder interactions and tested the ability of these visual representations to limit identification via a crowdsourced study. An evaluation across three months of data gathered within our organization also showed that the identity-obscured data could still be leveraged to accurately count group size. Our contributions lay the groundwork for sensing frameworks that bring utility to all stakeholders of shared spaces while being cognizant of their diverse privacy expectations.
References
[1]
Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y. Lim, and Mohan Kankanhalli. 2018. Trends and trajectories for explainable, accountable and intelligible systems: An HCI research agenda. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI’18). Association for Computing Machinery, 1–18. DOI:
[2]
Aakriti Adhikari, Hem Regmi, Sanjib Sur, and Srihari Nelakuditi. 2022. MiShape: Accurate human silhouettes and body joints from commodity millimeter-wave devices. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 3, Article 96 (sep 2022), 31 pages. DOI:
[3]
Prachi Agrawal and P. J. Narayanan. 2011. Person de-identification in videos. IEEE Transactions on Circuits and Systems for Video Technology 21, 3 (2011), 299–310. DOI:
[4]
Imtiaz Ahmad, Rosta Farzan, Apu Kapadia, and Adam J. Lee. 2020. Tangible privacy: Towards user-centric sensor designs for bystander privacy. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 116 (oct 2020), 28 pages. DOI:
[5]
Karan Ahuja, Yue Jiang, Mayank Goel, and Chris Harrison. 2021. Vid2Doppler: Synthesizing doppler radar data from videos for training privacy-preserving activity recognition. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI’21). Association for Computing Machinery, Article 292, 10 pages. DOI:
[6]
Karan Ahuja, Dohyun Kim, Franceska Xhakaj, Virag Varga, Anne Xie, Stanley Zhang, Jay Eric Townsend, Chris Harrison, Amy Ogan, and Yuvraj Agarwal. 2019. EduSense: Practical classroom sensing at scale. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 3, Article 71 (sep 2019), 26 pages. DOI:
[7]
A.R. Beresford and F. Stajano. 2003. Location privacy in pervasive computing. IEEE Pervasive Computing 2, 1 (2003), 46–55. DOI:
[8]
Dmitri Bitouk, Neeraj Kumar, Samreen Dhillon, Peter Belhumeur, and Shree K. Nayar. 2008. Face swapping: Automatically replacing faces in photographs. ACM Trans. Graph. 27, 3 (aug 2008), 1–8. DOI:
[9]
Michael Boyle, Christopher Edwards, and Saul Greenberg. 2000. The effects of filtered video on awareness and privacy. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW’00). Association for Computing Machinery, 1–10. DOI:
[10]
Daniel J. Butler, Justin Huang, Franziska Roesner, and Maya Cakmak. 2015. The privacy-utility tradeoff for remotely teleoperated robots. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human–Robot Interaction (HRI’15). Association for Computing Machinery, 27–34. DOI:
[11]
Bin Chen, Koki Hatada, Keiju Okabayashi, Hiroyuki Kuromiya, Ichiro Hidaka, Yoshiharu Yamamoto, and Kazumasa Togami. 2019. Group activity recognition to support collaboration in creative digital space. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing (CSCW’19) . Association for Computing Machinery, 175–179. DOI:
[12]
Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, and Liang-Chieh Chen. 2020. Panoptic-DeepLab: A simple, strong, and fast baseline for bottom-Up panoptic segmentation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Seattle, Washington, 12472–12482. DOI:
[13]
Wongun Choi, Yu-Wei Chao, Caroline Pantofaru, and Silvio Savarese. 2014. Discovering groups of people in images. In Computer Vision—ECCV 2014. David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 417–433.
[14]
Ivan Culjak, David Abram, Tomislav Pribanic, Hrvoje Dzapo, and Mario Cifrek. 2012. A brief introduction to OpenCV. In 2012 Proceedings of the 35th International Convention MIPRO. IEEE, Opatija, Croatia, 1725–1730.
[15]
Julia Dietlmeier, Joseph Antony, Kevin McGuinness, and Noel E. O’Connor. 2021. How important are faces for person re-identification?. In 2020 25th International Conference on Pattern Recognition (ICPR). IEEE Computer Society, 6912–6919. DOI:
[16]
Benedikt Driessen and Markus Dürmuth. 2013. Achieving anonymity against major face recognition algorithms. In Communications and Multimedia Security. Bart De Decker, Jana Dittmann, Christian Kraetzer, and Claus Vielhauer (Eds.). Springer, Berlin, 18–33.
[17]
Darren Ellis, David Harper, and Ian Tucker. 2013. The dynamics of impersonal trust and distrust in surveillance systems. Sociological Research Online 18, 3 (2013), 85–96. DOI:
[18]
Ethan Fast and Eric Horvitz. 2017. Long-term trends in the public perception of artificial intelligence. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI’17). AAAI Press, San Francisco, California, 963–969.
[19]
Oran Gafni, Lior Wolf, and Yaniv Taigman. 2019. Live face de-identification in video. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Seoul, Korea (South), 9377–9386. DOI:
[20]
Andrew C. Gallagher and Tsuhan Chen. 2009. Understanding images of groups of people. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Miami, Florida, 256–263. DOI:
[21]
Ralph Gross, Edoardo Airoldi, Bradley Malin, and Latanya Sweeney. 2006. Integrating utility into face de-identification. In Privacy Enhancing Technologies. George Danezis and David Martin (Eds.), Springer, Berlin, 227–242.
[22]
R. Gross, L. Sweeney, F. de la Torre, and S. Baker. 2006. Model-based face de-identification. In 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’06). IEEE, New York City, NY, 161–161. DOI:
[23]
Ralph Gross, Latanya Sweeney, Fernando de la Torre, and Simon Baker. 2008. Semi-supervised learning of multi-factor models for face de-identification. In 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Anchorage, Alaska, 1–8. DOI:
[24]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, Venice, Italy, 2980–2988. DOI:
[25]
Peter Hevesi, Sebastian Wille, Gerald Pirkl, Norbert Wehn, and Paul Lukowicz. 2014. Monitoring household activities and user location with a cheap, unobtrusive thermal sensor array. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’14). Association for Computing Machinery, 141–145. DOI:
[26]
Håkon Hukkelås, Rudolf Mester, and Frank Lindseth. 2019. DeepPrivacy: A generative adversarial network for face anonymization. In Advances in Visual Computing: 14th International Symposium on Visual Computing, ISVC 2019, Lake Tahoe, NV, October 7–9, 2019, Proceedings, Part I (Lake Tahoe, NV, USA). Springer-Verlag, Berlin, 565–578. DOI:
[27]
Yasha Iravantchi, Karan Ahuja, Mayank Goel, Chris Harrison, and Alanson Sample. 2021. PrivacyMic: Utilizing inaudible frequencies for privacy preserving daily activity recognition. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI’21). Association for Computing Machinery, Article 198, 13 pages. DOI:
[28]
Suman Jana, Arvind Narayanan, and Vitaly Shmatikov. 2013. A scanner darkly: Protecting user privacy from perceptual applications. In 2013 IEEE Symposium on Security and Privacy. IEEE, Berkeley, CA, 349–363. DOI:
[29]
E. Karthika, K. Amutha, S. Janardhana Prabhu, S.J. Kristin Jerushia, and H. Padmapriya. 2018. Object detection using radar. In 2018 International Conference on Current Trends towards Converging Technologies (ICCTCT). IEEE, Coimbatore, India, 1–4. DOI:
[30]
Aftab Khan, James Nicholson, Sebastian Mellor, Daniel Jackson, Karim Ladha, Cassim Ladha, Jon Hand, Joseph Clarke, Patrick Olivier, and Thomas Plötz. 2014. Occupancy monitoring using environmental & context sensors and a hierarchical analysis framework. In Proceedings of the 1st ACM Conference on Embedded Systems for Energy-Efficient Buildings (BuildSys’14) . Association for Computing Machinery, 90–99. DOI:
[31]
Gierad Laput, Yang Zhang, and Chris Harrison. 2017. Synthetic sensors: Towards general-purpose sensing. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI’17). Association for Computing Machinery, 3986–3999. DOI:
[32]
Bokyung Lee, Michael Lee, Jeremy Mogk, Rhys Goldstein, Jacobo Bibliowicz, Frederik Brudy, and Alexander Tessier. 2021. Designing a multi-agent occupant simulation system to support facility planning and analysis for COVID-19. In Proceedings of the 2021 ACM Designing Interactive Systems Conference (Virtual Event, USA) (DIS’21). Association for Computing Machinery, New York, NY, 15–30. DOI:
[33]
Bokyung Lee, Michael Lee, Pan Zhang, Alexander Tessier, and Azam Khan. 2019. An empirical study of how socio-spatial formations are influenced by interior elements and displays in an office context. Proceedings of the ACM on Human–Computer Interaction 3, CSCW (2019), 1–26. DOI:
[34]
Bokyung Lee, Michael Lee, Pan Zhang, Alexander Tessier, Daniel Saakes, and Azam Khan. 2021. Socio-spatial comfort. Proceedings of the ACM on Human–Computer Interaction 4, CSCW3 (2021), 1–33. DOI:
[35]
Boning Li, Xiangbo Shu, and Rui Yan. 2021. Storyboard relational model for group activity recognition. In Proceedings of the 2nd ACM International Conference on Multimedia in Asia (MMAsia’20). Association for Computing Machinery, Article 2, 7 pages. DOI:
[36]
Jiacheng Lin, Yang Li, and Guanci Yang. 2021. FPGAN: Face de-identification method with generative adversarial networks for social robots. Neural Netw. 133 (Jan. 2021), 132–147.
[37]
Beth Logan, Jennifer Healey, Matthai Philipose, Emmanuel Munguia Tapia, and Stephen Intille. 2007. A long-term evaluation of sensing modalities for activity recognition. In Proceedings of the 9th International Conference on Ubiquitous Computing (UbiComp’07). Springer-Verlag, Berlin, 483–500.
[38]
Ching-Hu Lu and Li-Chen Fu. 2009. Robust location-aware activity recognition using wireless sensor network in an attentive home. IEEE Transactions on Automation Science and Engineering 6, 4 (2009), 598–609. DOI:
[39]
Dimitrios Lymberopoulos, Athanasios Bamis, and Andreas Savvides. 2008. Extracting spatiotemporal human activity patterns in assisted living using a home sensor network. In Proceedings of the 1st International Conference on PErvasive Technologies Related to Assistive Environments (PETRA’08). Association for Computing Machinery, Article 29, 8 pages. DOI:
[40]
G. M. Lyons, K. M. Culhane, D. Hilton, P. A. Grace, and D. Lyons. 2005. A description of an accelerometer-based mobility monitoring technique. Med. Eng. Phys. 27, 6 (July 2005), 497–504.
[41]
Naresh K. Malhotra, Sung S. Kim, and James Agarwal. 2004. Internet users’ information privacy concerns (IUIPC): The construct, the scale, and a causal model. Information Systems Research 15, 4 (2004), 336–355.
[42]
Blaž Meden, Žiga Emeršič, Vitomir Štruc, and Peter Peer. 2018. K-same-net: K-anonymity with generative deep neural networks for face deidentification. Entropy (Basel) 20, 1 (Jan. 2018), 60.
[43]
Samantha Mitchell Finnigan and Adrian K. Clear. 2020. “No powers, man!”: A student perspective on designing university smart building interactions. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI’20). Association for Computing Machinery, New York, NY, 1–14. DOI:
[44]
Sowmya Myneni, Garima Agrawal, Yuli Deng, Ankur Chowdhary, Neha Vadnere, and Dijiang Huang. 2022. SCVS: On AI and edge clouds enabled privacy-preserved smart-city video surveillance services. ACM Trans. Internet Things 3, 4, Article 28 (sep 2022), 26 pages. DOI:
[45]
M. Najibi, P. Samangouei, R. Chellappa, and L. S. Davis. 2017. SSH: Single stage headless face detector. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE Computer Society, Los Alamitos, CA, 4885–4894. DOI:
[46]
Florian Nothdurft, Felix Richter, and Wolfgang Minker. 2014. Probabilistic human-computer trust handling. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL). Association for Computational Linguistics, Philadelphia, PA, 51–59. DOI:
[47]
José Ramón Padilla-López, Alexandros Andre Chaaraoui, and Francisco Flórez-Revuelta. 2015. Visual privacy protection methods: A survey. Expert Systems with Applications 42, 9 (2015), 4177–4195. DOI:
[48]
Francesco Pittaluga and Sanjeev Jagannatha Koppal. 2017. Pre-capture privacy for small vision sensors. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 11 (2017), 2215–2226. DOI:
[49]
Rui Qian, Divyansh Garg, Yan Wang, Yurong You, Serge Belongie, Bharath Hariharan, Mark Campbell, Kilian Q. Weinberger, and Wei-Lun Chao. 2020. End-to-end pseudo-LiDAR for image-based 3D object detection. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Seattle, WA, 5880–5889. DOI:
[50]
Nisarg Raval, Animesh Srivastava, Kiron Lebeck, Landon Cox, and Ashwin Machanavajjhala. 2014. MarkIt: Privacy markers for protecting visual secrets. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp’14 Adjunct). Association for Computing Machinery, 1289–1295. DOI:
[51]
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You only look once: Unified, real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, 779–788. DOI:
[52]
Matthew Rueben, Frank J. Bernieri, Cindy M. Grimm, and William D. Smart. 2016. User feedback on physical marker interfaces for protecting visual privacy from mobile robots. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, Christchurch, New Zealand, 507–508. DOI:
[53]
Nazmus Saquib, Ayesha Bose, Dwyane George, and Sepandar Kamvar. 2018. Sensei: Sensing educational interaction. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4, Article 161 (jan 2018), 27 pages. DOI:
[54]
Jeremy Schiff, Marci Meingast, Deirdre K. Mulligan, Shankar Sastry, and Ken Goldberg. 2009. Respectful Cameras: Detecting Visual Markers in Real-Time to Address Privacy Concerns. Springer London, London, 65–89. DOI:
[55]
Holger Schnädelbach, Nils Jäger, and Lachlan Urquhart. 2019. Adaptive architecture and personal data. ACM Transactions on Computer–Human Interaction 26, 2 (2019), 1–31. DOI:
[56]
Yansong Tang, Zian Wang, Peiyang Li, Jiwen Lu, Ming Yang, and Jie Zhou. 2018. Mining semantics-preserving attention for group activity recognition. In Proceedings of the 26th ACM International Conference on Multimedia (MM’18). Association for Computing Machinery, 1283–1291. DOI:
[57]
Robert Templeman, Mohammed Korayem, David Crandall, and Apu Kapadia. 2014. PlaceAvoider: Steering first-person cameras away from sensitive spaces. DOI:
[58]
Nava Tintarev and Judith Masthoff. 2011. Designing and Evaluating Explanations for Recommender Systems. Springer, 479–510. DOI:
[59]
Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. 2020. SOLOv2: Dynamic and fast instance segmentation. In Advances in Neural Information Processing Systems. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 17721–17732. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2020/file/cd3afef9b8b89558cd56638c3631868a-Paper.pdf
[60]
Yizhou Wang, Zhongyu Jiang, Xiangyu Gao, Jenq-Neng Hwang, Guanbin Xing, and Hui Liu. 2021. RODNet: Radar object detection using cross-modal supervision. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, Waikoloa, HI, 504–513. DOI: ISSN: 2642-9381.
[61]
Li-Fang Wu, Qi Wang, Meng Jian, Yu Qiao, and Bo-Xuan Zhao. 2021. A comprehensive review of group activity recognition in videos. Int. J. Autom. Comput. 18, 3 (jun 2021), 334–350. DOI:
[62]
Rui Yan, Jinhui Tang, Xiangbo Shu, Zechao Li, and Qi Tian. 2018. Participation-contributed temporal dynamic model for group activity recognition. In Proceedings of the 26th ACM International Conference on Multimedia (MM’18). Association for Computing Machinery, 1292–1300. DOI:
[63]
Guanci Yang, Jing Yang, Weihua Sheng, Francisco Erivaldo Fernandes Junior, and Shaobo Li. 2018. Convolutional neural network-based embarrassing situation detection under camera for social robot in smart homes. Sensors 18, 5 (2018). DOI:
[64]
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. 2018. BiSeNet: Bilateral segmentation network for real-time semantic segmentation. In Computer Vision—ECCV 2018. Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.), Springer International Publishing, Cham, 334–349.
[65]
Qiang Alex Zhao and John T. Stasko. 1998. Evaluating image filtering based techniques in media space applications. In Proceedings of the 1998 ACM Conference on Computer Supported Cooperative Work (CSCW’98). Association for Computing Machinery, 11–18. DOI:
[66]
Sailin Zhong, Denis Lalanne, and Hamed Alavi. 2021. The complexity of indoor air quality forecasting and the simplicity of interacting with it—a case study of 1007 office meetings. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI’21). Association for Computing Machinery, Article 47, 19 pages. DOI:
[67]
Hanwei Zhu, Sid Chi-Kin Chau, Gladhi Guarddin, and Weifa Liang. 2022. Integrating IoT-sensing and crowdsensing with privacy: Privacy-preserving hybrid sensing for smart cities. ACM Trans. Internet Things 3, 4, Article 31 (sep 2022), 30 pages. DOI:
Index Terms
- Getting it Just Right: Towards Balanced Utility, Privacy, and Equity in Shared Space Sensing
Recommendations
Preserving Privacy in Smart Homes: A Socio-Cultural Approach
CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing SystemsSmart homes are expected to reduce the time we spend on routine activities. Data becomes an enabler and asset, as smart devices collect and process it. Smart homes also have the potential to exaggerate bewilderment and resistance, feelings people express ...
Towards a Harmonised Approach for Security and Privacy Management in Smart Home Contexts
HCI for Cybersecurity, Privacy and TrustAbstractSmart homes are based upon Internet-connected versions of household devices that have traditionally operated in a standalone manner. In recent years, the proliferation of Internet-connected devices has increased dramatically, particularly in the ...
Towards an Architecture for Balancing Privacy and Traceability in Ubiquitous Computing Environments
ARES '08: Proceedings of the 2008 Third International Conference on Availability, Reliability and SecurityPrivacy Preservation has been identified as an important factor to the success and acceptance of Ubiquitous Compu-ting systems. Traceability, i.e.~attributing events and actions to those who caused them, seems to be a directly contradicting goal. ...
Comments
Information & Contributors
Information
Published In
May 2024
214 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Journal Family
Publication History
Published: 15 May 2024
Online AM: 29 February 2024
Accepted: 19 January 2024
Revised: 03 October 2023
Received: 16 December 2022
Published in TIOT Volume 5, Issue 2
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 190Total Downloads
- Downloads (Last 12 months)188
- Downloads (Last 6 weeks)11
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in