skip to main content
research-article

Unravelling Spatial Privacy Risks of Mobile Mixed Reality Data

Published: 30 March 2021 Publication History

Abstract

Previously, 3D data---particularly, spatial data---have primarily been utilized in the field of geo-spatial analyses, or robot navigation (e.g. self-automated cars) as 3D representations of geographical or terrain data (usually extracted from lidar). Now, with the increasing user adoption of augmented, mixed, and virtual reality (AR/MR/VR; we collectively refer to as MR) technology on user mobile devices, spatial data has become more ubiquitous. However, this ubiquity also opens up a new threat vector for adversaries: aside from the traditional forms of mobile media such as images and video, spatial data poses additional and, potentially, latent risks to users of AR/MR/VR. Thus, in this work, we analyse MR spatial data using various spatial complexity metrics---including a cosine similarity-based, and a Euclidean distance-based metric---as heuristic or empirical measures that can signify the inference risk a captured space has. To demonstrate the risk, we utilise 3D shape recognition and classification algorithms for spatial inference attacks over various 3D spatial data captured using mobile MR platforms: i.e. Microsoft HoloLens, and Android with Google ARCore. Our experimental evaluation and investigation shows that the cosine similarity-based metric is a good spatial complexity measure of captured 3D spatial maps and can be utilised as an indicator of spatial inference risk.

Supplementary Material

de-guzman (de-guzman.zip)
Supplemental movie, appendix, image and software files for, Unravelling Spatial Privacy Risks of Mobile Mixed Reality Data

References

[1]
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, 308--318.
[2]
Alessandro Acquisti. 2011. Privacy in the age of augmented reality. (2011).
[3]
Paarijaat Aditya, Rijurekha Sen, Peter Druschel, Seong Joon Oh, Rodrigo Benenson, Mario Fritz, Bernt Schiele, Bobby Bhattacharjee, and Tong Tong Wu. 2016. I-pic: A platform for privacy-compliant image capture. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 235--248.
[4]
Jalal Al-Muhtadi, Roy Campbell, Apu Kapadia, M Dennis Mickunas, and Seung Yi. 2002. Routing through the mist: Privacy preserving communication in ubiquitous computing environments. In Proceedings 22nd International Conference on Distributed Computing Systems. IEEE, 74--83.
[5]
Mihael Ankerst, Gabi Kastenmüller, Hans-Peter Kriegel, and Thomas Seidl. 1999. 3D shape histograms for similarity search and classification in spatial databases. In International symposium on spatial databases. Springer, 207--226.
[6]
Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. 2016. NetVLAD: CNN architecture for weakly supervised place recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5297--5307.
[7]
Anoop Aryan and Sanjay Singh. 2010. Protecting location privacy in augmented reality using k-anonymization and pseudo-id. In 2010 International Conference on Computer and Communication Technology (ICCCT). IEEE, 119--124.
[8]
Anoop Aryan and Sanjay Singh. 2010. Securing location privacy in Augmented Reality. In 2010 5th International Conference on Industrial and Information Systems. IEEE, 172--176.
[9]
Stefano Baldassi, Tadayoshi Kohno, Franziska Roesner, and Moqian Tian. 2018. Challenges and New Directions in Augmented Reality, Computer Security, and Neuroscience-Part 1: Risks to Sensation and Perception. arXiv preprint arXiv:1806.10557 (2018).
[10]
Steve Benford, Chris Brown, Gail Reynard, and Chris Greenhalgh. 1996. Shared spaces: transportation, artificiality, and spatiality. In Proceedings of the 1996 ACM conference on Computer supported cooperative work. ACM, 77--86.
[11]
Steve Benford, Chris Greenhalgh, Gail Reynard, Chris Brown, and Boriana Koleva. 1998. Understanding and constructing shared spaces with mixed-reality boundaries. ACM Transactions on computer-human interaction (TOCHI) 5, 3 (1998), 185--223.
[12]
Cara Bloom, Joshua Tan, Javed Ramjohn, and Lujo Bauer. 2017. Self-driving cars and data collection: Privacy perceptions of networked autonomous vehicles. In Thirteenth Symposium on Usable Privacy and Security ({SOUPS} 2017). 357--375.
[13]
Michael Bosse and Robert Zlot. 2009. Keypoint design and evaluation for place recognition in 2D lidar maps. Robotics and Autonomous Systems 57, 12 (2009), 1211--1224.
[14]
Michael Bosse and Robert Zlot. 2013. Place recognition using keypoint voting in large 3D lidar datasets. In 2013 IEEE International Conference on Robotics and Automation. IEEE, 2677--2684.
[15]
Alexander M Bronstein, Michael M Bronstein, Benjamin Bustos, Umberto Castellani, Marco Crisani, Bianca Falcidieno, Leonidas J Guibas, Iasonas Kokkinos, Vittorio Murino, et al. 2010. SHREC 2010: robust feature detection and description benchmark. Proc. EUROGRAPHICS Workshop on 3D Object Retrieval (3DOR) (2010).
[16]
Jaybie A de Guzman, Kanchana Thilakarathna, and Aruna Seneviratne. 2019. A First Look into Privacy Leakage in 3D Mixed Reality Data. In European Symposium on Research in Computer Security. Springer, 149--169.
[17]
Jaybie A de Guzman, Kanchana Thilakarathna, and Aruna Seneviratne. 2019. SafeMR: Privacy-aware Visual Information Protection for Mobile Mixed Reality. In 2019 IEEE 41st Conference on Local Computer Networks (LCN). IEEE.
[18]
Jaybie A. De Guzman, Kanchana Thilakarathna, and Aruna Seneviratne. 2019. Security and Privacy Approaches in Mixed Reality: A Literature Survey. ACM Comput. Surv. 52, 6, Article 110 (Oct. 2019), 37 pages. https://doi.org/10.1145/3359626
[19]
Jaybie A de Guzman, Kanchana Thilakarathna, and Aruna Seneviratne. 2020. Conservative Plane Releasing for Spatial Privacy Protection in Mixed Reality. arXiv preprint arXiv:2004.08029 (2020).
[20]
H Quynh Dinh and Liefei Xu. 2008. Measuring the similarity of vector fields using global distributions. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR). Springer, 187--196.
[21]
Cynthia Dwork, Aaron Roth, et al. 2014. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science 9, 3-4 (2014), 211--407.
[22]
Lucas Silva Figueiredo, Benjamin Livshits, David Molnar, and Margus Veanes. 2016. Prepose: Privacy, Security, and Reliability for Gesture-Based Programming. In Security and Privacy (SP), 2016 IEEE Symposium on. IEEE, 122--137.
[23]
Martin A Fischler and Robert C Bolles. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6 (1981), 381--395.
[24]
Batya Friedman and Peter H Kahn Jr. 2000. New directions: A Value-Sensitive Design approach to augmented reality. In Proceedings of DARE 2000 on designing augmented reality environments. ACM, 163--164.
[25]
Bugra Gedik and Ling Liu. 2007. Protecting location privacy with personalized k-anonymity: Architecture and algorithms. IEEE Transactions on Mobile Computing 7, 1 (2007), 1--18.
[26]
Marco Gruteser and Dirk Grunwald. 2003. Anonymous usage of location-based services through spatial and temporal cloaking. In Proceedings of the 1st international conference on Mobile systems, applications and services. 31--42.
[27]
Olli I Heimo, Kai K Kimppa, Seppo Helle, Timo Korkalainen, and Teijo Lehtonen. 2014. Augmented reality-Towards an ethical fantasy?. In Ethics in Science, Technology and Engineering, 2014 IEEE International Symposium on. IEEE, 1--7.
[28]
Jing Huang and Suya You. 2012. Point cloud matching based on 3D self-similarity. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on. IEEE, 41--48.
[29]
Suman Jana, David Molnar, Alexander Moshchuk, Alan M Dunn, Benjamin Livshits, Helen J Wang, and Eyal Ofek. 2013. Enabling Fine-Grained Permissions for Augmented Reality Applications with Recognizers. In USENIX Security.
[30]
Suman Jana, Arvind Narayanan, and Vitaly Shmatikov. 2013. A Scanner Darkly: Protecting user privacy from perceptual applications. In Security and Privacy (SP), 2013 IEEE Symposium on. IEEE, 349--363.
[31]
Andrew Edie Johnson and Martial Hebert. 1998. Surface matching for object recognition in complex three-dimensional scenes. Image and Vision Computing 16, 9-10 (1998), 635--651.
[32]
Andrew E Johnson and Martial Hebert. 1999. Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Transactions on Pattern Analysis & Machine Intelligence 5 (1999), 433--449.
[33]
Marc Langheinrich. 2001. Privacy by design---principles of privacy-aware ubiquitous systems. In International conference on Ubiquitous Computing. Springer, 273--291.
[34]
Kiron Lebeck, Tadayoshi Kohno, and Franziska Roesner. 2016. How to safely augment reality: Challenges and Directions. In Proceedings of the 17th International Workshop on Mobile Computing Systems and Applications. ACM, 45--50.
[35]
Kiron Lebeck, Kimberly Ruth, Tadayoshi Kohno, and Franziska Roesner. 2017. Securing Augmented Reality Output. In Security and Privacy (SP), 2017 IEEE Symposium on. IEEE, 320--337.
[36]
Kiron Lebeck, Kimberly Ruth, Tadayoshi Kohno, and Franziska Roesner. 2018. Towards Security and Privacy for Multi-User Augmented Reality: Foundations with End Users. In Towards Security and Privacy for Multi-User Augmented Reality: Foundations with End Users. IEEE, 0.
[37]
Ang Li, Qinghua Li, and Wei Gao. 2016. PrivacyCamera: Cooperative Privacy-Aware Photographing with Mobile Phones. In Sensing, Communication, and Networking (SECON), 2016 13th Annual IEEE International Conference on. IEEE, 1--9.
[38]
Frank McSherry and Kunal Talwar. 2007. Mechanism design via differential privacy. In Foundations of Computer Science, 2007. FOCS'07. 48th Annual IEEE Symposium on. IEEE, 94--103.
[39]
Abedelaziz Mohaisen, Dowon Hong, and DaeHun Nyang. 2008. Privacy in location based services: Primitives toward the solution. In 2008 Fourth International Conference on Networked Computing and Advanced Information Management, Vol. 1. IEEE, 572--579.
[40]
Ryutarou Ohbuchi, Takahiro Minamitani, and Tsuyoshi Takei. 2005. Shape-similarity search of 3D models by using enhanced shape functions. International Journal of Computer Applications in Technology 23, 2-4 (2005), 70--85.
[41]
Robert Osada, Thomas Funkhouser, Bernard Chazelle, and David Dobkin. 2002. Shape distributions. ACM Transactions on Graphics (TOG) 21, 4 (2002), 807--832.
[42]
Francesco Pittaluga, Sanjeev J Koppal, Sing Bing Kang, and Sudipta N Sinha. 2019. Revealing scenes by inverting structure from motion reconstructions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 145--154.
[43]
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 652--660.
[44]
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in neural information processing systems. 5099--5108.
[45]
Nisarg Raval, Animesh Srivastava, Kiron Lebeck, Landon Cox, and Ashwin Machanavajjhala. 2014. Markit: Privacy markers for protecting visual secrets. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. ACM, 1289--1295.
[46]
Nisarg Raval, Animesh Srivastava, Ali Razeen, Kiron Lebeck, Ashwin Machanavajjhala, and Lanodn P Cox. 2016. What you mark is what apps see. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 249--261.
[47]
Jingjing Ren, Ashwin Rao, Martina Lindorfer, Arnaud Legout, and David Choffnes. 2016. Recon: Revealing and controlling pii leaks in mobile network traffic. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 361--374.
[48]
Franziska Roesner, David Molnar, Alexander Moshchuk, Tadayoshi Kohno, and Helen J Wang. 2014. World-driven access control for continuous sensing. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security. ACM, 1169--1181.
[49]
Yoones A Sekhavat. 2017. Privacy preserving cloth try-on using mobile augmented reality. IEEE Transactions on Multimedia 19, 5 (2017), 1041--1049.
[50]
Reza Shokri and Vitaly Shmatikov. 2015. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security. ACM, 1310--1321.
[51]
Jiayu Shu, Rui Zheng, and Pan Hui. 2016. Cardea: Context-Aware Visual Privacy Protection from Pervasive Cameras. arXiv preprint arXiv:1610.00889 (2016).
[52]
Pablo Speciale, Johannes L Schonberger, Sing Bing Kang, Sudipta N Sinha, and Marc Pollefeys. 2019. Privacy preserving image-based localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5493--5503.
[53]
Przemys&lstroke;aw Spurek, Sebastian Winczowski, Jacek Tabor, Maciej Zamorski, Maciej Zieba, and Tomasz Trzcinski. 2020. Hypernetwork approach to generating point clouds. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and Aarti Singh (Eds.). PMLR, 9099--9108. http://proceedings.mlr.press/v119/spurek20a.html
[54]
Julian Steil, Marion Koelle, Wilko Heuten, Susanne Boll, and Andreas Bulling. 2018. PrivacEye: Privacy-Preserving First-Person Vision Using Image Features and Eye Movement Analysis. CoRR abs/1801.04457 (2018). arXiv:1801.04457 http://arxiv.org/abs/1801.04457
[55]
Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. 2009. A concise and provably informative multi-scale signature based on heat diffusion. In Computer graphics forum, Vol. 28. Wiley Online Library, 1383--1392.
[56]
Latanya Sweeney. 2002. k-anonymity: A model for protecting privacy. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 10, 05 (2002), 557--570.
[57]
Mikaela Angelina Uy and Gim Hee Lee. 2018. PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[58]
John Vilk, David Molnar, Benjamin Livshits, Eyal Ofek, Chris Rossbach, Alexander Moshchuk, Helen J Wang, and Ran Gal. 2015. Surroundweb: Mitigating privacy concerns in a 3d web browser. In Security and Privacy (SP), 2015 IEEE Symposium on. IEEE, 431--446.
[59]
John Vilk, David Molnar, Eyal Ofek, Chris Rossbach, Ben Livshits, Alexander Moshchuk, Helen J. Wang, and Ran Gal. 2014. Least Privilege Rendering in a 3D Web Browser. https://www.microsoft.com/en-us/research/publication/least-privilege-rendering-3d-web-browser/ Microsoft Research Technical Report MSR-TR-2014-25.
[60]
Junjue Wang, Brandon Amos, Anupam Das, Padmanabhan Pillai, Norman Sadeh, and Mahadev Satyanarayanan. 2017. A Scalable and Privacy-Aware IoT Service for Live Video Analytics. (2017).
[61]
Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. 2019. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE International Conference on Computer Vision. 4541--4550.
[62]
Chunyong Yin, Jinwen Xi, and Ruxia Sun. 2017. Location privacy protection based on improved-value method in augmented reality on mobile devices. Mobile Information Systems (2017).
[63]
Chunyong Yin, Jinwen Xi, Ruxia Sun, and Jin Wang. 2017. Location privacy protection based on differential privacy strategy for big data in industrial internet of things. IEEE Transactions on Industrial Informatics 14, 8 (2017), 3628--3636.
[64]
Eisa Zarepour, Mohammadreza Hosseini, Salil S Kanhere, and Arcot Sowmya. 2016. A context-based privacy preserving framework for wearable visual lifeloggers. In Pervasive Computing and Communication Workshops (PerCom Workshops), 2016 IEEE International Conference on. IEEE, 1--4.

Cited By

View all
  • (2024)Adaptive Mobile Diminished Reality Framework for 3D Visual Privacy2024 International Conference on Intelligent Systems and Computer Vision (ISCV)10.1109/ISCV60512.2024.10620084(1-8)Online publication date: 8-May-2024
  • (2024)The Security and Privacy Concerns on MetaverseComputer Networks and IoT10.1007/978-981-97-1332-5_23(287-305)Online publication date: 3-Apr-2024
  • (2023)Mobile Diminished Reality for Preserving 3D Visual Privacy2023 International Conference on Intelligent Metaverse Technologies & Applications (iMETA)10.1109/iMETA59369.2023.10294436(01-07)Online publication date: 18-Sep-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 5, Issue 1
March 2021
1272 pages
EISSN:2474-9567
DOI:10.1145/3459088
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 March 2021
Published in IMWUT Volume 5, Issue 1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. and resource sharing
  2. mixed or augmented reality
  3. object detection
  4. privacy

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)117
  • Downloads (Last 6 weeks)10
Reflects downloads up to 28 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Adaptive Mobile Diminished Reality Framework for 3D Visual Privacy2024 International Conference on Intelligent Systems and Computer Vision (ISCV)10.1109/ISCV60512.2024.10620084(1-8)Online publication date: 8-May-2024
  • (2024)The Security and Privacy Concerns on MetaverseComputer Networks and IoT10.1007/978-981-97-1332-5_23(287-305)Online publication date: 3-Apr-2024
  • (2023)Mobile Diminished Reality for Preserving 3D Visual Privacy2023 International Conference on Intelligent Metaverse Technologies & Applications (iMETA)10.1109/iMETA59369.2023.10294436(01-07)Online publication date: 18-Sep-2023
  • (2023)DiOS—An Extended Reality Operating System for the MetaverseIEEE MultiMedia10.1109/MMUL.2022.321135130:2(70-80)Online publication date: 1-Apr-2023
  • (2023)The Dark Side of Augmented Reality: Exploring Manipulative Designs in ARInternational Journal of Human–Computer Interaction10.1080/10447318.2023.218879940:13(3449-3464)Online publication date: 27-Mar-2023
  • (2023)Post-Covid-19 Metaverse Cybersecurity and Data Privacy: Present and Future ChallengesData Protection in a Post-Pandemic Society10.1007/978-3-031-34006-2_1(1-48)Online publication date: 8-May-2023
  • (2023)Privacy and Security Issues and Solutions for Mixed Reality ApplicationsSpringer Handbook of Augmented Reality10.1007/978-3-030-67822-7_7(157-183)Online publication date: 1-Jan-2023
  • (2023)The Metaverse with Life and Everything: An Overview of Privacy, Ethics, and Governance*Metaverse Communication and Computing Networks10.1002/9781394160013.ch12(293-310)Online publication date: 6-Oct-2023
  • (2022)Vision: Usable Privacy for XR in the Era of the MetaverseProceedings of the 2022 European Symposium on Usable Security10.1145/3549015.3554212(111-116)Online publication date: 29-Sep-2022
  • (2022)Stepping Into the Next Decade of Ubiquitous and Pervasive Computing: UbiComp and ISWC 2021IEEE Pervasive Computing10.1109/MPRV.2022.316006321:2(87-99)Online publication date: 1-Apr-2022
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media