research-article

Public Access

HoloSet - A Dataset for Visual-Inertial Pose Estimation in Extended Reality: Dataset

Authors:

Fatima M. AnwarAuthors Info & Claims

SenSys '22: Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems

Pages 1014 - 1019

https://doi.org/10.1145/3560905.3567763

Published: 24 January 2023 Publication History

Abstract

There is a lack of datasets for visual-inertial odometry applications in Extended Reality (XR). To the best of our knowledge, there is no dataset available that is captured from an XR headset with a human as a carrier. To bridge this gap, we present a novel pose estimation dataset --- called HoloSet --- collected using Microsoft Hololens 2, which is a state-of-the-art head mounted device for XR. Potential applications for HoloSet include visual-inertial odometry, simultaneous localization and mapping (SLAM), and additional applications in XR that leverage visual-inertial data.

HoloSet captures both macro and micro movements. For macro movements, the dataset consists of more than 66,000 samples of visual, inertial, and depth camera data in a variety of environments (indoor, outdoor) and scene setups (trails, suburbs, downtown) under multiple user action scenarios (walk, jog). For micro movements, the dataset consists of more than 12,000 samples of additional articulated hand depth camera images while a user plays games that exercise fine motor skills and hand-eye coordination. We present basic visualizations and high-level statistics of the data and outline the potential research use cases for HoloSet.

References

[1]

2019. Real life Dangers of Augmented Reality. https://spectrum.ieee.org/consumer-electronics/portable-devices/the-reallife-dangers-of-augmented-reality

[2]

Tim Bailey and Hugh Durrant-Whyte. 2006. Simultaneous localization and mapping (SLAM): Part II. IEEE robotics & automation magazine 13, 3 (2006), 108--117.

[3]

James Birt, Zane Stromberga, Michael Cowling, and Christian Moro. 2018. Mobile mixed reality for experiential learning and simulation in medical and health sciences education. Information 9, 2 (2018), 31.

[4]

José-Luis Blanco-Claraco, Francisco-Angel Moreno-Duenas, and Javier González-Jiménez. 2014. The Málaga urban dataset: High-rate stereo and LiDAR in a realistic urban scenario. The International Journal of Robotics Research 33, 2 (2014), 207--214.

Digital Library

[5]

Martin Brossard, Axel Barrau, and Silvere Bonnabel. 2019. RINS-W: Robust inertial navigation system on wheels. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2068--2075.

Digital Library

[6]

Michael Burri, Janosch Nikolic, Pascal Gohl, Thomas Schneider, Joern Rehder, Sammy Omari, Markus W Achtelik, and Roland Siegwart. 2016. The EuRoC micro aerial vehicle datasets. The International Journal of Robotics Research 35, 10 (2016), 1157--1163.

Digital Library

[7]

Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. 2019. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8748--8757.

[8]

Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R Millán, and Daniel Roggen. 2013. The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters 34, 15 (2013), 2033--2042.

Digital Library

[9]

Changhao Chen, Stefano Rosa, Yishu Miao, Chris Xiaoxuan Lu, Wei Wu, Andrew Markham, and Niki Trigoni. 2019. Selective sensor fusion for neural visual-inertial odometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10542--10551.

[10]

Changhao Chen, Peijun Zhao, Chris Xiaoxuan Lu, Wei Wang, Andrew Markham, and Niki Trigoni. 2018. OxIOD: The Dataset for Deep Inertial Odometry. arXiv preprint arXiv:1809.07491 (2018).

[11]

Adrian David Cheok, Anuroop Sreekumar, Cao Lei, and Le Nam Thang. 2006. Capture the flag: mixed-reality social gaming with smart phones. IEEE Pervasive Computing 5, 2 (2006), 62--69.

Digital Library

[12]

Wenyan Ci, Yingping Huang, and Xing Hu. 2019. Stereo visual odometry based on motion decoupling and special feature screening for navigation of autonomous vehicles. IEEE Sensors Journal 19, 18 (2019), 8047--8056.

[13]

Ronald Clark, Sen Wang, Hongkai Wen, A. Markham, and Agathoniki Trigoni. 2017. VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem. In AAAI.

[14]

Santiago Cortés, Arno Solin, Esa Rahtu, and Juho Kannala. 2018. ADVIO: An Authentic Dataset for Visual-Inertial Odometry. In Proceedings of the European Conference on Computer Vision (ECCV).

Digital Library

[15]

Fernando De la Torre, Jessica Hodgins, Adam Bargteil, Xavier Martin, Justin Macey, Alex Collado, and Pep Beltran. 2009. Guide to the carnegie mellon university multimodal activity (cmu-mmac) database. (2009).

[16]

European Organization For Nuclear Research and OpenAIRE. 2013. Zenodo.

[17]

Tobias Feigl, Sebastian Kram, Philipp Woller, Ramiz H Siddiqui, Michael Philippsen, and Christopher Mutschler. 2020. RNN-aided human velocity estimation from a single IMU. Sensors 20, 13 (2020), 3656.

[18]

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. 2013. Vision Meets Robotics: The KITTI Dataset. The International Journal of Robotics Research (2013).

[19]

Mar Gonzalez-Franco, Rodrigo Pizarro, Julio Cermeron, Katie Li, Jacob Thorn, Windo Hutabarat, Ashutosh Tiwari, and Pablo Bermell-Garcia. 2017. Immersive mixed reality for manufacturing training. Frontiers in Robotics and AI 4 (2017), 3.

[20]

Mert Gurturk, Abdullah Yusefi, Muhammet Fatih Aslan, Metin Soycan, Akif Durdu, and Andrea Masiero. 2021. The YTU Dataset and Recurrent Neural Network based Visual-inertial Odometry. Measurement (2021).

[21]

Ankur Handa, Thomas Whelan, John McDonald, and Andrew J Davison. 2014. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In 2014 IEEE international conference on Robotics and automation (ICRA). IEEE, 1524--1531.

[22]

Patrick Hübner, Kate Clintworth, Qingyi Liu, Martin Weinmann, and Sven Wursthorn. 2020. Evaluation of HoloLens tracking and depth sensing for indoor mapping applications. Sensors 20, 4 (2020), 1021.

[23]

Charles E Hughes, Christopher B Stapleton, Darin E Hughes, and Eileen M Smith. 2005. Mixed reality in education, entertainment, and training. IEEE computer graphics and applications 25, 6 (2005), 24--30.

Digital Library

[24]

Microsoft Inc. 2020. Hololens 2. https://www.microsoft.com/en-us/hololens/hardware. (2020). "[Online; accessed 13th-June-2022]".

[25]

SJ Ingram, D Harmer, and M Quinlan. 2004. Ultrawideband indoor positioning systems and their use in emergencies. In PLANS 2004. Position Location and Navigation Symposium (IEEE Cat. No. 04CH37556). IEEE, 706--715.

[26]

Max Juraschek, Lennart Büth, Gerrit Posselt, and Christoph Herrmann. 2018. Mixed reality in learning factories. Procedia Manufacturing 23 (2018), 153--158.

[27]

Siddhartha Khandelwal and Nicholas Wickström. 2017. Evaluation of the performance of accelerometer-based gait event detection algorithms in different real-world scenarios using the MAREA gait database. Gait & posture 51 (2017), 84--90.

[28]

Itzik Klein. 2019. Smartphone location recognition: A deep learning-based approach. Sensors 20, 1 (2019), 214.

[29]

Nicola Krombach, David Droeschel, Sebastian Houben, and Sven Behnke. 2018. Feature-based visual odometry prior for real-time semi-dense stereo SLAM. Robotics and Autonomous Systems 109 (2018), 38--58.

[30]

Cristian A Linte, Katherine P Davenport, Kevin Cleary, Craig Peters, Kirby G Vosburgh, Nassir Navab, Philip Eddie Edwards, Pierre Jannin, Terry M Peters, David R Holmes, and Richard A Robb. 2013. On Mixed Reality Environments for Minimally Invasive Therapy Guidance: Systems Architecture, Successes and Challenges In Their Implementation From Laboratory To Clinic. Computerized Medical Imaging and Graphics 37, 2 (March 2013), 83--97.

[31]

Melanie J Maas and Janette M Hughes. 2020. Virtual, augmented and mixed reality in K-12 education: A review of the literature. Technology, Pedagogy and Education 29, 2 (2020), 231--249.

[32]

Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman. 2017. 1 year, 1000 km: The Oxford RobotCar dataset. The International Journal of Robotics Research 36, 1 (2017), 3--15.

Digital Library

[33]

Adi Manos, Tamir Hazan, and Itzik Klein. 2022. Walking Direction Estimation Using Smartphone Sensors: A Deep Network-Based Framework. IEEE Transactions on Instrumentation and Measurement 71 (2022), 1--12.

[34]

Pierre Merriaux, Yohan Dupuis, Rémi Boutteau, Pascal Vasseur, and Xavier Savatier. 2017. A study of vicon system positioning performance. Sensors 17, 7 (2017), 1591.

[35]

Dimitris Mourtzis, John Angelopoulos, and Nikos Panopoulos. 2021. Collaborative manufacturing design: a mixed reality and cloud-based framework for part design. Procedia CIRP 100 (2021), 97--102.

[36]

Dimitris Mourtzis, Aikaterini Vlachou, and Vasilios Zogopoulos. 2017. Cloud-based augmented reality remote maintenance through shop-floor monitoring: a product-service system approach. Journal of Manufacturing Science and Engineering 139, 6 (2017).

[37]

Dimitris Mourtzis, Vasilios Zogopoulos, and E Vlachou. 2017. Augmented reality application to support remote maintenance as a service in the robotics industry. Procedia Cirp 63 (2017), 46--51.

[38]

Friedrich Niemann, Christopher Reining, Fernando Moya Rueda, Nilah Ravi Nair, Janine Anika Steffens, Gernot A Fink, and Michael Ten Hompel. 2020. Lara: Creating a dataset for human activity recognition in logistics using semantic attributes. Sensors 20, 15 (2020), 4083.

[39]

A Alan B Pritsker. 1984. Introduction to Simulation and SLAM II. Halsted Press.

[40]

David Robert, Ryan Wistorrt, Jesse Gray, and Cynthia Breazeal. 2010. Exploring mixed reality robot gaming. In Proceedings of the fifth international conference on tangible, embedded, and embodied interaction. 125--128.

Digital Library

[41]

David Schubert, Thore Goll, Nikolaus Demmel, Vladyslav Usenko, Jörg Stückler, and Daniel Cremers. 2018. The TUM VI Benchmark for Evaluating Visual-inertial Odometry. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE.

[42]

Thomas Schöps, Torsten Sattler, and Marc Pollefeys. 2019. BAD SLAM: Bundle Adjusted Direct RGB-D SLAM. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 134--144.

[43]

Inês Soares, Ricardo B. Sousa, Marcelo Petry, and António Paulo Moreira. 2021. Accuracy and repeatability tests on HoloLens 2 and HTC Vive. Multimodal Technologies and Interaction 5, 8 (2021), 47.

[44]

John Spinello. 1965. Operation Game. https://hasbrogaming.hasbro.com/en-us/product/classic-operation-game:03B9FF5E-5056-9047-F5FF-D57592C4E2FE.

[45]

Anna Syberfeldt, Oscar Danielsson, and Patrik Gustavsson. 2017. Augmented reality smart glasses in the smart factory: Product evaluation guidelines and review of available products. Ieee Access 5 (2017), 9118--9130.

[46]

Zachary Teed and Jia Deng. 2021. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in Neural Information Processing Systems 34 (2021), 16558--16569.

[47]

Dorin Ungureanu, Federica Bogo, S. Galliani, Pooja Sama, Xin Duan, Casey Meekhof, Jan Stühmer, Thomas J. Cashman, Bugra Tekin, Johannes L. Schönberger, Pawel Olszta, and Marc Pollefeys. 2020. HoloLens 2 Research Mode as a Tool for Computer Vision Research. ArXiv abs/2008.11239 (2020).

[48]

Tom Van Haute, Eli De Poorter, Pieter Crombez, Filip Lemic, Vlado Handziski, Niklas Wirström, Adam Wolisz, Thiemo Voigt, and Ingrid Moerman. 2016. Performance analysis of multiple Indoor Positioning Systems in a healthcare environment. International journal of health geographics 15, 1 (2016), 1--15.

[49]

Petr Vávra, Jan Roman, P. Zonča, Peter Ihnát, M. Němec, Kumar Jayant, Nagy Habib, and Ahmed El-Gendi. 2017. Recent Development of Augmented Reality in Surgery: A Review. Journal of Healthcare Engineering 2017 (08 2017), 1--9.

[50]

Qu Wang, Haiyong Luo, Langlang Ye, Aidong Men, Fang Zhao, Yan Huang, and Changhai Ou. 2019. Pedestrian heading estimation based on spatial transformer networks and hierarchical LSTM. IEEE Access 7 (2019), 162309--162322.

[51]

Ze Wang, Kailun Yang, Hao Shi, and Kaiwei Wang. 2022. LF-VIO: A Visual-Inertial-Odometry Framework for Large Field-of-View Cameras with Negative Plane. arXiv preprint arXiv:2202.12613 (2022).

[52]

Mi Zhang and Alexander A Sawchuk. 2012. USC-HAD: A daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the 2012 ACM conference on ubiquitous computing. 1036--1043.

Digital Library

[53]

Mingming Zhang, Xingxing Zuo, Yiming Chen, Yong Liu, and Mingyang Li. 2021. Pose estimation for ground robots: On manifold representation, integration, reparameterization, and optimization. IEEE Transactions on Robotics 37, 4 (2021), 1081--1099.

Digital Library

[54]

David Zuñiga-Noël, Alberto Jaenal, Ruben Gomez-Ojeda, and Javier Gonzalez-Jimenez. 2020. The uma-vi dataset: Visual-inertial odometry in low-textured and dynamic illumination environments. The International Journal of Robotics Research 39, 9 (2020), 1052--1060.

Digital Library

Cited By

Alekseeva DMezina ABurget RArponen OLohan EOmetov A(2025)Enhancing Extended Reality Assisted Surgery through a Field-of-View Video Delivery OptimizationComputer Networks10.1016/j.comnet.2025.111093259(111093)Online publication date: Mar-2025
https://doi.org/10.1016/j.comnet.2025.111093
Wang TZhong SGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Fingerprinting in EEG Model IP Protection Using Diffusion ModelProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658057(120-128)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658057
Guo DLi KHu BZhang YWang M(2024)Benchmarking Micro-Action Recognition: Dataset, Methods, and ApplicationsIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.335841534:7(6238-6252)Online publication date: Jul-2024
https://doi.org/10.1109/TCSVT.2024.3358415
Show More Cited By

Index Terms

HoloSet - A Dataset for Visual-Inertial Pose Estimation in Extended Reality: Dataset

Recommendations

Vision meets robotics: The KITTI dataset

We present a novel dataset captured from a VW station wagon for use in mobile robotics and autonomous driving research. In total, we recorded 6 hours of traffic scenarios at 10-100 Hz using a variety of sensor modalities such as high-resolution color and ...
HMDPose: A large-scale trinocular IR Augmented Reality Glasses Pose Dataset
VRST '20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology

Augmented Reality Glasses usually implement an Inside-Out tracking. In case of a driving scenario or glasses with less computation capabilities, an Outside-In tracking approach is required. However, to the best of our knowledge, no public datasets ...
The Visual-Inertial Canoe Dataset

We present a dataset collected from a canoe along the Sangamon River in Illinois. The canoe was equipped with a stereo camera, an inertial measurement unit IMU, and a global positioning system GPS device, which provide visual data suitable for stereo or ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SenSys '22: Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems

November 2022

1280 pages

ISBN:9781450398862

DOI:10.1145/3560905

General Chairs:
Jeremy Gummeson
University of Massachusetts Amherst
,
Sunghoon Ivan Lee
University of Massachusetts Amherst
,
Program Chairs:
Jie Gao
Rutgers University
,
Guoliang Xing
The Chinese University of Hong Kong

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 January 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

SenSys '22

Sponsor:

SenSys '22: The 20th ACM Conference on Embedded Networked Sensor Systems

November 6 - 9, 2022

Massachusetts, Boston

Acceptance Rates

SenSys '22 Paper Acceptance Rate 52 of 187 submissions, 28%;

Overall Acceptance Rate 198 of 990 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
389
Total Downloads

Downloads (Last 12 months)215
Downloads (Last 6 weeks)35

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Alekseeva DMezina ABurget RArponen OLohan EOmetov A(2025)Enhancing Extended Reality Assisted Surgery through a Field-of-View Video Delivery OptimizationComputer Networks10.1016/j.comnet.2025.111093259(111093)Online publication date: Mar-2025
https://doi.org/10.1016/j.comnet.2025.111093
Wang TZhong SGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Fingerprinting in EEG Model IP Protection Using Diffusion ModelProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658057(120-128)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658057
Guo DLi KHu BZhang YWang M(2024)Benchmarking Micro-Action Recognition: Dataset, Methods, and ApplicationsIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.335841534:7(6238-6252)Online publication date: Jul-2024
https://doi.org/10.1109/TCSVT.2024.3358415
Chandio YKhan MSelialia KGarcia LDeGol JAnwar F(2024)A Neurosymbolic Approach to Adaptive Feature Extraction in SLAM2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS58592.2024.10802379(4941-4948)Online publication date: 14-Oct-2024
https://doi.org/10.1109/IROS58592.2024.10802379
Chandio YBashir NAnwar F(2024)Stealthy and Practical Multi-modal Attacks on Mixed Reality Tracking2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR59861.2024.00010(11-20)Online publication date: 17-Jan-2024
https://doi.org/10.1109/AIxVR59861.2024.00010
Plizzari CGoletto GFurnari ABansal SRagusa FFarinella GDamen DTommasi T(2024)An Outlook into the Future of Egocentric VisionInternational Journal of Computer Vision10.1007/s11263-024-02095-7132:11(4880-4936)Online publication date: 28-May-2024
https://doi.org/10.1007/s11263-024-02095-7

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten