Hiroto Oshimi
Monica Perusquía-Hernández
ABSTRACT
Item-finding tasks due to memory lapse are costly activities commonly experienced by many people. However, conventional systems are not suitable for use in a collaborative environment. Therefore, we propose a multi-functional, pre-registration-free, and 3D location-based item management system. The system has two main functions: registration and search. The automatic registration is performed by image-based item movement recognition from the user’s grasping and placing motions. The registered item movement data comprises the item category, and the start and end locations. We ensure privacy protection by storing item movement data without images. Also, we provide a user interaction to refuse to share the items with other users. The search is based on the item list or item location. The location-based search is performed by specifying where the user last saw the item. To optimize and test the performance of the system, we first performed parameter optimization and then conducted a user study investigating the performance of a search task. The parameter optimization performed in the registration system led to the discovery of optimal values that are difficult to reach empirically. The search experiment showed that the proposed system’s search and guidance functions are effective as an assistance system for finding items, both in terms of search time and user experience. Overall, our system demonstrated the potential to be a useful assistance system for managing items in a shared space. We further discuss the possibility of further exploiting the limited registered information by treating item location as an identifier of the moved item.
Supplemental Material
- Ronald T. Azuma. 1997. A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6, 4(1997), 355–385. https://doi.org/10.1162/pres.1997.6.4.355Google Scholar
Digital Library
- Patrick Baudisch and Ruth Rosenholtz. 2003. Halo: a technique for visualizing off-screen objects. In Proceedings of the SIGCHI conference on Human factors in computing systems. 481–488.Google ScholarDigital Library
- Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. 2022. YOLACT + + Better Real-Time Instance Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 2(2022), 1108–1121. https://doi.org/10.1109/tpami.2020.3014297Google ScholarDigital Library
- Gaetano Borriello, Waylon Brunette, Matthew Hall, Carl Hartung, and Cameron Tangney. 2004. Reminding About Tagged Objects Using Passive RFIDs. In Proceedings of the 6th International Conference on Ubiquitous Computing (Nottingham, England) (UbiComp ’04). 36–53. https://doi.org/10.1007/978-3-540-30119-6_3Google ScholarCross Ref
- Kaixin Chen, Yongzhi Huang, Yicong Chen, Haobin Zhong, Lihua Lin, Lu Wang, and Kaishun Wu. 2022. LiSee: A Headphone That Provides All-Day Assistance for Blind and Low-Vision Users to Reach Surrounding Objects. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1–30. https://doi.org/10.1145/3550282Google ScholarDigital Library
- Rudolph P. Darken and John L. Sibert. 1996. Wayfinding Strategies and Behaviors in Large Virtual Worlds. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, British Columbia, Canada) (CHI ’96). 142–149. https://doi.org/10.1145/238386.238459Google ScholarDigital Library
- Muhammad Elsayeh, Mohamed Haroon, Bassel Tawfik, and Ahmed Fahmy. 2010. RFID-based Indoors Localization of Tag-less Objects. In Proceedings of the 5th Cairo International Biomedical Engineering Conference (Cairo, Egypt) (CIBEC ’10). 61–65. https://doi.org/10.1109/CIBEC.2010.5716049Google ScholarCross Ref
- Alessandro Farasin, Francesco Peciarolo, Marco Grangetto, Elena Gianaria, and Paolo Garza. 2020. Real-time Object Detection and Tracking in Mixed Reality using Microsoft HoloLens. In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (Valletta, Malta) (VISIGRAPP ’20, Vol. 4). 165–172. https://doi.org/10.5220/0008877901650172Google ScholarCross Ref
- Markus Funk, Robin Boldt, Bastian Pfleging, Max Pfeiffer, Niels Henze, and Albrecht Schmidt. 2014. Representing Indoor Location of Objects on Wearable Computers with Head-Mounted Displays. In Proceedings of the 5th Augmented Human International Conference (Kobe, Japan) (AH ’14). Article 18, 4 pages. https://doi.org/10.1145/2582051.2582069Google ScholarDigital Library
- Markus Funk, Albrecht Schmidt, and Lars Erik Holmquist. 2013. Antonius: A Mobile Search Engine for the Physical World. In Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication (Zurich, Switzerland) (UbiComp ’13 Adjunct). 179–182. https://doi.org/10.1145/2494091.2494149Google ScholarDigital Library
- Uwe Gruenefeld, Daniel Lange, Lasse Hammer, Susanne Boll, and Wilko Heuten. 2018. Flyingarrow: Pointing towards out-of-view objects on augmented reality devices. In Proceedings of the 7th ACM international symposium on pervasive displays. 1–6.Google ScholarDigital Library
- Sean Gustafson, Patrick Baudisch, Carl Gutwin, and Pourang Irani. 2008. Wedge: clutter-free visualization of off-screen locations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 787–796.Google ScholarDigital Library
- Roberto Hoyle, Robert Templeman, Steven Armes, Denise Anthony, David Crandall, and Apu Kapadia. 2014. Privacy Behaviors of Lifeloggers Using Wearable Cameras. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing(Seattle, Washington) (UbiComp ’14). 571–582. https://doi.org/10.1145/2632048.2632079Google ScholarDigital Library
- Julie A. Kientz, Shwetak N. Patel, Arwa Z. Tyebkhan, Brian Gane, Jennifer Wiley, and Gregory D. Abowd. 2006. Where’s My Stuff? Design and Evaluation of a Mobile System for Locating Lost Items for the Visually Impaired. In Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Portland, Oregon, USA) (Assets ’06). 103–110. https://doi.org/10.1145/1168987.1169006Google ScholarDigital Library
- Franklin Mingzhe Li, Di Laura Chen, Mingming Fan, and Khai N. Truong. 2019. FMT: A Wearable Camera-Based Object Tracking Memory Aid for Older Adults. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1–30. https://doi.org/10.1145/3550282Google ScholarDigital Library
- Xiaotao Liu, Mark D. Corner, and Prashant Shenoy. 2006. Ferret: RFID Localization for Pervasive Multimedia. In Proceedings of the 8th International Conference on Ubiquitous Computing (Orange County, CA) (UbiComp ’06). 422–440. https://doi.org/10.1007/11853565_25Google ScholarDigital Library
- Yuri Álvarez López, Jacqueline Franssen, Guillermo Álvarez Narciandi, Janet Pagnozzi, Ignacio González-Pinto Arrillaga, and Fernando Las-Heras Andrés. 2018. RFID Technology for Management and Tracking: e-Health Applications. Sensors 18, 8: 2663 (2018), 1–17. https://doi.org/10.3390/s18082663Google Scholar
- Takuya Maekawa, Yutaka Yanagisawa, Yasue Kishino, Katsuhiko Ishiguro, Koji Kamei, Yasushi Sakurai, and Takeshi Okadome. 2010. Object-Based Activity Recognition with Heterogeneous Sensors on Wrist. In Proceedings of the 8th International Conference on Pervasive Computing (Helsinki, Finland) (Pervasive’10). 246–264. https://doi.org/10.1007/978-3-642-12654-3_15Google ScholarDigital Library
- Toyohisa Nakada, Hideaki Kanai, and Susumu Kunifuji. 2005. A Support System for Finding Lost Objects Using Spotlight. In Proceedings of the 7th International Conference on Human Computer Interaction with Mobile Devices & Services(Salzburg, Austria) (MobileHCI ’05). 321–322. https://doi.org/10.1145/1085777.1085846Google ScholarDigital Library
- Niklas Osmers and Michael Prilla. 2020. Getting out of Out of Sight: Evaluation of AR Mechanisms for Awareness and Orientation Support in Occluded Multi-Room Settings. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). 1–11. https://doi.org/10.1145/3313831.3376742Google ScholarDigital Library
- Ling Pei, Ruizhi Chen, Jingbin Liu, Tomi Tenhunen, Heidi Kuusniemi, and Yuwei Chen. 2010. Inquiry-Based Bluetooth Indoor Positioning via RSSI Probability Distributions. In Proceedings of the 2nd International Conference on Advances in Satellite and Space Communications (Athens/Glyfada, Greece) (SPACOMM ’10). 151–156. https://doi.org/10.1109/SPACOMM.2010.18Google ScholarDigital Library
- Bradley J. Rhodes. 1997. The Wearable Remembrance Agent: A System for Augmented Memory. Personal Technologies 1, 4 (1997), 218–224. https://doi.org/10.1007/bf01682024Google ScholarCross Ref
- David Schwarz, Max Schwarz, Jörg Stückler, and Sven Behnke. 2015. Cosero, Find My Keys! Object Localization and Retrieval Using Bluetooth Low Energy Tags. In Proceedings of the RoboCup 2014: Robot World Cup XVIII. 195–206. https://doi.org/10.1007/978-3-319-18615-3_16Google ScholarCross Ref
- Thad Starner, Steve Mann, Brad Rhodes, Je rey Levine, Jennifer Healey, Dana Kirsch, Rosalind W. Picard, and Alex Pentland. 1997. Augmented Reality through Wearable Computing. Presence: Teleoperators & Virtual Environments 6 (1997), 386–398. https://doi.org/10.1162/pres.1997.6.4.386Google ScholarDigital Library
- Riku Suomela and Juha Lehikoinen. 2004. Taxonomy for Visualizing Location-based Information. Virtual Reality 8, 2 (2004), 71–82. https://doi.org/10.1007/s10055-004-0139-8Google ScholarCross Ref
- Masaya Tanbo, Ryoma Nojiri, Yuusuke Kawakita, and Haruhisa Ichikawa. 2017. Active RFID Attached Object Clustering Method with New Evaluation Criterion for Finding Lost Objects. Mobile Information Systems 2017, Article 3637814(2017), 12 pages. https://doi.org/10.1155/2017/3637814Google Scholar
- Zachary Teed and Jia Deng. 2020. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In Proceedings of the 16th European Conference on Computer Vision(ECCV ’20). 402–419. https://doi.org/10.1007/978-3-030-58536-5_24Google ScholarDigital Library
- Takahiro Ueoka, Tatsuyuki Kawamura, Yasuyuki Kono, and Masatsugu Kidode. 2003. I’m Here!: A Wearable Object Remembrance Support System. In Proceedings of the 5th Human-Computer Interaction with Mobile Devices and Services (Udine, Italy) (Mobile HCI ’03). 422–427. https://doi.org/10.1007/978-3-540-45233-1_40Google ScholarCross Ref
- Paul Wilson, Daniel Prashanth, and Hamid Aghajan. 2007. Utilizing RFID Signaling Scheme for Localization of Stationary Objects and Speed Estimation of Mobile Objects. In Proceedings of the 2007 IEEE international conference on RFID (Grapevine, TX, USA) (IEEE RFID ’07). 94–99. https://doi.org/10.1109/RFID.2007.346155Google ScholarCross Ref
- Erwin Wu, Ye Yuan, Hui-Shyong Yeo, Aaron Quigley, Hideki Koike, and Kris M. Kitani. 2020. Back-Hand-Pose: 3D Hand Pose Estimation for a Wrist-Worn Camera via Dorsum Deformation Network. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). 1147–1160. https://doi.org/10.1145/3379337.3415897Google ScholarDigital Library
- Takuma Yagi, Md Tasnimul Hasan, and Yoichi Sato. 2021. Hand-Object Contact Prediction via Motion-Based Pseudo-Labeling and Guided Progressive Label Correction. In Proceedings of the 32nd British Machine Vision Conference (Online) (BMVC ’21). 1–14.Google Scholar
- Takuma Yagi, Takumi Nishiyasu, Kunimasa Kawasaki, Moe Matsuki, and Yoichi Sato. 2021. GO-Finder: A Registration-Free Wearable System for Assisting Users in Finding Lost Objects via Hand-Held Object Discovery. In Proceedings of the 26th International Conference on Intelligent User Interfaces (College Station, TX, USA) (IUI ’21). 139–149. https://doi.org/10.1145/3397481.3450664Google ScholarDigital Library
- Ge Yan, Chao Zhang, Jiadi Wang, Zheng Xu, Jianhui Liu, Jintao Nie, Fangtian Ying, and Cheng Yao. 2022. CamFi: An AI-Driven and Camera-Based System for Assisting Users in Finding Lost Objects in Multi-Person Scenarios. In Proceedings of the Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Article 457, 7 pages. https://doi.org/10.1145/3491101.3519780Google ScholarDigital Library
Index Terms
- LocatAR: An AR Object Search Assistance System for a Shared Space
Recommendations
GO-Finder: A Registration-Free Wearable System for Assisting Users in Finding Lost Objects via Hand-Held Object Discovery
IUI '21: Proceedings of the 26th International Conference on Intelligent User InterfacesPeople spend an enormous amount of time and effort looking for lost objects. To help remind people of the location of lost objects, various computational systems that provide information on their locations have been developed. However, prior systems for ...
GO-Finder: A Registration-free Wearable System for Assisting Users in Finding Lost Hand-held Objects
People spend an enormous amount of time and effort looking for lost objects. To help remind people of the location of lost objects, various computational systems that provide information on their locations have been developed. However, prior systems for ...
AR-CF: Augmenting Virtual Users and Items in Collaborative Filtering for Addressing Cold-Start Problems
SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information RetrievalCold-start problems are arguably the biggest challenges faced by collaborative filtering (CF) used in recommender systems. When few ratings are available, CF models typically fail to provide satisfactory recommendations for cold-start users or to ...
Comments