skip to main content
10.1145/2742647.2742662acmconferencesArticle/Chapter ViewAbstractPublication PagesmobisysConference Proceedingsconference-collections
research-article

Turning a Mobile Device into a Mouse in the Air

Published: 18 May 2015 Publication History

Abstract

A mouse has been one of the most successful user interfaces due to its intuitive use. As more devices are equipped with displays and offer rich options for users to choose from, a traditional mouse that requires a surface to operate is no longer sufficient. While different types of air mice are available in the market, they rely on accelerometers and gyroscopes, which significantly limit the accuracy and ease of use.
In this paper, we develop a system that can accurately track hand movement to realize a mouse. A unique advantage of our scheme is that it achieves high tracking accuracy using the existing hardware already available in the mobile devices (e.g., smart phones and smart watches) and equipment to be controlled (e.g., smart TVs). More specifically, our approach sends inaudible sound pulses at a few selected frequencies, and uses the frequency shifts to estimate the speed and distance traveled. We then develop techniques to quickly calibrate the distance between speakers and narrow down the device's initial position using its movement trajectory. Based on the information, we continuously track the device's new position in real time. This is feasible because many devices, such as smart TVs, PCs, and laptops, already have multiple speakers. When only one speaker is available, we can leverage the frequency shift of sound along with the phase of received WiFi signal to enable tracking. Our evaluation and user study demonstrate that our system achieves high tracking accuracy (e.g., median error of around 1.4 cm) and ease of use.

References

[1]
Microsoft X-box Kinect. http://xbox.com.
[2]
Nintendo Wii. http://www.nintendo.com/wii.
[3]
Circle-circle intersection. http://mathworld.wolfram.com/Circle-CircleIntersection.html.
[4]
S. Agrawal, I. Constandache, S. Gaonkar, R. Roy Choudhury, K. Caves, and F. DeRuyter. Using mobile phones to write in air. In Proc. of ACM MobiSys, pages 15--28, 2011.
[5]
Logitech, air mouse. http://www.logitech.com.
[6]
M. T. I. Aumi, S. Gupta, M. Goel, E. Larson, and S. Patel. Doplink: Using the doppler effect for multi-device interaction. In Proc. of ACM UbiComp, pages 583--586, 2013.
[7]
G. Bradski. The OpenCV library. Doctor Dobbs Journal, 25(11):120--126, 2000.
[8]
K.-Y. Chen, D. Ashbrook, M. Goel, S.-H. Lee, and S. Patel. Airlink: Sharing files between multiple devices using in-air gestures. In Proc. of ACM Ubicomp, 2014.
[9]
F. Dellaert, D. Fox, W. Burgard, and S. Thrun. Monte carlo localization for mobile robots. In Proc. of Robotics and Automation, volume 2, pages 1322--1328, 1999.
[10]
Frame rate. http://en.wikipedia.org/wiki/Frame_rate.
[11]
D. Goehl and D. Sachs. Motion sensors gaining inertia with popular consumer electronics. White Paper, IvenSense Inc, 2007.
[12]
S. Gupta, D. Morris, S. Patel, and D. Tan. Soundwave: using the doppler effect to sense gestures. In Proc. of the SIGCHI, pages 1911--1914, 2012.
[13]
R. Heydon and N. Hunn. Bluetooth low energy. CSR Presentation, Bluetooth SIG https://www.bluetooth.org/DocMan/handlers/DownloadDoc. ashx, 2012.
[14]
W. Huang, Y. Xiong, X.-Y. Li, H. Lin, X. Mao, P. Yang, and Y. Liu. Shake and walk: Acoustic direction finding and fine-grained indoor localization using smartphones. In Proc. of IEEE INFOCOM, 2014.
[15]
InvenSense. http://www.invensense.com.
[16]
E. Jacobsen and R. Lyons. The sliding DFT. IEEE Signal Processing Magazine, 20(2):74--80, 2003.
[17]
S. Kumar, D. Cifuentes, S. Gollakota, and D. Katabi. Bringing cross-layer MIMO to today's wireless lans. In In Proc. of ACM SIGCOMM, August 2013.
[18]
S. Kumar, S. Gil, D. Katabi, and D. Rus. Accurate indoor localization with zero start-up cost. In Proc. of ACM MobiCom, pages 483--494, 2014.
[19]
P. Lazik and A. Rowe. Indoor pseudo-ranging of mobile devices using ultrasonic chirps. In Proc. of ACM SenSys, pages 99--112, 2012.
[20]
C.-H. Lee, J.-L. Shih, K.-M. Yu, and H.-S. Lin. Automatic music genre classification based on modulation spectral analysis of spectral and cepstral features. IEEE Transactions on Multimedia, 11(4):670--682, 2009.
[21]
LG magic motion remote. http://www.lg.com/us/tv-accessories/lg-AN-MR200-motion-remote.
[22]
F. Li, C. Zhao, G. Ding, J. Gong, C. Liu, and F. Zhao. A reliable and accurate indoor localization method using phone inertial sensors. In Proc. of UbiComp, pages 421--430, 2012.
[23]
Apple 13-inch MacBook Pro technical specifications. https://www.apple.com/macbook-pro/specs/.
[24]
R. A. Meyers. Encyclopedia of physical science and technology. Facts on File, 1987.
[25]
Top EDM 2014 music playlist tracklist. https://www.youtube.com/watch?v=PHIRcu3Ero0.
[26]
H. Nyqvist and F. Gustafsson. A high-performance tracking system based on camera and IMU. In Proc. of 16th IEEE International Conference on Information Fusion (FUSION), pages 2065--2072, 2013.
[27]
A. V. Oppenheim, R. W. Schafer, J. R. Buck, et al. Discrete-time signal processing, volume 2. Prentice-hall Englewood Cliffs, 1989.
[28]
T. Park, J. Lee, I. Hwang, C. Yoo, L. Nachman, and J. Song. E-gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices. In Proc. of ACM SenSys, pages 260--273, 2011.
[29]
C. Peng, G. Shen, Y. Zhang, Y. Li, and K. Tan. Beepbeep: A high accuracy acoustic ranging system using cots mobile devices. In Proc. of ACM SenSys, 2007.
[30]
Q. Pu, S. Gupta, S. Gollakota, and S. Patel. Whole-home gesture recognition using wireless signals. In Proc. of ACM MobiCom, 2013.
[31]
J. Qiu, D. Chu, X. Meng, and T. Moscibroda. On the feasibility of real-time phone-to-phone 3D localization. In Proc. of ACM MobiSys, 2011.
[32]
H. S. Rahul, S. Kumar, and D. Katabi. MegaMIMO: scaling wireless capacity with user demands. In Proc. of ACM SIGCOMM, pages 235--246, 2012.
[33]
A. Rai, K. K. Chintalapudi, V. N. Padmanabhan, and R. Sen. Zee: zero-effort crowdsourcing for indoor localization. In Proc. of ACM MobiCom, 2012.
[34]
Roku 3 streaming player. https://www.roku.com/products/roku-3.
[35]
CES 2014: Samsung shows off new gyroscopic remote control. http://www.digitalversus.com/tv-television/ces-2014-samsung-shows-off-new-gyroscopic-remote-control-n32491.html.
[36]
K. Seifert and O. Camacho. Implementing positioning algorithms using accelerometers. Freescale Semiconductor, 2007.
[37]
CES 2014 trends: New remotes and interfaces to make smart TVs actually usable. http://spectrum.ieee.org/tech-talk/consumer-electronics/audiovideo/ces-2014-trends-getting-smart-tv-under-control.
[38]
A. Smith, H. Balakrishnan, M. Goraczko, and N. Priyantha. Tracking moving devices with the cricket location system. In Proc. of ACM MobiSys, 2005.
[39]
Z. Sun, A. Purohit, R. Bose, and P. Zhang. Spartacus: spatially-aware interaction for mobile devices through energy-efficient audio sensing. In Proc. of ACM Mobisys, pages 263--276, 2013.
[40]
J. Wang and D. Katabi. Dude, where's my card?: RFID positioning that works with multipath and non-line of sight. In Proc. of the ACM SIGCOMM, pages 51--62, 2013.
[41]
J. Wang, D. Vasisht, and D. Katabi. RF-IDraw: virtual touch screen in the air using rf signals. In Proc. of ACM SIGCOMM, 2014.
[42]
P. D. Welch. The use of fast fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Transactions on audio and electroacoustics, 15(2):70--73, 1967.
[43]
C. A. Wingrave, B. Williamson, P. D. Varcholik, J. Rose, A. Miller, E. Charbonneau, J. Bott, and J. LaViola. The wiimote and beyond: Spatially convenient devices for 3d user interfaces. IEEE Computer Graphics and Applications, 30(2):71--85, 2010.
[44]
G. Woo, P. Kheradpour, D. Shen, and D. Katabi. Beyond the bits: Cooperative packet recovery using physical layer information. In Proc. of ACM MobiCom, 2007.
[45]
J. Xiong and K. Jamieson. Arraytrack: A fine-grained indoor location system. In Proc. of NSDI, pages 71--84, 2013.
[46]
L. Yang, Y. Chen, X.-Y. Li, C. Xiao, M. Li, and Y. Liu. Tagoram: Real-time tracking of mobile RFID tags to high precision using cots devices. In Proc. of ACM MobiCom, 2014.
[47]
S. Yun, L. Qiu, and A. Bhartia. Multi-point to multi-point MIMO in wireless LANs. In Proc. of INFOCOM Mini-Conference, April 2013.
[48]
Z. Zhang, D. Chu, X. Chen, and T. Moscibroda. Swordfight: Enabling a new class of phone-to-phone action games on commodity phones. In Proc. of ACM MobiSys, 2012.

Cited By

View all
  • (2024)Visar: Projecting Virtual Sound Spots for Acoustic Augmented Reality Using Air NonlinearityProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785468:3(1-30)Online publication date: 9-Sep-2024
  • (2024)Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic SignalsACM Transactions on Sensor Networks10.1145/367712220:5(1-32)Online publication date: 26-Aug-2024
  • (2024)Locating Your Smart Devices with a Single SpeakerProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699320(28-40)Online publication date: 4-Nov-2024
  • Show More Cited By

Index Terms

  1. Turning a Mobile Device into a Mouse in the Air

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      MobiSys '15: Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services
      May 2015
      516 pages
      ISBN:9781450334945
      DOI:10.1145/2742647
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      In-Cooperation

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 18 May 2015

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. accelerometer
      2. doppler effect
      3. gyroscope
      4. tracking

      Qualifiers

      • Research-article

      Conference

      MobiSys'15
      Sponsor:

      Acceptance Rates

      MobiSys '15 Paper Acceptance Rate 29 of 219 submissions, 13%;
      Overall Acceptance Rate 274 of 1,679 submissions, 16%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)122
      • Downloads (Last 6 weeks)14
      Reflects downloads up to 17 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Visar: Projecting Virtual Sound Spots for Acoustic Augmented Reality Using Air NonlinearityProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785468:3(1-30)Online publication date: 9-Sep-2024
      • (2024)Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic SignalsACM Transactions on Sensor Networks10.1145/367712220:5(1-32)Online publication date: 26-Aug-2024
      • (2024)Locating Your Smart Devices with a Single SpeakerProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699320(28-40)Online publication date: 4-Nov-2024
      • (2024)DisMouse: Disentangling Information from Mouse Movement DataProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676411(1-13)Online publication date: 13-Oct-2024
      • (2024)FusionTrack: Towards Accurate Device-free Acoustic Motion Tracking with Signal FusionACM Transactions on Sensor Networks10.1145/365466620:3(1-30)Online publication date: 30-Mar-2024
      • (2024)Room-scale Location Trace Tracking via Continuous Acoustic WavesACM Transactions on Sensor Networks10.1145/364913620:3(1-23)Online publication date: 13-Apr-2024
      • (2024)UTrack3D: 3D Tracking Using Ultra-wideband (UWB) RadiosProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661881(345-358)Online publication date: 3-Jun-2024
      • (2024)Enabling 6D Pose Tracking on Your Acoustic DevicesProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661875(15-28)Online publication date: 3-Jun-2024
      • (2024)Adaptive Metasurface-Based Acoustic Imaging using Joint OptimizationProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661863(492-504)Online publication date: 3-Jun-2024
      • (2024)GPMS: Enabling Indoor GNSS Positioning using Passive MetasurfacesProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3690702(1424-1438)Online publication date: 4-Dec-2024
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      EPUB

      View this article in ePub.

      ePub

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media