Abstract
Research has shown that the location of touch screen taps on modern smartphones and tablet computers can be identified based on sensor recordings from the device’s accelerometer and gyroscope. This security threat implies that an attacker could launch a background process on the mobile device and send the motion sensor readings to a third party vendor for further analysis. Even though the location inference is a non-trivial task requiring machine learning algorithms in order to predict the tap location, previous research was able to show that PINs and passwords of users could be successfully obtained. However, as the tap location inference was only shown for taps generated in a controlled setting not reflecting the environment users naturally engage with their smartphones, the attempts in this paper bridge this gap. We propose TapSensing, a data acquisition system designed to collect touch screen tap event information with corresponding accelerometer and gyroscope readings. Having performed a data acquisition study with 27 participants and 3 different iPhone models, a total of 25,000 labeled taps could be acquired from a laboratory and field environment enabling a direct comparison of both settings. The overall findings show that tap location inference is generally possible for data acquired in the field, hence, with a performance reduction of approximately 20% when comparing both environments. As the tap inference has therefore been shown for a more realistic data set, this work shows that smartphone motion sensors could potentially be used to comprise the user’s privacy in any surrounding user’s interact with the devices.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
National Security Agency.
References
Asonov, D., Agrawal, R.: Keyboard acoustic emanations. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 3–11, May 2004. https://doi.org/10.1109/SECPRI.2004.1301311
Backes, M., Chen, T., Duermuth, M., Lensch, H.P.A., Welk, M.: Tempest in a teapot: compromising reflections revisited. In: 2009 30th IEEE Symposium on Security and Privacy, pp. 315–327, May 2009. https://doi.org/10.1109/SP.2009.20
Backes, M., Dürmuth, M., Unruh, D.: Compromising reflections-or-how to read LCD monitors around the corner. In: 2008 IEEE Symposium on Security and Privacy (SP 2008), pp. 158–169, May 2008. https://doi.org/10.1109/SP.2008.25
Backes, M., Dürmuth, M., Gerling, S., Pinkal, M., Sporleder, C.: Acoustic side-channel attacks on printers. In: Proceedings of the 19th USENIX Conference on Security, USENIX Security 2010, p. 20. USENIX Association, Berkeley (2010). http://dl.acm.org/citation.cfm?id=1929820.1929847
Cai, L., Chen, H.: Touchlogger: inferring keystrokes on touch screen from smartphone motion. In: Proceedings of the 6th USENIX Conference on Hot Topics in Security, HotSec 2011, p. 9. USENIX Association, Berkeley (2011). http://dl.acm.org/citation.cfm?id=2028040.2028049
Case, M.A., Burwick, H.A., Volpp, K.G., Patel, M.S.: Accuracy of smartphone applications and wearable devices for tracking physical activity data. JAMA 313(6), 625–626 (2015)
van Eck, W.: Electromagnetic radiation from video display units: an eavesdropping risk? Comput. Secur. 4(4), 269–286 (1985). https://doi.org/10.1016/0167-4048(85)90046-X
Feijoo, C., Gómez-Barroso, J.L., Aguado, J.M., Ramos, S.: Mobile gaming: industry challenges and policy implications. Telecommun. Policy 36(3), 212–221 (2012)
James, G., Witten, D., Hastie, T., Tibshirani, R.: An Introduction to Statistical Learning: With Applications in R. Springer, New York (2014). https://doi.org/10.1007/978-1-4614-7138-7
Kuhn, M.G.: Optical time-domain eavesdropping risks of CRT displays. In: Proceedings 2002 IEEE Symposium on Security and Privacy, pp. 3–18 (2002). https://doi.org/10.1109/SECPRI.2002.1004358
Kuhn, M.G.: Electromagnetic eavesdropping risks of flat-panel displays. In: Martin, D., Serjantov, A. (eds.) PET 2004. LNCS, vol. 3424, pp. 88–107. Springer, Heidelberg (2005). https://doi.org/10.1007/11423409_7
Lane, N.D., Miluzzo, E., Lu, H., Peebles, D., Choudhury, T., Campbell, A.T.: A survey of mobile phone sensing. IEEE Commun. Mag. 48(9), 140–150 (2010). https://doi.org/10.1109/MCOM.2010.5560598
Link, J.A.B., Smith, P., Viol, N., Wehrle, K.: Footpath: accurate map-based indoor navigation using smartphones. In: 2011 International Conference on Indoor Positioning and Indoor Navigation (IPIN), pp. 1–8. IEEE (2011)
Marquardt, P., Verma, A., Carter, H., Traynor, P.: (sp)iphone: decoding vibrations from nearby keyboards using mobile phone accelerometers. In: Proceedings of the 18th ACM Conference on Computer and Communications Security, CCS 2011, pp. 551–562. ACM, New York (2011). https://doi.org/10.1145/2046707.2046771. http://doi.acm.org/10.1145/2046707.2046771
Miluzzo, E., Varshavsky, A., Balakrishnan, S., Choudhury, R.R.: Tapprints: your finger taps have fingerprints. In: Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services, MobiSys 2012, pp. 323–336. ACM, New York (2012). https://doi.org/10.1145/2307636.2307666. http://doi.acm.org/10.1145/2307636.2307666
Nalty, B.C.: The war against trucks aerial interdiction in southern laos 1968–1972. Technical report, Office of Air Force History Washington DC (2005)
NATO: Tempest equipment selection process. http://www.ia.nato.int/niapc/tempest/certification-scheme
Nowosielski, L., Wnuk, M.: Compromising emanations from USB 2 interface. In: PIERS Proceedings (2014)
NSA: Tempest: a signal problem (2007). https://www.nsa.gov/news-features/declassified-documents/cryptologic-spectrum/assets/files/tempest.pdf
Owusu, E., Han, J., Das, S., Perrig, A., Zhang, J.: Accessory: password inference using accelerometers on smartphones. In: Proceedings of the Twelfth Workshop on Mobile Computing Systems & #38; Applications, HotMobile 2012, pp. 9:1–9:6. ACM, New York (2012). https://doi.org/10.1145/2162081.2162095. http://doi.acm.org/10.1145/2162081.2162095
Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011). http://dl.acm.org/citation.cfm?id=1953048.2078195
Przesmycki, R.: Measurement and analysis of compromising emanation for laser printer. In: PIERS Proceedings, pp. 2661–2665 (2014)
Quisquater, J.-J., Samyde, D.: ElectroMagnetic analysis (EMA): measures and counter-measures for smart cards. In: Attali, I., Jensen, T. (eds.) E-smart 2001. LNCS, vol. 2140, pp. 200–210. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-45418-7_17. http://dl.acm.org/citation.cfm?id=646803.705980
Smulders, P.: The threat of information theft by reception of electromagnetic radiation from RS232 cables. Comput. Secur. 9, 53–58 (1990)
Vuagnoux, M., Pasini, S.: Compromising electromagnetic emanations of wired and wireless keyboards. In: Proceedings of the 18th Conference on USENIX Security Symposium, SSYM 2009, pp. 1–16. USENIX Association, Berkeley (2009). http://dl.acm.org/citation.cfm?id=1855768.1855769
Zeng, M., et al.: Convolutional neural networks for human activity recognition using mobile sensors. In: 2014 6th International Conference on Mobile Computing, Applications and Services (MobiCASE), pp. 197–205. IEEE (2014)
Zhuang, L., Zhou, F., Tygar, J.D.: Keyboard acoustic emanations revisited. ACM Trans. Inf. Syst. Secur. 13(1), 3:1–3:26 (2009). https://doi.org/10.1145/1609956.1609959. http://doi.acm.org/10.1145/1609956.1609959
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Schmitt, E., Voigt-Antons, JN. (2020). Predicting Tap Locations on Touch Screens in the Field Using Accelerometer and Gyroscope Sensor Readings. In: Moallem, A. (eds) HCI for Cybersecurity, Privacy and Trust. HCII 2020. Lecture Notes in Computer Science(), vol 12210. Springer, Cham. https://doi.org/10.1007/978-3-030-50309-3_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-50309-3_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-50308-6
Online ISBN: 978-3-030-50309-3
eBook Packages: Computer ScienceComputer Science (R0)