Abstract
Input latency adversely affects users experience when they interact with touchscreen devices, especially, when they perform such pointing tasks as writing or drawing with a stylus. To cope with this problem, we capitalize on the deep-learning latency compensation approach, which is considered effective for now, and propose GRU-CNN architecture that enables more accurate prediction of stylus nib future position based on the sequence of the latest input events. To improve prediction accuracy, we minimize the value of custom loss estimating not only distance but direction proximity of actual touches to predicted stylus positions. Additional usage of real-pen specific features generated with an active stylus (tilt, orientation, and pressure values) is also aimed at accuracy improvement. Experiments reveal that the models with proposed GRU-CNN architecture give 0.07, 0.24, and 0.47 mm of prediction error, which are 9.4, 5.3, and 3.8 times lower than the state-of-the-art LSTM-based ones have in cases of prediction in ~16.6, ~33.3, and 50 ms. The proposed solution provides low-latency interaction in real-time (about 4 ms on Galaxy Note 9) with no cost of hardware complexity.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Annett, M., Ng, A., Dietz, P., Bischof, W.F., Gupta. A.: How low should we go?: understanding the perception of latency while inking. In: Proceedings of Graphics Interface 2014, pp. 167–174. Canadian Information Processing Society, Toronto (2014)
Deber, J., Jota, R., Forlines, C., Wigdor, D.: How much faster is fast enough?: user perception of latency & latency improvements in direct and indirect touch. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1827–1836. ACM, New York (2015). https://doi.org/10.1145/2702123.2702300
Ng, A., Lepinski, J., Wigdor, D., Sanders, S., Dietz, P.: Designing for low-latency direct-touch input. In: Proceedings of the 25th annual ACM symposium on User Interface Software and Technology, pp. 453–464. ACM, New York (2012). https://doi.org/10.1145/2380116.2380174
Yun, M.H., He, S., Zhong, L.: POLYPATH: supporting multiple tradeoffs for interaction latency (2016). https://arxiv.org/pdf/1608.05654.pdf. Accessed 11 Mar 2020
Cattan, E., Rochet-Capellan, A., Bérard, F.: A predictive approach for an end-to-end touch-latency measurement. In: Proceedings of the 2015 International Conference on Interactive Tabletops and Surfaces, pp. 215–218. ACM, New York (2015). doi: https://doi.org/10.1145/2817721.2817747
Nancel, M., Vogel, D., Araujo, B., Jota, R., Casiez, G.: Next-point prediction metrics for perceived spatial errors. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pp. 271–285. ACM, New York (2016). doi: https://doi.org/10.1145/2984511.2984590
Nancel, M., et al.: Next-point prediction for direct touch using finite-time derivative estimation. In: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, pp. 793–807. ACM, New York (2018). doi: https://doi.org/10.1145/3242587.3242646
Henze, N., Funk, M., Shirazi, A.S.: Software-reduced touchscreen latency. In: Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 434–441. ACM, New York (2016). https://doi.org/10.1145/2935334.2935381
Henze, N., Mayer, S., Le, H.V., Schwind, V.: Improving software-reduced touchscreen latency. In: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, vol. 107. ACM, New York (2017). https://doi.org/10.1145/3098279.3122150
Cho, K., Merrienboer, B., Bahdanau, D., Bengio Y.: On the properties of neural machine translation: encoder–decoder approaches. In: Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103–111. Association for Computational Linguistics, Stroudsburg (2014). https://doi.org/10.3115/v1/w14-4012
Wang, Z., Yan, W., Oates, T.: Time series classification from scratch with deep neural networks: a strong baseline. In: 2017 International Joint Conference on Neural Networks, pp. 1578–1585. IEEE, Anchorage (2017). https://doi.org/10.1109/ijcnn.2017.7966039
Rumetshofer, E., Hofmarcher, M., Röhrl, C., Hochreiter, S., Klambauer, G.: Human-level protein localization with convolutional neural networks (2019). https://openreview.net/forum?id=ryl5khRcKm. Accessed 11 Mar 2020
Scherer, D., Müller, A., Behnke, S.: Evaluation of pooling operations in convolutional architectures for object recognition. In: Diamantaras, K., Duch, W., Iliadis, L.S. (eds.) ICANN 2010. LNCS, vol. 6354, pp. 92–101. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15825-4_10
LaViola, J.J.: Double exponential smoothing: an alternative to Kalman filter-based predictive tracking. In: Proceedings of the workshop on Virtual environments 2003, pp. 199–206. Association for Computing Machinery, New York (2003). https://doi.org/10.1145/769953.769976
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Kushnirenko, R., Alkhimova, S., Sydorenko, D., Tolmachov, I. (2020). Active Stylus Input Latency Compensation on Touch Screen Mobile Devices. In: Stephanidis, C., Antona, M. (eds) HCI International 2020 - Posters. HCII 2020. Communications in Computer and Information Science, vol 1224. Springer, Cham. https://doi.org/10.1007/978-3-030-50726-8_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-50726-8_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-50725-1
Online ISBN: 978-3-030-50726-8
eBook Packages: Computer ScienceComputer Science (R0)