Skip to main content
Springer Nature Link
Log in

Studying Human Factors Aspects of Text Classification Task Using Eye Tracking

  • Conference paper
  • First Online:
Augmented Cognition (HCII 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14019))

Included in the following conference series:

  • 1127 Accesses

Abstract

Text classification has a wide range of applications in today’s world including filtering spam emails, identifying health conditions, categorizing news articles, business intelligence, and finding relevant legal documents. This has become scalable due to the use of supervised machine learning models which are usually trained on manually labelled text data and their performance is heavily dependent on the quality of training data. Manual text classification tasks involve a person reading the text and assigning the most appropriate category, which can incur a significant amount of cognitive load. Therefore, an in-depth understanding of human factors aspects of the text classification task is important, and it can help in determining the expected level of accuracy of human-labelled text as well as identifying the challenging aspects of the task. To the best of our knowledge, previous studies have not studied the text classification task from a human computer interaction (HCI) and human factors perspective. Our study is an early effort towards studying text classification task using eye-tracking information captured during the manual labelling process. We aim to analyze ocular parameters to understand the manual text classification process from an HCI perspective. We designed an eye-tracking study that involved 30 human subjects reading narratives of injury-related texts and selecting the best-suited category for the cause of injury events. Ocular parameters such as fixation count, average fixation duration, and pupil dilation values were recorded for each participant. Preliminary results from our study indicate that (a) reasonable level of average classification accuracy (75%) was observed for study participants, (b) a positive correlation between fixation count and fixation duration, and fixation count and pupil diameter was observed, and (c) we did not observe a consistent pattern between ocular parameters representative of cognitive load, the time taken to complete the task, and the classification accuracy, maybe due to underlying variations among humans and interpretability of textual narratives.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Sebastiani, F.: Machine learning in automated text categorization. ACM Comput. Surv. 34(1), 1–47 (2002)

    Article  MathSciNet  Google Scholar 

  2. Li, Q., et al.: A survey on text classification: from traditional to deep learning. ACM Trans. Intell. Syst. Technol. 13(2), 1–41 (2022)

    Google Scholar 

  3. Karim, A., Azam, S., Shanmugam, B., Kannoorpatti, K., Alazab, M.: A comprehensive survey for intelligent spam email detection. IEEE Access 7, 168261–168295 (2019)

    Article  Google Scholar 

  4. Castano, S., Falduti, M., Ferrara, A., Montanelli, S.: A knowledge-centered framework for exploration and retrieval of legal documents. Inf. Syst. 106, 101842 (2022)

    Article  Google Scholar 

  5. Barberá, P., Boydstun, A.E., Linn, S., McMahon, R., Nagler, J.: Automated text classification of news articles: a practical guide. Polit. Anal. 29(1), 19–42 (2021)

    Article  Google Scholar 

  6. Hughes, M., Li, I., Kotoulas, S., Suzumura, T.: Medical text classification using convolutional neural networks. In: Informatics for Health: Connected Citizen-Led Wellness and Population Health, pp. 246–250. IOS Press (2017)

    Google Scholar 

  7. Nanda, G., Vallmuur, K., Lehto, M.: Semi-automated text mining strategies for identifying rare causes of injuries from emergency room triage data. IISE Trans. Healthc. Syst. Eng. 9(2), 157–171 (2019)

    Article  Google Scholar 

  8. Jain, A., et al.: Overview and importance of data quality for machine learning tasks. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 3561–3562 (2020)

    Google Scholar 

  9. Gupta, N., et al.: Data quality for machine learning tasks. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pp. 4040–4041 (2021)

    Google Scholar 

  10. Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 614–622 (2008)

    Google Scholar 

  11. Paun, S., Artstein, R., Poesio, M.: Statistical methods for annotation analysis. Synth. Lect. Hum. Lang. Technol. 15(1), 1–217 (2022)

    Article  Google Scholar 

  12. Nanda, G.: Improving the autocoding of injury narratives using a combination of machine learning methods and natural language processing techniques. Doctoral dissertation, Purdue University (2017)

    Google Scholar 

  13. Nanda, G., Vallmuur, K., Lehto, M.: Improving autocoding performance of rare categories in injury classification: is more training data or filtering the solution? Accid. Anal. Prev. 110, 115–127 (2018)

    Article  Google Scholar 

  14. Sen, C., Hartvigsen, T., Yin, B., Kong, X., Rundensteiner, E.: Human attention maps for text classification: do humans and neural networks focus on the same words? In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4596–4608 (2020)

    Google Scholar 

  15. Nguyen, D.: Comparing automatic and human evaluation of local explanations for text classification. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1 (Long Papers), pp. 1069–1078 (2018)

    Google Scholar 

  16. Wikimedia Foundation: Eye tracking. https://en.wikipedia.org/wiki/Eye_tracking

  17. Jiang, J., Zhou, X., Chan, S., Chen, S.: Appearance-based gaze tracking: a brief review. In: Yu, H., Liu, J., Liu, L., Ju, Z., Liu, Y., Zhou, D. (eds.) ICIRA 2019. LNCS (LNAI), vol. 11745, pp. 629–640. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27529-7_53

    Chapter  Google Scholar 

  18. Kooiker, M.J., Pel, J.J., van der Steen-Kant, S.P., van der Steen, J.: A method to quantify visual information processing in children using eye tracking. J. Vis. Exp. 113, e54031 (2016)

    Google Scholar 

  19. Babu, M.D., JeevithaShree, D.V., Prabhakar, G., Saluja, K.P.S., Pashilkar, A., Biswas, P.: Estimating pilots’ cognitive load from ocular parameters through simulation and in-flight studies. J. Eye Mov. Res. 12(3) (2019)

    Google Scholar 

  20. King, A.J., Bol, N., Cummins, R.G., John, K.K.: Improving visual behavior research in communication science: an overview, review, and reporting recommendations for using eye-tracking methods. Commun. Methods Meas. 13(3), 149–177 (2019)

    Article  Google Scholar 

  21. Li, J., et al.: Identification and classification of construction equipment operators’ mental fatigue using wearable eye-tracking technology. Autom. Constr. 109, 103000 (2020)

    Article  Google Scholar 

  22. Rello, L., Ballesteros, M.: Detecting readers with dyslexia using machine learning with eye tracking measures. In: Proceedings of the 12th International Web for All Conference, pp. 1–8 (2015)

    Google Scholar 

  23. Saluja, K.S., Dv, J., Arjun, S., Biswas, P., Paul, T.: Analyzing eye gaze of users with learning disability. In: Proceedings of the 3rd International Conference on Graphics and Signal Processing, pp. 95–99 (2019)

    Google Scholar 

  24. Tzafilkou, K., Protogeros, N.: Diagnosing user perception and acceptance using eye tracking in web-based end-user development. Comput. Hum. Behav. 72, 23–37 (2017)

    Article  Google Scholar 

  25. Zagermann, J., Pfeil, U., Reiterer, H.: Measuring cognitive load using eye tracking technology in visual computing. In: Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization, pp. 78–85 (2016)

    Google Scholar 

  26. Kruger, J.L., Hefer, E., Matthew, G.: Measuring the impact of subtitles on cognitive load: eye tracking and dynamic audiovisual texts. In: Proceedings of the 2013 Conference on Eye Tracking South Africa, pp. 62–66 (2013)

    Google Scholar 

  27. Raney, G.E., Campbell, S.J., Bovee, J.C.: Using eye movements to evaluate the cognitive processes involved in text comprehension. J. Vis. Exp. 83, e50780 (2014)

    Google Scholar 

  28. Tomanek, K., Hahn, U., Lohmann, S., Ziegler, J.: A cognitive cost model of annotations based on eye-tracking data. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 1158–1167 (2010)

    Google Scholar 

  29. Mishra, A., Bhattacharyya, P.: Scanpath complexity: modeling reading/annotation effort using gaze information. In: Cognitively Inspired Natural Language Processing. CIR, pp. 77–98. Springer, Singapore (2018). https://doi.org/10.1007/978-981-13-1516-9_4

    Chapter  Google Scholar 

  30. Joshi, A., Mishra, A., Senthamilselvan, N., Bhattacharyya, P.: Measuring sentiment annotation complexity of text. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, vol. 2 (Short Papers), pp. 36–41 (2014)

    Google Scholar 

  31. Hart, S.G.: NASA-task load index (NASA-TLX); 20 years later. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 50, no. 9, pp. 904–908. Sage Publications, Los Angeles, CA (2006)

    Google Scholar 

  32. Catchpoole, J., Nanda, G., Vallmuur, K., Nand, G., Lehto, M.: Application of a machine learning–based decision support tool to improve an injury surveillance system workflow. Appl. Clin. Inform. 13(03), 700–710 (2022)

    Article  Google Scholar 

  33. Salvucci, D.D., Goldberg, J.H.: Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the 2000 Symposium on Eye Tracking Research Applications, pp. 71–78 (2000)

    Google Scholar 

  34. Olsen, A.: The Tobii I-VT fixation filter. Tobii Technol. 21, 4–19 (2012)

    Google Scholar 

  35. Will, T.: Measuring and interpreting system usability scale (SUS) (2021). https://uiuxtrend.com/measuring-system-usability-scale-sus/

  36. Chen, W., Sawaragi, T., Hiraoka, T.: Comparing eye-tracking metrics of mental workload caused by NDRTs in semi-autonomous driving. Transp. Res. F Traffic Psychol. Behav. 89, 109–128 (2022)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Engineering Technology, Purdue University, West Lafayette, IN, 47907, USA

    Jeevithashree Divya Venkatesh, Meet Tusharbhai Suthar & Gaurav Nanda

  2. Center for Intercultural Learning, Mentorship, Assessment and Research (CILMAR), Purdue University, West Lafayette, IN, 47907, USA

    Aparajita Jaiswal

  3. Department of Computer and Information Technology, Purdue University, West Lafayette, IN, 47907, USA

    Romila Pradhan

Authors
  1. Jeevithashree Divya Venkatesh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aparajita Jaiswal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meet Tusharbhai Suthar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Romila Pradhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gaurav Nanda
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gaurav Nanda .

Editor information

Editors and Affiliations

  1. Soar Technology Inc., Orlando, FL, USA

    Dylan D. Schmorrow

  2. Katmai Government Services, Orlando, FL, USA

    Cali M. Fidopiastis

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Divya Venkatesh, J., Jaiswal, A., Suthar, M.T., Pradhan, R., Nanda, G. (2023). Studying Human Factors Aspects of Text Classification Task Using Eye Tracking. In: Schmorrow, D.D., Fidopiastis, C.M. (eds) Augmented Cognition. HCII 2023. Lecture Notes in Computer Science(), vol 14019. Springer, Cham. https://doi.org/10.1007/978-3-031-35017-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-35017-7_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-35016-0

  • Online ISBN: 978-3-031-35017-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics