Skip to main content

Advertisement

Springer Nature Link
Log in

Investigating human reading behavior during sentiment judgment

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Sentiment analysis is an essential task in natural language processing researches. Although existing works have gained much success with both statistical and neural-based solutions, little is known about the human decision process while performing this kind of complex cognitive task. Considering recent advances in human-inspired model design for NLP tasks, it is necessary to investigate the human reading and judging behavior in sentiment classification and adopt these findings to reconsider the sentiment analysis problem. In this paper, we carefully design a lab-based user study in which users’ fine-grained reading behaviors during microblog sentiment classification are recorded with an eye-track device. Through systematic analysis of the collected data, we look into the differences between human and machine attention distributions and the differences in human attention while performing different tasks. We find that (1) sentiment judgment is more like an auxiliary task of content comprehension for humans. (2) people have different reading behavior patterns while reading microblog posts with varying labels of sentiment. Based on these findings, we build a human behavior-inspired sentiment prediction model for microblog posts. Experiment results on public-available benchmarks show that the proposed classification model outperforms existing solutions over 2.13% in terms of macro F1-score by introducing behavior features. Our findings may bring insight into the research of designing more effective and explainable sentiment analysis methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. https://www.weibo.com/

  2. http://tcci.ccf.org.cn/conference/2013/

  3. http://www.keenage.com/

    http://nlg.csie.ntu.edu.tw/nlpresource/NTUSD-Fin/

    http://nlp.csai.tsinghua.edu.cn/site2/index.php/13-sms

    https://bosonnlp.com/dev/resource

References

  1. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:14090473

  2. Barrett M, Bingel J, Hollenstein N, Rei M, Søgaard A (2018) Sequence classification with human attention. In: Proceedings of the 22nd conference on computational natural language learning, pp 302–312

  3. Berger A, Della Pietra SA, Della Pietra VJ (1996) A maximum entropy approach to natural language processing. Comput Linguist 22(1):39–71

    Google Scholar 

  4. Bicknell K, Levy R (2010) A rational model of eye movement control in reading. In: Proceedings of the 48th annual meeting of the association for computational linguistics, pp 1168–1178

  5. Bolotova V, Blinov V, Zheng Y, Croft WB, Scholer F, Sanderson M (2020) Do people and neural nets pay attention to the same words: studying eye-tracking data for non-factoid qa evaluation. In: Proceedings of the 29th ACM international conference on information & knowledge management, pp 85–94

  6. Craswell N, Zoeter O, Taylor M, Ramsey B (2008) An experimental comparison of click position-bias models. In: Proceedings of the 2008 international conference on web search and data mining, pp 87–94

  7. Engbert R, Nuthmann A, Richter EM, Kliegl R (2005) Swift: a dynamical model of saccade generation during reading. Psychol Rev 112(4):777

    Article  Google Scholar 

  8. Gao S, Alawad M, Young MT, Gounley J, Schaefferkoetter N, Yoon HJ, Wu XC, Durbin EB, Doherty J, Stroup A et al (2021) Limitations of transformers on clinical text classification. IEEE J Biomed Health Inform 25:3596–3607

    Article  Google Scholar 

  9. Granka LA, Joachims T, Gay G (2004) Eye-tracking analysis of user behavior in www search. In: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pp 478–479

  10. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  11. Hosmer DW Jr, Lemeshow S, Sturdivant RX (2013) Applied logistic regression, vol 398. Wiley, Hoboken

    Book  MATH  Google Scholar 

  12. Jain PK, Quamer W, Pamula R, Saravanan V (2021) SpSAN: Sparse self-attentive network-based aspect-aware model for sentiment analysis. J Ambient Intell Humaniz Comput 1–18

  13. Jain S, Wallace BC (2019) Attention is not explanation. arXiv preprint arXiv:190210186

  14. Just MA, Carpenter PA (1987) The psychology of reading and language comprehension. Allyn & Bacon, Boston

    Google Scholar 

  15. Kim Y (2014) Convolutional neural networks for sentence classification. arXiv preprint arXiv:14085882

  16. Klerke S, Goldberg Y, Søgaard A (2016) Improving sentence compression by learning to predict gaze. arXiv preprint arXiv:160403357

  17. Li X, Pollatsek A (2020) An integrated model of word processing and eye-movement control during Chinese reading. Psychol Rev 127(6):1139

    Article  Google Scholar 

  18. Li X, Liu Y, Mao J, He Z, Zhang M, Ma S (2018) Understanding reading attention distribution during relevance judgement. In: Proceedings of the 27th ACM international conference on information and knowledge management, pp 733–742

  19. Liaw A, Wiener M et al (2002) Classification and regression by randomforest. R News 2(3):18–22

    Google Scholar 

  20. Liu KL, Li WJ, Guo M (2012) Emoticon smoothed language models for twitter sentiment analysis. Aaai Citeseer 12:22–26

    Google Scholar 

  21. Liu Y, Chen F, Kong W, Yu H, Zhang M, Ma S, Ru L (2012) Identifying web spam with the wisdom of the crowds. ACM Trans Web TWEB 6(1):1–30

    Article  Google Scholar 

  22. Liu Y, Wang C, Zhou K, Nie J, Zhang M, Ma S (2014) From skimming to reading: a two-stage examination model for web search. In: Proceedings of the 23rd ACM international conference on conference on information and knowledge management, pp 849–858

  23. Liversedge SP, Findlay JM (2000) Saccadic eye movements and cognition. Trends Cognit Sci 4(1):6–14

    Article  Google Scholar 

  24. Maas A, Daly RE, Pham PT, Huang D, Ng AY, Potts C (2011) Learning word vectors for sentiment analysis. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp 142–150

  25. Marimuthu K, Devi SL (2012) How human analyse lexical indicators of sentiments-a cognitive analysis using reaction-time. In: Proceedings of the 2nd workshop on sentiment analysis where AI meets psychology, pp 81–90

  26. McCallum A, Nigam K, et al. (1998) A comparison of event models for Naive Bayes text classification. In: AAAI-98 workshop on learning for text categorization, Citeseer, vol 752, pp 41–48

  27. McDonald SA, Shillcock RC (2003) Low-level predictive inference in reading: the influence of transitional probabilities on eye movements. Vis Res 43(16):1735–1751

    Article  Google Scholar 

  28. Mishra A, Dey K, Bhattacharyya P (2017) Learning cognitive features from gaze data for sentiment and sarcasm classification using convolutional neural network. In: Proceedings of the 55th annual meeting of the association for computational linguistics (Volume 1: Long Papers), pp 377–387

  29. Mishra A, Tamilselvam S, Dasgupta R, Nagar S, Dey K (2018) Cognition-cognizant sentiment analysis with multitask subjectivity summarization based on annotators’ gaze behavior. In: Thirtysecond AAAI conference on artificial intelligence

  30. Osman NA, Mohd Noah SA, Darwich M, Mohd M (2021) Integrating contextual sentiment analysis in collaborative recommender systems. PLoS One 16(3):e0248695

    Article  Google Scholar 

  31. Pang B, Lee L, Vaithyanathan S (2002) Thumbs up? Sentiment classification using machine learning techniques. arXiv preprint cs/0205070

  32. Rayner K (2009) The 35th sir Frederick Bartlett lecture: eye movements and attention in reading, scene perception, and visual search. Q J Exp Psychol 62(8):1457–1506

    Article  Google Scholar 

  33. Reichle ED, Pollatsek A, Fisher DL, Rayner K (1998) Toward a model of eye movement control in reading. Psychol Rev 105(1):125

    Article  Google Scholar 

  34. Reichle ED, Rayner K, Pollatsek A (2003) The ez reader model of eye-movement control in reading: comparisons to other models. Behav Brain Sci 26(4):445

    Article  Google Scholar 

  35. Salmerón L, Delgado P, Mason L (2020) Using eye-movement modelling examples to improve critical reading of multiple webpages on a conflicting topic. J Comput Assisted Learn 36(6):1038–1051

    Article  Google Scholar 

  36. Sen C, Hartvigsen T, Yin B, Kong X, Rundensteiner E (2020) Human attention maps for text classification: Do humans and neural networks focus on the same words? In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 4596–4608

  37. Sergio GC, Lee M (2021) Stacked debert: all attention in incomplete data for text classification. Neural Netw 136:87–96

    Article  Google Scholar 

  38. Suykens JA, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300

    Article  Google Scholar 

  39. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  40. Wiegreffe S, Pinter Y (2019) Attention is not not explanation. arXiv preprint arXiv:190804626

  41. Yadav RK, Jiao L, Granmo OC, Goodwin M (2021) Human-level interpretable learning for aspect-based sentiment analysis. In: The thirty-fifth AAAI conference on artificial intelligence (AAAI-21). AAAI

  42. Yan G, Wang L, Wu J, Bai X (2011) A study on eye movements of different grade students’ reading perception span and parafoveal preview (in Chinese). Acta Psychol Sin 03:249–263

    Google Scholar 

  43. Yang Z, Yang D, Dyer C, He X, Smola A, Hovy E (2016) Hierarchical attention networks for document classification. In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pp 1480–1489

  44. Yayla M, Demirkol MD, Alqaraleh S (2021) Cnn vs. lstm for turkish text classification. In: 2021 international conference on innovations in intelligent systems and applications (INISTA), IEEE, pp 1–6

  45. Yu K, Liu Y, Schwing AG, Peng J (2018) Fast and accurate text classification: skimming, rereading and early stopping. In: International conference on learning representations (ICLR)

  46. Zhao J, Dong L, Wu J, Xu K (2012) Moodlens: an emoticon-based sentiment analysis system for Chinese tweets. In: Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining, pp 1528–1531

  47. Zheng WL, Liu W, Lu Y, Lu BL, Cichocki A (2018) Emotionmeter: a multimodal framework for recognizing human emotions. IEEE Trans Cybern 49(3):1110–1122

    Article  Google Scholar 

  48. Zheng Y, Mao J, Liu Y, Ye Z, Zhang M, Ma S (2019) Human behavior inspired machine reading comprehension. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp 425–434

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, Tsinghua University, Beijing, 100084, China

    Xuesong Chen, Jiaxin Mao, Yiqun Liu, Min Zhang & Shaoping Ma

Authors
  1. Xuesong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiaxin Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yiqun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Min Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shaoping Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yiqun Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, X., Mao, J., Liu, Y. et al. Investigating human reading behavior during sentiment judgment. Int. J. Mach. Learn. & Cyber. 13, 2283–2296 (2022). https://doi.org/10.1007/s13042-022-01523-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-022-01523-9

Keywords

Search

Navigation