Skip to main content
SpringerLink
Log in

When attention is not enough to unveil a text’s author profile: Enhancing a transformer with a wide branch

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Author profiling (AP) is a highly relevant natural language processing (NLP) problem; it deals with predicting features of authors such as gender, age and personality traits. It is done by analyzing texts written by the authors themselves; take for instance documents such as books, articles, and more recently posts in social media platforms. In the present study, we focus in the latter, which is an scenario with a number of applications in marketing, security, health and others. Surprisingly, given the achievements of deep learning (DL) strategies on other NLP tasks, for AP DL architectures regularly underperform, left behind by classical machine learning (ML) approaches. In this study we show how a deep learning architecture based on transformers offers competitive results by exploiting a joint-intermediate fusion strategy called the Wide & Deep Transformer (WD-T). Our methodology implements a fusion of contextualized word vector representations and handcrafted features, by using a self-attention mechanism and a novel encoding technique that incorporates stylistic, topic, and personal information from authors. This allows for the creation of more accurate, fine-grained predictions. Our approach attained competitive performance against top-quartile results from the 2017–2019 editions at the Plagiarism analysis, Authorship identification, and Near-duplicate detection forum (PAN) in English and Spanish languages for gender and language variety predictions, and the Kaggle Myers–Briggs-type indicator (MBTI) dataset for personality forecasting. Our proposal consistently surpasses all other deep learning methods in PAN collections by as much as 2.4%, and up to 3.4% in the MBTI dataset. These results suggest that this DL strategy effectively addresses and improves upon the limitations of previous techniques and paves the way for new avenues of inquiry.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availibility statement

The datasets analyzed during the current study are available in the PAN at CLEF & Kaggle repositories. These datasets were obtained from the following public domain resources: https://pan.webis.de/shared-tasks.html, https://www.kaggle.com/datasets/datasnaek/mbti-type

Notes

  1. https://pan.webis.de/.

  2. The CLEF Initiative (Conference and Labs of the Evaluation Forum, formerly known as Cross-Language Evaluation Forum).

  3. In the 2019 collection [86], the maximum length of the concatenated documents is the largest of all datsets. This might be caused in part by the task of detecting bots from humans. Since most of the posts produced by bots are very large due to the excessive use of hashtags and re-tweets. In addition, to cope with the computational complexity of the self-attention module [\(O(n^2)\)], the 2019 dataset was truncated to a maximum of 3500 tokens (WEs) of length.

References

  1. Potthast M, Rosso P, Stamatatos E, Stein B (2019) A decade of shared tasks in digital text forensics at pan. In: Azzopardi L, Stein B, Fuhr N, Mayr P, Hauff C, Hiemstra D (eds) Advances in Information Retrieval. Springer, Cham, pp 291–300

  2. Rezaeinia SM, Rahmani R, Ghodsi A, Veisi H (2019) Sentiment analysis based on improved pre-trained word embeddings. Expert Syst Appl 117:139–147. https://doi.org/10.1016/j.eswa.2018.08.044

    Article  Google Scholar 

  3. Fatima M, Hasan K, Anwar S, Nawab RMA (2017) Multilingual author profiling on facebook. Inform Process Manag 53(4):886–904. https://doi.org/10.1016/j.ipm.2017.03.005

    Article  Google Scholar 

  4. Sengupta S, Basak S, Saikia P, Paul S, Tsalavoutis V, Atiah F, Ravi V, Peters A (2020) A review of deep learning with special emphasis on architectures, applications and recent trends. Knowl-Based Syst 194:105596. https://doi.org/10.1016/j.knosys.2020.105596

    Article  Google Scholar 

  5. Rangel F, Rosso P (2016) On the impact of emotions on author profiling. Information Processing & Management 52(1):73–92. https://doi.org/10.1016/j.ipm.2015.06.003. cited By 41

  6. Aragón ME. López-Monroy AP, González-Gurrola LC, Montes-y-Gómez M (2019) Detecting depression in social media using fine-grained emotions. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1481–1486. Association for Computational Linguistics, Minneapolis, Minnesota. https://doi.org/10.18653/v1/N19-1151. https://www.aclweb.org/anthology/N19-1151

  7. Aragón ME, López-Monroy AP, González LC, Montes-y-Gómez M (2020) Attention to emotions: Detecting mental disorders in social media. In: Sojka P, Kopeček I, Pala K, Horák A (eds) Text, Speech, and Dialogue. Springer, Cham, pp 231–239

  8. Hazrati N, Ricci F (2022) Recommender systems effect on the evolution of users’ choices distribution. Inform Process Manag 59(1):102766. https://doi.org/10.1016/j.ipm.2021.102766

    Article  Google Scholar 

  9. Liao G, Deng X, Wan C, Liu X (2022) Group event recommendation based on graph multi-head attention network combining explicit and implicit information. Inform Process Manag 59(2):102797. https://doi.org/10.1016/j.ipm.2021.102797

    Article  Google Scholar 

  10. Zhang H, Zang Z, Zhu H, Uddin MI, Amin MA (2022) Big data-assisted social media analytics for business model for business decision making system competitive analysis. Inform Process Manag 59(1):102762. https://doi.org/10.1016/j.ipm.2021.102762

    Article  Google Scholar 

  11. Yang J, Xiu P, Sun L, Ying L, Muthu B (2022) Social media data analytics for business decision making system to competitive analysis. Inform Process Manag 59(1):102751. https://doi.org/10.1016/j.ipm.2021.102751

    Article  Google Scholar 

  12. Rangel F, Rosso P, Potthast M, Stein B (2017) Overview of the 5th author profiling task at pan 2017: Gender and language variety identification in twitter. In: L., C., N., F., L, G., T, M. (eds.) CLEF 2017 Labs and Workshops, Notebook Papers. CEUR Workshop Proceedings. CEUR-WS.org

  13. Biswas S, Bhat S, Jaiswal G, Sharma A (2022) Critical insights into machine learning and deep learning approaches for personality prediction. In: Lecture Notes in Electrical Engineering. Springer, pp. 707–718. https://doi.org/10.1007/978-981-19-2828-4_63

  14. Rangel F, Rosso P, Verhoeven B, Daelemans W, Pottast M, Stein B (2016) Overview of the 4th author profiling task at pan 2016: Cross-genre evaluations. In: Balog, Capellato, Ferro, Macdonald (eds.) CLEF 2016 Labs and Workshops, Notebook Papers. CEUR Workshop Proceedings. CEUR-WS.org

  15. Cheng H-T, Koc L, Harmsen J, Shaked T, Chandra T, Aradhye H, Anderson G, Corrado G, Chai W, Ispir M, Anil R, Haque Z, Hong L, Jain V, Liu X, Shah H (2016) Wide & Deep Learning for Recommender Systems

  16. Pennebaker JW (2013) The Secret Life of Pronouns: What Our Words Say About Us. Bloomsbury Publishing, 1385 Broadway, Fifth Floor, New York, NY 10018. https://www.secretlifeofpronouns.com

  17. Ortega-Mendoza R, López-Monroy A, Franco-Arcega A, Montes M (2018) Emphasizing personal information for author profiling: new approaches for term selection and weighting. Knowl-Based Syst 145:169–181

    Article  Google Scholar 

  18. Schwartz HA, Eichstaedt JC, Kern ML, Dziurzynski L, Ramones SM, Agrawal M, Shah A, Kosinski M, Stillwell D, Seligman MEP, Ungar LH (2013) Personality, gender, and age in the language of social media: the open-vocabulary approach. PLoS ONE 8(9):1–16. https://doi.org/10.1371/journal.pone.0073791

    Article  Google Scholar 

  19. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Lu, Polosukhin I (2017) Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 5998–6008. Curran Associates, Inc., 57 Morehouse Lane Red Hook NY 12571 US. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf

  20. Pardo FMR, Giachanou A, Ghanem B, Rosso P (2020) Overview of the 8th author profiling task at pan 2020: Profiling fake news spreaders on twitter. In: Cappellato, L., Eickhoff, C., 0001, N.F., Névéol, A. (eds.) Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020. CEUR Workshop Proceedings, vol. 2696. CEUR-WS.org, Thessaloniki, Greece. http://ceur-ws.org/Vol-2696/paper_267.pdf

  21. Rangel F, la Peña Sarracén GLD, Chulvi B, Fersini E, Rosso P (2021) Profiling hate speech spreaders on twitter task at PAN 2021. In: Faggioli, G., Ferro, N., Joly, A., Maistro, M., Piroi, F. (eds.) Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum (CLEF 2021). CEUR Workshop Proceedings, vol. 2936, pp. 1772–1789. CEUR-WS.org, Aachen. http://ceur-ws.org/Vol-2936/

  22. Rangel F, Rosso FCP, Potthast M, Stein B, Daelemans W (2015) Daelemans, w.: Overview of the 3rd author profiling task at pan 2015. In: CLEF 2015 Labs and Workshops, Notebook Papers. CEUR Workshop Proceedings, CEUR-WS.org (Sep 2015)

  23. J M (2017) (MBTI) myers-briggs personality type dataset . https://www.kaggle.com/datasets/datasnaek/mbti-type

  24. Habic V, Semenov A, Pasiliao EL (2020) Multitask deep learning for native language identification. Knowl-Based Syst 209:106440. https://doi.org/10.1016/j.knosys.2020.106440

    Article  Google Scholar 

  25. Niu M, Cai J (2019) A label informative wide & deep classifier for patents and papers. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3438–3443. Association for Computational Linguistics, Hong Kong, China. https://doi.org/10.18653/v1/D19-1344. https://www.aclweb.org/anthology/D19-1344

  26. Fonseca E, Alvarenga JPR (2019) Wide and deep transformers applied to semantic relatedness and textual entailment. In: Oliveira, H.G., Real, L., Fonseca, E. (eds.) Proceedings of the ASSIN 2 Shared Task: Evaluating Semantic Textual Similarity and Textual Entailment in Portuguese. CEUR Workshop Proceedings, vol. 2583. CEUR-WS.org, Salvador, BA, Brazil. http://ceur-ws.org/Vol-2583/7_Stilingue.pdf

  27. Burel G, Saif H, Alani H (2017) Semantic wide and deep learning for detecting crisis-information categories on social media. In: d’Amato C, Fernandez M, Tamma V, Lecue F, Cudré-Mauroux P, Sequeda J, Lange C, Heflin J (eds) The Semantic Web - ISWC 2017. Springer, Cham, pp 138–155

  28. Anderson M (2021) Three unique architectures for deep learning based recommendation systems. WIDTH.AI - Martin Anderson. https://bit.ly/3aSYv4Q

  29. López-Monroy AP, Montes-y-Gómez M, Escalante HJ, Villaseñor-Pineda L, Stamatatos E (2015) Discriminative subprofile-specific representations for author profiling in social media. Knowl-Based Syst 89:134–147. https://doi.org/10.1016/j.knosys.2015.06.024

    Article  Google Scholar 

  30. López-Monroy AP, Montes-y-Gómez M, Escalante HJ, Pineda LV, Villatoro-Tello E (2013) Inaoe’s participation at pan’13: Author profiling task notebook for PAN at CLEF 2013. In: Working Notes for CLEF 2013 Conference, Valencia, Spain, September 23-26, 2013. http://ceur-ws.org/Vol-1179/CLEF2013wn-PAN-LopezMonroyEt2013.pdf

  31. López-Monroy AP, MontesY-Gómez M, Escalante HJ, Villaseñor-Pineda L (2014) Using intra-profile information for author profiling. In: Working Notes of CLEF 2014 - Conference and Labs of the Evaluation Forum, vol. 1180

  32. Álvarez-Carmona MA, López-Monroy AP, MontesY-Gómez M, Villaseñor-Pineda L, Escalante HJ (2015) Inaoe’s participation at pan’15: Author profiling task. In: Cappellato, L., Ferro, N., Jones, G.J.F., SanJuan, E. (eds.) CLEF (Working Notes). CEUR Workshop Proceedings, vol. 1391. CEUR-WS.org, Toulouse, France. http://ceur-ws.org/Vol-1391/122-CR.pdf

  33. Vollenbroek MBO, Carlotto T, Kreutz T, Medvedeva M, Pool C, Bjerva J, Haagsma H, Nissim M (2016) Gronup: Groningen user profiling. In: Working Notes of CLEF 2016 - Conference and Labs of the Evaluation Forum, Évora, Portugal, 5-8 September, 2016., pp. 846–857. http://ceur-ws.org/Vol-1609/16090846.pdf

  34. Basile A, Dwyer G, Medvedeva M, Rawee J, Haagsma H, Nissim M (2017) N-gram: New groningen author-profiling model. CoRR arXiv:1707.03764

  35. Daneshvar S, Inkpen D (2018) Gender identification in twitter using n-grams and lsa: Notebook for pan at clef 2018. In: Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018

  36. Pizarro J (2019) Using n-grams to detect bots on twitter. In: CLEF

  37. López-Santillán R, Montes-Y-Gómez M, González-Gurrola LC, Ramírez-Alonso G, Prieto-Ordaz O (2020) Richer document embeddings for author profiling tasks based on a heuristic search. Inform Process Manag 57(4):102227

    Article  Google Scholar 

  38. Sierra S, Montes-y-Gómez M, Solorio T, González FA (2017) Convolutional neural networks for author profiling in PAN 2017. In: Cappellato, L., Ferro, N., Goeuriot, L., Mandl, T. (eds.) Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, Dublin, Ireland, September 11-14. CEUR Workshop Proceedings, vol. 1866. CEUR-WS.org, Dublin, Ireland (2017). http://ceur-ws.org/Vol-1866/paper_93.pdf

  39. Franco-Salvador M, Plotnikova N, Pawar N, Benajiba Y (2017) Subword-based deep averaging networks for author profiling in social media. In: Cappellato, L., Ferro, N., Goeuriot, L., Mandl, T. (eds.) Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, Dublin, Ireland, September 11-14, 2017. CEUR Workshop Proceedings, vol. 1866. CEUR-WS.org, Dublin, Ireland

  40. Bojanowski P, Grave E, Joulin A, Mikolov T (2017) Enriching word vectors with subword information. Trans Assoc Comput Linguist 5:135–146

    Article  Google Scholar 

  41. Miura Y, Taniguchi T, Taniguchi M, Ohkuma T (2017) Author profiling with word+character neural attention network. In: Cappellato, L., Ferro, N., Goeuriot, L., Mandl, T. (eds.) Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, Dublin, Ireland, September 11-14, 2017. CEUR Workshop Proceedings, vol. 1866. CEUR-WS.org, Dublin, Ireland. http://ceur-ws.org/Vol-1866/paper_90.pdf

  42. Polignano M, de Pinto MG, Lops P, Semeraro G (2019) Identification of bot accounts in twitter using 2d cnns on user-generated contents. In: Cappellato, L., Ferro, N., Losada, D.E., Müller, H. (eds.) Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12. CEUR Workshop Proceedings, vol. 2380. CEUR-WS.org, Lugano, Switzerland

  43. Ashraf MA, Adeel Nawab RM, Nie F (2020) A study of deep learning methods for same-genre and cross-genre author profiling 39:2353–2363. https://doi.org/10.3233/JIFS-179896

  44. HaCohen-Kerner Y (2022) Survey on profiling age and gender of text authors. Expert Syst Appl 199:117140. https://doi.org/10.1016/j.eswa.2022.117140

    Article  Google Scholar 

  45. Joo Y, Hwang I (2019) Author profiling on social media: An ensemble learning model using various features notebook for pan at clef 2019:2380

  46. Devlin J, Chang M, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171–4186

  47. Zhang C, Abdul-Mageed M (2019) Bert-based arabic social media author profiling. In: Mehta, P., Rosso, P., Majumder, P., Mitra, M. (eds.) Working Notes of FIRE 2019 - Forum for Information Retrieval Evaluation, Kolkata, India, December 12-15, 2019. CEUR Workshop Proceedings, vol. 2517, pp. 84–91. CEUR-WS.org, Kolkata, India. http://ceur-ws.org/Vol-2517/T2-2.pdf

  48. Foundation TMB: Myers-Briggs / MBTI in the age of the big five. https://www.myersbriggs.org/my-mbti-personality-type/mbti-basics/

  49. International T What are the big 5 personality traits? https://www.thomas.co/resources/type/hr-guides/what-are-big-5-personality-traits

  50. Bharadwaj S, Sridhar S, Choudhary R, Srinath R (2018) Persona traits identification based on Myers-Briggs type indicator(MBTI) - a text classification approach. In: 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI). https://doi.org/10.1109/icacci.2018.8554828

  51. Patel S, Nimje M, Shetty A, Kulkarni S (2021) Personality analysis using social media. IJERT-International Journal of Engineering Research & Technology. https://www.ijert.org/personality-analysis-using-social-media

  52. Kaushal P, P NBB, S PM, S K, Koundinya AK (2021) Myers-briggs personality prediction and sentiment analysis of twitter using machine learning classifiers and BERT. International Journal of Information Technology and Computer Science 13(6), 48–60. https://doi.org/10.5815/ijitcs.2021.06.04

  53. Kishore Kumar R, Jeeva Surya V, Shana J (2022) Personality prediction based on twitter tweets. In: Bansal JC, Engelbrecht A, Shukla PK (eds) Computer Vision and Robotics. Springer, Singapore, pp 25–34

  54. Maulidah M, Pardede HF (2021) Prediction of Myers-Briggs type indicator personality using long short-term memory. Jurnal Elektronika dan Telekomunikasi 21(2), 104. https://doi.org/10.14203/jet.v21.104-111

  55. Khan AS, Ahmad H, Zubair M, Khan F, Arif A, Ali H (2020) Personality classification from online text using machine learning approach. Int J Adv Comput Sci Appl 11(3). https://doi.org/10.14569/ijacsa.2020.0110358

  56. Amirhosseini MH, Kazemian H (2020) Machine learning approach to personality type prediction based on the Myers–Briggs type indicator®. Multimodal Technol Interaction. MDPI AG 4(1):9. https://doi.org/10.3390/mti4010009

    Article  Google Scholar 

  57. Mushtaq Z, Ashraf S, Sabahat N (2020) Predicting MBTI personality type with k-means clustering and gradient boosting. In: 2020 IEEE 23rd International Multitopic Conference (INMIC). https://doi.org/10.1109/inmic50486.2020.9318078

  58. Nisha KA, Kulsum U, Rahman S, Hossain MF, Chakraborty P, Choudhury T (2021) A comparative analysis of machine learning approaches in personality prediction using MBTI. In: Computational Intelligence in Pattern Recognition. Springer, pp. 13–23. https://doi.org/10.1007/978-981-16-2543-5_2

  59. Choong EJ, Varathan KD (2021) Predicting judging-perceiving of Myers-Briggs type indicator (MBTI) in online social forum 9, 11382. https://doi.org/10.7717/peerj.11382

  60. Mehta Y, Fatehi S, Kazameini A, Stachl C, Cambria E, Eetemadi S (2020) Bottom-up and top-down: Predicting personality with psycholinguistic and language model features. In: 2020 IEEE International Conference on Data Mining (ICDM). https://doi.org/10.1109/icdm50108.2020.00146

  61. Cui B, Qi C (2017) Survey analysis of machine learning methods for natural language processing for mbti personality type prediction

  62. Ontoum S, Chan JH (2022) personality type based on Myers-Briggs type indicator with text posting style by using traditional and deep learning. arXiv. https://doi.org/10.48550/ARXIV.2201.08717. arXiv:abs/2201.08717

  63. Frkovic M, Cerkez N, Vrdoljak B, Skansi S (2020) Evaluation of structural hyperparameters for text classification with LSTM networks. In: 2020 43rd International Convention on Information, Communication and Electronic Technology (MIPRO). https://doi.org/10.23919/mipro48935.2020.9245216

  64. Kosse R, Schuur Y, Cnossen G (2018) Mixing traditional methods with neural networks for gender prediction: Notebook for pan at clef 2018. In: Cappellato, L., Ferro, N., Nie, J.-Y., Soulier, L. (eds.) Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018. CEUR Workshop Proceedings, vol. 2125. CEUR-WS.org, Avignon, France. http://ceur-ws.org/Vol-2125/paper_189.pdf

  65. Veenhoven R, Snijders S, van der Hall D, van Noord R (2018) Using translated data to improve deep learning author profiling models. In: Proceedings of the Ninth International Conference of the CLEF Association (CLEF). CLEF, Avignon, France

  66. Sierra S, González FA (2018) Combining textual and visual representations for multimodal author profiling: Notebook for pan at clef 2018. In: Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018). CLEF, Avignon, France

  67. Takahashi T, Tahara T, Nagatani K, Miura Y, Taniguchi T, Ohkuma T (2018) Text and image synergy with feature cross technique for gender identification: Notebook for pan at clef 2018. In: Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018)

  68. Martinc M, Skrlj B, Pollak S (2018) Multilingual gender classification with multi-view deep learning: Notebook for pan at clef 2018. In: Cappellato, L., Ferro, N., Nie, J., Soulier, L. (eds.) Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018. CEUR Workshop Proceedings, vol. 2125. CEUR-WS.org, Avignon, France. http://ceur-ws.org/Vol-2125/paper_156.pdf

  69. Aragón ME, López-Monroy AP (2018) A straightforward multimodal approach for author profiling: Notebook for pan at clef 2018. In: Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018). CLEF, Avignon, France

  70. Schaetti N (2018) Character-based convolutional neural network and resnet18 for twitter author profiling: Notebook for PAN at CLEF 2018. In: Cappellato, L., Ferro, N., Nie, J., Soulier, L. (eds.) Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018. CEUR Workshop Proceedings, vol. 2125. CEUR-WS.org, Avignon, France. http://ceur-ws.org/Vol-2125/paper_100.pdf

  71. Sezerer E, Polatbilek O, Sevgili Ö, Tekir S (2018) Gender prediction from tweets with convolutional neural networks: Notebook for PAN at CLEF 2018. In: Cappellato, L., Ferro, N., Nie, J., Soulier, L. (eds.) Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018. CEUR Workshop Proceedings, vol. 2125. CEUR-WS.org, Avignon, France. http://ceur-ws.org/Vol-2125/paper_116.pdf

  72. Raiyani K, Quaresma TGP, Beires-Nogueira V (2018) Multi-language neural network model with advance preprocessor for gender classification over social media. In: Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018). CLEF, Avignon, France

  73. Petrik J, Chuda D (2019) Bots and gender profiling with convolutional hierarchical recurrent neural network notebook for pan at clef 2019:2380

  74. la Peña Sarracén GLD, Fontcuberta JRP (2019) Bots and gender profiling using a deep learning approach. notebook for pan at clef 2019 2380

  75. Bolonyai F, Buda J, Katona E (2019) Bot or not: A two-level approach in author profiling notebook for pan at clef 2019:2380

  76. Onose C, Nedelcu C, Cercel D, Trausan-Matu S (2019) A hierarchical attention network for bots and gender profiling. In: Cappellato, L., Ferro, N., Losada, D.E., Müller, H. (eds.) Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019. CEUR Workshop Proceedings, vol. 2380. CEUR-WS.org, Lugano, Switzerland. http://ceur-ws.org/Vol-2380/paper_228.pdf

  77. Halvani O, Marquardt P (2019) An unsophisticated neural bots and gender profiling system. In: Cappellato, L., Ferro, N., Losada, D.E., Müller, H. (eds.) Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019. CEUR Workshop Proceedings, vol. 2380. CEUR-WS.org, Lugano, Switzerland. http://ceur-ws.org/Vol-2380/paper_206.pdf

  78. Zhechev I, Andonov Z, Bozhilov I (2019) Bot and gender profiling based on voting lstm. notebook for pan at clef 2019. In: Cappellato, L., Ferro, N., Losada, D.E., Müller, H. (eds.) Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019. CEUR Workshop Proceedings, vol. 2380. CEUR-WS.org, Lugano, Switzerland

  79. Dias RFS, Paraboni I (2019) Combined cnn+rnn bot and gender profiling. In: Cappellato, L., Ferro, N., Losada, D.E., Müller, H. (eds.) Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019. CEUR Workshop Proceedings, vol. 2380. CEUR-WS.org, Lugano, Switzerland

  80. Färber M, Qurdina A, Ahmedi L (2019) Identifying twitter bots using a convolutional neural network. In: Cappellato, L., Ferro, N., Losada, D.E., Müller, H. (eds.) Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019. CEUR Workshop Proceedings, vol. 2380. CEUR-WS.org, Lugano, Switzerland

  81. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 26, pp. 3111–3119. Curran Associates, Inc., 57 Morehouse Lane Red Hook NY 12571 US. http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf

  82. Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. In: Proc. of NAACL

  83. Geron A (2017) Hands-on Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. OReilly Media, Sebastopol, CA

    Google Scholar 

  84. López-Santillán R, Montes-Y-Gómez M, González-Gurrola LC, Ramírez-Alonso G, Prieto-Ordaz O (2020) Richer document embeddings for author profiling tasks based on a heuristic search. Inform Process Manag 57(4):102227. https://doi.org/10.1016/j.ipm.2020.102227

    Article  Google Scholar 

  85. Rangel F, Rosso P, Montes-y-Gómez M, Potthast M, Stein B Overview of the 6th author profiling task at pan 2018: Multimodal gender identification in twitter. In: Cappellato, L., Ferro, N., Nie, J.-Y., Soulier, L. (eds.) Working Notes Papers of the CLEF 2018 Evaluation Labs. CEUR Workshop Proceedings. CLEF and CEUR-WS.org

  86. Pardo FMR, Rosso P (2019) Overview of the 7th author profiling task at pan 2019: Bots and gender profiling in twitter. In: CLEF

  87. Loper E, Bird S (2002) Nltk: The natural language toolkit. CoRR cs.CL/0205028

  88. Qi P, Zhang Y, Zhang Y, Bolton J, Manning CD (2020) Stanza: A Python natural language processing toolkit for many human languages. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. https://nlp.stanford.edu/pubs/qi2020stanza.pdf

  89. Dai Z, Yang Z, Yang Y, Carbonell J, Le QV, Salakhutdinov R (2019) Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

  90. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ (2019) Exploring the Limits of transfer learning with a unified text-to-text transformer. arXiv . https://doi.org/10.48550/ARXIV.1910.10683. arXiv:abs/1910.10683

  91. Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler DM, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D (2020) Language Models are Few-Shot Learners. arXiv. https://doi.org/10.48550/ARXIV.2005.14165. arXiv:abs/2005.14165

  92. Kitaev N, Łukasz Kaiser, Levskaya A (2020) Reformer: The Efficient Transformer

Download references

Acknowledgements

The research in this paper was partially supported by the National Council of Science and Technology (CONACYT) of Mexico, via the project FC-2410 2017-2019.

Author information

Authors and Affiliations

  1. Facultad de Ingeniería, Universidad Autónoma de Chihuahua, Circuito Universitario Campus II, 31125, Chihuahua, Mexico

    Roberto López-Santillán & Luis C. González

  2. Coordinación de Ciencias Computacionales, Instituto Nacional de Astrofísica, Óptica y Electrónica, Luis Enrique Erro No. 1 Sta. Ma. Tonantzintla, 72840, Puebla, Mexico

    Manuel Montes-y-Gómez

  3. Departamento de Ciencias de la Computación, Centro de Investigación en Matemáticas AC, Jalisco S/N, Col. Valenciana, 36023, Guanajuato, Mexico

    A. Pastor López-Monroy

Authors
  1. Roberto López-Santillán
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luis C. González
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manuel Montes-y-Gómez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. A. Pastor López-Monroy
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Roberto López-Santillán contributed to conceptualization, methodology, software, writing - original draft preparation, visualization. Luis C. González contributed to conceptualization, writing - review & editing. Manuel Montes-Y-Gómez contributed to conceptualization, resources, writing - review & editing. A. Pastor López-Monroy contributed to writing - review & editing.

Corresponding authors

Correspondence to Roberto López-Santillán or Luis C. González.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

López-Santillán, R., González, L.C., Montes-y-Gómez, M. et al. When attention is not enough to unveil a text’s author profile: Enhancing a transformer with a wide branch. Neural Comput & Applic 35, 9607–9626 (2023). https://doi.org/10.1007/s00521-023-08198-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08198-5

Keywords

Search

Navigation