skip to main content
10.1145/3136755.3136770acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

Multimodal gender detection

Published: 03 November 2017 Publication History

Abstract

Automatic gender classification is receiving increasing attention in the computer interaction community as the need for personalized, reliable, and ethical systems arises. To date, most gender classification systems have been evaluated on textual and audiovisual sources. This work explores the possibility of enhancing such systems with physiological cues obtained from thermography and physiological sensor readings. Using a multimodal dataset consisting of audiovisual, thermal, and physiological recordings of males and females, we extract features from five different modalities, namely acoustic, linguistic, visual, thermal, and physiological. We then conduct a set of experiments where we explore the gender prediction task using single and combined modalities. Experimental results suggest that physiological and thermal information can be used to recognize gender at reasonable accuracy levels, which are comparable to the accuracy of current gender prediction systems. Furthermore, we show that the use of non-contact physiological measurements, such as thermography readings, can enhance current systems that are based on audio or visual input. This can be particularly useful for scenarios where non-contact approaches are preferred, i.e., when data is captured under noisy audiovisual conditions or when video or speech data are not available due to ethical considerations.

References

[1]
Mohamed Abouelenien, Veronica Pérez-Rosas, Rada Mihalcea, and Mihai Burzo. 2014. Deception Detection Using a Multimodal Approach. In Proceedings of the 16th International Conference on Multimodal Interaction (ICMI ’14). ACM, Istanbul, Turkey, 58–65.
[2]
M. Abouelenien, V. Pérez-Rosas, R. Mihalcea, and M. Burzo. 2017. Detecting Deceptive Behavior via Integration of Discriminative Features From Multiple Modalities. IEEE Transactions on Information Forensics and Security 12, 5 (May 2017), 1042–1055.
[3]
Jens Allwood, Loredana Cerrato, Kristiina Jokinen, Costanza Navarretta, and Patrizia Paggio. 2007. The MUMIN coding scheme for the annotation of feedback, turn management and sequencing phenomena. Language Resources and Evaluation 41, 3-4 (2007), 273–287.
[4]
Yasmina Andreu, Pedro Garcia-Sevilla, and R.A. Mollineda. 2014. Face gender classification: A statistical study when neutral and distorted faces are combined for training and testing purposes. Image and Vision Computing 32, 1 (2014), 27–36.
[5]
Juan Bekios-Calfa, J. Buenaposada, and Luis Baumela. 2014. Robust gender recognition by exploiting facial attributes dependencies. Pattern Recognition Letters 36 (2014), 228 – 234.
[6]
J. Bishop and P. Keating. 2009. Perception of pitch location within a speakerâĂŹs range: Fundamental Frequency, voice quality and speaker sex. The Journal of the Acoustical Society of America 132, 2 (2009), 1100–1112. 1749-818X.2009.00125.x
[7]
Constantinos Boulis and Mari Ostendorf. 2005. A Quantitative Analysis of Lexical Differences Between Genders in Telephone Conversations. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL ’05). Association for Computational Linguistics, Stroudsburg, PA, USA, 435–442.
[8]
Michael Brookes. 2003. VOICEBOX: Speech Processing Toolbox for MATLAB. (2003).
[9]
D. John Burger, John Henderson, George Kim, and Guido Zarrella. 2011. Discriminating Gender on Twitter. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1301–1309. http://aclweb.org/anthology/D11-1120
[10]
C. Chen and A. Ross. 2011. Evaluation of gender classification methods on thermal and near-infrared face images. In 2011 International Joint Conference on Biometrics (IJCB). 1–8.
[11]
Na Cheng, R. Chandramouli, and K. P. Subbalakshmi. 2011. Author Gender Identification from Text. Digit. Investig. 8, 1 (July 2011), 78–88.
[12]
Florian Eyben, Martin Wöllmer, and Björn Schuller. 2009. OpenEAR Introducing the Munich open-source emotion and affect recognition toolkit. In 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops. IEEE, 1–6.
[13]
Aparna Garimella and Rada Mihalcea. 2016. Proceedings of the Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media (PEOPLES). The COLING 2016 Organizing Committee, Chapter Zooming in on Gender Differences in Social Media, 1–10. http://aclweb.org/anthology/ W16-4301
[14]
Travis Gault and Aly Farag. 2013. A fully automatic method to extract the heart rate from thermal video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 336–341.
[15]
Abdenour Hadid and Matti Pietikäinen. 2009. Combining Appearance and Motion for Face and Gender Recognition from Videos. Pattern Recogn. 42, 11 (Nov. 2009), 2818–2827.
[16]
R. Hartley and A. Zisserman. 2003. Multiple View Geometry in Computer Vision. Cambridge University Press.
[17]
Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. 2007. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments. Technical Report 07-49. University of Massachusetts, Amherst.
[18]
G. Levi and T. Hassncer. 2015. Age and gender classification using convolutional neural networks. In 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 34–42.
[19]
[20]
Ita Sarah Levitan, Yocheved Levitan, Guozhen An, Michelle Levine, Rivka Levitan, Andrew Rosenberg, and Julia Hirschberg. 2016. Proceedings of the Second Workshop on Computational Approaches to Deception Detection. Association for Computational Linguistics, Chapter Identifying Individual Differences in Gender, Ethnicity, and Personality from Dialogue for Deception Detection, 40–44.
[21]
Sarah Ita Levitan, Taniya Mishra, and Srinivas Bangalore. 2016. Automatic Identification of Gender from Speech. In Speech Prosody.
[22]
Gregory F Lewis, Rodolfo G Gatto, and Stephen W Porges. 2011. A novel method for extracting respiration rate and relative tidal volume from infrared thermography. Psychophysiology 48, 7 (2011), 877–887.
[23]
X. Li, X. Zhao, Y. Fu, and Y. Liu. 2010. Bimodal gender recognition from face and fingerprint. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2590–2597.
[24]
C. A.and Papadelis C.and Vivas Ana B.and Klados M. A.and Kourtidou-Papadeli C.and Pappas C.and Ioannides A. A.and Bamidis P. D. Lithari, C.and Frantzidis. 2010. Are Females More Responsive to Emotional Stimuli? A Neurophysiological Study Across Arousal and Valence Dimensions. Brain Topography 23, 1 (01 Mar 2010), 27–40.
[25]
L. Lu, Z. Xu, and P. Shi. 2009. Gender Classification of Facial Images Based on Multiple Facial Regions. In 2009 WRI World Congress on Computer Science and Information Engineering, Vol. 6. 48–52.
[26]
Xiaofei Lu. 2010. Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics 15, 4 (2010), 474–496.
[27]
E. Makinen and R. Raisamo. 2008. Evaluation of Gender Classification Methods with Automatically Detected and Aligned Faces. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 3 (March 2008), 541–547.
[28]
Rada Mihalcea and Stephen Pulman. 2009. Linguistic ethnography: Identifying dominant word classes in text. In Computational Linguistics and Intelligent Text Processing. Springer, 594–602.
[29]
Matthew L Newman, Carla J Groom, Lori D Handelman, and James W Pennebaker. 2008. Gender differences in language use: An analysis of 14,000 text samples. Discourse Processes 45, 3 (2008), 211–236.
[30]
Dat Tien Nguyen and Kang Ryoung Park. 2016. Body-Based Gender Recognition Using Images from Visible and Thermal Cameras. Sensors 16, 2 (2016).
[31]
P. Nguyen, D. Tran, X. Huang, and W. Ma. 2013. Age and gender classification using EEG paralinguistic features. In 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER). 1295–1298.
[32]
[33]
J. Pennebaker and M. Francis. 1999. Linguistic Inquiry and Word Count: LIWC. (1999). Erlbaum Publishers.
[34]
Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 433–440.
[35]
P.Jonathon Phillips, Harry Wechsler, Jeffery Huang, and Patrick J. Rauss. 1998. The FERET database and evaluation procedure for face-recognition algorithms. Image and Vision Computing 16, 5 (1998), 295 – 306.
[36]
Daniel Reid, Sina Samangooei, Cunjian Chen, Mark Nixon, and Arun Ross. 2013. Soft biometrics for surveillance: an overview. Machine learning: theory and applications. Elsevier (2013), 327–352.
[37]
Mickael Rouvier, Grégor Dupuy, Paul Gay, Elie Khoury, Teva Merlin, and Sylvain Meignier. 2013. An open-source state-of-the-art toolbox for broadcast news diarization. Technical Report. Idiap.
[38]
Ruchita Sarawgi, Kailash Gajulapalli, and Yejin Choi. 2011. Gender attribution: tracing stylometric evidence beyond topic and genre. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning. Association for Computational Linguistics, 78–86.
[39]
Alexandra Schofield and Leo Mehr. 2016. Proceedings of the Fifth Workshop on Computational Linguistics for Literature. Association for Computational Linguistics, Chapter Gender-Distinguishing Features in Film Dialogue, 32–39.
[40]
Caifeng Shan. 2012. Learning Local Binary Patterns for Gender Classification on Real-world Face Images. Pattern Recognition Letters 33, 4 (March 2012), 431–437.
[41]
Maha Sharkas and Mohamed Abouelenien. 2008. Eigenfaces vs. fisherfaces vs. ICA for face recognition; a comparative study. In 2008 9th International Conference on Signal Processing. 914–919.
[42]
Adrian P. Simpson. 2009. Phonetic differences between male and female speech. Language and Linguistics Compass 3, 2 (2009), 621–640.
[43]
Sudipta N. Sinha, Jan-michael Frahm, Marc Pollefeys, and Yakup Genc. 2006. GPUbased Video Feature Tracking and Matching. Technical Report. The University of North Carolina at Chapel Hill.
[44]
David Smith and Roy D. Patterson. 2005. The Interaction of Glottal-Pulse Rate and Vocal-Tract Length in Judgements of Speaker Size, Sex, and Age. Journal of the Acoustical Society of America 118, 5 (2005), 3177âĂŞ3186.
[45]
Pattanawit Soanboon, Somsong Nanakorn, and Wibhu Kutanan. 2016. Determination of sex difference from fingerprint ridge density in northeastern Thai teenagers. Egyptian Journal of Forensic Sciences 6, 2 (2016), 185 – 193.
[46]
J. Tang, X. Liu, H. Cheng, and K. M. Robinette. 2011. Gender Recognition Using 3-D Human Body Shapes. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 41, 6 (Nov 2011), 898–908.
[47]
I. Ullah, M. Hussain, G. Muhammad, H. Aboalsamh, G. Bebis, and A. M. Mirza. 2012. Gender recognition from face images with local WLD descriptor. In 2012 ICMI’17, November 13–17, 2017, Glasgow, UK Mohamed Abouelenien, Verónica Pérez-Rosas, Rada Mihalcea, and Mihai Burzo 19th International Conference on Systems, Signals and Image Processing (IWSSIP). 417–420.
[48]
Paul Viola and MichaelJ. Jones. 2004. Robust Real-Time Face Detection. International Journal of Computer Vision 57, 2 (2004), 137–154.
[49]
Adam Vogel and Dan Jurafsky. 2012. Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries. Association for Computational Linguistics, Chapter He Said, She Said: Gender in the ACL Anthology, 33–41. http://aclweb. org/anthology/W12-3204

Cited By

View all
  • (2024)Beyond the visible: thermal data for facial soft biometric estimationEURASIP Journal on Image and Video Processing10.1186/s13640-024-00640-52024:1Online publication date: 6-Sep-2024
  • (2023)On the Role of Thermal Imaging in Automotive Applications: A Critical ReviewIEEE Access10.1109/ACCESS.2023.325511011(25152-25173)Online publication date: 2023
  • (2022)Identifikasi Jenis Kelamin Secara Real Time Berdasarkan Suara Pada Raspberry PiJurnal Komputer Terapan10.35143/jkt.v8i1.53208:1(158-167)Online publication date: 26-Jun-2022
  • Show More Cited By

Index Terms

  1. Multimodal gender detection

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction
    November 2017
    676 pages
    ISBN:9781450355438
    DOI:10.1145/3136755
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 03 November 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. gender detection
    2. linguistic
    3. multimodal
    4. physiological
    5. thermal
    6. visual
    7. vocal

    Qualifiers

    • Research-article

    Conference

    ICMI '17
    Sponsor:

    Acceptance Rates

    ICMI '17 Paper Acceptance Rate 65 of 149 submissions, 44%;
    Overall Acceptance Rate 453 of 1,080 submissions, 42%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)22
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Beyond the visible: thermal data for facial soft biometric estimationEURASIP Journal on Image and Video Processing10.1186/s13640-024-00640-52024:1Online publication date: 6-Sep-2024
    • (2023)On the Role of Thermal Imaging in Automotive Applications: A Critical ReviewIEEE Access10.1109/ACCESS.2023.325511011(25152-25173)Online publication date: 2023
    • (2022)Identifikasi Jenis Kelamin Secara Real Time Berdasarkan Suara Pada Raspberry PiJurnal Komputer Terapan10.35143/jkt.v8i1.53208:1(158-167)Online publication date: 26-Jun-2022
    • (2022)User Profiling Based on Nonlinguistic Audio DataACM Transactions on Information Systems10.1145/347482640:1(1-23)Online publication date: 31-Jan-2022
    • (2022)Leth-Gait: A Fusion Formula for Gait-Based Gender Classification Using Depth Images2022 International Conference on Computer Technologies (ICCTech)10.1109/ICCTech55650.2022.00008(1-6)Online publication date: Feb-2022
    • (2022)A real-time multi view gait-based automatic gender classification system using kinect sensorMultimedia Tools and Applications10.1007/s11042-022-13704-382:8(11993-12016)Online publication date: 16-Sep-2022
    • (2021)User Profiling based on Nonlinguistic Audio Data2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00241(2303-2308)Online publication date: Apr-2021
    • (2020)Towards detecting levels of alertness in drivers using multiple modalitiesProceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3389189.3389192(1-9)Online publication date: 30-Jun-2020
    • (2020)Performance estimation of the state-of-the-art convolution neural networks for thermal images-based gender classification systemJournal of Electronic Imaging10.1117/1.JEI.29.6.06300429:06Online publication date: 1-Nov-2020
    • (2020)Speech Age-Gender Classification Using Long Short-Term Memory2020 3rd International Conference on Information and Communications Technology (ICOIACT)10.1109/ICOIACT50329.2020.9331995(358-361)Online publication date: 24-Nov-2020
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media