research-article

Automatic assessment of communication skill in non-conventional interview settings: a comparative study

Authors:

Pooja Rao S. B,

Sowmya Rasipuram,

Dinesh Babu JayagopiAuthors Info & Claims

ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

Pages 221 - 229

https://doi.org/10.1145/3136755.3136756

Published: 03 November 2017 Publication History

Abstract

Effective communication is an important social skill that facilitates us to interpret and connect with people around us and is of utmost importance in employment based interviews. This paper presents a methodical study and automatic measurement of communication skill of candidates in different modes of behavioural interviews. It demonstrates a comparative analysis of non-conventional methods of employment interviews namely 1) Interface-based asynchronous video interviews and 2) Written interviews (including a short essay). In order to achieve this, we have collected a dataset of 100 structured interviews from participants. These interviews are evaluated independently by two human expert annotators on rubrics specific to each of the settings. We, then propose a predictive model using automatically extracted multimodal features like audio, visual and lexical, applying classical machine learning algorithms. Our best model performs with an accuracy of 75% for a binary classification task in all the three contexts. We also study the differences between the expert perception and the automatic prediction across the settings.

References

[1]

2016. Discover Big Voice Intelligence. https://www.voicebase.com/ products-features/. (2016).

[2]

Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater® V. 2. The Journal of Technology, Learning and Assessment 4, 3 (2006).

[3]

Ligia Maria Batrinca, Nadia Mana, Bruno Lepri, Fabio Pianesi, and Nicu Sebe. 2011. Please, tell me about yourself: automatic personality assessment using short self-presentations. In Proceedings of the 13th international conference on multimodal interfaces. ACM, 255–262.

Digital Library

[4]

Joan-Isaac Biel, Lucía Teijeiro-Mosquera, and Daniel Gatica-Perez. 2012. Facetube: predicting personality from facial expressions of emotion in online conversational video. In Proceedings of the 14th ACM international conference on Multimodal interaction. ACM, 53–56.

Digital Library

[5]

Paulus Petrus Gerardus Boersma et al. 2002. Praat, a system for doing phonetics by computer. Glot international 5 (2002).

[6]

Steven Burrows, Iryna Gurevych, and Benno Stein. 2015. The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education 25, 1 (2015), 60–117.

[7]

Timothy DeGroot and Janaki Gooty. 2009. Can nonverbal cues be used to make meaningful personality attributions in employment interviews? Journal of Business and Psychology 24, 2 (2009), 179–192.

[8]

Scott Elliot. 2003. IntelliMetric: From here to validity. Automated essay scoring: A cross-disciplinary perspective (2003), 71–86.

[9]

Florian Eyben, Felix Weninger, Florian Gross, and Björn Schuller. 2013. Recent developments in opensmile, the munich open-source multimedia feature extractor. In Proceedings of the 21st ACM international conference on Multimedia. ACM, 835– 838.

Digital Library

[10]

Noura Farra, Swapna Somasundaran, and Jill Burstein. 2015. Scoring persuasive essays using opinions and their targets. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications. 64–74.

[11]

Michelle Fung, Yina Jin, RuJie Zhao, and Mohammed Ehsan Hoque. 2015. ROC speak: semi-automated personalized feedback on nonverbal behavior from recorded videos. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 1167–1178.

Digital Library

[12]

Theodoros Giannakopoulos. 2015. pyAudioAnalysis: An Open-Source Python Library for Audio Signal Analysis. PloS one 10, 12 (2015).

[13]

Theodoros Giannakopoulos and Aggelos Pikrakis. 2014. Introduction to Audio Analysis: A MATLAB® Approach. Academic Press.

Digital Library

[14]

Derrick Higgins, Chris Brew, Michael Heilman, Ramon Ziai, Lei Chen, Aoife Cahill, Michael Flor, Nitin Madnani, Joel Tetreault, Daniel Blanchard, et al. 2014. Is getting the right answer just about choosing the right words? The role of syntactically-informed features in short answer scoring. arXiv preprint arXiv:1403.0801 (2014).

[15]

Mohammed Ehsan Hoque, Matthieu Courgeon, Jean-Claude Martin, Bilge Mutlu, and Rosalind W Picard. 2015. Mach: My automated conversation coach. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, 385–396.

Digital Library

[16]

Allen I Huffcutt, James M Conway, Philip L Roth, and Nancy J Stone. 2001. Identification and meta-analytic assessment of psychological constructs measured in employment interviews. Journal of Applied Psychology 86, 5 (2001), 897.

[17]

Miron B Kursa and Witold R Rudnicki. 2010. Feature Selection with the Boruta Package. (2010).

[18]

Thomas K Landauer, Darrell Laham, and Peter W Foltz. 2000. The intelligent essay assessor. (2000).

[19]

Gwen Littlewort, Jacob Whitehill, Tingfan Wu, Ian Fasel, Mark Frank, Javier Movellan, and Marian Bartlett. 2011. The computer expression recognition toolbox (CERT). In Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on. IEEE, 298–305.

[20]

Detmar Meurers, Ramon Ziai, Niels Ott, and Janina Kopp. 2011. Evaluating answers to reading comprehension questions in context: Results for German and the role of information structure. In Proceedings of the TextInfer 2011 Workshop on Textual Entailment. Association for Computational Linguistics, 1–9.

Digital Library

[21]

George A Miller. 1995. WordNet: a lexical database for English. Commun. ACM 38, 11 (1995), 39–41.

Digital Library

[22]

Michael Mohler, Razvan Bunescu, and Rada Mihalcea. 2011. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, 752–762.

Digital Library

[23]

Iftekhar Naim, M Iftekhar Tanveer, Daniel Gildea, and Mohammed Ehsan Hoque. 2015. Automated prediction and analysis of job interview performance: The role of what you say and how you say it. In Automatic Face and Gesture Recognition (FG), 2015 11th IEEE International Conference and Workshops on, Vol. 1. IEEE, 1–6.

[24]

Laurent Son Nguyen, Denise Frauendorfer, Marianne Schmid Mast, and Daniel Gatica-Perez. 2014. Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior. IEEE transactions on multimedia 16, 4 (2014), 1018–1031.

Digital Library

[25]

Ellis B Page. 1966. The imminence of... grading essays by computer. The Phi Delta Kappan 47, 5 (1966), 238–243.

[26]

Ellis Batten Page. 1994. Computer grading of student prose, using modern concepts and software. The Journal of experimental education 62, 2 (1994), 127– 142.

[27]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, Oct (2011), 2825–2830.

Digital Library

[28]

James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: LIWC 2001.

[29]

Mahway: Lawrence Erlbaum Associates 71, 2001 (2001), 2001.

[30]

Lakshmi Ramachandran, Jian Cheng, and Peter Foltz. 2015. Identifying patterns for short answer scoring using graph-based lexico-semantic text matching. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications. 97–106.

[31]

Sowmya Rasipuram and Dinesh Babu Jayagopi. 2016. Automatic assessment of communication skill in interface-based employment interviews using audiovisual cues. In Multimedia & Expo Workshops (ICMEW), 2016 IEEE International Conference on. IEEE, 1–6.

[32]

Sowmya Rasipuram, Dinesh Babu Jayagopi, et al. 2016. Asynchronous video interviews vs. face-to-face interviews for communication skill measurement: a systematic study. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. ACM, 370–377.

Digital Library

[33]

Lawrence M Rudner and Tahung Liang. 2002. Automated essay scoring using Bayes’ theorem. The Journal of Technology, Learning and Assessment 1, 2 (2002).

[34]

Brian H Spitzberg and Thomas W Adams. 2007. CSRS, the conversational skills rating scale: an instructional assessment of interpersonal competence. NCA, National Communication Association.

[35]

Jana Zuheir Sukkarieh and John Blackmore. 2009. c-rater: Automatic Content Scoring for Short Constructed Responses. In FLAIRS Conference. 290–295.

[36]

Jana Z Sukkarieh and Stephen G Pulman. 2005. Information extraction and machine learning: Auto-marking short free text responses to science questions. In Proceedings of the 2005 conference on artificial intelligence in education: Supporting learning through intelligent and socially informed technology. IOS Press, 629–637.

Digital Library

[37]

Hiroki Tanaka, Sakriani Sakti, Graham Neubig, Tomoki Toda, Hideki Negoro, Hidemi Iwasaka, and Satoshi Nakamura. 2013. Automated social skills trainer. In Proceedings of the 20th International Conference on Intelligent User Interfaces. ACM, 697–706.

Digital Library

[38]

Louis Tandalla. 2012. Scoring Short Answer Essays. (2012).

[39]

https://kaggle2.blob.core.windows.net/competitions/kaggle/2959/media/ TechnicalMethodsPaper.pdf.

[40]

M Iftekhar Tanveer, Emy Lin, and Mohammed Ehsan Hoque. 2015. Rhema: A real-time in-situ intelligent interface to help people with public speaking. In Proceedings of the 20th International Conference on Intelligent User Interfaces. ACM, 286–295.

Digital Library

[41]

M Iftekhar Tanveer, Ru Zhao, Kezhen Chen, Zoe Tiet, and Mohammed Ehsan Hoque. 2016. Automanner: An automated interface for making public speakers aware of their mannerisms. In Proceedings of the 21st International Conference on Intelligent User Interfaces. ACM, 385–396.

Digital Library

[42]

Yla R Tausczik and James W Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology 29, 1 (2010), 24–54.

[43]

Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the conference on human language technology and empirical methods in natural language processing. Association for Computational Linguistics, 347–354.

Digital Library

Cited By

Rao PRenier LBoldi MMast MJayagopi DCherubini M(2024)On the potential of supporting autonomy in online video interview training platformsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103326191:COnline publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103326
Kleinlogel ESchmid Mast MJayagopi DShubham KButera A(2023)“The interviewer is a machine!” Investigating the effects of conventional and technology‐mediated interview methods on interviewee reactions and behaviorInternational Journal of Selection and Assessment10.1111/ijsa.1243331:3(403-419)Online publication date: 8-May-2023
https://doi.org/10.1111/ijsa.12433
Asan USoyer A(2022)A Weighted Bonferroni-OWA Operator Based Cumulative Belief Degree Approach to Personnel Selection Based on Automated Video Interview Assessment DataMathematics10.3390/math1009158210:9(1582)Online publication date: 7-May-2022
https://doi.org/10.3390/math10091582
Show More Cited By

Index Terms

Automatic assessment of communication skill in non-conventional interview settings: a comparative study
1. Applied computing
  1. Law, social and behavioral sciences
2. Information systems
  1. Information retrieval
    1. Document representation
      1. Content analysis and feature selection
    2. Specialized information retrieval
      1. Multimedia and multimodal retrieval
        Speech / audio search

Recommendations

Automatic assessment of communication skill in interview-based interactions

Understanding and modeling people's behavior in social interactions is an important problem in Social Computing. In this work, we automatically predict the communication skill of a person in two kinds of interview-based social interactions namely ...
Prediction/Assessment of communication skill using multimodal cues in social interactions
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

Understanding people’s behavior in social interactions is a very interesting problem in Social Computing. In this work, we automatically predict the communication skill of a person in various kinds of social interactions. We consider in particular, 1) ...
Integrating communication skills into the computer science curriculum
SIGCSE '12: Proceedings of the 43rd ACM technical symposium on Computer Science Education

Computer Science majors must be able to communicate effectively. Industry surveys identify the development of communication and critical thinking skills as key to the reform of the higher education sector. However, academics are challenged by time and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

November 2017

676 pages

ISBN:9781450355438

DOI:10.1145/3136755

General Chairs:
Edward Lank
University of Waterloo, Canada
,
Alessandro Vinciarelli
University of Glasgow, UK
,
Program Chairs:
Eve Hoggan
Aarhus University, Denmark
,
Sriram Subramanian
University of Sussex, UK
,
Stephen A. Brewster
University of Glasgow, UK

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '17

Sponsor:

SIGCHI

ICMI '17: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 13 - 17, 2017

Glasgow, UK

Acceptance Rates

ICMI '17 Paper Acceptance Rate 65 of 149 submissions, 44%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
368
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)1

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rao PRenier LBoldi MMast MJayagopi DCherubini M(2024)On the potential of supporting autonomy in online video interview training platformsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103326191:COnline publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103326
Kleinlogel ESchmid Mast MJayagopi DShubham KButera A(2023)“The interviewer is a machine!” Investigating the effects of conventional and technology‐mediated interview methods on interviewee reactions and behaviorInternational Journal of Selection and Assessment10.1111/ijsa.1243331:3(403-419)Online publication date: 8-May-2023
https://doi.org/10.1111/ijsa.12433
Asan USoyer A(2022)A Weighted Bonferroni-OWA Operator Based Cumulative Belief Degree Approach to Personnel Selection Based on Automated Video Interview Assessment DataMathematics10.3390/math1009158210:9(1582)Online publication date: 7-May-2022
https://doi.org/10.3390/math10091582
Thakkar JRao PShubham KJain VJayagopi D(2022)Understanding Interviewees’ Perceptions and Behaviour towards Verbally and Non-verbally Expressive Virtual Interviewing AgentsCompanion Publication of the 2022 International Conference on Multimodal Interaction10.1145/3536220.3558802(61-69)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536220.3558802
Griswold KPhillips JKim MMondragon NLiff JGully S(2021)Global differences in applicant reactions to virtual interview synchronicityThe International Journal of Human Resource Management10.1080/09585192.2021.191764133:15(2991-3018)Online publication date: 1-Jul-2021
https://doi.org/10.1080/09585192.2021.1917641
Inoue KHara KLala DNakamura STakanashi KTatsuya K(2020)A Job Interview Dialogue System That Asks Follow-up Questions: Implementation and Evaluation with an Autonomous Android掘り下げ質問を行う就職面接対話システムの自律型アンドロイドでの実装と評価Transactions of the Japanese Society for Artificial Intelligence10.1527/tjsai.35-5_D-K4335:5(D-K43_1-10)Online publication date: 1-Sep-2020
https://doi.org/10.1527/tjsai.35-5_D-K43
Rasipuram SJayagopi D(2020)Automatic multimodal assessment of soft skills in social interactions: a reviewMultimedia Tools and Applications10.1007/s11042-019-08561-6Online publication date: 24-Jan-2020
https://doi.org/10.1007/s11042-019-08561-6

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten