Chee Wee Leong
ABSTRACT
Supervised systems require human labels for training. But, are humans themselves always impartial during the annotation process? We examine this question in the context of automated assessment of human behavioral tasks. Specifically, we investigate whether human ratings themselves can be trusted at their face value when scoring video-based structured interviews, and whether such ratings can impact machine learning models that use them as training data. We present preliminary empirical evidence that indicates there are biases in such annotations, most of which are visual in nature.
- Lei Chen, Gary Feng, Jilliam Joe, Chee Wee Leong, Christopher Kitchen, and Chong Min Lee. 2014. Towards automated assessment of public speaking skills using multimodal cues. In Proceedings of the 16th International Conference on Multimodal Interaction. ACM, 200–203.Google Scholar
Digital Library
- Lei Chen, Ru Zhao, Chee Wee Leong, Blair Lehman, Gary Feng, and Mohammed Ehsan Hoque. 2017. Automated video interview judgment on a large-sized corpus collected online. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 504–509.Google ScholarCross Ref
- Jose M Cortina, Nancy B Goldstein, Stephanie C Payne, H Kristl Davison, and Stephen W Gilliland. 2000. The incremental validity of interview scores over and above cognitive ability and conscientiousness scores. Personnel Psychology 53, 2 (2000), 325–351.Google ScholarCross Ref
- Allen I Huffcutt, James M Conway, Philip L Roth, and Nancy J Stone. 2001. Identification and meta-analytic assessment of psychological constructs measured in employment interviews.Journal of Applied Psychology 86, 5 (2001), 897.Google Scholar
- Julia Levashina, Christopher J Hartwell, Frederick P Morgeson, and Michael A Campion. 2014. The structured employment interview: Narrative and quantitative review of the research literature. Personnel Psychology 67, 1 (2014), 241–293.Google ScholarCross Ref
- Iftekhar Naim, M Iftekhar Tanveer, Daniel Gildea, and Mohammed Ehsan Hoque. 2015. Automated prediction and analysis of job interview performance: The role of what you say and how you say it. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 1. IEEE, 1–6.Google ScholarCross Ref
- Laurent Son Nguyen, Denise Frauendorfer, Marianne Schmid Mast, and Daniel Gatica-Perez. 2014. Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior. IEEE transactions on multimedia 16, 4 (2014), 1018–1031.Google Scholar
- Laurent Son Nguyen and Daniel Gatica-Perez. 2015. I would hire you in a minute: Thin slices of nonverbal behavior in job interviews. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM, 51–58.Google ScholarDigital Library
- Terence Parr. 2018. Feature importances for scikit random forests. https://github.com/parrt/random-forest-importancesGoogle Scholar
- Vikram Ramanarayanan, Chee Wee Leong, Lei Chen, Gary Feng, and David Suendermann-Oeft. 2015. Evaluating speech, face, emotion and body movement time-series features for automated multimodal presentation scoring. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM, 23–30.Google ScholarDigital Library
- Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 1135–1144.Google ScholarDigital Library
- Lisa M Schreiber, Gregory D Paul, and Lisa R Shibley. 2012. The development and test of the public speaking competence rubric. Communication Education 61, 3 (2012), 205–233.Google ScholarCross Ref
- Rutuja Ubale, Yao Qian, and Keelan Evanini. 2018. Exploring End-To-End Attention-Based Neural Networks For Native Language Identification. In 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 84–91.Google Scholar
- David H Wolpert. 1992. Stacked generalization. Neural networks 5, 2 (1992), 241–259.Google Scholar
Recommendations
Dyadic Behavior Analysis in Depression Severity Assessment Interviews
ICMI '14: Proceedings of the 16th International Conference on Multimodal InteractionPrevious literature suggests that depression impacts vocal timing of both participants and clinical interviewers but is mixed with respect to acoustic features. To investigate further, 57 middle-aged adults (men and women) with Major Depression Disorder ...
Modeling human annotation errors to design bias-aware systems for social stream processing
ASONAM '19: Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and MiningHigh-quality human annotations are necessary to create effective machine learning systems for social media. Low-quality human annotations indirectly contribute to the creation of inaccurate or biased learning systems. We show that human annotation ...
Human-centered neural reasoning for subjective content processing: Hate speech, emotions, and humor
AbstractSome tasks in content processing, e.g., natural language processing (NLP), like hate or offensive speech and emotional or funny text detection, are subjective by nature. Each human may perceive some content individually. The existing reasoning ...
Highlights- Human-centered neural architectures suitable for subjective NLP problems are introduced.
- Personalized NLP requires dedicated validation procedures.
- Personalized methods revealed their superiority over generalized approaches for 14 ...
Comments