Skip to main content
SpringerLink
Log in

Deception detection using multimodal fusion approaches

  • 1186: Pattern Recognition and Artificial Intelligence based Multimedia Systems and Applications
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Automatic deception detection is an important task that has gained a huge interest in different fields due to its potential applications. Particularly, it can improve justice and security in society by helping in detecting deceivers in high-stakes situations across jurisprudence, law enforcement, and national security domains, among others. However, the existing deception detection systems until today are not as accurate as it is expected, which makes their use very risky especially in critical fields. This article outlines an approach for automatically distinguishing between deceit and truth based on audio, video and text modalities and explores the possibility of combining them together in order to detect deception more accurately. First, each modality has been evaluated separately and then a feature and decision-level fusion approaches have been proposed to combine the considered modalities. The proposed feature level fusion approach investigates a diversified feature selection techniques to select the most relevant ones among the whole used feature set, while the decision level fusion approach is based on the belief theory considering information about the certainty degree of each modality. To do so, we used a real-life video dataset of people communicating truthfully or deceptively collected from public american court trials. Unimodal models trained on audio, video and text separately achieved an accuracy rate of 60%, 94% and 58% respectively. When using the feature level fusion approach, the best accuracy deception detection result reaches 93% using only 19 combined features, while a 100% deception recognition rate has been obtained with the decision-level fusion proposed approach, outperforming the results obtained in the literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Abouelenien M, Pérez-Rosas V, Mihalcea R, Burzo M (2014) Deception detection using a multimodal approach. In: Proc. of international conference on multimodal interaction, pp 58–65

  2. Abouelenien M, Pérez-Rosas V, Mihalcea R, Burzo M (2016) Detecting deceptive behavior via integration of discriminative features from multiple modalities. IEEE Trans Inform Forens Secur 12(5):1042–1055

    Article  Google Scholar 

  3. Adelson R (2004) Detecting deception. Monitor Psychol 35(7):70

    Google Scholar 

  4. Ahmad K, Mekhalfi ML, Conci N, Melgani F, Natale FD (2018) Ensemble of deep models for event recognition. ACM Trans Multimed Comput Commun Appl (TOMM) 14(2):1–20

    Article  Google Scholar 

  5. Avola D, Cinque L, De Marsico M, Fagioli A, Foresti GL (2020) Lietome: Preliminary study on hand gestures for deception detection via fisher-lstm. Pattern Recogn Lett 138:455–461

    Article  Google Scholar 

  6. Baltrušaitis T, Robinson P, Morency L-P (2016) Openface: an open source facial behavior analysis toolkit. In: 2016 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 1–10

  7. Ben-Shakhar G (2002) A critical review of the control questions test (cqt). Handbook of polygraph testing 103–126

  8. Bond CF Jr, DePaulo BM (2006) Accuracy of deception judgments. Person Soc Psychol Rev 10(3):214–234

    Article  Google Scholar 

  9. Burgoon JK, Kelley DL, Newton DA, Keeley-Dyreson MP (1989) The nature of arousal and nonverbal indices. Human Commun Res 16(2):217–255

    Article  Google Scholar 

  10. Burgoon JK, Twitchell DP, Jensen ML, Meservy TO, Adkins M, Kruse J, Deokar AV, Tsechpenakis G, Lu S, Metaxas DN et al (2009) Detecting concealment of intent in transportation screening: a proof of concept. Trans Intell Transport Syst 10(1):103–112

    Article  Google Scholar 

  11. Caso L, Maricchiolo F, Bonaiuto M, Vrij A, Mann S (2006) The impact of deception and suspicion on different hand movements. J Nonverb Behav 30(1):1–19

    Article  Google Scholar 

  12. Chebbi S, Jebara SB (2018) On the use of pitch-based features for fear emotion detection from speech. In: Proc. of international conference on advanced technologies for signal and image processing (ATSIP). IEEE, pp 1–6

  13. Chebbi S, Jebara SB (2018) On the selection of relevant features for fear emotion detection from speech. In: Proc. of international symposium on signal, image, video and communications (ISIVC). IEEE, pp 82–86

  14. Chebbi S, Jebara SB (2020) An audio-visual based feature level fusion approach applied to deception detection. In: Proc. of international joint conference on computer vision, imaging and computer graphics theory and applications (VISAPP), pp 1–9

  15. Chebbi S, Jebara SB (2020) Towards detecting simultaneous fear emotion and deception behavior in speech. In: Proc. of international joint conference on computer vision imaging and computer graphics theory and applications (VISAPP), pp 1–10

  16. Cohen D, Beattie G, Shovelton H (2010) Nonverbal indicators of deception: How iconic gestures reveal thoughts that cannot be suppressed. Semiotica 2010(182):133–174

    Article  Google Scholar 

  17. Conroy NJ, Rubin VL, Chen Y (2015) Automatic deception detection: Methods for finding fake news. Proc. Assoc Inform Sci Technol 52(1):1–4

    Article  Google Scholar 

  18. Crockett K, O’Shea J, Khan W (2020) Automated deception detection of males and females from non-verbal facial micro-gestures. In: 2020 international joint conference on neural networks (IJCNN). IEEE, pp 1–7

  19. De Silva LC, Ng PC (2000) Bimodal emotion recognition. In: Proc. of international conference on automatic face and gesture recognition (ICAFGR). IEEE, pp 332–335

  20. DePaulo BM, Lindsay JJ, Malone BE, Muhlenbruck L, Charlton K, Cooper H (2003) Cues to deception. Psychol Bullet 129(1):74

    Article  Google Scholar 

  21. Dibazar AA, Narayanan S, Berger TW (2002) Feature analysis for automatic detection of pathological speech. In: Proc. of annual conference and annual fall meeting of the biomedical engineering society Engineering in Medicine and Biology, vol 1. IEEE, pp 182–183

  22. Ekman P (2001) Telling lies clues to deceit in the marketplace, politics, and marriage revised edition

  23. Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32(1):88–106

    Article  Google Scholar 

  24. Ekman P, Friesen WV (1972) Hand movements. J Commun 22 (4):353–374

    Article  Google Scholar 

  25. Ekman P, Friesen WV, Scherer KR (1976) Body movement and voice pitch in deceptive interaction. Semiotica 16(1):23–28

    Article  Google Scholar 

  26. Ekman P, O’Sullivan M, Friesen WV, Scherer KR (1991) Face, voice, and body in detecting deceit. J Nonverb Behav 15(2):125–135

    Article  Google Scholar 

  27. Eyben F, Schuller B (2015) opensmile:) the munich open-source large-scale multimedia feature extractor. ACM SIGMultimedia Records 6(4):4–13

    Article  Google Scholar 

  28. Feng S, Banerjee R, Choi Y (2012) Syntactic stylometry for deception detection. In: Proc. of annual meeting of the association for computational linguistics, Association for computational linguistics, pp 171–175

  29. Forkosch MD (1975) The lie detector and mechanical jurisprudence. Oklahoma Law Rev 28(2):288–311

    Google Scholar 

  30. Fox Tree JE (2002) Interpreting pauses and ums at turn exchanges. Discourse Process 34(1):37–55

    Article  Google Scholar 

  31. Francis M, Booth RJ (1993) Linguistic inquiry and word count, Dallas TX: Southern Methodist University

  32. Gogate M, Adeel A, Hussain A (2017) Deep learning driven multimodal fusion for automated deception detection. In: Proc. of symposium series on computational intelligence (SSCI). IEEE, pp 1–6

  33. Granhag PA, Strömwall LA (2004) The detection of deception in forensic contexts. Cambridge University Press, Cambridge

    Book  Google Scholar 

  34. Hartwig M, Granhag PA, Strömwall LA, Kronkvist O (2006) Strategic use of evidence during police interviews: when training to detect deception works. Law Human Behav 30(5):603–619

    Article  Google Scholar 

  35. Hassan SZ, Ahmad K, Al-Fuqaha A, Conci N (2019) Sentiment analysis from images of natural disasters. In: International conference on image analysis and processing. Springer, pp 104–113

  36. Hirschberg JB, Benus S, Brenier JM, Enos F, Friedman S, Gilman S, Girand C, Graciarena M, Kathol A, Michaelis L et al (2005) Distinguishing deceptive from non-deceptive speech. In: Proc. of INTERSPEECH conference, pp 1833–1836

  37. Honts CR, Raskin DC, Kircher JC (1987) Effects of physical countermeasures and their electromyographic detection during polygraph tests for deception. Journal of Psychophysiology

  38. Jaiswal M, Tabibu S, Bajpai R (2016) The truth and nothing but the truth: Multimodal analysis for deception detection. In: Proc. of international conference on data mining workshops (ICDMW). IEEE, pp 938–943

  39. Jensen ML, Meservy TO, Burgoon JK, Nunamaker JF (2010) Automatic, multimodal evaluation of human interaction. Group Decis Negot 19 (4):367–389

    Article  Google Scholar 

  40. Jensen ML, Meservy TO, Kruse J, Burgoon JK, Nunamaker JF (2005) Identification of deceptive behavioral cues extracted from video. In: Proceedings. 2005 IEEE intelligent transportation systems, vol 2005. IEEE, pp 1135–1140

  41. Jordan S, Brimbal L, Wallace DB, Kassin SM, Hartwig M, Street CN (2019) A test of the micro-expressions training tool: Does it improve lie detection? J Investig Psychol Offender Profiling 16(3):222–235

    Article  Google Scholar 

  42. Karimi H, Tang J, Li Y (2018) Toward end-to-end deception detection in videos, in 2018. In: IEEE international conference on big data (big data). IEEE, 1278–1283

  43. Kirchhübel C., Howard DM (2013) Detecting suspicious behavior using speech: acoustic correlates of deceptive speech–an exploratory investigation. Appl Ergon 44(5):694–702

    Article  Google Scholar 

  44. Köhnken G (1996) Social psychology and the law. Appl Soc Psychol 257–282

  45. Lee B (1975) The polygraph and pre-employment screening. Hous L Rev 13:551

    Google Scholar 

  46. Levitan SI, An G, Ma M, Levitan R, Rosenberg A, Hirschberg J (2016) Combining acoustic-prosodic, lexical, and phonotactic features for automatic deceptiondetection. In: Proc. of INTERSPEECH conference, pp 2006–2010

  47. Levitan SI, Maredia A, Hirschberg J (2018) Acoustic-prosodic indicators of deception and trust in interview dialogues. In: Proc. of INTERSPEECH conference, pp 416–420

  48. Lu S, Tsechpenakis G, Metaxas DN, Jensen ML, Kruse J (2005) Blob analysis of the head and hands: A method for deception detection. In: Proc. of annual hawaii international conference on system sciences. IEEE, p 20c

  49. Maschke GW, Scalabrini GJ (2005) The lie behind the lie detector Antipolygraph org

  50. Masip J, Bethencourt M, Lucas G, Segundo MS, Herrero C (2012) Deception detection from written accounts. Scandin J Psychol 53(2):103–111

    Article  Google Scholar 

  51. Mbaziira A, Jones J (2016) A text-based deception detection model for cybercrime. In: Proc of internation conference on technology and management

  52. Mendels G, Levitan SI, Lee K-Z, Hirschberg J (2017) Hybrid acoustic-lexical deep learning approach for deception detection. In: Proc. of INTERSPEECH conference, pp 1472–1476

  53. Meservy TO, Jensen ML, Kruse J, Burgoon JK, Nunamaker JF, Twitchell DP, Tsechpenakis G, Metaxas DN (2005) Deception detection through automatic, unobtrusive analysis of nonverbal behavior. IEEE Intell Syst 20(5):36–43

    Article  Google Scholar 

  54. Mihalcea R, Strapparava C (2009) The lie detector: explorations in the automatic recognition of deceptive language. In: Proc. of international joint conference on natural language processing (IJCNLP). Association for computational linguistics, pp 309–312

  55. Motley MT (1974) Acoustic correlates of lies. Western J Commun (Includes Commun Rep) 38(2):81–87

    Google Scholar 

  56. Newman ML, Pennebaker JW, Berry DS, Richards JM (2003) Lying words: predicting deception from linguistic styles. Personal Soc Psychol Bullet 29 (5):665–675

    Article  Google Scholar 

  57. Nunamaker JF, Burgoon JK, Twyman NW, Proudfoot JG, Schuetzler R, Giboney JS (2012) Establishing a foundation for automated human credibility screening. In: Proc. of international conference on intelligence and security informatics. IEEE, pp 202–211

  58. Owayjan M, Kashour A, Al Haddad N, Fadel M, Al Souki G (2012) The design and development of a lie detection system using facial micro-expressions. In: Proc. of International conference on advances in computational tools for engineering applications (ACTEA). IEEE, pp 33–38

  59. Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543

  60. Pérez-Rosas V, Abouelenien M, Mihalcea R, Burzo M (2015) Deception detection using real-life trial data. In: Proc. of ACM on international conference on multimodal interaction, pp 59–66

  61. Pérez-Rosas V, Abouelenien M, Mihalcea R, Xiao Y, Linton C, Burzo M (2015) Verbal and nonverbal clues for real-life deception detection. In: Proc. of conference on empirical methods in natural language processing, pp 2336–2346

  62. Porter S, Ten Brinke L (2010) The truth about lies: what works in detecting high-stakes deception? Legal Criminol Psychol 15(1):57–75

    Article  Google Scholar 

  63. Porter S, Yuille JC (1996) The language of deceit: an investigation of the verbal clues to deception in the interrogation context. Law Human Behav 20 (4):443–458

    Article  Google Scholar 

  64. Qi J, Yang P, Newcombe L, Peng X, Yang Y, Zhao Z (2020) An overview of data fusion techniques for internet of things enabled physical activity recognition and measure. Inform Fusion 55:269–280

    Article  Google Scholar 

  65. Qin T, Burgoon J, Nunamaker JF (2004) An exploratory study on promising cues in deception detection and application of decision tree. In: Proc. of annual hawaii international conference on system sciences. IEEE, pp 23–32

  66. Randhavane T, Bhattacharya U, Kapsaskis K, Gray K, Bera A, Manocha D (2019) The liar’s walk: Detecting deception with gait and gesture, arXiv:1912.06874

  67. Rill-Garcia R, Jair Escalante H, Villasenor-Pineda L, Reyes-Meza V (2019) High-level features for multimodal deception detection in videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 0–0

  68. Schuetze P, Zeskind PS (2001) Relations between women’s depressive symptoms and perceptions of infant distress signals varying in pitch. Infancy 2 (4):483–499

    Article  Google Scholar 

  69. Sen UM, Perez-Rosas V, Yanikoglu B, Abouelenien M, Burzo M, Mihalcea R (2020) Multimodal deception detection using real-life trial data. IEEE Transactions on Affective Computing

  70. Shafer G (1976) A mathematical theory of evidence, vol 42. Princeton University Press, Princeton

    Book  MATH  Google Scholar 

  71. Sharma R, Pavlović VI, Huang TS (2002) Toward multimodal human–computer interface. In: Advances in image processing and understanding: a festschrift for Thomas S Huang. World Scientific, pp 349–365

  72. Sondhi S, Vijay R, Khan M, Salhan AK (2016) Voice analysis for detection of deception. In: Proc. of international conference on knowledge, information and creativity support systems (KICSS). IEEE, pp 1–6

  73. Sporer SL (1997) The less travelled road to truth: Verbal cues in deception detection in accounts of fabricated and self-experienced events. Appl Cognit Psychol Official J Soc Appl Res Memory Cognit 11(5):373–397

    Google Scholar 

  74. Steller M (1989) Criteria-based content analysism. Psychological methods in criminal investigation and evidence 217–245

  75. Streeter LA, Krauss RM, Geller V, Olson C, Apple W (1977) Pitch changes during attempted deception. J Personal Soc Psychol 35(5):345

    Article  Google Scholar 

  76. Su L, Levine MD (2014) High-stakes deception detection based on facial expressions. In: 2014 22nd international conference on pattern recognition. IEEE, pp 2519–2524

  77. Talkin D, Kleijn WB (1995) A robust algorithm for pitch tracking (rapt). Speech coding and synthesis 497–518

  78. Tausczik YR, Pennebaker JW (2010) The psychological meaning of words: Liwc and computerized text analysis methods. J Lang Soc Psychol 29(1):24–54

    Article  Google Scholar 

  79. Thannoon HH, Ali WH, Hashim IA (2019) Design and implementation of deception detection system based on reliable facial expression. J Eng Appl Sci 14(15):5002–5011

    Article  Google Scholar 

  80. Tian Y-L, Kanade T, Cohn JF (2005) Facial expression analysis. In: Handbook of face recognition. Springer, pp 247–275

  81. Tsechpenakis G, Metaxas D, Adkins M, Kruse J, Burgoon JK, Jensen ML, Meservy T, Twitchell DP, Deokar A, Nunamaker JF (2005) Hmm-based deception recognition from visual cues. In: Proc. of international conference on multimedia and expo. IEEE, pp 824–827

  82. Undeutsch U (1967) Beurteilung der glaubhaftigkeit von aussagen. Handbuch der psychologie 11:26–181

    Google Scholar 

  83. Undeutsch U (1989) The development of statement reality analysis. In: Credibility assessment. Springer, pp 101–119

  84. Ververidis D, Kotropoulos C, Pitas I (2004) Automatic emotional speech classification. In: Proc. of international conference on acoustics, speech, and signal processing, vol 1. IEEE, pp I–593

  85. Vrij A (1998) Nonverbal communication and credibility. In: Psychology and law: truthfulness, accuracy and credibility. McGraw-Hill, New York, pp 32–53

  86. Vrij A (2000) Detecting lies and deceit. The psychology of lying and implications for professional practice. Wiley, Hoboken

    Google Scholar 

  87. Vrij A, Edward K, Roberts KP, Bull R (2000) Detecting deceit via analysis of verbal and nonverbal behavior. J Nonverb Behav 24(4):239–263

    Article  Google Scholar 

  88. Vrij A, Semin GR (1996) Lie experts’ beliefs about nonverbal indicators of deception. J Nonverb Behav 20(1):65–80

    Article  Google Scholar 

  89. Wagner J, André E, Jung F (2009) Smart sensor integration: A framework for multimodal emotion recognition in real-time. In: IEEE 3rd international conference on affective computing and intelligent interaction and workshops, pp 1–8

  90. Wilcox DT, Wilcox T, Wiley CJ (2009) The use of the polygraph in assessing, treating and supervising sex offenders. A practitioner’s guide, International Journal of Police Science and Management

  91. Wilson N (1993) Decision-making with belief functions and pignistic probabilities. In: European conference on symbolic and quantitative approaches to reasoning and uncertainty. Springer, pp 364–371

  92. Wu Z, Singh B, Davis LS, Subrahmanian V (2018) Deception detection in videos. In: Proc of AAAI conference on artificial intelligence

  93. Yuzsever S (2015) Mutual information based feature selection for acoustic and diagnosis. Ph.D. dissertation, Bogaziċi University

  94. Zhang J, Levitan SI, Hirschberg J (2020) Multimodal deception detection using automatically extracted acoustic, visual, and lexical features. In: Proc Interspeech, vol 2020, pp 359–363

  95. Zhang Z, Singh V, Slowe TE, Tulyakov S, Govindaraju V (2007) Real-time automatic deceit detection from involuntary facial expressions. In: Proc. of conference on computer vision and pattern recognition (CCVPR). IEEE, pp 1–6

  96. Zhou L, Twitchell DP, Qin T, Burgoon JK, Nunamaker JF (2003) An exploratory study into deception detection in text-based computer-mediated communication. In: Proc. of annual hawaii international conference on system sciences. IEEE, pp 10–pp

  97. Zloteanu M (2019) Emotion recognition and deception detection

Download references

Author information

Authors and Affiliations

  1. University of Carthage, SUP’COM, 2083, Ariana, Tunisia

    Safa Chebbi & Sofia Ben Jebara

Authors
  1. Safa Chebbi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sofia Ben Jebara
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Safa Chebbi.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 8 Features investigated for audio, video and text modalities
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chebbi, S., Jebara, S.B. Deception detection using multimodal fusion approaches. Multimed Tools Appl 82, 13073–13102 (2023). https://doi.org/10.1007/s11042-021-11148-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11148-9

Keywords

Search

Navigation