Abstract
The ability to detect lies is a crucial skill in essential situations like police interrogations and court trials. At present, several devices, such as polygraphs and magnetic resonance, can ease the deception detection task. However, the effectiveness of these tools can be compromised by intentional behavioral changes due to the subject awareness of such appliances, suggesting that alternative ways must be explored to detect lies without using physical devices. In this context, this paper presents an approach focused on the extraction of meaningful features from hand gestures. The latter provide cues on the person’s behavior and are used to address the deception detection task in RGB videos of trials. Specifically, the proposed system extracts hands skeletons from an RGB video sequence and generates novel handcrafted features from the extrapolated keypoints to reflect the subject behavior through hand movements. Then, a long short-term memory (LSTM) neural network is used to classify these features and estimate whether the person is lying or not. Extensive experiments were performed to assess the quality of the derived features on a public collection of famous real-life trials. On this dataset, the proposed system sets new state-of-the-art performance on the unimodal hand-gesture deception detection task, demonstrating the effectiveness of the proposed approach and its handcrafted features.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alaskar, H., Sbaï, Z., Khan, W., Hussain, A., Alrawais, A.: Intelligent techniques for deception detection: a survey and critical study. Soft. Comput. 27(7), 3581–3600 (2023)
Avola, D., Bacciu, A., Cinque, L., Fagioli, A., Marini, M.R., Taiello, R.: Study on transfer learning capabilities for pneumonia classification in chest-x-rays images. Comput. Methods Programs Biomed. 221, 106833 (2022)
Avola, D., Cascio, M., Cinque, L., Fagioli, A., Foresti, G.L.: LieToMe: an ensemble approach for deception detection from facial cues. Int. J. Neural Syst. 31(02), 2050068 (2021)
Avola, D., Cascio, M., Cinque, L., Fagioli, A., Foresti, G.L.: Affective action and interaction recognition by multi-view representation learning from handcrafted low-level skeleton features. Int. J. Neural Syst. 32, 2250040 (2022)
Avola, D., Cinque, L., De Marsico, M., Fagioli, A., Foresti, G.L.: LieToMe: preliminary study on hand gestures for deception detection via fisher-LSTM. Pattern Recogn. Lett. 138, 455–461 (2020)
Avola, D., Cinque, L., Fagioli, A., Filetti, S., Grani, G., Rodolà, E.: Multimodal feature fusion and knowledge-driven learning via experts consult for thyroid nodule classification. IEEE Trans. Circuits Syst. Video Technol. 32(5), 2527–2534 (2021)
Avola, D., Cinque, L., Fagioli, A., Foresti, G.L.: Sire-networks: convolutional neural networks architectural extension for information preservation via skip/residual connections and interlaced auto-encoders. Neural Netw. 153, 386–398 (2022)
Avola, D., et al.: Medicinal boxes recognition on a deep transfer learning augmented reality mobile application. In: Proceedings of the International Conference on Image Analysis and Processing (ICIAP), pp. 489–499 (2022)
Avola, D., Cinque, L., Fagioli, A., Foresti, G.L., Massaroni, C.: Deep temporal analysis for non-acted body affect recognition. IEEE Trans. Affect. Comput. 13(3), 1366–1377 (2020)
Avola, D., Cinque, L., Foresti, G.L., Pannone, D.: Automatic deception detection in RGB videos using facial action units. In: Proceedings of the International Conference on Distributed Smart Cameras (ICDSC), pp. 1–6 (2019)
Bond, C.F., Jr., DePaulo, B.M.: Accuracy of deception judgments. Pers. Soc. Psychol. Rev. 10(3), 214–234 (2006)
Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: OpenPose: realtime multi-person 2d pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 43(1), 172–186 (2021)
Constâncio, A.S., Tsunoda, D.F., Silva, H.d.F.N., Silveira, J.M.d., Carvalho, D.R.: Deception detection with machine learning: a systematic review and statistical analysis. Plos One 18(2), e0281323 (2023)
DePaulo, B.M., Kashy, D.A., Kirkendol, S.E., Wyer, M.M., Epstein, J.A.: Lying in everyday life. J. Pers. Soc. Psychol. 70(5), 979 (1996)
Dzedzickis, A., Kaklauskas, A., Bucinskas, V.: Human emotion recognition: review of sensors and methods. Sensors 20(3), 592 (2020)
Gannon, T.A., Beech, A.R., Ward, T.: Risk assessment and the polygraph. The Use of the Polygraph in Assessing, Treating and Supervising Sex Offenders: A Practitioner’s Guide, pp. 129–154 (2009)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Karnati, M., Seal, A., Yazidi, A., Krejcar, O.: LieNet: a deep convolution neural network framework for detecting deception. IEEE Trans. Cogn. Develop. Syst. 14(3), 971–984 (2021)
Krishnamurthy, G., Majumder, N., Poria, S., Cambria, E.: A deep learning approach for multimodal deception detection. In: Proceedings of the International Conference on Computational Linguistics and Intelligent Text Processing (CICLing), pp. 87–96 (2023)
Lai, Y.F., Chen, M.Y., Chiang, H.S.: Constructing the lie detection system with fuzzy reasoning approach. Granul. Comput. 3, 169–176 (2018)
Li, F., Yang, W., Liu, X., Sun, G., Liu, J.: Using high-resolution UAV-borne thermal infrared imagery to detect coal fires in Majiliang mine, Datong coalfield, Northern China. Remote Sens. Lett. 9(1), 71–80 (2018)
Mihalcea, R., Pulman, S.: Linguistic ethnography: identifying dominant word classes in text. In: Proceedings of the International Conference on Computational Linguistics and Intelligent Text Processing (CICLing), pp. 594–602 (2009)
Nasri, H., Ouarda, W., Alimi, A.M.: ReLiDSS: novel lie detection system from speech signal. In: Proceedings of the IEEE/ACS International Conference of Computer Systems and Applications (AICCSA), pp. 1–8 (2016)
Nurçin, F.V., Imanov, E., Işın, A., Ozsahin, D.U.: Lie detection on pupil size by back propagation neural network. Procedia Comput. Sci. 120, 417–421 (2017)
Pennebaker, J.W., Francis, M.E., Booth, R.J.: Linguistic inquiry and word count: LIWC 2001. Mahway Lawrence Erlbaum Associates 71(2001), 2001 (2001)
Pérez-Rosas, V., Abouelenien, M., Mihalcea, R., Burzo, M.: Deception detection using real-life trial data. In: Proceedings of the ACM on International Conference on Multimodal Interaction (ICMI), pp. 59–66 (2015)
Rusconi, E., Mitchener-Nissen, T.: Prospects of functional magnetic resonance imaging as lie detector. Front. Hum. Neurosci. 7, 594 (2013)
Van der Velden, B.H., Kuijf, H.J., Gilhuijs, K.G., Viergever, M.A.: Explainable artificial intelligence (XAI) in deep learning-based medical image analysis. Med. Image Anal. 79, 102470 (2022)
Vrij, A., Edward, K., Roberts, K.P., Bull, R.: Detecting deceit via analysis of verbal and nonverbal behavior. J. Nonverbal Behav. 24, 239–263 (2000)
Vrij, A., Semin, G.R.: Lie experts’ beliefs about nonverbal indicators of deception. J. Nonverbal Behav. 20, 65–80 (1996)
Wu, Z., Singh, B., Davis, L., Subrahmanian, V.: Deception detection in videos. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 32 (2018)
Zaidi, S.S.A., Ansari, M.S., Aslam, A., Kanwal, N., Asghar, M., Lee, B.: A survey of modern deep learning based object detection models. Digit. Signal Process. 126, 103514 (2022)
Zuo, J., Gedeon, T., Qin, Z.: Your eyes say you’re lying: an eye movement pattern analysis for face familiarity and deceptive cognition. In: Proceedings of the International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2019)
Acknowledgements
This work was supported by “Smart unmannEd AeRial vehiCles for Human likE monitoRing (SEARCHER)” project of the Italian Ministry of Defence (CIG: Z84333EA0D); and “A Brain Computer Interface (BCI) based System for Transferring Human Emotions inside Unmanned Aerial Vehicles (UAVs)” Sapienza Research Projects (Protocol number: RM1221816C1CF63B); and “DRrone Aerial imaGe SegmentatiOn System (DRAGONS)” (CIG: Z71379B4EA); and Departmental Strategic Plan (DSP) of the University of Udine - Interdepartmental Project on Artificial Intelligence (2020–25); and “A proactive counter-UAV system to protect army tanks and patrols in urban areas (PROACTIVE COUNTER-UAV)” project of the Italian Ministry of Defence (Number 2066/16.12.2019); and the MICS (Made in Italy - Circular and Sustainable) Extended Partnership and received funding from Next-Generation EU (Italian PNRR - M4 C2, Invest 1.3 - D.D. 1551.11-10-2022, PE00000004). CUP MICS B53C22004130001.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Avola, D. et al. (2023). LieToMe: An LSTM-Based Method for Deception Detection by Hand Movements. In: Foresti, G.L., Fusiello, A., Hancock, E. (eds) Image Analysis and Processing – ICIAP 2023. ICIAP 2023. Lecture Notes in Computer Science, vol 14233. Springer, Cham. https://doi.org/10.1007/978-3-031-43148-7_33
Download citation
DOI: https://doi.org/10.1007/978-3-031-43148-7_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43147-0
Online ISBN: 978-3-031-43148-7
eBook Packages: Computer ScienceComputer Science (R0)