MultiModal Deception Detection: Accuracy, Applicability and Generalizability*

Vibha Belavadi, Yan Zhou, Jonathan Z. Bakdash, Murat Kantarcioglu, Daniel C. Krawczyk, Linda Nguyen, Jelena Rakic, Bhavani Thuriasingham

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The increasing use of Artificial Intelligence (AI) systems in face recognition and video processing in recent times creates higher stakes for their application in daily life. Increasingly, critical decisions are being made using these AI systems in application domains such as employment, finance, and crime prevention. These applications are done through the use of more abstract concepts such as emotions, trait evaluations (e.g., trustworthiness), and behavior (e.g., deception). These abstract concepts are learned by the AI system using the verbal and non-verbal cues from the human subject stimuli (e,g., facial expressions, movements, audio, text) for inference. Because the use of AI systems often happens in high stakes scenarios, it is of utmost importance that the AI system participating in the decision-making process is highly reliable and credible. In this paper, we specifically consider the feasibility of using such an AI system for deception detection. We examine if deception can be caught using multimodal aspects such as facial expressions and movements, audio cues, video cues, etc. We experiment using three different datasets with varying degrees of deception to explore the problem of deception detection. We also study state-of-the-art deception detection systems and investigate whether we can extend their algorithm into new datasets. We conclude that there is a lack of reasonable evidence that AI-based deception detection is generalizable over different scenarios of lying (lying deliberately, lying under duress, and lying through half-truths) and that in the future additional factors will need to be considered to make such a claim.

Original languageEnglish (US)
Title of host publicationProceedings - 2020 2nd IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications, TPS-ISA 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages99-106
Number of pages8
ISBN (Electronic)9781728185439
DOIs
StatePublished - Oct 2020
Externally publishedYes
Event2nd IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications, TPS-ISA 2020 - Virtual, Atlanta, United States
Duration: Dec 1 2020Dec 3 2020

Publication series

NameProceedings - 2020 2nd IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications, TPS-ISA 2020

Conference

Conference2nd IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications, TPS-ISA 2020
Country/TerritoryUnited States
CityVirtual, Atlanta
Period12/1/2012/3/20

Keywords

  • deception detection
  • ethics
  • facial expressions
  • machine learning
  • multi-modal data analysis

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Information Systems and Management
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'MultiModal Deception Detection: Accuracy, Applicability and Generalizability*'. Together they form a unique fingerprint.

Cite this