XAI-CBIR: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos

Deepak Roy Chittajallu, Bo Dong, Paul Tunison, Roddy Collins, Katerina Wells, James Fleshman, Ganesh Sankaranarayanan, Steven Schwaitzberg, Lora Cavuoto, Andinet Enquobahrie

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we present a human-in-the-loop explainable AI (XAI) system for content based image retrieval (CBIR) of video frames similar to a query image from minimally invasive surgery (MIS) videos for surgical education. It extracts semantic descriptors from MIS video frames using a self-supervised deep learning model. It then employs an iterative query refinement strategy where in a binary classifier trained online based on relevance feedback from the user is used to iteratively refine the search results. Lastly, it uses an XAI technique to generate a saliency map that provides a visual explanation of why the system considers a retrieved image to be similar to the query image. We evaluated the proposed XAI-CBIR system on the public Cholec80 dataset containing 80 videos of minimally invasive cholecystectomy surgeries with encouraging results.

Original languageEnglish (US)
Title of host publicationISBI 2019 - 2019 IEEE International Symposium on Biomedical Imaging
PublisherIEEE Computer Society
Pages66-69
Number of pages4
ISBN (Electronic)9781538636411
DOIs
StatePublished - Apr 2019
Externally publishedYes
Event16th IEEE International Symposium on Biomedical Imaging, ISBI 2019 - Venice, Italy
Duration: Apr 8 2019Apr 11 2019

Publication series

NameProceedings - International Symposium on Biomedical Imaging
Volume2019-April
ISSN (Print)1945-7928
ISSN (Electronic)1945-8452

Conference

Conference16th IEEE International Symposium on Biomedical Imaging, ISBI 2019
Country/TerritoryItaly
CityVenice
Period4/8/194/11/19

Keywords

  • Content based video retrieval
  • Deep learning
  • Explainable AI
  • Laparoscopy
  • Minimally invasive surgery
  • Self-supervised learning
  • Surgical data science

ASJC Scopus subject areas

  • Biomedical Engineering
  • Radiology Nuclear Medicine and imaging

Fingerprint

Dive into the research topics of 'XAI-CBIR: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos'. Together they form a unique fingerprint.

Cite this