XAI-CBIR: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos

Deepak Roy Chittajallu; Bo Dong; Paul Tunison; Roddy Collins; Katerina Wells; James Fleshman; Ganesh Sankaranarayanan; Steven Schwaitzberg; Lora Cavuoto; Andinet Enquobahrie

doi:10.1109/ISBI.2019.8759428

XAI-CBIR: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos

Deepak Roy Chittajallu, Bo Dong, Paul Tunison, Roddy Collins, Katerina Wells, James Fleshman, Ganesh Sankaranarayanan, Steven Schwaitzberg, Lora Cavuoto, Andinet Enquobahrie

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

26 Scopus citations

Abstract

In this paper, we present a human-in-the-loop explainable AI (XAI) system for content based image retrieval (CBIR) of video frames similar to a query image from minimally invasive surgery (MIS) videos for surgical education. It extracts semantic descriptors from MIS video frames using a self-supervised deep learning model. It then employs an iterative query refinement strategy where in a binary classifier trained online based on relevance feedback from the user is used to iteratively refine the search results. Lastly, it uses an XAI technique to generate a saliency map that provides a visual explanation of why the system considers a retrieved image to be similar to the query image. We evaluated the proposed XAI-CBIR system on the public Cholec80 dataset containing 80 videos of minimally invasive cholecystectomy surgeries with encouraging results.

Original language	English (US)
Title of host publication	ISBI 2019 - 2019 IEEE International Symposium on Biomedical Imaging
Publisher	IEEE Computer Society
Pages	66-69
Number of pages	4
ISBN (Electronic)	9781538636411
DOIs	https://doi.org/10.1109/ISBI.2019.8759428
State	Published - Apr 2019
Externally published	Yes
Event	16th IEEE International Symposium on Biomedical Imaging, ISBI 2019 - Venice, Italy Duration: Apr 8 2019 → Apr 11 2019

Publication series

Name	Proceedings - International Symposium on Biomedical Imaging
Volume	2019-April
ISSN (Print)	1945-7928
ISSN (Electronic)	1945-8452

Conference

Conference	16th IEEE International Symposium on Biomedical Imaging, ISBI 2019
Country/Territory	Italy
City	Venice
Period	4/8/19 → 4/11/19

Keywords

Content based video retrieval
Deep learning
Explainable AI
Laparoscopy
Minimally invasive surgery
Self-supervised learning
Surgical data science

ASJC Scopus subject areas

Biomedical Engineering
Radiology Nuclear Medicine and imaging

Access to Document

10.1109/ISBI.2019.8759428

Cite this

Chittajallu, D. R., Dong, B., Tunison, P., Collins, R., Wells, K., Fleshman, J., Sankaranarayanan, G., Schwaitzberg, S., Cavuoto, L., & Enquobahrie, A. (2019). XAI-CBIR: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos. In ISBI 2019 - 2019 IEEE International Symposium on Biomedical Imaging (pp. 66-69). Article 8759428 (Proceedings - International Symposium on Biomedical Imaging; Vol. 2019-April). IEEE Computer Society. https://doi.org/10.1109/ISBI.2019.8759428

XAI-CBIR: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos. / Chittajallu, Deepak Roy; Dong, Bo; Tunison, Paul et al.
ISBI 2019 - 2019 IEEE International Symposium on Biomedical Imaging. IEEE Computer Society, 2019. p. 66-69 8759428 (Proceedings - International Symposium on Biomedical Imaging; Vol. 2019-April).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Chittajallu, DR, Dong, B, Tunison, P, Collins, R, Wells, K, Fleshman, J, Sankaranarayanan, G, Schwaitzberg, S, Cavuoto, L & Enquobahrie, A 2019, XAI-CBIR: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos. in ISBI 2019 - 2019 IEEE International Symposium on Biomedical Imaging., 8759428, Proceedings - International Symposium on Biomedical Imaging, vol. 2019-April, IEEE Computer Society, pp. 66-69, 16th IEEE International Symposium on Biomedical Imaging, ISBI 2019, Venice, Italy, 4/8/19. https://doi.org/10.1109/ISBI.2019.8759428

Chittajallu DR, Dong B, Tunison P, Collins R, Wells K, Fleshman J et al. XAI-CBIR: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos. In ISBI 2019 - 2019 IEEE International Symposium on Biomedical Imaging. IEEE Computer Society. 2019. p. 66-69. 8759428. (Proceedings - International Symposium on Biomedical Imaging). doi: 10.1109/ISBI.2019.8759428

@inproceedings{d3a786a764eb430bb9044b8179b3241a,

title = "XAI-CBIR: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos",

abstract = "In this paper, we present a human-in-the-loop explainable AI (XAI) system for content based image retrieval (CBIR) of video frames similar to a query image from minimally invasive surgery (MIS) videos for surgical education. It extracts semantic descriptors from MIS video frames using a self-supervised deep learning model. It then employs an iterative query refinement strategy where in a binary classifier trained online based on relevance feedback from the user is used to iteratively refine the search results. Lastly, it uses an XAI technique to generate a saliency map that provides a visual explanation of why the system considers a retrieved image to be similar to the query image. We evaluated the proposed XAI-CBIR system on the public Cholec80 dataset containing 80 videos of minimally invasive cholecystectomy surgeries with encouraging results.",

keywords = "Content based video retrieval, Deep learning, Explainable AI, Laparoscopy, Minimally invasive surgery, Self-supervised learning, Surgical data science",

author = "Chittajallu, {Deepak Roy} and Bo Dong and Paul Tunison and Roddy Collins and Katerina Wells and James Fleshman and Ganesh Sankaranarayanan and Steven Schwaitzberg and Lora Cavuoto and Andinet Enquobahrie",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 16th IEEE International Symposium on Biomedical Imaging, ISBI 2019 ; Conference date: 08-04-2019 Through 11-04-2019",

year = "2019",

month = apr,

doi = "10.1109/ISBI.2019.8759428",

language = "English (US)",

series = "Proceedings - International Symposium on Biomedical Imaging",

publisher = "IEEE Computer Society",

pages = "66--69",

booktitle = "ISBI 2019 - 2019 IEEE International Symposium on Biomedical Imaging",

}

TY - GEN

T1 - XAI-CBIR

T2 - 16th IEEE International Symposium on Biomedical Imaging, ISBI 2019

AU - Chittajallu, Deepak Roy

AU - Dong, Bo

AU - Tunison, Paul

AU - Collins, Roddy

AU - Wells, Katerina

AU - Fleshman, James

AU - Sankaranarayanan, Ganesh

AU - Schwaitzberg, Steven

AU - Cavuoto, Lora

AU - Enquobahrie, Andinet

PY - 2019/4

Y1 - 2019/4

N2 - In this paper, we present a human-in-the-loop explainable AI (XAI) system for content based image retrieval (CBIR) of video frames similar to a query image from minimally invasive surgery (MIS) videos for surgical education. It extracts semantic descriptors from MIS video frames using a self-supervised deep learning model. It then employs an iterative query refinement strategy where in a binary classifier trained online based on relevance feedback from the user is used to iteratively refine the search results. Lastly, it uses an XAI technique to generate a saliency map that provides a visual explanation of why the system considers a retrieved image to be similar to the query image. We evaluated the proposed XAI-CBIR system on the public Cholec80 dataset containing 80 videos of minimally invasive cholecystectomy surgeries with encouraging results.

AB - In this paper, we present a human-in-the-loop explainable AI (XAI) system for content based image retrieval (CBIR) of video frames similar to a query image from minimally invasive surgery (MIS) videos for surgical education. It extracts semantic descriptors from MIS video frames using a self-supervised deep learning model. It then employs an iterative query refinement strategy where in a binary classifier trained online based on relevance feedback from the user is used to iteratively refine the search results. Lastly, it uses an XAI technique to generate a saliency map that provides a visual explanation of why the system considers a retrieved image to be similar to the query image. We evaluated the proposed XAI-CBIR system on the public Cholec80 dataset containing 80 videos of minimally invasive cholecystectomy surgeries with encouraging results.

KW - Content based video retrieval

KW - Deep learning

KW - Explainable AI

KW - Laparoscopy

KW - Minimally invasive surgery

KW - Self-supervised learning

KW - Surgical data science

UR - http://www.scopus.com/inward/record.url?scp=85073896747&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85073896747&partnerID=8YFLogxK

U2 - 10.1109/ISBI.2019.8759428

DO - 10.1109/ISBI.2019.8759428

M3 - Conference contribution

AN - SCOPUS:85073896747

T3 - Proceedings - International Symposium on Biomedical Imaging

SP - 66

EP - 69

BT - ISBI 2019 - 2019 IEEE International Symposium on Biomedical Imaging

PB - IEEE Computer Society

Y2 - 8 April 2019 through 11 April 2019

ER -

XAI-CBIR: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this