Stereoscopic augmented reality for laparoscopic surgery

Xin Kang, Mahdi Azizian, Emmanuel Wilson, Kyle Wu, Aaron D. Martin, Timothy D. Kane, Craig A Peters, Kevin Cleary, Raj Shekhar

Research output: Contribution to journalArticle

43 Citations (Scopus)

Abstract

Background: Conventional laparoscopes provide a flat representation of the three-dimensional (3D) operating field and are incapable of visualizing internal structures located beneath visible organ surfaces. Computed tomography (CT) and magnetic resonance (MR) images are difficult to fuse in real time with laparoscopic views due to the deformable nature of soft-tissue organs. Utilizing emerging camera technology, we have developed a real-time stereoscopic augmented-reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound (LUS) with stereoscopic video. The system creates two new visual cues: (1) perception of true depth with improved understanding of 3D spatial relationships among anatomical structures, and (2) visualization of critical internal structures along with a more comprehensive visualization of the operating field. Methods: The stereoscopic AR system has been designed for near-term clinical translation with seamless integration into the existing surgical workflow. It is composed of a stereoscopic vision system, a LUS system, and an optical tracker. Specialized software processes streams of imaging data from the tracked devices and registers those in real time. The resulting two ultrasound-augmented video streams (one for the left and one for the right eye) give a live stereoscopic AR view of the operating field. The team conducted a series of stereoscopic AR interrogations of the liver, gallbladder, biliary tree, and kidneys in two swine. Results: The preclinical studies demonstrated the feasibility of the stereoscopic AR system during in vivo procedures. Major internal structures could be easily identified. The system exhibited unobservable latency with acceptable image-to-video registration accuracy. Conclusions: We presented the first in vivo use of a complete system with stereoscopic AR visualization capability. This new capability introduces new visual cues and enhances visualization of the surgical anatomy. The system shows promise to improve the precision and expand the capacity of minimally invasive laparoscopic surgeries.

Original languageEnglish (US)
Pages (from-to)2227-2235
Number of pages9
JournalSurgical Endoscopy and Other Interventional Techniques
Volume28
Issue number7
DOIs
StatePublished - Jan 1 2014

Fingerprint

Laparoscopy
Depth Perception
Cues
Laparoscopes
Optical Devices
Minimally Invasive Surgical Procedures
Workflow
Biliary Tract
Feasibility Studies
Gallbladder
Anatomy
Magnetic Resonance Spectroscopy
Swine
Software
Tomography
Technology
Kidney
Equipment and Supplies
Liver

Keywords

  • Augmented reality
  • Laparoscopic surgery
  • Multimodality image fusion
  • Stereoscopic visualization
  • Surgical visualization

ASJC Scopus subject areas

  • Surgery

Cite this

Kang, X., Azizian, M., Wilson, E., Wu, K., Martin, A. D., Kane, T. D., ... Shekhar, R. (2014). Stereoscopic augmented reality for laparoscopic surgery. Surgical Endoscopy and Other Interventional Techniques, 28(7), 2227-2235. https://doi.org/10.1007/s00464-014-3433-x

Stereoscopic augmented reality for laparoscopic surgery. / Kang, Xin; Azizian, Mahdi; Wilson, Emmanuel; Wu, Kyle; Martin, Aaron D.; Kane, Timothy D.; Peters, Craig A; Cleary, Kevin; Shekhar, Raj.

In: Surgical Endoscopy and Other Interventional Techniques, Vol. 28, No. 7, 01.01.2014, p. 2227-2235.

Research output: Contribution to journalArticle

Kang, X, Azizian, M, Wilson, E, Wu, K, Martin, AD, Kane, TD, Peters, CA, Cleary, K & Shekhar, R 2014, 'Stereoscopic augmented reality for laparoscopic surgery', Surgical Endoscopy and Other Interventional Techniques, vol. 28, no. 7, pp. 2227-2235. https://doi.org/10.1007/s00464-014-3433-x
Kang, Xin ; Azizian, Mahdi ; Wilson, Emmanuel ; Wu, Kyle ; Martin, Aaron D. ; Kane, Timothy D. ; Peters, Craig A ; Cleary, Kevin ; Shekhar, Raj. / Stereoscopic augmented reality for laparoscopic surgery. In: Surgical Endoscopy and Other Interventional Techniques. 2014 ; Vol. 28, No. 7. pp. 2227-2235.
@article{619571a79538491dbf77b32a7b0d74d6,
title = "Stereoscopic augmented reality for laparoscopic surgery",
abstract = "Background: Conventional laparoscopes provide a flat representation of the three-dimensional (3D) operating field and are incapable of visualizing internal structures located beneath visible organ surfaces. Computed tomography (CT) and magnetic resonance (MR) images are difficult to fuse in real time with laparoscopic views due to the deformable nature of soft-tissue organs. Utilizing emerging camera technology, we have developed a real-time stereoscopic augmented-reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound (LUS) with stereoscopic video. The system creates two new visual cues: (1) perception of true depth with improved understanding of 3D spatial relationships among anatomical structures, and (2) visualization of critical internal structures along with a more comprehensive visualization of the operating field. Methods: The stereoscopic AR system has been designed for near-term clinical translation with seamless integration into the existing surgical workflow. It is composed of a stereoscopic vision system, a LUS system, and an optical tracker. Specialized software processes streams of imaging data from the tracked devices and registers those in real time. The resulting two ultrasound-augmented video streams (one for the left and one for the right eye) give a live stereoscopic AR view of the operating field. The team conducted a series of stereoscopic AR interrogations of the liver, gallbladder, biliary tree, and kidneys in two swine. Results: The preclinical studies demonstrated the feasibility of the stereoscopic AR system during in vivo procedures. Major internal structures could be easily identified. The system exhibited unobservable latency with acceptable image-to-video registration accuracy. Conclusions: We presented the first in vivo use of a complete system with stereoscopic AR visualization capability. This new capability introduces new visual cues and enhances visualization of the surgical anatomy. The system shows promise to improve the precision and expand the capacity of minimally invasive laparoscopic surgeries.",
keywords = "Augmented reality, Laparoscopic surgery, Multimodality image fusion, Stereoscopic visualization, Surgical visualization",
author = "Xin Kang and Mahdi Azizian and Emmanuel Wilson and Kyle Wu and Martin, {Aaron D.} and Kane, {Timothy D.} and Peters, {Craig A} and Kevin Cleary and Raj Shekhar",
year = "2014",
month = "1",
day = "1",
doi = "10.1007/s00464-014-3433-x",
language = "English (US)",
volume = "28",
pages = "2227--2235",
journal = "Surgical Endoscopy and Other Interventional Techniques",
issn = "0930-2794",
publisher = "Springer New York",
number = "7",

}

TY - JOUR

T1 - Stereoscopic augmented reality for laparoscopic surgery

AU - Kang, Xin

AU - Azizian, Mahdi

AU - Wilson, Emmanuel

AU - Wu, Kyle

AU - Martin, Aaron D.

AU - Kane, Timothy D.

AU - Peters, Craig A

AU - Cleary, Kevin

AU - Shekhar, Raj

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Background: Conventional laparoscopes provide a flat representation of the three-dimensional (3D) operating field and are incapable of visualizing internal structures located beneath visible organ surfaces. Computed tomography (CT) and magnetic resonance (MR) images are difficult to fuse in real time with laparoscopic views due to the deformable nature of soft-tissue organs. Utilizing emerging camera technology, we have developed a real-time stereoscopic augmented-reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound (LUS) with stereoscopic video. The system creates two new visual cues: (1) perception of true depth with improved understanding of 3D spatial relationships among anatomical structures, and (2) visualization of critical internal structures along with a more comprehensive visualization of the operating field. Methods: The stereoscopic AR system has been designed for near-term clinical translation with seamless integration into the existing surgical workflow. It is composed of a stereoscopic vision system, a LUS system, and an optical tracker. Specialized software processes streams of imaging data from the tracked devices and registers those in real time. The resulting two ultrasound-augmented video streams (one for the left and one for the right eye) give a live stereoscopic AR view of the operating field. The team conducted a series of stereoscopic AR interrogations of the liver, gallbladder, biliary tree, and kidneys in two swine. Results: The preclinical studies demonstrated the feasibility of the stereoscopic AR system during in vivo procedures. Major internal structures could be easily identified. The system exhibited unobservable latency with acceptable image-to-video registration accuracy. Conclusions: We presented the first in vivo use of a complete system with stereoscopic AR visualization capability. This new capability introduces new visual cues and enhances visualization of the surgical anatomy. The system shows promise to improve the precision and expand the capacity of minimally invasive laparoscopic surgeries.

AB - Background: Conventional laparoscopes provide a flat representation of the three-dimensional (3D) operating field and are incapable of visualizing internal structures located beneath visible organ surfaces. Computed tomography (CT) and magnetic resonance (MR) images are difficult to fuse in real time with laparoscopic views due to the deformable nature of soft-tissue organs. Utilizing emerging camera technology, we have developed a real-time stereoscopic augmented-reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound (LUS) with stereoscopic video. The system creates two new visual cues: (1) perception of true depth with improved understanding of 3D spatial relationships among anatomical structures, and (2) visualization of critical internal structures along with a more comprehensive visualization of the operating field. Methods: The stereoscopic AR system has been designed for near-term clinical translation with seamless integration into the existing surgical workflow. It is composed of a stereoscopic vision system, a LUS system, and an optical tracker. Specialized software processes streams of imaging data from the tracked devices and registers those in real time. The resulting two ultrasound-augmented video streams (one for the left and one for the right eye) give a live stereoscopic AR view of the operating field. The team conducted a series of stereoscopic AR interrogations of the liver, gallbladder, biliary tree, and kidneys in two swine. Results: The preclinical studies demonstrated the feasibility of the stereoscopic AR system during in vivo procedures. Major internal structures could be easily identified. The system exhibited unobservable latency with acceptable image-to-video registration accuracy. Conclusions: We presented the first in vivo use of a complete system with stereoscopic AR visualization capability. This new capability introduces new visual cues and enhances visualization of the surgical anatomy. The system shows promise to improve the precision and expand the capacity of minimally invasive laparoscopic surgeries.

KW - Augmented reality

KW - Laparoscopic surgery

KW - Multimodality image fusion

KW - Stereoscopic visualization

KW - Surgical visualization

UR - http://www.scopus.com/inward/record.url?scp=84903633127&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84903633127&partnerID=8YFLogxK

U2 - 10.1007/s00464-014-3433-x

DO - 10.1007/s00464-014-3433-x

M3 - Article

C2 - 24488352

AN - SCOPUS:84903633127

VL - 28

SP - 2227

EP - 2235

JO - Surgical Endoscopy and Other Interventional Techniques

JF - Surgical Endoscopy and Other Interventional Techniques

SN - 0930-2794

IS - 7

ER -