Toward long-term and accurate augmented-reality for monocular endoscopic videos

Gustavo A. Puerto-Souza, Jeffrey A Cadeddu, Gian Luca Mariottini

Research output: Contribution to journalArticle

25 Citations (Scopus)

Abstract

By overlaying preoperative radiological 3-D models onto the intraoperative laparoscopic video, augmented-reality (AR) displays promise to increase surgeons' visual awareness of high-risk surgical targets (e.g., the location of a tumor). Existing AR surgical systems lack in robustness and accuracy because of the many challenges in endoscopic imagery, such as frequent changes in illumination, rapid camera motions, prolonged organ occlusions, and tissue deformations. The frequent occurrence of these events can cause the loss of image (anchor) points, and thus, the loss of the AR display after a few frames. In this paper, we present the design of a new AR system that represents a first step toward long term and accurate augmented surgical display for monocular (calibrated and uncalibrated) endoscopic videos. Our system uses correspondence-search methods, and a new weighted sliding-window registration approach, to automatically and accurately recover the overlay by predicting the image locations of a high number of anchor points that were lost after a sudden image change. The effectiveness of the proposed system in maintaining a long term (over 2 min) and accurate (less than 1 mm) augmentation has been documented over a set of real partial-nephrectomy laparascopic videos.

Original languageEnglish (US)
Article number2323999
Pages (from-to)2609-2620
Number of pages12
JournalIEEE Transactions on Biomedical Engineering
Volume61
Issue number10
DOIs
StatePublished - Oct 1 2014

Fingerprint

Augmented reality
Imagery (Psychotherapy)
Nephrectomy
Lighting
Display devices
Anchors
Neoplasms
Tumors
Cameras
Tissue
Surgeons

Keywords

  • Augmented reality (AR)
  • endoscopic vision
  • feature tracking

ASJC Scopus subject areas

  • Biomedical Engineering
  • Medicine(all)

Cite this

Toward long-term and accurate augmented-reality for monocular endoscopic videos. / Puerto-Souza, Gustavo A.; Cadeddu, Jeffrey A; Mariottini, Gian Luca.

In: IEEE Transactions on Biomedical Engineering, Vol. 61, No. 10, 2323999, 01.10.2014, p. 2609-2620.

Research output: Contribution to journalArticle

Puerto-Souza, Gustavo A. ; Cadeddu, Jeffrey A ; Mariottini, Gian Luca. / Toward long-term and accurate augmented-reality for monocular endoscopic videos. In: IEEE Transactions on Biomedical Engineering. 2014 ; Vol. 61, No. 10. pp. 2609-2620.
@article{8bdb75fc73fa446abe67131a590802fd,
title = "Toward long-term and accurate augmented-reality for monocular endoscopic videos",
abstract = "By overlaying preoperative radiological 3-D models onto the intraoperative laparoscopic video, augmented-reality (AR) displays promise to increase surgeons' visual awareness of high-risk surgical targets (e.g., the location of a tumor). Existing AR surgical systems lack in robustness and accuracy because of the many challenges in endoscopic imagery, such as frequent changes in illumination, rapid camera motions, prolonged organ occlusions, and tissue deformations. The frequent occurrence of these events can cause the loss of image (anchor) points, and thus, the loss of the AR display after a few frames. In this paper, we present the design of a new AR system that represents a first step toward long term and accurate augmented surgical display for monocular (calibrated and uncalibrated) endoscopic videos. Our system uses correspondence-search methods, and a new weighted sliding-window registration approach, to automatically and accurately recover the overlay by predicting the image locations of a high number of anchor points that were lost after a sudden image change. The effectiveness of the proposed system in maintaining a long term (over 2 min) and accurate (less than 1 mm) augmentation has been documented over a set of real partial-nephrectomy laparascopic videos.",
keywords = "Augmented reality (AR), endoscopic vision, feature tracking",
author = "Puerto-Souza, {Gustavo A.} and Cadeddu, {Jeffrey A} and Mariottini, {Gian Luca}",
year = "2014",
month = "10",
day = "1",
doi = "10.1109/TBME.2014.2323999",
language = "English (US)",
volume = "61",
pages = "2609--2620",
journal = "IEEE Transactions on Biomedical Engineering",
issn = "0018-9294",
publisher = "IEEE Computer Society",
number = "10",

}

TY - JOUR

T1 - Toward long-term and accurate augmented-reality for monocular endoscopic videos

AU - Puerto-Souza, Gustavo A.

AU - Cadeddu, Jeffrey A

AU - Mariottini, Gian Luca

PY - 2014/10/1

Y1 - 2014/10/1

N2 - By overlaying preoperative radiological 3-D models onto the intraoperative laparoscopic video, augmented-reality (AR) displays promise to increase surgeons' visual awareness of high-risk surgical targets (e.g., the location of a tumor). Existing AR surgical systems lack in robustness and accuracy because of the many challenges in endoscopic imagery, such as frequent changes in illumination, rapid camera motions, prolonged organ occlusions, and tissue deformations. The frequent occurrence of these events can cause the loss of image (anchor) points, and thus, the loss of the AR display after a few frames. In this paper, we present the design of a new AR system that represents a first step toward long term and accurate augmented surgical display for monocular (calibrated and uncalibrated) endoscopic videos. Our system uses correspondence-search methods, and a new weighted sliding-window registration approach, to automatically and accurately recover the overlay by predicting the image locations of a high number of anchor points that were lost after a sudden image change. The effectiveness of the proposed system in maintaining a long term (over 2 min) and accurate (less than 1 mm) augmentation has been documented over a set of real partial-nephrectomy laparascopic videos.

AB - By overlaying preoperative radiological 3-D models onto the intraoperative laparoscopic video, augmented-reality (AR) displays promise to increase surgeons' visual awareness of high-risk surgical targets (e.g., the location of a tumor). Existing AR surgical systems lack in robustness and accuracy because of the many challenges in endoscopic imagery, such as frequent changes in illumination, rapid camera motions, prolonged organ occlusions, and tissue deformations. The frequent occurrence of these events can cause the loss of image (anchor) points, and thus, the loss of the AR display after a few frames. In this paper, we present the design of a new AR system that represents a first step toward long term and accurate augmented surgical display for monocular (calibrated and uncalibrated) endoscopic videos. Our system uses correspondence-search methods, and a new weighted sliding-window registration approach, to automatically and accurately recover the overlay by predicting the image locations of a high number of anchor points that were lost after a sudden image change. The effectiveness of the proposed system in maintaining a long term (over 2 min) and accurate (less than 1 mm) augmentation has been documented over a set of real partial-nephrectomy laparascopic videos.

KW - Augmented reality (AR)

KW - endoscopic vision

KW - feature tracking

UR - http://www.scopus.com/inward/record.url?scp=84907452204&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84907452204&partnerID=8YFLogxK

U2 - 10.1109/TBME.2014.2323999

DO - 10.1109/TBME.2014.2323999

M3 - Article

C2 - 24835126

AN - SCOPUS:84907452204

VL - 61

SP - 2609

EP - 2620

JO - IEEE Transactions on Biomedical Engineering

JF - IEEE Transactions on Biomedical Engineering

SN - 0018-9294

IS - 10

M1 - 2323999

ER -