Construct and face validity of MIST-VR, Endotower, and CELTS: Are we ready for skills assessment using simulators?

S. Maithel, R. Sierra, J. Korndorffer, P. Neumann, S. Dawson, M. Callery, D. Jones, D. Scott

Research output: Contribution to journalArticle

85 Citations (Scopus)

Abstract

Background: Video trainers may best offer visually realistic laparoscopic simulation, whereas virtual reality (VR) modules may best provide multidimensional objective measures of performance. This study compares the construct and face validity of three different laparoscopic simulators. Methods: Subjects were voluntarily enrolled at the Learning Center during the 2004 SAGES annual meeting. Each subject completed two repetitions of a single task on each of three simulators, MIST-VR, Endotower, and CELTS; performance scores were automatically generated and recorded. Scores of individuals with various levels of experience were compared to determine construct validity for each simulator. Experience was defined according to four parameters: (a) PGY level, (b) fellowship training, (c) basic laparoscopic cases, and (d) advanced laparoscopic cases. Subjects rated each simulator regarding six face validity (realism of simulation) parameters using a 10-point Likert scale (10 = best rating) and participant scores were compared to previously established expert scores (proficiency goals for training). Results: Ninety-one attendees completed the study. Construct validity was demonstrated for all three simulators; significant differences in scores were detected according to one parameter for MIST-VR, two param-eters for Endotower, and all four parameters for CELTS. Face validity was rated as good to excellent for all three simulators (7.0 ± 0.3 for MIST-VR, 7.9 ± 0.3 for Endotower [p < 0.001 vs MIST-VR], and 8.7 ± 0.1 for CELTS [p = 0.001 vs MIST-VR, p = 0.01 vs Endotower]); 6%, 0%, and 36% of "expert" participants obtained expert scores on MIST-VR, Endotower, and CELTS, respectively. Conclusions: All three simulators demonstrated significant construct and reasonable face validity. Although virtual reality holds great promise to expand the scope of laparoscopic simulation, current interfaces may limit their utility for assessment. Computer-enhanced video trainers may offer an improved interface while incorporating useful multidimensional metrics. Further work is needed to establish standards for appropriate skills assessment methods and performance levels using simulators.

Original languageEnglish (US)
Pages (from-to)104-112
Number of pages9
JournalSurgical Endoscopy and Other Interventional Techniques
Volume20
Issue number1
DOIs
StatePublished - Jan 2006

Fingerprint

Reproducibility of Results
Learning

Keywords

  • CELTS
  • Endotower
  • Laparoscopy
  • MIST-VR
  • Simulator
  • Validity

ASJC Scopus subject areas

  • Surgery

Cite this

Construct and face validity of MIST-VR, Endotower, and CELTS : Are we ready for skills assessment using simulators? / Maithel, S.; Sierra, R.; Korndorffer, J.; Neumann, P.; Dawson, S.; Callery, M.; Jones, D.; Scott, D.

In: Surgical Endoscopy and Other Interventional Techniques, Vol. 20, No. 1, 01.2006, p. 104-112.

Research output: Contribution to journalArticle

Maithel, S. ; Sierra, R. ; Korndorffer, J. ; Neumann, P. ; Dawson, S. ; Callery, M. ; Jones, D. ; Scott, D. / Construct and face validity of MIST-VR, Endotower, and CELTS : Are we ready for skills assessment using simulators?. In: Surgical Endoscopy and Other Interventional Techniques. 2006 ; Vol. 20, No. 1. pp. 104-112.
@article{f3ed81b1658a4a7a95f6ec34f4ee38d8,
title = "Construct and face validity of MIST-VR, Endotower, and CELTS: Are we ready for skills assessment using simulators?",
abstract = "Background: Video trainers may best offer visually realistic laparoscopic simulation, whereas virtual reality (VR) modules may best provide multidimensional objective measures of performance. This study compares the construct and face validity of three different laparoscopic simulators. Methods: Subjects were voluntarily enrolled at the Learning Center during the 2004 SAGES annual meeting. Each subject completed two repetitions of a single task on each of three simulators, MIST-VR, Endotower, and CELTS; performance scores were automatically generated and recorded. Scores of individuals with various levels of experience were compared to determine construct validity for each simulator. Experience was defined according to four parameters: (a) PGY level, (b) fellowship training, (c) basic laparoscopic cases, and (d) advanced laparoscopic cases. Subjects rated each simulator regarding six face validity (realism of simulation) parameters using a 10-point Likert scale (10 = best rating) and participant scores were compared to previously established expert scores (proficiency goals for training). Results: Ninety-one attendees completed the study. Construct validity was demonstrated for all three simulators; significant differences in scores were detected according to one parameter for MIST-VR, two param-eters for Endotower, and all four parameters for CELTS. Face validity was rated as good to excellent for all three simulators (7.0 ± 0.3 for MIST-VR, 7.9 ± 0.3 for Endotower [p < 0.001 vs MIST-VR], and 8.7 ± 0.1 for CELTS [p = 0.001 vs MIST-VR, p = 0.01 vs Endotower]); 6{\%}, 0{\%}, and 36{\%} of {"}expert{"} participants obtained expert scores on MIST-VR, Endotower, and CELTS, respectively. Conclusions: All three simulators demonstrated significant construct and reasonable face validity. Although virtual reality holds great promise to expand the scope of laparoscopic simulation, current interfaces may limit their utility for assessment. Computer-enhanced video trainers may offer an improved interface while incorporating useful multidimensional metrics. Further work is needed to establish standards for appropriate skills assessment methods and performance levels using simulators.",
keywords = "CELTS, Endotower, Laparoscopy, MIST-VR, Simulator, Validity",
author = "S. Maithel and R. Sierra and J. Korndorffer and P. Neumann and S. Dawson and M. Callery and D. Jones and D. Scott",
year = "2006",
month = "1",
doi = "10.1007/s00464-005-0054-4",
language = "English (US)",
volume = "20",
pages = "104--112",
journal = "Surgical Endoscopy and Other Interventional Techniques",
issn = "0930-2794",
publisher = "Springer New York",
number = "1",

}

TY - JOUR

T1 - Construct and face validity of MIST-VR, Endotower, and CELTS

T2 - Are we ready for skills assessment using simulators?

AU - Maithel, S.

AU - Sierra, R.

AU - Korndorffer, J.

AU - Neumann, P.

AU - Dawson, S.

AU - Callery, M.

AU - Jones, D.

AU - Scott, D.

PY - 2006/1

Y1 - 2006/1

N2 - Background: Video trainers may best offer visually realistic laparoscopic simulation, whereas virtual reality (VR) modules may best provide multidimensional objective measures of performance. This study compares the construct and face validity of three different laparoscopic simulators. Methods: Subjects were voluntarily enrolled at the Learning Center during the 2004 SAGES annual meeting. Each subject completed two repetitions of a single task on each of three simulators, MIST-VR, Endotower, and CELTS; performance scores were automatically generated and recorded. Scores of individuals with various levels of experience were compared to determine construct validity for each simulator. Experience was defined according to four parameters: (a) PGY level, (b) fellowship training, (c) basic laparoscopic cases, and (d) advanced laparoscopic cases. Subjects rated each simulator regarding six face validity (realism of simulation) parameters using a 10-point Likert scale (10 = best rating) and participant scores were compared to previously established expert scores (proficiency goals for training). Results: Ninety-one attendees completed the study. Construct validity was demonstrated for all three simulators; significant differences in scores were detected according to one parameter for MIST-VR, two param-eters for Endotower, and all four parameters for CELTS. Face validity was rated as good to excellent for all three simulators (7.0 ± 0.3 for MIST-VR, 7.9 ± 0.3 for Endotower [p < 0.001 vs MIST-VR], and 8.7 ± 0.1 for CELTS [p = 0.001 vs MIST-VR, p = 0.01 vs Endotower]); 6%, 0%, and 36% of "expert" participants obtained expert scores on MIST-VR, Endotower, and CELTS, respectively. Conclusions: All three simulators demonstrated significant construct and reasonable face validity. Although virtual reality holds great promise to expand the scope of laparoscopic simulation, current interfaces may limit their utility for assessment. Computer-enhanced video trainers may offer an improved interface while incorporating useful multidimensional metrics. Further work is needed to establish standards for appropriate skills assessment methods and performance levels using simulators.

AB - Background: Video trainers may best offer visually realistic laparoscopic simulation, whereas virtual reality (VR) modules may best provide multidimensional objective measures of performance. This study compares the construct and face validity of three different laparoscopic simulators. Methods: Subjects were voluntarily enrolled at the Learning Center during the 2004 SAGES annual meeting. Each subject completed two repetitions of a single task on each of three simulators, MIST-VR, Endotower, and CELTS; performance scores were automatically generated and recorded. Scores of individuals with various levels of experience were compared to determine construct validity for each simulator. Experience was defined according to four parameters: (a) PGY level, (b) fellowship training, (c) basic laparoscopic cases, and (d) advanced laparoscopic cases. Subjects rated each simulator regarding six face validity (realism of simulation) parameters using a 10-point Likert scale (10 = best rating) and participant scores were compared to previously established expert scores (proficiency goals for training). Results: Ninety-one attendees completed the study. Construct validity was demonstrated for all three simulators; significant differences in scores were detected according to one parameter for MIST-VR, two param-eters for Endotower, and all four parameters for CELTS. Face validity was rated as good to excellent for all three simulators (7.0 ± 0.3 for MIST-VR, 7.9 ± 0.3 for Endotower [p < 0.001 vs MIST-VR], and 8.7 ± 0.1 for CELTS [p = 0.001 vs MIST-VR, p = 0.01 vs Endotower]); 6%, 0%, and 36% of "expert" participants obtained expert scores on MIST-VR, Endotower, and CELTS, respectively. Conclusions: All three simulators demonstrated significant construct and reasonable face validity. Although virtual reality holds great promise to expand the scope of laparoscopic simulation, current interfaces may limit their utility for assessment. Computer-enhanced video trainers may offer an improved interface while incorporating useful multidimensional metrics. Further work is needed to establish standards for appropriate skills assessment methods and performance levels using simulators.

KW - CELTS

KW - Endotower

KW - Laparoscopy

KW - MIST-VR

KW - Simulator

KW - Validity

UR - http://www.scopus.com/inward/record.url?scp=30744477478&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=30744477478&partnerID=8YFLogxK

U2 - 10.1007/s00464-005-0054-4

DO - 10.1007/s00464-005-0054-4

M3 - Article

C2 - 16333535

AN - SCOPUS:30744477478

VL - 20

SP - 104

EP - 112

JO - Surgical Endoscopy and Other Interventional Techniques

JF - Surgical Endoscopy and Other Interventional Techniques

SN - 0930-2794

IS - 1

ER -