The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial

Jordan D. Bohnen, Brian C. George, Reed G. Williams, Mary C. Schuller, Debra A. DaRosa, Laura Torbeck, John T. Mullen, Shari L. Meyerson, Edward D. Auyang, Jeffrey G. Chipman, Jennifer N. Choi, Michael A. Choti, Eric D. Endean, Eugene F. Foley, Samuel P. Mandell, Andreas H. Meier, Douglas S. Smink, Kyla P. Terhune, Paul E. Wise, Nathaniel J. SoperJoseph B. Zwischenberger, Keith D. Lillemoe, Gary L. Dunnington, Jonathan P. Fryer

Research output: Contribution to journalReview article

43 Citations (Scopus)

Abstract

Purpose Intraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs. Methods Between September 1, 2015 and February 29, 2016, 15 U.S. general surgery residency programs were enrolled in an institutional review board-approved trial. SIMPL was made available after 70% of faculty and residents completed a 1-hour training session. Descriptive and univariate statistics analyzed multiple dimensions of feasibility, including training rates, volume of assessments, response rates/times, and dictation rates. The 20 most active residents and attendings were evaluated in greater detail. Results A total of 90% of eligible users (1267/1412) completed training. Further, 13/15 programs began using SIMPL. Totally, 6024 assessments were completed by 254 categorical general surgery residents (n = 3555 assessments) and 259 attendings (n = 2469 assessments), and 3762 unique operations were assessed. There was significant heterogeneity in participation within and between programs. Mean percentage (range) of users who completed ≥1, 5, and 20 assessments were 62% (21%-96%), 34% (5%-75%), and 10% (0%-32%) across all programs, and 96%, 75%, and 32% in the most active program. Overall, response rate was 70%, dictation rate was 24%, and mean response time was 12 hours. Assessments increased from 357 (September 2015) to 1146 (February 2016). The 20 most active residents each received mean 46 assessments by 10 attendings for 20 different procedures. Conclusions SIMPL can be feasibly integrated into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. We believe SIMPL could help facilitate a national competency-based surgical training system, although local and systemic challenges still need to be addressed.

Original languageEnglish (US)
Pages (from-to)e118-e130
JournalJournal of Surgical Education
Volume73
Issue number6
DOIs
StatePublished - Nov 1 2016

Fingerprint

performance assessment
Learning
learning
resident
experience
trainee
surgery
Education
training program
Research Ethics Committees
Internship and Residency
Reaction Time
participation
time
statistics
evaluation

Keywords

  • autonomy and Zwisch scale
  • Interpersonal and Communication Skills
  • intraoperative feedback
  • Medical Knowledge
  • mobile technology
  • operative performance assessment
  • Patient Care
  • Practice-Based Learning and Improvement
  • Professionalism
  • SIMPL
  • smartphone

ASJC Scopus subject areas

  • Surgery
  • Education

Cite this

The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning) : Early Experience From a Multi-institutional Trial. / Bohnen, Jordan D.; George, Brian C.; Williams, Reed G.; Schuller, Mary C.; DaRosa, Debra A.; Torbeck, Laura; Mullen, John T.; Meyerson, Shari L.; Auyang, Edward D.; Chipman, Jeffrey G.; Choi, Jennifer N.; Choti, Michael A.; Endean, Eric D.; Foley, Eugene F.; Mandell, Samuel P.; Meier, Andreas H.; Smink, Douglas S.; Terhune, Kyla P.; Wise, Paul E.; Soper, Nathaniel J.; Zwischenberger, Joseph B.; Lillemoe, Keith D.; Dunnington, Gary L.; Fryer, Jonathan P.

In: Journal of Surgical Education, Vol. 73, No. 6, 01.11.2016, p. e118-e130.

Research output: Contribution to journalReview article

Bohnen, JD, George, BC, Williams, RG, Schuller, MC, DaRosa, DA, Torbeck, L, Mullen, JT, Meyerson, SL, Auyang, ED, Chipman, JG, Choi, JN, Choti, MA, Endean, ED, Foley, EF, Mandell, SP, Meier, AH, Smink, DS, Terhune, KP, Wise, PE, Soper, NJ, Zwischenberger, JB, Lillemoe, KD, Dunnington, GL & Fryer, JP 2016, 'The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial', Journal of Surgical Education, vol. 73, no. 6, pp. e118-e130. https://doi.org/10.1016/j.jsurg.2016.08.010
Bohnen, Jordan D. ; George, Brian C. ; Williams, Reed G. ; Schuller, Mary C. ; DaRosa, Debra A. ; Torbeck, Laura ; Mullen, John T. ; Meyerson, Shari L. ; Auyang, Edward D. ; Chipman, Jeffrey G. ; Choi, Jennifer N. ; Choti, Michael A. ; Endean, Eric D. ; Foley, Eugene F. ; Mandell, Samuel P. ; Meier, Andreas H. ; Smink, Douglas S. ; Terhune, Kyla P. ; Wise, Paul E. ; Soper, Nathaniel J. ; Zwischenberger, Joseph B. ; Lillemoe, Keith D. ; Dunnington, Gary L. ; Fryer, Jonathan P. / The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning) : Early Experience From a Multi-institutional Trial. In: Journal of Surgical Education. 2016 ; Vol. 73, No. 6. pp. e118-e130.
@article{a0a68b7f66d44f2c9b24bbf045c5d782,
title = "The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial",
abstract = "Purpose Intraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs. Methods Between September 1, 2015 and February 29, 2016, 15 U.S. general surgery residency programs were enrolled in an institutional review board-approved trial. SIMPL was made available after 70{\%} of faculty and residents completed a 1-hour training session. Descriptive and univariate statistics analyzed multiple dimensions of feasibility, including training rates, volume of assessments, response rates/times, and dictation rates. The 20 most active residents and attendings were evaluated in greater detail. Results A total of 90{\%} of eligible users (1267/1412) completed training. Further, 13/15 programs began using SIMPL. Totally, 6024 assessments were completed by 254 categorical general surgery residents (n = 3555 assessments) and 259 attendings (n = 2469 assessments), and 3762 unique operations were assessed. There was significant heterogeneity in participation within and between programs. Mean percentage (range) of users who completed ≥1, 5, and 20 assessments were 62{\%} (21{\%}-96{\%}), 34{\%} (5{\%}-75{\%}), and 10{\%} (0{\%}-32{\%}) across all programs, and 96{\%}, 75{\%}, and 32{\%} in the most active program. Overall, response rate was 70{\%}, dictation rate was 24{\%}, and mean response time was 12 hours. Assessments increased from 357 (September 2015) to 1146 (February 2016). The 20 most active residents each received mean 46 assessments by 10 attendings for 20 different procedures. Conclusions SIMPL can be feasibly integrated into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. We believe SIMPL could help facilitate a national competency-based surgical training system, although local and systemic challenges still need to be addressed.",
keywords = "autonomy and Zwisch scale, Interpersonal and Communication Skills, intraoperative feedback, Medical Knowledge, mobile technology, operative performance assessment, Patient Care, Practice-Based Learning and Improvement, Professionalism, SIMPL, smartphone",
author = "Bohnen, {Jordan D.} and George, {Brian C.} and Williams, {Reed G.} and Schuller, {Mary C.} and DaRosa, {Debra A.} and Laura Torbeck and Mullen, {John T.} and Meyerson, {Shari L.} and Auyang, {Edward D.} and Chipman, {Jeffrey G.} and Choi, {Jennifer N.} and Choti, {Michael A.} and Endean, {Eric D.} and Foley, {Eugene F.} and Mandell, {Samuel P.} and Meier, {Andreas H.} and Smink, {Douglas S.} and Terhune, {Kyla P.} and Wise, {Paul E.} and Soper, {Nathaniel J.} and Zwischenberger, {Joseph B.} and Lillemoe, {Keith D.} and Dunnington, {Gary L.} and Fryer, {Jonathan P.}",
year = "2016",
month = "11",
day = "1",
doi = "10.1016/j.jsurg.2016.08.010",
language = "English (US)",
volume = "73",
pages = "e118--e130",
journal = "Journal of Surgical Education",
issn = "1931-7204",
publisher = "Elsevier Inc.",
number = "6",

}

TY - JOUR

T1 - The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning)

T2 - Early Experience From a Multi-institutional Trial

AU - Bohnen, Jordan D.

AU - George, Brian C.

AU - Williams, Reed G.

AU - Schuller, Mary C.

AU - DaRosa, Debra A.

AU - Torbeck, Laura

AU - Mullen, John T.

AU - Meyerson, Shari L.

AU - Auyang, Edward D.

AU - Chipman, Jeffrey G.

AU - Choi, Jennifer N.

AU - Choti, Michael A.

AU - Endean, Eric D.

AU - Foley, Eugene F.

AU - Mandell, Samuel P.

AU - Meier, Andreas H.

AU - Smink, Douglas S.

AU - Terhune, Kyla P.

AU - Wise, Paul E.

AU - Soper, Nathaniel J.

AU - Zwischenberger, Joseph B.

AU - Lillemoe, Keith D.

AU - Dunnington, Gary L.

AU - Fryer, Jonathan P.

PY - 2016/11/1

Y1 - 2016/11/1

N2 - Purpose Intraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs. Methods Between September 1, 2015 and February 29, 2016, 15 U.S. general surgery residency programs were enrolled in an institutional review board-approved trial. SIMPL was made available after 70% of faculty and residents completed a 1-hour training session. Descriptive and univariate statistics analyzed multiple dimensions of feasibility, including training rates, volume of assessments, response rates/times, and dictation rates. The 20 most active residents and attendings were evaluated in greater detail. Results A total of 90% of eligible users (1267/1412) completed training. Further, 13/15 programs began using SIMPL. Totally, 6024 assessments were completed by 254 categorical general surgery residents (n = 3555 assessments) and 259 attendings (n = 2469 assessments), and 3762 unique operations were assessed. There was significant heterogeneity in participation within and between programs. Mean percentage (range) of users who completed ≥1, 5, and 20 assessments were 62% (21%-96%), 34% (5%-75%), and 10% (0%-32%) across all programs, and 96%, 75%, and 32% in the most active program. Overall, response rate was 70%, dictation rate was 24%, and mean response time was 12 hours. Assessments increased from 357 (September 2015) to 1146 (February 2016). The 20 most active residents each received mean 46 assessments by 10 attendings for 20 different procedures. Conclusions SIMPL can be feasibly integrated into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. We believe SIMPL could help facilitate a national competency-based surgical training system, although local and systemic challenges still need to be addressed.

AB - Purpose Intraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs. Methods Between September 1, 2015 and February 29, 2016, 15 U.S. general surgery residency programs were enrolled in an institutional review board-approved trial. SIMPL was made available after 70% of faculty and residents completed a 1-hour training session. Descriptive and univariate statistics analyzed multiple dimensions of feasibility, including training rates, volume of assessments, response rates/times, and dictation rates. The 20 most active residents and attendings were evaluated in greater detail. Results A total of 90% of eligible users (1267/1412) completed training. Further, 13/15 programs began using SIMPL. Totally, 6024 assessments were completed by 254 categorical general surgery residents (n = 3555 assessments) and 259 attendings (n = 2469 assessments), and 3762 unique operations were assessed. There was significant heterogeneity in participation within and between programs. Mean percentage (range) of users who completed ≥1, 5, and 20 assessments were 62% (21%-96%), 34% (5%-75%), and 10% (0%-32%) across all programs, and 96%, 75%, and 32% in the most active program. Overall, response rate was 70%, dictation rate was 24%, and mean response time was 12 hours. Assessments increased from 357 (September 2015) to 1146 (February 2016). The 20 most active residents each received mean 46 assessments by 10 attendings for 20 different procedures. Conclusions SIMPL can be feasibly integrated into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. We believe SIMPL could help facilitate a national competency-based surgical training system, although local and systemic challenges still need to be addressed.

KW - autonomy and Zwisch scale

KW - Interpersonal and Communication Skills

KW - intraoperative feedback

KW - Medical Knowledge

KW - mobile technology

KW - operative performance assessment

KW - Patient Care

KW - Practice-Based Learning and Improvement

KW - Professionalism

KW - SIMPL

KW - smartphone

UR - http://www.scopus.com/inward/record.url?scp=84997771388&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84997771388&partnerID=8YFLogxK

U2 - 10.1016/j.jsurg.2016.08.010

DO - 10.1016/j.jsurg.2016.08.010

M3 - Review article

C2 - 27886971

AN - SCOPUS:84997771388

VL - 73

SP - e118-e130

JO - Journal of Surgical Education

JF - Journal of Surgical Education

SN - 1931-7204

IS - 6

ER -