Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth from Ten Pediatrics Residency Programs

Yoon Soo Park, Patricia J. Hicks, Carol Carraccio, Melissa Margolis, Alan Schwartz

Research output: Contribution to journalArticlepeer-review

5 Scopus citations

Abstract

Purpose This study investigates the impact of incorporating observer-reported workload into workplace-based assessment (WBA) scores on (1) psychometric characteristics of WBA scores and (2) measuring changes in performance over time using workloadunadjusted versus workload-adjusted scores. Method Structured clinical observations and multisource feedback instruments were used to collect WBA data from first-year pediatrics residents at 10 residency programs between July 2016 and June 2017. Observers completed items in 8 subcompetencies associated with Pediatrics Milestones. Faculty and resident observers assessed workload using a sliding scale ranging from low to high; all item scores were rescaled to a 1-5 scale to facilitate analysis and interpretation. Workloadadjusted WBA scores were calculated at the item level using three different approaches, and aggregated for analysis at the competency level. Mixed-effects regression models were used to estimate variance components. Longitudinal growth curve analyses examined patterns of developmental score change over time. Results On average, participating residents (n = 252) were assessed 5.32 times (standard deviation = 3.79) by different raters during the data collection period. Adjusting for workload yielded better discrimination of learner performance, and higher reliability, reducing measurement error by 28%. Projections in reliability indicated needing up to twice the number of raters when workloadunadjusted scores were used. Longitudinal analysis showed an increase in scores over time, with significant interaction between workload and time; workload also increased significantly over time. Conclusions Incorporating a measure of observerreported workload could improve the measurement properties and the ability to interpret WBA scores.

Original languageEnglish (US)
Pages (from-to)S21-S29
JournalAcademic Medicine
Volume93
Issue number11 S
DOIs
StatePublished - 2018

ASJC Scopus subject areas

  • Education

Fingerprint

Dive into the research topics of 'Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth from Ten Pediatrics Residency Programs'. Together they form a unique fingerprint.

Cite this