Faculty prediction of in-training examination scores of emergency medicine residents

Background The Emergency Medicine In-Training Examination (EMITE) is one of the only valid tools for medical knowledge assessment in current use by emergency medicine (EM) residencies. However, EMITE results return late in the academic year, providing little time to institute potential remediation. Objective The goal of this study was to determine the ability of EM faculty to accurately predict resident EMITE scores prior to results return. Methods We asked EM faculty at the study site to predict the 2012 EMITE scores of the 50 EM residents 2 weeks prior to results being available. The primary outcome was prediction accuracy, defined as the proportion of predictions within 6% of the actual score. The secondary outcome was prediction precision, defined as the mean deviation of predictions from the actual scores. We assessed several faculty background variables, including years of experience, educational leadership status, and clinical hours worked, for correlation with the two outcomes. Results Thirty-two of the 38 faculty (84.2%, 95% confidence interval [CI] 69.6-92.6) participated in the study, rendering a total of 1600 predictions for 50 residents. Mean resident EMITE score was 81.1% (95% CI 79.5-82.8%). Mean prediction accuracy for all faculty participants was 69% (95% CI 65.9-72.1%). Mean prediction precision was 5.2% (95% CI 4.9-5.5%). Education leadership status was the only background variable correlated with the primary and secondary outcomes (Spearman's ρ = 0.51 and -0.53, respectively). Conclusion Faculty possess only moderate accuracy at predicting resident EMITE scores. We recommend a multicenter study to evaluate the generalizability of the present results.