A novel workplace-based assessment for competency-based decisions and learner feedback

the PMAC Module 1 Study Group

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Background: Increased recognition of the importance of competency-based education and assessment has led to the need for practical and reliable methods to assess relevant skills in the workplace. Methods: A novel milestone-based workplace assessment system was implemented in 15 pediatrics residency programs. The system provided: (1) web-based multisource feedback (MSF) and structured clinical observation (SCO) instruments that could be completed on any computer or mobile device; and (2) monthly feedback reports that included competency-level scores and recommendations for improvement. Results: For the final instruments, an average of five MSF and 3.7 SCO assessment instruments were completed for each of 292 interns; instruments required an average of 4–8 min to complete. Generalizability coefficients >0.80 were attainable with six MSF observations. Users indicated that the new system added value to their existing assessment program; the need to complete the local assessments in addition to the new assessments was identified as a burden of the overall process. Conclusions: Outcomes–including high participation rates and high reliability compared to what has traditionally been found with workplace-based assessment–provide evidence for the validity of scores resulting from this novel competency-based assessment system. The development of this assessment model is generalizable to other specialties.

Original languageEnglish (US)
Pages (from-to)1143-1150
Number of pages8
JournalMedical Teacher
Volume40
Issue number11
DOIs
StatePublished - Nov 2 2018
Externally publishedYes

Fingerprint

workplace
value added
participation
evidence
education

ASJC Scopus subject areas

  • Education

Cite this

A novel workplace-based assessment for competency-based decisions and learner feedback. / the PMAC Module 1 Study Group.

In: Medical Teacher, Vol. 40, No. 11, 02.11.2018, p. 1143-1150.

Research output: Contribution to journalArticle

the PMAC Module 1 Study Group. / A novel workplace-based assessment for competency-based decisions and learner feedback. In: Medical Teacher. 2018 ; Vol. 40, No. 11. pp. 1143-1150.
@article{f9b7f33413574720bfd253627bf44ef9,
title = "A novel workplace-based assessment for competency-based decisions and learner feedback",
abstract = "Background: Increased recognition of the importance of competency-based education and assessment has led to the need for practical and reliable methods to assess relevant skills in the workplace. Methods: A novel milestone-based workplace assessment system was implemented in 15 pediatrics residency programs. The system provided: (1) web-based multisource feedback (MSF) and structured clinical observation (SCO) instruments that could be completed on any computer or mobile device; and (2) monthly feedback reports that included competency-level scores and recommendations for improvement. Results: For the final instruments, an average of five MSF and 3.7 SCO assessment instruments were completed for each of 292 interns; instruments required an average of 4–8 min to complete. Generalizability coefficients >0.80 were attainable with six MSF observations. Users indicated that the new system added value to their existing assessment program; the need to complete the local assessments in addition to the new assessments was identified as a burden of the overall process. Conclusions: Outcomes–including high participation rates and high reliability compared to what has traditionally been found with workplace-based assessment–provide evidence for the validity of scores resulting from this novel competency-based assessment system. The development of this assessment model is generalizable to other specialties.",
author = "{the PMAC Module 1 Study Group} and Hicks, {Patricia J.} and Margolis, {Melissa J.} and Carraccio, {Carol L.} and Clauser, {Brian E.} and Kathleen Donnelly and Fromme, {H. Barrett} and Gifford, {Kimberly A.} and Poynter, {Sue E.} and Schumacher, {Daniel J.} and Alan Schwartz",
year = "2018",
month = "11",
day = "2",
doi = "10.1080/0142159X.2018.1461204",
language = "English (US)",
volume = "40",
pages = "1143--1150",
journal = "Medical Teacher",
issn = "0142-159X",
publisher = "Informa Healthcare",
number = "11",

}

TY - JOUR

T1 - A novel workplace-based assessment for competency-based decisions and learner feedback

AU - the PMAC Module 1 Study Group

AU - Hicks, Patricia J.

AU - Margolis, Melissa J.

AU - Carraccio, Carol L.

AU - Clauser, Brian E.

AU - Donnelly, Kathleen

AU - Fromme, H. Barrett

AU - Gifford, Kimberly A.

AU - Poynter, Sue E.

AU - Schumacher, Daniel J.

AU - Schwartz, Alan

PY - 2018/11/2

Y1 - 2018/11/2

N2 - Background: Increased recognition of the importance of competency-based education and assessment has led to the need for practical and reliable methods to assess relevant skills in the workplace. Methods: A novel milestone-based workplace assessment system was implemented in 15 pediatrics residency programs. The system provided: (1) web-based multisource feedback (MSF) and structured clinical observation (SCO) instruments that could be completed on any computer or mobile device; and (2) monthly feedback reports that included competency-level scores and recommendations for improvement. Results: For the final instruments, an average of five MSF and 3.7 SCO assessment instruments were completed for each of 292 interns; instruments required an average of 4–8 min to complete. Generalizability coefficients >0.80 were attainable with six MSF observations. Users indicated that the new system added value to their existing assessment program; the need to complete the local assessments in addition to the new assessments was identified as a burden of the overall process. Conclusions: Outcomes–including high participation rates and high reliability compared to what has traditionally been found with workplace-based assessment–provide evidence for the validity of scores resulting from this novel competency-based assessment system. The development of this assessment model is generalizable to other specialties.

AB - Background: Increased recognition of the importance of competency-based education and assessment has led to the need for practical and reliable methods to assess relevant skills in the workplace. Methods: A novel milestone-based workplace assessment system was implemented in 15 pediatrics residency programs. The system provided: (1) web-based multisource feedback (MSF) and structured clinical observation (SCO) instruments that could be completed on any computer or mobile device; and (2) monthly feedback reports that included competency-level scores and recommendations for improvement. Results: For the final instruments, an average of five MSF and 3.7 SCO assessment instruments were completed for each of 292 interns; instruments required an average of 4–8 min to complete. Generalizability coefficients >0.80 were attainable with six MSF observations. Users indicated that the new system added value to their existing assessment program; the need to complete the local assessments in addition to the new assessments was identified as a burden of the overall process. Conclusions: Outcomes–including high participation rates and high reliability compared to what has traditionally been found with workplace-based assessment–provide evidence for the validity of scores resulting from this novel competency-based assessment system. The development of this assessment model is generalizable to other specialties.

UR - http://www.scopus.com/inward/record.url?scp=85046012909&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85046012909&partnerID=8YFLogxK

U2 - 10.1080/0142159X.2018.1461204

DO - 10.1080/0142159X.2018.1461204

M3 - Article

C2 - 29688108

AN - SCOPUS:85046012909

VL - 40

SP - 1143

EP - 1150

JO - Medical Teacher

JF - Medical Teacher

SN - 0142-159X

IS - 11

ER -