Abstract
We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater's score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater's performance characteristics, including bias, discrimination, and measurement error in the ratings. The proposed approach aims at three goals. First, three Bayesian estimators are introduced to estimate the ranks of items. They all show a substantial improvement over the widely used score sums by using the information on the variable skill of the raters. Second, rater performance can be compared based on rater bias, discrimination, and measurement error. Third, a simulation-based decision-theoretic approach is described to determine the number of raters to employ. A simulation study and an analysis based on a grant review data set are presented.
| Original language | English (US) |
|---|---|
| Pages (from-to) | 194-214 |
| Number of pages | 21 |
| Journal | Journal of Educational and Behavioral Statistics |
| Volume | 35 |
| Issue number | 2 |
| DOIs | |
| State | Published - Jun 2010 |
Fingerprint
Keywords
- Bayesian hierarchical model
- Multirater ordinal data
- Number of raters
- Ranking estimator
- Rater performance
ASJC Scopus subject areas
- Education
- Social Sciences (miscellaneous)
Cite this
A Bayesian approach to ranking and rater evaluation : An application to grant reviews. / Cao, Jing; Stokes, S. Lynne; Zhang, Song.
In: Journal of Educational and Behavioral Statistics, Vol. 35, No. 2, 06.2010, p. 194-214.Research output: Contribution to journal › Article
}
TY - JOUR
T1 - A Bayesian approach to ranking and rater evaluation
T2 - An application to grant reviews
AU - Cao, Jing
AU - Stokes, S. Lynne
AU - Zhang, Song
PY - 2010/6
Y1 - 2010/6
N2 - We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater's score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater's performance characteristics, including bias, discrimination, and measurement error in the ratings. The proposed approach aims at three goals. First, three Bayesian estimators are introduced to estimate the ranks of items. They all show a substantial improvement over the widely used score sums by using the information on the variable skill of the raters. Second, rater performance can be compared based on rater bias, discrimination, and measurement error. Third, a simulation-based decision-theoretic approach is described to determine the number of raters to employ. A simulation study and an analysis based on a grant review data set are presented.
AB - We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater's score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater's performance characteristics, including bias, discrimination, and measurement error in the ratings. The proposed approach aims at three goals. First, three Bayesian estimators are introduced to estimate the ranks of items. They all show a substantial improvement over the widely used score sums by using the information on the variable skill of the raters. Second, rater performance can be compared based on rater bias, discrimination, and measurement error. Third, a simulation-based decision-theoretic approach is described to determine the number of raters to employ. A simulation study and an analysis based on a grant review data set are presented.
KW - Bayesian hierarchical model
KW - Multirater ordinal data
KW - Number of raters
KW - Ranking estimator
KW - Rater performance
UR - http://www.scopus.com/inward/record.url?scp=77952527478&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=77952527478&partnerID=8YFLogxK
U2 - 10.3102/1076998609353116
DO - 10.3102/1076998609353116
M3 - Article
AN - SCOPUS:77952527478
VL - 35
SP - 194
EP - 214
JO - Journal of Educational and Behavioral Statistics
JF - Journal of Educational and Behavioral Statistics
SN - 1076-9986
IS - 2
ER -