A Bayesian approach to ranking and rater evaluation: An application to grant reviews

Jing Cao, S. Lynne Stokes, Song Zhang

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater's score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater's performance characteristics, including bias, discrimination, and measurement error in the ratings. The proposed approach aims at three goals. First, three Bayesian estimators are introduced to estimate the ranks of items. They all show a substantial improvement over the widely used score sums by using the information on the variable skill of the raters. Second, rater performance can be compared based on rater bias, discrimination, and measurement error. Third, a simulation-based decision-theoretic approach is described to determine the number of raters to employ. A simulation study and an analysis based on a grant review data set are presented.

Original languageEnglish (US)
Pages (from-to)194-214
Number of pages21
JournalJournal of Educational and Behavioral Statistics
Volume35
Issue number2
DOIs
StatePublished - Jun 2010

Fingerprint

ranking
discrimination
simulation
trend
evaluation
performance
grant
rating

Keywords

  • Bayesian hierarchical model
  • Multirater ordinal data
  • Number of raters
  • Ranking estimator
  • Rater performance

ASJC Scopus subject areas

  • Education
  • Social Sciences (miscellaneous)

Cite this

A Bayesian approach to ranking and rater evaluation : An application to grant reviews. / Cao, Jing; Stokes, S. Lynne; Zhang, Song.

In: Journal of Educational and Behavioral Statistics, Vol. 35, No. 2, 06.2010, p. 194-214.

Research output: Contribution to journalArticle

@article{98756886e5d943789dc3a398e2e63cc2,
title = "A Bayesian approach to ranking and rater evaluation: An application to grant reviews",
abstract = "We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater's score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater's performance characteristics, including bias, discrimination, and measurement error in the ratings. The proposed approach aims at three goals. First, three Bayesian estimators are introduced to estimate the ranks of items. They all show a substantial improvement over the widely used score sums by using the information on the variable skill of the raters. Second, rater performance can be compared based on rater bias, discrimination, and measurement error. Third, a simulation-based decision-theoretic approach is described to determine the number of raters to employ. A simulation study and an analysis based on a grant review data set are presented.",
keywords = "Bayesian hierarchical model, Multirater ordinal data, Number of raters, Ranking estimator, Rater performance",
author = "Jing Cao and Stokes, {S. Lynne} and Song Zhang",
year = "2010",
month = "6",
doi = "10.3102/1076998609353116",
language = "English (US)",
volume = "35",
pages = "194--214",
journal = "Journal of Educational and Behavioral Statistics",
issn = "1076-9986",
publisher = "SAGE Publications Inc.",
number = "2",

}

TY - JOUR

T1 - A Bayesian approach to ranking and rater evaluation

T2 - An application to grant reviews

AU - Cao, Jing

AU - Stokes, S. Lynne

AU - Zhang, Song

PY - 2010/6

Y1 - 2010/6

N2 - We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater's score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater's performance characteristics, including bias, discrimination, and measurement error in the ratings. The proposed approach aims at three goals. First, three Bayesian estimators are introduced to estimate the ranks of items. They all show a substantial improvement over the widely used score sums by using the information on the variable skill of the raters. Second, rater performance can be compared based on rater bias, discrimination, and measurement error. Third, a simulation-based decision-theoretic approach is described to determine the number of raters to employ. A simulation study and an analysis based on a grant review data set are presented.

AB - We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater's score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater's performance characteristics, including bias, discrimination, and measurement error in the ratings. The proposed approach aims at three goals. First, three Bayesian estimators are introduced to estimate the ranks of items. They all show a substantial improvement over the widely used score sums by using the information on the variable skill of the raters. Second, rater performance can be compared based on rater bias, discrimination, and measurement error. Third, a simulation-based decision-theoretic approach is described to determine the number of raters to employ. A simulation study and an analysis based on a grant review data set are presented.

KW - Bayesian hierarchical model

KW - Multirater ordinal data

KW - Number of raters

KW - Ranking estimator

KW - Rater performance

UR - http://www.scopus.com/inward/record.url?scp=77952527478&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77952527478&partnerID=8YFLogxK

U2 - 10.3102/1076998609353116

DO - 10.3102/1076998609353116

M3 - Article

AN - SCOPUS:77952527478

VL - 35

SP - 194

EP - 214

JO - Journal of Educational and Behavioral Statistics

JF - Journal of Educational and Behavioral Statistics

SN - 1076-9986

IS - 2

ER -