TY - JOUR
T1 - Exploring the behavior of hospital composite performance measures
T2 - An example from coronary artery bypass surgery
AU - O'Brien, Sean M.
AU - Delong, Elizabeth R.
AU - Dokholyan, Rachel S.
AU - Edwards, Fred H.
AU - Peterson, Eric D.
PY - 2007/12
Y1 - 2007/12
N2 - BACKGROUND - Composite scores that combine several performance measures into a single ranking are becoming the accepted metric for assessing hospital performance. In particular, the Centers for Medicare & Medicaid Services Hospital Quality Incentive Demonstration (HQID) project bases financial rewards and penalties on these scores. Although the HQID composite calculation is straightforward and easily understood, its method of combining process and outcome measures has not been validated. METHODS AND RESULTS - Using data on 530 hospitals from the Society of Thoracic Surgeons National Cardiac Database, we replicated the HQID methodology with 6 nationally endorsed performance measures (5 process measures plus survival) for coronary artery bypass surgery. Composite scores were essentially determined by process measure performance alone; the survival component explained only 4% of the composite score's total variance. This result persisted even when the survival component was allowed a 5-fold greater weighting in the composite summary. The popular "all-or-none" measurement approach was also dominated by the process component. Substantial disagreement was found among hospital rankings when several alternative methods were used; up to 60% of hospitals eligible for the top financial reward under HQID would change designation depending on the composite methodology used. The application of a simple statistical adjustment (standardization) to each method would provide more consistent results and a more balanced assessment of performance based on both process and outcomes. CONCLUSIONS - Existing methods used to create composite performance measures have remarkably different weighting of process versus outcomes metrics and lead to highly divergent provider rankings. Simple alternative methods can create more balanced process-outcome performance assessments.
AB - BACKGROUND - Composite scores that combine several performance measures into a single ranking are becoming the accepted metric for assessing hospital performance. In particular, the Centers for Medicare & Medicaid Services Hospital Quality Incentive Demonstration (HQID) project bases financial rewards and penalties on these scores. Although the HQID composite calculation is straightforward and easily understood, its method of combining process and outcome measures has not been validated. METHODS AND RESULTS - Using data on 530 hospitals from the Society of Thoracic Surgeons National Cardiac Database, we replicated the HQID methodology with 6 nationally endorsed performance measures (5 process measures plus survival) for coronary artery bypass surgery. Composite scores were essentially determined by process measure performance alone; the survival component explained only 4% of the composite score's total variance. This result persisted even when the survival component was allowed a 5-fold greater weighting in the composite summary. The popular "all-or-none" measurement approach was also dominated by the process component. Substantial disagreement was found among hospital rankings when several alternative methods were used; up to 60% of hospitals eligible for the top financial reward under HQID would change designation depending on the composite methodology used. The application of a simple statistical adjustment (standardization) to each method would provide more consistent results and a more balanced assessment of performance based on both process and outcomes. CONCLUSIONS - Existing methods used to create composite performance measures have remarkably different weighting of process versus outcomes metrics and lead to highly divergent provider rankings. Simple alternative methods can create more balanced process-outcome performance assessments.
KW - Coronary artery bypass
KW - Program evaluation
KW - Quality of health care
UR - http://www.scopus.com/inward/record.url?scp=37349083875&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=37349083875&partnerID=8YFLogxK
U2 - 10.1161/CIRCULATIONAHA.107.703553
DO - 10.1161/CIRCULATIONAHA.107.703553
M3 - Article
C2 - 18056529
AN - SCOPUS:37349083875
SN - 0009-7322
VL - 116
SP - 2969
EP - 2975
JO - Circulation
JF - Circulation
IS - 25
ER -