Method agreement analysis and interobserver reliability of the ISTH proposed definitions for effective hemostasis in management of major bleeding

Rahat A. Abdoellakhan, Jan Beyer-Westendorf, Sam Schulman, Ravindra Sarode, Karina Meijer, Nakisa Khorsand

Research output: Contribution to journalArticle

Abstract

Essentials In 2016 the SSC proposed definitions for effective hemostasis in management of major bleeding. To validate these definitions, we studied the use in three large anticoagulant-reversal studies. Method agreement analysis and interobserver reliability showed at least acceptable agreement. Recommendations were made, advising use of the definition in hemostatic effectiveness studies. Summary: Introduction In 2016 the Scientific and Standardization Subcommittee (SSC) on Control of Anticoagulation of the International Society on Thrombosis and Haemostasis (ISTH) proposed criteria to evaluate the effectiveness of anticoagulant reversal in major bleeding management. Testing and validation of these criteria are required. Objective To investigate the method agreement, interobserver reliability and applicability of the ISTH proposed definitions for hemostatic effectiveness. Methods Patient data from three anticoagulant-antidote studies were used for hemostatic effectiveness assessment using the ISTH-proposed definitions and clinical opinion. For every patient a case document was produced. For each cohort, four adjudicators were asked to assess the hemostatic effectiveness independently on a case-by-case basis. Agreement between the two methods of hemostatic effectiveness assessment was calculated using Cohen's kappa (κ), with a calculated sample size of at least 73 cases. Results The full dataset consisted of 116 cases, resulting in 464 assessments. Method agreement in outcome was observed in 364 of 464 assessments (78.5%), resulting in κ of 0.634 (95% CI: 0.575–0.694), or “substantial agreement.” Interobserver reliability analysis of the proposed definitions computed an overall agreement of 54.2% with κ of 0.312 (“fair agreement”). Discussion Method agreement analysis shows that the conclusions drawn using the ISTH definitions have “substantial agreement” with clinical opinion. Interobserver reliability analysis demonstrated acceptable agreement. In-depth analysis provided minor opportunities for further improvement and correct application of the definition. The definition is recommended to be used in all future studies evaluating hemostatic effectiveness, taking the suggested recommendations into account.

Original languageEnglish (US)
JournalJournal of Thrombosis and Haemostasis
DOIs
StatePublished - Jan 1 2019

Fingerprint

Hemostatics
Hemostasis
Thrombosis
Hemorrhage
Anticoagulants
Antidotes
Sample Size

Keywords

  • anticoagulants
  • bleeding
  • hemostasis
  • outcome assessment
  • prothrombin complex concentrates

ASJC Scopus subject areas

  • Hematology

Cite this

Method agreement analysis and interobserver reliability of the ISTH proposed definitions for effective hemostasis in management of major bleeding. / Abdoellakhan, Rahat A.; Beyer-Westendorf, Jan; Schulman, Sam; Sarode, Ravindra; Meijer, Karina; Khorsand, Nakisa.

In: Journal of Thrombosis and Haemostasis, 01.01.2019.

Research output: Contribution to journalArticle

@article{0db24dc871d94482b4b9b3f66fb34da4,
title = "Method agreement analysis and interobserver reliability of the ISTH proposed definitions for effective hemostasis in management of major bleeding",
abstract = "Essentials In 2016 the SSC proposed definitions for effective hemostasis in management of major bleeding. To validate these definitions, we studied the use in three large anticoagulant-reversal studies. Method agreement analysis and interobserver reliability showed at least acceptable agreement. Recommendations were made, advising use of the definition in hemostatic effectiveness studies. Summary: Introduction In 2016 the Scientific and Standardization Subcommittee (SSC) on Control of Anticoagulation of the International Society on Thrombosis and Haemostasis (ISTH) proposed criteria to evaluate the effectiveness of anticoagulant reversal in major bleeding management. Testing and validation of these criteria are required. Objective To investigate the method agreement, interobserver reliability and applicability of the ISTH proposed definitions for hemostatic effectiveness. Methods Patient data from three anticoagulant-antidote studies were used for hemostatic effectiveness assessment using the ISTH-proposed definitions and clinical opinion. For every patient a case document was produced. For each cohort, four adjudicators were asked to assess the hemostatic effectiveness independently on a case-by-case basis. Agreement between the two methods of hemostatic effectiveness assessment was calculated using Cohen's kappa (κ), with a calculated sample size of at least 73 cases. Results The full dataset consisted of 116 cases, resulting in 464 assessments. Method agreement in outcome was observed in 364 of 464 assessments (78.5{\%}), resulting in κ of 0.634 (95{\%} CI: 0.575–0.694), or “substantial agreement.” Interobserver reliability analysis of the proposed definitions computed an overall agreement of 54.2{\%} with κ of 0.312 (“fair agreement”). Discussion Method agreement analysis shows that the conclusions drawn using the ISTH definitions have “substantial agreement” with clinical opinion. Interobserver reliability analysis demonstrated acceptable agreement. In-depth analysis provided minor opportunities for further improvement and correct application of the definition. The definition is recommended to be used in all future studies evaluating hemostatic effectiveness, taking the suggested recommendations into account.",
keywords = "anticoagulants, bleeding, hemostasis, outcome assessment, prothrombin complex concentrates",
author = "Abdoellakhan, {Rahat A.} and Jan Beyer-Westendorf and Sam Schulman and Ravindra Sarode and Karina Meijer and Nakisa Khorsand",
year = "2019",
month = "1",
day = "1",
doi = "10.1111/jth.14388",
language = "English (US)",
journal = "Journal of Thrombosis and Haemostasis",
issn = "1538-7933",
publisher = "Wiley-Blackwell",

}

TY - JOUR

T1 - Method agreement analysis and interobserver reliability of the ISTH proposed definitions for effective hemostasis in management of major bleeding

AU - Abdoellakhan, Rahat A.

AU - Beyer-Westendorf, Jan

AU - Schulman, Sam

AU - Sarode, Ravindra

AU - Meijer, Karina

AU - Khorsand, Nakisa

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Essentials In 2016 the SSC proposed definitions for effective hemostasis in management of major bleeding. To validate these definitions, we studied the use in three large anticoagulant-reversal studies. Method agreement analysis and interobserver reliability showed at least acceptable agreement. Recommendations were made, advising use of the definition in hemostatic effectiveness studies. Summary: Introduction In 2016 the Scientific and Standardization Subcommittee (SSC) on Control of Anticoagulation of the International Society on Thrombosis and Haemostasis (ISTH) proposed criteria to evaluate the effectiveness of anticoagulant reversal in major bleeding management. Testing and validation of these criteria are required. Objective To investigate the method agreement, interobserver reliability and applicability of the ISTH proposed definitions for hemostatic effectiveness. Methods Patient data from three anticoagulant-antidote studies were used for hemostatic effectiveness assessment using the ISTH-proposed definitions and clinical opinion. For every patient a case document was produced. For each cohort, four adjudicators were asked to assess the hemostatic effectiveness independently on a case-by-case basis. Agreement between the two methods of hemostatic effectiveness assessment was calculated using Cohen's kappa (κ), with a calculated sample size of at least 73 cases. Results The full dataset consisted of 116 cases, resulting in 464 assessments. Method agreement in outcome was observed in 364 of 464 assessments (78.5%), resulting in κ of 0.634 (95% CI: 0.575–0.694), or “substantial agreement.” Interobserver reliability analysis of the proposed definitions computed an overall agreement of 54.2% with κ of 0.312 (“fair agreement”). Discussion Method agreement analysis shows that the conclusions drawn using the ISTH definitions have “substantial agreement” with clinical opinion. Interobserver reliability analysis demonstrated acceptable agreement. In-depth analysis provided minor opportunities for further improvement and correct application of the definition. The definition is recommended to be used in all future studies evaluating hemostatic effectiveness, taking the suggested recommendations into account.

AB - Essentials In 2016 the SSC proposed definitions for effective hemostasis in management of major bleeding. To validate these definitions, we studied the use in three large anticoagulant-reversal studies. Method agreement analysis and interobserver reliability showed at least acceptable agreement. Recommendations were made, advising use of the definition in hemostatic effectiveness studies. Summary: Introduction In 2016 the Scientific and Standardization Subcommittee (SSC) on Control of Anticoagulation of the International Society on Thrombosis and Haemostasis (ISTH) proposed criteria to evaluate the effectiveness of anticoagulant reversal in major bleeding management. Testing and validation of these criteria are required. Objective To investigate the method agreement, interobserver reliability and applicability of the ISTH proposed definitions for hemostatic effectiveness. Methods Patient data from three anticoagulant-antidote studies were used for hemostatic effectiveness assessment using the ISTH-proposed definitions and clinical opinion. For every patient a case document was produced. For each cohort, four adjudicators were asked to assess the hemostatic effectiveness independently on a case-by-case basis. Agreement between the two methods of hemostatic effectiveness assessment was calculated using Cohen's kappa (κ), with a calculated sample size of at least 73 cases. Results The full dataset consisted of 116 cases, resulting in 464 assessments. Method agreement in outcome was observed in 364 of 464 assessments (78.5%), resulting in κ of 0.634 (95% CI: 0.575–0.694), or “substantial agreement.” Interobserver reliability analysis of the proposed definitions computed an overall agreement of 54.2% with κ of 0.312 (“fair agreement”). Discussion Method agreement analysis shows that the conclusions drawn using the ISTH definitions have “substantial agreement” with clinical opinion. Interobserver reliability analysis demonstrated acceptable agreement. In-depth analysis provided minor opportunities for further improvement and correct application of the definition. The definition is recommended to be used in all future studies evaluating hemostatic effectiveness, taking the suggested recommendations into account.

KW - anticoagulants

KW - bleeding

KW - hemostasis

KW - outcome assessment

KW - prothrombin complex concentrates

UR - http://www.scopus.com/inward/record.url?scp=85061497416&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061497416&partnerID=8YFLogxK

U2 - 10.1111/jth.14388

DO - 10.1111/jth.14388

M3 - Article

C2 - 30657628

AN - SCOPUS:85061497416

JO - Journal of Thrombosis and Haemostasis

JF - Journal of Thrombosis and Haemostasis

SN - 1538-7933

ER -