Strength in Numbers or Quality over Quantity? Examining the Importance of Criterion Measure Selection to Define Validity Groups in Performance Validity Test (PVT) Research

Jason R. Soble, W. Alexander Alverson, Jacob I. Phillips, Edan A. Critchfield, Chrystal Fullen, Justin J.F. O’Rourke, Johanna Messerly, Jonathan M. Highsmith, K. Chase Bailey, Troy A. Webber, Janice C. Marceaux

Research output: Contribution to journalArticlepeer-review

13 Scopus citations

Abstract

Mirroring clinical guidelines, recent Performance Validity Test (PVT) research emphasizes using ≥ 2 criterion PVTs to optimally identify validity groups when validating/cross-validating PVTs; however, even with multiple measures, the effect of which specific PVTs are used as criterion measures remains incompletely explored. This study investigated the accuracy of varying two-PVT combinations for establishing validity status and how adding a third PVT or applying more liberal failure cut-scores affects overall false-positive (FP)/-negative (FN) rates. Clinically referred veterans (N = 114; 30% clinically identified as invalid) completing a six-PVT protocol as during their evaluation were included. Concordance rates were calculated across all possible two-and three-PVT combinations at conservative and liberal cutoffs. Two-PVT combinations classified 72–91% of valid (0–4% FPs) and 17–74% of invalid (0–40% FNs) cases, and three-PVT combinations classified 67–86% of valid (0–6% FPs) and 57–97% of invalid (0–24% FNs) at conservative cutoffs. Liberal cutoffs classified 53–86% of valid (0–15% FPs) and 39–82% of invalid (0–30% FNs) cases for two-PVT combinations and 46–75% of valid (3–27% FPs) and 60–97% of invalid (0–17% FNs) cases for three-PVT combinations. Irrespective of whether a two-or three-PVT combination or conservative/liberal cutoffs were used, many valid and invalid cases failed only one PVT (3–68%).Two-PVT combinations produced high FNs and were less accurate than three-PVTs for detecting invalid cases, though variable accuracy was found within both types of combinations based on the specific PVTs in the combination. Thus, both PVT quantity and quality are important for accurate validity classification in research studies to ensure reliability and replicability of findings. Applying more liberal cutoffs yielded increased sensitivity, but with generally higher FPs yielding problematic specificity, particularly for three-PVT combinations.

Original languageEnglish (US)
Pages (from-to)44-56
Number of pages13
JournalPsychological Injury and Law
Volume13
Issue number1
DOIs
StatePublished - Mar 1 2020

Keywords

  • Assessment
  • Neuropsychology
  • Performance validity
  • Psychometrics
  • Veterans

ASJC Scopus subject areas

  • Psychiatry and Mental health
  • Law

Fingerprint Dive into the research topics of 'Strength in Numbers or Quality over Quantity? Examining the Importance of Criterion Measure Selection to Define Validity Groups in Performance Validity Test (PVT) Research'. Together they form a unique fingerprint.

Cite this