TitleAn examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales
Publication TypeJournal Article
Year of Publication2001
AuthorsBorman, WC, Buck, DE, Hanson, MA, Motowidlo, SJ, Stark, S, Drasgow, F
JournalJournal of Applied Psychology
Publication Languageeng
Accession Number11596812
Keywords*Computer Simulation, *Employee Performance Appraisal, *Personnel Selection, Adult, Automatic Data Processing, Female, Human, Male, Reproducibility of Results, Sensitivity and Specificity, Support, U.S. Gov't, Non-P.H.S., Task Performance and Analysis, Video Recording

This laboratory research compared the reliability, validity, and accuracy of a computerized adaptive rating scale (CARS) format and 2 relatively common and representative rating formats. The CARS is a paired-comparison rating task that uses adaptive testing principles to present pairs of scaled behavioral statements to the rater to iteratively estimate a ratee's effectiveness on 3 dimensions of contextual performance. Videotaped vignettes of 6 office workers were prepared, depicting prescripted levels of contextual performance, and 112 subjects rated these vignettes using the CARS format and one or the other competing format. Results showed 23%-37% lower standard errors of measurement for the CARS format. In addition, validity was significantly higher for the CARS format (d = .18), and Cronbach's accuracy coefficients showed significantly higher accuracy, with a median effect size of .08. The discussion focuses on possible reasons for the results.