Open Access. Powered by Scholars. Published by Universities.®
Articles 1 - 1 of 1
Full-Text Articles in Education
Inter-Rater Variability As Mutual Disagreement: Identifying Raters’ Divergent Points Of View, A. Gingerich, Susan E. Ramlo
Inter-Rater Variability As Mutual Disagreement: Identifying Raters’ Divergent Points Of View, A. Gingerich, Susan E. Ramlo
Susan E Ramlo
Whenever multiple observers provide ratings, even of the same performance,
inter-rater variation is prevalent. The resulting ‘idiosyncratic rater variance’ is considered
to be unusable error of measurement in psychometric models and is a threat to the
defensibility of our assessments. Prior studies of inter-rater variation in clinical assessments
have used open response formats to gather raters’ comments and justifications. This
design choice allows participants to use idiosyncratic response styles that could result in a
distorted representation of the underlying rater cognition and skew subsequent analyses. In
this study we explored rater variability using the structured response format of Q
methodology. …