Medicine and Health Sciences Commons^™

Grant Peer Review: Improving Inter-Rater Reliability With Training, David N. Sattler, Patrick E. Mcknight, Linda Naney, Randy Mathis

Psychology Faculty and Staff Publications

This study developed and evaluated a brief training program for grant reviewers that aimed to increase inter-rater reliability, rating scale knowledge, and effort to read the grant review criteria. Enhancing reviewer training may improve the reliability and accuracy of research grant proposal scoring and funding recommendations. Seventy-five Public Health professors from U.S. research universities watched the training video we produced and assigned scores to the National Institutes of Health scoring criteria proposal summary descriptions. For both novice and experienced reviewers, the training video increased scoring accuracy (the percentage of scores that reflect the true rating scale values), inter-rater reliability, and …