Operations Research, Systems Engineering and Industrial Engineering Commons^™

An Attribute Agreement Method For Hfacs Inter-Rater Reliability Assessment, Teddy Steven Cotter, Veysel Yesilbas

Engineering Management & Systems Engineering Faculty Publications

Inter-rater reliability can be regarded as the degree of agreement among raters on a given item or a circumstance. Multiple approaches have been taken to estimate and improve inter-rater reliability of the United States Department of Defense Human Factors Analysis and Classification System used by trained accident investigators. In this study, three trained instructor pilots used the DoD-HFACS to classify 347 U.S. Air Force Accident Investigation Board (AIB) Class-A reports between the years of 2000 and 2013. The overall method consisted of four steps: (1) train on HFACS definitions, (2) verify rating reliability, (3) rate HFACS reports, and (4) random …