Digital Commons Network^™

Promises And Pitfalls Of Machine Learning Classifiers For Inter-Rater Reliability Annotation, Lucille Dorothy Ayres

Browse all Theses and Dissertations

Qualitative data result from observation, video, and dialogue. These types of data are flexible and allow us to study behavior without imposing potentially disruptive data collection methods. However, subsequent quantitative analysis requires a time consuming, labor intensive initial coding process, and a second manual coding to calculate inter-rater reliability. I examined the use of machine learning algorithms to reduce the amount of manual annotation work required to perform inter-rater reliability measures on text data. By comparing machine-human and human-human raters using Cohen’s Kappa statistic and an informal analysis of the features used in machine learning classification, I identify the promise …