Other Computer Engineering Commons^™

Is It Worth It? Budget-Related Evaluation Metrics For Model Selection, Filip Klubicka, Giancarlo Salton, John D. Kelleher

Conference papers

Projects that set out to create a linguistic resource often do so by using a machine learning model that pre-annotates or filters the content that goes through to a human annotator, before going into the final version of the resource. However, available budgets are often limited, and the amount of data that is available exceeds the amount of annotation that can be done. Thus, in order to optimize the benefit from the invested human work, we argue that the decision on which predictive model one should employ depends not only on generalized evaluation metrics, such as accuracy and F-score, but …