Language and Literacy Education Commons^™

Lessons Learned About Evaluating Fairness From A Data Challenge To Automatically Score Naep Reading Items, Maggie Beiting-Parrish, John Whitmer

Chinese/English Journal of Educational Measurement and Evaluation | 教育测量与评估双语期刊

Natural language processing (NLP) is widely used to predict human scores for open-ended student assessment responses in various content areas (Johnson et al., 2022). Ensuring algorithmic fairness based on student demographic background factors is crucial (Madnani et al., 2017). This study presents a fairness analysis of six top-performing entries from a data challenge involving 20 NAEP reading comprehension items that were initially analyzed for fairness based on race/ethnicity and gender. This study describes additional fairness evaluation including English Language Learner Status (ELLs), Individual Education Plans, and Free/Reduced-Price Lunch. Several items showed lower accuracy for predicted scores, particularly for ELLs. This …