Engineering Commons^™

A Framework For Assessing And Designing Human Annotation Practices In Human-Ai Teaming, Suzanne Ashley Stevens

Theses and Dissertations

This thesis work examines how people accomplish annotation tasks (i.e., labelling data based on content) while working with an artificial intelligence (AI) system. When people and AI systems work together to accomplish a task, this is referred to as human-AI teaming. This study reports on the results of an interview and observation study of 15 volunteers from the Washington DC area as the volunteers annotated Twitter messages (tweets) about the COVID-19 pandemic. During the interviews, researchers observed the volunteers as they annotated tweets, noting any needs, frustrations, or confusion that the volunteers expressed about the task itself or when working …