Evaluating The Knowledge Of Conversational Agents, Mina Park, Milam Aiken, Mahesh Vanjani

Southwestern Business Administration Journal

Several studies have tested chatbots for their abilities to emulate human conversation, but few have evaluated the systems’ general knowledge. In this study, we asked two chatbots (Mitsuku and Tutor) and a digital assistant (Cortana) several questions and compared their answers to 67 humans’ answers. Results showed that while Tutor and Cortana performed poorly, the accuracies of Mitsuku and the humans were not significantly different. As expected, the chatbots and Cortana answered factual questions more accurately than abstract questions.