Science and Mathematics Education Commons^™

Random Number Simulations Reveal How Random Noise Affects The Measurements And Graphical Portrayals Of Self-Assessed Competency, Edward Nuhfer, Christopher Cogan, Steven Fleisher, Eric Gaze, Karl Wirth

Numeracy

Self-assessment measures of competency are blends of an authentic self-assessment signal that researchers seek to measure and random disorder or "noise" that accompanies that signal. In this study, we use random number simulations to explore how random noise affects critical aspects of self-assessment investigations: reliability, correlation, critical sample size, and the graphical representations of self-assessment data. We show that graphical conventions common in the self-assessment literature introduce artifacts that invite misinterpretation. Troublesome conventions include: (y minus x) vs. (x) scatterplots; (y minus x) vs. (x) column graphs aggregated as quantiles; line …