Physical Sciences and Mathematics Commons^™

Estimating Error And Bias In Offline Evaluation Results, Mucun Tian, Michael D. Ekstrand

Computer Science Faculty Publications and Presentations

Offline evaluations of recommender systems attempt to estimate users’ satisfaction with recommendations using static data from prior user interactions. These evaluations provide researchers and developers with first approximations of the likely performance of a new system and help weed out bad ideas before presenting them to users. However, offline evaluation cannot accurately assess novel, relevant recommendations, because the most novel items were previously unknown to the user, so they are missing from the historical data and cannot be judged as relevant.

We present a simulation study to estimate the error that such missing data causes in commonly-used evaluation metrics in …

Signal Passing Self-Assembly Simulates Tile Automata, Angel A. Cantu, Austin Luchsinger, Robert Schweller, Tim Wylie

Computer Science Faculty Publications and Presentations

The natural process of self-assembly has been studied through various abstract models due to the abundant applications that benefit from self-assembly. Many of these different models emerged in an effort to capture and understand the fundamental properties of different physical systems and the mechanisms by which assembly may occur. A newly proposed model, known as Tile Automata, offers an abstract toolkit to analyze and compare the algorithmic properties of different self-assembly systems. In this paper, we show that for every Tile Automata system, there exists a Signal-passing Tile Assembly system that can simulate it. Finally, we connect our result with …

Monte Carlo Estimates Of Evaluation Metric Error And Bias: Work In Progress, Mucun Tian, Michael D. Ekstrand

Computer Science Faculty Publications and Presentations

Traditional offline evaluations of recommender systems apply metrics from machine learning and information retrieval in settings where their underlying assumptions no longer hold. This results in significant error and bias in measures of top-N recommendation performance, such as precision, recall, and nDCG. Several of the specific causes of these errors, including popularity bias and misclassified decoy items, are well-explored in the existing literature. In this paper we survey a range of work on identifying and addressing these problems, and report on our work in progress to simulate the recommender data generation and evaluation processes to quantify the extent of …