Engineering Commons^™

Harnessing Confidence For Report Aggregation In Crowdsourcing Environments, Hadeel Alhosaini, Xianzhi Wang, Lina Yao, Zhong Yang, Farookh Hussain, Ee-Peng Lim

Research Collection School Of Computing and Information Systems

Crowdsourcing is an effective means of accomplishing human intelligence tasks by leveraging the collective wisdom of crowds. Given reports of various accuracy degrees from workers, it is important to make wise use of these reports to derive accurate task results. Intuitively, a task result derived from a sufficient number of reports bears lower uncertainty, and higher uncertainty otherwise. Existing report aggregation research, however, has largely neglected the above uncertainty issue. In this regard, we propose a novel report aggregation framework that defines and incorporates a new confidence measure to quantify the uncertainty associated with tasks and workers, thereby enhancing result …

Changing The Focus: Worker-Centric Optimization In Human-In-The-Loop Computations, Mohammadreza Esfandiari

Dissertations

A myriad of emerging applications from simple to complex ones involve human cognizance in the computation loop. Using the wisdom of human workers, researchers have solved a variety of problems, termed as “micro-tasks” such as, captcha recognition, sentiment analysis, image categorization, query processing, as well as “complex tasks” that are often collaborative, such as, classifying craters on planetary surfaces, discovering new galaxies (Galaxyzoo), performing text translation. The current view of “humans-in-the-loop” tends to see humans as machines, robots, or low-level agents used or exploited in the service of broader computation goals. This dissertation is developed to shift the focus back …

Smart Communities: From Sensors To Internet Of Things And To A Marketplace Of Services, Stephan Olariu, Nirwan Ansari (Editor), Andreas Ahrens (Editor), Cesar Benavente-Preces (Editor)

Computer Science Faculty Publications

Our paper was inspired by the recent Society 5.0 initiative of the Japanese Government that seeks to create a sustainable human-centric society by putting to work recent advances in technology: sensor networks, edge computing, IoT ecosystems, AI, Big Data, robotics, to name just a few. The main contribution of this work is a vision of how these technological advances can contribute, directly or indirectly, to making Society 5.0 reality. For this purpose we build on a recently-proposed concept of Marketplace of Services that, in our view, will turn out to be one of the cornerstones of Society 5.0. Instead of …

Worker Demographics And Earnings On Amazon Mechanical Turk: An Exploratory Analysis, Kotaro Hara, Kristy Milland, Benjamin V. Hanrahan, Chris Callison-Burch, Abigail Adams, Saiph Savage, Jeffrey P. Bigham

Research Collection School Of Computing and Information Systems

Prior research reported that workers on Amazon Mechanical Turk (AMT) are underpaid, earning about $2/h. But the prior research did not investigate the difference in wage due to worker characteristics (e.g., country of residence). We present the first data-driven analysis on wage gap on AMT. Using work log data and demographic data collected via online survey, we analyse the gap in wage due to different factors. We show that there is indeed wage gap; for example, workers in the U.S. earn $3.01/h while those in India earn $1.41/h on average.

Deep Neural Ranking For Crowdsourced Geopolitical Event Forecasting, Giuseppe Nebbione, Derek Doran, Srikanth Nadella, Brandon Minnery

Computer Science and Engineering Faculty Publications

There are many examples of “wisdom of the crowd” effects in which the large number of participants imparts confidence in the collective judgment of the crowd. But how do we form an aggregated judgment when the size of the crowd is limited? Whose judgments do we include, and whose do we accord the most weight? This paper considers this problem in the context of geopolitical event forecasting, where volunteer analysts are queried to give their expertise, confidence, and predictions about the outcome of an event. We develop a forecast aggregation model that integrates topical information about a question, meta-data about …

Effect Of Label Noise On The Machine-Learned Classification Of Earthquake Damage, Jared Frank, Umaa Rebbapragada, James Bialas, Thomas Oommen, Timothy C. Havens

Michigan Tech Publications

Automated classification of earthquake damage in remotely-sensed imagery using machine learning techniques depends on training data, or data examples that are labeled correctly by a human expert as containing damage or not. Mislabeled training data are a major source of classifier error due to the use of imprecise digital labeling tools and crowdsourced volunteers who are not adequately trained on or invested in the task. The spatial nature of remote sensing classification leads to the consistent mislabeling of classes that occur in close proximity to rubble, which is a major byproduct of earthquake damage in urban areas. In this study, …

Smartphone Sensing Meets Transport Data: A Collaborative Framework For Transportation Service Analytics, Yu Lu, Archan Misra, Wen Sun, Huayu Wu

Research Collection School Of Computing and Information Systems

We advocate for and introduce TRANSense, a framework for urban transportation service analytics that combines participatory smartphone sensing data with city-scale transportation-related transactional data (taxis, trains etc.). Our work is driven by the observed limitations of using each data type in isolation: (a) commonly-used anonymous city-scale datasets (such as taxi bookings and GPS trajectories) provide insights into the aggregate behavior of transport infrastructure, but fail to reveal individual-specific transport experiences (e.g., wait times in taxi queues); while (b) mobile sensing data can capture individual-specific commuting-related activities, but suffers from accuracy and energy overhead challenges due to usage artefacts and lack …

Video Annotation By Crowd Workers With Privacy-Preserving Local Disclosure, Apeksha Dipak Kumavat

Open Access Theses

Advancements in computer vision are still not reliable enough for detecting video content including humans and their actions. Microtask crowdsourcing on task markets such as Amazon Mechnical Turk and Upwork can bring humans into the loop. However, engaging crowd workers to annotate non-public video footage risks revealing the identities of people in the video who may have a right to anonymity.

This thesis demonstrates how we can engage untrusted crowd workers to detect behaviors and objects, while robustly concealing the identities of all faces. We developed a web-based system that presents obfuscated videos to crowd workers, and provides them with …

Multi-Agent Task Assignment For Mobile Crowdsourcing Under Trajectory Uncertainties, Cen Chen, Shih-Fen Cheng, Hoong Chuin Lau, Archan Misra

Research Collection School Of Computing and Information Systems

In this work, we investigate the problem of mobile crowdsourcing, where workers are financially motivated to perform location-based tasks physically. Unlike current industry practice that relies on workers to manually browse and filter tasks to perform, we intend to automatically make task recommendations based on workers' historical trajectories and desired time budgets. However, predicting workers' trajectories is inevitably faced with uncertainties, as no one will take exactly the same route every day; yet such uncertainties are oftentimes abstracted away in the known literature. In this work, we depart from the deterministic modeling and study the stochastic task recommendation problem where …

Modeling User Transportation Patterns Using Mobile Devices, Erfan Davami

Electronic Theses and Dissertations

Participatory sensing frameworks use humans and their computing devices as a large mobile sensing network. Dramatic accessibility and affordability have turned mobile devices (smartphone and tablet computers) into the most popular computational machines in the world, exceeding laptops. By the end of 2013, more than 1.5 billion people on earth will have a smartphone. Increased coverage and higher speeds of cellular networks have given these devices the power to constantly stream large amounts of data. Most mobile devices are equipped with advanced sensors such as GPS, cameras, and microphones. This expansion of smartphone numbers and power has created a sensing …

Traccs: Trajectory-Aware Coordinated Urban Crowd-Sourcing, Cen Chen, Shih-Fen Cheng, Aldy Gunawan, Archan Misra, Koustuv Dasgupta, Deepthi Chander

Research Collection School Of Computing and Information Systems

We investigate the problem of large-scale mobile crowd-tasking, where a large pool of citizen crowd-workers are used to perform a variety of location-specific urban logistics tasks. Current approaches to such mobile crowd-tasking are very decentralized: a crowd-tasking platform usually provides each worker a set of available tasks close to the worker's current location; each worker then independently chooses which tasks she wants to accept and perform. In contrast, we propose TRACCS, a more coordinated task assignment approach, where the crowd-tasking platform assigns a sequence of tasks to each worker, taking into account their expected location trajectory over a wider time …

Active Learning With Unreliable Annotations, Liyue Zhao

Electronic Theses and Dissertations

With the proliferation of social media, gathering data has became cheaper and easier than before. However, this data can not be used for supervised machine learning without labels. Asking experts to annotate sufficient data for training is both expensive and time-consuming. Current techniques provide two solutions to reducing the cost and providing sufficient labels: crowdsourcing and active learning. Crowdsourcing, which outsources tasks to a distributed group of people, can be used to provide a large quantity of labels but controlling the quality of labels is hard. Active learning, which requires experts to annotate a subset of the most informative or …

Interactionless Calendar-Based Training For 802.11 Localization, Mark Chang, Andrew J. Barry, Noah L. Tye

Mark L. Chang

This paper presents our work in solving one of the weakest links in 802.11-based indoor-localization: the training of ground-truth received signal strength data. While crowdsourcing this information has been demonstrated to be a viable alternative to the time consuming and accuracy-limited process of manual training, one of the chief drawbacks is the rate at which a system can be trained. We demonstrate an approach that utilizes users' calendar and appointment information to perform interactionless training of an 802.11-based indoor localization system. Our system automatically determines if a user attended a calendar event, resulting in accuracy comparable to our previously published …