Graphics and Human Computer Interfaces Commons^™

Comai: Enabling Lightweight, Collaborative Intelligence By Retrofitting Vision Dnns, Kasthuri Jayarajah, Dhanuja Wanniarachchige, Tarek Abdelzaher, Archan Misra

Research Collection School Of Computing and Information Systems

While Deep Neural Network (DNN) models have transformed machine vision capabilities, their extremely high computational complexity and model sizes present a formidable deployment roadblock for AIoT applications. We show that the complexity-vs-accuracy-vs-communication tradeoffs for such DNN models can be significantly addressed via a novel, lightweight form of “collaborative machine intelligence” that requires only runtime changes to the inference process. In our proposed approach, called ComAI, the DNN pipelines of different vision sensors share intermediate processing state with one another, effectively providing hints about objects located within their mutually-overlapping Field-of-Views (FoVs). CoMAI uses two novel techniques: (a) a secondary shallow ML …

Gesturelens: Visual Analysis Of Gestures In Presentation Videos, Haipeng Zeng, Xingbo Wang, Yong Wang, Aoyu Wu, Ting Chuen Pong, Huamin Qu

Research Collection School Of Computing and Information Systems

Appropriate gestures can enhance message delivery and audience engagement in both daily communication and public presentations. In this paper, we contribute a visual analytic approach that assists professional public speaking coaches in improving their practice of gesture training through analyzing presentation videos. Manually checking and exploring gesture usage in the presentation videos is often tedious and time-consuming. There lacks an efficient method to help users conduct gesture exploration, which is challenging due to the intrinsically temporal evolution of gestures and their complex correlation to speech content. In this paper, we propose GestureLens, a visual analytics system to facilitate gesture-based and …

On Size-Oriented Long-Tailed Graph Classification Of Graph Neural Networks, Zemin Liu, Qiheng Mao, Chenghao Liu, Yuan Fang, Jianling Sun

Research Collection School Of Computing and Information Systems

The prevalence of graph structures attracts a surge of investigation on graph data, enabling several downstream tasks such as multigraph classification. However, in the multi-graph setting, graphs usually follow a long-tailed distribution in terms of their sizes, i.e., the number of nodes. In particular, a large fraction of tail graphs usually have small sizes. Though recent graph neural networks (GNNs) can learn powerful graph-level representations, they treat the graphs uniformly and marginalize the tail graphs which suffer from the lack of distinguishable structures, resulting in inferior performance on tail graphs. To alleviate this concern, in this paper we propose a …

Immersivepov: Filming How-To Videos With A Head-Mounted 360° Action Camera, Kevin Huang, Jiannan Li, Maurício Sousa, Tovi Grossman

Research Collection School Of Computing and Information Systems

How-to videos are often shot using camera angles that may not be optimal for learning motor tasks, with a prevalent use of third-person perspective. We present immersivePOV, an approach to film how-to videos from an immersive first-person perspective using a head-mounted 360° action camera. immersivePOV how-to videos can be viewed in a Virtual Reality headset, giving the viewer an eye-level viewpoint with three Degrees of Freedom. We evaluated our approach with two everyday motor tasks against a baseline first-person perspective and a third-person perspective. In a between-subjects study, participants were assigned to watch the task videos and then replicate the …

Rescuecastr: Exploring Photos And Live Streaming To Support Contextual Awareness In The Wilderness Search And Rescue Command Post, Brennon Jones, Anthony Tang, Carman Neustaedter

Research Collection School Of Computing and Information Systems

Wilderness search and rescue (WSAR) is a command-and-control activity where a Command team manages field teams scattered across a large area looking for a lost person. The challenge is that it can be difficult for Command to maintain awareness of field teams and the conditions of the field. We designed RescueCASTR, an interface that explores the idea of deploying field teams with wearable cameras that stream live video or sequential photos periodically to Command that aid contextual awareness. We ran a remote user study with WSAR managers to understand the opportunities and challenges of such a system. We found that …

Trend: Temporal Event And Node Dynamics For Graph Representation Learning, Zhihao Wen, Yuan Fang

Research Collection School Of Computing and Information Systems

Temporal graph representation learning has drawn significant attention for the prevalence of temporal graphs in the real world. However, most existing works resort to taking discrete snapshots of the temporal graph, or are not inductive to deal with new nodes, or do not model the exciting effects which is the ability of events to influence the occurrence of another event. In this work, We propose TREND, a novel framework for temporal graph representation learning, driven by TempoRal Event and Node Dynamics and built upon a Hawkes process-based graph neural network (GNN). TREND presents a few major advantages: (1) it is …

Mask-Guided Deformation Adaptive Network For Human Parsing, Aihua Mao, Yuan Liang, Jianbo Jiao, Yongtuo Liu, Shengfeng He

Research Collection School Of Computing and Information Systems

Due to the challenges of densely compacted body parts, nonrigid clothing items, and severe overlap in crowd scenes, human parsing needs to focus more on multilevel feature representations compared to general scene parsing tasks. Based on this observation, we propose to introduce the auxiliary task of human mask and edge detection to facilitate human parsing. Different from human parsing, which exploits the discriminative features of each category, human mask and edge detection emphasizes the boundaries of semantic parsing regions and the difference between foreground humans and background clutter, which benefits the parsing predictions of crowd scenes and small human parts. …

Mrim: Enabling Mixed-Resolution Imaging For Low-Power Pervasive Vision Tasks, Jiyan Wu, Vithurson Subasharan, Tuan Tran, Archan Misra

Research Collection School Of Computing and Information Systems

While many pervasive computing applications increasingly utilize real-time context extracted from a vision sensing infrastructure, the high energy overhead of DNN-based vision sensing pipelines remains a challenge for sustainable in-the-wild deployment. One common approach to reducing such energy overheads is the capture and transmission of lower-resolution images to an edge node (where the DNN inferencing task is executed), but this results in an accuracy-vs-energy tradeoff, as the DNN inference accuracy typically degrades with a drop in resolution. In this work, we introduce MRIM, a simple but effective framework to tackle this tradeoff. Under MRIM, the vision sensor platform first executes …

Debiasing Nlu Models Via Causal Intervention And Counterfactual Reasoning, Bing Tian, Yixin Cao, Yong Zhang, Chunxiao Xing

Research Collection School Of Computing and Information Systems

Recent studies have shown that strong Natural Language Understanding (NLU) models are prone to relying on annotation biases of the datasets as a shortcut, which goes against the underlying mechanisms of the task of interest. To reduce such biases, several recent works introduce debiasing methods to regularize the training process of targeted NLU models. In this paper, we provide a new perspective with causal inference to fnd out the bias. On the one hand, we show that there is an unobserved confounder for the natural language utterances and their respective classes, leading to spurious correlations from training data. To remove …

State Graph Reasoning For Multimodal Conversational Recommendation, Yuxia Wu, Lizi Liao, Gangyi Zhang, Wenqiang Lei, Guoshuai Zhao, Xueming Qian, Tat-Seng Chua

Research Collection School Of Computing and Information Systems

Conversational recommendation system (CRS) attracts increasing attention in various application domains such as retail and travel. It offers an effective way to capture users’ dynamic preferences with multi-turn conversations. However, most current studies center on the recommendation aspect while over-simplifying the conversation process. The negligence of complexity in data structure and conversation flow hinders their practicality and utility. In reality, there exist various relationships among slots and values, while users’ requirements may dynamically adjust or change. Moreover, the conversation often involves visual modality to facilitate the conversation. These actually call for a more advanced internal state representation of the dialogue …

The Impact Of Visual Feedback And Control Configuration On Pilot-Aircraft Interface Using Head Tracking Technology, Christopher M. Arnold

Theses and Dissertations

Traditional control mechanisms restrict human input on the displays in 5th generation aircraft. This research explored methods for enhancing pilot interaction with large, information dense cockpit displays; specifically, the effects of visual feedback and control button configuration when augmenting cursor control with head tracking technology. Previous studies demonstrated that head tracking can be combined with traditional cursor control to decrease selection times but can increase pilot mental and physical workload. A human subject experiment was performed to evaluate two control button configurations and three visual feedback conditions. A Fitts Law analysis was performed to create predictive models of selection time …

A Thematic And Reference Analysis Of Touchless Technologies, Eric R. Curia

Theses and Dissertations

The purpose of this research is to explore the utility and current state of touchless technologies. Five categories of technologies are identified as a result of collecting and reviewing literature: facial/biometric recognition, gesture recognition, touchless sensing, personal devices, and voice recognition. A thematic analysis was conducted to evaluate the advantages and disadvantages of the five categories. A reference analysis was also conducted to determine the similarities between articles in each category. Touchless sensing showed to have the most advantages and least similar references. Gesture recognition was the opposite. Comparing analyses shows more reliable technology types are more beneficial and diverse.

Applying Models Of Circadian Stimulus To Explore Ideal Lighting Configurations, Alexander J. Price

Theses and Dissertations

Increased levels of time are spent indoors, decreasing human interaction with nature and degrading photoentrainment, the synchronization of circadian rhythms with daylight variation. Military imagery analysts, among other professionals, are required to work in low light level environments to limit power consumption or increase contrast on display screens to improve detail detection. Insufficient exposure to light in these environments results in inadequate photoentrainment which is associated with degraded alertness and negative health effects. Recent research has shown that both the illuminance (i.e., perceived intensity) and wavelength of light affect photoentrainment. Simultaneously, modern lighting technologies have improved our ability to construct …

The Application Of Virtual Reality In Firefighting Training, Dylan A. Gagnon

Theses and Dissertations

Immersive simulations such as virtual reality is becoming more prevalent for use in training environments for many professions. United States Air Force firefighters may benefit from incorporating VR technology into their training program to increase organizational commitment, job satisfaction, self-efficacy, and job performance. With implementing a new training platform, it is also important to understand the relationship between these variables and the perceived benefits and efficacy of the VR training, which has not yet been studied in previous research. This study addresses this issue by gathering data from fire departments currently fielding a VR fire training platform.

Real Time Evaluation Of Boom And Drogue Occlusion With Aar, Xiaoyang Wu

Theses and Dissertations

In recent years, Unmanned Aerial Vehicles (UAV) have seen a rise in popularity. Various navigational algorithms have been developed as a solution to estimate a UAV’s pose relative to the refueler aircraft. The result can be used to safely automate aerial refueling (AAR) to improve UAVs’ time-on-station and ensure the success of military operations. This research aims to reach real-time performance using a GPU accelerated approach. It also conducts various experiments to quantify the effects of refueling boom/drogue occlusion and image exposure on the pose estimation pipeline in a lab setting.

Deconfounded Visual Grounding, Jianqiang Huang, Yu Qin, Jiaxin Qi, Qianru Sun, Hanwang Zhang

Research Collection School Of Computing and Information Systems

We focus on the confounding bias between language and location in the visual grounding pipeline, where we find that the bias is the major visual reasoning bottleneck. For example, the grounding process is usually a trivial languagelocation association without visual reasoning, e.g., grounding any language query containing sheep to the nearly central regions, due to that most queries about sheep have groundtruth locations at the image center. First, we frame the visual grounding pipeline into a causal graph, which shows the causalities among image, query, target location and underlying confounder. Through the causal graph, we know how to break the …

Designing Respectful Tech: What Is Your Relationship With Technology?, Noreen Y. Whysel

Publications and Research

According to research at the Me2B Alliance, people feel they have a relationship with technology. It’s emotional. It’s embodied. And it’s very personal. We are studying digital relationships to answer questions like “Do people have a relationship with technology?” “What does that relationship feel like?” And “Do people understand the commitments that they are making when they explore, enter into and dissolve these relationships?” There are parallels between messy human relationships and the kinds of relationships that people develop with technology. As with human relationships, we move through states of discovery, commitment and breakup with digital applications as well. Technology …

Modeling Functional Similarity In Source Code With Graph-Based Siamese Networks, Nikita Mehrotra, Navdha Agarwal, Piyush Gupta, Saket Anand, David Lo, Rahul Purandare

Research Collection School Of Computing and Information Systems

Code clones are duplicate code fragments that share (nearly) similar syntax or semantics. Code clone detection plays an important role in software maintenance, code refactoring, and reuse. A substantial amount of research has been conducted in the past to detect clones. A majority of these approaches use lexical and syntactic information to detect clones. However, only a few of them target semantic clones. Recently, motivated by the success of deep learning models in other fields, including natural language processing and computer vision, researchers have attempted to adopt deep learning techniques to detect code clones. These approaches use lexical information (tokens) …

Usability Of Health-Related Websites By Filipino-American Adults And Nursing Informatics Experts, Kathleen Begonia

Dissertations, Theses, and Capstone Projects

Filipino-Americans are an understudied minority group with high prevalence and mortality from chronic conditions, such as cardiovascular disease and diabetes. Facing barriers to care and lack of culturally appropriate health resources, they frequently use the internet to obtain health information. It is unknown whether they perceive health-related websites to be useful or easy to use because there are no published usability studies involving this population. Using the Technology Acceptance Model as a theoretical framework, this study investigated the difference between website design ratings by experts and the perceptions of Filipino-American users to determine if usability guidelines influenced the perceived ease …

Modeling Sentiments And Preferences From Multimodal Data, Quoc Tuan Truong

Dissertations and Theses Collection (Open Access)

Online reviews are prevalent in many modern Web applications, such as e-commerce, crowd-sourced location and check-in platforms. Fueled by the rise of mobile phones that are often the only cameras on hand, reviews are increasingly multimodal, with photos in addition to textual content. In this thesis, we focus on modeling the subjectivity carried in this form of data, with two research objectives.

In the first part, we tackle the problem of detecting sentiment expressed by a review. This is a key unlocking many applications, e.g., analyzing opinions, monitoring consumer satisfaction, assessing product quality.
Traditionally, the task of sentiment analysis primarily …