Graphics and Human Computer Interfaces Commons^™

Holoboard: A Large-Format Immersive Teaching Board Based On Pseudo Holographics, Jiangtao Gong, Teng Han, Siling Guo, Jiannan Li, Siyu Zha, Liuxin Zhang, Feng Tian, Qianying Wang, Yong Rui

Research Collection School Of Computing and Information Systems

In this paper, we present HoloBoard, an interactive large-format pseduo-holographic display system for lecture based classes. With its unique properties of immersive visual display and transparent screen, we designed and implemented a rich set of novel interaction techniques like immersive presentation, role-play, and lecturing behind the scene that are potentially valuable for lecturing in class. We conducted a controlled experimental study to compare a HoloBoard class with a normal class through measuring students’ learning outcomes and three dimensions of engagement (i.e., behavioral, emotional, and cognitive engagement). We used pre-/post- knowledge tests and multimodal learning analytics to measure students’ learning outcomes …

Route Tapestries: Navigating 360° Virtual Tour Videos Using Slit-Scan Visualizations, Jiannan Li, Jiahe Lyu, Maurício Sousa, Ravin Balakrishnan, Anthony Tang, Tovi Grossman

Research Collection School Of Computing and Information Systems

An increasingly popular way of experiencing remote places is by viewing 360° virtual tour videos, which show the surrounding view while traveling through an environment. However, finding particular locations in these videos can be difficult because current interfaces rely on distorted frame previews for navigation. To alleviate this usability issue, we propose Route Tapestries, continuous orthographic-perspective projection of scenes along camera routes. We first introduce an algorithm for automatically constructing Route Tapestries from a 360° video, inspired by the slit-scan photography technique. We then present a desktop video player interface using a Route Tapestry timeline for navigation. An online evaluation …

Uncovering Patterns In Reviewers' Feedback To Scene Description Authors, Rosiana Natalie, Jolene Kar Inn Loh, Huei Suen Tan, Joshua Shi-Hao Tseng, Hernisa Kacorri, Kotaro Hara

Research Collection School Of Computing and Information Systems

Audio descriptions (ADs) can increase access to videos for blind people. Researchers have explored different mechanisms for generating ADs, with some of the most recent studies involving paid novices; to improve the quality of their ADs, novices receive feedback from reviewers. However, reviewer feedback is not instantaneous. To explore the potential for real-time feedback through automation, in this paper, we analyze 1,120 comments that 40 sighted novices received from a sighted or a blind reviewer. We find that feedback patterns tend to fall under four themes: (i) Quality; commenting on different AD quality variables, (ii) Speech Act; the utterance or …

Causal Attention For Unbiased Visual Recognition, Tan Wang, Chang Zhou, Qianru Sun, Hanwang Zhang

Research Collection School Of Computing and Information Systems

Attention module does not always help deep models learn causal features that are robust in any confounding context, e.g., a foreground object feature is invariant to different backgrounds. This is because the confounders trick the attention to capture spurious correlations that benefit the prediction when the training and testing data are IID (identical & independent distribution); while harm the prediction when the data are OOD (out-of-distribution). The sole fundamental solution to learn causal attention is by causal intervention, which requires additional annotations of the confounders, e.g., a “dog” model is learned within “grass+dog” and “road+dog” respectively, so the “grass” and …

Transporting Causal Mechanisms For Unsupervised Domain Adaptation, Zhongqi Yue, Qianru Sun, Xian-Sheng Hua, Hanwang Zhang

Research Collection School Of Computing and Information Systems

Existing Unsupervised Domain Adaptation (UDA) literature adopts the covariate shift and conditional shift assumptions, which essentially encourage models to learn common features across domains. However, due to the lack of supervision in the target domain, they suffer from the semantic loss: the feature will inevitably lose nondiscriminative semantics in source domain, which is however discriminative in target domain. We use a causal view—transportability theory [41]—to identify that such loss is in fact a confounding effect, which can only be removed by causal intervention. However, the theoretical solution provided by transportability is far from practical for UDA, because it requires the …

Self-Regulation For Semantic Segmentation, Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, Qianru Sun

Research Collection School Of Computing and Information Systems

In this paper, we seek reasons for the two major failure cases in Semantic Segmentation (SS): 1) missing small objects or minor object parts, and 2) mislabeling minor parts of large objects as wrong classes. We have an interesting finding that Failure-1 is due to the underuse of detailed features and Failure-2 is due to the underuse of visual contexts. To help the model learn a better trade-off, we introduce several Self-Regulation (SR) losses for training SS neural networks. By “self”, we mean that the losses are from the model per se without using any additional data or supervision. By …

A Large-Scale Benchmark For Food Image Segmentation, Xiongwei Wu, Xin Fu, Ying Liu, Ee-Peng Lim, Steven C. H. Hoi, Qianru Sun

Research Collection School Of Computing and Information Systems

Food image segmentation is a critical and indispensible task for developing health-related applications such as estimating food calories and nutrients. Existing food image segmentation models are underperforming due to two reasons: (1) there is a lack of high quality food image datasets with fine-grained ingredient labels and pixel-wise location masks—the existing datasets either carry coarse ingredient labels or are small in size; and (2) the complex appearance of food makes it difficult to localize and recognize ingredients in food images, e.g., the ingredients may overlap one another in the same image, and the identical ingredient may appear distinctly in different …

Learning To Adversarially Blur Visual Object Tracking, Qing Guo, Ziyi Cheng, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Yang Liu, Jianjun Zhao

Research Collection School Of Computing and Information Systems

Motion blur caused by the moving of the object or camera during the exposure can be a key challenge for visual object tracking, affecting tracking accuracy significantly. In this work, we explore the robustness of visual object trackers against motion blur from a new angle, i.e., adversarial blur attack (ABA). Our main objective is to online transfer input frames to their natural motion-blurred counterparts while misleading the state-of-the-art trackers during the tracking process. To this end, we first design the motion blur synthesizing method for visual tracking based on the generation principle of motion blur, considering the motion information and …

Aixfood'21: 3rd Workshop On Aixfood, Ricardo Guerrero, Michael Spranger, Shuqiang Jiang, Chong-Wah Ngo

Research Collection School Of Computing and Information Systems

Food and cooking analysis present exciting research and application challenges for modern AI systems, particularly in the context of multimodal data such as images or video. A meal that appears in a food image is a product of a complex progression of cooking stages, often described in the accompanying textual recipe form. In the cooking process, individual ingredients change their physical properties, become combined with other food components, all to produce a final, yet highly variable, appearance of the meal. Recognizing food items or meals on a plate from images or videos, their physical properties such as the amount, nutritional …

Prediction Of Synthetic Lethal Interactions In Human Cancers Using Multi-View Graph Auto-Encoder, Zhifeng Hao, Di Wu, Yuan Fang, Min Wu, Ruichu Cai, Xiaoli Li

Research Collection School Of Computing and Information Systems

Synthetic lethality (SL) is a very important concept for the development of targeted anticancer drugs. However, experimental methods for SL detection often suffer from various issues like high cost and low consistency across cell lines. Hence, computational methods for predicting novel SLs have recently emerged as complements for wet-lab experiments. In addition, SL data can be represented as a graph where nodes are genes and edges are the SL interactions. It is thus motivated to design advanced graph-based machine learning algorithms for SL prediction. In this paper, we propose a novel SL prediction method using Multi-view Graph Auto-Encoder (SLMGAE). We …

Visilence: An Interactive Visualization Tool For Error Resilience Analysis, Shaolun Ruan, Yong Wang, Qiang Guan

Research Collection School Of Computing and Information Systems

Soft errors have become one of the major concerns for HPC applications, as those errors can result in seriously corrupted outcomes, such as silent data corruptions (SDCs). Prior studies on error resilience have studied the robustness of HPC applications. However, it is still difficult for program developers to identify potential vulnerability to soft errors. In this paper, we present Visilence, a novel visualization tool to visually analyze error vulnerability based on the control-flow graph generated from HPC applications. Visilence efficiently visualizes the affected program states under injected errors and presents the visual analysis of the most vulnerable parts of an …

Semi-Supervised Semantic Visualization For Networked Documents, Delvin Ce Zhang, Hady W. Lauw

Research Collection School Of Computing and Information Systems

Semantic interpretability and visual expressivity are important objectives in exploratory analysis of text. On the one hand, while some documents may have explicit categories, we could develop a better understanding of a corpus by studying its finer-grained structures, which may be latent. By inferring latent topics and discovering keywords associated with each topic, one obtains a semantic interpretation of the corpus. One the other hand, by visualizing documents, latent topics, and category labels on the same plot, one gains a bird’s eye view of the relationships among documents, topics, and various categories. Semantic visualization is a class of methods that …

Unit 1 - Virtual Machine Creation And Configuration, George A. Nossa

Open Educational Resources

The COVID pandemic has challenged us in continuing our instructional mission and provide high quality learning activities remotely. A key requirement for advanced technology courses is to duplicate the BMCC Computer Lab environment on the student's home computers.

This document provides detailed instructions for creating a "virtual machine" on the student's home computer. This document assumes the home computer is a PC desktop or laptop but this document will also work on computers running MacOS with an Intel X86 compatible chip,

Once the virtual machine is created, this document details the steps to install and configure an Ubuntu 20.04 Operating …

An Automated Method To Enrich Consumer Health Vocabularies Using Glove Word Embeddings And An Auxiliary Lexical Resource, Mohammed Ibrahim, Susan Gauch, Omar Salman, Mohammed Alqahtani

Computer Science and Computer Engineering Faculty Publications and Presentations

Background

Clear language makes communication easier between any two parties. A layman may have difficulty communicating with a professional due to not understanding the specialized terms common to the domain. In healthcare, it is rare to find a layman knowledgeable in medical terminology which can lead to poor understanding of their condition and/or treatment. To bridge this gap, several professional vocabularies and ontologies have been created to map laymen medical terms to professional medical terms and vice versa.

Objective

Many of the presented vocabularies are built manually or semi-automatically requiring large investments of time and human effort and consequently the …

Transitioning From Vue 2 To Vue 3, Adele Kanley

Theses/Capstones/Creative Projects

Frontend development is a field that is constantly changing because of the vast amounts of tools that are made available each year. One of the most popular frameworks being utilized to create fluid user experience is the Vue framework. Branching from the well-known Angular.js, Vue.js is an independent open-source project that is making its mark in the user interface community.

Regardless of the popularity of a framework, updates are inevitable to keep up with the innovations required by the IT Field. To ensure that UNO IS&T students are being offered opportunities to learn and develop in the most update to …

Verification Of Piecewise Deep Neural Networks: A Star Set Approach With Zonotope Pre-Filter, Hoang-Dung Tran, Neelanjana Pal, Diego Manzanas Lopez, Patrick Musau, Xiaodong Yang, Luan Viet Nguyen, Weiming Xiang, Stanley Bak, Taylor T. Johnson

Computer Science Faculty Publications

Verification has emerged as a means to provide formal guarantees on learning-based systems incorporating neural network before using them in safety-critical applications. This paper proposes a new verification approach for deep neural networks (DNNs) with piecewise linear activation functions using reachability analysis. The core of our approach is a collection of reachability algorithms using star sets (or shortly, stars), an effective symbolic representation of high-dimensional polytopes. The star-based reachability algorithms compute the output reachable sets of a network with a given input set before using them for verification. For a neural network with piecewise linear activation functions, our approach can …

How Knowledge Graph And Attention Help? A Qualitative Analysis Into Bag-Level Relation Extraction, Zikun Hu, Yixin Cao, Lifu Huang, Tat-Seng Chua

Research Collection School Of Computing and Information Systems

Knowledge Graph (KG) and attention mechanism have been demonstrated effective in introducing and selecting useful information for weakly supervised methods. However, only qualitative analysis and ablation study are provided as evidence. In this paper, we contribute a dataset and propose a paradigm to quantitatively evaluate the effect of attention and KG on bag-level relation extraction (RE). We find that (1) higher attention accuracy may lead to worse performance as it may harm the model’s ability to extract entity mention features; (2) the performance of attention is largely influenced by various noise distribution patterns, which is closely related to real-world datasets; …

Fine-Grained Analysis Of Structured Output Prediction, Waleed Mustafa, Yunwen Lei, Antoine Ledent, Marius And Kloft

Research Collection School Of Computing and Information Systems

In machine learning we often encounter structured output prediction problems (SOPPs), i.e. problems where the output space admits a rich internal structure. Application domains where SOPPs naturally occur include natural language processing, speech recognition, and computer vision. Typical SOPPs have an extremely large label set, which grows exponentially as a function of the size of the output. Existing generalization analysis implies generalization bounds with at least a square-root dependency on the cardinality d of the label set, which can be vacuous in practice. In this paper, we significantly improve the state of the art by developing novel high-probability bounds with …

Bidding Mechanisms In Graph Games, Guy Avni, Thomas A. Henzinger, Dorde Zikelic

Research Collection School Of Computing and Information Systems

A graph game proceeds as follows: two players move a token through a graph to produce a finite or infinite path, which determines the payoff of the game. We study bidding games in which in each turn, an auction determines which player moves the token. Bidding games were largely studied in combination with two variants of first-price auctions called “Richman” and “poorman” bidding. We study taxman bidding, which span the spectrum between the two. The game is parameterized by a constant τ∈[0,1]: portion τ of the winning bid is paid to the other player, and portion 1−τ to the bank. …

Invertible Grayscale With Sparsity Enforcing Priors, Yong Du, Yangyang Xu, Taizhong Ye, Qiang Wen, Chufeng Xiao, Junyu Dong, Guoqiang Han, Shengfeng He

Research Collection School Of Computing and Information Systems

Color dimensionality reduction is believed as a non-invertible process, as re-colorization results in perceptually noticeable and unrecoverable distortion. In this article, we propose to convert a color image into a grayscale image that can fully recover its original colors, and more importantly, the encoded information is discriminative and sparse, which saves storage capacity. Particularly, we design an invertible deep neural network for color encoding and decoding purposes. This network learns to generate a residual image that encodes color information, and it is then combined with a base grayscale image for color recovering. In this way, the non-differentiable compression process (e.g., …

Forecasting Interaction Order On Temporal Graphs, Wenwen Xia, Yuchen Li, Jianwei Tian, Shenghong Li

Research Collection School Of Computing and Information Systems

Link prediction is a fundamental task for graph analysis and the topic has been studied extensively for static or dynamic graphs. Essentially, the link prediction is formulated as a binary classification problem about two nodes. However, for temporal graphs, links (or interactions) among node sets appear in sequential orders. And the orders may lead to interesting applications. While a binary link prediction formulation fails to handle such an order-sensitive case. In this paper, we focus on such an interaction order prediction (IOP) problem among a given node set on temporal graphs. For the technical aspect, we develop a graph neural …

Learning From Miscellaneous Other-Class Words For Few-Shot Named Entity Recognition, Meihan Tong, Shuai Wang, Bin Xu, Yixin Cao, Minghui Liu, Lei Hou, Juanzi Li

Research Collection School Of Computing and Information Systems

Few-shot Named Entity Recognition (NER) exploits only a handful of annotations to identify and classify named entity mentions. Prototypical network shows superior performance on few-shot NER. However, existing prototypical methods fail to differentiate rich semantics in other-class words, which will aggravate overfitting under few shot scenario. To address the issue, we propose a novel model, Mining Undefined Classes from Other-class (MUCO), that can automatically induce different undefined classes from the other class to improve few-shot NER. With these extra-labeled undefined classes, our method will improve the discriminative ability of NER classifier and enhance the understanding of predefined classes with stand-by …

A Survey On Ml4vis: Applying Machine Learning Advances To Data Visualization, Qianwen Wang, Zhutian Chen, Yong Wang, Huamin Qu

Research Collection School Of Computing and Information Systems

Inspired by the great success of machine learning (ML), researchers have applied ML techniques to visualizations to achieve a better design, development, and evaluation of visualizations. This branch of studies, known as ML4VIS, is gaining increasing research attention in recent years. To successfully adapt ML techniques for visualizations, a structured understanding of the integration of ML4VIS is needed. In this article, we systematically survey 88 ML4VIS studies, aiming to answer two motivating questions: “what visualization processes can be assisted by ML?” and “how ML techniques can be used to solve visualization problems? ” This survey reveals seven main processes where …

Learning Interpretable Concept Groups In Cnns, Saurabh Varshneya, Antoine Ledent, Rob Vandermeulen, Yunwen Lei, Matthias Enders, Damian Borth, Marius Kloft

Research Collection School Of Computing and Information Systems

We propose a novel training methodology---Concept Group Learning (CGL)---that encourages training of interpretable CNN filters by partitioning filters in each layer into concept groups, each of which is trained to learn a single visual concept. We achieve this through a novel regularization strategy that forces filters in the same group to be active in similar image regions for a given layer. We additionally use a regularizer to encourage a sparse weighting of the concept groups in each layer so that a few concept groups can have greater importance than others. We quantitatively evaluate CGL's model interpretability using standard interpretability evaluation …

An Empirical Study Of The Discreteness Prior In Low-Rank Matrix Completion, Rodrigo Alves, Antoine Ledent, Renato Assunção, Marius And Kloft

Research Collection School Of Computing and Information Systems

A reasonable assumption in recommender systems is that the rows (users) and columns (items) of the rating matrix can be split into groups (communities) with the following property: each entry of the matrix is the sum of components corresponding to community behavior and a purely low-rank component corresponding to individual behavior. We investigate (1) whether such a structure is present in real-world datasets, (2) whether the knowledge of the existence of such structure alone can improve performance, without explicit information about the community memberships. To these ends, we formulate a joint optimization problem over all (completed matrix, set of communities) …

Are Missing Links Predictable? An Inferential Benchmark For Knowledge Graph Completion, Yixin Cao, Xiang Ji, Xin Lv, Juanzi Li, Yonggang Wen, Hanwang Zhang

Research Collection School Of Computing and Information Systems

We present InferWiki, a Knowledge Graph Completion (KGC) dataset that improves upon existing benchmarks in inferential ability, assumptions, and patterns. First, each testing sample is predictable with supportive data in the training set. To ensure it, we propose to utilize rule-guided train/test generation, instead of conventional random split. Second, InferWiki initiates the evaluation following the open-world assumption and improves the inferential difficulty of the closed-world assumption, by providing manually annotated negative and unknown triples. Third, we include various inference patterns (e.g., reasoning path length and types) for comprehensive evaluation. In experiments, we curate two settings of InferWiki varying in sizes …

Leveraging Two Types Of Global Graph For Sequential Fashion Recommendation, Yujuan Ding, Yunshan Ma, Wai Keung Wong, Tat‑Seng Chua

Research Collection School Of Computing and Information Systems

Sequential fashion recommendation is of great significance in online fashion shopping, which accounts for an increasing portion of either fashion retailing or online e-commerce. The key to building an effective sequential fashion recommendation model lies in capturing two types of patterns: the personal fashion preference of users and the transitional relationships between adjacent items. The two types of patterns are usually related to user-item interaction and item-item transition modeling respectively. However, due to the large sets of users and items as well as the sparse historical interactions, it is difficult to train an effective and efficient sequential fashion recommendation model. …

Towards Generative Aspect-Based Sentiment Analysis, Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, Wai Lam

Research Collection School Of Computing and Information Systems

Aspect-based sentiment analysis (ABSA) has received increasing attention recently. Most existing work tackles ABSA in a discriminative manner, designing various task-specific classification networks for the prediction. Despite their effectiveness, these methods ignore the rich label semantics in ABSA problems and require extensive task-specific designs. In this paper, we propose to tackle various ABSA tasks in a unified generative framework. Two types of paradigms, namely annotation-style and extraction-style modeling, are designed to enable the training process by formulating each ABSA task as a text generation problem. We conduct experiments on four ABSA tasks across multiple benchmark datasets where our proposed generative …

Collaborative Development Of Spatial Audio Virtual Environments, George V. Landon, Austin K. Jaquith

Frameless

Access to the newest features of Virtual Reality headsets has become increasingly more accessible to student developers in recent years. Manufacturers are competing to support all available device features not just through their Software Development Kits (SDKs), but also integrated into industry-standard game engines. One particular feature, spatial sound, can now be deployed without directly accessing the SDK but instead modifying deployment settings and selecting checkboxes. This accessibility to new VR developers has opened up new opportunities for inter-disciplinary collaborations within constrained development cycles like an academic semester.

Rotateentry: Controller-Rolling-Style Text Entry For Three Degrees Of Freedom Virtual Reality Devices, Ziming Li, Roshan Peiris

Frameless

In this work, we propose RotateEntry, a controller-rolling-style method for text entry on three degrees of freedom virtual reality devices. To move the key-selecting cursor in two dimensions on a QWERTY layout virtual keyboard, we developed three variants of RotateEntry: Rotate Column Rotate, Rotate Key, and Rotate Column Point. We conducted a comparative empirical evaluation of the four text input methods, including three proposed controller-rolling-style text input methods and the standard raycasting-style one. Text entry performance, accuracy, workload, usability, and user experience were tested and evaluated. Due to the COVID-19 situation, our study was conducted remotely. The impact of using …