Computer Sciences Commons^™

Wav-Bert: Cooperative Acoustic And Linguistic Representation Learning For Low-Resource Speech Recognition, Guolin Zheng, Yubei Xiao, Ke Gong, Pan Zhou, Xiaodan Liang, Liang Lin

Research Collection School Of Computing and Information Systems

Unifying acoustic and linguistic representation learning has become increasingly crucial to transfer the knowledge learned on the abundance of high-resource language data for low-resource speech recognition. Existing approaches simply cascade pre-trained acoustic and language models to learn the transfer from speech to text. However, how to solve the representation discrepancy of speech and text is unexplored, which hinders the utilization of acoustic and linguistic information. Moreover, previous works simply replace the embedding layer of the pre-trained language model with the acoustic features, which may cause the catastrophic forgetting problem. In this work, we introduce Wav-BERT, a cooperative acoustic and linguistic …

Masked Face Analysis Via Multi-Task Deep Learning, Vatsa S. Patel, Zhongliang Nie, Trung-Nghia Le, Tam Van Nguyen

Computer Science Faculty Publications

Face recognition with wearable items has been a challenging task in computer vision and involves the problem of identifying humans wearing a face mask. Masked face analysis via multi-task learning could effectively improve performance in many fields of face analysis. In this paper, we propose a unified framework for predicting the age, gender, and emotions of people wearing face masks. We first construct FGNET-MASK, a masked face dataset for the problem. Then, we propose a multi-task deep learning model to tackle the problem. In particular, the multi-task deep learning model takes the data as inputs and shares their weight to …

The Efficacy Of Collaborative Authoring Of Video Scene Descriptions, Rosiana Natalie, Jolene Kar Inn Loh, Huei Suen Tan, Joshua Shi-Hao Tseng, Ian Luke Yi-Ren Chan, Ebrima H. Jarjue, Hernisa Kacorri, Kotaro Hara

Research Collection School Of Computing and Information Systems

The majority of online video contents remain inaccessible to people with visual impairments due to the lack of audio descriptions to depict the video scenes. Content creators have traditionally relied on professionals to author audio descriptions, but their service is costly and not readily-available. We investigate the feasibility of creating more cost-effective audio descriptions that are also of high quality by involving novices. Specifically, we designed, developed, and evaluated ViScene, a web-based collaborative audio description authoring tool that enables a sighted novice author and a reviewer either sighted or blind to interact and contribute to scene descriptions (SDs)—text that can …

Causal Attention For Unbiased Visual Recognition, Tan Wang, Chang Zhou, Qianru Sun, Hanwang Zhang

Research Collection School Of Computing and Information Systems

Attention module does not always help deep models learn causal features that are robust in any confounding context, e.g., a foreground object feature is invariant to different backgrounds. This is because the confounders trick the attention to capture spurious correlations that benefit the prediction when the training and testing data are IID (identical & independent distribution); while harm the prediction when the data are OOD (out-of-distribution). The sole fundamental solution to learn causal attention is by causal intervention, which requires additional annotations of the confounders, e.g., a “dog” model is learned within “grass+dog” and “road+dog” respectively, so the “grass” and …

Aixfood'21: 3rd Workshop On Aixfood, Ricardo Guerrero, Michael Spranger, Shuqiang Jiang, Chong-Wah Ngo

Research Collection School Of Computing and Information Systems

Food and cooking analysis present exciting research and application challenges for modern AI systems, particularly in the context of multimodal data such as images or video. A meal that appears in a food image is a product of a complex progression of cooking stages, often described in the accompanying textual recipe form. In the cooking process, individual ingredients change their physical properties, become combined with other food components, all to produce a final, yet highly variable, appearance of the meal. Recognizing food items or meals on a plate from images or videos, their physical properties such as the amount, nutritional …

Conquer: Contextual Query-Aware Ranking For Video Corpus Moment Retrieval, Zhijian Hou, Chong-Wah Ngo, W. K. Chan

Research Collection School Of Computing and Information Systems

This paper tackles a recently proposed Video Corpus Moment Retrieval task. This task is essential because advanced video retrieval applications should enable users to retrieve a precise moment from a large video corpus. We propose a novel CONtextual QUery-awarE Ranking (CONQUER) model for effective moment localization and ranking. CONQUER explores query context for multi-modal fusion and representation learning in two different steps. The first step derives fusion weights for the adaptive combination of multi-modal video content. The second step performs bi-directional attention to tightly couple video and query as a single joint representation for moment localization. As query context is …

Constrained Contrastive Distribution Learning For Unsupervised Anomaly Detection And Localisation In Medical Images, Yu Tian, Guansong Pang, Fengbei Liu, Yuanhong Chen, Seon Ho Shin, Johan W. Verjans, Rajvinder Singh

Research Collection School Of Computing and Information Systems

Unsupervised anomaly detection (UAD) learns one-class classifiers exclusively with normal (i.e., healthy) images to detect any abnormal (i.e., unhealthy) samples that do not conform to the expected normal patterns. UAD has two main advantages over its fully supervised counterpart. Firstly, it is able to directly leverage large datasets available from health screening programs that contain mostly normal image samples, avoiding the costly manual labelling of abnormal samples and the subsequent issues involved in training with extremely class-imbalanced data. Further, UAD approaches can potentially detect and localise any type of lesions that deviate from the normal patterns. One significant challenge faced …

Differentiated Learning For Multi-Modal Domain Adaptation, Jianming Lv, Kaijie Liu, Shengfeng He

Research Collection School Of Computing and Information Systems

Directly deploying a trained multi-modal classifier to a new environment usually leads to poor performance due to the well-known domain shift problem. Existing multi-modal domain adaptation methods treated each modality equally and optimize the sub-models of different modalities synchronously. However, as observed in this paper, the degrees of domain shift in different modalities are usually diverse. We propose a novel Differentiated Learning framework to make use of the diversity between multiple modalities for more effective domain adaptation. Specifically, we model the classifiers of different modalities as a group of teacher/student sub-models, and a novel Prototype based Reliability Measurement is presented …

Learning To Adversarially Blur Visual Object Tracking, Qing Guo, Ziyi Cheng, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Yang Liu, Jianjun Zhao

Research Collection School Of Computing and Information Systems

Motion blur caused by the moving of the object or camera during the exposure can be a key challenge for visual object tracking, affecting tracking accuracy significantly. In this work, we explore the robustness of visual object trackers against motion blur from a new angle, i.e., adversarial blur attack (ABA). Our main objective is to online transfer input frames to their natural motion-blurred counterparts while misleading the state-of-the-art trackers during the tracking process. To this end, we first design the motion blur synthesizing method for visual tracking based on the generation principle of motion blur, considering the motion information and …

Prediction Of Synthetic Lethal Interactions In Human Cancers Using Multi-View Graph Auto-Encoder, Zhifeng Hao, Di Wu, Yuan Fang, Min Wu, Ruichu Cai, Xiaoli Li

Research Collection School Of Computing and Information Systems

Synthetic lethality (SL) is a very important concept for the development of targeted anticancer drugs. However, experimental methods for SL detection often suffer from various issues like high cost and low consistency across cell lines. Hence, computational methods for predicting novel SLs have recently emerged as complements for wet-lab experiments. In addition, SL data can be represented as a graph where nodes are genes and edges are the SL interactions. It is thus motivated to design advanced graph-based machine learning algorithms for SL prediction. In this paper, we propose a novel SL prediction method using Multi-view Graph Auto-Encoder (SLMGAE). We …

A Large-Scale Benchmark For Food Image Segmentation, Xiongwei Wu, Xin Fu, Ying Liu, Ee-Peng Lim, Steven C. H. Hoi, Qianru Sun

Research Collection School Of Computing and Information Systems

Food image segmentation is a critical and indispensible task for developing health-related applications such as estimating food calories and nutrients. Existing food image segmentation models are underperforming due to two reasons: (1) there is a lack of high quality food image datasets with fine-grained ingredient labels and pixel-wise location masks—the existing datasets either carry coarse ingredient labels or are small in size; and (2) the complex appearance of food makes it difficult to localize and recognize ingredients in food images, e.g., the ingredients may overlap one another in the same image, and the identical ingredient may appear distinctly in different …

Route Tapestries: Navigating 360° Virtual Tour Videos Using Slit-Scan Visualizations, Jiannan Li, Jiahe Lyu, Maurício Sousa, Ravin Balakrishnan, Anthony Tang, Tovi Grossman

Research Collection School Of Computing and Information Systems

An increasingly popular way of experiencing remote places is by viewing 360° virtual tour videos, which show the surrounding view while traveling through an environment. However, finding particular locations in these videos can be difficult because current interfaces rely on distorted frame previews for navigation. To alleviate this usability issue, we propose Route Tapestries, continuous orthographic-perspective projection of scenes along camera routes. We first introduce an algorithm for automatically constructing Route Tapestries from a 360° video, inspired by the slit-scan photography technique. We then present a desktop video player interface using a Route Tapestry timeline for navigation. An online evaluation …

Holoboard: A Large-Format Immersive Teaching Board Based On Pseudo Holographics, Jiangtao Gong, Teng Han, Siling Guo, Jiannan Li, Siyu Zha, Liuxin Zhang, Feng Tian, Qianying Wang, Yong Rui

Research Collection School Of Computing and Information Systems

In this paper, we present HoloBoard, an interactive large-format pseduo-holographic display system for lecture based classes. With its unique properties of immersive visual display and transparent screen, we designed and implemented a rich set of novel interaction techniques like immersive presentation, role-play, and lecturing behind the scene that are potentially valuable for lecturing in class. We conducted a controlled experimental study to compare a HoloBoard class with a normal class through measuring students’ learning outcomes and three dimensions of engagement (i.e., behavioral, emotional, and cognitive engagement). We used pre-/post- knowledge tests and multimodal learning analytics to measure students’ learning outcomes …

Uncovering Patterns In Reviewers' Feedback To Scene Description Authors, Rosiana Natalie, Jolene Kar Inn Loh, Huei Suen Tan, Joshua Shi-Hao Tseng, Hernisa Kacorri, Kotaro Hara

Research Collection School Of Computing and Information Systems

Audio descriptions (ADs) can increase access to videos for blind people. Researchers have explored different mechanisms for generating ADs, with some of the most recent studies involving paid novices; to improve the quality of their ADs, novices receive feedback from reviewers. However, reviewer feedback is not instantaneous. To explore the potential for real-time feedback through automation, in this paper, we analyze 1,120 comments that 40 sighted novices received from a sighted or a blind reviewer. We find that feedback patterns tend to fall under four themes: (i) Quality; commenting on different AD quality variables, (ii) Speech Act; the utterance or …

Visionary Caption: Improving The Accessibility Of Presentation Slides Through Highlighting Visualization, Carmen Ji Yan Yip, Jie Mi Chong, Sin Yee Kwek, Yong Wang, Kotaro Hara

Research Collection School Of Computing and Information Systems

Presentation slides are widely used in occasions such as academic talks and business meetings. Captions placed on slides support deaf and hard of hearing (DHH) people to understand spoken contents, but simultaneously comprehending and associating visual contents on slides and caption text could be challenging. In this paper, we design and develop a visualization technique to highlight and associate chart on a slide and numerical data in caption. We first conduct a small formative study with people with and without hearing impairments to assess the value of the visualization technique using a lo-fidelity video prototype. We then develop Visionary Caption, …

Visilence: An Interactive Visualization Tool For Error Resilience Analysis, Shaolun Ruan, Yong Wang, Qiang Guan

Research Collection School Of Computing and Information Systems

Soft errors have become one of the major concerns for HPC applications, as those errors can result in seriously corrupted outcomes, such as silent data corruptions (SDCs). Prior studies on error resilience have studied the robustness of HPC applications. However, it is still difficult for program developers to identify potential vulnerability to soft errors. In this paper, we present Visilence, a novel visualization tool to visually analyze error vulnerability based on the control-flow graph generated from HPC applications. Visilence efficiently visualizes the affected program states under injected errors and presents the visual analysis of the most vulnerable parts of an …

Condensing A Sequence To One Informative Frame For Video Recognition, Qiu. Zhaofan, Ting Yao, Yan Shu, Chong-Wah Ngo, Tao Mei

Research Collection School Of Computing and Information Systems

Video is complex due to large variations in motion and rich content in fine-grained visual details. Abstracting useful information from such information-intensive media requires exhaustive computing resources. This paper studies a two-step alternative that first condenses the video sequence to an informative" frame" and then exploits off-the-shelf image recognition system on the synthetic frame. A valid question is how to define" useful information" and then distill it from a video sequence down to one synthetic frame. This paper presents a novel Informative Frame Synthesis (IFS) architecture that incorporates three objective tasks, ie, appearance reconstruction, video categorization, motion estimation, and two …

Transporting Causal Mechanisms For Unsupervised Domain Adaptation, Zhongqi Yue, Qianru Sun, Xian-Sheng Hua, Hanwang Zhang

Research Collection School Of Computing and Information Systems

Existing Unsupervised Domain Adaptation (UDA) literature adopts the covariate shift and conditional shift assumptions, which essentially encourage models to learn common features across domains. However, due to the lack of supervision in the target domain, they suffer from the semantic loss: the feature will inevitably lose nondiscriminative semantics in source domain, which is however discriminative in target domain. We use a causal view—transportability theory [41]—to identify that such loss is in fact a confounding effect, which can only be removed by causal intervention. However, the theoretical solution provided by transportability is far from practical for UDA, because it requires the …

Self-Regulation For Semantic Segmentation, Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, Qianru Sun

Research Collection School Of Computing and Information Systems

In this paper, we seek reasons for the two major failure cases in Semantic Segmentation (SS): 1) missing small objects or minor object parts, and 2) mislabeling minor parts of large objects as wrong classes. We have an interesting finding that Failure-1 is due to the underuse of detailed features and Failure-2 is due to the underuse of visual contexts. To help the model learn a better trade-off, we introduce several Self-Regulation (SR) losses for training SS neural networks. By “self”, we mean that the losses are from the model per se without using any additional data or supervision. By …

Semi-Supervised Semantic Visualization For Networked Documents, Delvin Ce Zhang, Hady W. Lauw

Research Collection School Of Computing and Information Systems

Semantic interpretability and visual expressivity are important objectives in exploratory analysis of text. On the one hand, while some documents may have explicit categories, we could develop a better understanding of a corpus by studying its finer-grained structures, which may be latent. By inferring latent topics and discovering keywords associated with each topic, one obtains a semantic interpretation of the corpus. One the other hand, by visualizing documents, latent topics, and category labels on the same plot, one gains a bird’s eye view of the relationships among documents, topics, and various categories. Semantic visualization is a class of methods that …

Unit 1 - Virtual Machine Creation And Configuration, George A. Nossa

Open Educational Resources

The COVID pandemic has challenged us in continuing our instructional mission and provide high quality learning activities remotely. A key requirement for advanced technology courses is to duplicate the BMCC Computer Lab environment on the student's home computers.

This document provides detailed instructions for creating a "virtual machine" on the student's home computer. This document assumes the home computer is a PC desktop or laptop but this document will also work on computers running MacOS with an Intel X86 compatible chip,

Once the virtual machine is created, this document details the steps to install and configure an Ubuntu 20.04 Operating …

An Automated Method To Enrich Consumer Health Vocabularies Using Glove Word Embeddings And An Auxiliary Lexical Resource, Mohammed Ibrahim, Susan Gauch, Omar Salman, Mohammed Alqahtani

Computer Science and Computer Engineering Faculty Publications and Presentations

Background

Clear language makes communication easier between any two parties. A layman may have difficulty communicating with a professional due to not understanding the specialized terms common to the domain. In healthcare, it is rare to find a layman knowledgeable in medical terminology which can lead to poor understanding of their condition and/or treatment. To bridge this gap, several professional vocabularies and ontologies have been created to map laymen medical terms to professional medical terms and vice versa.

Objective

Many of the presented vocabularies are built manually or semi-automatically requiring large investments of time and human effort and consequently the …

Transitioning From Vue 2 To Vue 3, Adele Kanley

Theses/Capstones/Creative Projects

Frontend development is a field that is constantly changing because of the vast amounts of tools that are made available each year. One of the most popular frameworks being utilized to create fluid user experience is the Vue framework. Branching from the well-known Angular.js, Vue.js is an independent open-source project that is making its mark in the user interface community.

Regardless of the popularity of a framework, updates are inevitable to keep up with the innovations required by the IT Field. To ensure that UNO IS&T students are being offered opportunities to learn and develop in the most update to …

Forecasting Interaction Order On Temporal Graphs, Wenwen Xia, Yuchen Li, Jianwei Tian, Shenghong Li

Research Collection School Of Computing and Information Systems

Link prediction is a fundamental task for graph analysis and the topic has been studied extensively for static or dynamic graphs. Essentially, the link prediction is formulated as a binary classification problem about two nodes. However, for temporal graphs, links (or interactions) among node sets appear in sequential orders. And the orders may lead to interesting applications. While a binary link prediction formulation fails to handle such an order-sensitive case. In this paper, we focus on such an interaction order prediction (IOP) problem among a given node set on temporal graphs. For the technical aspect, we develop a graph neural …

Verification Of Piecewise Deep Neural Networks: A Star Set Approach With Zonotope Pre-Filter, Hoang-Dung Tran, Neelanjana Pal, Diego Manzanas Lopez, Patrick Musau, Xiaodong Yang, Luan Viet Nguyen, Weiming Xiang, Stanley Bak, Taylor T. Johnson

Computer Science Faculty Publications

Verification has emerged as a means to provide formal guarantees on learning-based systems incorporating neural network before using them in safety-critical applications. This paper proposes a new verification approach for deep neural networks (DNNs) with piecewise linear activation functions using reachability analysis. The core of our approach is a collection of reachability algorithms using star sets (or shortly, stars), an effective symbolic representation of high-dimensional polytopes. The star-based reachability algorithms compute the output reachable sets of a network with a given input set before using them for verification. For a neural network with piecewise linear activation functions, our approach can …

Fine-Grained Analysis Of Structured Output Prediction, Waleed Mustafa, Yunwen Lei, Antoine Ledent, Marius And Kloft

Research Collection School Of Computing and Information Systems

In machine learning we often encounter structured output prediction problems (SOPPs), i.e. problems where the output space admits a rich internal structure. Application domains where SOPPs naturally occur include natural language processing, speech recognition, and computer vision. Typical SOPPs have an extremely large label set, which grows exponentially as a function of the size of the output. Existing generalization analysis implies generalization bounds with at least a square-root dependency on the cardinality d of the label set, which can be vacuous in practice. In this paper, we significantly improve the state of the art by developing novel high-probability bounds with …

Learning From Miscellaneous Other-Class Words For Few-Shot Named Entity Recognition, Meihan Tong, Shuai Wang, Bin Xu, Yixin Cao, Minghui Liu, Lei Hou, Juanzi Li

Research Collection School Of Computing and Information Systems

Few-shot Named Entity Recognition (NER) exploits only a handful of annotations to identify and classify named entity mentions. Prototypical network shows superior performance on few-shot NER. However, existing prototypical methods fail to differentiate rich semantics in other-class words, which will aggravate overfitting under few shot scenario. To address the issue, we propose a novel model, Mining Undefined Classes from Other-class (MUCO), that can automatically induce different undefined classes from the other class to improve few-shot NER. With these extra-labeled undefined classes, our method will improve the discriminative ability of NER classifier and enhance the understanding of predefined classes with stand-by …

How Knowledge Graph And Attention Help? A Qualitative Analysis Into Bag-Level Relation Extraction, Zikun Hu, Yixin Cao, Lifu Huang, Tat-Seng Chua

Research Collection School Of Computing and Information Systems

Knowledge Graph (KG) and attention mechanism have been demonstrated effective in introducing and selecting useful information for weakly supervised methods. However, only qualitative analysis and ablation study are provided as evidence. In this paper, we contribute a dataset and propose a paradigm to quantitatively evaluate the effect of attention and KG on bag-level relation extraction (RE). We find that (1) higher attention accuracy may lead to worse performance as it may harm the model’s ability to extract entity mention features; (2) the performance of attention is largely influenced by various noise distribution patterns, which is closely related to real-world datasets; …

Leveraging Two Types Of Global Graph For Sequential Fashion Recommendation, Yujuan Ding, Yunshan Ma, Wai Keung Wong, Tat‑Seng Chua

Research Collection School Of Computing and Information Systems

Sequential fashion recommendation is of great significance in online fashion shopping, which accounts for an increasing portion of either fashion retailing or online e-commerce. The key to building an effective sequential fashion recommendation model lies in capturing two types of patterns: the personal fashion preference of users and the transitional relationships between adjacent items. The two types of patterns are usually related to user-item interaction and item-item transition modeling respectively. However, due to the large sets of users and items as well as the sparse historical interactions, it is difficult to train an effective and efficient sequential fashion recommendation model. …

Towards Generative Aspect-Based Sentiment Analysis, Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, Wai Lam

Research Collection School Of Computing and Information Systems

Aspect-based sentiment analysis (ABSA) has received increasing attention recently. Most existing work tackles ABSA in a discriminative manner, designing various task-specific classification networks for the prediction. Despite their effectiveness, these methods ignore the rich label semantics in ABSA problems and require extensive task-specific designs. In this paper, we propose to tackle various ABSA tasks in a unified generative framework. Two types of paradigms, namely annotation-style and extraction-style modeling, are designed to enable the training process by formulating each ABSA task as a text generation problem. We conduct experiments on four ABSA tasks across multiple benchmark datasets where our proposed generative …