Digital Commons Network^™

Token Shift Transformer For Video Classification, Zhang Hao, Yanbin. Hao, Chong-Wah Ngo

Research Collection School Of Computing and Information Systems

Transformer achieves remarkable successes in understanding 1 and 2-dimensional signals (e.g., NLP and Image Content Understanding). As a potential alternative to convolutional neural networks, it shares merits of strong interpretability, high discriminative power on hyper-scale data, and flexibility in processing varying length inputs. However, its encoders naturally contain computational intensive operations such as pair-wise self-attention, incurring heavy computational burden when being applied on the complex 3-dimensional video signals. This paper presents Token Shift Module (i.e., TokShift), a novel, zero-parameter, zero-FLOPs operator, for modeling temporal relations within each transformer encoder. Specifically, the TokShift barely temporally shifts partial [Class] token features back-and-forth …

Uncovering Patterns In Reviewers' Feedback To Scene Description Authors, Rosiana Natalie, Jolene Kar Inn Loh, Huei Suen Tan, Joshua Shi-Hao Tseng, Hernisa Kacorri, Kotaro Hara

Research Collection School Of Computing and Information Systems

Audio descriptions (ADs) can increase access to videos for blind people. Researchers have explored different mechanisms for generating ADs, with some of the most recent studies involving paid novices; to improve the quality of their ADs, novices receive feedback from reviewers. However, reviewer feedback is not instantaneous. To explore the potential for real-time feedback through automation, in this paper, we analyze 1,120 comments that 40 sighted novices received from a sighted or a blind reviewer. We find that feedback patterns tend to fall under four themes: (i) Quality; commenting on different AD quality variables, (ii) Speech Act; the utterance or …

Holoboard: A Large-Format Immersive Teaching Board Based On Pseudo Holographics, Jiangtao Gong, Teng Han, Siling Guo, Jiannan Li, Siyu Zha, Liuxin Zhang, Feng Tian, Qianying Wang, Yong Rui

Research Collection School Of Computing and Information Systems

In this paper, we present HoloBoard, an interactive large-format pseduo-holographic display system for lecture based classes. With its unique properties of immersive visual display and transparent screen, we designed and implemented a rich set of novel interaction techniques like immersive presentation, role-play, and lecturing behind the scene that are potentially valuable for lecturing in class. We conducted a controlled experimental study to compare a HoloBoard class with a normal class through measuring students’ learning outcomes and three dimensions of engagement (i.e., behavioral, emotional, and cognitive engagement). We used pre-/post- knowledge tests and multimodal learning analytics to measure students’ learning outcomes …

Aixfood'21: 3rd Workshop On Aixfood, Ricardo Guerrero, Michael Spranger, Shuqiang Jiang, Chong-Wah Ngo

Research Collection School Of Computing and Information Systems

Food and cooking analysis present exciting research and application challenges for modern AI systems, particularly in the context of multimodal data such as images or video. A meal that appears in a food image is a product of a complex progression of cooking stages, often described in the accompanying textual recipe form. In the cooking process, individual ingredients change their physical properties, become combined with other food components, all to produce a final, yet highly variable, appearance of the meal. Recognizing food items or meals on a plate from images or videos, their physical properties such as the amount, nutritional …

Integrated Discourse Analysis & Learning Skills Framework For Class Conversations, Devyn Wei Hung Tan, Gottipati Swapna, Kyong Jin Shim, Shankararaman, Venky

Research Collection School Of Computing and Information Systems

Constructive interactions through discussion forums allow students to open their horizons and thought processes to acquire more knowledge and develop skills. Thus, discussion forums play an important role in supporting learning. Additionally, the discussion forum provides the content for creating a knowledge repository. It contains discussion threads related to key course topics that are debated by the students. One approach to understanding the student learning experience is through the analysis of the discussion threads. This research proposes the application of discourse analysis and collaborative learning frameworks to discussion forums to gain further insights into the student’s learning in a classroom. …

Users’ Sentiment Analysis Toward National Digital Library Of India: A Quantitative Approach For Understanding User Perception, Ritu Sharma, Sarita Gulati, Amanpreet Kaur, Rupak Chakravarty

Library Philosophy and Practice (e-journal)

Sentiment analysis is also known as opinion mining. Sentiment analysis is contextual mining of text which identifies and extracts subjective information in textual data. It is extremely used by business, educational organizations, and social media monitoring to gain the general outlook of the wide public regarding their product and policy. The current study looks for gaining insights into user reviews on the National Digital Library of India (NDLI) mobile app (android and iOS). For this purpose, sentiment analysis will be used. It yields an average of 3.64/5 ratings based on 11,861 reviews. The dataset includes a total of 4560 user …

Cognition-Enhanced Machine Learning For Better Predictions With Limited Data, Florian Sense, Ryan Wood, Michael G. Collins, Joshua Fiechter, Aihua W. Wood, Michael Krusmark, Tiffany Jastrzembski, Christopher W. Myers

Faculty Publications

The fields of machine learning (ML) and cognitive science have developed complementary approaches to computationally modeling human behavior. ML's primary concern is maximizing prediction accuracy; cognitive science's primary concern is explaining the underlying mechanisms. Cross-talk between these disciplines is limited, likely because the tasks and goals usually differ. The domain of e-learning and knowledge acquisition constitutes a fruitful intersection for the two fields’ methodologies to be integrated because accurately tracking learning and forgetting over time and predicting future performance based on learning histories are central to developing effective, personalized learning tools. Here, we show how a state-of-the-art ML model can …

Generative Adversarial Networks For Classic Cryptanalysis, Deanne Charan

Master's Projects

The necessity of protecting critical information has been understood for millennia. Although classic ciphers have inherent weaknesses in comparison to modern ciphers, many classic ciphers are extremely challenging to break in practice. Machine learning techniques, such as hidden Markov models (HMM), have recently been applied with success to various classic cryptanalysis problems. In this research, we consider the effectiveness of the deep learning technique CipherGAN---which is based on the well- established generative adversarial network (GAN) architecture---for classic cipher cryptanalysis. We experiment extensively with CipherGAN on a number of classic ciphers, and we compare our results to those obtained using HMMs.

In-Game Social Interactions To Facilitate Esl Students' Morphological Awareness, Language And Literacy Skills, Yolanda A. Rankin, Sana Tibi, Casey Kennington, Na-Eun Han

Computer Science Faculty Publications and Presentations

Video games that require players to utilize a target or second language to complete tasks have emerged as alternative pedagogical tools for Second Language Acquisition (SLA). With the exception of vocabulary acquisition, much of the prior research in game-based SLA fails to gauge students' literacy skills, specifically their morphological awareness or understanding of the smallest meaningful linguistic units (e.g., prefixes, suffixes, and roots). Given this shortcoming, we utilize a two-player online game to facilitate social interactions between Native English Speakers (NES) and English as a Second Language (ESL) students as a mechanism to generate ESL students' written output in the …

Does Bert Understand Idioms? A Probing-Based Empirical Study Of Bert Encodings Of Idioms, Minghuan Tan, Jing Jiang

Research Collection School Of Computing and Information Systems

Understanding idioms is important in NLP. In this paper, we study to what extent pre-trained BERT model can encode the meaning of a potentially idiomatic expression (PIE) in a certain context. We make use of a few existing datasets and perform two probing tasks: PIE usage classification and idiom paraphrase identification. Our experiment results suggest that BERT indeed can separate the literal and idiomatic usages of a PIE with high accuracy. It is also able to encode the idiomatic meaning of a PIE to some extent.

Enhancing Project Based Learning With Unsupervised Learning Of Project Reflections, Hua Leong Fwa

Research Collection School Of Computing and Information Systems

Natural Language Processing (NLP) is an area of research and application that uses computers to analyze human text. It has seen wide adoption within several industries but few studies have investigated it for use in evaluating the effectiveness of educational interventions and pedagogies. Pedagogies such as Project based learning (PBL) centers on learners solving an authentic problem or challenge which leads to knowledge creation and higher engagement. PBL also lends itself well in plugging the gap between what is taught in classrooms and applying the knowledge gained to the real working environment. In this study, we seek to investigate how …

Exploiting Group Structures To Infer Social Interactions From Videos, Maksim Bolonkin

Dartmouth College Ph.D Dissertations

In this thesis, we consider the task of inferring the social interactions between humans by analyzing multi-modal data. Specifically, we attempt to solve some of the problems in interaction analysis, such as long-term deception detection, political deception detection, and impression prediction. In this work, we emphasize the importance of using knowledge about the group structure of the analyzed interactions. Previous works on the matter mostly neglected this aspect and analyzed a single subject at a time. Using the new Resistance dataset, collected by our collaborators, we approach the problem of long-term deception detection by designing a class of histogram-based features …

Exploring The Personality Of Virtual Tutors In Conversational Foreign Language Practice, Johanna Dobbriner, Cathy Ennis, Robert J. Ross

Conference papers

Fluid interaction between virtual agents and humans requires the understanding of many issues of conversational pragmatics. One such issue is the interaction between communication strategy and personality. As a step towards developing models of personality driven pragmatics policies, in this paper, we present our initial experiment to explore differences in user interaction with two contrasting avatar personalities. Each user saw a single personality in a video-call setting and gave feedback on the interaction. Our expectations, that a more extroverted outgoing positive personality would be a more successful tutor, were only partially confirmed. While this personality did induce longer conversations in …

Biasheal: On-The-Fly Black-Box Healing Of Bias In Sentiment Analysis Systems, Zhou Yang, Harshit Jain, Jieke Shi, Muhammad Hilmi Asyrofi, David Lo

Research Collection School Of Computing and Information Systems

Although Sentiment Analysis (SA) is widely applied in many domains, existing research has revealed that the unfairness in SA systems can be harmful to the welfare of less privileged people. Several works propose pre-processing and in-processing methods to eliminate bias in SA systems, but little attention is paid to utilizing post-processing methods to heal bias. Postprocessing methods are particularly important for systems that use third-party SA services. Systems that use such services have no access to the SA engine or its training data and thus cannot apply pre-processing nor in-processing methods. Therefore, this paper proposes a black-box post-processing method to …

Learning And Evaluating Chinese Idiom Embeddings, Minghuan Tan, Jing Jiang

Research Collection School Of Computing and Information Systems

We study the task of learning and evaluating Chinese idiom embeddings. We first construct a new evaluation dataset that contains idiom synonyms and antonyms. Observing that existing Chinese word embedding methods may not be suitable for learning idiom embeddings, we further present a BERT-based method that directly learns embedding vectors for individual idioms. We empirically compare representative existing methods and our method. We find that our method substantially outperforms existing methods on the evaluation dataset we have constructed.

Injecting Descriptive Meta-Information Into Pre-Trained Language Models With Hypernetworks, Wenying Duan, Xiaoxi He, Zimu Zhou, Hong Rao, Lothar Thiele

Research Collection School Of Computing and Information Systems

Pre-trained language models have been widely adopted as backbones in various natural language processing tasks. However, existing pre-trained language models ignore the descriptive meta-information in the text such as the distinction between the title and the mainbody, leading to over-weighted attention to insignificant text. In this paper, we propose a hypernetwork-based architecture to model the descriptive meta-information and integrate it into pre-trained language models. Evaluations on three natural language processing tasks show that our method notably improves the performance of pre-trained language models and achieves the state-of-the-art results on keyphrase extraction.

Partially Speaker-Dependent Automatic Speech Recognition Using Deep Neural Networks, Christopher J. Li, Michelle Spigner

Journal of the South Carolina Academy of Science

No abstract provided.

Transfer Learning For Low-Resource Part-Of-Speech Tagging, Jeffrey Zhou, Neha Verma

The Yale Undergraduate Research Journal

Neural network approaches to Part-of-Speech tagging, like other supervised neural network tasks, benefit from larger quantities of labeled data. However, in the case of low-resource languages, additional methods are necessary to improve the performances of POS taggers. In this paper, we explore transfer learning approaches to improve POS tagging in Afrikaans using a neural network. We investigate the effect of transferring network weights that were originally trained for POS tagging in Dutch. We also test the use of pretrained word embeddings in our POS tagger, both independently and in conjunction with the transferred weights from a Dutch POS tagger. We …

An Automated Method To Enrich Consumer Health Vocabularies Using Glove Word Embeddings And An Auxiliary Lexical Resource, Mohammed Ibrahim, Susan Gauch, Omar Salman, Mohammed Alqahtani

Computer Science and Computer Engineering Faculty Publications and Presentations

Background

Clear language makes communication easier between any two parties. A layman may have difficulty communicating with a professional due to not understanding the specialized terms common to the domain. In healthcare, it is rare to find a layman knowledgeable in medical terminology which can lead to poor understanding of their condition and/or treatment. To bridge this gap, several professional vocabularies and ontologies have been created to map laymen medical terms to professional medical terms and vice versa.

Objective

Many of the presented vocabularies are built manually or semi-automatically requiring large investments of time and human effort and consequently the …

Classification Of Explainable Artificial Intelligence Methods Through Their Output Formats, Giulia Vilone, Luca Longo

Articles

Machine and deep learning have proven their utility to generate data-driven models with high accuracy and precision. However, their non-linear, complex structures are often difficult to interpret. Consequently, many scholars have developed a plethora of methods to explain their functioning and the logic of their inferences. This systematic review aimed to organise these methods into a hierarchical classification system that builds upon and extends existing taxonomies by adding a significant dimension—the output formats. The reviewed scientific papers were retrieved by conducting an initial search on Google Scholar with the keywords “explainable artificial intelligence”; “explainable machine learning”; and “interpretable machine learning”. …

Tweet-To-Act: Towards Tweet-Mining Framework For Extracting Terrorist Attack-Related Information And Reporting, Farkhund Iqbal, Rabia Batool, Benjamin C. M. Fung, Saiqa Aleem, Ahmed Abbasi, Abdul Rehman Javed

All Works

The widespread popularity of social networking is leading to the adoption of Twitter as an information dissemination tool. Existing research has shown that information dissemination over Twitter has a much broader reach than traditional media and can be used for effective post-incident measures. People use informal language on Twitter, including acronyms, misspelled words, synonyms, transliteration, and ambiguous terms. This makes incident-related information extraction a non-trivial task. However, this information can be valuable for public safety organizations that need to respond in an emergency. This paper proposes an early event-related information extraction and reporting framework that monitors Twitter streams, synthesizes event-specific …

Modeling Factual Claims With Semantic Frames: Definitions, Datasets, Tools, And Fact-Checking Applications, Fatma Arslan

Computer Science and Engineering Dissertations - Archive

As social media sites have become major channels for the quick dissemination of news, misinformation has become a significant challenge for our society to tackle. Today fact-checking rests primarily on the shoulders of human fact-checkers who laboriously sift through various trustworthy sources, interview subject experts, and check references before reaching a verdict regarding the degree of truthfulness of a factual claim. Compounded with the speed and scale at which misinformation spreads, the demanding process may leave many harmful factual claims unchecked. In the fight to curb the spread of misinformation, researchers from various disciplines have come forward to assist fact-checkers …

Boundary Detection With Bert For Span-Level Emotion Cause Analysis, Xiangju Li, Wei Gao, Shi Feng, Yifei Zhang, Daling Wang

Research Collection School Of Computing and Information Systems

Emotion cause analysis (ECA) has been anemerging topic in natural language processing,which aims to identify the reasons behind acertain emotion expressed in the text. MostECA methods intend to identify the clausewhich contains the cause of a given emotion,but such clause-level ECA (CECA) can be ambiguous and imprecise. In this paper, we aimat span-level ECA (SECA) by detecting theprecise boundaries of text spans conveying accurate emotion causes from the given context.We formulate this task as sequence labelingand position identification problems and design two neural methods to solve them. Experiments on two benchmark ECA datasets showthat the proposed methods substantially outperform the …

Modeling Transitions Of Focal Entities For Conversational Knowledge Base Question Answering, Yunshi Lan, Jing Jiang

Research Collection School Of Computing and Information Systems

Conversational KBQA is about answering a sequence of questions related to a KB. Follow-up questions in conversational KBQA often have missing information referring to entities from the conversation history. In this paper, we propose to model these implied entities, which we refer to as the focal entities of the conversation. We propose a novel graph-based model to capture the transitions of focal entities and apply a graph neural network to derive a probability distribution of focal entities for each question, which is then combined with a standard KBQA module to perform answer ranking. Our experiments on two datasets demonstrate the …

Into The Unknown: Exploration Of Search Engines' Responses To Users With Depression And Anxiety, Ashlee Milton

Boise State University Theses and Dissertations

Mental health disorders (MHD) are a rising, yet stigmatized, topic. With statistics reporting that one in five adults in the United States will be afflicted by a MHD in their lifetime, researchers have begun exploring the behavioral nuances that emerge from interactions of these individuals with persuasive technologies, mainly social media. Yet, there is a gap in the analysis pertaining to a persuasive technology that is part of their everyday lives: search engines (SE). Each day, users with MHD embark on information seeking journeys using SE. Every step of the search process for better or worse has the potential to …

Application Of Artificial Intelligence And Machine Learning In Libraries: A Systematic Review, Rajesh Kumar Das, Mohammad Sharif Ul Islam

Library Philosophy and Practice (e-journal)

As the concept and implementation of cutting-edge technologies like artificial intelligence and machine learning has become relevant, academics, researchers and information professionals involve research in this area. The objective of this systematic literature review is to provide a synthesis of empirical studies exploring application of artificial intelligence and machine learning in libraries. To achieve the objectives of the study, a systematic literature review was conducted based on the original guidelines proposed by Kitchenham et al. (2009). Data was collected from Web of Science, Scopus, LISA and LISTA databases. Following the rigorous/ established selection process, a total of thirty-two articles were …

Estimating Homophily In Social Networks Using Dyadic Predictions, George Berry, Antonio Sirianni, Ingmar Weber, Jisun An, Michael Macy

Research Collection School Of Computing and Information Systems

Predictions of node categories are commonly used to estimate homophily and other relational properties in networks. However, little is known about the validity of using predictions for this task. We show that estimating homophily in a network is a problem of predicting categories of dyads (edges) in the graph. Homophily estimates are unbiased when predictions of dyad categories are unbiased. Node-level prediction models, such as the use of names to classify ethnicity or gender, do not generally produce unbiased predictions of dyad categories and therefore produce biased homophily estimates. Bias comes from three sources: sampling bias, correlation between model errors …

Learning-Based Extraction Of First-Order Logic Representations Of Api Directives, Mingwei Liu, Xin Peng, Andrian Marcus, Christoph Treude, Xuefang Bai, Gang Lyu, Jiazhen Xie, Xiaoxin Zhang

Research Collection School Of Computing and Information Systems

Developers often rely on API documentation to learn API directives, i.e., constraints and guidelines related to API usage. Failing to follow API directives may cause defects or improper implementations. Since there are no industry-wide standards on how to document API directives, they take many forms and are often hard to understand by developers or challenging to parse with tools. In this paper, we propose a learning based approach for extracting first-order logic representations of API directives (FOL directives for short). The approach, called LeadFOL, uses a joint learning method to extract atomic formulas by identifying the predicates and arguments involved …

Towards Generative Aspect-Based Sentiment Analysis, Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, Wai Lam

Research Collection School Of Computing and Information Systems

Aspect-based sentiment analysis (ABSA) has received increasing attention recently. Most existing work tackles ABSA in a discriminative manner, designing various task-specific classification networks for the prediction. Despite their effectiveness, these methods ignore the rich label semantics in ABSA problems and require extensive task-specific designs. In this paper, we propose to tackle various ABSA tasks in a unified generative framework. Two types of paradigms, namely annotation-style and extraction-style modeling, are designed to enable the training process by formulating each ABSA task as a text generation problem. We conduct experiments on four ABSA tasks across multiple benchmark datasets where our proposed generative …

Biasrv: Uncovering Biased Sentiment Predictions At Runtime, Zhou Yang, Muhammad Hilmi Asyrofi, David Lo

Research Collection School Of Computing and Information Systems

Sentiment analysis (SA) systems, though widely applied in many domains, have been demonstrated to produce biased results. Some research works have been done in automatically generating test cases to reveal unfairness in SA systems, but the community still lacks tools that can monitor and uncover biased predictions at runtime. This paper fills this gap by proposing BiasRV, the first tool to raise an alarm when a deployed SA system makes a biased prediction on a given input text. To implement this feature, BiasRV dynamically extracts a template from an input text and from the template generates gender-discriminatory mutants (semanticallyequivalent texts …