Digital Commons Network^™

Hate-Clipper: Multimodal Hateful Meme Classification Based On Cross-Modal Interaction Of Clip Features, Gokul Karthik Kumar, Karthik Nandakumar

Computer Vision Faculty Publications

Hateful memes are a growing menace on social media. While the image and its corresponding text in a meme are related, they do not necessarily convey the same meaning when viewed individually. Hence, detecting hateful memes requires careful consideration of both visual and textual information. Multimodal pretraining can be beneficial for this task because it effectively captures the relationship between the image and the text by representing them in a similar feature space. Furthermore, it is essential to model the interactions between the image and text features through intermediate fusion. Most existing methods either employ multimodal pre-training or intermediate fusion, …

Artificial Intelligence In The Medical Field: Medical Review Sentiment Analysis, Nicholas Podlesak

Honors Capstones

In this research project, natural language processing techniques’ ability to accurately classify medical text was measured to reinforce the relevance of artificial intelligence in the medical field. Sentiment analyses (analyses to determine whether the text was positive or negative) were performed on the prescription drug reviews in an open-source dataset using four different models: lexical, a neural network, a support vector machine, and a logistic regression model. Each model’s effectiveness was gauged by its ability to correctly classify unlabeled drug reviews (i.e., a percentage representing accuracy). The machine learning models were able to accurately classify the text, while the lexical …

Ur-280 Speech Recognition, Rachnicha Rojjhanarittikorn

C-Day Computing Showcase

This project makes use of speech recognition in Python libraries from Google, such as Google Speech-to-Text. The program first installs the dependencies that are required to run the program. The program then asks the user for their input. The user has the option of either exit the program, having the text read back to them, or responding to the conversation. If the user decides to have the text read back to them, the program will use the Google Text-to-Speech library to convert the text to speech. The purpose of this project is to understand how to implement speech recognition in …

Predicting Publication Of Clinical Trials Using Structured And Unstructured Data: Model Development And Validation Study, Siyang Wang, Simon Šuster, Timothy Baldwin, Karin Verspoor

Natural Language Processing Faculty Publications

Background: Publication of registered clinical trials is a critical step in the timely dissemination of trial findings. However, a significant proportion of completed clinical trials are never published, motivating the need to analyze the factors behind success or failure to publish. This could inform study design, help regulatory decision-making, and improve resource allocation. It could also enhance our understanding of bias in the publication of trials and publication trends based on the research direction or strength of the findings. Although the publication of clinical trials has been addressed in several descriptive studies at an aggregate level, there is a lack …

Assisting The Human Fact-Checkers: Detecting All Previously Fact-Checked Claims In A Document, Shaden Shaar, Nikola Georgiev, Firoj Alam, Giovanni Da San Martino, Aisha Mohamed, Preslav Nakov

Natural Language Processing Faculty Publications

Given the recent proliferation of false claims online, there has been a lot of manual fact-checking effort. As this is very time-consuming, human fact-checkers can benefit from tools that can support them and make them more efficient. Here, we focus on building a system that could provide such support. Given an input document, it aims to detect all sentences that contain a claim that can be verified by some previously fact-checked claims (from a given database). The output is a re-ranked list of the document sentences, so that those that can be verified are ranked as high as possible, together …

Overview Of The Wanlp 2022 Shared Task On Propaganda Detection In Arabic, Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, Preslav Nakov

Natural Language Processing Faculty Publications

Propaganda is the expression of an opinion or an action by an individual or a group deliberately designed to influence the opinions or the actions of other individuals or groups with reference to predetermined ends, which is achieved by means of well-defined rhetorical and psychological devices. Propaganda techniques are commonly used in social media to manipulate or to mislead users. Thus, there has been a lot of recent research on automatic detection of propaganda techniques in text as well as in memes. However, so far the focus has been primarily on English. With the aim to bridge this language gap, …

Supervised Acoustic Embeddings And Their Transferability Across Languages, Sreepratha Ram, Hanan Aldarmaki

Natural Language Processing Faculty Publications

In speech recognition, it is essential to model the phonetic content of the input signal while discarding irrelevant factors such as speaker variations and noise, which is challenging in low-resource settings. Self-supervised pretraining has been proposed as a way to improve both supervised and unsupervised speech recognition, including frame-level feature representations and Acoustic Word Embeddings (AWE) for variable-length segments. However, self-supervised models alone cannot learn perfect separation of the linguistic content as they are trained to optimize indirect objectives. In this work, we experiment with different pre-trained self-supervised features as input to AWE models and show that they work best …

Camelira: An Arabic Multi-Dialect Morphological Disambiguator, Ossama Obeid, Go Inoue, Nizar Habash

Computer Vision Faculty Publications

We present Camelira, a web-based Arabic multi-dialect morphological disambiguation tool that covers four major variants of Arabic: Modern Standard Arabic, Egyptian, Gulf, and Levantine. Camelira offers a user-friendly web interface that allows researchers and language learners to explore various linguistic information, such as part-of-speech, morphological features, and lemmas. Our system also provides an option to automatically choose an appropriate dialect-specific disambiguator based on the prediction of a dialect identification component. Camelira is publicly accessible at http://camelira.camel-lab.com.

Pasta: Table-Operations Aware Fact Verification Via Sentence-Table Cloze Pre-Training, Zihui Gu, Ju Fan, Nan Tang, Preslav Nakov, Xiaoman Zhao, Xiaoyong Du

Natural Language Processing Faculty Publications

Fact verification has attracted a lot of research attention recently, e.g., in journalism, marketing, and policymaking, as misinformation and disinformation online can sway one's opinion and affect one's actions. While fact-checking is a hard task in general, in many cases, false statements can be easily debunked based on analytics over tables with reliable information. Hence, table-based fact verification has recently emerged as an important and growing research area. Yet, progress has been limited due to the lack of datasets that can be used to pre-train language models (LMs) to be aware of common table operations, such as aggregating a column …

Greener: Graph Neural Networks For News Media Profiling, Panayot Panayotov, Utsav Shukla, Husrev T. Sencar, Mohamed Nabeel, Preslav Nakov

Natural Language Processing Faculty Publications

We study the problem of profiling news media on the Web with respect to their factuality of reporting and bias. This is an important but under-studied problem related to disinformation and “fake news” detection, but it addresses the issue at a coarser granularity compared to looking at an individual article or an individual claim. This is useful as it allows to profile entire media outlets in advance. Unlike previous work, which has focused primarily on text (e.g., on the articles published by the target website, or on the textual description in their social media profiles or in Wikipedia), here we …

Asdot: Any-Shot Data-To-Text Generation With Pretrained Language Models, Jiannan Xiang, Zhengzhong Liu, Yucheng Zhou, Eric P. Xing, Zhiting Hu

Machine Learning Faculty Publications

Data-to-text generation is challenging due to the great variety of the input data in terms of domains (e.g., finance vs sports) or schemata (e.g., diverse predicates). Recent end-to-end neural methods thus require substantial training examples to learn to disambiguate and describe the data. Yet, real-world data-to-text problems often suffer from various data-scarce issues: one may have access to only a handful of or no training examples, and/or have to rely on examples in a different domain or schema. To fill this gap, we propose Any-Shot Data-to-Text (ASDOT), a new approach flexibly applicable to diverse settings by making efficient use of …

Resel: N-Ary Relation Extraction From Scientific Text And Tables By Learning To Retrieve And Select, Yuchen Zhuang, Yinghao Li, Jerry Junyang Cheung, Yue Yu, Yingjun Mou, Xiang Chen, Le Song, Chao Zhang

Machine Learning Faculty Publications

We study the problem of extracting N-ary relation tuples from scientific articles. This task is challenging because the target knowledge tuples can reside in multiple parts and modalities of the document. Our proposed method RESEL decomposes this task into a two-stage procedure that first retrieves the most relevant paragraph/table and then selects the target entity from the retrieved component. For the high-level retrieval stage, RESEL designs a simple and effective feature set, which captures multilevel lexical and semantic similarities between the query and components. For the low-level selection stage, RESEL designs a cross-modal entity correlation graph along with a multi-view …

Efficient (Soft) Q-Learning For Text Generation With Limited Good Data, Han Guo, Bowen Tan, Zhengzhong Liu, Eric P. Xing, Zhiting Hu

Machine Learning Faculty Publications

Maximum likelihood estimation (MLE) is the predominant algorithm for training text generation models. This paradigm relies on direct supervision examples, which is not applicable to many emerging applications, such as generating adversarial attacks or generating prompts to control language models. Reinforcement learning (RL) on the other hand offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. Yet previous RL algorithms for text generation, such as policy gradient (on-policy RL) and Q-learning (off-policy RL), are often notoriously inefficient or unstable to train due to the large sequence space and the sparse reward received only …

Eureka: Euphemism Recognition Enhanced Through Knn-Based Methods And Augmentation, Sedrick Scott Keh, Rohit Bharadwaj, Emmy Liu, Simone Tedeschi, Varun Gangal, Roberto Navigli

Computer Vision Faculty Publications

We introduce EUREKA, an ensemble-based approach for performing automatic euphemism detection. We (1) identify and correct potentially mislabelled rows in the dataset, (2) curate an expanded corpus called EuphAug, (3) leverage model representations of Potentially Euphemistic Terms (PETs), and (4) explore using representations of semantically close sentences to aid in classification. Using our augmented dataset and kNN-based methods, EUREKA was able to achieve state-of-the-art results on the public leaderboard of the Euphemism Detection Shared Task, ranking first with a macro F1 score of 0.881.

Predicting Startup Success Using Publicly Available Data, Emily Gavrilenko

Master's Theses

Predicting the success of an early-stage startup has always been a major effort for investors and venture funds. Statistically, there are about 305 million total startups created in a year, but less than 10% of them succeed to become profitable businesses. Accurately identifying the signs of startup growth is the work of countless investors, and in recent years, research has turned to machine learning in hopes of improving the accuracy and speed of startup success prediction.

To learn about a startup, investors have to navigate many different internet sources and often rely on personal intuition to determine the startup’s potential …

Drug Abuse Ontology To Harness Web-Based Data For Substance Use Epidemiology Research, Usha Lokala, Francois Lamy, Raminta Daniulaityte, Manas Gaur, Amelie Gyrard, Krishnaprasad Thirunarayan, Ugur Kursuncu, Amit Sheth

Computer Science and Engineering Faculty Publications

Background: Web-based resources and social media platforms play an increasingly important role in health-related knowledge and experience sharing. There is a growing interest in the use of these novel data sources for epidemiological surveillance of substance use behaviors and trends. Objective: The key aims were to describe the development and application of the drug abuse ontology (DAO) as a framework for analyzing web-based and social media data to inform public health and substance use research in the following areas: determining user knowledge, attitudes, and behaviors related to nonmedical use of buprenorphine and illicitly manufactured opioids through the analysis of web …

Learning Generalizable Models For Vehicle Routing Problems Via Knowledge Distillation, Jieyi Bi, Yining Ma, Jiahai Wang, Zhiguang Cao, Jinbiao Chen, Yuan Sun, Yeow Meng Chee

Research Collection School Of Computing and Information Systems

Recent neural methods for vehicle routing problems always train and test the deep models on the same instance distribution (i.e., uniform). To tackle the consequent cross-distribution generalization concerns, we bring the knowledge distillation to this field and propose an Adaptive Multi-Distribution Knowledge Distillation (AMDKD) scheme for learning more generalizable deep models. Particularly, our AMDKD leverages various knowledge from multiple teachers trained on exemplar distributions to yield a light-weight yet generalist student model. Meanwhile, we equip AMDKD with an adaptive strategy that allows the student to concentrate on difficult distributions, so as to absorb hard-to-master knowledge more effectively. Extensive experimental results …

What Should Streamers Communicate In Livestream E-Commerce? The Effects Of Social Interactions On Live Streaming Performance, Danyang Song, Xi Chen, Zhiling Guo, Xiao Liu Liu, Ruijin. Jin

Research Collection School Of Computing and Information Systems

Compared with traditional e-commerce, livestreaming e-commerce is characterized by direct and intimate communication between streamers and consumers that stimulates instant social interactions. This study focuses on streamers’ three types of information exchange (i.e., product information, social conversation, and social solicitation) and examines their roles in driving both short-term and long-term livestreaming performance (i.e., sales and customer base growth). We find that the informational role of product information (nonpromotional and promotional) is beneficial not only to sales performance, but also to the growth of the customer base. We also find that social conversation has a relationship-building effect that positively impacts both …

Implementation Of Empath X Sla Predictive Tool For A Government Agency, Hui Shan Lee, Shankararaman, Venky, Eng Lieh Ouh

Research Collection School Of Computing and Information Systems

Service Level Agreement (SLA) plays a significant role in the relationship between citizens and the government. It stipulates the quality levels required for the meaningful interaction between the two parties. Most SLA predictive models consider end-to-end duration and frequency of failed service requests as model inputs with little research on the analysis of textual details of the service request. This is an issue for government bodies as the latter do not just want to meet SLA, but also be proactive by knowing the citizens before assisting them. Inclusion of textual data potentially answer to this requirement of knowing the citizen …

Assessment Design For Digital Education: An Analytics-Based Authentic Assessment Approach, Lim Ming Soon Tristan, Gottipati Swapna, Michelle L. F. Cheong, Christopher Pang, Jun Wei Ng

Research Collection School Of Computing and Information Systems

This study looks to identify an assessment design construct that overcomes known issues in authentic assessment design practices in digital education. These include lack of "freedom-of-choice", lack of focus on multimodal nature of the digital process, and shortage of effective feedbacks. This study proposes an authentic assessment, that combines gamification (G) with heutagogy (H) and multimodality (M) of assessments, building upon learning analytics (A), known as GHMA. Proposed assessment design is a skills-oriented game-based assessment approach. Learners can determine their own goals and create individualized multimodal artefacts; receive cognitive challenge through cognitively complex tasks structured in gamified non-linear learning paths; …

Learning Dynamic Multimodal Implicit And Explicit Networks For Multiple Financial Tasks, Meng Kiat Gary Ang, Ee-Peng Lim

Research Collection School Of Computing and Information Systems

Many financial f orecasting d eep l earning w orks focus on the single task of predicting stock returns for trading with unimodal numerical inputs. Investment and risk management however involves multiple financial t asks - f orecasts o f expected returns, risks and correlations of multiple stocks in portfolios, as well as important events affecting different stocks - to support decision making. Moreover, stock returns are influenced by large volumes of non-stationary time-series information from a variety of modalities and the propagation of such information across inter-company relationship networks. Such networks could be explicit - observed co-occurrences in online …

Towards Reinterpreting Neural Topic Models Via Composite Activations, Jia Peng Lim, Hady Wirawan Lauw

Research Collection School Of Computing and Information Systems

Most Neural Topic Models (NTM) use a variational auto-encoder framework producing K topics limited to the size of the encoder’s output. These topics are interpreted through the selection of the top activated words via the weights or reconstructed vector of the decoder that are directly connected to each neuron. In this paper, we present a model-free two-stage process to reinterpret NTM and derive further insights on the state of the trained model. Firstly, building on the original information from a trained NTM, we generate a pool of potential candidate “composite topics” by exploiting possible co-occurrences within the original set of …

Biasfinder: Metamorphic Test Generation To Uncover Bias For Sentiment Analysis Systems, Muhammad Hilmi Asyrofi, Zhou Yang, Imam Nur Bani Yusuf, Hong Jin Kang, Thung Ferdian, David Lo

Research Collection School Of Computing and Information Systems

Artificial intelligence systems, such as Sentiment Analysis (SA) systems, typically learn from large amounts of data that may reflect human bias. Consequently, such systems may exhibit unintended demographic bias against specific characteristics (e.g., gender, occupation, country-of-origin, etc.). Such bias manifests in an SA system when it predicts different sentiments for similar texts that differ only in the characteristic of individuals described. To automatically uncover bias in SA systems, this paper presents BiasFinder, an approach that can discover biased predictions in SA systems via metamorphic testing. A key feature of BiasFinder is the automatic curation of suitable templates from any given …

A Unified Dialogue User Simulator For Few-Shot Data Augmentation, Dazhen Wan, Zheng Zhang, Qi Zhu, Lizi Liao, Minlie Huang

Research Collection School Of Computing and Information Systems

Pre-trained language models have shown superior performance in task-oriented dialogues. However, existing datasets are on limited scales, which cannot support large-scale pre-training. Fortunately, various data augmentation methods have been developed to augment largescale task-oriented dialogue corpora. However, they heavily rely on annotated data in the target domain, which require a tremendous amount of data collection and human labeling work. In this paper, we build a unified dialogue user simulation model by pre-training on several publicly available datasets. The model can then be tuned on a target domain with fewshot data. The experiments on a target dataset across multiple domains show …

Multi-Relation Extraction Via A Global-Local Graph Convolutional Network, Harry Cheng, Lizi Liao, Linmei Hu, Liqiang Nie

Research Collection School Of Computing and Information Systems

Relation extraction (RE) extracts the semantic relations among entities in a sentence, which converts the unstructured text into structured and easy-to-understand information. Although RE has been studied over decades, it still faces two kinds of research challenges that are not well addressed thus far: 1) joint consideration of the global sentence structure and the local entity interaction, and 2) effective solution to the overlapping triplets within the same sentence. To tackle these issues, in this paper, we present globallocal graph-based convolutional network towards multi-relation extraction, GAME for short. In particular, we devise two layers of graph convolutional network (GCN) with …

R2f: A General Retrieval, Reading And Fusion Framework For Document-Level Natural Language Inference, Hao Wang, Yixin Cao, Yangguang Li, Zhen Huang, Kun Wang, Jing Shao

Research Collection School Of Computing and Information Systems

Document-level natural language inference (DocNLI) is a new challenging task in natural language processing, aiming at judging the entailment relationship between a pair of hypothesis and premise documents. Current datasets and baselines largely follow sentence-level settings, but fail to address the issues raised by longer documents. In this paper, we establish a general solution, named Retrieval, Reading and Fusion (R2F) framework, and a new setting, by analyzing the main challenges of DocNLI: interpretability, long-range dependency, and cross-sentence inference. The basic idea of the framework is to simplify document-level task into a set of sentence-level tasks, and improve both performance and …

Conversation Disentanglement With Bi-Level Contrastive Learning, Chengyu Huang, Zheng Zhang, Hao Fei, Lizi Liao

Research Collection School Of Computing and Information Systems

Conversation disentanglement aims to group utterances into detached sessions, which is a fundamental task in processing multi-party conversations. Existing methods have two main drawbacks. First, they overemphasize pairwise utterance relations but pay inadequate attention to the utterance-to-context relation modeling. Second, a huge amount of human annotated data is required for training, which is expensive to obtain in practice. To address these issues, we propose a general disentangle model based on bi-level contrastive learning. It brings closer utterances in the same session while encourages each utterance to be near its clustered session prototypes in the representation space. Unlike existing approaches, our …

Wordmuse, John M. Nelson

Computer Science and Software Engineering

Wordmuse is an application that allows users to enter a song and a list of keywords to create a new song. Built on Spotify's API, this project showcases the fusion of music composition and artificial intelligence. This paper also discusses the motivation, design, and creation of Wordmuse.

Hate Speech Detection Using Textual And User Features, Rohan Raut

Boise State University Theses and Dissertations

Social media platforms provide users with a powerful platform to share their ideas. Using one’s right to expression to incite hatred toward a particular group of people is inappropriate. However, hate speech is pervasive in our society. Spreading hate through online social networks like Facebook, Twitter, Tiktok, and Instagram is commonplace in today’s milieu. One such case is the unprecedented COVID-19 pandemic, which engendered anti-Asian hate.

In current literature, there is limited study on using user features in conjunction with textual features to detect hate. This thesis aims to combine textual features with user features to improve the state-of-the-art hate …

Conreader: Exploring Implicit Relations In Contracts For Contract Clause Extraction, Weiwen Xu, Yang Deng, Wenqiang Lei, Wenlong Zhao, Tat-Seng Chua, Wai Lam

Research Collection School Of Computing and Information Systems

We study automatic Contract Clause Extraction (CCE) by modeling implicit relations in legal contracts. Existing CCE methods mostly treat contracts as plain text, creating a substantial barrier to understanding contracts of high complexity. In this work, we first comprehensively analyze the complexity issues of contracts and distill out three implicit relations commonly found in contracts, namely, 1) Long-range Context Relation that captures the correlations of distant clauses; 2) Term-Definition Relation that captures the relation between important terms with their corresponding definitions; and 3) Similar Clause Relation that captures the similarities between clauses of the same type. Then we propose a …