Artificial Intelligence and Robotics Commons^™

Deep Learning-Based Human Action Understanding In Videos, Elahe Vahdani

Dissertations, Theses, and Capstone Projects

The understanding of human actions in videos holds immense potential for technological advancement and societal betterment. This thesis explores fundamental aspects of this field, including action recognition in trimmed clips and action localization in untrimmed videos. Trimmed videos contain only one action instance, with moments before or after the action excluded from the video. However, the majority of videos captured in unconstrained environments, often referred to as untrimmed videos, are naturally unsegmented. Untrimmed videos are typically lengthy and may encompass multiple action instances, along with the moments preceding or following each action, as well as transitions between actions. In the …

Optimization And Application Of Graph Neural Networks, Shuo Zhang

Dissertations, Theses, and Capstone Projects

Graph Neural Networks (GNNs) are widely recognized for their potential in learning from graph-structured data and solving complex problems. However, optimal performance and applicability of GNNs have been an open-ended challenge. This dissertation presents a series of substantial advances addressing this problem. First, we investigate attention-based GNNs, revealing a critical shortcoming: their ignorance of cardinality information that impacts their discriminative power. To rectify this, we propose Cardinality Preserved Attention (CPA) models that can be applied to any attention-based GNNs, which exhibit a marked improvement in performance. Next, we introduce the Directional Node Pair (DNP) descriptor and the Robust Molecular Graph …

Out-Of-Distribution Generalization Of Deep Learning To Illuminate Dark Protein Functional Space, Tian Cai

Dissertations, Theses, and Capstone Projects

Dark protein illumination is a fundamental challenge in drug discovery where majority human proteins are understudied, i.e. with only known protein sequence but no known small molecule binder. It's a major road block to enable drug discovery paradigm shift from single-targeted which looks to identify a single target and design drug to regulate the single target to multi-targeted in a Systems Pharmacology perspective. Diseases such as Alzheimer's and Opioid-Use-Disorder plaguing millions of patients call for effective multi-targeted approach involving dark proteins. Using limited protein data to predict dark protein property requires deep learning systems with OOD generalization capacity. Out-of-Distribution (OOD) …

Ai-Supported Academic Advising: Exploring Chatgpt’S Current State And Future Potential Toward Student Empowerment, Daisuke Akiba, Michelle C. Fraboni

Publications and Research

Artificial intelligence (AI), once a phenomenon primarily in the world of science fiction, has evolved rapidly in recent years, steadily infiltrating into our daily lives. ChatGPT, a freely accessible AI-powered large language model designed to generate human-like text responses to users, has been utilized in several areas, such as the healthcare industry, to facilitate interactive dissemination of information and decision-making. Academic advising has been essential in promoting success among university students, particularly those from disadvantaged backgrounds. Unfortunately, however, student advising has been marred with problems, with the availability and accessibility of adequate advising being among the hurdles. The current study …

Evaluating Neural Networks As Cognitive Models For Learning Quasi-Regularities In Language, Xiaomeng Ma

Dissertations, Theses, and Capstone Projects

Many aspects of language can be categorized as quasi-regular: the relationship between the inputs and outputs is systematic but allows many exceptions. Common domains that contain quasi-regularity include morphological inflection and grapheme-phoneme mapping. How humans process quasi-regularity has been debated for decades. This thesis implemented modern neural network models, transformer models, on two tasks: English past tense inflection and Chinese character naming, to investigate how transformer models perform quasi-regularity tasks. This thesis focuses on investigating to what extent the models' performances can represent human behavior. The results show that the transformers' performance is very similar to human behavior in many …

Artificial Intelligence In Neuroradiology: A Scoping Review Of Some Ethical Challenges, Pegah Khosravi, Mark Schweitzer

Publications and Research

Artificial intelligence (AI) has great potential to increase accuracy and efficiency in many aspects of neuroradiology. It provides substantial opportunities for insights into brain pathophysiology, developing models to determine treatment decisions, and improving current prognostication as well as diagnostic algorithms. Concurrently, the autonomous use of AI models introduces ethical challenges regarding the scope of informed consent, risks associated with data privacy and protection, potential database biases, as well as responsibility and liability that might potentially arise. In this manuscript, we will first provide a brief overview of AI methods used in neuroradiology and segue into key methodological and ethical challenges. …

Towards An Unsupervised Bayesian Network Pipeline For Explainable Prediction, Decision Making And Discovery, Daniel Mallia

Theses and Dissertations

An unsupervised learning pipeline for discrete Bayesian networks is proposed to facilitate prediction, decision making, discovery of patterns, and transparency in challenging real-world AI applications, and contend with data limitations. We explore methods for discretizing data, and notably apply the pipeline to prediction and prevention of preterm birth.

Artificial Intelligence And The Situational Rationality Of Diagnosis: Human Problem-Solving And The Artifacts Of Health And Medicine, Michael W. Raphael

Publications and Research

What is the problem-solving capacity of artificial intelligence (AI) for health and medicine? This paper draws out the cognitive sociological context of diagnostic problem-solving for medical sociology regarding the limits of automation for decision-based medical tasks. Specifically, it presents a practical way of evaluating the artificiality of symptoms and signs in medical encounters, with an emphasis on the visualization of the problem-solving process in doctor-patient relationships. In doing so, the paper details the logical differences underlying diagnostic task performance between man and machine problem-solving: its principle of rationality, the priorities of its means of adaptation to abstraction, and the effects …

Data-Centric Machine Learning For Speech And Audio, Ali Raza Syed

Dissertations, Theses, and Capstone Projects

There is growing recognition of the importance of data-centric methods for building machine learning systems. Data-centric methods assume a fixed model and iterate over the data to improve system performance. This is in contrast to traditional model-centric approaches, which assume a fixed dataset and iterate over models for the same ends. Data-centric machine learning is driven by the observation that, beyond the size of the training data, model performance depends on factors such as the quality of the annotations, and whether the data are representative of conditions in which models will be deployed. This is particularly of interest in the …

Influence Level Prediction On Social Media Through Multi-Task And Sociolinguistic User Characteristics Modeling, Denys Katerenchuk

Dissertations, Theses, and Capstone Projects

Prediction of a user’s influence level on social networks has attracted a lot of attention as human interactions move online. Influential users have the ability to influence others’ behavior to achieve their own agenda. As a result, predicting users’ level of influence online can help to understand social networks, forecast trends, prevent misinformation, etc. The research on user influence in social networks has attracted much attention across multiple disciplines, from social sciences to mathematics, yet it is still not well understood. One of the difficulties is that the definition of influence is specific to a particular problem or a domain, …

The Interaction Of Different Primary Producers And Physical And Chemical Dynamics Of An Urban Shallow Lake, Majid Sahin

Dissertations, Theses, and Capstone Projects

An artificial urban shallow lake, Prospect Park Lake (PPL), is situated on a terminal moraine in Brooklyn New York, and supplied with municipal water treated with ortho-phosphates. The constant input of the phosphate nutrient is the primary source of eutrophication in the lake. The numerous pools along the water course houses various aquatic phototrophs, which influence the water quality and the state of the system, driving conditions into favoring the survival of their species. In the first half of the dissertation, the focus of the project is on analyzing how the different primary producers in different regions of PPL affect …

Finite Gaussian Neurons: Defending Against Adversarial Attacks By Making Neural Networks Say "I Don’T Know", Felix Grezes

Dissertations, Theses, and Capstone Projects

In this work, I introduce the Finite Gaussian Neuron (FGN), a novel neuron architecture for artificial neural networks aimed at protecting against adversarial attacks.
Since 2014, artificial neural networks have been known to be vulnerable to adversarial attacks, which can fool the network into producing wrong or nonsensical outputs by making humanly imperceptible alterations to inputs. While defenses against adversarial attacks have been proposed, they usually involve retraining a new neural network from scratch, a costly task.

My works aims to:
- easily convert existing models to Finite Gaussian Neuron architecture,
- while preserving the existing model's behavior on real …

Computer-Aided Response-To-Intervention For Reading Comprehension Based On Recommender System, Ming-Chi Liu, Wei-Yang Lin, Chia-Ling Tsai

Publications and Research

In 2019, New York State Education Department announced 54.6% of all students in grades 3 to 8 not meeting the standard of reading proficiency. Motivated by the need for a more efficient intervention model, we propose a recommender system to leverage the technology in machine learning to recommend suitable reading materials for effective intervention. The recommendation is based on the student's prior reading comprehension assessments and also assessments of other students at the same grade level using collaborative filtering. No other prior academic or demographic information of students is available. Two main challenges are lack of explicit ratings of reading …

Challenges In Migrating Imperative Deep Learning Programs To Graph Execution: An Empirical Study, Tatiana Castro Vélez, Raffi T. Khatchadourian, Mehdi Bagherzadeh, Anita Raja

Publications and Research

Efficiency is essential to support responsiveness w.r.t. ever-growing datasets, especially for Deep Learning (DL) systems. DL frameworks have traditionally embraced deferred execution-style DL code that supports symbolic, graph-based Deep Neural Network (DNN) computation. While scalable, such development tends to produce DL code that is error-prone, non-intuitive, and difficult to debug. Consequently, more natural, less error-prone imperative DL frameworks encouraging eager execution have emerged at the expense of run-time performance. While hybrid approaches aim for the "best of both worlds," the challenges in applying them in the real world are largely unknown. We conduct a data-driven analysis of challenges—and resultant bugs—involved …

The Significance Of Sonic Branding To Strategically Stimulate Consumer Behavior: Content Analysis Of Four Interviews From Jeanna Isham’S “Sound In Marketing” Podcast, Ina Beilina

Student Theses and Dissertations

Purpose:
Sonic branding is not just about composing jingles like McDonald’s “I’m Lovin’ It.” Sonic branding is an industry that strategically designs a cohesive auditory component of a brand’s corporate identity. This paper examines the psychological impact of music and sound on consumer behavior reviewing studies from the past 40 years and investigates the significance of stimulating auditory perception by infusing sound in consumer experience in the modern 2020s.

Design/methodology/approach:
Qualitative content analysis of audio media was used to test two hypotheses. Four archival oral interview recordings from Jeanna Isham’s podcast “Sound in Marketing” featuring the sonic branding experts …

Diagnosis Of Polypoidal Choroidal Vasculopathy From Fluorescein Angiography Using Deep Learning, Yu-Yeh Tsai, Wei-Yang Ling, Shih-Jen Chen, Paisan Ruamviboonsuk, Cheng-Ho King, Chia-Ling Tsai

Publications and Research

Purpose: To differentiate polypoidal choroidal vasculopathy (PCV) from choroidal neovascularization (CNV) and to determine the extent of PCV from fluorescein angiography (FA) using attention-based deep learning networks.

Methods: We build two deep learning networks for diagnosis of PCV using FA, one for detection and one for segmentation. Attention-gated convolutional neural network (AG-CNN) differentiates PCV from other types of wet age-related macular degeneration. Gradient-weighted class activation map (Grad-CAM) is generated to highlight important regions in the image for making the prediction, which offers explainability of the network. Attention-gated recurrent neural network (AG-PCVNet) for spatiotemporal prediction is applied for segmentation …

Identifying, Evaluating And Applying Importance Maps For Speech, Viet Anh Trinh

Dissertations, Theses, and Capstone Projects

Like many machine learning systems, speech models often perform well when employed on data in the same domain as their training data. However, when the inference is on out-of-domain data, performance suffers. With a fast-growing number of applications of speech models in healthcare, education, automotive, automation, etc., it is essential to ensure that speech models can generalize to out-of-domain data, especially to noisy environments in real-world scenarios. In contrast, human listeners are quite robust to noisy environments. Thus, a thorough understanding of the differences between human listeners and speech models is urgently required to enhance speech model performance in noise. …

Diagnosis Of Polypoidal Choroidal Vasculopathy From Fluorescein Angiography Using Deep Learning, Yu-Yeh Tsai, Wei-Yang Lin, Shih-Jen Chen, Paisan Ruamviboonsuk, Cheng-Ho King, Chia-Ling Tsai

Publications and Research

Purpose: To differentiate polypoidal choroidal vasculopathy (PCV) from choroidal neovascularization (CNV) and to determine the extent of PCV from fluorescein angiography (FA) using attention-based deep learning networks.

Methods: We build two deep learning networks for diagnosis of PCV using FA, one for detection and one for segmentation. Attention-gated convolutional neural network (AG-CNN) differentiates PCV from other types of wet age-related macular degeneration. Gradient-weighted class activation map (Grad-CAM) is generated to highlight important regions in the image for making the prediction, which offers explainability of the network. Attention-gated recurrent neural network (AG-PCVNet) for spatiotemporal prediction is applied for segmentation of PCV. …

Representation Learning For Chemical Activity Predictions, Mohamed S. Ayed

Dissertations, Theses, and Capstone Projects

Computational prediction of a phenotypic response upon the chemical perturbation on a biological system plays an important role in drug discovery and many other applications. Chemical fingerprints derived from chemical structures are a widely used feature to build machine learning models. However, the fingerprints ignore the biological context, thus, they suffer from several problems such as the activity cliff and curse of dimensionality. Fundamentally, the chemical modulation of biological activities is a multi-scale process. It is the genome-wide chemical-target interactions that modulate chemical phenotypic responses. Thus, the genome-scale chemical-target interaction profile will more directly correlate with in vitro and in …

Automatic Cephalometric Landmark Detection On X-Ray Images Using Object Detection, Cheng-Ho King, Yin-Lin Wang, Chia-Ling Tsai

Publications and Research

We propose a new deep convolutional cephalometric landmark detection framework for orthodontic treatment. Our proposed method consists of two major steps: landmark detection using a deep neural network for object detection, and landmark repair to ensure one instance per landmark class. For landmark detection, we modify the loss function of the backbone network YOLOv3 to eliminate the constrains on the bounding box and incorporate attention mechanism to improve the detection accuracy. For landmark repair, a triangle mesh is generated from the average face to eliminate superfluous instances, followed by estimation of missing landmarks from the detected ones using Laplacian Mesh. …

A Novel Tropical Geometry-Based Interpretable Machine Learning Method: Pilot Application To Delivery Of Advanced Heart Failure Therapies, Heming Yao, Harm Derkson, Jessica R. Golbus, Justin Zhang, Keith D. Aaronson, Jonathan Gryak, Kayvan Najarian

Publications and Research

Abstract—A model’s interpretability is essential to many practical applications such as clinical decision support systems. In this paper, a novel interpretable machine learning method is presented, which can model the relationship between input variables and responses in humanly understandable rules. The method is built by applying tropical geometry to fuzzy inference systems, wherein variable encoding functions and salient rules can be discovered by supervised learning. Experiments using synthetic datasets were conducted to demonstrate the performance and capacity of the proposed algorithm in classification and rule discovery. Furthermore, we present a pilot application in identifying heart failure patients that are eligible …

Behavioral Predictive Analytics Towards Personalization For Self-Management – A Use Case On Linking Health-Related Social Needs, Bon Sy, Michael Wassil, Helene Connelly, Alisha Hassan

Publications and Research

The objective of this research is to investigate the feasibility of applying behavioral predictive analytics to optimize patient engagement in diabetes self-management, and to gain insights on the potential of infusing a chatbot with NLP technology for discovering health-related social needs. In the U.S., less than 25% of patients actively engage in self-health management even though self-health management has been reported to associate with improved health outcomes and reduced healthcare costs. The proposed behavioral predictive analytics relies on manifold clustering to identify subpopulations segmented by behavior readiness characteristics that exhibit non-linear properties. For each subpopulation, an individualized auto-regression model and …

A Citizen-Science Approach For Urban Flood Risk Analysis Using Data Science And Machine Learning, Candace Agonafir

Dissertations and Theses

Street flooding is problematic in urban areas, where impervious surfaces, such as concrete, brick, and asphalt prevail, impeding the infiltration of water into the ground. During rain events, water ponds and rise to levels that cause considerable economic damage and physical harm. The main goal of this dissertation is to develop novel approaches toward the comprehension of urban flood risk using data science techniques on crowd-sourced data. This is accomplished by developing a series of data-driven models to identify flood factors of significance and localized areas of flood vulnerability in New York City (NYC). First, the infrastructural (catch basin clogs, …

Treatment Selection Using Prototyping In Latent-Space With Application To Depression Treatment, Akiva Kleinerman, Ariel Rosenfeld, David Benrimoh, Robert Fratila, Caitrin Armstrong, Joseph Mehltretter, Eliyahu Shneider, Amit Yaniv-Rosenfeld, Jordan Karp, Charles F. Reynolds, Gustavo Turecki, Adam Kapelner

Publications and Research

Machine-assisted treatment selection commonly follows one of two paradigms: a fully personalized paradigm which ignores any possible clustering of patients; or a sub-grouping paradigm which ignores personal differences within the identified groups. While both paradigms have shown promising results, each of them suffers from important limitations. In this article, we propose a novel deep learning-based treatment selection approach that is shown to strike a balance between the two paradigms using latent-space prototyping. Our approach is specifically tailored for domains in which effective prototypes and sub-groups of patients are assumed to exist, but groupings relevant to the training objective are not …

Piecewise Linear Manifold Clustering, Artyom Diky

Dissertations, Theses, and Capstone Projects

This work studies the application of topological analysis to non-linear manifold clustering. A novel method, that exploits the data clustering structure, allows to generate a topological representation of the point dataset. An analysis of topological construction under different simulated conditions is performed to explore the capabilities and limitations of the method, and demonstrated statistically significant improvements in performance. Furthermore, we introduce a new information-theoretical validation measure for clustering, that exploits geometrical properties of clusters to estimate clustering compressibility, for evaluation of the clustering goodness-of-fit without any prior information about true class assignments. We show how the new validation measure, when …

Solving Multiple Inference In Graphical Models, Cong Chen

Dissertations, Theses, and Capstone Projects

For inference problems in graphical models, much effort has been directed at algorithms for obtaining one single optimal prediction. In practice, the data is often noisy or incomplete, which makes one single optimal solution unreliable. To address this problem, multiple Inference is proposed to find several best solutions, M-Best, where multiple hypotheses are preferred for advanced reasoning. People use oracle accuracy as an evaluation criterion expecting one of the solutions has high accuracy with the ground truth. It has been shown that it is beneficial for the top solutions to be diverse. Approaches for solving diverse multiple inference are proposed …

Adversarial Training For Skill Learning In A Mobile Robot, Todd W. Flyr

Dissertations, Theses, and Capstone Projects

Machine Learning in mobile robotics is sometimes hampered by the difficulties associated with the creation of a large corpus of labeled data that most neural network based learning algorithms demand. In recent years, advances in the field of machine learning have been facilitated via the creation of large collaboratively-created labeled training datasets that researchers can use as the basis for experiments to validate and improve their candidate neural network architectures. For the field of robotics, however, tasks are so disparate and the physical devices so varied that in most cases the creation of collaborative benchmark datasets are impractical. Obtaining data …

Teaching Machine Learning For The Physical Sciences: A Summary Of Lessons Learned And Challenges, Viviana Acquaviva

Publications and Research

This paper summarizes some challenges encountered and best practices established in several years of teaching Machine Learning for the Physical Sciences at the undergraduate and graduate level. I discuss motivations for teaching ML to physicists, desirable properties of pedagogical materials, such as accessibility, relevance, and likeness to real-world research problems, and give examples of components of teaching units.

Learn Biologically Meaningful Representation With Transfer Learning, Di He

Dissertations, Theses, and Capstone Projects

Machine learning has made significant contributions to bioinformatics and computational biol­ogy. In particular, supervised learning approaches have been widely used in solving problems such as bio­marker identification, drug response prediction, and so on. However, because of the limited availability of comprehensively labeled and clean data, constructing predictive models in super­ vised settings is not always desirable or possible, especially when using data­hunger, red­hot learning paradigms such as deep learning methods. Hence, there are urgent needs to develop new approaches that could leverage more readily available unlabeled data in driving successful machine learning ap­ plications in this area.

In my dissertation, …

Metareasoning, Opportunistic Exploration, And Explanations For Autonomous Indoor Navigation, Raj Korpan

Dissertations, Theses, and Capstone Projects

Autonomous indoor navigation is an important task for mobile robots deployed without a map in real-world environments, such as museums or offices. While it travels, an autonomous robot navigator must contend with lack of prior knowledge, sensor noise, actuator error, and inquisitive people. This dissertation addresses these challenges with a cognitively-based hierarchical reasoning architecture that incorporates learning, exploration, reactivity, planning, heuristics, and explanations. Evaluation by simulation in large, complex, indoor environments shows that a robot controller can successfully navigate without a detailed map of every obstruction's location when it performs limited initial global exploration and plans in its learned spatial …