Graphics and Human Computer Interfaces Commons^™

Occlusion-Insensitive Talking Head Video Generation Via Facelet Compensation, Yuhui Deng, Yuqin Lu, Yangyang Xu, Yongwei Nie, Shengfeng He

Research Collection School Of Computing and Information Systems

Talking head video generation involves animating a still face image using facial motion cues derived from a driving video to replicate target poses and expressions. Traditional methods often rely on the assumption that the relative positions of facial keypoints remain unchanged. However, this assumption fails when keypoints are occluded or when the head is in a profile pose, leading to inconsistencies in identity and blurring in certain facial regions. In this paper, we introduce Occlusion-Insensitive Talking Head Video Generation, a novel approach that eliminates the reliance on spatial correlation of keypoints and instead leverages semantic correlation. Our method transforms facial …

Seven Hci Grand Challenges Revisited: Five-Year Progress, Constantine Stephanidis, Gavriel Salvendy, Margherita Antona, Vincent G Duffy, Qin Gao, Waldemar Karwowski, Fiona Nah, Stavroula Ntoa, Pei-Luen Patrick Rau, Keng Siau, Jia Zhou

Research Collection School Of Computing and Information Systems

Motivated by the rapid technological advancements achieved in the last five years, and the pervasiveness of Artificial Intelligence, the paper investigates the evolving role of Human-Computer Interaction and revisits the seven grand challenges outlined in 2019: human-technology symbiosis, human-environment interactions, ethics, privacy and security, well-being, health and eudaimonia, accessibility and universal access, learning and creativity, and social organization and democracy. Through literature analysis, the paper reevaluates the status of each challenge and highlights emerging requirements. Key findings reveal the widespread impact of Artificial Intelligence across all domains and emphasize the need for improved AI transparency, alignment with human values, and …

Heartdj - Music Recommendation And Generation Through Biofeedback From Heart Rate Variability, Egemen Şahin

Dartmouth College Master’s Theses

This study investigates the integration of real-time physiological data with AI-generated music to enhance emotional well-being, stress regulation, and focus, using Heart Rate Variability (HRV) as a biomarker of autonomic function. Conducted in two phases—Stable Audio Open (SAO) and Suno (SUNO)—the research evaluates biofeedback-driven music interventions across varying daily music-listening habits.

In the SAO phase, short AI-generated instrumental tracks were compared with Spotify recommendations and guided meditation. Modest HRV improvements were observed in biofeedback conditions, but participants noted emotional limitations, citing short track lengths and abrupt transitions.

The SUNO phase addressed these limitations with longer, more complex AI-generated compositions combined …

A Deep Reinforcement Learning Framework For Sequential Art Creation, Asmin Pothula

Computer Science and Engineering Theses - Archive

Most computational art systems rely on generative models that produce a complete artwork in a single pass, without capturing the gradual, decision-driven process through which human artists construct visual pieces. Prior research in sequential, stroke-based image generation, including differentiable neural painters and model-based reinforcement learning agents, has explored step-by-step creation, but these systems typically aim to reconstruct the input image within the same visual representation space, closely matching brushstrokes, textures, or colors to the target. In contrast, this thesis investigates sequential art creation in a different artistic representation, where the final artwork does not share the same visual form as …

Openmuse: Integrating Open-Source Models Into Music Creation Workflows, Tyler K. Vergho

Dartmouth College Master’s Theses

This master's thesis introduces OpenMUSE (Open Multimodal Unified Sound Engine), a platform that demonstrates the potential of open-source AI music generation by integrating state-of-the-art deep learning models into a unified system. By unifying ten different open-source models, including MusicGen, AudioLDM2, and custom-trained text-to-symbolic music generation models, OpenMUSE aims to create a user-friendly interface that empowers artists to produce complex, adaptive musical compositions. The system enhances accessibility by providing a simple web interface and natural language controls, while improving controllability through features like melody conditioning and semantic audio editing. Specifically, OpenMUSE offers a digital audio workstation (DAW)-inspired interface that lowers the …

Generalizing Classification Of Pilot Workload: Transfer Learning Versus A Jepa-Inspired Transformer Architecture, Naim Barnett, Shivani Nagrecha, Morgan Glover, Clayton Harper, Justin Wilson, James Maher, Eric C. Larson

International Journal of Aviation, Aeronautics, and Aerospace

Within the context of learning, there poses difficulty when objectively measuring human performance. In this work, we investigate the evaluation of human performance via its relation to the individual's mental capacity by classification of cognitive load within the domain of aviation. By utilizing a mixed virtual and physical flight simulation environment in conjunction with biometric sensing, we create and evaluate the predictive capabilities of a Joint-Embedding Predictive Architecture (JEPA) and compare the architecture and results to traditional methods for transfer learning and domain adaptation. We find that our JEPA inspired architecture can achieve more than 70% accuracy of cognitive workload, …

Weakly-Supervised Semantic Segmentation With Image-Level Labels: From Traditional Models To Foundation Models, Zhaozheng Chen, Qianru Sun

Research Collection School Of Computing and Information Systems

The rapid development of deep learning has driven significant progress in image semantic segmentation—a fundamental task in computer vision. Semantic segmentation algorithms often depend on the availability of pixel-level labels (i.e., masks of objects), which are expensive, time consuming, and labor intensive. Weakly supervised semantic segmentation (WSSS) is an effective solution to avoid such labeling. It utilizes only partial or incomplete annotations and provides a cost-effective alternative to fully supervised semantic segmentation. In this article, our focus is on the WSSS with image-level labels, which is the most challenging form of WSSS. Our work has two parts. First, we conduct …

Nexus: Network Exploration For Exploiting Unsafe Sequences In Multi-Turn Llm Jailbreaks, Javad Rafiei Asl, Sidhant Narula, Mohammad Ghasemigol, Eduardo Blanco, Daniel Takabi

School of Cybersecurity Faculty Publications

Large Language Models (LLMs) have revolutionized natural language processing, yet remain vulnerable to jailbreak attacks—particularly multi-turn jailbreaks that distribute malicious intent across benign exchanges, thereby bypassing alignment mechanisms. Existing approaches often suffer from limited exploration of the adversarial space, rely on hand-crafted heuristics, or lack systematic query refinement. We propose NEXUS (Network Exploration for eXploiting Unsafe Sequences), a modular framework for constructing, refining, and executing optimized multi-turn attacks. NEXUS comprises: (1) ThoughtNet, which hierarchically expands a harmful intent into a structured semantic network of topics, entities, and query chains; (2) a feedback-driven Simulator that iteratively refines and prunes these chains …

Flexible Hybrid Self-Powered Piezo-Triboelectric Nanogenerator Based On Bto-Pvdf/Pdms Nanocomposites For Human Machine Interaction, Wentao Dong, Mengyun Li, Chang Chen, Kun Xie, Jinhua Hong, Lin Yang

Civil & Environmental Engineering Faculty Publications

As flexible and wearable electronics play more and more important role in smart watches, smart glass and virtual reality, and the power supply to the wearable electronics have been revealed more attentions for long-term usage and continuous healthy monitoring. To overcome the challenge, flexible self-powered BTO-PVDF/PDMS piezoelectric-triboelectric electric hybrid generators (BPP-HNG) are developed to human gesture monitoring and human machine interaction (HMI) application without external power supply. BPP-HNG based on BTO-PVDF and PDMS films are prepared by sol-gel and spin-coating method. When the BTO content is 20 wt.%, BPP-HNG exhibits better electrical performance with an output voltage of 20.51 V. …

Automatic Scoring Cornhole Board System, Eric Diffendal, Jonah Harsh, Connor Lengel, Brett Sukie

Williams Honors College, Honors Research Projects

The objective is to create a self-scoring cornhole board that can detect and calculate each team's score based on the bags thrown each round and to be created at a low cost/eventually being sold at the current cost of a normal board. When playing cornhole, the game is simple: throw a bag on the board; however, the scores are variable (deduct and add) across each round. The most common issue when playing cornhole is miscalculations of the scores and forgetting the correct scores. Thus, this invention will make gameplay easy for all to play.

Personalized Physics Learning Through Ai: Insights From Problem Generation, Chatbot Dialogues, And Intelligent Tutoring Systems, Atharva Dange

Physics Dissertations - Archive

Artificial intelligence (AI) is poised to transform science education, yet questions remain on how best to integrate these technologies into teaching and learning. This dissertation investigates the use of AI-driven tools in university physics courses through three complementary studies. In the first study, a generative language model (ChatGPT) was used to create novel physics homework problems aligned with course objectives. Analysis showed that, after expert vetting, AI-generated questions can foster higher-order problem-solving and reduce student reliance on solution memorization, though careful instructor oversight is required to ensure accuracy. The second study embedded an AI chatbot as a learning aid in …

Recdreamer: Consistent Text-To-3d Generation Via Uniform Score Distillation, Chenxi Zheng, Yihong Lin, Bangzhen Liu, Xuemiao Xu, Yongwei Nie, Shengfeng He

Research Collection School Of Computing and Information Systems

Current text-to-3D generation methods based on score distillation often suffer from geometric inconsistencies, leading to repeated patterns across different poses of 3D assets. This issue, known as the Multi-Face Janus problem, arises because existing methods struggle to maintain consistency across varying poses and are biased toward a canonical pose. While recent work has improved pose control and approximation, these efforts are still limited by this inherent bias, which skews the guidance during generation. To address this, we propose a solution called RecDreamer, which reshapes the underlying data distribution to achieve more consistent pose representation. The core idea behind our method …

Synthesizing Multi-Person And Rare Pose Images For Human Pose Estimation, Liuqing Zhao, Zichen Tian, Zou Peng, Richang Hong, Qianru Sun

Research Collection School Of Computing and Information Systems

Human pose estimation (HPE) models underperform in recognizing rare poses because they suffer from data imbalance problems (i.e., there are few image samples for rare poses) in their training datasets. From a data perspective, the most intuitive solution is to synthesize data for rare poses. Specifically, the rule-based methods apply manual manipulations (such as Cutout and GridMask) to the existing data, so the limited diversity of the data constrains the model. An alternative method is to learn the underlying data distribution via deep generative models (such as ControlNet and HumanSD) and then sample “new data” from the distribution. This works …

Human Perception Of Ai Capabilities At Classifying Perturbed Roadway Signs, Katherine R. Garcia, Jing Chen, Yanru Xiao, Scott Mishler, Cong Wang, Bin Hu

Computer Science Faculty Publications

Artificial Intelligence (AI) is crucial to numerous functions required for driving automation systems, including the computer vision techniques used to detect the roadway environment and make real-time decisions. However, the images used as inputs to the AI system may be maliciously perturbed, or manipulated, causing the AI system to make an incorrect classification. In this study, we examined humans’ perception of the AI’s computer vision capability of classifying various road sign images, including the original images, images with two different types of malicious attacks, and images that are scrambled randomly at the pixel level. Our results showed that participants rated …

Optiselect And Enshap: Integrating Machine Learning And Game Theory For Ischemic Stroke Prediction, Pritam Chakraborty, Anjan Bandyopadhyay, Sricheta Parul, Sujata Swain, Partha Sarathy Banerjee, Tapas Si, Hong Qin, Saurav Mallik

Computer Science Faculty Publications

Stroke analysis using game theory and machine learning techniques. The study investigates the use of the Shapley value in predictive ischemic brain stroke analysis. Initially, preference algorithms identify the most important features in various machine learning models, including logistic regression, K-nearest neighbor, decision tree, support vector machine (linear kernel), support vector machine ( RBF kernel), neural networks, etc. For each sample, the top 3, 4, and 5 features are evaluated and selected to evaluate their performance. The Shapley value method was used to rank the models using their best four features based on their predictive capabilities. As a result, better-performing …

Adapting Online Customer Reviews For Blind Users: A Case Study Of Restaurant Reviews, Mohan Sunkara, Akshay Kolgar Nayak, Sandeep Kalari, Yash Prakash, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Online reviews have become an integral aspect of consumer decision-making on e-commerce websites, especially in the restaurant industry. Unlike sighted users who can visually skim through the reviews, perusing reviews remains challenging for blind users, who rely on screen reader assistive technology that supports predominantly one-dimensional narration of content via keyboard shortcuts. In an interview study, we uncovered numerous pain points of blind screen reader users with online restaurant reviews, notably, the listening fatigue and frustration after going through only the first few reviews. To address these issues, we developed QuickCue assistive tool that performs aspect-focused sentiment-driven summarization to reorganize …

Surveying The Role Of Visual Analytics In Human-Machine Teaming, Naga Datha Saikiran Battula

Theses

Humans and machines both possess their unique capabilities and have their strengths and weaknesses, which can be complementary to one another and allow them to achieve a common goal. Teaming in the modern era involves text prompts, voice commands, gesture recognition, touch interfaces, and the latest visualization techniques that allow parties/agents to interact. Communication through visualization plays a vital role in allowing robust insights to be gained through a glance. Using visualization as a medium between humans and machines can increase the communication bandwidth. Human-machine teaming has witnessed much progress, with many theories and practical examples emerging. In the report, …

La Creatividad En Peligro: Como La Inteligencia Artificial Es Un Reto Para Los Artistas., Nathaly Cisneros

Capstones

Los artistas digitales han creado obras maestras que nos han dejado sin aliento con sus pinceles digitales, lápices y pinturas. Desde retratos que parecen saltar de la pantalla hasta paisajes que nos transportan a mundos desconocidos, su arte ha sido una fuente constante de inspiración.

Pero en los últimos años, una nueva fuerza ha comenzado a cambiar el juego. La inteligencia artificial ha estado avanzando a pasos agigantados y ahora se perfila como una amenaza para el futuro de los artistas digitales. ¿Qué significa esto para el arte y la creatividad?

Link: https://docs.google.com/document/d/1xe8UxDMekX_SwiIppyt_JppK8M-lB-YWNWGyeyShlJM/edit?usp=sharing

Real-Time Feedback-Driven Framework For Automated Cybersickness Mitigation, Md Jahirul Islam

Master's Theses

As technologies are becoming more advanced day by day, the embracement of virtual reality (VR) technology among users is also increasing in daily activities for various purposes, and subsequently, the barrier between the real and virtual world is fading. Despite the versatile uses, cybersickness (CS) is a major problem which is induced among users due to the immersive VR experience. There is a plethora of research findings and methods to measure the users’ CS such as virtual reality sickness questionnaire (VRSQ), simulator sickness questionnaire (SSQ), fast motion scale questionnaire (FMS), and others. Recently, machine learning approaches have also been adopted …

On The Benefits Of Directness In Virtual Characters For Motivational Interviews, Michael O'Mahony, Cathy Ennis, Robert Ross

Conference papers

Understanding the factors influencing successful engagement with Embodied Conversational Agents (ECAs) remains a significant challenge. This understanding could be used to personalise agents to users to improve interactions. Some studies have shown that simulating personalities in healthcare agents can improve effectiveness and engagement. However, it is not yet well understood how variations of agent personality can be leveraged to improve user engagement with Motivational Interviewing (MI) ECAs. Specifically how the balance between agent warmth and directness can be controlled in an MI agent to improve likeability and engagement. We conducted an online Wizard-of-Oz (WoZ) mediated study of two variants of …

Using Llms To Establish Implicit User Sentiment Of Software Desirability, Sherri Weitl-Harms, John D. Hastings, Jonah Lum

Research & Publications

This study explores the use of LLMs for providing quantitative zero-shot sentiment analysis of implicit software desirability, addressing a critical challenge in product evaluation where traditional review scores, though convenient, fail to capture the richness of qualitative user feedback. Innovations include establishing a method that 1) works with qualitative user experience data without the need for explicit review scores, 2) focuses on implicit user satisfaction, and 3) provides scaled numerical sentiment analysis, offering a more nuanced understanding of user sentiment, instead of simply classifying sentiment as positive, neutral, or negative.

Data is collected using the Microsoft Product Desirability Toolkit (PDT), …

Mvgamba : Unify 3d Content Generation As State Space Sequence Modeling, Xuanyu Yi, Zike Wu, Qiuhong Shen, Qingshan Xu, Pan Zhou, Joo-Hwee Lim, Shuicheng Yan, Xinchao Wang, Hanwang Zhang

Research Collection School Of Computing and Information Systems

Recent 3D large reconstruction models (LRMs) can generate high-quality 3D content in sub-seconds by integrating multi-view diffusion models with scalable multi-view reconstructors. Current works further leverage 3D Gaussian Splatting as 3D representation for improved visual quality and rendering efficiency. However, we observe that existing Gaussian reconstruction models often suffer from multi-view inconsistency and blurred textures. We attribute this to the compromise of multi-view information propagation in favor of adopting powerful yet computationally intensive architectures (e.g., Transformers). To address this issue, we introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor based on the RNN-like State …

3d Snapshot: Invertible Embedding Of 3d Neural Representations In A Single Image, Yuqin Lu, Bailin Deng, Zhixuan Zhong, Tianle Zhang, Yuhui Quan, Hongmin Cai, Shengfeng He

Research Collection School Of Computing and Information Systems

3D neural rendering enables photo-realistic reconstruction of a specific scene by encoding discontinuous inputs into a neural representation. Despite the remarkable rendering results, the storage of network parameters is not transmission-friendly and not extendable to metaverse applications. In this paper, we propose an invertible neural rendering approach that enables generating an interactive 3D model from a single image (i.e., 3D Snapshot). Our idea is to distill a pre-trained neural rendering model (e.g., NeRF) into a visualizable image form that can then be easily inverted back to a neural network. To this end, we first present a neural image distillation method …

User Acceptance Of Advice By Ai Agents: Expectation-System Fit Perspective, Jingyuan Cai, Fiona Fui-Hoon Nah

Research Collection School Of Computing and Information Systems

Algorithms have increasing influence on our daily decisions, especially when the recommendations are presented by human-like AI agents. This study applies the Theory of Effective Use to investigate how the fit between the user’s role expectation for an AI agent and the agent’s interaction style impacts AI advice adoption. We proposed a new concept termed Perceived Expectation-System Fit (PESF) and empirically examined its impact on user perceptions and advice acceptance. We found that low PESF reduces advice acceptance by diminishing cognitive and affective trust in the AI agent. Furthermore, increased algorithm transparency increases PESF's impact on decision-making. Our findings provide …

Habit Coach: Customising Rag-Based Chatbots To Support Behavior Change, Arian Fooroogh Mand Arabi, Cansu Koyuturk, Michael O'Mahony, Raffaella Calati, Dimitri Ognibene

Conference papers

This paper presents the iterative development of Habit Coach, a GPT-based chatbot designed to support users in habit change through personalized interaction. Employing a user-centered design approach, we developed the chatbot using a Retrieval-Augmented Generation (RAG) system, which enables behavior personalization without retraining the underlying language model (GPT-4). The system leverages document retrieval and specialized prompts to tailor interactions, drawing from Cognitive Behavioral Therapy (CBT) and narrative therapy techniques. A key challenge in the development process was the difficulty of translating declarative knowledge into effective interaction behaviors. In the initial phase, the chatbot was provided with declarative knowledge about CBT …

Cirp: Cross‑Item Relational Pre‑Training For Multimodal Product Bundling, Yunshan Ma, Yingzhi He, Wenjun Zhong, Xiang Wang, Roger Zimmermann, Tat-Seng Chua

Research Collection School Of Computing and Information Systems

Product bundling has been a prevailing marketing strategy that is beneficial in the online shopping scenario. Effective product bundling methods depend on high-quality item representations capturing both the individual items' semantics and cross-item relations. However, previous item representation learning methods, either feature fusion or graph learning, suffer from inadequate cross-modal alignment and struggle to capture the cross-item relations for cold-start items. Multimodal pre-train models could be the potential solutions given their promising performance on various multimodal downstream tasks. However, the cross-item relations have been under-explored in the current multimodal pre-train models.To bridge this gap, we propose a novel and simple …

The Psychological Impacts Of Algorithmic And Ai-Driven Social Media On Teenagers: A Call To Action, Sunil Arora, Sahil Arora, John Hastings

Research & Publications

This study investigates the meta-issues surrounding social media, which, while theoretically designed to enhance social interactions and improve our social lives by facilitating the sharing of personal experiences and life events, often results in adverse psychological impacts. Our investigation reveals a paradoxical outcome: rather than fostering closer relationships and improving social lives, the algorithms and structures that underlie social media platforms inadvertently contribute to a profound psychological impact on individuals, influencing them in unforeseen ways. This phenomenon is particularly pronounced among teenagers, who are disproportionately affected by curated online personas, peer pressure to present a perfect digital image, and the …

Digital Twin For Shelf Intelligence: Ai-Driven Inventory Management For Minimizing Food Waste, Charlotte Maples, Marvin Velazquez

College of Engineering Summer Undergraduate Research Program

This project aims to develop a solution for improving grocery store inventory management by leveraging AI-driven image recognition. Traditional inventory methods, which rely on manual counting or barcode scanning, are inefficient, labor-intensive, and prone to human error. Over an 8-week period, we designed and developed a basic iPad app capable of identifying specific types of fruit and automatically updating inventory records in real time. By utilizing the iPad’s camera and machine learning algorithms, the app demonstrates the potential to streamline inventory tracking, reduce manual labor, and improve accuracy in managing perishable goods. Future work will focus on expanding the app’s …

Enhancing Place-Based Interaction With Emotion Ai And Augmented Reality, Jake Maier, Ivan Martinez

College of Engineering Summer Undergraduate Research Program

This project explores the integration of augmented reality (AR) and Emotion AI technologies to enhance user experiences in physical environments. By seamlessly merging virtual elements with real-world contexts, we aim to deepen individuals’ interactions and perceptions of their surroundings. Leveraging AR technology enables users to access contextual information, engage with interactive content, and navigate spaces with heightened immersion and understanding. Additionally, Emotion AI enhances these experiences by detecting and responding to users’ emotional states, fostering personalized and emotionally resonant interactions. We aim to integrate digital content within physical environments using mixed-reality headsets equipped with eye-tracking capabilities and consumer-grade wireless EEG …

Leveraging Tradespace-Exploration For A Senior Project Team Formation Application, Miguel Saenz

College of Engineering Summer Undergraduate Research Program

This project revolves around the development of an app in MATLAB that leverages the VASSAR rule-based system and a genetic algorithm to form groups of teams for the Mechanical Engineering Senior Design project class. We leveraged the iterative design process to eventually attain a functional app with a reasonable runtime that works provided correctly formatted rulesheets describing student project preference and member preference.