Graphics and Human Computer Interfaces Commons^™

Graph Foundation Models: Concepts, Opportunities And Challenges, Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Fang Yuan, Lichao Sun, Philip S. Yu, Chuan Shi

Research Collection School Of Computing and Information Systems

Foundation models have emerged as critical components in a variety of artificial intelligence applications, and showcase significant success in natural language processing and several other domains. Meanwhile, the field of graph machine learning is witnessing a paradigm transition from shallow methods to more sophisticated deep learning approaches. The capabilities of foundation models in generalization and adaptation motivate graph machine learning researchers to discuss the potential of developing a new graph learning paradigm. This paradigm envisions models that are pre-trained on extensive graph data and can be adapted for various graph tasks. Despite this burgeoning interest, there is a noticeable lack …

Occlusion-Insensitive Talking Head Video Generation Via Facelet Compensation, Yuhui Deng, Yuqin Lu, Yangyang Xu, Yongwei Nie, Shengfeng He

Research Collection School Of Computing and Information Systems

Talking head video generation involves animating a still face image using facial motion cues derived from a driving video to replicate target poses and expressions. Traditional methods often rely on the assumption that the relative positions of facial keypoints remain unchanged. However, this assumption fails when keypoints are occluded or when the head is in a profile pose, leading to inconsistencies in identity and blurring in certain facial regions. In this paper, we introduce Occlusion-Insensitive Talking Head Video Generation, a novel approach that eliminates the reliance on spatial correlation of keypoints and instead leverages semantic correlation. Our method transforms facial …

Improving Multimodal Human Pose Estimation By Adversarial Modality Enhancement, Jiangnan Xia, Qilong Wu, Yanyin Guo, Yi Li, Jianghan Cheng, Junwei Li, Zhiyuan Zhang

Research Collection School Of Computing and Information Systems

Human pose estimation in computer vision predominantly focuses on the visible modality, with limited research on the infrared modality. No existing methods demonstrate robust performance across both modalities, missing their complementary strengths. This gap arises from the lack of a multimodal benchmark and the difficulty of developing robust multimodal capabilities. To address this, we introduce MMPD, a novel visible-infrared multimodal pose benchmark with high-quality annotations for both modalities. Leveraging MMPD, we expose the limitations of state-of-the-art methods due to modality variance. To overcome this challenge, we propose a novel method-agnostic scheme called AMMPE. By employing the Modality Adversarial Enhancement Stage …

Imageinthat: Manipulating Images To Convey User Instructions To Robots, Karthik Mahadevan, Blaine Lewis, Jiannan Li, Bilge Mutlu, Anthony Tang, Tovi Grossman

Research Collection School Of Computing and Information Systems

Foundation models are rapidly improving the capability of robots in performing everyday tasks autonomously such as meal preparation, yet robots will still need to be instructed by humans due to model performance, the difficulty of capturing user preferences, and the need for user agency. Robots can be instructed using various methods---natural language conveys immediate instructions but can be abstract or ambiguous, whereas end-user programming supports longer-horizon tasks but interfaces face difficulties in capturing user intent. In this work, we propose using direct manipulation of images as an alternative paradigm to instruct robots, and introduce a specific instantiation called ImageInThat which …

Lightprof: A Lightweight Reasoning Framework For Large Language Model On Knowledge Graph, Tu Ao, Yanhua Yu, Yuling Wang, Yang Deng, Zirui Guo, Liang Pang, Pinghui Wang, Tat-Seng Chua, Xiao Zhang, Zhen Cai

Research Collection School Of Computing and Information Systems

Large Language Models (LLMs) have impressive capabilities in text understanding and zero-shot reasoning. However, delays in knowledge updates may cause them to reason incorrectly or produce harmful results. Knowledge Graphs (KGs) provide rich and reliable contextual information for the reasoning process of LLMs by structurally organizing and connecting a wide range of entities and relations. Existing KG-based LLM reasoning methods only inject KGs’ knowledge into prompts in a textual form, ignoring its structural information. Moreover, they mostly rely on close-source models or open-source models with large parameters, which poses challenges to high resource consumption. To address this, we propose a …

Adversarial Attacks On Event-Based Pedestrian Detectors: A Physical Approach, Guixu Lin, Muyao Niu, Qingtian Zhu, Zhengwei Yin, Zhuoxiao Li, Shengfeng He, Yinqiang Zheng

Research Collection School Of Computing and Information Systems

Event cameras, known for their low latency and high dynamic range, show great potential in pedestrian detection applications. However, while recent research has primarily focused on improving detection accuracy, the robustness of event-based visual models against physical adversarial attacks has received limited attention. For example, adversarial physical objects, such as specific clothing patterns or accessories, can exploit inherent vulnerabilities in these systems, leading to misdetections or misclassifications. This study is the first to explore physical adversarial attacks on event-driven pedestrian detectors, specifically investigating whether certain clothing patterns worn by pedestrians can cause these detectors to fail, effectively rendering them unable …

Promoting Digital Agriculture Adoption In Community-Based Agricultural Organizations, Jean Hardy, Abbey Palmer

Journal of Extension

Existing research and practice related to digital agriculture technology adoption is largely focused on large-scale producers. In this paper, we describe a case of adopting an advanced soil monitoring system in a community-based agricultural organization. We provide guidance for Extension professionals seeking to implement or promote digital agriculture technology adoption on: selecting appropriate technology, incorporating new technology into existing practices, harnessing local technology champions, and avoiding data-driven mission creep.

Seven Hci Grand Challenges Revisited: Five-Year Progress, Constantine Stephanidis, Gavriel Salvendy, Margherita Antona, Vincent G Duffy, Qin Gao, Waldemar Karwowski, Fiona Nah, Stavroula Ntoa, Pei-Luen Patrick Rau, Keng Siau, Jia Zhou

Research Collection School Of Computing and Information Systems

Motivated by the rapid technological advancements achieved in the last five years, and the pervasiveness of Artificial Intelligence, the paper investigates the evolving role of Human-Computer Interaction and revisits the seven grand challenges outlined in 2019: human-technology symbiosis, human-environment interactions, ethics, privacy and security, well-being, health and eudaimonia, accessibility and universal access, learning and creativity, and social organization and democracy. Through literature analysis, the paper reevaluates the status of each challenge and highlights emerging requirements. Key findings reveal the widespread impact of Artificial Intelligence across all domains and emphasize the need for improved AI transparency, alignment with human values, and …

Learning An Interpretable Stylized Subspace For 3d-Aware Animatable Artforms, Chenxi Zheng, Bangzhen Liu, Xuemiao Xu, Huaidong Zhang, Shengfeng He

Research Collection School Of Computing and Information Systems

Throughout history, static paintings have captivated viewers within display frames, yet the possibility of making these masterpieces vividly interactive remains intriguing. This research paper introduces 3DArtmator, a novel approach that aims to represent artforms in a highly interpretable stylized space, enabling 3D-aware animatable reconstruction and editing. Our rationale is to transfer the interpretability and 3D controllability of the latent space in a 3D-aware GAN to a stylized sub-space of a customized GAN, revitalizing the original artforms. To this end, the proposed two-stage optimization framework of 3DArtmator begins with discovering an anchor in the original latent space that accurately mimics the …

Density Boosts Everything: A One-Stop Strategy For Improving Performance, Robustness, And Sustainability Of Malware Detectors, Jianwen Tian, Wei Kong, Debin Gao, Tong Wang, Taotao Gu, Kefan Qiu, Zhi Wang, Xiaohui Kuang

Research Collection School Of Computing and Information Systems

In the contemporary landscape of cybersecurity, AI-driven detectors have emerged as pivotal in the realm of malware detection. However, existing AI-driven detectors encounter a myriad of challenges, including poisoning attacks, evasion attacks, and concept drift, which stem from the inherent characteristics of AI methodologies. While numerous solutions have been proposed to address these issues, they often concentrate on isolated problems, neglecting the broader implications for other facets of malware detection. This paper diverges from the conventional approach by not targeting a singular issue but instead identifying one of the fundamental causes of these challenges, sparsity. Sparsity refers to a scenario …

Heartdj - Music Recommendation And Generation Through Biofeedback From Heart Rate Variability, Egemen Şahin

Dartmouth College Master’s Theses

This study investigates the integration of real-time physiological data with AI-generated music to enhance emotional well-being, stress regulation, and focus, using Heart Rate Variability (HRV) as a biomarker of autonomic function. Conducted in two phases—Stable Audio Open (SAO) and Suno (SUNO)—the research evaluates biofeedback-driven music interventions across varying daily music-listening habits.

In the SAO phase, short AI-generated instrumental tracks were compared with Spotify recommendations and guided meditation. Modest HRV improvements were observed in biofeedback conditions, but participants noted emotional limitations, citing short track lengths and abrupt transitions.

The SUNO phase addressed these limitations with longer, more complex AI-generated compositions combined …

Display System Interface Using Visually-Evoked Cortical Potentials, Michael E. Miller, Brett J. Borghetti, Kellie D. Kennedy, Chad L. Stephens, Alan T. Pope

AFIT Patents

A brain-computer interface system includes a video processor for producing a display signal, a temporal controller for producing a plurality of repetitive visual stimulus (RVS) signals with different respective temporal aspects, a display device that receives the display signal and displays a corresponding image on a plurality of different display regions and receives the RVS signals and displays corresponding RVS in respective ones of the display regions, an electroencephalographic (EEG) sensor for sensing a visually-evoked cortical potential (VECP) signal in a user with eyes fixated on a viewed one of the display regions, and a VECP processor for processing the …

Dashar: An Implementation Of Augmented Reality Technology For Automotive Applications, Trevor D. Brown

Masters Theses & Specialist Projects

Since the advent of the modern automobile, manufacturers have provided means of tracking various critical data points associated with automobile operation, with the most prominent and standardized method being the instrument cluster. These data points include, but are not limited to, automobile speed, engine speed, fuel level, oil temperature, radiator (water) temperature, and battery charge. While this data is updated in real-time as the automobile is running, traditional instrument clusters cannot be modified or adjusted to the automobile driver’s needs, unless extensive after-market modifications are made. These modifications can be expensive, and require great understanding of the automobile’s assembly.

Alongside …

A Deep Reinforcement Learning Framework For Sequential Art Creation, Asmin Pothula

Computer Science and Engineering Theses - Archive

Most computational art systems rely on generative models that produce a complete artwork in a single pass, without capturing the gradual, decision-driven process through which human artists construct visual pieces. Prior research in sequential, stroke-based image generation, including differentiable neural painters and model-based reinforcement learning agents, has explored step-by-step creation, but these systems typically aim to reconstruct the input image within the same visual representation space, closely matching brushstrokes, textures, or colors to the target. In contrast, this thesis investigates sequential art creation in a different artistic representation, where the final artwork does not share the same visual form as …

Teaming With Technology: Adaptive Automation In Joint Cognitive Systems For Industry 5.0, Jessica Johnson

Virginia Digital Maritime Center (VDMC) Faculty Publications

Adaptive automation enables dynamic reallocation of functions between people and autonomous agents to improve performance in complex work. This paper presents a meta-analysis of experimental and quasi-experimental studies (2000-2025) on joint cognitive systems in industrially relevant contexts, quantifying effects on task performance, safety/failure management, workload, trust, and learning. Across studies, adaptive automation reliably reduces operator workload and shows moderate gains in task performance and safety, with healthier trust dynamics when adaptations are triggered by human-state or event cues, made transparent to the user, and remain rapidly overridable. Risks emerge when performance-triggered switching is opaque or poorly timed, which can erode …

Interaction Mechanism Between Health Anxiety And Information Seeking Behavior From The Perspective Of Phenomenology, Yanfeng Zhang, Minqian Yu

Journal of Scientific Information Research

[Purpose/significance]To analyze the evolution characteristics of health anxiety before and after information search behavior from the perspective of phenomenological graph analysis, and to explain the internal mechanism of the interaction between health anxiety and information search behavior. [Method/process]By using the phenomenological qualitative research method, the interactive mechanism between health anxiety and information search behavior was deeply explored. Based on the I-PACE theoretical model framework, the model elements of users' health anxiety and information search behavior were analyzed from the four dimensions of "Person-Affect-Cognition-Execution". To construct a mechanistic relationship model between health anxiety and information search behavior. [Result/conclusion]The research results revealed …

Optimizing Radial Interfaces For Eye-Movement Authentication On Smartphones, Trey V. Tuscai

Honors Theses

Radial authentication interfaces offer privacy-preserving, calibration-free eye-movement authentication. While their effectiveness has been demonstrated on large displays, their performance on smartphones remains underexplored. This study investigates seven radial interface configurations on the iPhone 13, varying the number of radial indicators and password lengths to examine trade-offs between accuracy, security, and entry time. Through a controlled eye-tracking experiment with 27 participants, we evaluate each configuration’s performance and collect user prioritizations of the three factors. Our findings reveal that shorter passwords with fewer indicators improve speed and accuracy but reduce security, while longer configurations enhance security at the cost of usability. Based …

Revealing Spatiotemporal Neural Activation Patterns In Electrocorticography Recordings Of Human Speech Production By Mutual Information, Julio Kovacs, Dean Krusienski, Minu Maninder, Willy Wriggers

Mechanical & Aerospace Engineering Faculty Publications

Background

Spatiotemporal mapping of neural activity during continuous speech production has been traditionally approached using correlation coefficient (CC) analysis between cortical signals and speech recordings. A prior study employed this approach using electrocorticography (ECoG) data from participants who underwent invasive intracranial monitoring for epilepsy. However, CC cannot detect nonlinear relationships and is dominated by the correspondence between periods of silence and of non-silence.

New Method

We introduce the mutual information (MI) measure, which can capture both linear and nonlinear dependencies. We validated CC and MI on the sub-second spatiotemporal brain activity recorded during continuous speech tasks. To refine the results, …

Gnnsynergy: A Multi-View Graph Neural Network For Predicting Anti-Cancer Drug Synergy, Zhifeng Hao, Jianming Zhan, Yuan Fang, Min Wu, Ruichu Cai

Research Collection School Of Computing and Information Systems

Drug combinations play very important roles in cancer therapy, as they can enhance curative efficacy and overcome drug resistance. Due to the increasing size of combinatorial space, experimental screening for all the drug combinations becomes infeasible in practice. Therefore, there is a great need to develop accurate computational approaches that can predict potential drug combinations to direct the experimental screening. In this paper, we propose a novel method called GNNSynergy to learn drug embeddings for drug synergy prediction. Given a specific cancer cell line, we propose a multi-view graph neural network framework which considers the current cell line as main …

Clinician Perspectives On Virtual Reality Use In Physical Therapy Practice In The United States, Danielle T. Felsberg, Jared T. Mcguirt, Scott E. Ross, Louisa D. Raisbeck, Charlend K. Howard, Christopher K. Rhea

Rehabilitation Sciences Faculty Publications

The primary goal of physical rehabilitation is to assess movement impairments and restore function to improve overall quality of life. Virtual reality (VR) may provide the optimal environment to promote these goals due to its motivating and modifiable nature which can be difficult to accomplish through traditional real-world therapeutic methods. Current research of VR for rehabilitation has demonstrated that VR interventions can produce clinically meaningful change in motor outcomes. Despite this, adoption and usage of VR by physical therapy professionals is unclear due to the limited research in this area. Thus, the purpose of this study was to identify the …

Automatic Scoring Cornhole Board System, Eric Diffendal, Jonah Harsh, Connor Lengel, Brett Sukie

Williams Honors College, Honors Research Projects

The objective is to create a self-scoring cornhole board that can detect and calculate each team's score based on the bags thrown each round and to be created at a low cost/eventually being sold at the current cost of a normal board. When playing cornhole, the game is simple: throw a bag on the board; however, the scores are variable (deduct and add) across each round. The most common issue when playing cornhole is miscalculations of the scores and forgetting the correct scores. Thus, this invention will make gameplay easy for all to play.

User Interface For Custom Car Infotainment Systems, Dylan Miller

Williams Honors College, Honors Research Projects

The infotainment system is often considered one of the most functional and luxurious aspects of modern cars, containing functions that are useful to drivers in ways that range from convenient to safety-enhancing. However, modern infotainment systems can have some drawbacks such as making it more difficult to repair the vehicles they are in, helping to artificially limit the lifespan of the vehicles they are in, and not being present in most vehicles more than 15 years old. The software described in this paper, OpenQarUI, seeks to be a part of a solution to these problems. It is a piece of …

Procedural Terrain Generation: Noise Functions, Modern Methods, And Style Transfer, Hunter Barton

EWU Masters Thesis Collection

Procedural terrain generation, the algorithmic creation of digital terrain, finds use in multiple types of digital media. As the capabilities of modern computation increase, the ability to create more and more realistic terrains fully procedurally at scale improves. Modern methods of procedural generation have also overlapped with these advances, most notably advances in hardware. To account for this, a survey was done of modern methods for procedural terrain generation. Smooth procedural noise functions are one of the backbones of procedural terrain generation. Perlin noise, value noise, and fractal noise were explored in-depth. These noise functions were also tested for capabilities …

Flowing Together Or Alone: Impact Of Collaboration In The Metaverse, Fiona Fui-Hoon Nah, Brenda Eschenbrenner, Langtao Chen

Research Collection School Of Computing and Information Systems

The metaverse is the next-generation Internet (Web3) that facilitates social connections and collaborations in a virtual world environment. Given the potential of the metaverse to provide more satisfying and effective means of remote collaborations, exploring the possibility of leveraging the metaverse for these endeavors is warranted. Therefore, an important question to address is whether greater engagement occurs when tasks are completed collaboratively versus individually in the metaverse. We address this question by drawing on flow and transportation theories to hypothesize the effect of carrying out a creative task in the metaverse collaboratively versus alone on one's cognitive absorption, a contextually …

Recdreamer: Consistent Text-To-3d Generation Via Uniform Score Distillation, Chenxi Zheng, Yihong Lin, Bangzhen Liu, Xuemiao Xu, Yongwei Nie, Shengfeng He

Research Collection School Of Computing and Information Systems

Current text-to-3D generation methods based on score distillation often suffer from geometric inconsistencies, leading to repeated patterns across different poses of 3D assets. This issue, known as the Multi-Face Janus problem, arises because existing methods struggle to maintain consistency across varying poses and are biased toward a canonical pose. While recent work has improved pose control and approximation, these efforts are still limited by this inherent bias, which skews the guidance during generation. To address this, we propose a solution called RecDreamer, which reshapes the underlying data distribution to achieve more consistent pose representation. The core idea behind our method …

Weakly-Supervised Semantic Segmentation With Image-Level Labels: From Traditional Models To Foundation Models, Zhaozheng Chen, Qianru Sun

Research Collection School Of Computing and Information Systems

The rapid development of deep learning has driven significant progress in image semantic segmentation—a fundamental task in computer vision. Semantic segmentation algorithms often depend on the availability of pixel-level labels (i.e., masks of objects), which are expensive, time consuming, and labor intensive. Weakly supervised semantic segmentation (WSSS) is an effective solution to avoid such labeling. It utilizes only partial or incomplete annotations and provides a cost-effective alternative to fully supervised semantic segmentation. In this article, our focus is on the WSSS with image-level labels, which is the most challenging form of WSSS. Our work has two parts. First, we conduct …

Synthesizing Multi-Person And Rare Pose Images For Human Pose Estimation, Liuqing Zhao, Zichen Tian, Zou Peng, Richang Hong, Qianru Sun

Research Collection School Of Computing and Information Systems

Human pose estimation (HPE) models underperform in recognizing rare poses because they suffer from data imbalance problems (i.e., there are few image samples for rare poses) in their training datasets. From a data perspective, the most intuitive solution is to synthesize data for rare poses. Specifically, the rule-based methods apply manual manipulations (such as Cutout and GridMask) to the existing data, so the limited diversity of the data constrains the model. An alternative method is to learn the underlying data distribution via deep generative models (such as ControlNet and HumanSD) and then sample “new data” from the distribution. This works …

Ultrasonic Sensor-Based Sound Synthesis Using Raspberry Pi Pico W, Niraj Jaishwal

Williams Honors College, Honors Research Projects

At the intersection of Human Computer Interaction and digital art, this project transforms simple motion into musical expression. It explores an interactive real-time sound synthesis system using ultrasonic sensors to generate continuous audio. The objective is to design a system that maps physical distances into musical parameters such as pitch and amplitude, which will create a responsive audio environment. Two ultrasonic sensors are used in combination with the Raspberry Pi Pico W microcontroller running CircuitPython and Adafruit Audio Hat for real-time sound output. One sensor controls the pitch of the generated tone, while the other controls volume. This enables expressive …

Personalized Physics Learning Through Ai: Insights From Problem Generation, Chatbot Dialogues, And Intelligent Tutoring Systems, Atharva Dange

Physics Dissertations - Archive

Artificial intelligence (AI) is poised to transform science education, yet questions remain on how best to integrate these technologies into teaching and learning. This dissertation investigates the use of AI-driven tools in university physics courses through three complementary studies. In the first study, a generative language model (ChatGPT) was used to create novel physics homework problems aligned with course objectives. Analysis showed that, after expert vetting, AI-generated questions can foster higher-order problem-solving and reduce student reliance on solution memorization, though careful instructor oversight is required to ensure accuracy. The second study embedded an AI chatbot as a learning aid in …

Polarimetric Capture And Differentiable Rendering, Katherine Anne Salesin

Dartmouth College Ph.D Dissertations

Many scientific fields rely on the capture and modeling of light to extract underlying information about the world. Often, more information can be extracted by capturing more about the nature of the light, such as its spectral shape or polarization state. While polarization is a relatively unexplored topic in computer graphics, when used in tandem with other recent advancements in the field it has enormous potential to improve both forward and inverse models in other scientific disciplines. We demonstrate this potential in two distinct settings in this thesis.

First, we apply the capture of polarized light to an inverse problem …