Graphics and Human Computer Interfaces Commons^™

Generalizing Classification Of Pilot Workload: Transfer Learning Versus A Jepa-Inspired Transformer Architecture, Naim Barnett, Shivani Nagrecha, Morgan Glover, Clayton Harper, Justin Wilson, James Maher, Eric C. Larson

International Journal of Aviation, Aeronautics, and Aerospace

Within the context of learning, there poses difficulty when objectively measuring human performance. In this work, we investigate the evaluation of human performance via its relation to the individual's mental capacity by classification of cognitive load within the domain of aviation. By utilizing a mixed virtual and physical flight simulation environment in conjunction with biometric sensing, we create and evaluate the predictive capabilities of a Joint-Embedding Predictive Architecture (JEPA) and compare the architecture and results to traditional methods for transfer learning and domain adaptation. We find that our JEPA inspired architecture can achieve more than 70% accuracy of cognitive workload, …

Robust Text Input For Smartwatches: Compensating For Imprecise Tapping And Swiping, Jianwei Lai, Lina Zhou, Kanlun Wang, Dongsong Zhang

Faculty Publications - Information Technology

Entering text on a smartwatch is challenging due to the difficulty of tapping tiny keys. This study introduces a novel keyboard, Tap’nSwipe, to address the challenge. The keyboard features nine areas, each containing up to four characters. To enter a character, users swipe in a specific direction within the area containing the character, freeing them from precisely tapping on the target key. In addition, Tap’nSwipe leverages word predictions to enter words by allowing users to tap anywhere in the areas containing the target characters. The results of a user experiment show that Tap’nSwipe improves text entry accuracy and reduces error …

Automation To Autonomy: Temporal Dynamics Of Trust And Visual Attention Allocation Did Not Evolve, Tetsuya Sato, Eric Chancey, Yusuke Yamani

Psychology Faculty Publications

Emerging work environments are expected to implement autonomy that performs various functions without human input. Previous works has shown that trust in automation is negatively correlated with visual attention allocation, indicating that trust is a dynamic construct. Moreover, trust in automation and trust in autonomy appears to evolve in similar ways. However, recent work has demonstrated differences between trust in automation and trust in autonomy within Kaber’s (2018) theoretical framework (Sato et al., 2023b). Yet, it is uncertain whether the development of trust and visual attention allocation differs between automation and autonomy. The present study examined the temporal dynamics of …

The Interacting Roles Of Attention Allocation And Trust In Highly Automated Aam Environments, Yusuke Yamani

Psychology Faculty Publications

[First slide]

Mechanisms of attentive visual processing

• Attention control
• Visual search
• Eye movement
• Aging and individual differences

Limits of human performance in applied environment

• Complex displays
• Machine operation
• Surface transportation
• Advanced air mobility
• Nuclear operation

Methods to ameliorate human cognitive performance

• Human-machine interface
• Human autonomy/AI teaming
• Human-systems integration
• Training

Flexible Hybrid Self-Powered Piezo-Triboelectric Nanogenerator Based On Bto-Pvdf/Pdms Nanocomposites For Human Machine Interaction, Wentao Dong, Mengyun Li, Chang Chen, Kun Xie, Jinhua Hong, Lin Yang

Civil & Environmental Engineering Faculty Publications

As flexible and wearable electronics play more and more important role in smart watches, smart glass and virtual reality, and the power supply to the wearable electronics have been revealed more attentions for long-term usage and continuous healthy monitoring. To overcome the challenge, flexible self-powered BTO-PVDF/PDMS piezoelectric-triboelectric electric hybrid generators (BPP-HNG) are developed to human gesture monitoring and human machine interaction (HMI) application without external power supply. BPP-HNG based on BTO-PVDF and PDMS films are prepared by sol-gel and spin-coating method. When the BTO content is 20 wt.%, BPP-HNG exhibits better electrical performance with an output voltage of 20.51 V. …

Nexus: Network Exploration For Exploiting Unsafe Sequences In Multi-Turn Llm Jailbreaks, Javad Rafiei Asl, Sidhant Narula, Mohammad Ghasemigol, Eduardo Blanco, Daniel Takabi

School of Cybersecurity Faculty Publications

Large Language Models (LLMs) have revolutionized natural language processing, yet remain vulnerable to jailbreak attacks—particularly multi-turn jailbreaks that distribute malicious intent across benign exchanges, thereby bypassing alignment mechanisms. Existing approaches often suffer from limited exploration of the adversarial space, rely on hand-crafted heuristics, or lack systematic query refinement. We propose NEXUS (Network Exploration for eXploiting Unsafe Sequences), a modular framework for constructing, refining, and executing optimized multi-turn attacks. NEXUS comprises: (1) ThoughtNet, which hierarchically expands a harmful intent into a structured semantic network of topics, entities, and query chains; (2) a feedback-driven Simulator that iteratively refines and prunes these chains …

Insights In Adaptation: Examining Self-Reflection Strategies Of Job Seekers With Visual Impairments In India, Akshay Kolgar Nayak, Yash Prakash, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Significant changes in the digital employment landscape, driven by rapid technological advancements and the COVID-19 pandemic, have introduced new opportunities for blind and visually impaired (BVI) individuals in developing countries like India. However, a significant portion of the BVI population in India remains unemployed despite extensive accessibility advancements and job search interventions. Therefore, we conducted semi-structured interviews with 20 BVI persons who were either pursuing or recently sought employment in the digital industry. Our findings reveal that despite gaining digital literacy and extensive training, BVI individuals struggle to meet industry requirements for fulfilling job openings. While they engage in self-reflection …

Optiselect And Enshap: Integrating Machine Learning And Game Theory For Ischemic Stroke Prediction, Pritam Chakraborty, Anjan Bandyopadhyay, Sricheta Parul, Sujata Swain, Partha Sarathy Banerjee, Tapas Si, Hong Qin, Saurav Mallik

Computer Science Faculty Publications

Stroke analysis using game theory and machine learning techniques. The study investigates the use of the Shapley value in predictive ischemic brain stroke analysis. Initially, preference algorithms identify the most important features in various machine learning models, including logistic regression, K-nearest neighbor, decision tree, support vector machine (linear kernel), support vector machine ( RBF kernel), neural networks, etc. For each sample, the top 3, 4, and 5 features are evaluated and selected to evaluate their performance. The Shapley value method was used to rank the models using their best four features based on their predictive capabilities. As a result, better-performing …

Human Perception Of Ai Capabilities At Classifying Perturbed Roadway Signs, Katherine R. Garcia, Jing Chen, Yanru Xiao, Scott Mishler, Cong Wang, Bin Hu

Computer Science Faculty Publications

Artificial Intelligence (AI) is crucial to numerous functions required for driving automation systems, including the computer vision techniques used to detect the roadway environment and make real-time decisions. However, the images used as inputs to the AI system may be maliciously perturbed, or manipulated, causing the AI system to make an incorrect classification. In this study, we examined humans’ perception of the AI’s computer vision capability of classifying various road sign images, including the original images, images with two different types of malicious attacks, and images that are scrambled randomly at the pixel level. Our results showed that participants rated …

A Real-Time Approach To Capture Ambient And Focal Attention In Visual Search, Gavindya Jayawardena, Yasith Jayawardana, Yasasi Abeysinghe, Bhanuka Mahanama, Sampath Jayarathna, Jacek Gwizdka

Computer Science Faculty Publications

During visual search, individuals’ attention shifts between ambient and focal states in response to task demands and stimuli. The ambient/focal coefficient K is a statistically validated measure of these states, computed offline from fixation duration and saccade amplitude data. While current methods compute K offline, real-time computation could enable applications such as monitoring user attention, creating attention-adaptive user interfaces, and optimizing graphics rendering. However, real-time computation of K requires stable estimates for the parameters of fixation duration and saccade amplitude distributions. Since these distributions are heavy-tailed, the real-time estimates exhibit high variance and slow convergence. To overcome this, we propose …

Quickque: Enabling Quick Access To Information In User Reviews For Screen Reader Users, Mohan Sunkara, Akshay Kolgar Nayak, Sandeep Kalari, Yash Prakash, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Efficiently perusing online customer reviews is presently challenging for blind users, who rely on a screen reader that supports predominantly one-dimensional narration of content via keyboard shortcuts. To address this, with restaurant reviews as seminal case study, we developed QuickCue, a browser extension prototype that enables screen reader users to quickly obtain the positives and negatives regarding different aspects of a restaurant (e.g., food quality, hygiene, ambiance), without having to sift through numerous reviews containing redundant information. At its core, QuickCue utilizes a large language model to perform aspect and sentiment-based joint classification of reviews to group them based on …

Accessmenu: Enhancing Usability Of Online Restaurant Menus For Screen Reader Users, Nithiya Venkatraman, Akshay Kolgar Nayak, Suyog Dahal, Yash Prakash, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Online food ordering has become commonplace due to its convenience. The wide variety of culinary choices, combined with fast and economical door-delivery services, encourages more people to order food online. To facilitate this process, food vendors, including restaurants, often provide full menus on their websites, typically in visual formats such as images or PDFs. While this is convenient for sighted users, blind and visually impaired (BVI) individuals face significant challenges accessing these visual menus with their screen reader assistive technology. An interview study with 12 BVI screen reader users revealed that present assistive tools do not adequately satisfy the needs …

Adapting Online Customer Reviews For Blind Users: A Case Study Of Restaurant Reviews, Mohan Sunkara, Akshay Kolgar Nayak, Sandeep Kalari, Yash Prakash, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Online reviews have become an integral aspect of consumer decision-making on e-commerce websites, especially in the restaurant industry. Unlike sighted users who can visually skim through the reviews, perusing reviews remains challenging for blind users, who rely on screen reader assistive technology that supports predominantly one-dimensional narration of content via keyboard shortcuts. In an interview study, we uncovered numerous pain points of blind screen reader users with online restaurant reviews, notably, the listening fatigue and frustration after going through only the first few reviews. To address these issues, we developed QuickCue assistive tool that performs aspect-focused sentiment-driven summarization to reorganize …

An Optimized Generalized Multi-Color Point Implicit Solver For Intel Gpus Using Oneapi Esimd, Joseph Wassell, Mohammad Zubair, Aaron Walden, Gabriel Nastac, Eric Nielsen, Timothée Ewart

Computer Science Faculty Publications

This paper presents an efficient implementation of a linear-solver kernel relevant to FUN3D, a suite of computational fluid dynamics software developed at NASA’s Langley Research Center. The linear solver is optimized for a range of block sizes commonly used in FUN3D. The implementation targets Aurora, the Argonne Leadership Computing Facility’s (ALCF) exascale machine featuring Intel Data Center Max 1550 GPUs. The linear solver’s performance is memory bandwidth-bound due to its low arithmetic intensity. The primary performance challenges stem from variable matrix row lengths and indirect memory access patterns inherent in unstructured-grid applications. Variable block sizes introduce additional complexity through differing …

Openmuse: Integrating Open-Source Models Into Music Creation Workflows, Tyler K. Vergho

Dartmouth College Master’s Theses

This master's thesis introduces OpenMUSE (Open Multimodal Unified Sound Engine), a platform that demonstrates the potential of open-source AI music generation by integrating state-of-the-art deep learning models into a unified system. By unifying ten different open-source models, including MusicGen, AudioLDM2, and custom-trained text-to-symbolic music generation models, OpenMUSE aims to create a user-friendly interface that empowers artists to produce complex, adaptive musical compositions. The system enhances accessibility by providing a simple web interface and natural language controls, while improving controllability through features like melody conditioning and semantic audio editing. Specifically, OpenMUSE offers a digital audio workstation (DAW)-inspired interface that lowers the …

An End-To-End Bi-Objective Approach To Deep Graph Partitioning, Pengcheng Wei, Yuan Fang, Zhihao Wen, Zheng Xiao, Binbin Chen

Research Collection School Of Computing and Information Systems

Graphs are ubiquitous in real-world applications, such as computation graphs and social networks. Partitioning large graphs into smaller, balanced partitions is often essential, with the biobjective graph partitioning problem aiming to minimize both the“cut” across partitions and the imbalance in partition sizes. However, existing heuristic methods face scalability challenges or overlook partition balance, leading to suboptimal results. Recent deep learning approaches, while promising, typically focus only on node-level features and lack a truly end-to-end framework, resulting in limited performance. In this paper, we introduce a novel method based on graph neural networks (GNNs) that leverages multilevel graph features and addresses …

Eulerian Smoke Simulation With Multiple Fields, Diyang Zhang

Dartmouth College Master’s Theses

Fluid simulation is a cornerstone of computer graphics, enabling the realistic depiction of dynamic phenomena such as smoke, fire, and other gaseous behaviours. This thesis focuses on advancing Eulerian smoke simulation techniques, with a particular emphasis on grid-based simulations that capture intricate vortical structures and fine visual details.

We propose several detail-preserving frameworks that incorporate various scalar and vector fields within the simulation pipeline, including velocity, impulse, and Lamb vectors, along with their decompositions and transformed representations. By mathematically analyzing the properties of impulse, we derive its scalar fields decomposition (ImpSFD), which introduces an alternative numerical interpretation, and Vortex-Particles in …

Surveying The Role Of Visual Analytics In Human-Machine Teaming, Naga Datha Saikiran Battula

Theses

Humans and machines both possess their unique capabilities and have their strengths and weaknesses, which can be complementary to one another and allow them to achieve a common goal. Teaming in the modern era involves text prompts, voice commands, gesture recognition, touch interfaces, and the latest visualization techniques that allow parties/agents to interact. Communication through visualization plays a vital role in allowing robust insights to be gained through a glance. Using visualization as a medium between humans and machines can increase the communication bandwidth. Human-machine teaming has witnessed much progress, with many theories and practical examples emerging. In the report, …

La Creatividad En Peligro: Como La Inteligencia Artificial Es Un Reto Para Los Artistas., Nathaly Cisneros

Capstones

Los artistas digitales han creado obras maestras que nos han dejado sin aliento con sus pinceles digitales, lápices y pinturas. Desde retratos que parecen saltar de la pantalla hasta paisajes que nos transportan a mundos desconocidos, su arte ha sido una fuente constante de inspiración.

Pero en los últimos años, una nueva fuerza ha comenzado a cambiar el juego. La inteligencia artificial ha estado avanzando a pasos agigantados y ahora se perfila como una amenaza para el futuro de los artistas digitales. ¿Qué significa esto para el arte y la creatividad?

Link: https://docs.google.com/document/d/1xe8UxDMekX_SwiIppyt_JppK8M-lB-YWNWGyeyShlJM/edit?usp=sharing

Real-Time Feedback-Driven Framework For Automated Cybersickness Mitigation, Md Jahirul Islam

Master's Theses

As technologies are becoming more advanced day by day, the embracement of virtual reality (VR) technology among users is also increasing in daily activities for various purposes, and subsequently, the barrier between the real and virtual world is fading. Despite the versatile uses, cybersickness (CS) is a major problem which is induced among users due to the immersive VR experience. There is a plethora of research findings and methods to measure the users’ CS such as virtual reality sickness questionnaire (VRSQ), simulator sickness questionnaire (SSQ), fast motion scale questionnaire (FMS), and others. Recently, machine learning approaches have also been adopted …

On The Benefits Of Directness In Virtual Characters For Motivational Interviews, Michael O'Mahony, Cathy Ennis, Robert Ross

Conference papers

Understanding the factors influencing successful engagement with Embodied Conversational Agents (ECAs) remains a significant challenge. This understanding could be used to personalise agents to users to improve interactions. Some studies have shown that simulating personalities in healthcare agents can improve effectiveness and engagement. However, it is not yet well understood how variations of agent personality can be leveraged to improve user engagement with Motivational Interviewing (MI) ECAs. Specifically how the balance between agent warmth and directness can be controlled in an MI agent to improve likeability and engagement. We conducted an online Wizard-of-Oz (WoZ) mediated study of two variants of …

Visualization Of Paleocurrents On A Web Application Using Gplates, Anjan Sapkota

MS in Computer Science Theses

Paleocurrents are flow directions derived from features of sedimentary rocks that reveal the direction of the current of wind or water that deposited the sediment. In 2015, Brand et al. created a global database of paleocurrents, which contains over 1,000,000 measurements worldwide: North America, South America, Australia, Great Britain, parts of Western Europe, China, Africa are fairly well represented; Antarctica, Eastern Europe, and Asia are modestly represented and Russia is poorly represented. The contribution of this thesis is a web application that uses the GPlates’ Application Programming Interface (API) to visualize global paleocurrents through time in an interactive way based …

Using Llms To Establish Implicit User Sentiment Of Software Desirability, Sherri Weitl-Harms, John D. Hastings, Jonah Lum

Research & Publications

This study explores the use of LLMs for providing quantitative zero-shot sentiment analysis of implicit software desirability, addressing a critical challenge in product evaluation where traditional review scores, though convenient, fail to capture the richness of qualitative user feedback. Innovations include establishing a method that 1) works with qualitative user experience data without the need for explicit review scores, 2) focuses on implicit user satisfaction, and 3) provides scaled numerical sentiment analysis, offering a more nuanced understanding of user sentiment, instead of simply classifying sentiment as positive, neutral, or negative.

Data is collected using the Microsoft Product Desirability Toolkit (PDT), …

Real-Time Motion Augmentation And Synthesis For Animating The Hands And Eyes Of Virtual Humans And Avatars, Ryan Canales

All Dissertations

Virtual Reality (VR) enables users to interact within virtual worlds via an embodied virtual representation of themselves called an “avatar”. Because avatars are essential for immersive experiences, it is important to consider how altering or augmenting avatar motion affects virtual experiences. This dissertation aims to improve virtual experiences by addressing some of the many challenges in animating avatars and virtual humans.

In our first study, we addressed the lack of tactile feedback during virtual grasping by using visual feedback techniques. We augmented the avatar’s hand motion to remain outside virtual objects (“outer hand”) even when the user’s hand penetrated them. …

User Acceptance Of Advice By Ai Agents: Expectation-System Fit Perspective, Jingyuan Cai, Fiona Fui-Hoon Nah

Research Collection School Of Computing and Information Systems

Algorithms have increasing influence on our daily decisions, especially when the recommendations are presented by human-like AI agents. This study applies the Theory of Effective Use to investigate how the fit between the user’s role expectation for an AI agent and the agent’s interaction style impacts AI advice adoption. We proposed a new concept termed Perceived Expectation-System Fit (PESF) and empirically examined its impact on user perceptions and advice acceptance. We found that low PESF reduces advice acceptance by diminishing cognitive and affective trust in the AI agent. Furthermore, increased algorithm transparency increases PESF's impact on decision-making. Our findings provide …

Replay-And-Forget-Free Graph Class-Incremental Learning: A Task Profiling And Prompting Approach, Chaoxi Niu, Guansong Pang, Ling Chen, Bing Liu

Research Collection School Of Computing and Information Systems

Class-incremental learning (CIL) aims to continually learn a sequence of tasks, with each task consisting of a set of unique classes. Graph CIL (GCIL) follows the same setting but needs to deal with graph tasks (e.g., node classification in a graph). The key characteristic of CIL lies in the absence of task identifiers (IDs) during inference, which causes a significant challenge in separating classes from different tasks (i.e., inter-task class separation). Being able to accurately predict the task IDs can help address this issue, but it is a challenging problem. In this paper, we show theoretically that accurate task ID …

Interactive Known-Item Search In Large Video Corpora, Zhixin Ma

Dissertations and Theses Collection (Open Access)

The surge in video volume makes it challenging to locate a specific target with a single query using automatic video retrieval systems. The interactive video retrieval offers a solution by enabling users to iteratively refine a search. Nevertheless, existing systems often present users with an overwhelming number of similar videos, which can lead to mental fatigue while inspecting results and increase difficulty in providing feedback. This dissertation studies known-item video search and addresses four key challenges. First and foremost, as the link between users and the system, the interaction must be both efficient and effective. To ensure effectiveness, the user’s …

Shifting Perspectives With Procedural Modeling Techniques And Vertex Animated Textures For Real-Time Interactive Morphing, Stephanie Schulze

All Theses

As humans experience reality, they intake external stimuli using sensory receptors to process information, forming a perception. Perceptions are subjective and shape each individual’s reality. Naturally then, by changing one’s perceptions, their experience of reality is altered, for better or worse. To emphasize the idea that perception is malleable and encourage a mindset of questioning alternative ways to look at a situation, an interactive experience in Unreal Engine is developed where a dull cityscape morphs and transforms into a surreal oversized nature scene. Procedural modeling techniques are used to create a variety of morphing assets, while Vertex Animated Textures are …

Triadic Temporal-Semantic Alignment For Weakly-Supervised Video Moment Retrieval, Jin Liu, Jialong Xie, Fengyu Zhou, Shengfeng He

Research Collection School Of Computing and Information Systems

Video Moment Retrieval (VMR) aims to identify specific event moments within untrimmed videos based on natural language queries. Existing VMR methods have been criticized for relying heavily on moment annotation bias rather than true multi-modal alignment reasoning. Weakly supervised VMR approaches inherently overcome this issue by training without precise temporal location information. However, they struggle with fine-grained semantic alignment and often yield multiple speculative predictions with prolonged video spans. In this paper, we take a step forward in the context of weakly supervised VMR by proposing a triadic temporalsemantic alignment model. Our proposed approach augments weak supervision by comprehensively addressing …

Mvgamba : Unify 3d Content Generation As State Space Sequence Modeling, Xuanyu Yi, Zike Wu, Qiuhong Shen, Qingshan Xu, Pan Zhou, Joo-Hwee Lim, Shuicheng Yan, Xinchao Wang, Hanwang Zhang

Research Collection School Of Computing and Information Systems

Recent 3D large reconstruction models (LRMs) can generate high-quality 3D content in sub-seconds by integrating multi-view diffusion models with scalable multi-view reconstructors. Current works further leverage 3D Gaussian Splatting as 3D representation for improved visual quality and rendering efficiency. However, we observe that existing Gaussian reconstruction models often suffer from multi-view inconsistency and blurred textures. We attribute this to the compromise of multi-view information propagation in favor of adopting powerful yet computationally intensive architectures (e.g., Transformers). To address this issue, we introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor based on the RNN-like State …