Graphics and Human Computer Interfaces Commons^™

A Macrocognitive Design Taxonomy For Simulation-Based Training Systems: Bridging Cognitive Theory And Human-Computer Interaction, Jessica M. Johnson

Virginia Digital Maritime Center (VDMC) Faculty Publications

Simulation-based training systems are increasingly deployed to prepare learners for complex, safety-critical, and dynamic work environments. While advances in computing have enabled immersive and data-rich simulations, many systems remain optimized for procedural accuracy and surface-level task performance rather than the macrocognitive processes that underpin adaptive expertise. Macrocognition encompasses higher-order cognitive processes that are essential for performance transfer beyond controlled training conditions. When these processes are insufficiently supported, training systems risk fostering brittle strategies and negative training effects. This paper introduces a macrocognitive design taxonomy for simulation-based training systems derived from a large-scale meta-analysis examining the transfer of macrocognitive skills from …

Micro-Behavioral Analysis Of Online Shopping Patterns For Blind Users, Yash Prakash, Akshay Kolgar Nayak, Nithiya Venkatraman, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

While online shopping platforms provide convenience and autonomy to blind users, their non-visual interactions remain underexplored at a micro-behavioral level. Existing studies have primarily emphasized accessibility and usability challenges but have overlooked how fine-grained, screen reader-driven keystroke-level behaviors reflect users’ cognitive strategies. In this paper, we present the findings of a longitudinal study with 25 blind participants to examine their micro-behavioral patterns, using keyboard activity and screen reader logs on both familiar and unfamiliar e-commerce websites. We complemented this study with semi-structured interviews to contextualize the uncovered micro-behavioral patterns. Our results revealed patterns in how blind users draw upon cognitive …

Shaping The Future: Emerging Technologies And Their Role In Industry 4.0 And Beyond, Liuliu Qin

Information Technology & Decision Sciences Faculty Publications

This paper provides a comprehensive review of emerging technologies driving the transition from Industry 4.0 to Industry 5.0. It examines the foundational concepts and pillars of Industry 4.0 and explores the transformative roles of Artificial Intelligence (AI), Extended Reality (XR), Collaborative Cobots (Cobots), Brain–Computer Interfaces (BCIs), quantum technologies, and next-generation connectivity (5G/6G). By integrating technological, human-centric, and sustainability perspectives, the study outlines how these emerging technologies reshape industrial systems and enable intelligent, adaptive, and inclusive futures.

Learning Design To Advance Human-Ai Collaboration In K-12 Education, Wing Sha Chan, Jinhee Kim, Seongryeong Yu, Rita Kay Detrick

STEMPS Faculty Publications

This chapter explores key components for designing effective Human-AI Collaboration (HAC) in K–12 education, addressing the current lack of theoretical and conceptual frameworks for structuring and implementing HAC in teaching and learning. It examines four essential areas: curriculum design, student and teacher–AI interaction, learning environments, and the evolution of HAC over time. The chapter introduces the concept of HAC in K–12 contexts, highlighting how humans and AI can leverage each other's strengths through co-evolutionary processes that foster mutual learning and collaboration. It reviews current HAC practices in schools and discusses their contributions to both teaching and learning. Finally, it presents …

Impacts Of Segmenting Principle On Learner Performance And Attitude In A 3d Environment: A Mixed-Method Multiple Case Study, Kristin Herman, Mohan Yang, Jim Shifflet, Noah Glaser

STEMPS Faculty Publications

This study presents a conceptual replication of Moreno’s (Appl Cogn Psychol 21:765–781. 10.1002/acp.1348, 2007) study on the benefits of adhering to the segmentation principle when utilizing multimedia learning objects. Furthermore, this study expands upon the original by taking place in a low-immersive virtual reality environment, allowing for further understanding on the extent to which multimedia principles are still relevant. Both a synchronous and an asynchronous case are presented. Results indicate benefits for both cases in far transfer of learning. Furthermore, synchronous learners indicated a significant reduction in cognitive load and increased overall attitudes towards learning due to segmented instruction.

Wearable Sensor-Based Phase Segmentation Analysis Of Front Crawl Swimming: A Scoping Review, Jonathan Simoes, Samuel Aylward, Daniel Hamze, Daniel James Goble, Daniel M. Russell, Joshua Haworth

Exercise Science Faculty Publications

Front crawl swimming stroke phase segmentation has historically relied on video analysis, but the development of wearable sensor technology has created new opportunities for automated phase segmentation. This scoping review mapped the available evidence on wearable sensor-based stroke phase segmentation methods in front crawl swimming, following PRISMA-ScR guidelines. A systematic search of SPORTDiscus, Web of Science, and IEEE Xplore conducted from January to June 2026, identified 15 eligible peer-reviewed studies published between 2000 and 2024. The review revealed an emerging field of research that has converged methodologically around inertial measurement units (IMUs) and the Chollet phase segmentation framework while remaining …

Improving Medical Diagnostics With Vision-Language Models: Convex Hull-Based Uncertainty Analysis, Ferhat Ozgur Catak, Murat Kuzlu, Taylor Patrick, Michel Audette

Engineering Technology Faculty Publications

In recent years, vision-language models (VLMs) have been applied to various fields, including healthcare, education, finance, and manufacturing, with remarkable performance. However, concerns remain regarding VLMs' consistency and uncertainty, particularly in critical applications such as healthcare, which demand a high level of trust and reliability. This paper proposes a novel approach to evaluate uncertainty in VLMs' responses using a convex hull approach on a healthcare application for visual question answering (VQA). For any VLM, temperature refers to a sampling parameter used in probabilistic generation, which controls the randomness of the model's output. The LLM-CXR model is selected as the medical …

Eeg And Imu Gait Signal Processing: A Comparative Assessment Of The "Reza" Exponential Filter And Classic Filters, Reza Pousti, Daniel M. Russell, Derek C. Monroe, Christopher K. Rhea

Rehabilitation Sciences Faculty Publications

Noise degrades both EEG and gait signals, and classical IIR filters (Butterworth, Chebyshev, elliptic) involve trade-offs between passband flatness, ripple, and roll-off. This study compared a novel exponential "Reza" filter with these designs for neural and locomotor data. We analyzed an open-source mobile brain-body imaging dataset with EEG and gait data from 49 healthy adults (EEG: 256-channel, 512 Hz; IMUs: six APDM Opals, 128 Hz). EEG channels were grand-averaged and band-pass filtered at 0.5-50 Hz, while IMU axes were averaged and band-pass filtered at 0.5-5 Hz. The outcomes were signal-to-noise ratio SNR (dB) and band-integrated Welch PSD (EEG:0.5-50 Hz; IMU:0.5-5 …

Seeing The Invisible Load: Xr+ Multimodal Sensing For Cognitive Ergonomics In Industrial Training, Jessica M. Johnson, Andwele Grant

Virginia Digital Maritime Center (VDMC) Faculty Publications

Extended reality (XR) technologies are increasingly positioned as disruptive Industry 5.0 tools for human-centric industrial training and intelligent human–system integration. Coupled with multimodal sensing (eye tracking, EEG, HRV, GSR, and other physiological signals), XR environments promise to make otherwise invisible cognitive demands observable, especially for novice trainees entering complex industrial settings. Yet the evidence base is fragmented: (1) there is no quantitative synthesis of the cognitive ergonomics benefits of XR plus sensing; (2) little is known about which XR–sensor configurations yield the strongest effects; (3) prior reviews rarely focus on industrial and manufacturing tasks; (4) multimodal signals are used predominantly …

Gaze Transition Entropy And Automation Trust In Multitasking Workspace, Yusuke Yamani, Austin Jackson, Tetsuya Sato, Feyishola Ashimi, Michael S. Politowicz, Eric T. Chancey, Makoto Itoh

Psychology Faculty Publications

Safe flight operation requires visual scanning across multiple displays in a cockpit, which collectively represent the state of the aircraft and supporting automation. Trust is a crucial factor that drives human-automation interaction, and recent work has suggested a relationship between an operator's visual attention and automation trust. One index that captures predictability of eye movements between different areas of interest is gaze transition entropy. The current work reanalyzed data from Sato et al., which examined eye movement patterns and trust in automation associated with the system monitoring task of the Multi-Attribute Task Battery. Results showed credible positive correlations between the …

Cognitive Prosthetic: An Ai-Enabled Multimodal System For Episodic Recall In Knowledge Work, Lawrence Obiuwevwi, Krzystof J. Rechowicz, Vikas Ashok, Sachin Shetty, Sampath Jayarathna

Computer Science Faculty Publications

Modern knowledge workplaces increasingly strain human episodic memory as individuals navigate fragmented attention, overlapping meetings, and multimodal information streams. Existing workplace tools provide partial support through note-taking or analytics but rarely integrate cognitive, physiological, and attentional context into retrievable memory representations. This paper presents the Cognitive Prosthetic Multimodal System (CPMS)—an AI-enabled proof-of-concept designed to support episodic recall in knowledge work through structured episodic capture and natural language retrieval. CPMS synchronizes speech transcripts, physiological signals, and gaze behavior into temporally aligned, JSON-based episodic records processed locally for privacy. Beyond data logging, the system includes a web-based retrieval interface that allows users …

Contextual Scaffolding And Self-Efficacy: Supporting Computer Skill Development Among Blind Learners In India, Akshay Kolgar Nayak, Yash Prakash, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Inclusive computer literacy education efforts, broadening the participation of blind or visually impaired (BVI) individuals, have gained traction in recent years. Existing literature investigating these efforts primarily draws evidence from affluent Global North contexts, where accessibility resources and legal frameworks are relatively more mature. Little is known about the in-situ teaching and learning challenges faced by trainers and BVI students, respectively, in resource-constrained, multicultural Global South countries like India. To address this knowledge gap, we conducted a four-month contextual inquiry at two computer training centers catering to 94 BVI students in India. We notably observed a rigid, experience-driven training environment …

Voxvista: Enhancing Screen Reading Experience For Online User Comments, Yash Prakash, Akshay Kolgar Nayak, Mohammed Shoaib Alyaan, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Online discussions have become integral to how people exchange ideas, form opinions, and participate in collective deliberation. While sighted users can comfortably engage with online discussions, blind users who are dependent on screen readers are forced to listen to long threads narrated in a single, monotonic voice that lacks prosodic variation, rhythm, or emotion. This robotic auditory experience not only deteriorates the user engagement with the content but also increases cognitive strain, by making it difficult to remain attentive and discern meaning beyond literal words. In an interview study, most blind participants reported that monotonous narration hindered their ability to …

Examining Inclusive Computing Education For Blind Students In India, Akshay Kolgar Nayak, Yash Prakash, Md Javedul Ferdous, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

The growing demand for computer professionals, driven by the expanding Information Technology industry, has led to numerous inclusive computing education efforts. These efforts have even included blind or visually-impaired (BVI) students, who are being increasingly encouraged to pursue education and a career in computing, despite the visually-oriented nature of the discipline. Extant literature has predominantly focused on identifying and addressing the accessibility barriers faced by BVI students to promote more inclusive learning environments. While few studies have also investigated the accessibility of computing education from the perspectives of BVI learners and instructors, these have been primarily situated in the Global …

Modeling Joint Visual Attention In Naturalistic Dyadic Interactions, Kuushini Thennakoon, Yasasi Abeysinghe, Bhanuka Mahanama, Vikas Ashok, Sampath Jayarathna

Computer Science Faculty Publications

Joint visual attention (JVA) provides important insight into how individuals coordinate attention during social interaction. Egocentric eye tracking enables the study of JVA in natural, multi-user settings. This work presents a multi-stage framework to identify and analyze JVA using egocentric video and gaze data. The approach consists of three steps: spatiotemporal tube-based visual similarity, gaze-guided object detection, and attention pattern analysis using the ambient–focal coefficient K. Results show that object-focused collaborative activities exhibit high JVA, with object detection capturing higher joint attention than visual similarity, whereas conversation-based or independent activities show lower and more fragmented joint attention. Analysis of K …

Finding The Signal In The Noise: An Exploratory Study On Assessing The Effectiveness Of Ai And Accessibility Forums For Blind Users' Support Needs, Satwik Ram Kodandaram, Jiawei Zhou, Xiaojun Bi, Iv Ramakrishnan, Vikas Ashok

Computer Science Faculty Publications

Accessibility forums and, more recently, generative AI tools have become vital resources for blind users seeking solutions to computer-interaction issues and learning about new assistive technologies, screen reader features, tutorials, and software updates. Understanding user experiences with these resources is essential for identifying and addressing persistent support gaps. Towards this, we interviewed 14 blind users who regularly engage with forums and GenAI tools. Findings revealed that forums often overwhelm users with multiple overlapping topics, redundant or irrelevant content, and fragmented responses that must be mentally pieced together, increasing cognitive load. GenAI tools, while offering more direct assistance, introduce new barriers …

Lost In Instructions: Study Of Blind Users' Experiences With Diy Manuals And Ai-Rewritten Instructions For Assembly, Operation, And Troubleshooting Of Tangible Products, Monalika Padma Reddy, Aruna Balasubramanian, Jiawei Zhou, Xiaojun Bi, Iv Ramakrishnan, Vikas Ashok

Computer Science Faculty Publications

AI tools like ChatGPT and Be-My-AI are increasingly being used by blind individuals. Although prior work has explored their use in some Do-It-Yourself (DIY) tasks by blind individuals, little is known about how they use these tools and the available product-manual resources to assemble, operate, and troubleshoot physical/tangible products – tasks requiring spatial reasoning, structural understanding, and precise execution. We address this knowledge gap via an interview study and a usability study with blind participants, investigating how they leverage AI tools and product manuals for DIY tasks with physical products. Findings show that manuals are essential resources, but product-manual instructions …

A Comparative Analysis Of Explainable Ai (Xai) Techniques For Transparent And Reliable Image Classification, Sovon Chakraborty, Shakib Mahmud Dipto, Kevin R. Pilkiewicz, Michael L. Mayo, Pratip Rana

Computer Science Faculty Publications

Evaluating the trustworthiness of black-box machine learning models remains a significant methodological challenge. Their lack of transparency and interpretability limits applicability, because stakeholders often seek transparency before trusting the results of black-box machine learning models. Explainable AI (XAI) methods provide for human-understandable justifications and informed decision-making of these black-box architectures. Therefore, it is imperative to select the proper XAI model tailored to specific tasks. In this research, we focus on examining four XAI techniques: PEEK, LRP, GRAD-CAM, and LIME to understand how they perform against each other for image classification tasks. We evaluate the performance, robustness, generalizability, noise stability, and …

Susceptibility To High-Fidelity Misinformation: An Eye-Tracking Analysis, Yasasi Abeysinghe, Gavindya Jayawardena, Enkelejda Kasneci, Sampath Jayarathna

Computer Science Faculty Publications

With the rise of online misinformation and AI-generated text, understanding human perception of news truthfulness is critical. In this study, we examine visual attention and cognitive processing using eye-tracking measures as individuals read fake and real news articles sharing nearly identical structure and imagery, differing only in subtle textual changes. Using the public FakeNewsPerception dataset, we analyze advanced gaze measures, including scanpaths, AOI transitions, and luminance-corrected pupil measures, beyond basic gaze features, in relation to news truthfulness and perceived believability. Results show that, given the high fidelity of the fake news, readers exhibited comparable visual scanning patterns, attention allocation across …

Memebuddy: Dialog-Style Audio Representations For Engaging Non-Visual Meme Experiences, Chirag Bhansali, Vikas Ashok, Hae-Na Lee

Computer Science Faculty Publications

Image memes are a pervasive form of online communication, widely used to convey humor, opinions, and cultural references. Prior work has explored making memes accessible to blind users, primarily through auto-generated descriptive captions. While these approaches improve comprehensibility and sometimes incorporate prosodic or emotional cues, they often fail to capture the humor, narrative structure, and contextual nuances that make memes engaging. We present MemeBuddy, a system that models memes as dialog, generating structured, multi-turn audio representations using role-based speakers. MemeBuddy reinterprets a meme as a conversation between two speakers, integrating extracted meme text with contextual knowledge implicitly inferred by a …

Teaming With Technology: Adaptive Automation In Joint Cognitive Systems For Industry 5.0, Jessica Johnson

Virginia Digital Maritime Center (VDMC) Faculty Publications

Adaptive automation enables dynamic reallocation of functions between people and autonomous agents to improve performance in complex work. This paper presents a meta-analysis of experimental and quasi-experimental studies (2000-2025) on joint cognitive systems in industrially relevant contexts, quantifying effects on task performance, safety/failure management, workload, trust, and learning. Across studies, adaptive automation reliably reduces operator workload and shows moderate gains in task performance and safety, with healthier trust dynamics when adaptations are triggered by human-state or event cues, made transparent to the user, and remain rapidly overridable. Risks emerge when performance-triggered switching is opaque or poorly timed, which can erode …

Revealing Spatiotemporal Neural Activation Patterns In Electrocorticography Recordings Of Human Speech Production By Mutual Information, Julio Kovacs, Dean Krusienski, Minu Maninder, Willy Wriggers

Mechanical & Aerospace Engineering Faculty Publications

Background

Spatiotemporal mapping of neural activity during continuous speech production has been traditionally approached using correlation coefficient (CC) analysis between cortical signals and speech recordings. A prior study employed this approach using electrocorticography (ECoG) data from participants who underwent invasive intracranial monitoring for epilepsy. However, CC cannot detect nonlinear relationships and is dominated by the correspondence between periods of silence and of non-silence.

New Method

We introduce the mutual information (MI) measure, which can capture both linear and nonlinear dependencies. We validated CC and MI on the sub-second spatiotemporal brain activity recorded during continuous speech tasks. To refine the results, …

Clinician Perspectives On Virtual Reality Use In Physical Therapy Practice In The United States, Danielle T. Felsberg, Jared T. Mcguirt, Scott E. Ross, Louisa D. Raisbeck, Charlend K. Howard, Christopher K. Rhea

Rehabilitation Sciences Faculty Publications

The primary goal of physical rehabilitation is to assess movement impairments and restore function to improve overall quality of life. Virtual reality (VR) may provide the optimal environment to promote these goals due to its motivating and modifiable nature which can be difficult to accomplish through traditional real-world therapeutic methods. Current research of VR for rehabilitation has demonstrated that VR interventions can produce clinically meaningful change in motor outcomes. Despite this, adoption and usage of VR by physical therapy professionals is unclear due to the limited research in this area. Thus, the purpose of this study was to identify the …

Nexus: Network Exploration For Exploiting Unsafe Sequences In Multi-Turn Llm Jailbreaks, Javad Rafiei Asl, Sidhant Narula, Mohammad Ghasemigol, Eduardo Blanco, Daniel Takabi

School of Cybersecurity Faculty Publications

Large Language Models (LLMs) have revolutionized natural language processing, yet remain vulnerable to jailbreak attacks—particularly multi-turn jailbreaks that distribute malicious intent across benign exchanges, thereby bypassing alignment mechanisms. Existing approaches often suffer from limited exploration of the adversarial space, rely on hand-crafted heuristics, or lack systematic query refinement. We propose NEXUS (Network Exploration for eXploiting Unsafe Sequences), a modular framework for constructing, refining, and executing optimized multi-turn attacks. NEXUS comprises: (1) ThoughtNet, which hierarchically expands a harmful intent into a structured semantic network of topics, entities, and query chains; (2) a feedback-driven Simulator that iteratively refines and prunes these chains …

Flexible Hybrid Self-Powered Piezo-Triboelectric Nanogenerator Based On Bto-Pvdf/Pdms Nanocomposites For Human Machine Interaction, Wentao Dong, Mengyun Li, Chang Chen, Kun Xie, Jinhua Hong, Lin Yang

Civil & Environmental Engineering Faculty Publications

As flexible and wearable electronics play more and more important role in smart watches, smart glass and virtual reality, and the power supply to the wearable electronics have been revealed more attentions for long-term usage and continuous healthy monitoring. To overcome the challenge, flexible self-powered BTO-PVDF/PDMS piezoelectric-triboelectric electric hybrid generators (BPP-HNG) are developed to human gesture monitoring and human machine interaction (HMI) application without external power supply. BPP-HNG based on BTO-PVDF and PDMS films are prepared by sol-gel and spin-coating method. When the BTO content is 20 wt.%, BPP-HNG exhibits better electrical performance with an output voltage of 20.51 V. …

Quickque: Enabling Quick Access To Information In User Reviews For Screen Reader Users, Mohan Sunkara, Akshay Kolgar Nayak, Sandeep Kalari, Yash Prakash, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Efficiently perusing online customer reviews is presently challenging for blind users, who rely on a screen reader that supports predominantly one-dimensional narration of content via keyboard shortcuts. To address this, with restaurant reviews as seminal case study, we developed QuickCue, a browser extension prototype that enables screen reader users to quickly obtain the positives and negatives regarding different aspects of a restaurant (e.g., food quality, hygiene, ambiance), without having to sift through numerous reviews containing redundant information. At its core, QuickCue utilizes a large language model to perform aspect and sentiment-based joint classification of reviews to group them based on …

Accessmenu: Enhancing Usability Of Online Restaurant Menus For Screen Reader Users, Nithiya Venkatraman, Akshay Kolgar Nayak, Suyog Dahal, Yash Prakash, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Online food ordering has become commonplace due to its convenience. The wide variety of culinary choices, combined with fast and economical door-delivery services, encourages more people to order food online. To facilitate this process, food vendors, including restaurants, often provide full menus on their websites, typically in visual formats such as images or PDFs. While this is convenient for sighted users, blind and visually impaired (BVI) individuals face significant challenges accessing these visual menus with their screen reader assistive technology. An interview study with 12 BVI screen reader users revealed that present assistive tools do not adequately satisfy the needs …

Automation To Autonomy: Temporal Dynamics Of Trust And Visual Attention Allocation Did Not Evolve, Tetsuya Sato, Eric Chancey, Yusuke Yamani

Psychology Faculty Publications

Emerging work environments are expected to implement autonomy that performs various functions without human input. Previous works has shown that trust in automation is negatively correlated with visual attention allocation, indicating that trust is a dynamic construct. Moreover, trust in automation and trust in autonomy appears to evolve in similar ways. However, recent work has demonstrated differences between trust in automation and trust in autonomy within Kaber’s (2018) theoretical framework (Sato et al., 2023b). Yet, it is uncertain whether the development of trust and visual attention allocation differs between automation and autonomy. The present study examined the temporal dynamics of …

The Interacting Roles Of Attention Allocation And Trust In Highly Automated Aam Environments, Yusuke Yamani

Psychology Faculty Publications

[First slide]

Mechanisms of attentive visual processing

• Attention control
• Visual search
• Eye movement
• Aging and individual differences

Limits of human performance in applied environment

• Complex displays
• Machine operation
• Surface transportation
• Advanced air mobility
• Nuclear operation

Methods to ameliorate human cognitive performance

• Human-machine interface
• Human autonomy/AI teaming
• Human-systems integration
• Training

Human Perception Of Ai Capabilities At Classifying Perturbed Roadway Signs, Katherine R. Garcia, Jing Chen, Yanru Xiao, Scott Mishler, Cong Wang, Bin Hu

Computer Science Faculty Publications

Artificial Intelligence (AI) is crucial to numerous functions required for driving automation systems, including the computer vision techniques used to detect the roadway environment and make real-time decisions. However, the images used as inputs to the AI system may be maliciously perturbed, or manipulated, causing the AI system to make an incorrect classification. In this study, we examined humans’ perception of the AI’s computer vision capability of classifying various road sign images, including the original images, images with two different types of malicious attacks, and images that are scrambled randomly at the pixel level. Our results showed that participants rated …