Graphics and Human Computer Interfaces Commons^™

Modeling Joint Visual Attention In Naturalistic Dyadic Interactions, Kuushini Thennakoon, Yasasi Abeysinghe, Bhanuka Mahanama, Vikas Ashok, Sampath Jayarathna

Computer Science Faculty Publications

Joint visual attention (JVA) provides important insight into how individuals coordinate attention during social interaction. Egocentric eye tracking enables the study of JVA in natural, multi-user settings. This work presents a multi-stage framework to identify and analyze JVA using egocentric video and gaze data. The approach consists of three steps: spatiotemporal tube-based visual similarity, gaze-guided object detection, and attention pattern analysis using the ambient–focal coefficient K. Results show that object-focused collaborative activities exhibit high JVA, with object detection capturing higher joint attention than visual similarity, whereas conversation-based or independent activities show lower and more fragmented joint attention. Analysis of K …

Cognitive Prosthetic: An Ai-Enabled Multimodal System For Episodic Recall In Knowledge Work, Lawrence Obiuwevwi, Krzystof J. Rechowicz, Vikas Ashok, Sachin Shetty, Sampath Jayarathna

Computer Science Faculty Publications

Modern knowledge workplaces increasingly strain human episodic memory as individuals navigate fragmented attention, overlapping meetings, and multimodal information streams. Existing workplace tools provide partial support through note-taking or analytics but rarely integrate cognitive, physiological, and attentional context into retrievable memory representations. This paper presents the Cognitive Prosthetic Multimodal System (CPMS)—an AI-enabled proof-of-concept designed to support episodic recall in knowledge work through structured episodic capture and natural language retrieval. CPMS synchronizes speech transcripts, physiological signals, and gaze behavior into temporally aligned, JSON-based episodic records processed locally for privacy. Beyond data logging, the system includes a web-based retrieval interface that allows users …

Display System Interface Using Visually-Evoked Cortical Potentials, Michael E. Miller, Brett J. Borghetti, Kellie D. Kennedy, Chad L. Stephens, Alan T. Pope

AFIT Patents

A brain-computer interface system includes a video processor for producing a display signal, a temporal controller for producing a plurality of repetitive visual stimulus (RVS) signals with different respective temporal aspects, a display device that receives the display signal and displays a corresponding image on a plurality of different display regions and receives the RVS signals and displays corresponding RVS in respective ones of the display regions, an electroencephalographic (EEG) sensor for sensing a visually-evoked cortical potential (VECP) signal in a user with eyes fixated on a viewed one of the display regions, and a VECP processor for processing the …

A Real-Time Approach To Capture Ambient And Focal Attention In Visual Search, Gavindya Jayawardena, Yasith Jayawardana, Yasasi Abeysinghe, Bhanuka Mahanama, Sampath Jayarathna, Jacek Gwizdka

Computer Science Faculty Publications

During visual search, individuals’ attention shifts between ambient and focal states in response to task demands and stimuli. The ambient/focal coefficient K is a statistically validated measure of these states, computed offline from fixation duration and saccade amplitude data. While current methods compute K offline, real-time computation could enable applications such as monitoring user attention, creating attention-adaptive user interfaces, and optimizing graphics rendering. However, real-time computation of K requires stable estimates for the parameters of fixation duration and saccade amplitude distributions. Since these distributions are heavy-tailed, the real-time estimates exhibit high variance and slow convergence. To overcome this, we propose …

Eyetraes : Fine-Grained, Low-Latency Eye Tracking Via Adaptive Event Slicing, Argha Sen, Panahetipola Mudiyanselage Nuwan Bandara, Ila Gokarn, Thivya Kandappu, Archan Misra

Research Collection School Of Computing and Information Systems

Eye-tracking technology has gained significant attention in recent years due to its wide range of applications in humancomputer interaction, virtual and augmented reality, and wearable health. Traditional RGB camera-based eye-tracking systems often struggle with poor temporal resolution and computational constraints, limiting their effectiveness in capturing rapid eye movements. To address these limitations, we propose EyeTrAES, a novel approach using neuromorphic event cameras for high-fidelity tracking of natural pupillary movement that shows significant kinematic variance. One of EyeTrAES’s highlights is the use of a novel adaptive windowing/slicing algorithm that ensures just the right amount of descriptive asynchronous event data accumulation within …

X-Disetrac: Distributed Eye-Tracking With Extended Realities, Bhanuka Mahanama, Sampath Jayarathna

College of Sciences Posters

Humans use heterogeneous collaboration mediums such as in-person, online, and extended realities for day-to-day activities. Identifying patterns in viewpoints and pupillary responses (a.k.a eye-tracking data) provide informative cues on individual and collective behavior during collaborative tasks. Despite the increasing ubiquity of these different mediums, the aggregation and analysis of eye-tracking data in heterogeneous collaborative environments remain unexplored. Our study proposes X-DisETrac: Extended Distributed Eye Tracking, a versatile framework for eye tracking in heterogeneous environments. Our approach tackles the complexity by establishing a platform-agnostic communication protocol encompassing three data streams to simplify data aggregation and …

Evaluating Human Eye Features For Objective Measure Of Working Memory Capacity, Yasasi Abeysinghe, Enkelejda Kasneci (Ed.), Frederick Shic (Ed.), Mohamed Khamis (Ed.)

Computer Science Faculty Publications

Eye tracking measures can provide means to understand the underlying development of human working memory. In this study, we propose to develop machine learning algorithms to find an objective relationship between human eye movements via oculomotor plant and their working memory capacity, which determines subjective cognitive load. Here we evaluate oculomotor plant features extracted from saccadic eye movements, traditional positional gaze metrics, and advanced eye metrics such as ambient/focal coefficient , gaze transition entropy, low/high index of pupillary activity (LHIPA), and real-time index of pupillary activity (RIPA). This paper outlines the proposed approach of evaluating eye movements for obtaining an …

Driving Simulator : Driving Performance Under Distraction, Kaushik Pilligundla

Master's Projects

This pilot study used a driving simulator experiment to look into how podcast consumption affects driving performance as a continuous distraction. Three volunteers conducted three trials in the study, each with a different driving scenario. Data analysis was done to compare two conditions. The first condition is the Audio, where volunteers listen to podcasts while driving. The second condition is no-audio condition.. The no-audi condition had nothing to play in the background. We used eye-tracking technology to gather gaze data. The study's findings using the post survey and eye fixation data indicate that listening to podcasts leads to continuous distraction …

Eye Movements Behaviors In A Driving Simulator During Simple And Complex Distractions, Pradeep Narayana

Master's Projects

Road accidents occur frequently due to driving distractions all around the world. A driving simulator has been created to explore the cognitive effects of distractions while driving in order to address this problem. The purpose of this study is to discover the distraction-causing elements and how they affect driving performance. The simulator offers a secure and regulated setting for carrying out tests while being distracted by different visual distractions, such as solving mathematical equations and number memorizations.

Several trials have been conducted in the studies, which were carried out under varied circumstances like varying driving sceneries and by displaying different …

Multi-User Eye-Tracking, Bhanuka Mahanama

Computer Science Faculty Publications

The human gaze characteristics provide informative cues on human behavior during various activities. Using traditional eye trackers, assessing gaze characteristics in the wild requires a dedicated device per participant and therefore is not feasible for large-scale experiments. In this study, we propose a commodity hardware-based multi-user eye-tracking system. We leverage the recent advancements in Deep Neural Networks and large-scale datasets for implementing our system. Our preliminary studies provide promising results for multi-user eye-tracking on commodity hardware, providing a cost-effective solution for large-scale studies.

Toward A Real-Time Index Of Pupillary Activity As An Indicator Of Cognitive Load, Gavindya Jayawardena, Yasith Jayawardana, Sampath Jayarathna, Jonas Högström, Thomas Papa, Deepak Akkil, Andrew T. Duchowski, Vsevolod Peysakhovich, Izabela Krejtz, Nina Gehrer, Krzysztof Krejtz

Computer Science Faculty Publications

The Low/High Index of Pupillary Activity (LHIPA), an eye-tracked measure of pupil diameter oscillation, is redesigned and implemented to function in real-time. The novel Real-time IPA (RIPA) is shown to discriminate cognitive load in re-streamed data from earlier experiments. Rationale for the RIPA is tied to the functioning of the human autonomic nervous system yielding a hybrid measure based on the ratio of Low/High frequencies of pupil oscillation. The paper's contribution is drawn from provision of documentation of the calculation of the RIPA. As with the LHIPA, it is possible for researchers to apply this metric to their own experiments …

Eye Movement And Pupil Measures: A Review, Bhanuka Mahanama, Yasith Jayawardana, Sundararaman Rengarajan, Gavindya Jayawardena, Leanne Chukoskie, Joseph Snider, Sampath Jayarathna

Computer Science Faculty Publications

Our subjective visual experiences involve complex interaction between our eyes, our brain, and the surrounding world. It gives us the sense of sight, color, stereopsis, distance, pattern recognition, motor coordination, and more. The increasing ubiquity of gaze-aware technology brings with it the ability to track gaze and pupil measures with varying degrees of fidelity. With this in mind, a review that considers the various gaze measures becomes increasingly relevant, especially considering our ability to make sense of these signals given different spatio-temporal sampling capacities. In this paper, we selectively review prior work on eye movements and pupil measures. We first …

Using Ai-Enabled Gaze Tracking System To Understand Comprehension Patterns Of Computer Science Students, Bradley Boswell

College of Graduate Studies: Theses & Dissertations

Academic institutions and instructors lack the ability to accurately assess the moment-to-moment attentiveness of students in classrooms where students’ faces are obscured by computer monitors. This can cause the lectures of Computer Science, Information Technology, or other lab-based courses to be incorrectly-paced, which can lead to students having an overall poorer grasp of the subject material. We propose a system for real-time accurate detection of classroom attentiveness using monitor-mounted webcams and eye trackers, along with a Convolutional Neural Network machine learning model (NiCATS). Through the use of a neural network, we produce an initial attentiveness score based on student webcam …

A Review On Eye-Tracking Metrics For Sleepiness, Debasis Roy, Fiona Fui-Hoon Nah

Research Collection School Of Computing and Information Systems

Sleepiness that can arise from sleep deprivation can increase human errors in task performance and create workplace hazards and accidents. Hence, it is critical to detect sleepiness to minimize hazards and human errors. This paper provides a review of the literature on eye tracking metrics that can be used to detect sleepiness. These metrics include blink duration, blink frequency, saccade latency, saccade peak velocity, saccade accuracy, smooth pursuit velocity gain, fixation rate, pupil size, and latency to pupil constriction.

Use Of Eye-Tracking To Identify Psychological Indicators Of Sleepiness, Debasis Roy, Fiona Fui-Hoon Nah, Matthew Thimgan

Research Collection School Of Computing and Information Systems

Sleepiness or sleep deprivation creates a serious hazard or obstacle to task execution and performance. Sleep deprivation can be life-threating (e.g., when driving or executing attention-critical tasks). We are interested to examine if eye-tracking technology can be used to assess and detect sleepiness in an online environment. In this research proposal, we will focus on examining the relationships between sleepiness and parameters involving pupil size, blinks, and saccades.

Automatic Gaze Classification For Aviators: Using Multi-Task Convolutional Networks As A Proxy For Flight Instructor Observation, Justin Wilson, Sandro Scielzo, Sukumaran Nair, Eric C. Larson

International Journal of Aviation, Aeronautics, and Aerospace

In this work, we investigate how flight instructors observe aviator scan patterns and assign quality to an aviator's gaze. We first establish the reliability of instructors to assign similar quality to an aviator's scan patterns, and then investigate methods to automate this quality using machine learning. In particular, we focus on the classification of gaze for aviators in a mixed-reality flight simulation. We create and evaluate two machine learning models for classifying gaze quality of aviators: a task-agnostic model and a multi-task model. Both models use deep convolutional neural networks to classify the quality of pilot gaze patterns for 40 …

Effect Of Timing And Source Of Online Product Recommendations: An Eye-Tracking Study, Yan Shi, Qing Zeng, Fiona Fui-Hoon Nah, Chuan-Hoo Tan, Choon Ling Sia, Keng Siau, Jiaqi Yan

Research Collection School Of Computing and Information Systems

Online retail business has become an emerging market for almost all business owners. Online recommender systems provide better service to consumers during their decision making processes. In this study, a controlled lab experiment was conducted to assess the effect of recommendation timing (early, mid, and late) and recommendation source (expert reviews vs. consumer reviews) on online consumers’ interest and attention. Eye-tracking data was extracted from the experiment and analyzed. The results suggest that consumers show more interest in recommendation based on consumer reviews than expert reviews. Earlier recommendations do not receive greater attention than later recommendations.

Reducing Peak Workload In The Cockpit: A Human In The Loop Simulation Evaluating New Runway Selection Tool, Tanja Bos, Rolf Zon, Wilfred Rouwhorst

H-Workload 2017: Models and Applications (Works in Progress)

In efforts to increase safety and reduce peak workload situations in the cockpit, a tool with a different interaction style was developed for use in case of a runway change instructed by Air Traffic Control during approach. In an experiment a workload comparison was made between the new tool and the conventional cockpit. Workload was measured by means of a self-rating after each experiment run, as well as eye blink frequency during each run. Results show that the self-rated workload decreases with the new tool for one of the two crew members and the blink frequency suggests a workload decrease …

Using Eye And Head Movements As A Control Mechanism For Tele-Operating A Ground-Based Robot And Its Payload, Kathryn C. Hicks

Computational Modeling & Simulation Engineering Theses & Dissertations

To date, eye and head tracking has been used to indicate users' attention patterns while performing a task or as an aid for disabled persons, to allow hands-free interaction with a computer. The increasing accuracy and the reduced cost of eye- and head-tracking equipment make utilizing this technology feasible for explicit control tasks, especially in cases where there is confluence between the visual task and control.

The goal of this research was to investigate the use of eye-tracking as a more natural interface for the control of a camera-equipped, remotely operated robot in tasks that require the operator to simultaneously …

Understanding Effects Of Presentation On Concept Learning In Technology Supported Learning, Nivedita Singh

Theses

The world of technology has had a significant impact on learning and instructional domain. Today, a large number of devices and software are specifically designed to afford faster and effective learning and instruction. They have not only erased the physical boundaries to resources in education but have also helped create new interactions and engagements for learners and instructors. With this changed scenario, the content or instructional material also needs our attention to become usable and compatible with the changed learning styles and preferences of the learners today. Not only does the content need to seamlessly integrate with the delivery methodology …