Vision Science Commons^™

A Webcam Eye Tracking Infrastructure For Software Engineering Tasks, Zachary M. Kozak

School of Computing: Dissertations, Theses, and Student Research

Performing eye tracking utilizing commodity webcams has been explored for over a decade, but limited camera quality and sensitivity to head movements have hindered its adoption in research settings. Recent advances in consumer-grade webcams and machine learning methods present an opportunity to improve the accuracy of webcam eye tracking and extend the feasibility of studies beyond controlled laboratory environments.

Current popular webcam eye tracking methods restrict implementations to the browser and rely on continuous user interactions for calibration, limiting the kinds of studies that can be conducted. This thesis presents a feature-based gaze prediction system that incorporates eye geometry and …

Memebuddy: Dialog-Style Audio Representations For Engaging Non-Visual Meme Experiences, Chirag Bhansali, Vikas Ashok, Hae-Na Lee

Computer Science Faculty Publications

Image memes are a pervasive form of online communication, widely used to convey humor, opinions, and cultural references. Prior work has explored making memes accessible to blind users, primarily through auto-generated descriptive captions. While these approaches improve comprehensibility and sometimes incorporate prosodic or emotional cues, they often fail to capture the humor, narrative structure, and contextual nuances that make memes engaging. We present MemeBuddy, a system that models memes as dialog, generating structured, multi-turn audio representations using role-based speakers. MemeBuddy reinterprets a meme as a conversation between two speakers, integrating extracted meme text with contextual knowledge implicitly inferred by a …

Adversarial Robustness In Biomedical Time-Series Models, Rohan Tiwari

Bioengineering Theses

This study investigates adversarial vulnerabilities in deep learning models for biomedical time-series classification across two clinically important modalities: electrocardiography (ECG) and electroencephalography (EEG). Using the MIT-BIH Arrhythmia and CHB-MIT seizure datasets, I evaluate time-domain attacks (FGSM, PGD), Fourier-domain constrained attacks, and learned spectral perturbations designed to reveal modality-specific sensitivity patterns. Across both tasks, a consistent trend emerges low-frequency components (0–5 Hz) constitute a dominant axis of adversarial vulnerability, with perturbations in this range producing the steepest degradation in classification performance. In ECG models, protecting the physiologically relevant QRS band (5–20 Hz) significantly improves robustness, whereas EEG models remain highly sensitive …

Insights In Adaptation: Examining Self-Reflection Strategies Of Job Seekers With Visual Impairments In India, Akshay Kolgar Nayak, Yash Prakash, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Significant changes in the digital employment landscape, driven by rapid technological advancements and the COVID-19 pandemic, have introduced new opportunities for blind and visually impaired (BVI) individuals in developing countries like India. However, a significant portion of the BVI population in India remains unemployed despite extensive accessibility advancements and job search interventions. Therefore, we conducted semi-structured interviews with 20 BVI persons who were either pursuing or recently sought employment in the digital industry. Our findings reveal that despite gaining digital literacy and extensive training, BVI individuals struggle to meet industry requirements for fulfilling job openings. While they engage in self-reflection …

Accessible Real-Time Eye-Gaze Tracking For Neurocognitive Health Assessments, A Multimodal Web-Based Approach, Daniel C. Tisdale

Master's Theses

We introduce a novel integration of real-time, predictive eye-gaze tracking models into a multimodal dialogue system tailored for remote health assessments. This system is designed to be highly accessible requiring only a conventional webcam for video input along with minimal cursor interaction and utilizes engaging gaze-based tasks that can be performed directly in a web browser. We have crafted dynamic subsystems that capture high-quality data efficiently and maintain quality through instances of user attrition and incomplete calls. Additionally, these subsystems are designed with the foresight to allow for future re-analysis using improved predictive models, as well as enable the creation …

Implementing Unmanned Aerial Vehicles To Collect Human Gait Data At Distance And Altitude For Identification And Re-Identification, Donn E. Bartram

Graduate Theses, Dissertations, and Problem Reports (ETD)

Gait patterns are a class of biometric information pertaining to the way a person moves and poses. Gait information is unique to each person and can be used to identify and reidentify people. Historically, this task has been achieved through the use of multiple ground-based imaging sensors. However, as Unmanned Aerial Vehicles (UAVs) advance, they present the opportunity to evolve the process of persons identification and re-identification. Collecting human gait data using UAVs at distances ranging from 20m to 500m and altitudes ranging from 0m to 120m is a challenging task. The current biometric data collection methods, primarily designed for …

2d Respiratory Sound Analysis To Detect Lung Abnormalities, Rafia Sharmin Alice, Kc Santosh

SDSU Data Science Symposium

Abstract. In this paper, we analyze deep visual features from 2D data representation(s) of the respiratory sound to detect evidence of lung abnormalities. The primary motivation behind this is that visual cues are more important in decision-making than raw data (lung sound). Early detection and prompt treatments are essential for any future possible respiratory disorders, and respiratory sound is proven to be one of the biomarkers. In contrast to state-of-the-art approaches, we aim at understanding/analyzing visual features using our Convolutional Neural Networks (CNN) tailored Deep Learning Models, where we consider all possible 2D data such as Spectrogram, Mel-frequency Cepstral Coefficients …

On The Pursuit Of Developer Happiness: Webcam-Based Eye Tracking And Affect Recognition In The Ide, Tamsin Rogers

Honors Theses

Recent research highlights the viability of webcam-based eye tracking as a low-cost alternative to dedicated remote eye trackers. Simultaneously, research shows the importance of understanding emotions of software developers, where it was found that emotions have significant effects on productivity, code quality, and team dynamics. In this paper, we present our work towards an integrated eye-tracking and affect recognition tool for use during software development. This combined approach could enhance our understanding of software development by combining information about the code developers are looking at, along with the emotions they experience. The presented tool utilizes an unmodified webcam to capture …

Human Interaction With Fake News, Autumn Woodson, Sampath Jayarathna (Mentor)

Computer & Information Science: Research Experiences for Undergraduates in Disinformation Detection and Analytics

No abstract provided.

Eye Movement And Pupil Measures: A Review, Bhanuka Mahanama, Yasith Jayawardana, Sundararaman Rengarajan, Gavindya Jayawardena, Leanne Chukoskie, Joseph Snider, Sampath Jayarathna

Computer Science Faculty Publications

Our subjective visual experiences involve complex interaction between our eyes, our brain, and the surrounding world. It gives us the sense of sight, color, stereopsis, distance, pattern recognition, motor coordination, and more. The increasing ubiquity of gaze-aware technology brings with it the ability to track gaze and pupil measures with varying degrees of fidelity. With this in mind, a review that considers the various gaze measures becomes increasingly relevant, especially considering our ability to make sense of these signals given different spatio-temporal sampling capacities. In this paper, we selectively review prior work on eye movements and pupil measures. We first …

Customer Gaze Estimation In Retail Using Deep Learning, Shashimal Senarath, Primesh Pathirana, Dulani Meedeniya, Sampath Jayarathna

Computer Science Faculty Publications

At present, intelligent computing applications are widely used in different domains, including retail stores. The analysis of customer behaviour has become crucial for the benefit of both customers and retailers. In this regard, the concept of remote gaze estimation using deep learning has shown promising results in analyzing customer behaviour in retail due to its scalability, robustness, low cost, and uninterrupted nature. This study presents a three-stage, three-attention-based deep convolutional neural network for remote gaze estimation in retail using image data. In the first stage, we design a mechanism to estimate the 3D gaze of the subject using image data …

Seeing Eye To Eye: A Machine Learning Approach To Automated Saccade Analysis, Maigh Attre

Honors Scholar Theses

Abnormal ocular motility is a common manifestation of many underlying pathologies particularly those that are neurological. Dynamics of saccades, when the eye rapidly changes its point of fixation, have been characterized for many neurological disorders including concussions, traumatic brain injuries (TBI), and Parkinson’s disease. However, widespread saccade analysis for diagnostic and research purposes requires the recognition of certain eye movement parameters. Key information such as velocity and duration must be determined from data based on a wide set of patients’ characteristics that may range in eye shapes and iris, hair and skin pigmentation [36]. Previous work on saccade analysis has …

Open Source Artificial Intelligence In A Biological/Ecological Context, Trevor Grant

Annual Symposium on Biomathematics and Ecology Education and Research

No abstract provided.

Automatic Detection Of Aircraft Emergency Landing Sites, Yu-Fei Shen, Zia-Ur Rahman, Dean Krusienski, Jiang Li, Zia-Ur Rahman (Ed.), Stephen E. Reichenbach (Ed.), Mark Allen Neifeld (Ed.)

Electrical & Computer Engineering Faculty Publications

An automatic landing site detection algorithm is proposed for aircraft emergency landing. Emergency landing is an unplanned event in response to emergency situations. If, as is unfortunately usually the case, there is no airstrip or airfield that can be reached by the un-powered aircraft, a crash landing or ditching has to be carried out. Identifying a safe landing site is critical to the survival of passengers and crew. Conventionally, the pilot chooses the landing site visually by looking at the terrain through the cockpit. The success of this vital decision greatly depends on the external environmental factors that can impair …