Graphics and Human Computer Interfaces Commons^™

Micro-Behavioral Analysis Of Online Shopping Patterns For Blind Users, Yash Prakash, Akshay Kolgar Nayak, Nithiya Venkatraman, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

While online shopping platforms provide convenience and autonomy to blind users, their non-visual interactions remain underexplored at a micro-behavioral level. Existing studies have primarily emphasized accessibility and usability challenges but have overlooked how fine-grained, screen reader-driven keystroke-level behaviors reflect users’ cognitive strategies. In this paper, we present the findings of a longitudinal study with 25 blind participants to examine their micro-behavioral patterns, using keyboard activity and screen reader logs on both familiar and unfamiliar e-commerce websites. We complemented this study with semi-structured interviews to contextualize the uncovered micro-behavioral patterns. Our results revealed patterns in how blind users draw upon cognitive …

Contextual Scaffolding And Self-Efficacy: Supporting Computer Skill Development Among Blind Learners In India, Akshay Kolgar Nayak, Yash Prakash, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Inclusive computer literacy education efforts, broadening the participation of blind or visually impaired (BVI) individuals, have gained traction in recent years. Existing literature investigating these efforts primarily draws evidence from affluent Global North contexts, where accessibility resources and legal frameworks are relatively more mature. Little is known about the in-situ teaching and learning challenges faced by trainers and BVI students, respectively, in resource-constrained, multicultural Global South countries like India. To address this knowledge gap, we conducted a four-month contextual inquiry at two computer training centers catering to 94 BVI students in India. We notably observed a rigid, experience-driven training environment …

Voxvista: Enhancing Screen Reading Experience For Online User Comments, Yash Prakash, Akshay Kolgar Nayak, Mohammed Shoaib Alyaan, Sampath Jayarathna, Hae-Na Lee, Vikas Ashok

Computer Science Faculty Publications

Online discussions have become integral to how people exchange ideas, form opinions, and participate in collective deliberation. While sighted users can comfortably engage with online discussions, blind users who are dependent on screen readers are forced to listen to long threads narrated in a single, monotonic voice that lacks prosodic variation, rhythm, or emotion. This robotic auditory experience not only deteriorates the user engagement with the content but also increases cognitive strain, by making it difficult to remain attentive and discern meaning beyond literal words. In an interview study, most blind participants reported that monotonous narration hindered their ability to …

Modeling Joint Visual Attention In Naturalistic Dyadic Interactions, Kuushini Thennakoon, Yasasi Abeysinghe, Bhanuka Mahanama, Vikas Ashok, Sampath Jayarathna

Computer Science Faculty Publications

Joint visual attention (JVA) provides important insight into how individuals coordinate attention during social interaction. Egocentric eye tracking enables the study of JVA in natural, multi-user settings. This work presents a multi-stage framework to identify and analyze JVA using egocentric video and gaze data. The approach consists of three steps: spatiotemporal tube-based visual similarity, gaze-guided object detection, and attention pattern analysis using the ambient–focal coefficient K. Results show that object-focused collaborative activities exhibit high JVA, with object detection capturing higher joint attention than visual similarity, whereas conversation-based or independent activities show lower and more fragmented joint attention. Analysis of K …

A Comparative Analysis Of Explainable Ai (Xai) Techniques For Transparent And Reliable Image Classification, Sovon Chakraborty, Shakib Mahmud Dipto, Kevin R. Pilkiewicz, Michael L. Mayo, Pratip Rana

Computer Science Faculty Publications

Evaluating the trustworthiness of black-box machine learning models remains a significant methodological challenge. Their lack of transparency and interpretability limits applicability, because stakeholders often seek transparency before trusting the results of black-box machine learning models. Explainable AI (XAI) methods provide for human-understandable justifications and informed decision-making of these black-box architectures. Therefore, it is imperative to select the proper XAI model tailored to specific tasks. In this research, we focus on examining four XAI techniques: PEEK, LRP, GRAD-CAM, and LIME to understand how they perform against each other for image classification tasks. We evaluate the performance, robustness, generalizability, noise stability, and …

Susceptibility To High-Fidelity Misinformation: An Eye-Tracking Analysis, Yasasi Abeysinghe, Gavindya Jayawardena, Enkelejda Kasneci, Sampath Jayarathna

Computer Science Faculty Publications

With the rise of online misinformation and AI-generated text, understanding human perception of news truthfulness is critical. In this study, we examine visual attention and cognitive processing using eye-tracking measures as individuals read fake and real news articles sharing nearly identical structure and imagery, differing only in subtle textual changes. Using the public FakeNewsPerception dataset, we analyze advanced gaze measures, including scanpaths, AOI transitions, and luminance-corrected pupil measures, beyond basic gaze features, in relation to news truthfulness and perceived believability. Results show that, given the high fidelity of the fake news, readers exhibited comparable visual scanning patterns, attention allocation across …

Memebuddy: Dialog-Style Audio Representations For Engaging Non-Visual Meme Experiences, Chirag Bhansali, Vikas Ashok, Hae-Na Lee

Computer Science Faculty Publications

Image memes are a pervasive form of online communication, widely used to convey humor, opinions, and cultural references. Prior work has explored making memes accessible to blind users, primarily through auto-generated descriptive captions. While these approaches improve comprehensibility and sometimes incorporate prosodic or emotional cues, they often fail to capture the humor, narrative structure, and contextual nuances that make memes engaging. We present MemeBuddy, a system that models memes as dialog, generating structured, multi-turn audio representations using role-based speakers. MemeBuddy reinterprets a meme as a conversation between two speakers, integrating extracted meme text with contextual knowledge implicitly inferred by a …

Designing Narrative-Based Ai Assistance For Sensemaking In Collaborative Environments: Case Studies In Education And Dementia Care, Dylan Edward Moore

Dartmouth College Ph.D Dissertations

This thesis addresses a gap in the human-computer interaction literature regarding the design, development, and evaluation of narrative-based AI assistance for collaborative, complex problem solving. I explore this design space through three case studies across the domains of education and dementia care. This work encompasses multi-year industry partnerships and longitudinal fieldwork, user-centered design, dataset curation, model training, and system evaluation.

Specifically, the first case study considers a story-based web platform for teaching AI literacy through peer-generated, personalized narrative scaffolding. Learners on the platform showed significant knowledge gains and other learning-related outcomes. To describe the novel design of this system, I …

From Physical Correlation To Emotional Connection: The Role Of Passive Haptics On Empathy In Virtual Reality, Jemely Robles

Dartmouth College Master’s Theses

Virtual reality is increasingly explored as a tool for cultivating empathy, and haptic feedback as a tool for enhancing immersion. This paper investigates the effects of combining the two.  Fifty-two participants experienced a custom-built VR scene in which a character was shown packing up a room. Participants were assigned to either a haptic condition, receiving passive haptic feedback corresponding to the character's actions, or a non-haptic control condition that did not receive any haptic input. Trait empathy was measured beforehand, and state empathy and engagement were measured after the experience. Thematic analysis was conducted on post-study interviews, and headset recordings …

Purified Zero-Shot Sketch-Based Image Retrieval, Yang Zhou, Jingru Yang, Jin Wang, Kaixiang Huang, Guodong Lu, Shengfeng He

Research Collection School Of Computing and Information Systems

Sketches, as a new solution in multimedia systems that can replace natural language, are characterized by sparse visual cues such as simple strokes that differ significantly from natural images containing complex elements such as background, foreground, and texture. This misalignment poses substantial challenges for zero-shot sketch-based image retrieval (ZS-SBIR). Prior approaches match sketches to full images and tend to overlook redundant elements in natural images, leading to model distraction and semantic ambiguity. To address this issue, we introduce a distraction-agnostic framework, purified cross-domain matching (PuXIM), which operates on a straightforward principle: masking and matching. We devise a visual-cross-linguistic (VxL) sampler …

Tests Without Borders: A Global Approach To Measuring Visualization Literacy, Olivia A. Guess

McKelvey School of Engineering Graduate Student Theses & Dissertations

Visualization literacy assessments shape how we understand people's ability to interpret data, yet most existing instruments embed Western datasets and assumptions that limit their relevance for global audiences. This thesis argues that because data is personal, assessments must also be culturally grounded. We introduce a unified framework for adapting the Mini-VLAT into 22 regionally responsive short-form assessments, each retaining the structure of the original test while incorporating datasets and scenarios tailored to specific regions around the world. To demonstrate how such adaptations can be customized and validated, we present a detailed case study of a Ghana-adapted Mini-VLAT, developed in collaboration …

Developing An Ai-Assisted Grading System Using Large Language Models, Andrei Modiga

MS in Computer Science Project Reports

We present a grading system that accelerates evaluation of open-ended student work across scanned and digital workflows. The system crops answer regions from PDFs, assigns submissions via OCR on identity regions only, and groups answers by visual semantics using a vision LLM. Instructors review and edit groups, apply rubric items once per group, and export grades from an on-screen table. The solution integrates Ghostscript rasterization, PdfPig page orchestration, SkiaSharp region extraction, Tesseract identity OCR, and GPT-4o Vision for grouping. We detail the architecture, token-budgeted batching strategy, and persistence design, then describe testing results for grouping quality, time-on-task, and usability. The …

My First Conversation With Chatgpt (February 22, 2023): Origins Of A Generative Dialogue, David Smith

Publications and Research

This working paper presents the first recorded interaction between the author and the generative AI system ChatGPT, written on February 22, 2023 during the initial weeks of a faculty sabbatical in Boston. The document preserves a complete and unedited transcript of an exploratory conversation conducted without predetermined research aims, marking the author’s first encounter with a large-language-model conversational interface. Although the exchange includes creative experimentation—including musical and poetic prompts—the discussion remains informal and wide-ranging, and no theoretical framework is articulated at this stage. Rather, this transcript is published as primary-source material documenting the moment of discovery and experimentation that precedes …

Developing Accessible Narrative-Based Stem Learning Software For K-6 Braille Display Users, Dylan Ravel, Daniel Tsivkovski, Brandon Foley, Maryam Etezad, Franceli Cibrian, Ariel Han, Rajeev Joshi

Student Scholar Symposium Abstracts and Posters

This research develops a free, accessible web application that enables K-6 students who are blind or visually impaired (BVI) to learn STEM concepts using refreshable braille displays. Currently, most online learning tools are not designed for BVI students, creating a significant educational barrier.

The application interfaces with commercial braille displays and uses narrative-based learning to make STEM content approachable and engaging. By presenting material as interactive stories, students can connect with concepts while developing braille reading skills. The curriculum design prioritizes accessibility through the Accessible Rich Internet Applications (ARIA) standards and screen reader support.

The goal is to provide BVI …

Decoding The Chameleon Game, Tri Dang '25, Hieu Tran, Brian T. Howard, Sutthirut Charoenphon, Dat Nguyen '25

Student Research

The Chameleon game is a challenging word association activity where players are given a secret word and must respond with words relevant to that secret word. It requires strategic thinking and deduction. The Chameleon must cleverly guess the secret keyword in this game while avoiding suspicion. Our research aims to create an advanced artificial intelligence (AI) model that can play the Chameleon game from both perspectives: as the Chameleon and as a Human. This AI is designed to guess secret keywords based on the information provided by the players, choose the best strategies to avoid detection as the Chameleon, identify …

Registration Is A Powerful Rotation-Invariance Learner For 3d Anomaly Detection, Yuyang Yu, Zhengwei Chen, Xuemiao Xu, Lei Zhang, Haoxin Yang, Yongwei Nie, Shengfeng He

Research Collection School Of Computing and Information Systems

3D anomaly detection in point-cloud data is critical for industrial quality control, aiming to identify structural defects with high reliability. However, current memory bank-based methods often suffer from inconsistent feature transformations and limited discriminative capacity, particularly in capturing local geometric details and achieving rotation invariance. These limitations become more pronounced when registration fails, leading to unreliable detection results. We argue that point-cloud registration plays an essential role not only in aligning geometric structures but also in guiding feature extraction toward rotation-invariant and locally discriminative representations. To this end, we propose a registration-induced, rotation-invariant feature extraction framework that integrates the objectives …

Instance-Level Video Depth In Groups Beyond Occlusions, Yuan Liang, Yang Zhou, Ziming Sun, Tianyi Xiang, Guiqing Li, Shengfeng He

Research Collection School Of Computing and Information Systems

Depth estimation in dynamic, multi-object scenes remains a major challenge, especially under severe occlusions. Existing monocular models, including foundation models, struggle with instance-wise depth consistency due to their reliance on global regression. We tackle this problem from two key aspects: data and methodology. First, we introduce the Group Instance Depth (GID) dataset, the first large-scale video depth dataset with instance-level annotations, featuring 101,500 frames from real-world activity scenes. GID bridges the gap between synthetic and real-world depth data by providing high-fidelity depth supervision for multi-object interactions. Second, we propose InstanceDepth, the first occlusion-aware depth estimation framework for multi-object environments. Our …

Marsanywhere: Dataset And Cross-View Diffusion Model For Satellite-To-Ground View Synthesis With Mars Data, Benjamin T. Hinchliff

Master's Theses

Satellite-to-ground view synthesis aims to create a realistic ground view image from a corresponding satellite view image. This is a well-studied problem for street level imagery, with good results being achieved by using modern image synthesis techniques such as diffusion models. However, despite the public availability of satellite and ground level imagery on Mars, these techniques have yet to be applied to the domain due to difficulties in collating and processing the data into a usable form. We address this deficiency by creating a dataset consisting of ground view panorama imagery from the Perseverance rover, along with associated satellite view …

Upright-Net+: Enhanced Learning Of Upright Orientation For 3d Point Clouds, Xufang Pang, Feng Li, Hongjie Zhuang, Ning Ding, Xiaopin Zhong, Shengfeng He, Wenxi Liu, Bo Jiang

Research Collection School Of Computing and Information Systems

Automatic 3D shape analysis is heavily influenced by the pose of input 3D models, as the continuous nature of pose space introduces complexities that usually exceed the encoding capacities of standard deep learning frameworks. To tackle this challenge, we present Upright-Net+, an enhancement of our previous model, Upright-Net, specifically developed for estimating upright orientation in 3D point clouds. Our approach is grounded in the design principle that ”form ever follows function,” treating the natural base of an object as a functional structure that stabilizes it in its typical pose, influenced by physical laws and geometric properties. We reformulate the continuous …

Jury-And-Judge Chain-Of-Thought For Uncovering Toxic Data In 3d Visual Grounding, Kaixiang Huang, Qifeng Zhang, Jin Wang, Jingru Yang, Yang Zhou, Huan Yu, Guodong Lu, Shengfeng He

Research Collection School Of Computing and Information Systems

3D Visual Grounding (3DVG) faces persistent challenges due to coarse scene-level observations and logically inconsistent annotations, which introduce ambiguities that compromise data quality and hinder effective model supervision. To address these challenges, we introduce Refer-Judge, a novel framework that harnesses the reasoning capabilities of Multimodal Large Language Models (MLLMs) to identify and mitigate toxic data. At the core of Refer-Judge is a Jury-and-Judge Chain-of-Thought paradigm, inspired by the deliberative process of the judicial system. This framework targets the root causes of annotation noise: jurors collaboratively assess 3DVG samples from diverse perspectives, providing structured, multi-faceted evaluations. Judges then consolidate these insights …