Graphics and Human Computer Interfaces Commons^™

Stprompt++: Prompting Vision-Language Models For Weakly Supervised Video Anomaly Detection And Fine-Grained Localization, Peng Wu, Chengyu Pan, Guansong Pang, Xiangteng He, Zhiwei Yang, Peng Wang, Yanning Zhang

Research Collection School Of Computing and Information Systems

Traditional weakly supervised video anomaly detection (WSVAD) tasks typically rely on coarse-grained frame-level labels for training. Although this approach reduces annotation costs, it results in weak semantic understanding and spatial localization capabilities due to the absence of fine-grained annotations, hindering precise pixel-level anomaly detection and localization. Thanks to the success of vision-language models (VLMs), e.g., CLIP, recent approaches leveraging large VLMs focus on exploiting their strong semantic understanding capabilities, but they typically feed only keyframes or short video segments into the models, without supplying sufficient prior contextual information (e.g., contextual frames around anomalies, zoomed-in anomaly regions, and detailed anomaly descriptions), …

Generalizing Unsupervised Lidar Motion Estimation Via Patch-Aware Feature Reweighting Under Snowy Conditions, Beibei Zhou, Zhiyuan Zhang, Zhenbo Song, Jianhui Guo, Tongpo Zhang, Shaojing Song, Qinge Wu, Hui Kong

Research Collection School Of Computing and Information Systems

Unsupervised LiDAR motion estimation has attracted significant attention in pattern recognition and robotic vision. However, ensuring robust performance under adverse conditions — particularly structured noise like heavy snowfall — remains a challenging problem. Existing models struggle with domain generalization due to the sensitivity of feature extractors to spatial outliers, limiting their reliability in complex real-world scenarios. In this work, we present an unsupervised LiDAR motion estimation framework via patch-aware feature reweighting to mitigate the domain shift between clear and snowy conditions. Our approach prioritizes reliability-aware reweighting to reduce the influence of snowflake noise and outlier points on pose estimation, while …

Drag As Evidence: Motion-Grounded Latent Recomposition For Drag-Based Editing, Xinyu Pu, Hongsong Wang, Jie Gui, Pan Zhou

Research Collection School Of Computing and Information Systems

Modern image editors excel at semantic manipulation and visual synthesis, yet remain limited in precise spatial control, motivating the development of drag-based editing. However, existing drag-based methods often struggle to balance drag accuracy with natural, plausible, and intent-aligned generation. We propose MoReDrag, a motion-grounded drag-based editing method. Our key insight is to treat pixel-space warping as coarse motion evidence, and to inject this evidence into the generative sampling trajectory. Specifically, MoRe-Drag performs region-aware latent recomposition over refinement, inpainting, and anchor regions, coupled with stage-adaptive conditioning that progressively shifts from motion-grounded structure formation to semantic refinement. We further support an instruction-free …

Defense-To-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks In Vision-Language Models, Yunhan Zhao, Xiang Zheng, Yige Li, Xingjun Ma

Research Collection School Of Computing and Information Systems

Despite their superb capabilities, Vision-Language Models (VLMs) have been shown to be vulnerable to jailbreak attacks. While recent jailbreaks have achieved notable progress, their effectiveness and efficiency can still be improved. In this work, we reveal an interesting phenomenon: incorporating weak defense cues into the attack pipeline can significantly enhance both the effectiveness and efficiency of jailbreaks on VLMs. Building on this insight, we propose Defense2Attack, a novel jailbreak method that bypasses the safety guardrails of VLMs by leveraging defensive patterns to guide jailbreak prompt construction. Specifically, Defense2Attack consists of three key components: (1) a visual optimizer that embeds universal …

Characterizing And Repairing Obsolete Android Gui Tests Under Ui Evolution, Shiwen Song, Yiheng Xiong, Wenbo Guo, Manqi Sun, Jiaolong Kong, Xiaofei Xie

Research Collection School Of Computing and Information Systems

Graphical user interface (GUI) tests are widely used in regression testing of mobile applications (apps) tovalidate app behavior from the user’s perspective. However, frequent app evolution, such as UI redesignsand feature updates, often renders existing GUI tests obsolete, even when underlying functionality remainsunchanged. Automatically repairing such tests is critical for maintaining test suites and reducing substantialmanual effort. Despite the practical importance, there is still a lack of a systematic understanding of thecharacteristics of obsolete GUI tests and a publicly available benchmark to support their study. To fill thisgap, we construct a benchmark comprising 736 obsolete GUI tests collected from 36 …

Restoring Linguistic Grounding In Vla Models Via Train-Free Attention Recalibration, Ninghao Zhang, Bin Zhu, Shijie Zhou, Jingjing Chen

Research Collection School Of Computing and Information Systems

Vision-Language-Action (VLA) models enable robots to perform manipulation tasks directly from natural language instructions and are increasingly viewed as a foundation for generalist robotic policies. However, their reliability under Out-Of-Distribution (OOD) instructions remains underexplored. In this paper, we reveal a critical failure mode in which VLA policies continue executing visually plausible actions even when the language instruction contradicts the scene. We refer to this phenomenon as linguistic blindness, where VLA policies prioritize visual priors over instruction semantics during action generation. To systematically analyze this issue, we introduce ICBench, a diagnostic benchmark constructed from the LIBERO dataset that probes language–action coupling …

Facevalue: Exploring Real-Time Self-View Overlays To Prompt Meaning-Oriented Self-Awareness In Remote Meetings, Gun Woo (Warren) Park, Anthony Tang, Fanny Chevalier

Research Collection School Of Computing and Information Systems

In remote video meetings, visual non-verbal cues, such as facial expressions or head movements, are seen continuously but often only partially. This increases ambiguity compared to in-person settings and can cause misinterpretation or misalignment between intended and perceived meaning. Motivated by communication theories, we designed FaceValue, a technology probe that augments the self-view with private, real-time overlays. These overlays are subtle, suggestive prompts intended to help attendees reflect on how their cues might be interpreted by others. To invite personal interpretation, FaceValue avoids behavioral labeling and instead aims to support meaning-oriented self-awareness: recognizing when visible cues may unintentionally (mis)communicate intent. …

Learning 1-Bit Lidar-Based Localization With Auxiliary Objective, Kaijie Yin, Zhiyuan Zhang, Tian Gao, Wentao Zhu, Cheng-Zhong Xu, Hui Kong

Research Collection School Of Computing and Information Systems

6-DoF LiDAR-based localization is a fundamental capability for autonomous systems operating in large-scale outdoor environments. Many deep-learning-based localization methods have achieved promising performance so far. However, as one of the always-on modules competing for limited on-board computational resources, the localization module is expected to consume only a small portion of the overall compute budget. Most existing learning-based methods are still too heavy for this purpose. In contrast, binary neural networks (BNNs) offer an appealing solution, but the 1-bit compression causes severe information loss and performance drop. In this paper, we address this challenge by proposing Binarized LiDAR-based Localization (BiLoc), the …

Neural Symphony Of Flow Experience: Evidence For High-Dimensional Metastable Dynamics, Abdelrahman B. M. Eldaly, Kris Zhangguang Kang, Fiona Fui-Hoon Nah, Leanne Lai-Hang Chan, Keng Siau, Xiao Fan Liu, Richard Huskey, Langtao Chen, Tejaswini Yelamanchili, Rene Weber

Research Collection School Of Computing and Information Systems

Flow, an optimal experience characterized by deep immersion and engagement in an activity, has been extensively studied in behavioral research. However, its neural dynamic mechanism remains poorly understood. In a within-subject video gaming experiment, we captured neural activity underlying flow, boredom, and anxiety using a 64-channel electroencephalogram (EEG) system. Compared to boredom and anxiety, flow exhibits the highest global functional connectivity, metastability, and dimensionality of dynamic functional connectivity patterns, suggesting that flow is a highly adaptable process that is supported by high-dimensional neural dynamics. Unlike previous studies that focused on identifying static or localized brain activity, we examine the neural …

Generalized Logit Adjustment: Improved Fine-Tuning By Mitigating Label Bias In Zero-Shot Vision Models, Beier Zhu, Qianru Sun, Xun Yang, Hanwang Zhang

Research Collection School Of Computing and Information Systems

Foundation models like CLIP allow zero-shot transfer on various tasks without additional training data. Yet, the zero-shot performance is less competitive than a fully supervised one. Thus, fine-tuning and ensembling are also commonly adopted to better fit the downstream tasks. However, we argue that such prior work has overlooked the inherent biases in foundation models. Due to the highly imbalanced Web-scale training set, foundation models are inevitably skewed toward frequent semantics, and thus the subsequent fine-tuning or ensembling is still biased. In this study, we systematically examine the biases in foundation models and demonstrate the efficacy of our proposed Generalized …

Hvi-Cidnet+: Beyond Extreme Darkness For Low-Light Image Enhancement, Kangbiao Shi, Xiaowen Ma, Yixu Feng, Tao Hu, Peng Wu, Guansong Pang, Qingsen Yan

Research Collection School Of Computing and Information Systems

Low-Light Image Enhancement (LLIE) aims to recover visually pleasing content and details from degraded low-light images. However, existing RGB-based methods often suffer from color bias and brightness artifacts due to inherent high color sensitivity. Although the HSV color space can decouple brightness and color, it introduces noticeable red and black noise artifacts. To address these challenges, we adopt the Horizontal/Vertical-Intensity (HVI) color space for LLIE, which is defined by the HV color map and learnable intensity. The former enforces small distances for red coordinates to alleviate red noise artifacts, while the latter adaptively compresses low-light regions to suppress black noise …

Task-Aligned Haze Removal With Semantic-Aware Fusion And Contrast Self-Correction, Jinbin Wang, Aiping Yang, Guosong Jiang, Wenlong Yu, Dongwei Ren, Qinghua Hu

Research Collection School Of Computing and Information Systems

Adverse haze conditions introduce complex degradations that obscure scene details and distort structural cues critical for object detection, posing persistent challenges for vision‐based sensing systems. Although existing haze removal methods have achieved notable improvements in visual clarity, their optimisation objectives are often misaligned with downstream detection requirements, leading to limited detection performance in real‐world scenarios. To address this issue, this work proposes a task‐aligned weakly supervised haze removal framework, termed Dehaze4Detection, which explicitly aligns low‐level restoration with high‐level detection objectives. The framework incorporates a Semantic‐Aware Multi‐Scale Fusion Module (SMFM) that embeds pixel‐level semantic knowledge into the dehazing process, enabling selective …

Activity Transition Graph Generation: How Far Are We?, Jiakun Liu, Peixin Zhang, Han Hu, Yonghui Liu, Wei Minn, Ferdian Thung, Shahar Maoz, Eran Toch, Debin Gao, David Lo

Research Collection School Of Computing and Information Systems

Android applications (i.e., apps) are indispensable nowadays and are getting bigger and bigger with an increasing number offunctionalities. To understand how to access functionalities in an app, prior studies proposed tools to model the transitionsbetween functionalities with the activity transition graph (ATG). ATG is an important data structure and has been used forvarious Android app analyses, including app design, understanding, and testing. However, there is no benchmarking work onATG generation. It is still unclear whether the transitions identified by tools are correct and how many transitions are missed.To fill this gap, we manually identified all transitions in 98 applications to …

Generation-Augmented Video Corpus Moment Retrieval, Mingjin Kuai, Qianyin Xiao, Juncheng Li, Jin Peng, Lizi Liao, Wei Ji

Research Collection School Of Computing and Information Systems

Video Corpus Moment Retrieval (VCMR) requires models to efficiently retrieve and precisely locate specific moments relevant to natural language queries within a massive, untrimmed video corpus. However, existing discriminative approaches typically rely on shallow visual-textual feature matching mechanisms, which often struggle to capture fine-grained semantic differences. To address this limitation, we propose Video-GAR, a novel framework that reframes the conventional retrieval task from superficial matching to generative understanding, positing that the capability for query reconstruction evidences deep semantic comprehension. Specifically, Video-GAR orchestrates three synergistic components: To overcome the computational efficiency bottleneck, we construct a Bi-Mamba backbone that leverages the linear …

Oscbench: Benchmarking Object State Change In Text-To-Video Generation, Xianjing Han, Bin Zhu, Shiqi Hu, Franklin Mingzhe Li, Patrick Carrington, Roger Zimmermann, Jingjing Chen

Research Collection School Of Computing and Information Systems

Text-to-video (T2V) generation models have made rapid progress in producing visually high-quality and temporally coherent videos. However, existing benchmarks primarily focus on perceptual quality, text–video alignment, or physical plausibility, leaving a critical aspect of action understanding largely unexplored: object state change (OSC) explicitly specified in the text prompt. OSC refers to the transformation of an object’s state induced by an action, such as peeling a potato or slicing a lemon. In this paper, we introduce OSCBench, a benchmark specifically designed to assess OSC performance in T2V models. OSCBench is constructed from instructional cooking data and systematically organizes action–object interactions into …

Tranx-Adapter: Bridging Artifacts And Semantics Within Mllms For Robust Ai-Generated Image Detection, Wenbin Wang, Yuge Huang, Jianqing Xu, Yue Yu, Jiangtao Yan, Shouhong Ding, Pan Zhou, Yong Luo

Research Collection School Of Computing and Information Systems

Rapid advances in AI-generated image (AIGI) technology enable highly realistic synthesis, threatening public information integrity and security. Recent studies have demonstrated that incorporating texture-level artifact features alongside semantic features into multimodal large language models (MLLMs) can enhance their AIGI detection capability. However, our preliminary analyses reveal that artifact features exhibit high intra-feature similarity, leading to an almost uniform attention map after the softmax operation. This phenomenon causes attention dilution, thereby hindering effective fusion between semantic and artifact features. To overcome this limitation, we propose a lightweight fusion adapter, TranX-Adapter, which integrates a Task-aware Optimal-Transport Fusion that leverages the Jensen-Shannon divergence …

Dual-Diffusional Generative Fashion Recommendation, Mingzhe Yu, Lei Wu, Qianru Sun, Yunshan Ma

Research Collection School Of Computing and Information Systems

Personalized generative recommender systems have emerged as a promising solution for fashion recommendation. However, existing methods primarily rely on implicit visual embeddings from historical interactions, which often contain preference-irrelevant information and result in insufficient user behavior modeling. Moreover, these models typically generate only item images, providing limited interpretability. To address these limitations, we propose DualFashion, a Dual-Diffusional Generative Fashion Recommendation Architecture that jointly models image and text modalities for personalized and explainable recommendation. DualFashion adopts a dual-diffusion Transformer with image and text branches, where structured attribute-level captions and visual outfit information are jointly used as conditioning signals to model user …

Chi Meta-Project Ecosystem Overview - Spring 2026, David B. Smith

Publications and Research

This paper offers a high-level account of the Center for Holistic Integration’s (CHI) meta-project ecosystem as visualized in the included system map. CHI provides an organizational structure framed around persistent meta-projects that support and extend individual initiatives across curriculum, scholarly and applied research, infrastructure, artistic production, AI development, cultural inquiry, and external partnerships. Rather than presenting the map as a static inventory of projects, the paper examines how its core domains function as living systems through which knowledge, tools, documentation, participants, and collaborations can accumulate over time. It also considers how CHI-mediated connectivity, institutional integration, and external funding allow the …

Videocreator: An Agentic System For Multi-Turn Video Production, Zhengyang Liang, Yan Shu, Cathal Gurrin, Nicu Sebe, Lizi Liao

Research Collection School Of Computing and Information Systems

Recent advances in video generation models enable visually compelling single clips. However, real-world video creation is inherently continuous and iterative: creators refine content over multiple rounds while maintaining narrative, style, and entity consistency. Existing standalone generators are largely stateless and lack memory of previously generated segments, making it difficult to produce a coherent and consistent video project. To address this gap, we present VideoCreator, a unified video agent that integrates generation and understanding with a project-level memory system. VideoCreator leverages understanding capabilities to perform fine-grained analysis of newly produced content and uses persistent memory to retain and reuse prior context …

Enhancing Pointing Gestures Of Non-Hmd Users In Asymmetric Collocated Mixed Reality Collaboration, Nam-Dang Vo, Van-Vinh Thai, Anthony Tang, Khanh-Duy Le

Research Collection School Of Computing and Information Systems

A common collocated group setting in mixed-reality (MR) collaboration is a person wearing a MR headset (HMD user) and presenting MR contents to audiences who are not provided with such specialized devices (Non-HMD users). In this setting, while Non-HMD users can view the MR environment shown on a large physical display, it still remains challenging for the HMD user to interpret their pointing gesture when they spatially refer to objects in the MR environment. To address this, we designed and evaluated two pointing techniques—SCREEN and SCREEN+SPACE—that support Non-HMD users in referring to MR content. Screen pointing allows users to refer …

“From Remembering To Shaping”: Narrating Shared Experiences By Co-Designing Cultural Heritage Artifacts In Collaborative Vr, Yushang Yang, Fanxu Meng, Fiona Fui-Hoon Nah, L. C. Ray

Research Collection School Of Computing and Information Systems

The ways people remember and recall places reveal an invisible aspect of cultural heritage (CH), reflecting how individuals and communities relate to these places. Heritage is communal, emerging through collaboratively constructed narratives rather than individual records. To probe how people may share collective memories, we designed an immersive two-person workflow for collaboratively co-designing 3D artifacts and environments in virtual heritage locations, using Generative AI (GenAI) to instantiate these intangible memories. Observations of the co-creation process revealed that participants merged prompts and model placements when negotiating different perspectives. They used spatial operations to compose scenes, and also to express personal and …

Not Too Early, Not All At Once: Design Tensions In Ai-Mediated Self-Disclosure In Online Dating, Pei-Hua Tsai, Tianyi Zhang, Emran Bin Elias Poh, Anthony Tang, Yung-Ju Chang

Research Collection School Of Computing and Information Systems

Online dating relies on self-disclosure, yet initial conversations are fragile: users must navigate uncertainty around timing, boundaries, and reciprocity with little shared context. While advances in AI raise the possibility of mediating disclosure, how such support might reshape the experience of early-stage relational disclosure remains underexplored. We conducted 29 semi-structured interviews to examine how daters envision AI-mediated self-disclosure in online dating. Our findings surface recurring design tensions rather than simple opportunities or risks. Participants welcomed guidance that could pace disclosure, support reflection, and reduce social awkwardness, but stressed preserving agency and authorship. They valued interpretive assistance for sense-making of ambiguous …

Happycal: Designing Text And Image-Based Supports For Savouring Positive Work Experiences, Molly Stewart, Minghao Cai, Anthony Tang, Sam Liu, Chris Mosunic, Sowmya Somanath

Research Collection School Of Computing and Information Systems

Savouring positive work experiences can promote positive affect and well-being at work, yet there is limited guidance on how digital applications can support workers to engage in savouring. We developed HappyCal, a work-focused savouring application offering two forms of savouring support: text-based, a common modality in workplace reflection tools, and images, a largely unexplored approach in work-related savouring. We conducted an exploratory qualitative study where participants (N=36) used HappyCal over five days and engaged in savouring through either a text-only modality (n=17) or text input paired with image output (n=19). We found that (1) participants in both groups reported heightened …

A Pruning-Based Question-Answering For Interactive Video Search: A Simple Baseline, Yu Tong Cheng, Phuong Anh Nguyen, Chong-Wah Ngo

Research Collection School Of Computing and Information Systems

There are various factors affecting the performance of video search. An imprecise query will enlarge search space and reduce the discriminative power of ranking functions. This problem is further exacerbated by the presence of numerous visually or semantically similar videos in large datasets. Consequently, users need to painstakingly browse through many highly similar candidates to locate the search target, leading to increased cognitive load and inefficient searching. Ideally, engaging users through interactive questioning to resolve uncertainties in the search process is an effective strategy for progressively narrowing down the search space. However, despite rapid advances in deep learning, generating informative …

Sam3-Litetext: An Anatomical Study Of The Sam3 Text Encoder For Efficient Vision-Language Segmentation, Chengxi Zeng, Yuxuan Jiang, Ge Gao, Shuai Wang, Duolikun Danier, Bin Zhu, Stevan Rudinac, David Bull, Fan Zhang

Research Collection School Of Computing and Information Systems

Vision-language segmentation models such as SAM3 enable flexible, prompt-driven visual grounding, but inherit large, general-purpose text encoders originally designed for open-ended language understanding. In practice, segmentation prompts are short, structured, and semantically constrained, leading to substantial over-provisioning in text encoder capacity and persistent computational and memory overhead. In this paper, we perform a large-scale anatomical analysis of text prompting in vision–language segmentation, covering 404,796 real prompts across multiple benchmarks. Our analysis reveals severe redundancy: most context windows are underutilized, vocabulary usage is highly sparse, and text embeddings lie on a low-dimensional manifold despite high-dimensional representations. Motivated by these findings, we …

Co-Designing With Autistic Livestreamers: Care, Constraints, And Trade-Offs In Livestreaming, Terrance Mok, Anthony Tang, Lora Oehlberg

Research Collection School Of Computing and Information Systems

Autistic livestreamers use platforms like Twitch for social connection, self-expression, and community, but these spaces also impose ongoing social and emotional demands. Prior work has documented these experiences, but less is known about what autistic creators themselves envision for the tools and platforms they use. We address this gap through a Research through Design (RtD) co-design study with three autistic Twitch streamers, using speculative artefacts as discussion prompts to explore how participants reasoned about potential livestreaming technologies. Across three co-design activities, we identify three overarching tensions shaping autistic streaming practice: Expression versus Misinterpretation and Harm; Public Participation versus Control and …

History To Future: Evolving Agent With Experience And Thought For Zero-Shot Vision-And-Language Navigation, Guangzhao Dai, Shuo Wang, Zihan Wang, Guo-Sen Xie, Yang Yang, Jinshan Pan, Qianru Sun, Xiangbo Shu

Research Collection School Of Computing and Information Systems

Vision-and-Language Navigation in Continuous Environment (VLN-CE) requires an agent to follow language instructions to navigate the target destination. With the advancement of large language models (LLMs), recent efforts have explored adapting them for zero-shot VLN-CE, offering a promising solution in addressing the drawbacks of poor generalization in the training-based paradigm. However, existing LLM-based works primarily perform naive reasoning for decision-making and lack feedback, e.g., reviewing historical errors and predicting future potentials. Consequently, it may suffer from continuous failure for those initial error tasks. In this paper, we rethink LLM-based zero-shot VLN-CE and propose a new paradigm, named EvoNav, to improve …

Teacher-Student Diffusion Model For Text-Driven 3d Hand Motion Generation, Ching Lam Cheng, Bin Zhu, Shengfeng He

PhD Student’s Publications Collection

Generating realistic 3D hand motion from natural language is vital for VR, robotics, and human-computer interaction. Existing methods either focus on full-body motion, overlooking detailed hand gestures, or require explicit 3D object meshes, limiting generality. We propose TSHaMo, a model-agnostic teacher-student diffusion framework for text-driven hand motion generation. The student model learns to synthesize motions from text alone, while the teacher leverages auxiliary signals (e.g., MANO parameters) to provide structured guidance during training. A co-training strategy enables the student to benefit from the teacher’s intermediate predictions while remaining text-only at inference. Evaluated using two diffusion backbones on GRAB and H2O, …

It's Not Nde Without U And X: Preparing For Change With Inclusive Research, Sanjeet Mann, Heather L. Cribbs

Library Faculty Publications & Presentations

The upcoming Next Discovery Experience (NDE) introduces major changes to how users search, interpret information, and navigate Primo. Preparing for NDE is an opportunity to center the diverse students and faculty who rely on discovery systems every day, ensuring that their lived experiences, accessibility needs, and research practices guide interface design, configuration, and communication. This session presents a consortial approach to NDE readiness that positions students and community members as partners in the development process. We describe strategies for creating ethical and rigorous UX research workflows that include IRB approval, purposeful recruitment, accessible study design, and clear documentation on the …

Spatial Future Ahead! Augmented Reality And Anticipated Life Consequences, Sergio Barta, Reto Felix, Chris Hinsch, Mahdokht Kalantari, Nina Krey, Philipp A. Rauschnabel

Marketing Faculty Publications

Purpose: This study explores how initial exposure to immersive spatial computing experiences using AR headsets generates lasting inspiration and shapes consumers expected long-term life consequences (i.e., enhancement of reality, perceived substitutability and social impact).

Design/methodology/approach: The study uses a time-lagged research design based on 148 first-time users of spatial computing devices (AR headsets). Respondents were interviewed once shortly after being exposed to AR and a few days later. Data is analyzed using partial least squares structural equation modeling (PLS-SEM).

Findings: Users' immediate “inspired-by” experiences predict increased “inspired-to” intentions days later. Such inspiration translates into anticipated consequences such as virtually customizing …