Graphics and Human Computer Interfaces Commons^™

Hvi: A New Color Space For Low-Light Image Enhancement, Qingsen Yan, Yixu Feng, Cheng Zhang, Guansong Pang, Kangbiao Shi, Peng Wu, Wei Dong, Jinqiu Sun, Yanning Zhang

Research Collection School Of Computing and Information Systems

Low-Light Image Enhancement (LLIE) is a crucial computer vision task that aims to restore detailed visual information from corrupted low-light images. Many existing LLIE methods are based on standard RGB (sRGB) space, which often produce color bias and brightness artifacts due to inherent high color sensitivity in sRGB. While converting the images using Hue, Saturation and Value (HSV) color space helps resolve the brightness issue, it introduces significant red and black noise artifacts. To address this issue, we propose a new color space for LLIE, namely Horizontal/Vertical-Intensity (HVI), defined by polarized HS maps and learnable inten sity. The former enforces …

Event-Based Eye Tracking: Event-Based Vision Workshop 2025, Qinyu Chen, Et. Al.

Research Collection School Of Computing and Information Systems

No abstract provided.

“I Can Run At Night!”: Using Augmented Reality To Support Nighttime Guided Running For Low-Vision Runners, Yuki Abe, Keisuke Matsushima, Kotaro Hara, Daisuke Sakamoto, Tetsuo Ono

Research Collection School Of Computing and Information Systems

Dark environment challenges low-vision (LV) individuals to engage in running by following sighted guide—a Caller-style guided running—due to insufficient illumination, because it prevents them from using their residual vision to follow the guide and be aware about their environment. We design, develop, and evaluate RunSight, an augmented reality (AR)-based assistive tool to support LV individuals to run at night. RunSight combines see-through HMD and image processing to enhance one’s visual awareness of the surrounding environment (e.g., potential hazard) and visualize the guide’s position with AR-based visualization. To demonstrate RunSight’s efficacy, we conducted a user study with 8 LV runners. The …

Worldcuisines: A Massive-Scale Benchmark For Multilingual And Multicultural Visual Question Answering On Global Cuisines, Genta Indra Winata, Et. Al

Research Collection School Of Computing and Information Systems

Vision Language Models (VLMs) often struggle with culture-specific knowledge, particularly in languages other than English and in underrepresented cultural contexts. To evaluate their understanding of such knowledge, we introduce WorldCuisines, a massive-scale benchmark for multilingual and multicultural, visually grounded language understanding. This benchmark includes a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects, spanning 9 language families and featuring over 1 million data points, making it the largest multicultural VQA benchmark to date. It includes tasks for identifying dish names and their origins. We provide evaluation datasets in two sizes (12k and 60k instances) alongside …

Oscar: Object Status And Contextual Awareness For Recipes To Support Non-Visual Cooking, Franklin Mingzhe Li, Kaitlyn Ng, Bin Zhu, Patrick Carrington

Research Collection School Of Computing and Information Systems

Following recipes while cooking is an important but difficult task for visually impaired individuals. We developed OSCAR (Object Status Context Awareness for Recipes), a novel approach that provides recipe progress tracking and context-aware feedback on the completion of cooking tasks through tracking object statuses. OSCAR leverages both Large-Language Models (LLMs) and Vision-Language Models (VLMs) to manipulate recipe steps, extract object status information, align visual frames with object status, and provide cooking progress tracking log. We evaluated OSCAR’s recipe following functionality using 173 YouTube cooking videos and 12 real-world non-visual cooking videos to demonstrate OSCAR’s capability to track cooking steps and …

Exploring The Potential Of Large Language Models For Heterophilic Graphs, Yuxia Wu, Shujie Li, Yuan Fang, Chuan Shi

Research Collection School Of Computing and Information Systems

Large language models (LLMs) have presented significant opportunities to enhance various machine learning applications, including graph neural networks (GNNs). By leveraging the vast open-world knowledge within LLMs, we can more effectively interpret and utilize textual data to better characterize heterophilic graphs, where neighboring nodes often have different labels. However, existing approaches for heterophilic graphs overlook the rich textual data associated with nodes, which could unlock deeper insights into their heterophilic contexts. In this work, we explore the potential of LLMs for modeling heterophilic graphs and propose a novel two-stage framework: LLM-enhanced edge discriminator and LLM-guided edge reweighting. In the first …

Guest Editorial: When Multimedia Meets Food: Multimedia Computing For Food Data Analysis And Applications, Weiqing Min, Shuqiang Jiang, Petia Radeva, Vladimir Pavlovic, Chong-Wah Ngo, Kiyoharu Aizawa, Wanqing Li

Research Collection School Of Computing and Information Systems

Food is central in our life for its fundamental role in our survival, health, mood and culture. The deployment of various networks (e.g., IoT and mobile networks), devices (e.g., hyperspectral imaging devices, electronic nose/tongue), databases (e.g., nutrition tables and food compositional databases), recipe-sharing websites (e.g., Yummly and Meishijie) and social media (e.g., Twitter and Weibo) has generated unprecedented volumes of multi-modal food data. Such multi-source multi-modal food data provides new perspectives to analyze and understand food consumption via multimedia computing. Riding on the wave of AI, food-oriented multimedia computing integrates AI, multimedia technology and food science to enable a wide …

Sans: Efficient Densest Subgraph Discovery Over Relational Graphs Without Materialization, Yudong Niu, Yuchen Li, Jiaxin Jiang, Laks V. S. Lakshmanan

Research Collection School Of Computing and Information Systems

How can we efficiently identify the densest subgraph over relational graphs? Existing dense subgraph discovery (DSD) approaches assume that a relational graph H is already derived from a heterogeneous data source and they focus on efficient discovery of the densest subgraph on the materialized H. Unfortunately, materializing relational graphs can be resource-intensive, which thus limits the practical usefulness of existing algorithms over large datasets. To mitigate this, we propose a novel Summary-bAsed deNsest Subgraph discovery (SANS) system. Our unique summary-based peeling algorithm forms the core of SANS. Following the peeling paradigm, it utilizes summaries of each node's neighborhood to efficiently …

Cyberoception: Finding A Painlessly-Measurable New Sense In The Cyberworld Towards Emotion-Awareness In Computing, Tadashi Okoshi, Zexiong Gao, Yi Zhen Tan, Takumi Karasawa, Takeshi Miki, Wataru Sasaki, Rajesh Krishna Balan

Research Collection School Of Computing and Information Systems

In Affective computing, recognizing users’ emotions accurately is the basis of affective human–computer interaction. Understanding users’ interoception contributes to a better understanding of individually different emotional abilities, which is essential for achieving inter-individually accurate emotion estimation. However, existing interoception measurement methods, such as the heart rate discrimination task, have several limitations, including their dependence on a well-controlled laboratory environment and precision apparatus, making monitoring users’ interoception challenging. This study aims to determine other forms of data that can explain users’ interoceptive or similar states in their real-world lives and propose a novel hypothetical concept “cyberoception,” a new sense (1) which …

Few-Shot Learning On Graphs: From Meta-Learning To Llm-Empowered Pre-Training And Beyond, Yuan Fang, Yuxia Wu, Xingtong Yu, Shirui Pan

Research Collection School Of Computing and Information Systems

Graph representation learning has become central to many graph-based tasks, driving advancements in various domains such as web search, recommendation systems, and social network analysis. Traditionally, these methods rely on end-to-end supervised learning paradigms that require abundant labeled data, which can be costly and difficult to obtain. To address this limitation, few-shot learning on graphs has emerged as a promising approach, allowing models to generalize with minimal supervision and overcome data scarcity in real-world applications. This tutorial offers an in-depth exploration of recent advancements in few-shot learning for graphs, providing a comparative analysis of state-of-the-art methods and identifying future research …

Sita: Structurally Imperceptible And Transferable Adversarial Attacks For Stylized Image Generation, Jingdan Kang, Haoxin Yang, Yan Cai, Huaidong Zhang, Xuemiao Xu, Yong Du, Shengfeng He

Research Collection School Of Computing and Information Systems

Image generation technology has brought significant advancements across various fields but has also raised concerns about data misuse and potential rights infringements, particularly with respect to creating visual artworks. Current methods aimed at safeguarding artworks often employ adversarial attacks. However, these methods face challenges such as poor transferability, high computational costs, and the introduction of noticeable noise, which compromises the aesthetic quality of the original artwork. To address these limitations, we propose a Structurally Imperceptible and Transferable Adversarial (SITA) attacks. SITA leverages a CLIP-based destylization loss, which decouples and disrupts the robust style representation of the image. This disruption hinders …

Minimum Multi-Service Fleet Size Problem: Shareability Graph And Network Flow Approach, Dingtong Yang, Yubin Liu, Hai Wang, Jinhua Zhao, Hamsa Balakrishnan

Research Collection School Of Computing and Information Systems

On-demand, vehicle-based services—such as ride-hailing, food, grocery, and parcel delivery—have become ubiquitous over the past decade. These services can be categorized into four types (Sun et al., 2023): passenger mobility, goods delivery, information acquisition (e.g., probe vehicle for traffic conditions), and mobile server (e.g., vehicle displaying advertisements). Passenger mobility and goods delivery are typically fulfilled by separate fleets, each dedicated to a single service. However, if various services can be pooled and handled simultaneously by a multi-functional fleet while maintaining service quality, the total number of required vehicles and overall vehicle mileage could be significantly reduced. This exciting potential motivates …

Samgpt: Text-Free Graph Foundation Model For Multi-Domain Pre-Training And Cross-Domain Adaptation, Xingtong Yu, Zechuan Gong, Chang Zhou, Yuan Fang, Hui Zhang

Research Collection School Of Computing and Information Systems

Graphs are able to model interconnected entities in many online services, supporting a wide range of applications on the Web. This raises an important question: How can we train a graph foundational model on multiple source domains and adapt to an unseen target domain? A major obstacle is that graphs from different domains often exhibit divergent characteristics. Some studies leverage large language models to align multiple domains based on textual descriptions associated with the graphs, limiting their applicability to text-attributed graphs. For text-free graphs, a few recent works attempt to align different feature distributions across domains, while generally neglecting structural …

Node-Time Conditional Prompt Learning In Dynamic Graphs, Xingtong Yu, Zhenghao Liu, Xinming Zhang, Yuan Fang

Research Collection School Of Computing and Information Systems

Dynamic graphs capture evolving interactions between entities, such as in social networks, online learning platforms, and crowdsourcing projects. For dynamic graph modeling, dynamic graph neural networks (DGNNs) have emerged as a mainstream technique. However, they are generally pre-trained on the link prediction task, leaving a significant gap from the objectives of downstream tasks such as node classification. To bridge the gap, prompt-based learning has gained traction on graphs, but most existing efforts focus on static graphs and neglect the evolution of dynamic graphs. In this paper, we propose DYGPROMPT, a novel pre-training and prompt learning framework for dynamic graph modeling. …

Towards Real-World Unsupervised Anomaly Detection For Images, Zhonghang Liu

Dissertations and Theses Collection (Open Access)

In the era of big data, data quality plays a critical role in computer vision, where the reliability and purity of training images are essential for optimal performance. When training models such as image classifiers and object detectors, the quality of the training data directly influences the success of the model. In other words, if the training dataset is contaminated, the model’s performance might accordingly decrease.

To address this challenge, unsupervised anomaly detection (UAD) has become an attractive research area. By automatically removing these anomalous data points, UAD can help improve the accuracy and robustness of machine learning models in …

Temporal Relational Graph Convolutional Networks For Financial Applications, Brindha Priyadarshini Jeyaraman

Dissertations and Theses Collection (Open Access)

The financial industry operates within a highly dynamic and interconnected ecosystem, presenting unique challenges for predictive modeling and decision-making. Accurately forecasting financial performance, assessing credit risk, detecting fraud, and ensuring compliance require methodologies that can capture complex temporal, relational, and contextual dependencies within financial data. This thesis investigates the use of Temporal Relational Graph Convolutional Networks (TRGCNs) combined with financial knowledge graphs (FKGs) to address these challenges and enable advanced analytics in the financial domain. We introduce FintechKG, a financial knowledge graph constructed through a threedimensional information extraction process, incorporating entities, temporal dimensions, and domain-specific financial relationships. A TRGCN-based framework …

One-For-All: Towards Universal Domain Translation With A Single Stylegan, Yong Du, Jiahui Zhan, Xinzhe Li, Junyu Dong, Sheng Chen, Ming-Hsuan Yang, Shengfeng He

Research Collection School Of Computing and Information Systems

In this paper, we propose a novel translation model, UniTranslator, for transforming representations between visually distinct domains under conditions of limited training data and significant visual differences. The main idea behind our approach is leveraging the domain-neutral capabilities of CLIP as a bridging mechanism, while utilizing a separate module to extract abstract, domain-agnostic semantics from the embeddings of both the source and target realms. Fusing these abstract semantics with target-specific semantics results in a transformed embedding within the CLIP space. To bridge the gap between the disparate worlds of CLIP and StyleGAN, we introduce a new non-linear mapper, the CLIP2P …

Open-Set Graph Anomaly Detection Via Normal Structure Regularisation, Qizhou Wang, Guansong Pang, Mahsa Salehi, Xiaokun Xia, Christopher Leckie

Research Collection School Of Computing and Information Systems

This paper considers an important Graph Anomaly Detection (GAD) task, namely open-set GAD, which aims to train a detection model using a small number of normal and anomaly nodes (referred to as *seen anomalies*) to detect both seen anomalies and *unseen anomalies* (*i.e*., anomalies that cannot be illustrated the training anomalies). Those labelled training data provide crucial prior knowledge about abnormalities for GAD models, enabling substantially reduced detection errors. However, current supervised GAD methods tend to over-emphasise fitting the seen anomalies, leading to many errors of detecting the unseen anomalies as normal nodes. Further, existing open-set AD models were introduced …

Hand1000: Generating Realistic Hands From Text With Only 1,000 Images, Haozhuo Zhang, Bin Zhu, Yu Cao, Yanbin Hao

Research Collection School Of Computing and Information Systems

Text-to-image generation models have achieved remarkable advancements in recent years, aiming to produce realistic images from textual descriptions. However, these models often struggle with generating anatomically accurate representations of human hands. The resulting images frequently exhibit issues such as incorrect numbers of fingers, unnatural twisting or interlacing of fingers, or blurred and indistinct hands. These issues stem from the inherent complexity of hand structures and the difficulty in aligning textual descriptions with precise visual depictions of hands. To address these challenges, we propose a novel approach named Hand1000 that enables the generation of realistic hand images with target gesture using …

Neurovig: Integrating Event Cameras For Resource-Efficient Video Grounding, Dulanga Weerakoon, Vigneshwaran Subbaraju, Joo Hwee Lim, Archan Misra

Research Collection School Of Computing and Information Systems

Spatio-Temporal Video Grounding (STVG) - the task of identifying the target object in the field-of-view that the language instruction refers to - is a fundamental vision-language task. Current STVG approaches typically utilize feeds from an RGB camera that is assumed to be always-on and process the video frames using complex neural network pipelines. As a result they often impose prohibitive system overheads (energy latency) on pervasive devices. To address this we propose NeuroViG with two key innovations: (a) leveraging on event streams from a low-power neuromorphic event camera sensor to perform selective triggering of the more energy-hungry RGB camera for …

Personamagic: Stage-Regulated High-Fidelity Face Customization With Tandem Equilibrium, Xinzhe Li, Jiahui Zhan, Shengfeng He, Yangyang Xu, Junyu Dong, Huaidong Zhang, Yong Du

Research Collection School Of Computing and Information Systems

Personalized image generation has made significant strides in adapting content to novel concepts. However, a persistent challenge remains: balancing the accurate reconstruction of unseen concepts with the need for editability according to the prompt, especially when dealing with the complex nuances of facial features. In this study, we delve into the temporal dynamics of the text-to-image conditioning process, emphasizing the crucial role of stage partitioning in introducing new concepts. We present PersonaMagic, a stage-regulated generative technique designed for high-fidelity face customization. Using a simple MLP network, our method learns a series of embeddings within a specific timestep interval to capture …

Graph Foundation Models: Concepts, Opportunities And Challenges, Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Fang Yuan, Lichao Sun, Philip S. Yu, Chuan Shi

Research Collection School Of Computing and Information Systems

Foundation models have emerged as critical components in a variety of artificial intelligence applications, and showcase significant success in natural language processing and several other domains. Meanwhile, the field of graph machine learning is witnessing a paradigm transition from shallow methods to more sophisticated deep learning approaches. The capabilities of foundation models in generalization and adaptation motivate graph machine learning researchers to discuss the potential of developing a new graph learning paradigm. This paradigm envisions models that are pre-trained on extensive graph data and can be adapted for various graph tasks. Despite this burgeoning interest, there is a noticeable lack …

Occlusion-Insensitive Talking Head Video Generation Via Facelet Compensation, Yuhui Deng, Yuqin Lu, Yangyang Xu, Yongwei Nie, Shengfeng He

Research Collection School Of Computing and Information Systems

Talking head video generation involves animating a still face image using facial motion cues derived from a driving video to replicate target poses and expressions. Traditional methods often rely on the assumption that the relative positions of facial keypoints remain unchanged. However, this assumption fails when keypoints are occluded or when the head is in a profile pose, leading to inconsistencies in identity and blurring in certain facial regions. In this paper, we introduce Occlusion-Insensitive Talking Head Video Generation, a novel approach that eliminates the reliance on spatial correlation of keypoints and instead leverages semantic correlation. Our method transforms facial …

Improving Multimodal Human Pose Estimation By Adversarial Modality Enhancement, Jiangnan Xia, Qilong Wu, Yanyin Guo, Yi Li, Jianghan Cheng, Junwei Li, Zhiyuan Zhang

Research Collection School Of Computing and Information Systems

Human pose estimation in computer vision predominantly focuses on the visible modality, with limited research on the infrared modality. No existing methods demonstrate robust performance across both modalities, missing their complementary strengths. This gap arises from the lack of a multimodal benchmark and the difficulty of developing robust multimodal capabilities. To address this, we introduce MMPD, a novel visible-infrared multimodal pose benchmark with high-quality annotations for both modalities. Leveraging MMPD, we expose the limitations of state-of-the-art methods due to modality variance. To overcome this challenge, we propose a novel method-agnostic scheme called AMMPE. By employing the Modality Adversarial Enhancement Stage …

Imageinthat: Manipulating Images To Convey User Instructions To Robots, Karthik Mahadevan, Blaine Lewis, Jiannan Li, Bilge Mutlu, Anthony Tang, Tovi Grossman

Research Collection School Of Computing and Information Systems

Foundation models are rapidly improving the capability of robots in performing everyday tasks autonomously such as meal preparation, yet robots will still need to be instructed by humans due to model performance, the difficulty of capturing user preferences, and the need for user agency. Robots can be instructed using various methods---natural language conveys immediate instructions but can be abstract or ambiguous, whereas end-user programming supports longer-horizon tasks but interfaces face difficulties in capturing user intent. In this work, we propose using direct manipulation of images as an alternative paradigm to instruct robots, and introduce a specific instantiation called ImageInThat which …

Lightprof: A Lightweight Reasoning Framework For Large Language Model On Knowledge Graph, Tu Ao, Yanhua Yu, Yuling Wang, Yang Deng, Zirui Guo, Liang Pang, Pinghui Wang, Tat-Seng Chua, Xiao Zhang, Zhen Cai

Research Collection School Of Computing and Information Systems

Large Language Models (LLMs) have impressive capabilities in text understanding and zero-shot reasoning. However, delays in knowledge updates may cause them to reason incorrectly or produce harmful results. Knowledge Graphs (KGs) provide rich and reliable contextual information for the reasoning process of LLMs by structurally organizing and connecting a wide range of entities and relations. Existing KG-based LLM reasoning methods only inject KGs’ knowledge into prompts in a textual form, ignoring its structural information. Moreover, they mostly rely on close-source models or open-source models with large parameters, which poses challenges to high resource consumption. To address this, we propose a …

Adversarial Attacks On Event-Based Pedestrian Detectors: A Physical Approach, Guixu Lin, Muyao Niu, Qingtian Zhu, Zhengwei Yin, Zhuoxiao Li, Shengfeng He, Yinqiang Zheng

Research Collection School Of Computing and Information Systems

Event cameras, known for their low latency and high dynamic range, show great potential in pedestrian detection applications. However, while recent research has primarily focused on improving detection accuracy, the robustness of event-based visual models against physical adversarial attacks has received limited attention. For example, adversarial physical objects, such as specific clothing patterns or accessories, can exploit inherent vulnerabilities in these systems, leading to misdetections or misclassifications. This study is the first to explore physical adversarial attacks on event-driven pedestrian detectors, specifically investigating whether certain clothing patterns worn by pedestrians can cause these detectors to fail, effectively rendering them unable …

Seven Hci Grand Challenges Revisited: Five-Year Progress, Constantine Stephanidis, Gavriel Salvendy, Margherita Antona, Vincent G Duffy, Qin Gao, Waldemar Karwowski, Fiona Nah, Stavroula Ntoa, Pei-Luen Patrick Rau, Keng Siau, Jia Zhou

Research Collection School Of Computing and Information Systems

Motivated by the rapid technological advancements achieved in the last five years, and the pervasiveness of Artificial Intelligence, the paper investigates the evolving role of Human-Computer Interaction and revisits the seven grand challenges outlined in 2019: human-technology symbiosis, human-environment interactions, ethics, privacy and security, well-being, health and eudaimonia, accessibility and universal access, learning and creativity, and social organization and democracy. Through literature analysis, the paper reevaluates the status of each challenge and highlights emerging requirements. Key findings reveal the widespread impact of Artificial Intelligence across all domains and emphasize the need for improved AI transparency, alignment with human values, and …

Learning An Interpretable Stylized Subspace For 3d-Aware Animatable Artforms, Chenxi Zheng, Bangzhen Liu, Xuemiao Xu, Huaidong Zhang, Shengfeng He

Research Collection School Of Computing and Information Systems

Throughout history, static paintings have captivated viewers within display frames, yet the possibility of making these masterpieces vividly interactive remains intriguing. This research paper introduces 3DArtmator, a novel approach that aims to represent artforms in a highly interpretable stylized space, enabling 3D-aware animatable reconstruction and editing. Our rationale is to transfer the interpretability and 3D controllability of the latent space in a 3D-aware GAN to a stylized sub-space of a customized GAN, revitalizing the original artforms. To this end, the proposed two-stage optimization framework of 3DArtmator begins with discovering an anchor in the original latent space that accurately mimics the …

Density Boosts Everything: A One-Stop Strategy For Improving Performance, Robustness, And Sustainability Of Malware Detectors, Jianwen Tian, Wei Kong, Debin Gao, Tong Wang, Taotao Gu, Kefan Qiu, Zhi Wang, Xiaohui Kuang

Research Collection School Of Computing and Information Systems

In the contemporary landscape of cybersecurity, AI-driven detectors have emerged as pivotal in the realm of malware detection. However, existing AI-driven detectors encounter a myriad of challenges, including poisoning attacks, evasion attacks, and concept drift, which stem from the inherent characteristics of AI methodologies. While numerous solutions have been proposed to address these issues, they often concentrate on isolated problems, neglecting the broader implications for other facets of malware detection. This paper diverges from the conventional approach by not targeting a singular issue but instead identifying one of the fundamental causes of these challenges, sparsity. Sparsity refers to a scenario …