Digital Commons Network^™

A Pruning-Based Question-Answering For Interactive Video Search: A Simple Baseline, Yu Tong Cheng, Phuong Anh Nguyen, Chong-Wah Ngo

Research Collection School Of Computing and Information Systems

There are various factors affecting the performance of video search. An imprecise query will enlarge search space and reduce the discriminative power of ranking functions. This problem is further exacerbated by the presence of numerous visually or semantically similar videos in large datasets. Consequently, users need to painstakingly browse through many highly similar candidates to locate the search target, leading to increased cognitive load and inefficient searching. Ideally, engaging users through interactive questioning to resolve uncertainties in the search process is an effective strategy for progressively narrowing down the search space. However, despite rapid advances in deep learning, generating informative …

Rode: Linear Rectified Mixture Of Diverse Experts For Food Large Multi-Modal Models, Pengkun Jiao, Xinlan Wu, Bin Zhu, Jingjing Chen, Chong-Wah Ngo, Yu-Gang

Research Collection School Of Computing and Information Systems

Large Multi-modal Models (LMMs) have significantly advanced a variety of vision-language tasks. The scalability and availability of high-quality training data play a pivotal role in the success of LMMs. In the realm of food, while comprehensive food datasets such as Recipe1M offer an abundance of ingredient and recipe information, they often fall short of providing ample data for nutritional analysis. The Recipe1M+ dataset, despite offering a subset for nutritional evaluation, is limited in the scale and accuracy of nutrition information. To bridge this gap, we introduce Uni-Food, a unified food dataset that comprises over 100,000 images with various food labels, …

Benchmarking Gaslighting Negation Attacks Against Multimodal Large Language Models, Bin Zhu, Yinxuan Gui, Huiyan Qi, Jingjing Chen, Chong-Wah Ngo, Ee-Peng Lim

Research Collection School Of Computing and Information Systems

Multimodal Large Language Models (MLLMs) have exhibited remarkable advancements in integrating different modalities, excelling in complex understanding and generation tasks. Despite their success, MLLMs remain vulnerable to conversational adversarial inputs. In this paper, we systematically study gaslighting negation attacks—a phenomenon where models, despite initially providing correct answers, are persuaded by user-provided negations to reverse their outputs, often fabricating justifications. We conduct extensive evaluations of state-of-the-art MLLMs across diverse benchmarks and observe substantial performance drops when negation is introduced. Notably, we introduce the first benchmark GaslightingBench, specifically designed to evaluate the vulnerability of MLLMs to negation arguments. GaslightingBench consists of multiple-choice …

Sam3-Litetext: An Anatomical Study Of The Sam3 Text Encoder For Efficient Vision-Language Segmentation, Chengxi Zeng, Yuxuan Jiang, Ge Gao, Shuai Wang, Duolikun Danier, Bin Zhu, Stevan Rudinac, David Bull, Fan Zhang

Research Collection School Of Computing and Information Systems

Vision-language segmentation models such as SAM3 enable flexible, prompt-driven visual grounding, but inherit large, general-purpose text encoders originally designed for open-ended language understanding. In practice, segmentation prompts are short, structured, and semantically constrained, leading to substantial over-provisioning in text encoder capacity and persistent computational and memory overhead. In this paper, we perform a large-scale anatomical analysis of text prompting in vision–language segmentation, covering 404,796 real prompts across multiple benchmarks. Our analysis reveals severe redundancy: most context windows are underutilized, vocabulary usage is highly sparse, and text embeddings lie on a low-dimensional manifold despite high-dimensional representations. Motivated by these findings, we …

Frozen Lvlms For Micro-Video Recommendation: A Systematic Study Of Feature Extraction And Fusion, Huatuan Sun, Yunshan Ma, Changguang Wu, Yanxin Zhang, Pengfei Wang, Xiaoyu Du

Research Collection School Of Computing and Information Systems

Frozen Large Video Language Models (LVLMs) are increasingly employed in micro-video recommendation (MVR) due to their strong multimodal understanding. However, existing apporches typically deploy LVLMs as fixed black-box feature extractors without systematically comparing alternative representation strategies. To address this gap, we present the first systematic empirical study on various feature extraction paradigms and integration strategies, along with hierarchical representations from frozen LVLMs for MVR. Extensive experiments on representative LVLMs reveal that hidden states from multiple decoder layers provide richer and more effective representations for MVR. Guided by this insight, we propose the Dual Feature Fusion (DFF) Framework, a lightweight approach …

Co-Designing With Autistic Livestreamers: Care, Constraints, And Trade-Offs In Livestreaming, Terrance Mok, Anthony Tang, Lora Oehlberg

Research Collection School Of Computing and Information Systems

Autistic livestreamers use platforms like Twitch for social connection, self-expression, and community, but these spaces also impose ongoing social and emotional demands. Prior work has documented these experiences, but less is known about what autistic creators themselves envision for the tools and platforms they use. We address this gap through a Research through Design (RtD) co-design study with three autistic Twitch streamers, using speculative artefacts as discussion prompts to explore how participants reasoned about potential livestreaming technologies. Across three co-design activities, we identify three overarching tensions shaping autistic streaming practice: Expression versus Misinterpretation and Harm; Public Participation versus Control and …

Group Conversational Agents: A Review Of Designs That Support And Shape Group Interaction, Shunyi Yeo, Tianyi Zhang, Scott Bateman, Gary Hsieh, Young-Ho Kim, Simon Tangi Perrault, Jiannan Li, Anthony Tang

Research Collection School Of Computing and Information Systems

Conversational agents that participate in or mediate group interaction introduce challenges that extend beyond supporting individual users, raising new questions about how agents participate in and influence groups. To characterise this emerging design space, we present a systematic review of 53 peer-reviewed studies on group conversational agents (GCAs). We analyse how GCAs intervene in group-level processes, including participation regulation, conflict mediation, task alignment, and execution support. Using concepts from group research as an analytic lens, we organise prior GCA work around recurring group interactional challenges (orientation, conflict, alignment, and execution), and examine the roles agents are designed to play in …

Language Embeddings Meet Shallow Autoencoders, Rodrigo Alves, Vojtěch Vančura, Pavel Kordík, Antoine Ledent

Research Collection School Of Computing and Information Systems

Shallow autoencoders are appealing recommenders due to their simplicity, scalability, and competitive retrieval quality, but they struggle in strict cold-start settings where new items have no interactions. We propose an inductive shallow autoencoder that leverages item side information (language embeddings) by fixing the decoder to item features and learning only an encoder in the same semantic space. To prevent trivial self-reconstruction without enforcing a hard zero diagonal, we introduce diagonal gating: a leave-one-item-out objective that blocks the self-copy shortcut only for the item being updated while retaining context from the rest of the user history. An alternating-style optimization trains the …

Ppg-Sport: A Dataset For Reliable Heart Rate Monitoring From Wrist Ppg Under Dynamic Sports Conditions, Changshuo Hu, Hung Manh Pham, Yiming Zhang, Guanru Yan, Xiao Ma, Yuezhong Wu, Thivya Kandappu, Archan Misra, Dong Ma

Research Collection School Of Computing and Information Systems

Photoplethysmography (PPG) has become a cornerstone of physiological sensing in wearable devices, enabling non-invasive monitoring of heart rate and related biomarkers. However, its reliability deteriorates sharply under dynamic, high-intensity, or non-periodic motions such as those in sports, where existing datasets fail to capture realistic wrist dynamics. To address this gap, we introduce PPG-Sport, the first large-scale dataset designed for heart rate monitoring from wrist-worn PPG under real sports conditions. The PPG-Sport dataset includes synchronized PPG, inertial measurement unit (IMU), and electrocardiography (ECG) recordings from both wrists of 30 participants across six representative activities: stationary, walking, running, badminton, table tennis, and …

A Survey Of Earable Technology: Trends, Tools, And The Road Ahead, Changshuo Hu, Qiang Yang, Yang Liu, Tobias Röddiger, Kayla-Jade Butkow, Mathias Ciliberto, Adam Luke Pullin, Jake Stuchbury-Wass, Mahbub Hassan, Cecilia Mascolo, Dong Ma

Research Collection School Of Computing and Information Systems

Earable devices, wearables positioned in or around the ear, are undergoing a rapid transformation from audio-centric accessories into multifunctional systems for interaction, contextual awareness, and health monitoring. This evolution is driven by commercial trends emphasizing sensor integration and by a surge of academic interest exploring novel sensing capabilities. Building on the foundation established by earlier surveys, this work presents a timely and comprehensive review of earable research published since 2022. We attempt to answer three core questions: (1) how has earable research evolved in recent years, (2) what enabling resources are now available, and (3) what opportunities remain for future …

Anatomical Domain Shifts: Test-Time Heterogeneous Adaptation For 3d Human Pose Prediction, Qiongjie Cui, Pan Zhou, Jingjing Chen, Na Zhao

Research Collection School Of Computing and Information Systems

The research frontier in human pose prediction (HPP) is advancing toward continual test-time adaptation (TTA), where models must self-adapt to dynamic test distributions. To date, the homeostatic continual TTA remains the sole viable solution, which isolates the model parameters and update domain-sensitive ones. Despite mitigating full-body domain gaps, human anatomical heterogeneity (domain shifts often localize to specific regions) is ignored. This anatomical-agnostic approach forces uniform parameter adaptation across kinematically distinct segments, causing: over-adaptation of stable regions and under-adaptation of shift-prone articulations. To address it, we introduce TT-HA, a novel Test-Time Heterogeneous Adaptation that implicitly estimates domain changes for anatomical segments, …

History To Future: Evolving Agent With Experience And Thought For Zero-Shot Vision-And-Language Navigation, Guangzhao Dai, Shuo Wang, Zihan Wang, Guo-Sen Xie, Yang Yang, Jinshan Pan, Qianru Sun, Xiangbo Shu

Research Collection School Of Computing and Information Systems

Vision-and-Language Navigation in Continuous Environment (VLN-CE) requires an agent to follow language instructions to navigate the target destination. With the advancement of large language models (LLMs), recent efforts have explored adapting them for zero-shot VLN-CE, offering a promising solution in addressing the drawbacks of poor generalization in the training-based paradigm. However, existing LLM-based works primarily perform naive reasoning for decision-making and lack feedback, e.g., reviewing historical errors and predicting future potentials. Consequently, it may suffer from continuous failure for those initial error tasks. In this paper, we rethink LLM-based zero-shot VLN-CE and propose a new paradigm, named EvoNav, to improve …

Adaptive Outlier Detection Over Data Stream, Rui Zhu, Mingyuan Jiang, Xiaochun Yang, Baihua Zheng, Bin Wang, Tao Qiu

Research Collection School Of Computing and Information Systems

Continuous distance-based outlier detection in streaming data poses significant challenges and has a wide range of practical applications. Traditional threshold-based methods perform well under stable streaming conditions, where fixed parameters remain effective. However, they often struggle with dynamic data distributions and high stream speeds, leading to suboptimal performance, limited control over the number of returned outliers, and failure to meet real-time detection requirements. To address these issues, this paper introduces a novel Recall and Proportion-Aware Outlier Detection (RPA-OD) query. In RPA-OD, ρ defines a distance relaxation that enables real-time outlier detection. Specifically, objects with fewer than k neighbors within the …

“Alexa, Do Not Say That In Front Of My Boss!” A Cross-Cultural Comparison Of User And Ai Preferences For Privacy-Aware Smart Speaker Interactions Across Contexts, Lynne Warin, Emily Aurelia, Anthony Tang, Emily Aurelia, Delphine Reinhardt

Research Collection School Of Computing and Information Systems

Due to their limited ability to reason about the social context in which they are used, smart speakers pose significant privacy risks by responding in ways that may violate people's implicit social boundaries. We conducted a cross-cultural vignette study (N = 944) in Germany and Singapore to investigate how situational factors—specifically social context (bystander relationships and closeness), physical context (location), and interaction context (topic and deceptive intent)—regulate user preferences for smart speaker responses. Our results demonstrate that these factors are superior predictors of response preferences than dispositional user traits (i.e., intrinsic personal traits). We identify two distinct social dynamics: a …

Cfalr: Collaborative Filtering-Augmented Large Language Model For Personalized Fashion Outfit Recommendation, Yujuan Ding, Junrong Liao, Yunshan Ma, Yi Bin, Wenqi Fan, Tat-Seng Chua, Qing Li

Research Collection School Of Computing and Information Systems

Personalized outfit recommendation poses a significant challenge in e-commerce and social media platforms, requiring systems that balance user preferences with aesthetic compatibility. Collaborative filtering (CF) provides a traditional solution for this, but it struggles with data-sparse scenarios and complex user-item-outfit relationships. Meanwhile, existing template-based approaches are constrained by rigid pre-designed structures. To bridge these research gaps, we introduce CFALR (Collaborative Filtering-Augmented Large Language Model for Recommendation), a novel framework that synergizes collaborative filtering with large language models for personalized outfit recommendation. Specifically, CFALR describes user-outfit interactions in natural language and leverages LLMs to capture fashion semantics while employing CF-enhanced embeddings …

How Do Machine Learning Models Change?, Joel Castaño, Rafael Cabañas, Antonio Salmerón, David Lo, Silverio Martínez-Fernández

Research Collection School Of Computing and Information Systems

The proliferation of Machine Learning (ML) models and their open source implementations has transformed AI research and applications. Platforms like Hugging Face (HF) enable this evolving ecosystem, yet a large-scale longitudinal study of how these models change is lacking. This study addresses this gap by analyzing over 680,000 commits from 100,000 models and 2,251 releases from 202 of these models on HF using repository mining and longitudinal methods. We apply an extended ML change taxonomy to classify commits and use Bayesian networks to model temporal patterns in commit and release activities. Our findings show that commit activities align with established …

Expanding Links Between Work Values And Occupations: Development Of The Occupational Values Inventory (Ovi), Nicholas F. Heimpel, Kevin Hoff, Zihan Liu, Federick L. Oswald, James Rounds

Research Collection Lee Kong Chian School Of Business

This research developed the Occupational Values Inventory (OVI), a 30-item measure that expands the content coverage of work values by linking into a broad range of O*NET occupational descriptors, including Work Activities and Work Contexts, along with salary data from the Bureau of Labor Statistics. The OVI assesses 11 work values—Interpersonal, Outdoor, Physical, Leadership, Salary, Prestige, Variety, Interest, Work Hours, Knowledge Utilization, and Autonomy—providing an updated and refined coverage of work values relevant to people and occupations in the modern labor market. Study 1 (N = 768) developed and refined the OVI’s scales using conceptual and psychometric criteria, also reporting …

Sequential Robustness In Adversarial Reinforcement Learning, Roman Lok-Ming Belaire

Dissertations and Theses Collection (Open Access)

My goal is to build autonomous systems that expand the reach of human capability in challenging domains such as undersea and space exploration, disaster response, and large-scale infrastructure. In everyday settings, these systems will increasingly appear in safety-critical applications such as autonomous driving, robotics, and industrial manufacturing. A central requirement for these systems is the ability to operate reliably under uncertainty, particularly when the environment behaves in unanticipated ways.

The robust handling of unforeseen environment dynamics is therefore a technical cornerstone of autonomous decision-making; Adversarial attacks provide a useful and principled lens through which to study this problem. Adversarial \textit{robustness}, …

Perspectives On Interpretability For Neural Text Representations, Jia Peng Lim

Dissertations and Theses Collection (Open Access)

In this dissertation, we investigate interpretability in the three elements of learning neural text representations: inputs, passed into models, to produce probabilistic outputs. We emphasise perspectives as we present alternative novel methods to mine and organise meaning in this work.

Models. We initiate our investigation by examining Neural Topic Models (NTM), proposing an alternate angle of interpreting its word-topic distribution, producing better topic representations for interpretation. Our method maps the problem of finding these better interpretations to classical NP-hard graph problems, enabling examination of topic distributions in a composite manner. Next, we apply our previous findings to extract interpretations from …

Mease: Multi-Agent Episodic Action Sequence Explanation, Phyo Wai Khaing, Minghong Geng, Shubham Pateria, Budhitama Subagdja, Ah-Hwee Tan

Research Collection School Of Computing and Information Systems

Multi-agent reinforcement learning (MARL) achieves remarkable performance in complex coordination tasks, yet interpreting the emergent behaviors of trained agents remains a fundamental challenge. Most current explainability methods focus on individual agent decisions, overlooking the critical interplay of joint strategiesand temporal coordination patterns that define successful multi-agent policies. We present MEASE (Multi-agent Episodic Action Sequence Explanation), a novel explainable MARL (XMARL) framework that explains trained MARL policies as human-interpretable emergent cooperative joint behaviors. MEASE employs a cognition-inspired episodic memory model to learn spatio-temporal multi-agent interaction patterns, coupled with abstraction algorithms that identify significant cooperative agent behaviors. We evaluate MEASE on diverse …

Quantitative Bounds On Resource Usage Of Probabilistic Programs, Krishnendu Chatterjee, Amir Kafshdar Goharshady, Tobias Meggendorfer, Dorde Zikelic

Research Collection School Of Computing and Information Systems

Cost analysis, also known as resource usage analysis, is the task of finding bounds on the total cost of a program and is a well-studied problem in static analysis. In this work, we consider two classical quantitative problems in cost analysis for probabilistic programs. The first problem is to find a bound on the expected total cost of the program. This is a natural measure for the resource usage of the program and can also be directly applied to average-case runtime analysis. The second problem asks for a tail bound, i.e. ‍given a threshold t the goal is to find …

Market Reactions To Deceptive Language In Fake News: Implications From Language Expectancy Theory And Transfer Learning, Ka Chung Ng, Ping Fan Ke, Ping Fan, Mike So, Tam, Kar Yan

Research Collection School Of Computing and Information Systems

The advent of generative artificial intelligence (AI) has heightened the proliferation of fake news. A key challenge is the limited real-world data to investigate the societal impact of fake news produced by generative AI. In this paper, we examine stock market reactions to financial news articles that exhibit stylometric similarity to human-crafted and AI-crafted fake financial news. Grounded in language expectancy theory, we employ a style-based transfer learning model, pre-trained to recognizing deceptive language employed in various types of fake news intricacies. We then apply this model to a comprehensive dataset of financial news, assigning a “veracity style score” to …

Open Source Software Development Tool Installation: Challenges And Strategies For Novice Developers, Larissa Salerno, Christoph Treude, Patanamon Thongtanunam

Research Collection School Of Computing and Information Systems

As the world of technology advances, so do the tools that software developers use to create new programs. In recent years, software development tools have become more popular, allowing developers to work more efficiently and produce higher-quality software. Still, installing such tools can be challenging for novice developers at the early stage of their careers, as they may face issues such as compatibility problems (e.g., with operating systems) and unclear instructions. Therefore, this work aims to investigate the challenges novice developers face when installing software development tools and the strategies they employ to overcome them. To investigate these, we conducted …

Natural Adversaries: Fuzzing Autonomous Vehicles With Realistic Roadside Object Placements, Yang Sun, Haoyu Wang, Christopher M. Poskitt, Jun Sun

Research Collection School Of Computing and Information Systems

The emergence of Autonomous Vehicles (AVs) has spurred research into testing the resilience of their perception systems, i.e., ensuring that they are not susceptible to critical misjudgements. It is important that these systems are tested not only with respect to other vehicles on the road, but also with respect to objects placed on the roadside. Trash bins, billboards, and greenery are examples of such objects, typically positioned according to guidelines developed for the human visual system, which may not align perfectly with the needs of AVs. Existing tests, however, usually focus on adversarial objects with conspicuous shapes or patches, which …

Fighting Against Recruitment Scams: Theory-Driven Supervised Learning And Empirical Analysis For Digital Fraudulent Recruitment Posting Behavior, Tom (Tianteng) Wang, David (Jingjun) Xu, Keng Siau, Zhongju (John) Zhang

Research Collection School Of Computing and Information Systems

The number of recruitment postings on digital recruitment hiring platforms has increased since the COVID-19 pandemic. However, the weak surveillance and operations of these platforms, combined with the fact that most job seekers have relatively low vigilance and a strong desire for recruitment offers, enable scammers to easily deceive job seekers for their money and confidential information. In this work, we combine prevailing text mining techniques (i.e., ChatGPT with prompting engineering and supervised machine learning) with interpersonal deception theory (IDT) from social science to design an interpretable IT system to predict fraudulent recruitment postings on digital recruitment-hiring platforms. We compare …

Enhancing Action And Ingredient Modeling For Semantically Grounded Recipe Generation, Guoshan Liu, Bin Zhu, Yian Li, Jingjing Chen, Chong-Wah Ngo, Yu-Gang Jiang

Research Collection School Of Computing and Information Systems

Recent advances in Multimodal Large Language Models (MLMMs) have enabled recipe generation from food images, yet outputs often contain semantically incorrect actions or ingredients despite high lexical scores (e.g., BLEU, ROUGE). To address this gap, we propose a semantically grounded framework that predicts and validates actions and ingredients as internal context for instruction generation. Our two-stage pipeline combines supervised fine-tuning (SFT) with reinforcement fine-tuning (RFT): SFT builds foundational accuracy using an Action-Reasoning dataset and ingredient corpus, while RFT employs frequency-aware rewards to improve long-tail action prediction and ingredient generalization. A Semantic Confidence Scoring and Rectification (SCSR) module further filters and …

Benchmarking Gaslighting Attacks Against Speech Large Language Models, Jinyang Wu, Bin Zhu, Xiandong Zou, Qiquan Zhang

PhD Student’s Publications Collection

As Speech Large Language Models (Speech LLMs) become increasingly integrated into voice-based applications, ensuring their robustness against manipulative or adversarial input becomes critical. Although prior work has studied adversarial attacks in text-based LLMs and vision-language models, the unique cognitive and perceptual challenges of speech-based interaction remain underexplored. In contrast, speech presents inherent ambiguity, continuity, and perceptual diversity, which make adversarial attacks more difficult to detect. In this paper, we introduce gaslighting attacks, strategically crafted prompts designed to mislead, override, or distort model reasoning as a means to evaluate the vulnerability of Speech LLMs. Specifically, we construct five manipulation strategies: Anger, …

Teacher-Student Diffusion Model For Text-Driven 3d Hand Motion Generation, Ching Lam Cheng, Bin Zhu, Shengfeng He

PhD Student’s Publications Collection

Generating realistic 3D hand motion from natural language is vital for VR, robotics, and human-computer interaction. Existing methods either focus on full-body motion, overlooking detailed hand gestures, or require explicit 3D object meshes, limiting generality. We propose TSHaMo, a model-agnostic teacher-student diffusion framework for text-driven hand motion generation. The student model learns to synthesize motions from text alone, while the teacher leverages auxiliary signals (e.g., MANO parameters) to provide structured guidance during training. A co-training strategy enables the student to benefit from the teacher’s intermediate predictions while remaining text-only at inference. Evaluated using two diffusion backbones on GRAB and H2O, …

Gencode: A Generic Data Augmentation Framework For Boosting Deep Learning-Based Code Understanding, Zeming Dong, Qiang Hu, Xiaofei Xie, Maxime Cordy, Mike Papadakis, Yves Le Traon, Jianjun Zhao

Research Collection School Of Computing and Information Systems

Pre-trained code models lead the era of code intelligence, with multiple models designed with impressive performance. However, one important problem, data augmentation for code data that automatically helps developers prepare training data lacks study in this field. In this paper, we introduce a generic data augmentation framework, GenCode, to enhance the training of code understanding models. Simply speaking, GenCode follows a generation-and-selection paradigm to prepare useful training code data. Specifically, it employs code augmentation techniques to generate new code candidates first and then identifies important ones as the training data by influence scores. To evaluate the effectiveness of GenCode, we …

Causality-Driven Test Case Minimisation For Cyber-Physical Systems, Michael Foster, Christopher M. Poskitt, Nicholas R. Latimer, Neil Walkinshaw, Richard Somers, Robert M. Hierons

Research Collection School Of Computing and Information Systems

Cyber-physical systems allow digital control systems to interact with the physical world using sensors and actuators. They are increasingly being used to automate critical infrastructure, where software faults can have dire consequences. Due to the complex nature and unpredictability of these systems, their resilience is often tested using a technique called fuzzing, which generates quasi-random sequences of sensor and actuator manipulations with the goal of forcing a system into unsafe states. However, there is currently no way of determining which manipulations of a test case cause a failure without systematically removing each one and re-running the test, which can be …