Open Access. Powered by Scholars. Published by Universities.®

Programming Languages and Compilers Commons™

Open Access. Powered by Scholars. Published by Universities.®

Research Collection School Of Computing and Information Systems

Discipline
Keyword
Publication Year

Articles 1 - 30 of 379

Full-Text Articles in Programming Languages and Compilers

Larger Is Not Always Better: Exploring Small Open-Source Language Models In Logging Statement Generation, Renyi Zhong, Yichen Li, Guangba Yu, Wenwei Gu, Jinxi Kuang, Yintong Huo, Michael R. Lyu Jul 2026

Larger Is Not Always Better: Exploring Small Open-Source Language Models In Logging Statement Generation, Renyi Zhong, Yichen Li, Guangba Yu, Wenwei Gu, Jinxi Kuang, Yintong Huo, Michael R. Lyu

Research Collection School Of Computing and Information Systems

Developers use logging statements to create logs that document system behavior and aid in software maintenance. As such, high-quality logging is essential for effective maintenance; however, manual logging often leads to errors and inconsistency. Recent methods emphasize using large language models (LLMs) for automated logging statement generation, but these present privacy and resource issues, hindering their suitability for enterprise use. This paper presents the first large-scale empirical study evaluating small open-source language models (SOLMs) for automated logging statement generation. We evaluate four prominent SOLMs using various prompt strategies and parameter-efficient fine-tuning techniques, such as Low-Rank Adaptation (LoRA) and Retrieval-Augmented Generation …

Go to article

Quantitative Bounds On Resource Usage Of Probabilistic Programs, Krishnendu Chatterjee, Amir Kafshdar Goharshady, Tobias Meggendorfer, Dorde Zikelic May 2026

Quantitative Bounds On Resource Usage Of Probabilistic Programs, Krishnendu Chatterjee, Amir Kafshdar Goharshady, Tobias Meggendorfer, Dorde Zikelic

Research Collection School Of Computing and Information Systems

Cost analysis, also known as resource usage analysis, is the task of finding bounds on the total cost of a program and is a well-studied problem in static analysis. In this work, we consider two classical quantitative problems in cost analysis for probabilistic programs. The first problem is to find a bound on the expected total cost of the program. This is a natural measure for the resource usage of the program and can also be directly applied to average-case runtime analysis. The second problem asks for a tail bound, i.e. ‍given a threshold t the goal is to find …

Go to article

Generation Of Elaborated, Targeted And Effective Feedback For Novice Programmers Using Llm, Hua Leong Fwa May 2026

Generation Of Elaborated, Targeted And Effective Feedback For Novice Programmers Using Llm, Hua Leong Fwa

Research Collection School Of Computing and Information Systems

Programming errors and misconceptions are pervasive in novice programmers which causes difficulty in the learning of computer programming. Large Language Models (LLMs), with their ability to comprehend and generate programming codes have shown promising results in the automatic identification of errors. This can potentially benefit student programmers by providing them with timely formative feedback at efficiencies and scale that were not attainable previously. In this study, we leveraged an LLM - OpenAI o4-mini for the generation of elaborated, targeted feedback for novice programmers across PHP and JavaScript exercises. We contend that the feedback needs to be effective and targeted other …

Go to article

Automatic Generation Of Introductory Programming Exercises With Large Language Models, Nguyen Binh Duong Ta, Hua Gia Phuc Nguyen, Gottipati Swapna Jan 2026

Automatic Generation Of Introductory Programming Exercises With Large Language Models, Nguyen Binh Duong Ta, Hua Gia Phuc Nguyen, Gottipati Swapna

Research Collection School Of Computing and Information Systems

Despite recent advances in code generation made possible by large language models (LLMs), programming is still an essential skill that computing students need to master now and in the foreseeable future. In learning programming, frequent practices with exercises set at an appropriate difficulty and knowledge level is of crucial importance for students. However, it’s not a trivial task for instructors to create many good quality exercises customized for each student. Programming problems found on Internet sources such as LeetCode are mostly too challenging for novice programmers with no prior coding knowledge. Recent work in AI-enabled education has been leveraging LLMs …

Go to article

Backdoorllm: A Comprehensive Benchmark For Backdoor Attacks And Defenses On Large Language Models, Yige Li, Hanxun Huang, Yunhan Zhao, Xingjun Ma, Jun Sun Dec 2025

Backdoorllm: A Comprehensive Benchmark For Backdoor Attacks And Defenses On Large Language Models, Yige Li, Hanxun Huang, Yunhan Zhao, Xingjun Ma, Jun Sun

Research Collection School Of Computing and Information Systems

Generative large language models (LLMs) have achieved state-of-the-art results on a wide range of tasks, yet they remain susceptible to backdoor attacks: carefully crafted triggers in the input can manipulate the model to produce adversaryspecified outputs. While prior research has predominantly focused on backdoor risks in vision and classification settings, the vulnerability of LLMs in open-ended text generation remains underexplored. To fill this gap, we introduce BackdoorLLM1 , the first comprehensive benchmark for systematically evaluating backdoor threats in text-generation LLMs. BackdoorLLM provides: (i) a unified repository of benchmarks with a standardized training and evaluation pipeline; (ii) a diverse suite of …

Go to article

A Learning‑Augmented Dynamic Programming Approach For Orienteering Problem With Time Windows, Guansheng Peng, Lining Xing, Fuyan Song Ma, Aldy Gunawan, Aldy Gunawan Dec 2025

A Learning‑Augmented Dynamic Programming Approach For Orienteering Problem With Time Windows, Guansheng Peng, Lining Xing, Fuyan Song Ma, Aldy Gunawan, Aldy Gunawan

Research Collection School Of Computing and Information Systems

Recent years have witnessed a surge of interest in solving combinatorial optimization problems (COPs) using machine learning techniques. Motivated by this trend, we propose a learning-augmented exact approach for tackling an NP-hard COP, the Orienteering Problem with Time Windows, which aims to maximize the total score collected by visiting a subset of vertices in a graph within their time windows. Traditional exact algorithms rely heavily on domain expertise and meticulous design, making it hard to achieve further improvements. By leveraging deep learning models to learn effective relaxations of problem restrictions from data, our approach enables significant performance gains in an …

Go to article

The Rise Of Parameter Specialization For Knowledge Storage In Large Language Models, Yihuai Hong, Yiran Zhao, Wei Tang, Yang Deng, Yu Rong, Wenxuan Zhang Dec 2025

The Rise Of Parameter Specialization For Knowledge Storage In Large Language Models, Yihuai Hong, Yiran Zhao, Wei Tang, Yang Deng, Yu Rong, Wenxuan Zhang

Research Collection School Of Computing and Information Systems

Over time, a growing wave of large language models from various series has been introduced to the community. Researchers are striving to maximize the performance of language models with constrained parameter sizes. However, from a microscopic perspective, there has been limited research on how to better store knowledge in model parameters, particularly within MLPs, to enable more effective utilization of this knowledge by the model. In this work, we analyze twenty publicly available open-source large language models to investigate the relationship between their strong performance and the way knowledge is stored in their corresponding MLP parameters. Our findings reveal that …

Go to article

A Partition Cover Approach To Tokenization, Jia Peng Lim, Shawn Tan, Davin Choo, Hady Wirawan Lauw Dec 2025

A Partition Cover Approach To Tokenization, Jia Peng Lim, Shawn Tan, Davin Choo, Hady Wirawan Lauw

Research Collection School Of Computing and Information Systems

Tokenization is the process of encoding strings into tokens of a fixed vocabulary size, and is widely utilized in Natural Language Processing applications. The leading tokenization algorithm today is Byte Pair Encoding (BPE), which formulates the tokenization problem as a compression problem and tackles it by performing sequences of merges. In this work, we formulate tokenization as an optimization objective, show that it is NP-hard via a simple reduction from vertex cover, and propose a polynomial-time greedy algorithm GreedTok. Our formulation naturally relaxes to the well-studied weighted maximum coverage problem which has a simple -approximation algorithm GreedWMC. Through empirical evaluations …

Go to article

When Less Language Is More: Language-Reasoning Disentanglement Makes Llms Better Multilingual Reasoners, Weixiang Zhao, Jiahe Guo, Yang Deng, Tongtong Wu, Wenxuan Zhang, Yulin Hu, Xingyu Sui, Yanyan Zhao, Wanxiang Che, Bing Qin, Tat-Seng Chua, Ting Liu Dec 2025

When Less Language Is More: Language-Reasoning Disentanglement Makes Llms Better Multilingual Reasoners, Weixiang Zhao, Jiahe Guo, Yang Deng, Tongtong Wu, Wenxuan Zhang, Yulin Hu, Xingyu Sui, Yanyan Zhao, Wanxiang Che, Bing Qin, Tat-Seng Chua, Ting Liu

Research Collection School Of Computing and Information Systems

Multilingual reasoning remains a significant challenge for large language models (LLMs), with performance disproportionately favoring high-resource languages. Drawing inspiration from cognitive neuroscience, which suggests that human reasoning functions largely independently of language processing, we hypothesize that LLMs similarly encode reasoning and language as separable components that can be disentangled to enhance multilingual reasoning. To evaluate this, we perform a causal intervention by ablating language-specific representations at inference time. Experiments on 10 open-weight LLMs spanning 11 typologically diverse languages show that this language-specific ablation consistently boosts multilingual reasoning performance. Layer-wise analyses further confirm that language and reasoning representations can be effectively …

Go to article

Do Code Semantics Help? A Comprehensive Study On Execution Trace-Based Information For Code Large Language Models, Jian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, Yi Li Nov 2025

Do Code Semantics Help? A Comprehensive Study On Execution Trace-Based Information For Code Large Language Models, Jian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, Yi Li

Research Collection School Of Computing and Information Systems

Code Large Language Models (Code LLMs) have opened a new era in programming with their impressive capabilities. However, recent research has revealed critical limitations in their ability to reason about runtime behavior and understand the actual functionality of programs, which poses significant challenges for their post-training and practical deployment. Specifically, Code LLMs encounter two principal issues: (1) a lack of proficiency in reasoning about program execution behavior, as they struggle to interpret what programs actually do during runtime, and (2) inconsistent and fragmented representation of semantic information, such as execution traces, across existing methods, which hinders their ability to generalize …

Go to article

Mmlu-Prox: A Multilingual Benchmark For Advanced Large Language Model Evaluation, Weihao Xuan, Et. Al. Nov 2025

Mmlu-Prox: A Multilingual Benchmark For Advanced Large Language Model Evaluation, Weihao Xuan, Et. Al.

Research Collection School Of Computing and Information Systems

Existing large language model (LLM) evaluation benchmarks primarily focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-lingual reasoning abilities. This dual limitation makes it challenging to assess LLMs’ performance in the multilingual setting comprehensively. To fill this gap, we introduce MMLU-ProX, a comprehensive benchmark covering 29 languages, built on an English benchmark. Each language version consists of 11,829 identical questions, enabling direct cross-lingual comparisons. Additionally, to meet efficient evaluation needs, we provide a lite version containing 658 questions per language. To ensure the high quality of MMLU-ProX, we employ a rigorous development process that involves …

Go to article

Chain Of Strategy Optimization Makes Large Language Models Better Emotional Supporter, Weixiang Zhao, Xingyu Sui, Xinyang Han, Yang Deng, Yulin Hu, Jiahe Guo, Libo Qin, Qianyun Du, Shijin Wang, Yanyan Zhao, Bing Qin, Ting Liu Nov 2025

Chain Of Strategy Optimization Makes Large Language Models Better Emotional Supporter, Weixiang Zhao, Xingyu Sui, Xinyang Han, Yang Deng, Yulin Hu, Jiahe Guo, Libo Qin, Qianyun Du, Shijin Wang, Yanyan Zhao, Bing Qin, Ting Liu

Research Collection School Of Computing and Information Systems

The growing emotional stress in modern society has increased the demand for Emotional Support Conversations (ESC). While Large Language Models (LLMs) show promise for ESC, they face two key challenges: (1) low strategy selection accuracy, and (2) preference bias, limiting their adaptability to users’ emotional needs. Existing supervised fine-tuning (SFT) struggles to address these issues, as it rigidly trains models on single gold-standard responses without modeling nuanced strategy trade-offs. To overcome these limitations, we propose a novel two-stage framework that optimizes strategy selection preferences at each dialogue turn. We first leverage Monte Carlo Tree Search to construct ESC-Pro, a high-quality …

Go to article

Envisioning Future Interactive Web Development: Editing Webpage With Natural Language, Truong Hai Dang, Jingyu Xiao, Yintong Huo Nov 2025

Envisioning Future Interactive Web Development: Editing Webpage With Natural Language, Truong Hai Dang, Jingyu Xiao, Yintong Huo

Research Collection School Of Computing and Information Systems

The evolution of web applications relies on iterative code modifications, a process that is traditionally manual and time-consuming. While Large Language Models (LLMs) can generate UI code, their ability to edit existing code from new design requirements (e.g., ”center the logo”) remains a challenge. This is largely due to the absence of large-scale, high-quality tuning data to align model performance with human expectations. In this paper, we introduce a novel, automated data generation pipeline that uses LLMs to synthesize a high-quality fine-tuning dataset for web editing, named Instruct4Edit. Our approach generates diverse instructions, applies the corresponding code modifications, and performs …

Go to article

One Planner To Guide Them All! Learning Adaptive Conversational Planners For Goal-Oriented Dialogues, Huy Dao, Lizi Liao Nov 2025

One Planner To Guide Them All! Learning Adaptive Conversational Planners For Goal-Oriented Dialogues, Huy Dao, Lizi Liao

Research Collection School Of Computing and Information Systems

Goal-oriented dialogues, such as recommendation and negotiation, often require balancing multiple, conflicting objectives. Existing methods typically involve training separate models for specific combinations of objectives, leading to computational and scalability issues. In this work, we aim to develop a new dialogue policy method that can adapt to varying objective preferences at inference time without retraining. This raises several challenges in terms of both (1) optimization strategy and (2) knowledge utilization. To address these, we propose a novel learning framework, Preference Adaptive Dialogue Policy Planner (PADPP), for multi-objective goal-oriented dialogues. Specifically, to tackle the former, we introduce a novel policy optimization …

Go to article

Context-Aware Hierarchical Taxonomy Generation For Scientific Papers Via Llm-Guided Multi-Aspect Clustering, Kun Zhu, Lizi Liao, Yuxuan Gu, Lei Huang, Xiaocheng Feng, Bing Qin Nov 2025

Context-Aware Hierarchical Taxonomy Generation For Scientific Papers Via Llm-Guided Multi-Aspect Clustering, Kun Zhu, Lizi Liao, Yuxuan Gu, Lei Huang, Xiaocheng Feng, Bing Qin

Research Collection School Of Computing and Information Systems

The rapid growth of scientific literature demands efficient methods to organize and synthesize research findings. Existing taxonomy construction methods, leveraging unsupervised clustering or direct prompting of large language models (LLMs), often lack coherence and granularity. We propose a novel context-aware hierarchical taxonomy generation framework that integrates LLM-guided multi-aspect encoding with dynamic clustering. Our method leverages LLMs to identify key aspects of each paper (e.g., methodology, dataset, evaluation) and generates aspect-specific paper summaries, which are then encoded and clustered along each aspect to form a coherent hierarchy. In addition, we introduce a new evaluation benchmark of 156 expert-crafted taxonomies encompassing 11.6k …

Go to article

Distillcaps: Enhancing Audio-Language Alignment In Captioning Via Retrieval-Augmented Knowledge Distillation, Thinh Pham, Nghiem Diep, Lizi Liao, Binh Nguyen Nov 2025

Distillcaps: Enhancing Audio-Language Alignment In Captioning Via Retrieval-Augmented Knowledge Distillation, Thinh Pham, Nghiem Diep, Lizi Liao, Binh Nguyen

Research Collection School Of Computing and Information Systems

Automated audio captioning (AAC) benefits from incorporatingexternal context to interpret complex sounds, but doing so withretrieval-augmented generation (RAG) at inference is sometimesinfeasible due to data availability or incurs significant latency andcomplexity. We propose DistillCaps, a novel training-time frame-work that leverages RAG to guide knowledge distillation for im-proved audio-language alignment, while lessening the relianceon retrieval during inference. In our framework, a RAG-equippedteacher model retrieves relevant textual information (e.g., simi-lar captions) for each audio clip and uses it for training to gener-ate context-enriched captions. Simultaneously, a student model istrained to imitate this teacher, learning to produce high-qualitycaptions from audio alone. We further …

Go to article

Probabilistic Prototype Calibration Of Vision-Language Models For Generalized Few-Shot Semantic Segmentation, Jie Liu, Jiayi Shen, Pan Zhou, Jan-Jakob Sonke, Stratis Gavves Oct 2025

Probabilistic Prototype Calibration Of Vision-Language Models For Generalized Few-Shot Semantic Segmentation, Jie Liu, Jiayi Shen, Pan Zhou, Jan-Jakob Sonke, Stratis Gavves

Research Collection School Of Computing and Information Systems

Generalized Few-Shot Semantic Segmentation (GFSS) aims to extend a segmentation model to novel classes with only a few annotated examples while maintaining performance on base classes. Recently, pretrained vision-language models (VLMs) such as CLIP have been leveraged in GFSS to improve generalization on novel classes through multi-modal prototypes learning. However, existing prototype-based methods are inherently deterministic, limiting the adaptability of learned prototypes to diverse samples, particularly for novel classes with scarce annotations. To address this, we propose FewCLIP, a probabilistic prototype calibration framework over multi-modal prototypes from the pretrained CLIP, thus providing more adaptive prototype learning for GFSS. Specifically, FewCLIP …

Go to article

Boosting Chart-To-Code Generation In Mllm Via Dual Preference-Guided Refinement, Zhihan Zhang, Yixin Cao, Lizi Liao Oct 2025

Boosting Chart-To-Code Generation In Mllm Via Dual Preference-Guided Refinement, Zhihan Zhang, Yixin Cao, Lizi Liao

Research Collection School Of Computing and Information Systems

Translating chart images into executable plotting scripts-referred to as the chart-to-code generation task-requires Multimodal Large Language Models (MLLMs) to perform fine-grained visual parsing, precise code synthesis, and robust cross-modal reasoning. However, this task is inherently under-constrained: multiple valid code implementations can produce the same visual chart, and evaluation must consider both code correctness and visual fidelity across diverse dimensions. This makes it difficult to learn accurate and generalizable mappings through standard supervised fine-tuning. To address these challenges, we propose a dual preference-guided refinement framework that combines a feedback-driven, dual-modality reward mechanism with iterative preference learning. Our approach introduces a structured …

Go to article

Auxiliary Prompt Tuning Of Vision‑Language Models For Few‑Shot Out‑Of‑Distribution Detection, Wenjun Miao, Guansong Pang, Zihan Wang, Jin Zheng, Xiao Bai Oct 2025

Auxiliary Prompt Tuning Of Vision‑Language Models For Few‑Shot Out‑Of‑Distribution Detection, Wenjun Miao, Guansong Pang, Zihan Wang, Jin Zheng, Xiao Bai

Research Collection School Of Computing and Information Systems

Recent advancements in CLIP-based out-of-distribution (OOD) detection have shown promising results via regularization on prompt tuning, leveraging background features extracted from a few in-distribution (ID) samples as proxies for OOD features.However, these methods suffer from an inherent limitation: a lack of diversity in the extracted OOD features from the few-shot ID data.To address this issue, we propose to leverage external datasets as auxiliary outlier data (i.e., pseudo OOD samples) to extract rich, diverse OOD features, with the features from not only background regions but also foreground object regions, thereby supporting more discriminative prompt tuning for OOD detection. We further introduce …

Go to article

Contrastrepair: Enhancing Conversation-Based Automated Program Repair Via Contrastive Test Case Pairs, Jiaolong Kong, Xiaofei Xie, Mingfei Cheng, Shangqing Liu, Xiaoning Du, Qi Guo Oct 2025

Contrastrepair: Enhancing Conversation-Based Automated Program Repair Via Contrastive Test Case Pairs, Jiaolong Kong, Xiaofei Xie, Mingfei Cheng, Shangqing Liu, Xiaoning Du, Qi Guo

Research Collection School Of Computing and Information Systems

Automated Program Repair (APR) aims to automatically generate patches for rectifying software bugs. Recentstrides in Large Language Models (LLM), such as ChatGPT, have yielded encouraging outcomes in APR,especially within the conversation-driven APR framework. Nevertheless, the efficacy of conversation-drivenAPR is contingent on the quality of the feedback information. In this article, we propose ContrastRepair, anovel conversation-based APR approach that augments conversation-driven APR by providing LLMs withcontrastive test pairs. A test pair consists of a failing test and a passing test, which offer contrastive feedback tothe LLM. Our key insight is to minimize the difference between the generated passing test and the …

Go to article

Polymind: Parallel Visual Diagramming With Large Language Models To Support Prewriting Through Microtasks, Qian Wan, Jiannan Li, Huanchen Wang, Zhicong Lu Oct 2025

Polymind: Parallel Visual Diagramming With Large Language Models To Support Prewriting Through Microtasks, Qian Wan, Jiannan Li, Huanchen Wang, Zhicong Lu

Research Collection School Of Computing and Information Systems

Prewriting is the process of generating and organising ideas before a first draft. It consists of a combination of informal, iterative, and semi-structured strategies such as visual diagramming, which poses a challenge for collaborating with large language models (LLMs) in a turn-taking conversational manner. We present Polymind, a visual diagramming tool that leverages multiple LLM-powered agents to support prewriting. The system features a parallel collaboration workflow in place of the turn-taking conversational interactions. It defines multiple ''microtasks'' to simulate group collaboration scenarios such as collaborative writing and group brainstorming. Instead of repetitively prompting a chatbot for various purposes, Polymind enables …

Go to article

Exploring Parameter-Efficient Fine-Tuning Techniques For Code Generation With Large Language Models, Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, Houari A. Sahraoui Sep 2025

Exploring Parameter-Efficient Fine-Tuning Techniques For Code Generation With Large Language Models, Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, Houari A. Sahraoui

Research Collection School Of Computing and Information Systems

Large language models (LLMs) demonstrate impressive capabilities to generate accurate code snippets given natural language intents in a zero-shot manner, i.e., without the need for specific fine-tuning. While prior studies have highlighted the advantages of fine-tuning LLMs, this process incurs high computational costs, making it impractical in resource-scarce environments, particularly for models with billions of parameters. To address these challenges, previous research explored in-context learning (ICL) and retrieval-augmented generation (RAG) as strategies to guide the LLM generative process with task-specific prompt examples. However, ICL and RAG introduce inconveniences, such as the need for designing contextually relevant prompts and the absence …

Go to article

Detecting Defi Fraud With A Graph-Transformer Language Model, Wei Ma, Junjie Shi, Jiaxi Qiu, Cong Wu, Jing Chen, Lingxiao Jiang, Shangqing Liu, Yang Liu, Yang Xiang Sep 2025

Detecting Defi Fraud With A Graph-Transformer Language Model, Wei Ma, Junjie Shi, Jiaxi Qiu, Cong Wu, Jing Chen, Lingxiao Jiang, Shangqing Liu, Yang Liu, Yang Xiang

Research Collection School Of Computing and Information Systems

With the rapid development of blockchain technology, the widespread adoption of smart contracts—particularly in decentralized finance (DeFi) applications—has introduced significant security challenges, such as reentrancy attacks, phishing, and Sybil attacks. To address these issues, we propose a novel model called TrxGNNBERT, which combines Graph Neural Network (GNN) and the Transformer architecture to effectively handle both graph-structured and textual data. This combination enhances the detection of suspicious transactions and accounts on blockchain platforms like Ethereum. TrxGNNBERT was pre-trained using a masked language model (MLM) on a dataset of 60,000 Ethereum transactions by randomly masking the attributes of nodes and edges, thereby …

Go to article

Guiding Multiple Remote Users In Physical Tasks With Language-Driven Robotic Telepresence, Ruyi Li, Jingfei Guo, Xinyi Zhang, Xuji Zhang, Zeqing Li, Jiannan Li, Jiangtao Gong Sep 2025

Guiding Multiple Remote Users In Physical Tasks With Language-Driven Robotic Telepresence, Ruyi Li, Jingfei Guo, Xinyi Zhang, Xuji Zhang, Zeqing Li, Jiannan Li, Jiangtao Gong

Research Collection School Of Computing and Information Systems

Remote assistance through robotic telepresence could involve both control and memory challenges, particularly in one expert to multiple workers situation. In this work, we proposed a novelty language-driven interface to facilitate remote collaboration through telepresence robots. Through operations and maintenance expert interviews and a scenario simulation study, we identified key pain points in executing one-expert-multiple-workers remote guidance using the telepresence robot and proposed two design goals, which together consist of five sub-design goals with corresponding features. These features were integrated into a standard telepresence robot, resulting in the development of a Collaborative LLM-based Embodied Assistant Robot, named CLEAR Robot. A …

Go to article

Equivalence And Similarity Refutation For Probabilistic Programs, Krishnendu Chatterjee, Ehsan Kafshdar Goharshady, Petr Novotný, Dorde Zikelic Aug 2025

Equivalence And Similarity Refutation For Probabilistic Programs, Krishnendu Chatterjee, Ehsan Kafshdar Goharshady, Petr Novotný, Dorde Zikelic

Research Collection School Of Computing and Information Systems

We consider the problems of statically refuting equivalence and similarity of output distributions defined by a pair of probabilistic programs. Equivalence and similarity are two fundamental relational properties of probabilistic programs that are essential for their correctness both in implementation and in compilation. In this work, we present a new method for static equivalence and similarity refutation. Our method refutes equivalence and similarity by computing a function over program outputs whose expected value with respect to the output distributions of two programs is different. The function is computed simultaneously with an upper expectation supermartingale and a lower expectation submartingale for …

Go to article

Focus: Evaluating Pre-Trained Vision-Language Models On Underspecification Reasoning, Kankan Zhou, Yibin Lai, Kyriakos Mouratidis, Jing Jiang Aug 2025

Focus: Evaluating Pre-Trained Vision-Language Models On Underspecification Reasoning, Kankan Zhou, Yibin Lai, Kyriakos Mouratidis, Jing Jiang

Research Collection School Of Computing and Information Systems

Humans possess a remarkable ability to interpret underspecified ambiguous statements by inferring their meanings from contexts such as visual inputs. This ability, however, may not be as developed in recent pre-trained visionlanguage models (VLMs). In this paper, we introduce a novel probing dataset called FOCUS to evaluate whether state-of-the-art VLMs have this ability. FOCUS consists of underspecified sentences paired with image contexts and carefully designed probing questions. Our experiments reveal that VLMs still fall short in handling underspecification even when visual inputs that can help resolve the ambiguities are available. To further support research in underspecification, FOCUS will be released …

Go to article

Evaluating And Mitigating Linguistic Discrimination In Large Language Models: Perspectives On Safety Equity And Knowledge Equity, Guoliang Dong, Haoyu Wang, Jun Sun, Xinyu Wang Aug 2025

Evaluating And Mitigating Linguistic Discrimination In Large Language Models: Perspectives On Safety Equity And Knowledge Equity, Guoliang Dong, Haoyu Wang, Jun Sun, Xinyu Wang

Research Collection School Of Computing and Information Systems

By training on text in various languages, large language models (LLMs) typically possess multilingual support and demonstrate remarkable capabilities in solving tasks described in different languages. However, LLMs can exhibit linguistic discrimination due to the uneven distribution of training data across languages. That is, LLMs are hard to keep the consistency of responses when faced with the same task but depicted in different languages. In this study, we first explore the consistency in the LLMs’ outputs responding to queries in various languages from two aspects: safety and quality. We conduct this analysis with two datasets (AdvBench and NQ) based on …

Go to article

How To Enable Effective Cooperation Between Humans And Nlp Models: A Survey Of Principles, Formalizations, And Beyond, Chen Huang, Yang Deng, Wenqiang Lei, Jiancheng Lv, Tat-Seng Chua, Jimmy Huang Aug 2025

How To Enable Effective Cooperation Between Humans And Nlp Models: A Survey Of Principles, Formalizations, And Beyond, Chen Huang, Yang Deng, Wenqiang Lei, Jiancheng Lv, Tat-Seng Chua, Jimmy Huang

Research Collection School Of Computing and Information Systems

With the advancement of large language models (LLMs), intelligent models have evolved from mere tools to autonomous agents with their own goals and strategies for cooperating with humans. This evolution has birthed a novel paradigm in NLP, i.e., human-model cooperation, that has yielded remarkable progress in numerous NLP tasks in recent years. In this paper, we take the first step to present a thorough review of human-model cooperation, exploring its principles, formalizations, and open challenges. In particular, we introduce a new taxonomy that provides a unified perspective to summarize existing approaches. Also, we discuss potential frontier areas and their corresponding …

Go to article

Evowiki: Evaluating Llms On Evolving Knowledge, Wei Tang, Yixin Cao, Yang Deng, Jiahao Ying, Bo Wang, Yizhe Yang, Yuyue Zhao, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Yong Liao Aug 2025

Evowiki: Evaluating Llms On Evolving Knowledge, Wei Tang, Yixin Cao, Yang Deng, Jiahao Ying, Bo Wang, Yizhe Yang, Yuyue Zhao, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Yong Liao

Research Collection School Of Computing and Information Systems

Knowledge utilization is a critical aspect of LLMs, and understanding how they adapt to evolving knowledge is essential for their effective deployment. However, existing benchmarks are predominantly static, failing to capture the evolving nature of LLMs and knowledge, leading to inaccuracies and vulnerabilities such as contamination. In this paper, we introduce EvoWiki, an evolving dataset designed to reflect knowledge evolution by categorizing information into stable, evolved, and uncharted states. EvoWiki is fully auto-updated, enabling precise evaluation of continuously changing knowledge and newly released LLMs. Through experiments with Retrieval-Augmented Generation (RAG) and Continual Learning (CL), we evaluate how effectively LLMs adapt …

Go to article

Fact-Audit: An Adaptive Multi-Agent Framework For Dynamic Fact-Checking Evaluation Of Large Language Models, Hongzhan Lin, Yang Deng, Yuxuan Gu, Wenxuan Zhang, Jing Ma, See-Kiong Ng, Tat-Seng Chua Aug 2025

Fact-Audit: An Adaptive Multi-Agent Framework For Dynamic Fact-Checking Evaluation Of Large Language Models, Hongzhan Lin, Yang Deng, Yuxuan Gu, Wenxuan Zhang, Jing Ma, See-Kiong Ng, Tat-Seng Chua

Research Collection School Of Computing and Information Systems

Large Language Models (LLMs) have significantly advanced the fact-checking studies. However, existing automated fact-checking evaluation methods rely on static datasets and classification metrics, which fail to automatically evaluate the justification production and uncover the nuanced limitations of LLMs in fact-checking. In this work, we introduce FACT-AUDIT, an agent-driven framework that adaptively and dynamically assesses LLMs’ fact-checking capabilities. Leveraging importance sampling principles and multi-agent collaboration, FACT-AUDIT generates adaptive and scalable datasets, performs iterative model-centric evaluations, and updates assessments based on model-specific responses. By incorporating justification production alongside verdict prediction, this framework provides a comprehensive and evolving audit of LLMs’ factual reasoning …

Go to article

Next Last