Computer Sciences Commons^™

Evowiki: Evaluating Llms On Evolving Knowledge, Wei Tang, Yixin Cao, Yang Deng, Jiahao Ying, Bo Wang, Yizhe Yang, Yuyue Zhao, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Yong Liao

Research Collection School Of Computing and Information Systems

Knowledge utilization is a critical aspect of LLMs, and understanding how they adapt to evolving knowledge is essential for their effective deployment. However, existing benchmarks are predominantly static, failing to capture the evolving nature of LLMs and knowledge, leading to inaccuracies and vulnerabilities such as contamination. In this paper, we introduce EvoWiki, an evolving dataset designed to reflect knowledge evolution by categorizing information into stable, evolved, and uncharted states. EvoWiki is fully auto-updated, enabling precise evaluation of continuously changing knowledge and newly released LLMs. Through experiments with Retrieval-Augmented Generation (RAG) and Continual Learning (CL), we evaluate how effectively LLMs adapt …

Focus: Evaluating Pre-Trained Vision-Language Models On Underspecification Reasoning, Kankan Zhou, Yibin Lai, Kyriakos Mouratidis, Jing Jiang

Research Collection School Of Computing and Information Systems

Humans possess a remarkable ability to interpret underspecified ambiguous statements by inferring their meanings from contexts such as visual inputs. This ability, however, may not be as developed in recent pre-trained visionlanguage models (VLMs). In this paper, we introduce a novel probing dataset called FOCUS to evaluate whether state-of-the-art VLMs have this ability. FOCUS consists of underspecified sentences paired with image contexts and carefully designed probing questions. Our experiments reveal that VLMs still fall short in handling underspecification even when visual inputs that can help resolve the ambiguities are available. To further support research in underspecification, FOCUS will be released …

Equivalence And Similarity Refutation For Probabilistic Programs, Krishnendu Chatterjee, Ehsan Kafshdar Goharshady, Petr Novotný, Dorde Zikelic

Research Collection School Of Computing and Information Systems

We consider the problems of statically refuting equivalence and similarity of output distributions defined by a pair of probabilistic programs. Equivalence and similarity are two fundamental relational properties of probabilistic programs that are essential for their correctness both in implementation and in compilation. In this work, we present a new method for static equivalence and similarity refutation. Our method refutes equivalence and similarity by computing a function over program outputs whose expected value with respect to the output distributions of two programs is different. The function is computed simultaneously with an upper expectation supermartingale and a lower expectation submartingale for …

Xfinbench: Benchmarking Llms In Complex Financial Problem Solving And Reasoning, Zhihan Zhang, Yixin Cao, Lizi Liao

Research Collection School Of Computing and Information Systems

Solving financial problems demands complex reasoning, multimodal data processing, and a broad technical understanding, presenting unique challenges for current large language models (LLMs). We introduce **XFinBench**, a novel benchmark with 4,235 examples designed to evaluate LLM’s ability in solving comple**X**, knowledge-intensive **Fin**ancial problems across diverse graduate-level finance topics with multi-modal context. We identify five core capabilities of LLMs using XFinBench, i.e., _terminology understanding_, _temporal reasoning_, _future forecasting_, _scenario planning_, and _numerical modelling_. Upon XFinBench, we conduct extensive experiments on 18 leading models. The result shows that o1 is the best-performing text-only model with an overall accuracy of 67.3%, but still …

Llm2rec: Large Language Models Are Powerful Embedding Models For Sequential Recommendation, Yingzhi He, Xiaohao Liu, An Zhang, Yunshan Ma, Tat‑Seng Chua

Research Collection School Of Computing and Information Systems

Sequential recommendation aims to predict users' future interactions by modeling collaborative filtering (CF) signals from historical behaviors of similar users or items. Traditional sequential recommenders predominantly rely on ID-based embeddings, which capture CF signals through high-order co-occurrence patterns. However, these embeddings depend solely on past interactions, lacking transferable knowledge to generalize to unseen domains. Recent advances in large language models (LLMs) have motivated text-based recommendation approaches that derive item representations from textual descriptions. While these methods enhance generalization, they fail to encode CF signals-i.e., latent item correlations and preference patterns-crucial for effective recommendation. We argue that an ideal embedding model …

Fact-Audit: An Adaptive Multi-Agent Framework For Dynamic Fact-Checking Evaluation Of Large Language Models, Hongzhan Lin, Yang Deng, Yuxuan Gu, Wenxuan Zhang, Jing Ma, See-Kiong Ng, Tat-Seng Chua

Research Collection School Of Computing and Information Systems

Large Language Models (LLMs) have significantly advanced the fact-checking studies. However, existing automated fact-checking evaluation methods rely on static datasets and classification metrics, which fail to automatically evaluate the justification production and uncover the nuanced limitations of LLMs in fact-checking. In this work, we introduce FACT-AUDIT, an agent-driven framework that adaptively and dynamically assesses LLMs’ fact-checking capabilities. Leveraging importance sampling principles and multi-agent collaboration, FACT-AUDIT generates adaptive and scalable datasets, performs iterative model-centric evaluations, and updates assessments based on model-specific responses. By incorporating justification production alongside verdict prediction, this framework provides a comprehensive and evolving audit of LLMs’ factual reasoning …

Beware Of Your Po! Measuring And Mitigating Ai Safety Risks In Role-Play Fine-Tuning Of Llms, Weixiang Zhao, Yulin Hu, Yang Deng, Jiahe Guo, Xingyu Sui, Xinyang Han, An Zhang, Yanyan Zhao, Bing Qin, Tat-Seng Chua, Ting Liu

Research Collection School Of Computing and Information Systems

Although large language models (LLMs) store vast amount of knowledge in their parameters, they still have limitations in the memorization and utilization of certain knowledge, leading to undesired behaviors such as generating untruthful and inaccurate responses. This highlights the critical need to understand the knowledge boundary of LLMs, a concept that remains inadequately defined in existing research. In this survey, we propose a comprehensive definition of the LLM knowledge boundary and introduce a formalized taxonomy categorizing knowledge into four distinct types. Using this foundation, we systematically review the field through three key lenses: the motivation for studying LLM knowledge boundaries, …

Browsing Like Human: A Multimodal Web Agent With Experiential Fast-And-Slow Thinking, Haohao Luo, Jiayi Kuang, Wei Liu, Ying Shen, Jian Luan, Yang Deng

Research Collection School Of Computing and Information Systems

Automating web navigation which aims to build a web agent that follows user instructions to complete tasks like booking flights by interacting with websites, has received increasing attention due to its practical value. Although existing web agents are mostly equipped with visual perception, planning, and memory abilities, their reasoning process are still deviate from human cognition. In this work, we study the human thought pattern to empower agent with more human-like abilities in web navigation. To tackle this problem, we propose a novel multimodal web agent framework called WebExperT, which is designed to emulate the human planning process of “thinking …

Debate, Reflect, And Distill: Multi-Agent Feedback With Tree-Structured Preference Optimization For Efficient Language Model Enhancement, Xiaofeng Zhou, Heyan Huang, Lizi Liao

Research Collection School Of Computing and Information Systems

Large Language Models (LLMs) continue to set new standards in knowledge-intensive and complex reasoning tasks, yet their high computational demands limit widespread adoption. While distilling large models into smaller ones offers a sustainable solution, current techniques—such as static knowledge distillation, resource-intensive reinforcement learning from human feedback, or limited self-reflection—struggle to yield substantial and lasting performance gains. In this paper, we present a novel Debate and Reflect (D&R) framework that orchestrates multi-turn debates between smaller models and stronger teacher models, eliciting actionable feedback (e.g., error analysis, corrective strategies) to guide student models. Further, we introduce Tree-structured Direct Preference Optimization (T-DPO) to …

The Impact Of Artificial Intelligence As An Intervening Variable Between The Digital Government Strategy And Competency Development "An Applied Study At Sharjah Police Sciences Academy ", Elsayed Kamal Risha, Abd Al-Rahman Al-Naqbi

Journal of Police and Legal Sciences

The study aimed to determine the impact of the digital government strategy on competencies development, through artificial intelligence as an intervening variable, and to achieve the objectives, the study relied on the quantitative approach and the questionnaire was used as the main tool for collecting data. The study community represented officers, non-commissioned officers and individuals at the Sharjah Academy for Police Sciences, and the study sample amounted to 30 affiliates, i.e. the total number of employees in the Competency Development Department at the Academy. The study reached a set of results, the most prominent of which are:

- The existence …

Rattler Python, Samer Jabor

Systems Manuals - 2026

The Rattler Python project is an interactive game-based learning system that intends to teach the basic concepts of Python programming through guided instruction, gameplay challenges, and review-based assessments. The document contains a proposal for this system consisting of problem definition, background research, existing solutions, and the proposed product, together with the system scope, assumptions, and the organization of the remainder of this document.

Query Understanding In Llm-Based Conversational Information Seeking, Yifei Yuan, Zahra Abbasiantaeb, Mohammad Aliannejadi, Yang Deng

Research Collection School Of Computing and Information Systems

Query understanding in CIS involves accurately interpreting user intent through context-aware interactions. This includes resolving ambiguities, refining queries, and adapting to evolving information needs. LLM enhance this process by interpreting nuanced language and adapting dynamically, improving the relevance and precision of search results in real-time. In this tutorial, we explore advanced techniques to enhance query understanding in LLM-based CIS systems. We delve into LLM-driven methods for developing robust evaluation metrics to assess query understanding quality in multi-turn interactions, strategies for building more interactive systems, and applications like proactive query management and query reformulation. We also discuss key challenges in integrating …

Hps: Hard Preference Sampling For Human Preference Alignment, Xiandong Zou, Wanyu Lin, Yuchen Li, Pan Zhou

Research Collection School Of Computing and Information Systems

Aligning Large Language Model (LLM) responses with human preferences is vital for building safe and controllable AI systems. While preference optimization methods based on PlackettLuce (PL) and Bradley-Terry (BT) models have shown promise, they face challenges such as poor handling of harmful content, inefficient use of dispreferred responses, and, specifically for PL, high computational costs. To address these issues, we propose Hard Preference Sampling (HPS), a novel framework for robust and efficient human preference alignment. HPS introduces a training loss that prioritizes the most preferred response while rejecting all dispreferred and harmful ones. It emphasizes “hard” dispreferred responses — those …

Llm-Based Multi-Agent Systems For Software Engineering: Literature Review, Vision And The Road Ahead, Junda He, Christoph Treude, David Lo

Research Collection School Of Computing and Information Systems

Integrating Large Language Models (LLMs) into autonomous agents marks a significant shift in the research landscape by offering cognitive abilities that are competitive with human planning and reasoning. This paper explores the transformative potential of integrating Large Language Models into Multi-Agent (LMA) systems for addressing complex challenges in software engineering (SE). By leveraging the collaborative and specialized abilities of multiple agents, LMA systems enable autonomous problem-solving, improve robustness, and provide scalable solutions for managing the complexity of real-world software projects. In this paper, we conduct a systematic review of recent primary studies to map the current landscape of LMA applications …

Milpbench: A Large-Scale Benchmark Test Suite For Mixed Integer Linear Programming Problems, Huigen Ye, Yaoyang Cheng, Hua Xu, Zhiguang Cao, Hanzhang Qin

Research Collection School Of Computing and Information Systems

Mixed-integer linear programming (MILP) is a cornerstone of optimization with applications across numerous domains. However, the development and evaluation of MILP-solving algorithms are hindered by existing benchmark datasets, which are often limited in scale, lack diversity, and are poorly structured, making them inadequate for systematic testing across different solving approaches, especially for machine learning (ML)-based methods. To address these issues, we introduce MILPBench, a large-scale benchmark suite comprising 100,000 MILP instances organized into 60 well-categorized classes. Using structural properties and embedding similarity metrics, we developed a novel classification framework to ensure both intra-class homogeneity and inter-class diversity. In addition to …

Crow: Eliminating Backdoors From Large Language Models Via Internal Consistency Regularization, Nay Myat Min, Long H. Pham, Yige Li, Jun Sun

Research Collection School Of Computing and Information Systems

Large Language Models (LLMs) are vulnerable to backdoor attacks that manipulate outputs via hidden triggers. Existing defense methods—designed for vision/text classification tasks—fail for text generation. We propose Internal Consistency Regularization (CROW), a defense leveraging the observation that backdoored models exhibit unstable layer-wise hidden representations when triggered, while clean models show smooth transitions. CROW enforces consistency across layers via adversarial perturbations and regularization during finetuning, neutralizing backdoors without requiring clean reference models or trigger knowledge—only a small clean dataset. Experiments across Llama-2 (7B, 13B), CodeLlama (7B, 13B), and Mistral-7B demonstrate CROW’s effectiveness: it achieves significant reductions in attack success rates across …

Developing Sky Plots Of Rocket Launches From Gps Scintillation Data, Ishaan Dey, Kshitija Deshpande

Beyond: Undergraduate Research Journal

Sky plots displaying GPS satellite and rocket launch trajectories are developed to determine the spatial correlation between satellites that display ionospheric scintillations and heavy thrust-producing rockets. The trajectories of three major Falcon Heavy and Artemis 1 rocket launches are used within this paper. Python code is utilized to compute and plot Ionospheric Pierce Point (IPP) coordinates which are then used to produce satellite trajectories from the receiver's point of view. Rocket latitude, longitude, and altitude data is integrated within the code to provide extensive detail into the location of the rocket in relation to GPS satellites that displayed scintillations from …

Efficient And Green Large Language Models For Software Engineering: Literature Review, Vision, And The Road Ahead, Jieke Shi, Zhou Yang, David Lo

Research Collection School Of Computing and Information Systems

Large Language Models (LLMs) have recently shown remarkable capabilities in various software engineering tasks, spurring the rapid growth of the Large Language Models for Software Engineering (LLM4SE) area. However, limited attention has been paid to developing efficient LLM4SE techniques that demand minimal computational cost, time, and memory resources, as well as green LLM4SE solutions that reduce energy consumption, water usage, and carbon emissions. This article aims to redirect the focus of the research community toward the efficiency and greenness of LLM4SE, while also sharing potential research directions to achieve this goal. It commences with a brief overview of the significance …

Learning And Optimization Under Human-Centric Considerations, Qian Shao

Dissertations and Theses Collection (Open Access)

This dissertation investigates learning and optimization problems shaped by humancentric considerations, such as preferences, demonstrations, behavioral patterns, and resource constraints. As real-world decision-making increasingly involves interaction with human agents, data, and limitations, modeling these factors becomes critical for building practical, adaptive, and robust systems.

The research spans four domains. First, we study preference-aware delivery routing by learning implicit practitioner preferences and incorporating them into a hierarchical route optimization framework. Second, we develop imitation learning methods for cost-constrained settings, enabling agents to mimic expert behavior while respecting safety and resource limitations. Third,we explore early rumor detection in data-limited environments, integrating large …

Dupin: A Parallel Framework For Densest Subgraph Discovery In Fraud Detection On Massive Graphs, Jiaxin Jiang, Siyuan Yao, Yuchen Li, Qiange Wang, Bingsheng He, Min Chen

Research Collection School Of Computing and Information Systems

Detecting fraudulent activities in financial and e-commerce transaction networks is crucial. One effective method for this is Densest Subgraph Discovery (DSD). However, deploying DSD methods in production systems faces substantial scalability challenges due to the predominantly sequential nature of existing methods, which impedes their ability to handle large-scale transaction networks and results in significant detection delays. To address these challenges, we introduce Dupin, a novel parallel processing framework designed for efficient DSD processing in billion-scale graphs. Dupin is powered by a processing engine that exploits the unique properties of the peeling process, with theoretical guarantees on detection quality and efficiency. …

Reaccept: Automated Co-Evolution Of Production And Test Code Based On Dynamic Validation And Large Language Models, Jianlei Chi, Xiaotian Wang, Yuhan Huang, Lechen Yu, Di Cui, Jianguo Sun, Jun Sun

Research Collection School Of Computing and Information Systems

Synchronizing production and test code, known as PT co-evolution, is critical for software quality. Given the significant manual effort involved, researchers have tried automating PT co-evolution using predefined heuristics and machine learning models. However, existing solutions are still incomplete. Most approaches only detect and flag obsolete test cases, leaving developers to manually update them. Meanwhile, existing solutions may suffer from low accuracy, especially when applied to real-world software projects. In this paper, we propose ReAccept, a novel approach leveraging large language models (LLMs), retrievalaugmented generation (RAG), and dynamic validation to fully automate PT co-evolution with high accuracy. ReAccept employs an …

Large Language Models For Logical Fallacy Detection, Nicole Anne Hui-Ying Teo, Donghao Huang, Erik Cambria, Zhaoxia Wang

Research Collection School Of Computing and Information Systems

Identifying logical fallacies is essential for maintaining log-ical reasoning and reducing false information in a variety of domains, such as the media, law, and education. We present an extensive study on the use of large language models (LLMs) for logical fallacy detection and provide a comparative overview of model performance across various fallacy classes. We evaluate the logical fallacy detection capabilities of multiple state-of-the-art models (LLaMA, Qwen, Gemma, Phi) utilizing accuracy, precision, recall, and F1-score as assessment measures. Accord-ing to our findings, our models do well on simple fallacies like “circular reasoning,” but they have trouble with more interpretive reasoning …

Mat 301 - Applied Statistics And Data Analysis, Eric Aragundi

Open Educational Resources

Data analysis using standard statistical methods and relevant computer software. Emphasis on real-world data, interpretation, and misinterpretation of computer output.

This syllabus contains open source notebook about data analysis content.

Hotlangbench, A Tiny Benchmark Suite For Higher-Order Statically Typed Languages, Konstantin Laufer

Computer Science: Faculty Publications and Other Works

This work in progress aims to compare various HOT (higher-order and statically typed, a term coined by Phil Wadler) through reproducible course-grained, wall-time benchmarks. Our overall goals include simplicity, agility, and reproducibility.

There is currently only one benchmark, but it brings out substantial performance differences among the various languages and platforms. It uses function composition and other higher-order constructs to build a pipeline of transformations, along with a brute-force iteration that is computationally expensive for input files specifying large ranges as function domains. We currently include versions in Modern C++, C#, Go, Haskell, Kotlin, Modern (stream-based) Java (24), OCaml, Scala …

Praxly: An Online Ide For The Praxis Cs Test Pseudocode, Benjamin Saupp

James Madison Undergraduate Research Journal (JMURJ)

No abstract provided.

Hybridize Functions: A Tool For Automatically Refactoring Imperative Deep Learning Programs To Graph Execution, Raffi Khatchadourian, Tatiana Castro Vélez, Mehdi Bagherzadeh, Nan Jia, Anita Raja

Publications and Research

Efficiency is essential to support responsiveness w.r.t. ever-growing datasets, especially for Deep Learning (DL) systems. DL frameworks have traditionally embraced deferred execution-style DL code—supporting symbolic, graph-based Deep Neural Network (DNN) computation. While scalable, such development is error-prone, non-intuitive, and difficult to debug. Consequently, more natural, imperative DL frameworks encouraging eager execution have emerged but at the expense of run-time performance. Though hybrid approaches aim for the “best of both worlds,” using them effectively requires subtle considerations to make code amenable to safe, accurate, and efficient graph execution—avoiding performance bottlenecks and semantically inequivalent results. We discuss the engineering aspects of a …

Hybridize Functions: A Tool For Automatically Refactoring Imperative Deep Learning Programs To Graph Execution, Raffi Khatchadourian, Tatiana Castro Vélez, Mehdi Bagherzadeh, Nan Jia, Anita Raja

Publications and Research

Efficiency is essential to support responsiveness w.r.t. ever-growing datasets, especially for Deep Learning (DL) systems. DL frameworks have traditionally embraced deferred execution-style DL code—supporting symbolic, graph-based Deep Neural Network (DNN) computation. While scalable, such development is error-prone, non-intuitive, and difficult to debug. Consequently, more natural, imperative DL frameworks encouraging eager execution have emerged but at the expense of run-time performance. Though hybrid approaches aim for the “best of both worlds,” using them effectively requires subtle considerations to make code amenable to safe, accurate, and efficient graph execution—avoiding performance bottlenecks and semantically inequivalent results. We discuss the engineering aspects of a …

Hybridize Functions: A Tool For Automatically Refactoring Imperative Deep Learning Programs To Graph Execution, Raffi Khatchadourian, Tatiana Castro Vélez, Mehdi Bagherzadeh, Nan Jia, Anita Raja

Publications and Research

Efficiency is essential to support responsiveness w.r.t. ever-growing datasets, especially for Deep Learning (DL) systems. DL frameworks have traditionally embraced deferred execution-style DL code—supporting symbolic, graph-based Deep Neural Network (DNN) computation. While scalable, such development is error-prone, non-intuitive, and difficult to debug. Consequently, more natural, imperative DL frameworks encouraging eager execution have emerged but at the expense of run-time performance. Though hybrid approaches aim for the "best of both worlds," using them effectively requires subtle considerations to make code amenable to safe, accurate, and efficient graph execution—avoiding performance bottlenecks and semantically inequivalent results. We discuss the engineering aspects of a …

Compiling Haskell Into Lean: A Common Abstract Syntax For Haskell And Interactive Theorem Provers, Talitha Holcombe

Electrical Engineering and Computer Science (MS) Theses

In this work, we introduce a program conversion tool, HS-TO-LEAN, that uses GHC's ghc-lib-parser API to translate Haskell programs into Lean code, which is then validated by the Lean compiler. The repo can be found at https://github.com/holcombet/hs-to-lean/tree/main. The result is a successful compilation of a fragment of Haskell into correct and executable Lean code that users can prove theorems about. We conducted a case study using a heap sort algorithm to support our claim that HS-TO-LEAN produces verifiable Lean code. Our approach is inspired by recent advances in formal verification of Haskell programs in Coq, and we currently restrict our …

Evaluation Of Pre-Trained Vision Language Models In Challenging Contexts, Kankan Zhou

Dissertations and Theses Collection (Open Access)

The rapid advancement and proliferation of pre-trained vision-language models (VLMs) have heralded a new era in the realm of artificial intelligence (AI), opening up unprecedented opportunities and challenges alike. This dissertation sets forth on an ambitious and comprehensive journey to critically evaluate the performance and limitations of pre-trained VLMs, particularly in complex and challenging contexts that test the bounds of their capabilities. Our focus is twofold: to rigorously assess the extent of bias embedded in these models, and to meticulously scrutinize their reasoning abilities, highlighting parallels and disparities between machine and human cognition.

We initiate our exploration with a targeted …