Software Engineering Commons^™

Viewsrd: 3d Visual Grounding Via Structured Multi-View Decomposition, Ronggang Huang, Haoxin Yang, Yan Cai, Xuemiao Xu, Huaidong Zhang, Shengfeng He

Research Collection School Of Computing and Information Systems

3Dvisual grounding aims to identify and localize objects in a 3Dspacebasedontextualdescriptions. However, existing methods struggle with disentangling targets from anchors in complex multi-anchor queries and resolving inconsisten cies in spatial descriptions caused by perspective variations. To tackle these challenges, we propose ViewSRD, a frame work that formulates 3D visual grounding as a structured multi-view decomposition process. First, the Simple Rela tion Decoupling (SRD) module restructures complex multi anchor queries into a set of targeted single-anchor state ments, generating a structured set of perspective-aware de scriptions that clarify positional relationships. These de composed representations serve as the foundation for the Multi-view …

Emoshortcuts: Emotionally Expressive Body Augmentation For Social Mixed Reality Avatars, Hyuna Seo, Youngki Lee, Rajesh Krishna Balan, Thivya Kandappu

Research Collection School Of Computing and Information Systems

We present EmoShortcuts1, a novel social Mixed Reality (MR) framework that enhances emotional expression by dynamically augmenting avatar body gestures to reflect users’ emotional states. While social MR enables immersive remote interactions through avatars, conveying emotions remains challenging due to limitations in head-mounted display (HMD) tracking (e.g., missing lower-body movements, such as stomping or defensive postures), and users’ tendency to deprioritize nonverbal expressions during multitasking. EmoShortcuts addresses these challenges by introducing an augmentation framework that generates expressive body gestures even when users’ physical movements are restricted. We conducted a formative study with 12 participants to identify key challenges in emotional …

Morphology-Aware Hrv Estimation From Wrist Ppg In Sedentary Scenarios, Changshuo Hu, Hung Manh Pham, Dong Ma

Research Collection School Of Computing and Information Systems

Photoplethysmography (PPG) is widely used in wearable devices for non-invasive heart rate variability (HRV) monitoring. While most prior work focuses on mitigating motion artifacts, recent studies highlight that even subtle contact pressure variations can distort waveform morphology and lead to inaccurate HRV estimates. In this work, we propose a morphology-aware deep learning framework that conditions HRV estimation on beat-level waveform types. Our model jointly encodes the raw PPG waveform and a sequence of pressure-induced morphology labels using parallel encoders, integrates them via cross-attention, and predicts normal-to-normal (NN) intervals and beat count to support downstream HRV computation. Evaluated on the public …

Developing A Strong Cps Defender: An Evolutionary Approach, Qingyuan Hu, Christopher M. Poskitt, Jun Sun, Yuqi Chen

Research Collection School Of Computing and Information Systems

Cyber-physical systems (CPSs) are used extensively in critical infrastructure, underscoring the need for anomaly detection systems that are able to catch even the most motivated attackers. Traditional anomaly detection techniques typically do `one-off' training on datasets crafted by experts or generated by fuzzers, potentially limiting their ability to generalize to unseen and more subtle attack strategies. Stopping at this point misses a key opportunity: a defender can actively challenge the attacker to find more nuanced attacks, which in turn can lead to more effective detection capabilities. Building on this concept, we propose Evo-Defender, an evolutionary framework that iteratively strengthens CPS …

Tactile Data Comics: Combining Step-By-Step Presentation Of Tactile Graphics With Verbal Narration For The Blind And Visually Impaired, Yang Jiao, Ruoting Sun, Rong Luo, Xiwen Yao, Xinran She, Kotaro Hara, Yuewen Zhang, Xinyi Fu

Research Collection School Of Computing and Information Systems

Tactile graphics on a refreshable display have proven effective in enabling visually impaired people to comprehend pictorial content. To further evaluate the effectiveness of refreshable tactile displays in blind education, we designed tactile data comics, a method that combines step-by-step presentation of tactile graphics with verbal narration. We conducted a user study with sixteen visually impaired students to compare tactile data comics against verbal-only and static tactile graphics. Our findings show that tactile data comics significantly improve participants’ comprehension and engagement during the learning experience. These empirical results suggest that the integration of refreshable tactile displays and tactile data comics …

Teaching Diffusion Models To Ground Alpha Matte, Tianyi Xiang, Weiying Zheng, Yutao Jiang, Tingrui Shen, Hewei Yu, Yangyang Xu, Shengfeng He

Research Collection School Of Computing and Information Systems

The power of visual language models is showcased in visual understanding tasks, where language-guided models achieve impressive flexibility and precision. In this paper, we ex tend this capability to the challenging domain of image matting by framing it as a soft grounding problem, enabling a single diffusion model to handle diverse objects, textures, and transparencies, all directed by descriptive text prompts. Our method teaches the diffusion model to ground alpha mattes by guiding it through a process of instance-level localization and transparency estimation. First, we introduce an intermediate objective that trains the model to accurately localize semantic components of the …

Information-Bottleneck Driven Binary Neural Network For Change Detection, Kaijie Yin, Zhiyuan Zhang, Shu Kong, Tian Gao, Cheng-Zhong Xu, Hui Kong

Research Collection School Of Computing and Information Systems

In this paper, we propose Binarized Change Detection (BiCD), the first binary neural network (BNN) designed specifically for change detection. Conventional network binarization approaches, which directly quantize both weights and activations in change detection models, severely limit the network's ability to represent input data and distinguish between changed and unchanged regions. This results in significantly lower detection accuracy compared to real-valued networks. To overcome these challenges, BiCD enhances both the representational power and feature separability of BNNs, improving detection performance. Specifically, we introduce an auxiliary objective based on the Information Bottleneck (IB) principle, guiding the encoder to retain essential input …

From Release To Adoption: Challenges In Reusing Pre-Trained Ai Models For Downstream Developers, Peerachai Banyongrakkul, Mansooreh Zahedi, Patanamon Thongtanunam, Christoph Treude, Haoyu Gao

Research Collection School Of Computing and Information Systems

Pre-trained models (PTMs) have gained widespread popularity and achieved remarkable success across various fields, driven by their groundbreaking performance and easy accessibility through hosting providers. However, the challenges faced by downstream developers in reusing PTMs in software systems are less explored. To bridge this knowledge gap, we qualitatively created and analyzed a dataset of 840 PTM-related issue reports from 31 OSS GitHub projects. We systematically developed a comprehensive taxonomy of PTM-related challenges that developers face in downstream projects. Our study identifies seven key categories of challenges that downstream developers face in reusing PTMs, such as model usage, model performance, and …

Finding Safety Violations Of Ai-Enabled Control Systems Through The Lens Of Synthesized Proxy Programs, Jieke Shi, Zhou Yang, Junda He, Bowen Xu, Dongsun Kim, Donggyun Han, David Lo

Research Collection School Of Computing and Information Systems

Given the increasing adoption of modern AI-enabled control systems, ensuring their safety and reliability has become a critical task in software testing. One prevalent approach to testing control systems is falsification, which aims to find an input signal that causes the control system to violate a formal safety specification using optimization algorithms. However, applying falsification to AI-enabled control systems poses two significant challenges: (1) it requires the system to execute numerous candidate test inputs, which can be time-consuming, particularly for systems with AI models that have many parameters, and (2) multiple safety requirements are typically defined as a conjunctive specification, …

Boosting Symbolic Execution For Vulnerability Detection, Haoxin Tu

Dissertations and Theses Collection (Open Access)

Software systems written by humans tend to be unreliable and insecure, hence, bugs or vulnerabilities in them are inevitable. Symbolic execution has shown considerable potential in detecting diverse types of software bugs and also vulnerabilities that have severe security implications. However, existing symbolic execution engines still suffer from at least three fundamental limitations in memory modeling, path exploration, and structured input generation, which significantly impede existing engines from efficiently and effectively detecting software bugs and vulnerabilities.

The objective of this dissertation is to boost existing symbolic execution engines by designing a new memory model, two new path exploration strategies, and …

Apidocbooster: An Extract-Then-Abstract Framework Leveraging Large Language Models For Augmenting Api Documentation, Chengran Yang, Jiakun Liu, Bowen Xu, Christoph Treude, Yunbo Lyu, Junda He, Ming Li, David Lo

Research Collection School Of Computing and Information Systems

API documentation is often the most trusted resource for programming. Many approaches have been proposed to augment API documentation by summarizing complementary information from external resources such as Stack Overflow. Existing extractive-based summarization approaches excel in producing faithful summaries that accurately represent the source content without input length restrictions. Nevertheless, they suffer from inherent readability limitations. On the other hand, our empirical study on the abstractive-based summarization method, i.e., GPT-4, reveals that GPT-4 can generate coherent and concise summaries but presents limitations in terms of informativeness and faithfulness. We introduce APIDocBooster, an extract-then-abstract framework that seamlessly fuses the advantages of …

Static Analysis As A Feedback Loop: Enhancing Llm-Generated Code Beyond Correctness, Scott Blyth, Sherlock Licorish, Christoph Treude, Markus Wagner

Research Collection School Of Computing and Information Systems

Large language models (LLMs) have demonstrated impressive capabilities in code generation, achieving high scores on benchmarks such as HumanEval and MBPP. However, these benchmarks primarily assess functional correctness and neglect broader dimensions of code quality, including security, reliability, readability, and maintainability. In this work, we systematically evaluate the ability of LLMs to generate high-quality code across multiple dimensions using the PythonSecurityEval benchmark. We introduce an iterative static analysis-driven prompting algorithm that leverages Bandit and Pylint to identify and resolve code quality issues. Our experiments with GPT-4o show substantial improvements: security issues reduced from >40% to 13%, readability violations from >80% …

Educator Perceptions Of Devops Teaching Recommendations And Their Alignment With Common Challenges, Marcelo Romulo Fernandes, Pablo Paiva, Samuel Lucas De Moura Ferino, Roberta Coelho, Christoph Treude, Eduardo Aranha, Uirá Kulesza

Research Collection School Of Computing and Information Systems

DevOps education presents unique pedagogical challenges due to the diversity of tools, rapid technological change, and the multidisciplinary nature of the field. Although previous work has proposed recommendations to address these challenges, it is unclear how educators perceive these recommendations and whether they align with the challenges encountered in practice. In this paper, we present a quantitative and qualitative methods study involving 11 DevOps educators who interacted with Improve, a tool that presents a curated set of educational challenges and recommendations derived from previous literature. Educators indicated which recommendations they already use, which they intend to use, and which challenges …

Studying Satd In Drone Systems With Human-Ai Collaboration, Leevi Rantala, Lwin Khin Shar, Mäntylä Mika V., Wei Minn, Naing Tun Yan

Research Collection School Of Computing and Information Systems

Background: Self-Admitted Technical Debt (SATD) refers to sub-optimal solutions that developers acknowledge within the source code. SATD research originated on Java projects but is expanding to other domains. We focus on SATD in drones, which are used for various critical tasks.Aims: The primary objective is to investigate SATD in drone systems. The second aim is to explore the integration of AI and human collaboration for SATD labelling and classification.Method: We conducted a sample study of SATD comments in drone systems (14 open source, 4 SDKs) to analyse the quantity and types of SATD comments present. Our study incorporates collaboration between …

Stylegan-∞: Extending Stylegan To Arbitrary-Ratio Translation With Stylebook, Yihua Dai, Tianyi Xiang, Bailin Deng, Yong Du, Hongmin Cai, Jing Qin, Shengfeng He

Research Collection School Of Computing and Information Systems

Although pre-trained large-scale generative models StyleGAN series have proven to be effective in various editing and translation tasks, they are limited to pre-defined fixed aspect ratio. To overcome this limitation, we propose StyleGAN-∞, a model that enables pre-trained StyleGAN to perform arbitrary-ratio conditional synthesis. Our key insight is to distill the expressive StyleGAN features into a StyleBook, such that an arbitrary-ratio condition can be translated to other forms by properly assembling pre-defined StyleBook vectors. To learn and leverage the StyleBook, we employ a network with three distinct stages, each corresponding to StyleBook extraction, StyleBook correspondence learning, and arbitrary-ratio synthesis. Extensive …

Improving Co-Decoding Based Security Hardening Of Code Llms Leveraging Knowledge Distillation, Dong Li, Shanfu Shu, Meng Yan, Zhongxin Liu, Chao Liu, Xiaohong Zhang, David Lo

Research Collection School Of Computing and Information Systems

Large Language Models (LLMs) have been widely adopted by developers in software development. However, the massive pretraining code data is not rigorously filtered, allowing LLMs to learn unsafe coding patterns. Several prior studies have demonstrated that code LLMs tend to generate code with potential vulnerabilities. The widespread adoption of intelligent programming assistants poses a significant threat to the software development process. Existing approaches to mitigating this risk primarily involve constructing secure data that are free of vulnerabilities and then retraining or fine-tuning the models. However, such an effort is resource intensive and requires significant manual supervision. When the model parameters …

Exploring Parameter-Efficient Fine-Tuning Techniques For Code Generation With Large Language Models, Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, Houari A. Sahraoui

Research Collection School Of Computing and Information Systems

Large language models (LLMs) demonstrate impressive capabilities to generate accurate code snippets given natural language intents in a zero-shot manner, i.e., without the need for specific fine-tuning. While prior studies have highlighted the advantages of fine-tuning LLMs, this process incurs high computational costs, making it impractical in resource-scarce environments, particularly for models with billions of parameters. To address these challenges, previous research explored in-context learning (ICL) and retrieval-augmented generation (RAG) as strategies to guide the LLM generative process with task-specific prompt examples. However, ICL and RAG introduce inconveniences, such as the need for designing contextually relevant prompts and the absence …

Map As A By-Product: Collective Landmark Mapping From Imu Data And User-Provided Texts In Situated Tasks, Ryo Yonetani, Kotaro Hara

Research Collection School Of Computing and Information Systems

This paper presents Collective Landmark Mapper, a novel map-as-a-by-product system for generating semantic landmark maps of indoor environments. Consider users engaged in situated tasks that require them to navigate these environments and regularly take notes on their smartphones. Collective Landmark Mapper exploits the smartphone's IMU data and the user's free text input during these tasks to identify a set of landmarks encountered by the user. The identified landmarks are then aggregated across multiple users to generate a unified map representing the positions and semantic information of all landmarks. In developing the proposed system, we focused specifically on retail applications and …

Robface: A Test Suite For Efficient Robustness Evaluation Of Face Recognition Systems, Ruihan Zhang, Jun Sun

Research Collection School Of Computing and Information Systems

Face recognition is a widely used authentication technology in practice, where robustness is required. It is thus essential to have an efficient and easy-to-use method for evaluating the robustness of (possibly third-party) trained face recognition systems. Existing approaches to evaluating the robustness of face recognition systems are either based on empirical evaluation (e.g., measuring attacking success rate using state-of-the-art attacking methods) or formal analysis (e.g., measuring the Lipschitz constant). While the former demands significant user efforts and expertise, the latter is extremely time-consuming. In pursuit of a comprehensive, efficient, easy-to-use, and scalable estimation of the robustness of face recognition systems, …

Ponzilens+: Visualizing Bytecode Actions For Smart Ponzi Scheme Identification, Xiaolin Wen, Tai D. Nguyen, Shaolun Ruan, Qiaomu Shen, Jun Sun, Feida Zhu, Yong Wang

Research Collection School Of Computing and Information Systems

With the prevalence of smart contracts, smart Ponzi schemes have become a common fraud on blockchain and have caused significant financial loss to cryptocurrency investors in the past few years. Despite the critical importance of detecting smart Ponzi schemes, a reliable and transparent identification approach adaptive to various smart Ponzi schemes is still missing. To fill the research gap, we first extract semantic-meaningful actions to represent the execution behaviors specified in smart contract bytecodes, which are derived from a literature review and in-depth interviews with domain experts. We then propose PonziLens+, a novel visual analytic approach that provides an intuitive …

Preface, Special Issue For The 16th International Conference On Graph Transformation (Icgt 2023), Maribel Fernández, Christopher M. Poskitt

Research Collection School Of Computing and Information Systems

This special issue contains six extended versions of papers presented at the 16th International Conference on Graph Transformation (ICGT 2023), held in Leicester, UK, on 19–20 July 2023. The conference was part of STAF 2023 (Software Technologies: Applications and Foundations) and was held under the auspices of the European Association for Theoretical Computer Science (EATCS), the European Association of Software Science and Technology (EASST), and the IFIP Working Group 1.3, Foundations of Systems Specification.

Evaluating And Mitigating Linguistic Discrimination In Large Language Models: Perspectives On Safety Equity And Knowledge Equity, Guoliang Dong, Haoyu Wang, Jun Sun, Xinyu Wang

Research Collection School Of Computing and Information Systems

By training on text in various languages, large language models (LLMs) typically possess multilingual support and demonstrate remarkable capabilities in solving tasks described in different languages. However, LLMs can exhibit linguistic discrimination due to the uneven distribution of training data across languages. That is, LLMs are hard to keep the consistency of responses when faced with the same task but depicted in different languages. In this study, we first explore the consistency in the LLMs’ outputs responding to queries in various languages from two aspects: safety and quality. We conduct this analysis with two datasets (AdvBench and NQ) based on …

Assessing The Robustness Of Test Selection Methods For Deep Neural Networks, Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Wei Ma, Mike Papadakis, Lei Ma, Yves Le Traon

Research Collection School Of Computing and Information Systems

Regularly testing deep learning-powered systems on newly collected data is critical to ensure their reliability, robustness, and efficacy in real-world applications. This process is demanding due to the significant time and human effort required for labeling new data. While test selection methods alleviate manual labor by labeling and evaluating only a subset of data while meeting testing criteria, we observe that such methods with reported promising results are simply evaluated, e.g., testing on original test data. The question arises: are they always reliable? In this article, we explore when and to what extent test selection methods fail. First, we identify …

Information-Theoretic Detection Of Unusual Source Code Changes, Adriano Torres, Markus Wagner, Christoph Treude, Sebastian Baltes

Research Collection School Of Computing and Information Systems

The code base of software projects evolves essentially through inserting and removing information to and from the source code. We can measure this evolution via the elements of information—tokens, words, nodes—of the respective representation of the code. In this work, we approach the measurement of the information content of the source code of open-source projects from an information-theoretic standpoint. Our focus is on the entropy of two fundamental representations of code: tokens and abstract syntax tree nodes, from which we derive definitions of textual and structural entropy. We proceed with an empirical assessment where we evaluate the evolution patterns of …

Leveraging Reviewer Experience In Code Review Comment Generation, Hong Yi Lin, Patanamon Thongtanunam, Christoph Treude, Michael W. Godfrey, Chunhua Liu, Wachiraphan Charoenwet

Research Collection School Of Computing and Information Systems

Modern code review is a ubiquitous software quality assurance process aimed at identifying and resolving potential issues (e.g., functional, evolvability) within newly written code. Despite its effectiveness, the process demands large amounts of effort from the human reviewers involved. To help alleviate this workload, researchers have trained various deep learning based language models to imitate human reviewers in providing natural language code reviews for submitted code. Formally, this automation task is known as code review comment generation. Prior work has demonstrated improvements in code review comment generation by leveraging machine learning techniques and neural models, such as transfer learning and …

Freqllm: Frequency-Aware Large Language Models For Time Series Forecasting, Shunan Wang, Min Gao, Zongwei Wang, Yibing Bai, Feng Jiang, Guansong Pang

Research Collection School Of Computing and Information Systems

Large Language Models (LLMs) have recently shown promise in Time Series Forecasting (TSF) by effectively capturing intricate time-domain dependencies. However, our preliminary experiments reveal that standard LLM-based approaches often fail to capture global correlations, limiting predictive performance. We found that embedding frequency-domain signals smooths weight distributions and enhances structured correlations by clearly separating global trends (low-frequency components) from local variations (high-frequency components). Building on these insights, we propose FreqLLM, a novel framework that integrates frequency-domain semantic alignment into LLMs to refine prompts for improved time series analysis. By bridging the gap between frequency signals and textual embeddings, FreqLLM effectively captures …

Anomalygfm: Graph Foundation Model For Zero/Few-Shot Anomaly Detection, Hezhe Qiao, Chaoxi Niu, Ling Chen, Guansong Pang

Research Collection School Of Computing and Information Systems

Graph anomaly detection (GAD) aims to identify abnormal nodes that differ from the majority of the nodes in a graph, which has been attracting significant attention in recent years. Existing generalist graph models have achieved remarkable success in different graph tasks but struggle to generalize to the GAD task. This limitation arises from their difficulty in learning generalized knowledge for capturing the inherently infrequent, irregular and heterogeneous abnormality patterns in graphs from different domains. To address this challenge, we propose AnomalyGFM, a GAD-oriented graph foundation model that supports zero-shot inference and few-shot prompt tuning for GAD in diverse graph datasets. …

Gnncontext: Gnn-Based Code Context Prediction For Programming Tasks, Xiaoye Zheng, Zhiyuan Wan, Shun Liu, Kaiwen Yang, David Lo, Xiaohu Yang

Research Collection School Of Computing and Information Systems

A code context model comprises source code elements and their relations relevant to a programming task. The capture and use of code context models in software tools can benefit software development practices, such as code navigation and search. Prior research has explored approaches that leverage either the structural information of code or interaction histories of developers with integrated development environments to automate the construction of code context models. However, these approaches primarily capture shallow syntactic and lexical features of code elements, with limited ability to capture contextual and structural dependencies among neighboring code elements. In this paper, we propose GNNContext, …

Unified Neural Backdoor Removal With Only Few Clean Samples Through Unlearning And Relearning, Nay Myat Min, Long H. Pham, Jun Sun

Research Collection School Of Computing and Information Systems

Deep neural networks have achieved remarkable success across various applications; however, their vulnerability to backdoor attacks poses severe security risks—especially in situations where only a limited set of clean samples is available for defense. In this work, we address this critical challenge by proposing ULRL (UnLearn and ReLearn for backdoor removal), a novel two-phase approach for comprehensive backdoor removal. Our method first employs an unlearning phase, in which the network’s loss is intentionally maximized on a small clean dataset to expose neurons that are excessively sensitive to backdoor triggers. Subsequently, in the relearning phase, these suspicious neurons are recalibrated using …

Exploring The Capabilities Of Llms For Code-Change-Related Tasks, Lishui Fan, Jiakun Liu, Zhongxin Liu, David Lo, Xin Xia, Shanping Li

Research Collection School Of Computing and Information Systems

Developers deal with code-change-related tasks daily, e.g., reviewing code. Pre-trained code and code-change-oriented models have been adapted to help developers with such tasks. Recently, large language models (LLMs) have shown their effectiveness in code-related tasks. However, existing LLMs for code focus on general code syntax and semantics rather than the differences between two code versions. Thus, it is an open question how LLMs perform on code-change-related tasks.To answer this question, we conduct an empirical study using 1B parameters LLMs on three code-change-related tasks, i.e., code review generation, commit message generation, and just-in-time comment update, with in-context learning (ICL) and parameter-efficient …