Software Engineering Commons^™

Ai Failures In The Eyes Of The Downstream Developer: A First Look At Concerns, Practices, And Challenges, Haoyu Gao, Mansooreh Zahedi, Wenxin Jiang, Hong Yi Lin, James C. Davis, Christoph Treude

Research Collection School Of Computing and Information Systems

With the advancement of AI models, more software systems are adopting AI as a component to facilitate automation. Pre-trained models (PTMs) have become a cornerstone of AI-based software, allowing for rapid integration and development with lower training cost. However, their adoption also introduces failure modes such as data leakage and biased outputs, that may require careful handling by downstream developers. While previous research has proposed taxonomies of these technical concerns and various mitigation strategies, how downstream developers address these issues during the development of general AI-based software when reusing PTMs remains unexplored. Understanding downstream developers’ perspectives is essential because they …

Analyzing Developer Discussions On Eu And Us Privacy Legislation Compliance In Github Repositories, Georgia M. Kapitsaki, Maria Papoutsoglou, Christoph Treude, Ioanna Theophilou

Research Collection School Of Computing and Information Systems

Context: Privacy legislation has impacted the way software systems are developed, prompting practitioners to update their implementations. Specifically, the EU General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) have forced the community to focus on users’ data privacy. Objectives: Relying on the vast amount of data on developer issues available in GitHub repositories, our aim is to gather empirical evidence on the issues developers of Open Source Software discuss to comply with privacy legislation. Method: We examined such discussions by mining and analyzing 32,820 issues from GitHub repositories. We partially analyzed the dataset automatically to identify …

Mutation-Based Multi-Agent Test Case Update, Dawei Tian, Jiakun Liu, Yun Peng, Yichen Zhang, Jianlei Chi, Jun Sun, Xiaohong Su

Research Collection School Of Computing and Information Systems

Modern software systems evolve rapidly under CI/CD practices, where tests are critical for quality. However, substantial code changes often render existing test cases obsolete, causing pipeline disruptions, reduced productivity, and compromised quality. Recent automatic test update approaches leverage LLMs to refine test cases via execution feedback and exact-matching context retrieval, prioritizing executability and line coverage but suffering three limitations: (1) neglecting test assertion adequacy, weakening fault detection; (2) relying on coarse line coverage instead of specific uncovered lines/branches; (3) using exact-matching retrieval, which fails for LLM hallucinated queries. To address these, we propose MuMuTestUp, a mutation-guided multi-agent framework with three …

Ddor: Delta Debugging For Explainable Overrefusal Testing And Repair, Qinyan Zhou, Peixin Zhang, Jun Sun, Haonan Zhang, Dongxia Wang

Research Collection School Of Computing and Information Systems

While safety alignment and guardrails help large language models (LLMs) avoid harmful outputs, they can also induce overrefusal, i.e., unwarranted rejection of benign queries that merely appear risky. We present DDOR (Delta Debugging for OverRefusal), a fully automated and explainable framework for overrefusal testing and repair in a black-box setting, where only model inputs and outputs are accessible and internal safety mechanisms remain opaque. DDOR applies delta debugging to localize minimal refusal-triggering fragments (mRTFs) that provide phrase-level, explainable evidence for why a refusal occurs. Conditioned on these mRTFs, DDOR generates diverse, context-rich prompts and performs multi-oracle validation to filter intrinsically …

Bayesian And Multi-Objective Decision Support For Incident Mitigation In Cyber-Physical Systems, Shaofei Huang, Christopher M. Poskitt, Lwin Khin Shar

Research Collection School of Computing and Information Systems

Cyber-physical systems increasingly rely on interconnected physical and digital systems whose security incidents can escalate rapidly into safety and operational failures. Existing decision-support approaches struggle to support incident response because they rely on static assumptions, incomplete vulnerability data, and single-objective risk models that do not adequately capture trade-offs between attack success likelihood, impact severity, and system availability. This paper proposes an adaptive decision-support framework for incident mitigation in cyber-physical systems that integrates hierarchical Bayesian Network modelling, confidence-calibrated exposure estimation, and multi-objective optimisation into a unified, adaptive pipeline. The framework constructs probabilistic models from system architecture and vulnerability data, incorporating complementary …

Llm-As-A-Judge For Software Engineering: Literature Review, Vision, And The Road Ahead, Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, David Lo

Research Collection School Of Computing and Information Systems

The rapid integration of Large Language Models (LLMs) into software engineering (SE) has revolutionized tasks from code generation to program repair, producing a massive volume of software artifacts. This surge in automated creation has exposed a critical bottleneck: the lack of scalable and reliable methods to evaluate the quality of these outputs. Human evaluation, while effective, is very costly and time-consuming. Traditional automated metrics like BLEU rely on high-quality references and struggle to capture nuanced aspects of software quality, such as readability and usefulness. In response, the LLM-as-a-Judge paradigm, which employs LLMs for automated evaluation, has emerged. This approach leverages …

Configuring Agentic Ai Coding Tools: An Exploratory Study, Matthias Galster, Seyedmoein Mohsenimofidi, Jai Lal Lulla, Muhammad Auwal Abubakar, Christoph Treude, Sebastian Baltes

Research Collection School Of Computing and Information Systems

Agentic AI coding tools increasingly automate software development tasks. Developers can configure these tools through versioned repository-level artifacts such as Markdown and JSON files. We present a systematic analysis of configuration mechanisms for agentic AI coding tools, covering Claude Code, GitHub Copilot, Cursor, Gemini, and Codex. We identify eight configuration mechanisms spanning from static context to executable and external integrations and, in an empirical study of 2,853 GitHub repositories, examine whether and how they are adopted, with a detailed analysis of Context Files, Skills, and Subagents. First, Context Files dominate the configuration landscape and are often the sole mechanism in …

Anomaly Management In Unmanned Aerial Vehicles: A Systematic Literature Review, Ivan Tan Wei Han, Christopher M. Poskitt, Lingxiao Jiang, Lwin Khin Shar

Research Collection School Of Computing and Information Systems

Unmanned Aerial Vehicles (UAVs) are increasingly deployed in safety-critical applications such as logistics, surveillance, disaster response, and urban air mobility. While their autonomy enables powerful capabilities, it also introduces vulnerabilities due to hardware faults, software defects, communication failures, and adversarial interference. This survey presents a comprehensive review of research studies closely related to UAV anomalies published between 2015 and 2025, covering 111 papers from academic and industrial sources. We introduce a unified five-pillar taxonomy—anomaly generation, prevention, detection, recovery, and analysis—that organizes existing work across the full anomaly management lifecycle. In contrast to prior surveys that focus primarily on detection algorithms, …

Survey On Learning-Based Dynamic Fault Localization: From Traditional Machine Learning To Large Language Models, Chunyan Liu, Yan Lei, Huan Xie, Jinping Wang, Yue Yu, David Lo

Research Collection School Of Computing and Information Systems

Learning-based dynamic fault localization techniques play a crucial role in the field of software engineering. These techniques dynamically execute test cases to meticulously extract useful knowledge from the execution information in the program, with the aim of identifying fault locations by leveraging machine learning, deep learning, and large language models. Currently, there is already a flourishing body of research that is intensely focused on learning-based dynamic fault localization. Research literature can be categorized into two main aspects for learning-based dynamic fault localization: data-based enhancements (i.e., the datasets) and model-based enhancements (i.e., the suspiciousness algorithms). Thus, we conduct an extensive literature …

Accountable Agents In Software Engineering: An Analysis Of Terms Of Service And A Research Roadmap, Christoph Treude

Research Collection School Of Computing and Information Systems

AI coding assistants and autonomous agents are becoming integral to software development workflows, reshaping how code is produced, reviewed, and maintained. While recent research has focused mainly on the capabilities and impacts of productivity of these systems, much less attention has been paid to accountability: who is responsible when agents generate, modify, or recommend code? In practice, accountability is defined through the Terms of Service (ToS) and related policy documents that govern the use of AI-powered development tools.In this vision paper, we present a comparative analysis of the Terms of Service for widely used AI coding assistants and agent-enabled development …

Operationalizing Ethics For Ai Agents: How Developers Encode Values Into Repository Context Files, Christoph Treude, Sebastian Baltes, Marc Cheong

Research Collection School Of Computing and Information Systems

As AI coding agents become embedded in software development workflows, developers are beginning to operationalize ethical principles by encoding behavioral rules into repository-level context files for AI agents, such as AGENTS.md files. Rather than examining the ethics of AI agents in the abstract, this vision paper investigates how ethics and values are already being translated for AI agents into actionable instructions that shape agent behavior. Through a preliminary investigation, we find that developers are already embedding guidance related to fairness, accessibility, sustainability, tone, and privacy. These artifacts function as a developer-authored governance layer, translating abstract principles into situated, natural-language directives …

A Dataset Of Agentic Ai Coding Tool Configurations, Matthias Galster, Seyedmoein Mohsenimofidi, Levi Böhme, Jai Lal Lulla, Muhammad Auwal Abubakar, Christoph Treude, Sebastian Baltes

Research Collection School Of Computing and Information Systems

Agentic AI coding tools such as Claude Code and OpenAI Codex execute multi-step coding tasks with limited human oversight. To steer these tools, developers create repository-level configuration artifacts (e.g., Markdown files) for configuration mechanisms such as Context Files, Skills, Rules, and Hooks. There is no curated dataset yet that captures these configurations at scale. This dataset, collected from open-source GitHub repositories, fills that gap. We selected 40,585 actively maintained repositories through metadata filtering, classified them using GPT-5.2 to identify 36,710 as belonging to engineered software projects, and systematically detected configuration artifacts in these repositories. The dataset covers 4,738 repositories across …

Train In Vain: Functionality-Preserving Poisoning To Prevent Unauthorized Use Of Code Datasets, Yuan Xiao, Yuchen Chen, Jiaming Wang, Wei Song, Jun Sun, Shiqing Ma, Yanzhou Mu, Juan Zhai, Chunrong Fang, Jin Song Dong, Zhenyu Chen

Research Collection School Of Computing and Information Systems

The widespread availability of large-scale code datasets has accelerated the development of code large language models (CodeLLMs), raising concerns about unauthorized dataset usage. Dataset poisoning offers a proactive defense by reducing the utility of such unauthorized training. However, existing poisoning methods often require full-dataset poisoning and introduce transformations that break code compilability. In this paper, we introduce FunPoison, a functionality-preserving poisoning approach that injects short, compilable weak-use fragments into executed code paths. FunPoison leverages reusable statement-level templates with automatic repair and conservative safety checking to ensure side-effect freedom, while a type-aware synthesis module preserves type correctness, suppresses static-analysis warnings, and …

Activity Transition Graph Generation: How Far Are We?, Jiakun Liu, Peixin Zhang, Han Hu, Yonghui Liu, Wei Minn, Ferdian Thung, Shahar Maoz, Eran Toch, Debin Gao, David Lo

Research Collection School Of Computing and Information Systems

Android applications (i.e., apps) are indispensable nowadays and are getting bigger and bigger with an increasing number offunctionalities. To understand how to access functionalities in an app, prior studies proposed tools to model the transitionsbetween functionalities with the activity transition graph (ATG). ATG is an important data structure and has been used forvarious Android app analyses, including app design, understanding, and testing. However, there is no benchmarking work onATG generation. It is still unclear whether the transitions identified by tools are correct and how many transitions are missed.To fill this gap, we manually identified all transitions in 98 applications to …

How Do Machine Learning Models Change?, Joel Castaño, Rafael Cabañas, Antonio Salmerón, David Lo, Silverio Martínez-Fernández

Research Collection School Of Computing and Information Systems

The proliferation of Machine Learning (ML) models and their open source implementations has transformed AI research and applications. Platforms like Hugging Face (HF) enable this evolving ecosystem, yet a large-scale longitudinal study of how these models change is lacking. This study addresses this gap by analyzing over 680,000 commits from 100,000 models and 2,251 releases from 202 of these models on HF using repository mining and longitudinal methods. We apply an extended ML change taxonomy to classify commits and use Bayesian networks to model temporal patterns in commit and release activities. Our findings show that commit activities align with established …

Patchfuzz: Patch Fuzzing For Javascript Engines, Junjie Wang, Zhihua Xie, Xiaofei Xie, Xiaoning Du, Xiangwei Zhang

Research Collection School Of Computing and Information Systems

Context: Patch fuzzing is a technique aimed at identifying vulnerabilities that arise from newly patched code. While researchers have made efforts to apply patch fuzzing to testing JavaScript (JS) engines with considerable success, these efforts have been limited to using ordinary test cases or publicly available vulnerability PoCs (Proof of Concepts) as seeds, and the sustainability of these approaches is hindered by the challenges associated with automating the PoC collection. Objective: To address these limitations, we propose an end-to-end sustainable approach for JS engine patch fuzzing, named PatchFuzz. Method: It automates the collection of PoCs of a broader range of …

Hydpn: A Hybrid Deep Reinforcement Learning, Programming, And Neighborhood Operations Framework For Integrated Scheduling On Parallel Batch Processing Machines, Yuqi Wang, He Luo, Guoqiang Wang, Zhaoxia Wang

Research Collection School Of Computing and Information Systems

Batch processing machines (BPMs) are widely used in industries such as semiconductors, metal processing, and healthcare, where jobs are processed in batches. As production, inventory, and distribution become increasingly integrated to improve efficiency, research on their joint scheduling in parallel BPM environments remains scarce. This paper addresses the integrated scheduling problem in parallel BPMs, involving production, inventory, and distribution stages, with the objective of minimizing total costs. A unified cost-based model is first formulated, applicable to both in-facility and external distribution scenarios. A hybrid algorithm framework, HyDPN, combining deep reinforcement learning, dynamic programming, and neighborhood operations is proposed. Extensive experiments …

On-The-Fly Generation-Quality Enhancement Of Deep Code Models Via Model Collaboration, Weifeng Sun, Naiqi Huang, Meng Yan, Zhongxin Liu, Hongyan Li, Yan Lei, David Lo

Research Collection School Of Computing and Information Systems

The growing prominence of deep code models in automating software engineering tasks is undeniable. However, their deployment encounters significant challenges in on-the-fly performance enhancement, which refers to dynamically improving the performance of deep code models during real-time execution. Conventional techniques, such as retraining or fine-tuning, are effective in controlled pre-deployment scenarios but fall short when adapting to on-the-fly adjustments post-deployment. CodeDenoise, a notable on-the-fly performance enhancement technology, leverages uncertainty-based methods to identify misclassified inputs and applies an input modification strategy to rectify classification errors. While effective for classification tasks, this approach is inapplicable to generative tasks due to two key …

Hide-And-Sweep: Detecting Concealed Cameras Via Led Illumination Sweeps, Jonghyuk Yun, Jaeyoung Moon, Yunseo Park, Sean Rui Xiang Tan, Byunghyun Kim, Rajesh Krishna Balan, Jun Han

Research Collection School Of Computing and Information Systems

Hidden cameras have increasingly infiltrated hotel and Airbnb rooms, posing serious privacy risks. Detecting such cameras is challenging because they are visually inconspicuous and often embedded inside everyday objects. Even worse, existing handheld detectors are manual and also rely on single-angle illumination and hence suffer from high false-positive rates. We present SweepLED (pronounced "sweepled")1, a practical hidden camera detection system that operates on a commodity smartphone augmented with an unobtrusive LED-embedded case. SweepLED performs LED sweeping - a controlled sequence of multi-angle illumination - while the user simply holds the phone still by hand, enabling the camera to capture how …

Towards Auto-Evaluation For Large Language Models, Jiahao Ying

Dissertations and Theses Collection (Open Access)

The rapid advancement of large language models (LLMs) has created an urgent need for evaluation methodologies that are timely, scalable, reliable, and informative. Conventional evaluation benchmarks, although essential for measuring model capabilities and guiding model development, are often constructed and maintained through labor-intensive human annotation. As LLMs continue to improve through increases in model scale, training data, and computational resources, static benchmarks may quickly lose discriminative power. Moreover, the growing use of large and diverse training corpora increases the risk of benchmark leakage, which can inflate evaluation results and obscure the true capabilities of models. These challenges call for a …

“Alexa, Do Not Say That In Front Of My Boss!” A Cross-Cultural Comparison Of User And Ai Preferences For Privacy-Aware Smart Speaker Interactions Across Contexts, Lynne Warin, Emily Aurelia, Anthony Tang, Emily Aurelia, Delphine Reinhardt

Research Collection School Of Computing and Information Systems

Due to their limited ability to reason about the social context in which they are used, smart speakers pose significant privacy risks by responding in ways that may violate people's implicit social boundaries. We conducted a cross-cultural vignette study (N = 944) in Germany and Singapore to investigate how situational factors—specifically social context (bystander relationships and closeness), physical context (location), and interaction context (topic and deceptive intent)—regulate user preferences for smart speaker responses. Our results demonstrate that these factors are superior predictors of response preferences than dispositional user traits (i.e., intrinsic personal traits). We identify two distinct social dynamics: a …

Func: Reducing The Impact Of Android Framework Evolution On Malware Detection, Hailong Yu, Tiantian Wang, Lwin Khin Shar, Hanmeng Li, David Lo

Research Collection School Of Computing and Information Systems

Android malware detection approaches commonly use APIs and permissions as features for classifying malware. However, since the release of the first Android operating system in 2008, the Android framework has undergone numerous version updates. The evolution of the Android framework over time has led to changes in APIs and permissions, including deprecations and replacements. These changes can result in inaccurate characterization of Android malware, thereby affecting performance of malware detectors. There is a lack of methods to mitigate the impact of Android framework evolution on malware detection. To fill this gap, we conduct a systematic study of the impact of …

Open Source Software Development Tool Installation: Challenges And Strategies For Novice Developers, Larissa Salerno, Christoph Treude, Patanamon Thongtanunam

Research Collection School Of Computing and Information Systems

As the world of technology advances, so do the tools that software developers use to create new programs. In recent years, software development tools have become more popular, allowing developers to work more efficiently and produce higher-quality software. Still, installing such tools can be challenging for novice developers at the early stage of their careers, as they may face issues such as compatibility problems (e.g., with operating systems) and unclear instructions. Therefore, this work aims to investigate the challenges novice developers face when installing software development tools and the strategies they employ to overcome them. To investigate these, we conducted …

Natural Adversaries: Fuzzing Autonomous Vehicles With Realistic Roadside Object Placements, Yang Sun, Haoyu Wang, Christopher M. Poskitt, Jun Sun

Research Collection School Of Computing and Information Systems

The emergence of Autonomous Vehicles (AVs) has spurred research into testing the resilience of their perception systems, i.e., ensuring that they are not susceptible to critical misjudgements. It is important that these systems are tested not only with respect to other vehicles on the road, but also with respect to objects placed on the roadside. Trash bins, billboards, and greenery are examples of such objects, typically positioned according to guidelines developed for the human visual system, which may not align perfectly with the needs of AVs. Existing tests, however, usually focus on adversarial objects with conspicuous shapes or patches, which …

Securing Cloud-Native Systems: From Vulnerability Analysis To External And Insider Threat Detection, Jiongchi Yu

Dissertations and Theses Collection (Open Access)

Cloud-native systems have become the backbone of modern software infrastructure. However, their dynamic resource orchestration and complex configurability introduce a large attack surface and intricate security challenges. Adversaries can externally exploit vulnerabilities in cloud components or perform insider movement within cloud environments to launch attacks. As these systems increasingly support critical services, security breaches can lead to severe operational and economic consequences.

Despite extensive efforts in vulnerability detection and attack monitoring, existing approaches struggle to remain effective in cloud-native environments characterized by rapid evolution and inherent heterogeneity. In particular, they exhibit three fundamental limitations: (1) Insufficient understanding of defect patterns …

Selective Concolic Testing, Guofeng Zhang, Zhenbang Chen, Ziqi Shuai, Jun Sun, Weijiang Hong, Yufeng Zhang, Ji Wang, Yang Liu

Research Collection School Of Computing and Information Systems

The principled combination of symbolic execution and random testing lacks a formal foundation, especially in deciding which inputs to symbolize. We propose selective concolic testing, a cost-aware framework that formulates this choice as an optimized policy problem of a MDP (Markov Decision Process). We model program exploration over a finite control-flow graph, where MDP states represent covered statements, actions partition path constraints into symbolic and random fragments, rewards reflect coverage gain, and costs account for SMT solving effort and sampling inefficiency. Our framework yields the first formal characterization of selective symbolization as policy synthesis in a probabilistic system. We prove …

Gencode: A Generic Data Augmentation Framework For Boosting Deep Learning-Based Code Understanding, Zeming Dong, Qiang Hu, Xiaofei Xie, Maxime Cordy, Mike Papadakis, Yves Le Traon, Jianjun Zhao

Research Collection School Of Computing and Information Systems

Pre-trained code models lead the era of code intelligence, with multiple models designed with impressive performance. However, one important problem, data augmentation for code data that automatically helps developers prepare training data lacks study in this field. In this paper, we introduce a generic data augmentation framework, GenCode, to enhance the training of code understanding models. Simply speaking, GenCode follows a generation-and-selection paradigm to prepare useful training code data. Specifically, it employs code augmentation techniques to generate new code candidates first and then identifies important ones as the training data by influence scores. To evaluate the effectiveness of GenCode, we …

Collaborative Practices And Tool Utilization In Software Development Projects: A Student Perspective, Yi Meng Lau, Muhammad Syahmi Bin Abbas, Lingxiao Jiang

Research Collection School Of Computing and Information Systems

Software development is a collaborative activity that depends on effective teamwork, shared understanding, and coordinated use of development practices and tools. While these aspects are well studied in professional environments, they are less frequently examined within software engineering education. This study investigates how students collaborate in group projects, focusing on collaborative practices, tool usage, and their perceptions of software quality. We conducted a quantitative post-project survey with 143 second-year undergraduate students enrolled in a software development course. The results show that students actively share information and often establish team norms to support coordination and collaboration. However, students face challenges in …

Causality-Driven Test Case Minimisation For Cyber-Physical Systems, Michael Foster, Christopher M. Poskitt, Nicholas R. Latimer, Neil Walkinshaw, Richard Somers, Robert M. Hierons

Research Collection School Of Computing and Information Systems

Cyber-physical systems allow digital control systems to interact with the physical world using sensors and actuators. They are increasingly being used to automate critical infrastructure, where software faults can have dire consequences. Due to the complex nature and unpredictability of these systems, their resilience is often tested using a technique called fuzzing, which generates quasi-random sequences of sensor and actuator manipulations with the goal of forcing a system into unsafe states. However, there is currently no way of determining which manipulations of a test case cause a failure without systematically removing each one and re-running the test, which can be …

Trace: Securing Smart Contract Repository Against Access Control Vulnerability, Chong Chen, Lingfeng Bao, David Lo, Yanlin Wang, Zhenyu Shan, Ting Chen, Guangqiang Yin, Jianxing Yu, Zibin Zheng, Jiachi Chen

Research Collection School Of Computing and Information Systems

Smart contract vulnerabilities have led to billions of dollars in economic losses. Among these, improper Access Control, which allows unauthorized users to execute restricted functions, is particularly prevalent and has caused significant financial damage. Smart contract repositories contain source code, documentation, configuration files, and other artifacts necessary for building and deploying smart contracts. GitHub hosts numerous open-source repositories of this kind, which serve as intermediate artifacts in development and require compilation and packaging to produce deployable contracts. Third-party developers often reference, reuse, or fork code from these repositories during custom development. However, if the referenced code contains vulnerabilities, it can …