Computer Sciences Commons^™

Hierarchical Damage Correlations For Old Photo Restoration, Weiwei Cai, Xuemiao Xu, Jiajia Xu, Huaidong Zhang, Haoxin Yang, Kun Zhang, Shengfeng He

Research Collection School Of Computing and Information Systems

Restoring old photographs can preserve cherished memories. Previous methods handled diverse damages within the same network structure, which proved impractical. In addition, these methods cannot exploit correlations among artifacts, especially in scratches versus patch-misses issues. Hence, a tailored network is particularly crucial. In light of this, we propose a unified framework consisting of two key components: ScratchNet and PatchNet. In detail, ScratchNet employs the parallel Multi-scale Partial Convolution Module to effectively repair scratches, learning from multi-scale local receptive fields. In contrast, the patch-misses necessitate the network to emphasize global information. To this end, we incorporate a transformer-based encoder and decoder …

Ethical Considerations Toward Protestware, Marc Cheong, Raula Kula, Christoph Treude

Research Collection School Of Computing and Information Systems

This article looks into possible scenarios where developers might consider turning their free and open source software into protestware. Using different frameworks commonly used in artificial intelligence (AI) ethics, we extend the applications of AI ethics to the study of protestware.

Large Language Models For Qualitative Research In Software Engineering: Exploring Opportunities And Challenges, Muneera Bano, Rashina Hoda, Didar Zowghi, Christoph Treude

Research Collection School Of Computing and Information Systems

The recent surge in the integration of Large Language Models (LLMs) like ChatGPT into qualitative research in software engineering, much like in other professional domains, demands a closer inspection. This vision paper seeks to explore the opportunities of using LLMs in qualitative research to address many of its legacy challenges as well as potential new concerns and pitfalls arising from the use of LLMs. We share our vision for the evolving role of the qualitative researcher in the age of LLMs and contemplate how they may utilize LLMs at various stages of their research experience.

An Evaluation Of Heart Rate Monitoring With In-Ear Microphones Under Motion, Kayla-Jade Butkow, Ting Dang, Andrea Ferlini, Dong Ma, Yang Liu, Cecilia Mascolo

Research Collection School Of Computing and Information Systems

With the soaring adoption of in-ear wearables, the research community has started investigating suitable in-ear heart rate detection systems. Heart rate is a key physiological marker of cardiovascular health and physical fitness. Continuous and reliable heart rate monitoring with wearable devices has therefore gained increasing attention in recent years. Existing heart rate detection systems in wearables mainly rely on photoplethysmography (PPG) sensors, however, these are notorious for poor performance in the presence of human motion. In this work, leveraging the occlusion effect that enhances low-frequency bone-conducted sounds in the ear canal, we investigate for the first time in-ear audio-based motion-resilient …

Data Engineering: Building Software Efficiency In Medium To Large Organizations, Alessandro De La Torre

Whittier Scholars Program

The introduction of PoetHQ, a mobile application, offers an economical strategy for colleges, potentially ushering in significant cost savings. These savings could be redirected towards enhancing academic programs and services, enriching the educational landscape for students. PoetHQ aims to democratize access to crucial software, effectively removing financial barriers and facilitating a richer educational experience. By providing an efficient software solution that reduces organizational overhead while maximizing accessibility for students, the project highlights the essential role of equitable education and resource optimization within academic institutions.

Rescape: Transforming Coral-Reefscape Images For Quantitative Analysis, Zachary Ferris, Eraldo Ribeiro, Tomofumi Nagata, Robert Van Woesik

Ocean Engineering and Marine Sciences Faculty Publications

Ever since the first image of a coral reef was captured in 1885, people worldwide have been accumulating images of coral reefscapes that document the historic conditions of reefs. However, these innumerable reefscape images suffer from perspective distortion, which reduces the apparent size of distant taxa, rendering the images unusable for quantitative analysis of reef conditions. Here we solve this century-long distortion problem by developing a novel computer-vision algorithm, ReScape, which removes the perspective distortion from reefscape images by transforming them into top-down views, making them usable for quantitative analysis of reef conditions. In doing so, we demonstrate the …

Redriver: Runtime Enforcement For Autonomous Vehicles, Yang Sun, Christopher M. Poskitt, Xiaodong Zhang, Jun Sun

Research Collection School Of Computing and Information Systems

Autonomous driving systems (ADSs) integrate sensing, perception, drive control, and several other critical tasks in autonomous vehicles, motivating research into techniques for assessing their safety. While there are several approaches for testing and analysing them in high-fidelity simulators, ADSs may still encounter additional critical scenarios beyond those covered once they are deployed on real roads. An additional level of confidence can be established by monitoring and enforcing critical properties when the ADS is running. Existing work, however, is only able to monitor simple safety properties (e.g., avoidance of collisions) and is limited to blunt enforcement mechanisms such as hitting the …

Exploring The Potential Of Chatgpt In Automated Code Refinement: An Empirical Study, Qi Guo, Shangqing Liu, Junming Cao, Xiaohong Li, Xin Peng, Xiaofei Xie, Bihuan Chen

Research Collection School Of Computing and Information Systems

Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive performance in various natural language processing tasks, suggesting its potential to automate code review processes. However, it is still unclear how well ChatGPT performs in code review tasks. To fill this gap, in this paper, we conduct the first empirical study to understand the capabilities of ChatGPT in code review tasks, specifically focusing on automated code refinement based on given …

Teaching Software Development For Real-World Problems Using A Microservice-Based Collaborative Problem-Solving Approach, Yi Meng Lau, Christian Michael Koh, Lingxiao Jiang

Research Collection School Of Computing and Information Systems

Experienced and skillful software developers are needed in organizations to develop software products effective for their business with shortened time-to-market. Such developers will not only need to code but also be able to work in teams and collaboratively solve real-world problems that organizations arefacing. It is challenging for educators to nurture students to become such developers with strong technical, social, and cognitive skills. Towards addressing the challenge, this study presents a Collaborative Software Development Project Framework for a course that focuses on learning microservices architectures anddeveloping a software application for a real-world business. Students get to work in teams to …

The Social Pot: A Social Media Application, Reid Long

Honors Projects

The Social Pot is a web application that allows a user to post to Instagram and X simultaneously from one place. The user creates a Social Pot Account and from there can set their Instagram username and password within the home page. Once the user attempts to post, it will redirect them to login to X which once successful will make the tweet. Used the API 'instagram-private-api'. User needed to give access to my X Project which in turn gave an Auth token (via X redirect URL). The auth token was then sent to my endpoint in order to get …

Marco: A Stochastic Asynchronous Concolic Explorer, Jie Hu, Yue Duan, Heng Yin

Research Collection School Of Computing and Information Systems

Concolic execution is a powerful program analysis technique for code path exploration. Despite recent advances that greatly improved the efficiency of concolic execution engines, path constraint solving remains a major bottleneck of concolic testing. An intelligent scheduler for inputs/branches becomes even more crucial. Our studies show that the previously under-studied branch-flipping policy adopted by state-of-the-art concolic execution engines has several limitations. We propose to assess each branch by its potential for new code coverage from a global view, concerning the path divergence probability at each branch. To validate this idea, we implemented a prototype Marco and evaluated it against the …

Acav: A Framework For Automatic Causality Analysis In Autonomous Vehicle Accident Recordings, Huijia Sun, Christopher M. Poskitt, Yang Sun, Jun Sun, Yuqi Chen

Research Collection School Of Computing and Information Systems

The rapid progress of autonomous vehicles (AVs) has brought the prospect of a driverless future closer than ever. Recent fatalities, however, have emphasized the importance of safety validation through large-scale testing. Multiple approaches achieve this fully automatically using high-fidelity simulators, i.e., by generating diverse driving scenarios and evaluating autonomous driving systems (ADSs) against different test oracles. While effective at finding violations, these approaches do not identify the decisions and actions that caused them -- information that is critical for improving the safety of ADSs. To address this challenge, we propose ACAV, an automated framework designed to conduct causality analysis for …

The Social Pot: A Social Media Application, Reid Long

ASPIRE 2024

The Social Pot is a web application that allows a user to post to Instagram and X simultaneously from one place. The user creates a Social Pot Account and from there can set their Instagram username and password within the home page. Once the user attempts to post, it will redirect them to login to X which once successful will make the tweet. Used the API 'instagram-private-api'. User needed to give access to my X Project which in turn gave an Auth token (via X redirect URL). The auth token was then sent to my endpoint in order to get …

Terry Riley's "In C" For Mobile Ensemble, David B. Wetzel, Griffin Moe, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

This workshop presents a mobile-friendly Web Audio application for a “technology ensemble play-along” of Terry Riley’s 1964 composition In C. Attendees will join in a reading of In C using available web-enabled devices as musical instruments. We hope to demonstrate an accessible music-technology experience that relies on face-to-face interaction within a shared space. In this all-electronic implementation, no special musical or technical expertise is required.

Accepted for presentation and publication at WAC 2024.

Ur2m: Uncertainty And Resource-Aware Event Detection On Microcontrollers, Hong Jia, Young D. Kwon, Dong Ma, Nhat Pham, Lorena Qendro, Tam Vu, Cecilia Mascolo

Research Collection School Of Computing and Information Systems

Traditional machine learning techniques are prone to generating inaccurate predictions when confronted with shifts in the distribution of data between the training and testing phases. This vulnerability can lead to severe consequences, especially in applications such as mobile healthcare. Uncertainty estimation has the potential to mitigate this issue by assessing the reliability of a model's output. However, existing uncertainty estimation techniques often require substantial computational resources and memory, making them impractical for implementation on microcontrollers (MCUs). This limitation hinders the feasibility of many important on-device wearable event detection (WED) applications, such as heart attack detection. In this paper, we present …

Fixing Your Own Smells: Adding A Mistake-Based Familiarization Step When Teaching Code Refactoring, Ivan Wei Han Tan, Christopher M. Poskitt

Research Collection School Of Computing and Information Systems

Programming problems can be solved in a multitude of functionally correct ways, but the quality of these solutions (e.g. readability, maintainability) can vary immensely. When code quality is poor, symptoms emerge in the form of 'code smells', which are specific negative characteristics (e.g. duplicate code) that can be resolved by applying refactoring patterns. Many undergraduate computing curricula train students on this software engineering practice, often doing so via exercises on unfamiliar instructor-provided code. Our observation, however, is that this makes it harder for novices to internalise refactoring as part of their own development practices. In this paper, we propose a …

Ditmos: Delving Into Diverse Tiny-Model Selection On Microcontrollers, Xiao Ma, Shengfeng He, Hezhe Qiao, Dong Ma

Research Collection School Of Computing and Information Systems

Enabling efficient and accurate deep neural network (DNN) inference on microcontrollers is non-trivial due to the constrained on-chip resources. Current methodologies primarily focus on compressing larger models yet at the expense of model accuracy. In this paper, we rethink the problem from the inverse perspective by constructing small/weak models directly and improving their accuracy. Thus, we introduce DiTMoS, a novel DNN training and inference framework with a selectorclassifiers architecture, where the selector routes each input sample to the appropriate classifier for classification. DiTMoS is grounded on a key insight: a composition of weak models can exhibit high diversity and the …

Delving Into Multimodal Prompting For Fine-Grained Visual Classification, Xin Jiang, Hao Tang, Junyao Gao, Xiaoyu Du, Shengfeng He, Zechao Li

Research Collection School Of Computing and Information Systems

Fine-grained visual classification (FGVC) involves categorizing fine subdivisions within a broader category, which poses challenges due to subtle inter-class discrepancies and large intra-class variations. However, prevailing approaches primarily focus on uni-modal visual concepts. Recent advancements in pre-trained vision-language models have demonstrated remarkable performance in various high-level vision tasks, yet the applicability of such models to FGVC tasks remains uncertain. In this paper, we aim to fully exploit the capabilities of cross-modal description to tackle FGVC tasks and propose a novel multimodal prompting solution, denoted as MP-FGVC, based on the contrastive language-image pertaining (CLIP) model. Our MP-FGVC comprises a multimodal prompts …

Detecting Outdated Code Element References In Software Repository Documentation, Wen Siang Tan, Markus Wagner, Christoph Treude

Research Collection School Of Computing and Information Systems

Outdated documentation is a pervasive problem in software development, preventing effective use of software, and misleading users and developers alike. We posit that one possible reason why documentation becomes out of sync so easily is that developers are unaware of when their source code modifications render the documentation obsolete. Ensuring that the documentation is always in sync with the source code takes considerable effort, especially for large codebases. To address this situation, we propose an approach that can automatically detect code element references that survive in the documentation after all source code instances have been deleted. In this work, we …

Vibmilk: Non-Intrusive Milk Spoilage Detection Via Smartphone Vibration, Yuezhong Wu, Wei Song, Yanxiang Wang, Dong Ma, Weitao Xu, Mahbub Hassan, Wen Hu

Research Collection School Of Computing and Information Systems

Quantifying the chemical process of milk spoilage is challenging due to the need for bulky, expensive equipment that is not user-friendly for milk producers or customers. This lack of a convenient and accurate milk spoilage detection system can cause two significant issues. First, people who consume spoiled milk may experience serious health problems. Secondly, milk manufacturers typically provide a “best before” date to indicate freshness, but this date only shows the highest quality of the milk, not the last day it can be safely consumed, leading to significant milk waste. A practical and efficient solution to this problem is proposed …

Piecing Together Performance: Collaborative, Participatory Research-Through-Design For Better Diversity In Games, Daniel L. Gardner, Louanne Boyd, Reginald T. Gardner

Engineering Faculty Articles and Research

Digital games are a multi-billion-dollar industry whose production and consumption extend globally. Representation in games is an increasingly important topic. As those who create and consume the medium grow ever more diverse, it is essential that player or user-experience research, usability, and any consideration of how people interface with their technology is exercised through inclusive and intersectional lenses. Previous research has identified how character configuration interfaces preface white-male defaults [39, 40, 67]. This study relies on 1-on-1 play-interviews where diverse participants attempt to create “themselves” in a series of games and on group design activities to explore how participants may …

Remote Multi-Person Heart Rate Monitoring With Smart Speakers: Overcoming Separation Constraint, Ngoc Doan Thu Tran, Dong Ma, Rajesh Krishna Balan

Research Collection School Of Computing and Information Systems

Heart rate is a key vital sign that can be used to understand an individual’s health condition. Recently, remote sensing techniques, especially acoustic-based sensing, have received increasing attention for their ability to non-invasively detect heart rate via commercial mobile devices such as smartphones and smart speakers. However, due to signal interference, existing methods have primarily focused on monitoring a single user and required a large separation between them when monitoring multiple people. These limitations hinder many common use cases such as couples sharing the same bed or two or more people located in close proximity. In this paper, we present …

Provably Secure Decisions Based On Potentially Malicious Information, Dongxia Wang, Tim Muller, Jun Sun

Research Collection School Of Computing and Information Systems

There are various security-critical decisions routinely made, on the basis of information provided by peers: routing messages, user reports, sensor data, navigational information, blockchain updates, etc. Jury theorems were proposed in sociology to make decisions based on information from peers, which assume peers may be mistaken with some probability. We focus on attackers in a system, which manifest as peers that strategically report fake information to manipulate decision making. We define the property of robustness: a lower bound probability of deciding correctly, regardless of what information attackers provide. When peers are independently selected, we propose an optimal, robust decision mechanism …

Conceptthread: Visualizing Threaded Concepts In Mooc Videos, Zhiguang Zhou, Li Ye, Lihong Cai, Lei Wang, Yigang Wang, Yongheng Wang, Wei Chen, Yong Wang

Research Collection School Of Computing and Information Systems

Massive Open Online Courses (MOOCs) platforms are becoming increasingly popular in recent years. Online learners need to watch the whole course video on MOOC platforms to learn the underlying new knowledge, which is often tedious and time-consuming due to the lack of a quick overview of the covered knowledge and their structures. In this paper, we propose ConceptThread , a visual analytics approach to effectively show the concepts and the relations among them to facilitate effective online learning. Specifically, given that the majority of MOOC videos contain slides, we first leverage video processing and speech analysis techniques, including shot recognition, …

Big Code Search: A Bibliography, Kisub Kim, Sankalp Ghatpande, Dongsun Kim, Xin Zhou, Kui Liu, Tegawende F. Bissyande, Jacques Klein, Traon Yves Le

Research Collection School Of Computing and Information Systems

Code search is an essential task in software development. Developers often search the internet and other code databases for necessary source code snippets to ease the development efforts. Code search techniques also help learn programming as novice programmers or students can quickly retrieve (hopefully good) examples already used in actual software projects. Given the recurrence of the code search activity in software development, there is an increasing interest in the research community. To improve the code search experience, the research community suggests many code search tools and techniques. These tools and techniques leverage several different ideas and claim a better …

Clearspeech: Improving Voice Quality Of Earbuds Using Both In-Ear And Out-Ear Microphones, Dong Ma, Ting Dang, Ming Ding, Rajesh Krishna Balan

Research Collection School Of Computing and Information Systems

Wireless earbuds have been gaining increasing popularity and using them to make phone calls or issue voice commands requires the earbud microphones to pick up human speech. When the speaker is in a noisy environment, speech quality degrades significantly and requires speech enhancement (SE). In this paper, we present ClearSpeech, a novel deep-learningbased SE system designed for wireless earbuds. Specifically, by jointly using the earbud’s in-ear and out-ear microphones, we devised a suite of techniques to effectively fuse the two signals and enhance the magnitude and phase of the speech spectrogram. We built an earbud prototype to evaluate ClearSpeech under …

Conversational Localization: Indoor Human Localization Through Intelligent Conversation, Sheshadri Smitha, Kotaro Hara

Research Collection School Of Computing and Information Systems

We propose a novel sensorless approach to indoor localization by leveraging natural language conversations with users, which we call conversational localization. To show the feasibility of conversational localization, we develop a proof-of-concept system that guides users to describe their surroundings in a chat and estimates their position based on the information they provide. We devised a modular architecture for our system with four modules. First, we construct an entity database with available image-based floor maps. Second, we enable the dynamic identification and scoring of information provided by users through our utterance processing module. Then, we implement a conversational agent that …

Stealthy Backdoor Attack For Code Models, Zhou Yang, Bowen Xu, Jie M. Zhang, Hong Jin Kang, Jieke Shi, Junda He, David Lo

Research Collection School Of Computing and Information Systems

Code models, such as CodeBERT and CodeT5, offer general-purpose representations of code and play a vital role in supporting downstream automated software engineering tasks. Most recently, code models were revealed to be vulnerable to backdoor attacks. A code model that is backdoor-attacked can behave normally on clean examples but will produce pre-defined malicious outputs on examples injected with that activate the backdoors. Existing backdoor attacks on code models use unstealthy and easy-to-detect triggers. This paper aims to investigate the vulnerability of code models with backdoor attacks. To this end, we propose A (dversarial eature as daptive Back). A achieves stealthiness …

Active Code Learning: Benchmarking Sample-Efficient Training Of Code Models, Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Lei Ma, Mike Papadakis, Yves Le Traon

Research Collection School Of Computing and Information Systems

The costly human effort required to prepare the training data of machine learning (ML) models hinders their practical development and usage in software engineering (ML4Code), especially for those with limited budgets. Therefore, efficiently training models of code with less human effort has become an emergent problem. Active learning is such a technique to address this issue that allows developers to train a model with reduced data while producing models with desired performance, which has been well studied in computer vision and natural language processing domains. Unfortunately, there is no such work that explores the effectiveness of active learning for code …

Learning An Interpretable Stylized Subspace For 3d-Aware Animatable Artforms, Chenxi Zheng, Bangzhen Liu, Xuemiao Xu, Huaidong Zhang, Shengfeng He

Research Collection School Of Computing and Information Systems

Throughout history, static paintings have captivated viewers within display frames, yet the possibility of making these masterpieces vividly interactive remains intriguing. This research paper introduces 3DArtmator, a novel approach that aims to represent artforms in a highly interpretable stylized space, enabling 3D-aware animatable reconstruction and editing. Our rationale is to transfer the interpretability and 3D controllability of the latent space in a 3D-aware GAN to a stylized sub-space of a customized GAN, revitalizing the original artforms. To this end, the proposed two-stage optimization framework of 3DArtmator begins with discovering an anchor in the original latent space that accurately mimics the …