Software Engineering Commons^™

Lessons From The Club Homeschool Capstone: Testing, Data Discipline, And The Computer Science Curriculum, Shane Brown

University Honors Theses

This thesis looks at the CLUB Homeschool Capstone project to argue that Portland State University's Computer Science curriculum should introduce testing and data quality discipline earlier and more intentionally than it does now. As team lead of a seven-person team, I coordinated sprint planning, communicated with the sponsor, and developed custom Discourse plugins that enhanced an existing forum platform instead of creating a separate application database, as requested by the sponsor. The project's requirements document called for a formal testing plan, but our team lacked the practical experience to implement one. This gap became evident through my internships as a …

Weavecc: Symbolically-Guided Joint Exploration Of Inputs And Schedules For Concurrency Bug Detection, William Philip Dinauer

Dartmouth College Master’s Theses

Concurrent programs introduce a class of bugs that depend jointly on both program inputs and thread schedules. Exposing these bugs requires simultaneously reasoning about which code paths are reachable and which thread interleavings are possible. At the same time, many existing tools handle the problem insufficiently. Race detectors observe only the interleavings that the OS happens to produce. Fuzzers explore inputs without controlling schedules. Tools that address both dimensions together exist, but are built on interpretation-based symbolic executors that incur considerable overhead.

We present WeaveCC, a practical concurrency testing tool for C/C++ programs that jointly explores inputs and thread schedules. …

A Comparative Evaluation Of Feedback Strategies For Enhancing Student Software Test Suite Writing Outcomes, Ashton Alonge

Master's Theses

Background and Context

Software testing is a fundamental component of computer science education, forming the basis for students’ ability to ensure program correctness and reliability. Despite its importance, many students struggle to design test cases that effectively expose faults and achieve meaningful test coverage. Traditional instructional approaches often emphasize code coverage metrics such as line or branch coverage, but these metrics may not adequately capture the quality of student tests. Mutation analysis, which measures how well tests detect small, artificial faults (mutants) introduced into the program, offers a potentially richer measure of test effectiveness. However, little is known about how …

The Use Of Call Graphs And Deep Learning To Improve Software Testing, Ziad A. Al-Sharif, Hemanth G. Chintala, Safwan Omari

Engineering, Computing and Mathematical Sciences Faculty Conferences

Software testing is a critical part of software development, it is essential for preventing failures and enhancing software quality attributes. However, the testing process can be costly and time-consuming, often involving a large number of test cases. Over time, the accumulation of redundant and overlapping test cases can complicate and lengthen the testing time. To address these challenges, this paper utilizes graph similarity and deep learning techniques to optimize test suites. It uses call graphs from test cases to identify redundant and similar test cases. A machine learning model is used to calculate and predict the similarity scores between these …

Bugs In Pods: Understanding Bugs In Container Runtime Systems, Jiongchi Yu, Xiaofei Xie, Ceng Zhang, Sen Chen

Research Collection School Of Computing and Information Systems

Container Runtime Systems (CRSs), which form the foundational infrastructure of container clouds, are critically important due to their impact on the quality of container cloud implementations. However, a comprehensive understanding of the quality issues present in CRS implementations remains lacking. To bridge this gap, we conduct the first comprehensive empirical study of CRS bugs. Specifically, we gather 429 bugs from 8,271 commits across dominant CRS projects, including runc, gvisor, containerd, and cri-o. Through manual analysis, we develop taxonomies of CRS bug symptoms and root causes, comprising 16 and 13 categories, respectively. Furthermore, we evaluate the capability of popular testing approaches, …

Empirical Exploration Of Software Testing, Samia Alblwi

Dissertations

Despite several advances in software engineering research and development, the quality of software products remains a considerable challenge. For all its theoretical limitations, software testing remains the main method used in practice to control, enhance, and certify software quality. This doctoral work comprises several empirical studies aimed at analyzing and assessing common software testing approaches, methods, and assumptions. In particular, the concept of mutant subsumption is generalized by taking into account the possibility for a base program and its mutants to diverge for some inputs, demonstrating the impact of this generalization on how subsumption is defined. The problem of mutant …

Beyond A Joke: Dead Code Elimination Can Delete Live Code, Haoxin Tu, Lingxiao Jiang, Debin Gao, He Jiang

Research Collection School Of Computing and Information Systems

Dead Code Elimination (DCE) is a fundamental compiler optimization technique that removes dead code (e.g., unreachable or reachable but whose results are unused) in the program to produce smaller or faster executables. However, since compiler optimizations are typically aggressively performed and there are complex relationships/interplay among a vast number of compiler optimizations (including DCE), it is not known whether DCE is indeed correctly performed and will only delete dead code in practice. In this study, we open a new research problem to investigate: can DCE happen to erroneously delete live code? To tackle this problem, we design a new approach …

Software Architecture In Practice: Challenges And Opportunities, Zhiyuan Wan, Yun Zhang, Xin Xia, Yi Jiang, David Lo

Research Collection School Of Computing and Information Systems

Software architecture has been an active research field for nearly four decades, in which previous studies make significant progress such as creating methods and techniques and building tools to support software architecture practice. Despite past efforts, we have little understanding of how practitioners perform software architecture related activities, and what challenges they face. Through interviews with 32 practitioners from 21 organizations across three continents, we identified challenges that practitioners face in software architecture practice during software development and maintenance. We reported on common software architecture activities at software requirements, design, construction and testing, and maintenance stages, as well as corresponding …

Kape: Knn-Based Performance Testing For Deep Code Search, Yuejun Guo, Qiang Hu, Xiaofei Xie, Cordy Maxime, Mike Papadakis, Yves Le Traon

Research Collection School Of Computing and Information Systems

Code search is a common yet important activity of software developers. An efficient code search model can largely facilitate the development process and improve the programming quality. Given the superb performance of learning the contextual representations, deep learning models, especially pre-trained language models, have been widely explored for the code search task. However, studies mainly focus on proposing new architectures for ever-better performance on designed test sets but ignore the performance on unseen test data where only natural language queries are available. The same problem in other domains, e.g., CV and NLP, is usually solved by test input selection that …

Software Testing And Code Refactoring: A Survey With Practitioners, Danilo Leandro Lima, Ronnie Souza Santos, Guilherme Pires Garcia, Sildemir S. Silva, Cesar Franca, Luiz Fernando Capretz

Electrical and Computer Engineering Publications

Nowadays, software testing professionals are commonly required to develop coding skills to work on test automation. One essential skill required from those who code is the ability to implement code refactoring, a valued quality aspect of software development; however, software developers usually encounter obstacles in successfully applying this practice. In this scenario, the present study aims to explore how software testing professionals (e.g., software testers, test engineers, test analysts, and software QAs) deal with code refactoring to understand the benefits and limitations of this practice in the context of software testing. We followed the guidelines to conduct surveys in software …

Myths And Facts About A Career In Software Testing: A Comparison Between Students’ Beliefs And Professionals’ Experience, Ronnie Souza Santos, Luiz Fernando Capretz, Cleyton Magalhaes, Rodrigo Souza

Electrical and Computer Engineering Publications

Testing is an indispensable part of software development. However, a career in software testing is reported to be unpopular among students in computer science and related areas. This can potentially create a shortage of testers in the software industry in the future. The question is, whether the perception that undergraduate students have about software testing is accurate and whether it differs from the experience reported by those who work in testing activities in the software development industry. This investigation demonstrates that a career in software testing is more exciting and rewarding, as reported by professionals working in the field, than …

Revisiting The Identification Of The Co-Evolution Of Production And Test Code, Weifeng Sun, Meng Yan, Zhongxin Liu, Xin Xia, Yan Lei, David Lo

Research Collection School Of Computing and Information Systems

Many software processes advocate that the test code should co-evolve with the production code. Prior work usually studies such co-evolution based on production-test co-evolution samples mined from software repositories. A production-test co-evolution sample refers to a pair of a test code change and a production code change where the test code change triggers or is triggered by the production code change. The quality of the mined samples is critical to the reliability of research conclusions. Existing studies mined production-test co-evolution samples based on the following assumption: if a test class and its associated production class change together in one commit, …

Detecting C++ Compiler Front-End Bugs Via Grammar Mutation And Differential Testing, Haoxin Tu, He Jiang, Zhide Zhou, Yixuan Tang, Zhilei Ren, Lei Qiao, Lingxiao Jiang

Research Collection School Of Computing and Information Systems

C++ is a widely used programming language and the C++ front-end is a critical part of a C++ compiler. Although many techniques have been proposed to test compilers, few studies are devoted to detecting bugs in C++ compiler. In this study, we take the first step to detect bugs in C++ compiler front-ends. To do so, two main challenges need to be addressed, namely, the acquisition of test programs that are more likely to trigger bugs in compiler front-ends and the bug identification from complicated compiler outputs. In this article, we propose a novel framework named Ccoft to detect bugs …

What Pakistani Computer Science And Software Engineering Students Think About Software Testing?, Luiz Fernando Capretz, Abdul Rehman Gilal

Electrical and Computer Engineering Publications

Software testing is one of the crucial supporting processes of the software life cycle. Unfortunately for the software industry, the role is stigmatized, partly due to misperception and partly due to treatment of the role. The present study aims to analyze the situation to explore what restricts computer science and software engineering students from taking up a testing career in the software industry. To conduct this study, we surveyed 88 Pakistani students taking computer science or software engineering degrees. The results showed that the present study supports previous work into the unpopularity of testing compared to other software life cycle …

Comparing The Popularity Of Testing Careers Among Canadian, Indian, Chinese, And Malaysian Students, Luiz Fernando Capretz, Pradeep Waychal, Jingdong Jia, Shuib Basri

Electrical and Computer Engineering Publications

This study attempts to understand motivators and de-motivators that influence the decisions of software students to take up and sustain software testing careers across four different countries, Canada, India, China, and Malaysia. Towards that end, we have developed a cross-sectional, but simple, survey-based instrument. In this study we investigated how software engineering and computer science students perceive and value what they do and their environmental settings. This study found that very few students are keen to take up software testing careers - why is this happening with such an important task in the software life cycle? The common advantages of …

International Comparative Studies On The Software Testing Profession, Luiz Fernando Capretz, Pradeep Waychal, Jingdong Jia, Daniel Varona, Yadira Lizama

Electrical and Computer Engineering Publications

This work attempts to fill a gap by exploring the human dimension in particular, by trying to understand the motivation of software professionals for taking up and sustaining their careers as software testers. Towards that goal, four surveys were conducted in four countries—India, Canada, Cuba, and China—to try to understand how professional software engineers perceive and value work-related factors that could influence their motivation to start or move into software testing careers. From our sample of 220 software professionals, we observed that very few were keen to take up testing careers. Some aspects of software testing, such as the potential …

Graph-Based Seed Object Synthesis For Search-Based Unit Testing, Yun Lin, You Seng Ong, Jun Sun, Gordon Fraser, Jin Song Dong

Research Collection School Of Computing and Information Systems

Search-based software testing (SBST) generates tests using search algorithms guided by measurements gauging how far a test case is away from exercising a coverage goal. The effectiveness of SBST largely depends on the continuity and monotonicity of the fitness landscape decided by these measurements and the search operators. Unfortunately, the fitness landscape is challenging when the function under test takes object inputs, as classical measurements hardly provide guidance for constructing legitimate object inputs. To overcome this problem, we propose test seeds, i.e., test code skeletons of legitimate objects which enable the use of classical measurements. Given a target branch in …

What Malaysian Software Students Think About Testing?, Luiz Fernando Capretz, Shuib Basri, Maythem Adili, Aamir Amin

Electrical and Computer Engineering Publications

Software testing is one of the crucial supporting processes of software life cycle. Unfortunately for the software industry, the role is stigmatized, partly due to misperception and partly due to treatment of the role in the software industry. The present study aims to analyse this situation to explore what inhibit an individual from taking up a software testing career. In order to investigate this issue, we surveyed 82 senior students taking a degree in information technology, information and communication technology, and computer science at two Malaysian universities. The subjects were asked the PROs and CONs of taking up a career …

Towards Characterizing Adversarial Defects Of Deep Learning Software From The Lens Of Uncertainty, Xiyue Zhang, Xiaofei Xie, Lei Ma, Xiaoning Du, Qiang Hu, Yang Liu, Jianjun Zhao, Meng Sun

Research Collection School Of Computing and Information Systems

Over the past decade, deep learning (DL) has been successfully applied to many industrial domain-specific tasks. However, the current state-of-the-art DL software still suffers from quality issues, which raises great concern especially in the context of safety- and security-critical scenarios. Adversarial examples (AEs) represent a typical and important type of defects needed to be urgently addressed, on which a DL software makes incorrect decisions. Such defects occur through either intentional attack or physical-world noise perceived by input sensors, potentially hindering further industry deployment. The intrinsic uncertainty nature of deep learning decisions can be a fundamental reason for its incorrect behavior. …

Statistical Log Differencing, Lingfeng Bao, Nimrod Busany, David Lo, Shahar Maoz

Research Collection School Of Computing and Information Systems

Recent works have considered the problem of log differencing: given two or more system’s execution logs, output a model of their differences. Log differencing has potential applications in software evolution, testing, and security. In this paper we present statistical log differencing, which accounts for frequencies of behaviors found in the logs. We present two algorithms, s2KDiff for differencing two logs, and snKDiff, for differencing of many logs at once, both presenting their results over a single inferred model. A unique aspect of our algorithms is their use of statistical hypothesis testing: we let the engineer control the sensitivity of the …

Deviant: A Mutation Testing Tool For Solidity Smart Contracts, Patrick Chapman

Boise State University Theses and Dissertations

Blockchain in recent years has exploded in popularity with Ethereum being one of the leading blockchain platforms. Solidity is a widely used scripting language for creating smart contracts in Ethereum applications. Quality assurance in Solidity contracts is of critical importance because bugs or vulnerabilities can lead to a considerable loss of financial assets. However, it is unclear what level of quality assurance is provided in many of these applications.

Mutation testing is the process of intentionally injecting faults into a target program and then running the provided test suite against the various injected faults. Mutation testing is used to evaluate …

To The Attention Of Mobile Software Developers: Guess What, Test Your App!, Luis C. Cruz, Rui Abreu, David Lo

Research Collection School Of Computing and Information Systems

Software testing is an important phase in the software development lifecycle because it helps in identifying bugs in a software system before it is shipped into the hand of its end users. There are numerous studies on how developers test general-purpose software applications. The idiosyncrasies of mobile software applications, however, set mobile apps apart from general-purpose systems (e.g., desktop, stand-alone applications, web services). This paper investigates working habits and challenges of mobile software developers with respect to testing. A key finding of our exhaustive study, using 1000 Android apps, demonstrates that mobile apps are still tested in a very ad …

Evaluating Software Testing Techniques: A Systematic Mapping Study, Mitchell Mayeda

Electronic Theses and Dissertations

Software testing techniques are crucial for detecting faults in software and reducing the risk of using it. As such, it is important that we have a good understanding of how to evaluate these techniques for their efficiency, scalability, applicability, and effectiveness at finding faults. This thesis enhances our understanding of testing technique evaluations by providing an overview of the state of the art in research. To accomplish this we utilize a systematic mapping study; structuring the field and identifying research gaps and publication trends. We then present a small case study demonstrating how our mapping study can be used to …

Perflearner: Learning From Bug Reports To Understand And Generate Performance Test Frames, Xue Han, Tingting Yu, David Lo

Research Collection School Of Computing and Information Systems

Software performance is important for ensuring the quality of software products. Performance bugs, defined as programming errors that cause significant performance degradation, can lead to slow systems and poor user experience. While there has been some research on automated performance testing such as test case generation, the main idea is to select workload values to increase the program execution times. These techniques often assume the initial test cases have the right combination of input parameters and focus on evolving values of certain input parameters. However, such an assumption may not hold for highly configurable real-word applications, in which the combinations …

Code Coverage And Postrelease Defects: A Large-Scale Study On Open Source Projects, Pavneet Singh Kochhar, David Lo, Julia Lawall, Nachiappan Nagappan

Research Collection School Of Computing and Information Systems

Testing is a pivotal activity in ensuring the quality of software. Code coverage is a common metric used as a yardstick to measure the efficacy and adequacy of testing. However, does higher coverage actually lead to a decline in postrelease bugs? Do files that have higher test coverage actually have fewer bug reports? The direct relationship between code coverage and actual bug reports has not yet been analyzed via a comprehensive empirical study on real bugs. Past studies only involve a few software systems or artificially injected bugs (mutants). In this empirical study, we examine these questions in the context …

Security Testing With Misuse Case Modeling, Samer Yousef Khamaiseh

Boise State University Theses and Dissertations

Having a comprehensive model of security requirements is a crucial step towards developing a reliable software system. An effective model of security requirements which describes the possible scenarios that may affect the security aspects of the system under development can be an effective approach for subsequent use in generating security test cases.

Misuse case was first proposed by Sinder and Opdahl as an approach to extract the security requirements of the system under development [1]. A misuse case is a use case representing scenarios that might be followed by a system adversary in order to compromise the system; that is …

Why A Testing Career Is Not The First Choice Of Engineers, Pradeep Kashinath Waychal, Luiz Fernando Capretz

Electrical and Computer Engineering Publications

No abstract provided.

An Empirical Study On The Adequacy Of Testing In Open Source Projects, Pavneet Singh Kochhar, Ferdian Thung, David Lo, Julia Lawall

Research Collection School Of Computing and Information Systems

During software maintenance, testing is crucial to ensure the quality of code as it evolves. With the increasing size and complexity of software, adequate software testing has become increasingly important. Code coverage is an important metric to gauge the effectiveness of test cases and the adequacy of testing. However, what is the coverage level exhibited by large-scale open-source projects? What is the correlation between software metrics and the code coverage of the software?In this study, we investigate the state-of-the-practice of testing by measuring code coverage in open-source software projects. We examine over300 large open-source projects written in Java, coming from …

A Fault-Based Model Of Fault Localization Techniques, Mark A. Hays

Theses and Dissertations--Computer Science

Every day, ordinary people depend on software working properly. We take it for granted; from banking software, to railroad switching software, to flight control software, to software that controls medical devices such as pacemakers or even gas pumps, our lives are touched by software that we expect to work. It is well known that the main technique/activity used to ensure the quality of software is testing. Often it is the only quality assurance activity undertaken, making it that much more important.

In a typical experiment studying these techniques, a researcher will intentionally seed a fault (intentionally breaking the functionality of …

Automated Test Case Generation To Validate Non-Functional Software Requirements, Pingyu Zhang

School of Computing: Dissertations, Theses, and Student Research

A software system is bounded by a set of requirements. Functional requirements describe what the system must do, in terms of inputs, behavior, and outputs. We define non-functional requirements to be how well these functional requirements are satisfied, in terms of qualities or constraints on the design or on the implementation of a system. In practice, the validation of these kinds of requirements, does not receive equal emphasis. Techniques for validating functional requirements target all levels of software testing phases, and explore both black-box and white-box approaches. Techniques for validating non-functional requirements, on the other hand, largely operate in a …