Software Engineering Commons^™

Hotlangbench, A Tiny Benchmark Suite For Higher-Order Statically Typed Languages, Konstantin Laufer

Computer Science: Faculty Publications and Other Works

This work in progress aims to compare various HOT (higher-order and statically typed, a term coined by Phil Wadler) through reproducible course-grained, wall-time benchmarks. Our overall goals include simplicity, agility, and reproducibility.

There is currently only one benchmark, but it brings out substantial performance differences among the various languages and platforms. It uses function composition and other higher-order constructs to build a pipeline of transformations, along with a brute-force iteration that is computationally expensive for input files specifying large ranges as function domains. We currently include versions in Modern C++, C#, Go, Haskell, Kotlin, Modern (stream-based) Java (24), OCaml, Scala …

Improving The Reproducibility Of Deep Learning Software: An Initial Investigation Through A Case Study Analysis, Nikita Ravi, Abhinav Goel, James C. Davis, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

The field of deep learning has witnessed significant breakthroughs, spanning various applications, and fundamentally transforming current software capabilities. However, alongside these advancements, there have been increasing concerns about reproducing the results of these deep learning methods. This is significant because reproducibility is the foundation of reliability and validity in software development, particularly in the rapidly evolving domain of deep learning. The difficulty of reproducibility may arise due to several reasons, including having differences from the original execution environment, incompatible software libraries, proprietary data and source code, lack of transparency, and the stochastic nature in some software. A study conducted by …

Tla+ For All: Model Checking In A Python Notebook, Konstantin Laufer, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

TLA+ is widely recognized for its effectiveness in specifying and verifying concurrent and distributed systems. However, for educators and practitioners, barriers to adoption include installation complexity and tooling setup. In the proposed presentation, we demonstrate a lightweight, easily shareable, and fully reproducible approach to running TLA+ in a Python notebook hosted on Google Colab without requiring new tools or custom Jupyter kernel development. By creating an environment where users can experiment with TLA+ models instantly, we lower these barriers and demonstrate the suitability for education and outreach.

Can Large-Language Models Help Us Better Understand And Teach The Development Of Energy-Efficient Software?, Ryan Hasler, Konstantin Laufer, George K. Thiruvathukal, Huiyun Peng, Kyle Robinson, Kirsten Davis, Yung-Hisang Lu, James C. Davis

Computer Science: Faculty Publications and Other Works

Computing systems are consuming an increasing and unsustainable fraction of society's energy footprint, notably in data centers. Meanwhile, energy-efficient software engineering techniques are often absent from undergraduate curricula. We propose to develop a learning module for energy-efficient software, suitable for incorporation into an undergraduate software engineering class. There is one major problem with such an endeavor: undergraduate curricula have limited space for mastering energy-related systems programming aspects. To address this problem, we propose to leverage the domain expertise afforded by large language models (LLMs). In our preliminary studies, we observe that LLMs can generate energy-efficient variations of basic linear algebra …

What Do We Know About Hugging Face? A Systematic Literature Review And Quantitative Validation Of Qualitative Claims, Jason Jones, Wenxin Jiang, Nicholas Synovic, George K. Thiruvathukal, James C. Davis

Computer Science: Faculty Publications and Other Works

Background: Collaborative Software Package Registries (SPRs) are an integral part of the software supply chain. Much engineering work synthesizes SPR package into applications. Prior research has examined SPRs for traditional software, such as NPM (JavaScript) and PyPI (Python). Pre-Trained Model (PTM) Registries are an emerging class of SPR of increasing importance, because they support the deep learning supply chain.
Aims: Recent empirical research has examined PTM registries in ways such as vulnerabilities, reuse processes, and evolution. However, no existing research synthesizes them to provide a systematic understanding of the current knowledge. Some of the existing research includes qualitative …

Wip: An Engaging Undergraduate Intro To Model Checking In Software Engineering Using Tla+, Konstantin Laufer, Gunda Mertin, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

Background: In this Innovative Practice Work in Progress, we present our initial efforts to integrate formal methods, with a focus on model-checking specifications written in Temporal Logic of Actions (TLA⁺), into computer science education, targeting undergraduate juniors/seniors and graduate students. Many safety-critical systems and services crucially depend on correct and reliable behavior. Formal methods can play a key role in ensuring correct and safe system behavior, yet remain underutilized in educational and industry contexts.

Aims: We aim to (1) qualitatively assess the state of formal methods in computer science programs, (2) construct level-appropriate examples that could be included …

Interoperability In Deep Learning: A User Survey And Failure Analysis Of Onnx Model Converters, Purvish Jajal, Wenxin Jiang, Arav Tewari, Erik Kocinare, Joseph Woo, Anusha Sarraf, Yung-Hsiang Lu, George Thiruvathukal, James C. Davis

Computer Science: Faculty Publications and Other Works

Software engineers develop, fine-tune, and deploy deep learning (DL) models using a variety of development frameworks and runtime environments. DL model converters move models between frameworks and to runtime environments. Conversion errors compromise model quality and disrupt deployment. However, the failure characteristics of DL model converters are unknown, adding risk when using DL interoperability technologies. This paper analyzes failures in DL model converters. We survey software engineers about DL interoperability tools, use cases, and pain points (N=92). Then, we characterize failures in model converters associated with the main interoperability tool, ONNX (N=200 issues in PyTorch and TensorFlow). Finally, we formulate …

Challenges And Practices Of Deep Learning Model Reengineering: A Case Study On Computer Vision, Wenxin Jiang, Vishnu Banna, Naveen Vivek, Abhinav Goel, Nicholas Synovic, George K. Thiruvathukal, James C. Davis

Computer Science: Faculty Publications and Other Works

Many engineering organizations are reimplementing and extending deep neural networks from the research community. We describe this process as deep learning model reengineering. Deep learning model reengineering — reusing, replicating, adapting, and enhancing state-of-the-art deep learning approaches — is challenging for reasons including under-documented reference models, changing requirements, and the cost of implementation and testing.

Peatmoss: A Dataset And Initial Analysis Of Pre-Trained Models In Open-Source Software, Wenxin Jiang, Jerin Yasmin, Jason Jones, Nicholas Synovic, Jiashen Kuo, Nathaniel Bielanski, Yuan Tian, George K. Thiruvathukal, James C. Davis

Computer Science: Faculty Publications and Other Works

The development and training of deep learning models have become increasingly costly and complex. Consequently, software engineers are adopting pre-trained models (PTMs) for their downstream applications. The dynamics of the PTM supply chain remain largely unexplored, signaling a clear need for structured datasets that document not only the metadata but also the subsequent applications of these models. Without such data, the MSR community cannot comprehensively understand the impact of PTM adoption and reuse. This paper presents the PeaTMOSS dataset, which comprises metadata for 281,638 PTMs and detailed snapshots for all PTMs with over 50 monthly downloads (14,296 PTMs), along with …

Terry Riley's "In C" For Mobile Ensemble, David B. Wetzel, Griffin Moe, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

This workshop presents a mobile-friendly Web Audio application for a “technology ensemble play-along” of Terry Riley’s 1964 composition In C. Attendees will join in a reading of In C using available web-enabled devices as musical instruments. We hope to demonstrate an accessible music-technology experience that relies on face-to-face interaction within a shared space. In this all-electronic implementation, no special musical or technical expertise is required.

Peatmoss: Mining Pre-Trained Models In Open-Source Software, Wenxin Jiang, Jason Jones, Jerin Yasmin, Nicholas Synovic, Rajiv Sashti, Sophie Chen, George K. Thiruvathukal, Yuan Tian, James C. Davis

Computer Science: Faculty Publications and Other Works

Developing and training deep learning models is expensive, so software engineers have begun to reuse pre-trained deep learning models (PTMs) and fine-tune them for downstream tasks. Despite the widespread use of PTMs, we know little about the corresponding software engineering behaviors and challenges. To enable the study of software engineering with PTMs, we present the PeaTMOSS dataset: Pre-Trained Models in Open-Source Software. PeaTMOSS has three parts: a snapshot of (1) 281,638 PTMs, (2) 27,270 open-source software repositories that use PTMs, and (3) a mapping between PTMs and the projects that use them. We challenge PeaTMOSS miners to discover software engineering …

Ptmtorrent: A Dataset For Mining Open-Source Pre-Trained Model Packages, Wenxin Jiang, Nicholas Synovic, Purvish Jajal, Taylor R. Schorlemmer, Arav Tewari, Bhavesh Pareek, George K. Thiruvathukal, James C. Davis

Computer Science: Faculty Publications and Other Works

Due to the cost of developing and training deep learning models from scratch, machine learning engineers have begun to reuse pre-trained models (PTMs) and fine-tune them for downstream tasks. PTM registries known as “model hubs” support engineers in distributing and reusing deep learning models. PTM packages include pre-trained weights, documentation, model architectures, datasets, and metadata. Mining the information in PTM packages will enable the discovery of engineering phenomena and tools to support software engineers. However, accessing this information is difficult — there are many PTM registries, and both the registries and the individual packages may have rate limiting for accessing …

Conversations With Chatgpt About C Programming: An Ongoing Study, James C. Davis, Yung-Hsiang Lu, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

AI (Artificial Intelligence) Generative Models have attracted great attention in recent years. Generative models can be used to create new articles, visual arts, music composition, even computer programs from English specifications. Among all generative models, ChatGPT is becoming one of the most well-known since its public announcement in November 2022. GPT means {\it Generative Pre-trained Transformer}. ChatGPT is an online program that can interact with human users in text formats and is able to answer questions in many topics, including computer programming. Many computer programmers, including students and professionals, are considering the use of ChatGPT as an aid. The quality …

Snapshot Metrics Are Not Enough: Analyzing Software Repositories With Longitudinal Metrics, Nicholas Synovic, Matt Hyattt, Rohan Sethi, Sohini Thota, Shilpika, Allan J. Miller, Wenxin Jiang, Emmanuel S. Amobi, Austin Pinderski, Konstantin Läufer, Nicholas J. Hayward, Neil Klingensmith, James C. Davis, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

Software metrics capture information about software development processes and products. These metrics support decision-making, e.g., in team management or dependency selection. However, existing metrics tools measure only a snapshot of a software project. Little attention has been given to enabling engineers to reason about metric trends over time -- longitudinal metrics that give insight about process, not just product. In this work, we present PRiME (PRocess MEtrics), a tool for computing and visualizing process metrics. The currently-supported metrics include productivity, issue density, issue spoilage, and bus factor. We illustrate the value of longitudinal data and conclude with a research agenda. …

Unoapi: Balancing Performance, Portability, And Productivity (P3) In Hpc Education, Konstantin Laufer, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

oneAPI is a major initiative by Intel aimed at making it easier to program heterogeneous architectures used in high-performance computing using a unified application programming interface (API). While raising the abstraction level via a unified API represents a promising step for the current generation of students and practitioners to embrace high- performance computing, we argue that a curriculum of well- developed software engineering methods and well-crafted exem- plars will be necessary to ensure interest by this audience and those who teach them. We aim to bridge the gap by developing a curriculum—codenamed UnoAPI—that takes a more holistic approach by looking …

An Empirical Study Of Artifacts And Security Risks In The Pre-Trained Model Supply Chain, Wenxin Jiang, Nicholas Synovic, Rohan Sethi, Aryan Indarapu, Matt Hyattt, Taylor R. Schorlemmer, George K. Thiruvathukal, James C. Davis

Computer Science: Faculty Publications and Other Works

Deep neural networks achieve state-of-the-art performance on many tasks, but require increasingly complex architectures and costly training procedures. Engineers can reduce costs by reusing a pre-trained model (PTM) and fine-tuning it for their own tasks. To facilitate software reuse, engineers collaborate around model hubs, collections of PTMs and datasets organized by problem domain. Although model hubs are now comparable in popularity and size to other software ecosystems, the associated PTM supply chain has not yet been examined from a software engineering perspective.

We present an empirical study of artifacts and security features in 8 model hubs. We indicate the potential …

Requirements Metrics - A Working List, William L. Honig

Computer Science: Faculty Publications and Other Works

A working set of metrics for review of requirements materials including documents.

Establishing Suitable Metrics To Encourage Broader Use Of Atomic Requirements, William L. Honig

Computer Science: Faculty Publications and Other Works

Despite the apparent benefits of singular, individual, or atomic requirements, their use remains limited, and teaching their creation is difficult. The acceptance of a set of requirements metrics specifically designed to evaluate atomic requirements may lead to their better utilization and improved requirements engineering. Twelve metrics designed to measure atomic requirements are presented here: six used on individual requirements statements and six applied to a requirements document or set of requirement statements. Example metrics for individual requirements are Requirement Completeness and Requirement Atomicity; examples to measure multiple requirements include Requirements Traceablity and Requirements Purity. These metrics are designed to work …

Tests As Maintainable Assets Via Auto-Generated Spies: A Case Study Involving The Scala Collections Library's Iterator Trait, Konstantin Läufer, John O'Sullivan, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

In testing stateful abstractions, it is often necessary to record interactions, such as method invocations, and express assertions over these interactions. Following the Test Spy design pattern, we can reify such interactions programmatically through additional mutable state. Alternatively, a mocking framework, such as Mockito, can automatically generate test spies that allow us to record the interactions and express our expectations in a declarative domain-specific language. According to our study of the test code for Scala’s Iterator trait, the latter approach can lead to a significant reduction of test code complexity in terms of metrics such as code size (in some …

Use Of Software Process In Research Software Development:A Survey, Nasir U. Eisty, George K. Thiruvathukal, Jeffrey C. Carver

Computer Science: Faculty Publications and Other Works

Background: Developers face challenges in building high-quality research software due to its inherent complexity. These challenges can reduce the confidence users have in the quality of the result produced by the software. Use of a defined software development process, which divides the development into distinct phases, results in improved design, more trustworthy results, and better project management. Aims: This paper focuses on gaining a better understanding of the use of software development process for research software. Method: We surveyed research software developers to collect information about their use of software development processes. We analyze whether and demographic factors influence the …

A Survey Of Software Metric Use In Research Software Development, Nasir U. Eisty, George K. Thiruvathukal, Jeffrey C. Carver

Computer Science: Faculty Publications and Other Works

Background: Breakthroughs in research increasingly depend on complex software libraries, tools, and applications aimed at supporting specific science, engineering, business, or humanities disciplines. The complexity and criticality of this software motivate the need for ensuring quality and reliability. Software metrics are a key tool for assessing, measuring, and understanding software quality and reliability. Aims: The goal of this work is to better understand how research software developers use traditional software engineering concepts, like metrics, to support and evaluate both the software and the software development process. One key aspect of this goal is to identify how the set of metrics …

Reproducible Research For Computing In Science & Engineering, Lorena A. Barba, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

The editors of the new track for reproducible research outline the parameters for future peer review, submission, and access, highlighting the magazine’s previous work in this field and some of the challenges still to come.

Software Engineering For Science, Jeffrey C. Carver, Neil P. Chue Hong, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

Software Engineering for Science provides an in-depth collection of peer-reviewed chapters that describe experiences with applying software engineering practices to the development of scientific software. It provides a better understanding of how software engineering is and should be practiced, and which software engineering practices are effective for scientific software.

The book starts with a detailed overview of the Scientific Software Lifecycle, and a general overview of the scientific software development process. It highlights key issues commonly arising during scientific software development, as well as solutions to these problems.

The second part of the book provides examples of the use of …

Lack Of Attention To Singular (Or Atomic) Requirements Despite Benefits For Quality, Metrics And Management, William L. Honig, Shingo Takada, Natsuko Noda

Computer Science: Faculty Publications and Other Works

There are seemingly many advantages to being able to identify, document, test, and trace single or “atomic” requirements. Why then has there been little attention to the topic and no widely used definition or process on how to define atomic requirements?

Definitions of requirements and standards focus on user needs, system capabilities or functions; some definitions include making individual requirements singular or without the use of conjunctions. In a few cases there has been a description of atomic system events or requirements. This work is surveyed here although there is no well accepted and used best practice for generating atomic …

Metrics, Software Engineering, Small Systems – The Future Of Systems Development, William L. Honig

Computer Science: Faculty Publications and Other Works

In this talk I will introduce the importance of metrics, or measures, and the role they play in the development of high quality computer systems. I will review some key mega trends in computer science over the last three decades and then explain why I believe the trend to small networked systems, along with metrics and software engineering will define the future of high technology computer based systems.

I first learned about metrics at the Bell System where everything was measured. Metrics can be understood easily if you think of them as measures, for example of calories or salt in …

An Example Of Atomic Requirements - Login Screen, William L. Honig

Computer Science: Faculty Publications and Other Works

A simple example of what an atomic or individual or singular requirement statement should be. Using the example of the familiar login screen, shows the evolution from a low quality initial attempt at requirements to a complete atomic requirement statement. Introduces the idea of a system glossary to support the atomic requirement.

Atomic Requirements Quick Notes, William L. Honig, Shingo Takada

Computer Science: Faculty Publications and Other Works

Working paper on atomic requirements for systems development and the importance of singular, cohesive, individual requirements statements. Covers possible definitions of atomic requirements, and their characteristics. Atomic requirements improve many parts of the development process from requirements to testing and contracting.

Introduction To Atomic Requirements, William L. Honig

Computer Science: Faculty Publications and Other Works

An introduction to requirements and the importance of making single atomic requirements statements. Atomic requirements have advantages and improve the requirements process, support requirement verification and validation, enable traceability, support testability of systems, and provide management advantages.

Why has there been so little emphasis on atomic requirements?

Requirements Quick Notes, William L. Honig, Shingo Takada

Computer Science: Faculty Publications and Other Works

A short introduction to requirements and their role in system development. Includes industry definition of requirements, overview of basic requirements process including numbering of requirements, ties to testing, and traceability. An introduction to requirements quality attributes (correct, unambiguous, etc.) Includes references to requirements process, numbering, and quality papers.

Requirements Metrics - Definitions Of A Working List Of Possible Metrics For Requirements Quality, William L. Honig

Computer Science: Faculty Publications and Other Works

A work in progress to define a metrics set for requirements. Metrics are defined that apply to either the entire requirements set (requirements document as a whole) or individual atomic (or singular, individual) requirements statements. Requirements are identified with standard names and a identification scheme and include both subjective and objective measures.

An example metric for the full set of requirements: Rd2 - Requirements Consistency, Is the set of atomic requirements internally consistent, with no contradictions, no duplication between individual requirements? An example of a metric for a single requirement: Ra4 - Requirement Verifiability, How adequately can this requirement be …