Physical Sciences and Mathematics Commons^™

Evaluating Large Language Model Performance On Haskell, Andrew Chen

Undergraduate Honors Theses

I introduce HaskellEval, a Haskell evaluation benchmark for Large Language Models. HaskellEval’s curation leverages a novel synthetic generation framework, streamlining the process of dataset curation by minimizing manual intervention. The core of this research is an extensive analysis of the trustworthiness of synthetic generations, ensuring accuracy, realism, and diversity. Additional, I provide a comprehensive evaluation of existing open-source models on HaskellEval.

Security And Interpretability In Large Language Models, Lydia Danas

Undergraduate Honors Theses

Large Language Models (LLMs) have the capability to model long-term dependencies in sequences of tokens, and are consequently often utilized to generate text through language modeling. These capabilities are increasingly being used for code generation tasks; however, LLM-powered code generation tools such as GitHub's Copilot have been generating insecure code and thus pose a cybersecurity risk. To generate secure code we must first understand why LLMs are generating insecure code. This non-trivial task can be realized through interpretability methods, which investigate the hidden state of a neural network to explain model outputs. A new interpretability method is rationales, which obtains …

Improving The Scalability Of Neural Network Surface Code Decoders, Kevin Wu

Undergraduate Honors Theses

Quantum computers have recently gained significant recognition due to their ability to solve problems intractable to classical computers. However, due to difficulties in building actual quantum computers, they have large error rates. Thus, advancements in quantum error correction are urgently needed to improve both their reliability and scalability. Here, we first present a type of topological quantum error correction code called the surface code, and we discuss recent developments and challenges of creating neural network decoders for surface codes. In particular, the amount of training data needed to reach the performance of algorithmic decoders grows exponentially with the size of …

Code Syntax Understanding In Large Language Models, Cole Granger

Undergraduate Honors Theses

In recent years, tasks for automated software engineering have been achieved using Large Language Models trained on source code, such as Seq2Seq, LSTM, GPT, T5, BART and BERT. The inherent textual nature of source code allows it to be represented as a sequence of sub-words (or tokens), drawing parallels to prior work in NLP. Although these models have shown promising results according to established metrics (e.g., BLEU, CODEBLEU), there remains a deeper question about the extent of syntax knowledge they truly grasp when trained and fine-tuned for specific tasks.

To address this question, this thesis introduces a taxonomy of syntax …

Kfactorvae: Self-Supervised Regularization For Better A.I. Disentanglement, Joseph S. Lee

Undergraduate Honors Theses

Obtaining disentangled representations is a goal sought after to make A.I. models more interpretable. Studies have proven the impossibility of obtaining these kinds of representations with just unsupervised learning, or in other words, without strong inductive biases. One strong inductive bias is a regularization term that encourages the invariance of factors of variations across an image and a carefully selected augmentation. In this thesis, we build upon the existing Variational Autoencoder (VAE)-based disentanglement literature by utilizing the aforementioned inductive bias. We evaluate our method on the dSprites dataset, a well-known benchmark, and demonstrate its ability to achieve comparable or higher …

Quantum Federated Learning: Training Hybrid Neural Networks Collaboratively, Anneliese Brei

Undergraduate Honors Theses

This thesis explores basic concepts of machine learning, neural networks, federated learning, and quantum computing in an effort to better understand Quantum Machine Learning, an emerging field of research. We propose Quantum Federated Learning (QFL), a schema for collaborative distributed learning that maintains privacy and low communication costs. We demonstrate the QFL framework and local and global update algorithms with implementations that utilize TensorFlow Quantum libraries. Our experiments test the effectiveness of frameworks of different sizes. We also test the effect of changing the number of training cycles and changing distribution of training data. This thesis serves as a synoptic …