Artificial Intelligence and Robotics Commons^™

A Computer Vision Solution To Cross-Cultural Food Image Classification And Nutrition Logging​, Rohan Sethi, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

The US is a culturally and ethnically diverse country, and with this diversity comes a myriad of cuisines and eating habits that expand well beyond that of western culture. Each of these meals have their own good and bad effects when it comes to the nutritional value and its potential impact on human health. Thus, there is a greater need for people to be able to access the nutritional profile of their diverse daily meals and better manage their health. A revolutionary solution to democratize food image classification and nutritional logging is using deep learning to extract that information from …

Single-Class Target-Specific Attack Against Interpretable Deep Learning Systems, Eldor Abdukhamidov, Mohammed Abuhamad, George K. Thiruvathukal, Hyoungshick Kim, Tamer Abuhmed

Computer Science: Faculty Publications and Other Works

In this paper, we present a novel Single-class target-specific Adversarial attack called SingleADV. The goal of SingleADV is to generate a universal perturbation that deceives the target model into confusing a specific category of objects with a target category while ensuring highly relevant and accurate interpretations. The universal perturbation is stochastically and iteratively optimized by minimizing the adversarial loss that is designed to consider both the classifier and interpreter costs in targeted and non-targeted categories. In this optimization framework, ruled by the first- and second-moment estimations, the desired loss surface promotes high confidence and interpretation score of adversarial samples. By …

Optimizing Uncertainty Quantification Of Vision Transformers In Deep Learning On Novel Ai Architectures, Erik Pautsch, John Li, Silvio Rizzi, George K. Thiruvathukal, Maria Pantoja

Computer Science: Faculty Publications and Other Works

Deep Learning (DL) methods have shown substantial efficacy in computer vision (CV) and natural language processing (NLP). Despite their proficiency, the inconsistency in input data distributions can compromise prediction reliability. This study mitigates this issue by introducing uncertainty evaluations in DL models, thereby enhancing dependability through a distribution of predictions. Our focus lies on the Vision Transformer (ViT), a DL model that harmonizes both local and global behavior. We conduct extensive experiments on the ImageNet-1K dataset, a vast resource with over a million images across 1,000 categories. ViTs, while competitive, are vulnerable to adversarial attacks, making uncertainty estimation crucial for …

Optimized Uncertainty Estimation For Vision Transformers: Enhancing Adversarial Robustness And Performance Using Selective Classification, Erik Pautsch, John Li, Silvio Rizzi, George K. Thiruvathukal, Maria Pantoja

Computer Science: Faculty Publications and Other Works

Deep Learning models often exhibit undue confidence when encountering out-of-distribution (OOD) inputs, misclassifying with high confidence. The ideal outcome, in these cases, would be an "I do not know" verdict. We enhance the trustworthiness of our models through selective classification, allowing the model to abstain from making predictions when facing uncertainty. Rather than a singular prediction, the model offers a prediction distribution, enabling users to gauge the model’s trustworthiness and determine the need for human intervention. We assess uncertainty in two baseline models: a Convolutional Neural Network (CNN) and a Vision Transformer (ViT). By leveraging these uncertainty values, we minimize …

Peatmoss: Mining Pre-Trained Models In Open-Source Software, Wenxin Jiang, Jason Jones, Jerin Yasmin, Nicholas Synovic, Rajiv Sashti, Sophie Chen, George K. Thiruvathukal, Yuan Tian, James C. Davis

Computer Science: Faculty Publications and Other Works

Developing and training deep learning models is expensive, so software engineers have begun to reuse pre-trained deep learning models (PTMs) and fine-tune them for downstream tasks. Despite the widespread use of PTMs, we know little about the corresponding software engineering behaviors and challenges. To enable the study of software engineering with PTMs, we present the PeaTMOSS dataset: Pre-Trained Models in Open-Source Software. PeaTMOSS has three parts: a snapshot of (1) 281,638 PTMs, (2) 27,270 open-source software repositories that use PTMs, and (3) a mapping between PTMs and the projects that use them. We challenge PeaTMOSS miners to discover software engineering …

Tree-Based Unidirectional Neural Networks For Low-Power Computer Vision, Abhinav Goel, Caleb Tung, Nick Eliopoulos, Amy Wang, Jamie C. Davis, George K. Thiruvathukal, Yung-Hisang Lu

Computer Science: Faculty Publications and Other Works

This article describes the novel Tree-based Unidirectional Neural Network (TRUNK) architecture. This architecture improves computer vision efficiency by using a hierarchy of multiple shallow Convolutional Neural Networks (CNNs), instead of a single very deep CNN. We demonstrate this architecture’s versatility in performing different computer vision tasks efficiently on embedded devices. Across various computer vision tasks, the TRUNK architecture consumes 65% less energy and requires 50% less memory than representative low-power CNN architectures, e.g., MobileNet v2, when deployed on the NVIDIA Jetson Nano.

Conversations With Chatgpt About C Programming: An Ongoing Study, James C. Davis, Yung-Hsiang Lu, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

AI (Artificial Intelligence) Generative Models have attracted great attention in recent years. Generative models can be used to create new articles, visual arts, music composition, even computer programs from English specifications. Among all generative models, ChatGPT is becoming one of the most well-known since its public announcement in November 2022. GPT means {\it Generative Pre-trained Transformer}. ChatGPT is an online program that can interact with human users in text formats and is able to answer questions in many topics, including computer programming. Many computer programmers, including students and professionals, are considering the use of ChatGPT as an aid. The quality …

Observing Human Mobility Internationally During Covid-19, Shane Allcroft, Mohammed Metwaly, Zachery Berg, Isha Ghodgaonkar, Fischer Bordwell, Xinxin Zhao, Xinglei Liu, Jiahao Xu, Subhankar Chakraborty, Vishnu Banna, Akhil Chinnakotla, Abhinav Goel, Caleb Tung, Gore Kao, Wei Zakharov, David A. Shoham, George K. Thiruvathukal, Yung-Hsiang Lu

Computer Science: Faculty Publications and Other Works

This article analyzes visual data captured from five countries and three U.S. states to evaluate the effectiveness of lockdown policies for reducing the spread of COVID-19. The main challenge is the scale: nearly six million images are analyzed to observe how people respond to the policy changes.

Evolution Of Winning Solutions In The 2021 Low-Power Computer Vision Challenge, Xiao Hu, Ziteng Jiao, Ayden Kocher, Zhenyu Wu, Junjie Liu, James C. Davis, George K. Thiruvathukal, Yung-Hsiang Lu

Computer Science: Faculty Publications and Other Works

Mobile and embedded devices are becoming ubiquitous. Applications such as rescue with autonomous robots and event analysis on traffic cameras rely on devices with limited power supply and computational sources. Thus, the demand for efficient computer vision algorithms increases. Since 2015, we have organized the IEEE Low-Power Computer Vision Challenge to advance the state of the art in low-power computer vision. We describe the competition organizing details including the challenge design, the reference solution, the dataset, the referee system, and the evolution of the solutions from two winning teams. We examine the winning teams’ development patterns and design decisions, focusing …

College Teaching And Ai, Leo Irakliotis

Computer Science: Faculty Publications and Other Works

Artificial Intelligence will reshape the way we assess student learning in ways that no one has prepared us for.

An Empirical Study Of Artifacts And Security Risks In The Pre-Trained Model Supply Chain, Wenxin Jiang, Nicholas Synovic, Rohan Sethi, Aryan Indarapu, Matt Hyattt, Taylor R. Schorlemmer, George K. Thiruvathukal, James C. Davis

Computer Science: Faculty Publications and Other Works

Deep neural networks achieve state-of-the-art performance on many tasks, but require increasingly complex architectures and costly training procedures. Engineers can reduce costs by reusing a pre-trained model (PTM) and fine-tuning it for their own tasks. To facilitate software reuse, engineers collaborate around model hubs, collections of PTMs and datasets organized by problem domain. Although model hubs are now comparable in popularity and size to other software ecosystems, the associated PTM supply chain has not yet been examined from a software engineering perspective.

We present an empirical study of artifacts and security features in 8 model hubs. We indicate the potential …

Directed Acyclic Graph-Based Neural Networks For Tunable Low-Power Computer Vision, Abhinav Goel, Caleb Tung, Nick Eliopoulos, Xiao Hu, George K. Thiruvathukal, James C. Davis, Yung-Hisang Lu

Computer Science: Faculty Publications and Other Works

Processing visual data on mobile devices has many applications, e.g., emergency response and tracking. State-of-the-art computer vision techniques rely on large Deep Neural Networks (DNNs) that are usually too power-hungry to be deployed on resource-constrained edge devices. Many techniques improve DNN efficiency of DNNs by compromising accuracy. However, the accuracy and efficiency of these techniques cannot be adapted for diverse edge applications with different hardware constraints and accuracy requirements. This paper demonstrates that a recent, efficient tree-based DNN architecture, called the hierarchical DNN, can be converted into a Directed Acyclic Graph-based (DAG) architecture to provide tunable accuracy-efficiency tradeoff options. We …

Are You Really Muted?: A Privacy Analysis Of Mute Buttons In Video Conferencing Apps, Yucheng Yang, Jack West, George K. Thiruvathukal, Neil Klingensmith, Kassem Fawaz

Computer Science: Faculty Publications and Other Works

In the post-pandemic era, video conferencing apps (VCAs) have converted previously private spaces — bedrooms, living rooms, and kitchens — into semi-public extensions of the office. And for the most part, users have accepted these apps in their personal space, without much thought about the permission models that govern the use of their personal data during meetings. While access to a device’s video camera is carefully controlled, little has been done to ensure the same level of privacy for accessing the microphone. In this work, we ask the question: what happens to the microphone data when a user clicks the …

Irrelevant Pixels Are Everywhere: Find And Exclude Them For More Efficient Computer Vision, Caleb Tung, Abhinav Goel, Xiao Hu, Nick Eliopoulos, Emmanuel Amobi, George K. Thiruvathukal, Vipin Chaudhary, Yung-Hisang Lu

Computer Science: Faculty Publications and Other Works

Computer vision is often performed using Convolutional Neural Networks (CNNs). CNNs are compute-intensive and challenging to deploy on power-constrained systems such as mobile and Internet-of-Things (IoT) devices. CNNs are compute-intensive because they indiscriminately compute many features on all pixels of the input image. We observe that, given a computer vision task, images often contain pixels that are irrelevant to the task. For example, if the task is looking for cars, pixels in the sky are not very useful. Therefore, we propose that a CNN be modified to only operate on relevant pixels to save computation and energy. We propose a …

Towards An Active Foveated Approach To Computer Vision, Dario Dematties, Silvio Rizzi, George K. Thiruvathukal, Alejandro Javier Wainselboim

Computer Science: Faculty Publications and Other Works

In this paper, a series of experimental methods are presented explaining a new approach towards active foveated Computer Vision (CV). This is a collaborative effort between researchers at CONICET Mendoza Technological Scientific Center from Argentina, Argonne National Laboratory (ANL), and Loyola University Chicago from the US. The aim is to advance new CV approaches more in line with those found in biological agents in order to bring novel solutions to the main problems faced by current CV applications. Basically this work enhance Self-supervised (SS) learning, incorporating foveated vision plus saccadic behavior in order to improve training and computational efficiency without …

Modular Neural Networks For Low-Power Image Classification On Embedded Devices, Abhinav Goel, Sara Aghajanzadeh, Caleb Tung, Shuo-Han Chen, George K. Thiruvathukal, Yung-Hisang Lu

Computer Science: Faculty Publications and Other Works

Embedded devices are generally small, battery-powered computers with limited hardware resources. It is difficult to run deep neural networks (DNNs) on these devices, because DNNs perform millions of operations and consume significant amounts of energy. Prior research has shown that a considerable number of a DNN’s memory accesses and computation are redundant when performing tasks like image classification. To reduce this redundancy and thereby reduce the energy consumption of DNNs, we introduce the Modular Neural Network Tree architecture. Instead of using one large DNN for the classifier, this architecture uses multiple smaller DNNs (called modules) to progressively classify images …

Exercises Integrating High School Mathematics With Robot Motion Planning, Ronald I. Greenberg, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

This paper presents progress in developing exercises for high school students incorporating level-appropriate mathematics into robotics activities. We assume mathematical foundations ranging from algebra to precalculus, whereas most prior work on integrating mathematics into robotics uses only very elementary mathematical reasoning or, at the other extreme, is comprised of technical papers or books using calculus and other advanced mathematics. The exercises suggested are relevant to any differerential-drive robot, which is an appropriate model for many different varieties of educational robots. They guide students towards comparing a variety of natural navigational strategies making use of typical movement primitives. The exercises align …

Mathematics And Programming Exercises For Educational Robot Navigation, Ronald I. Greenberg

Computer Science: Faculty Publications and Other Works

This paper points students towards ideas they can use towards developing a convenient library for robot navigation, with examples based on Botball primitives, and points educators towards mathematics and programming exercises they can suggest to students, especially advanced high school students.

Integrating Mathematics And Educational Robotics: Simple Motion Planning, Ronald I. Greenberg, George K. Thiruvathukal, Sara T. Greenberg

Computer Science: Faculty Publications and Other Works

This paper shows how students can be guided to integrate elementary mathematical analyses with motion planning for typical educational robots. Rather than using calculus as in comprehensive works on motion planning, we show students can achieve interesting results using just simple linear regression tools and trigonometric analyses. Experiments with one robotics platform show that use of these tools can lead to passable navigation through dead reckoning even if students have limited experience with use of sensors, programming, and mathematics.

Cross-Referencing Social Media And Public Surveillance Camera Data For Disaster Response, Chittayong Surakitbanharn,, Calvin Yau, Guizhen Wang, Aniesh Chawla, Yinuo Pan, Zhaoya Sun, Sam Yellin, David Ebert, Yung-Hsiang Lu, George K. Thiruvathukal

Computer Science: Faculty Publications and Other Works

Physical media (like surveillance cameras) and social media (like Instagram and Twitter) may both be useful in attaining on-the-ground information during an emergency or disaster situation. However, the intersection and reliability of both surveillance cameras and social media during a natural disaster are not fully understood. To address this gap, we tested whether social media is of utility when physical surveillance cameras went off-line during Hurricane Irma in 2017. Specifically, we collected and compared geo-tagged Instagram and Twitter posts in the state of Florida during times and in areas where public surveillance cameras went off-line. We report social media content …

Comparison Of Visual Datasets For Machine Learning, Kent Gauen, Ryan Dailey, John Laiman, Yuxiang Zi, Nirmal Asokan, Yung-Hsiang Lu, George K. Thiruvathukal, Mei-Ling Shyu, Shu-Ching Chen

Computer Science: Faculty Publications and Other Works

One of the greatest technological improvements in recent years is the rapid progress using machine learning for processing visual data. Among all factors that contribute to this development, datasets with labels play crucial roles. Several datasets are widely reused for investigating and analyzing different solutions in machine learning. Many systems, such as autonomous vehicles, rely on components using machine learning for recognizing objects. This paper compares different visual datasets and frameworks for machine learning. The comparison is both qualitative and quantitative and investigates object detection labels with respect to size, location, and contextual information. This paper also presents a new …

Intelligent Systems Development In A Non Engineering Curriculum, Emily A. Brand, William L. Honig, Matthew Wojtowicz

Computer Science: Faculty Publications and Other Works

Much of computer system development today is programming in the large - systems of millions of lines of code distributed across servers and the web. At the same time, microcontrollers have also become pervasive in everyday products, economical to manufacture, and represent a different level of learning about system development. Real world systems at this level require integrated development of custom hardware and software.

How can academic institutions give students a view of this other extreme - programming on small microcontrollers with specialized hardware? Full scale system development including custom hardware and software is expensive, beyond the range of any …