Engineering Commons^™

Online Aircraft System Identification Using A Novel Parameter Informed Reinforcement Learning Method, Nathan Schaff

Doctoral Dissertations and Master's Theses

This thesis presents the development and analysis of a novel method for training reinforcement learning neural networks for online aircraft system identification of multiple similar linear systems, such as all fixed wing aircraft. This approach, termed Parameter Informed Reinforcement Learning (PIRL), dictates that reinforcement learning neural networks should be trained using input and output trajectory/history data as is convention; however, the PIRL method also includes any known and relevant aircraft parameters, such as airspeed, altitude, center of gravity location and/or others. Through this, the PIRL Agent is better suited to identify novel/test-set aircraft.

First, the PIRL method is applied to …

Quantifying Balance: Computational And Learning Frameworks For The Characterization Of Balance In Bipedal Systems, Kubra Akbas

Dissertations

In clinical practice and general healthcare settings, the lack of reliable and objective balance and stability assessment metrics hinders the tracking of patient performance progression during rehabilitation; the assessment of bipedal balance plays a crucial role in understanding stability and falls in humans and other bipeds, while providing clinicians important information regarding rehabilitation outcomes. Bipedal balance has often been examined through kinematic or kinetic quantities, such as the Zero Moment Point and Center of Pressure; however, analyzing balance specifically through the body's Center of Mass (COM) state offers a holistic and easily comprehensible view of balance and stability.

Building upon …

Motion Control Simulation Of A Hexapod Robot, Weishu Zhan

Dartmouth College Master’s Theses

This thesis addresses hexapod robot motion control. Insect morphology and locomotion patterns inform the design of a robotic model, and motion control is achieved via trajectory planning and bio-inspired principles. Additionally, deep learning and multi-agent reinforcement learning are employed to train the robot motion control strategy with leg coordination achieves using a multi-agent deep reinforcement learning framework. The thesis makes the following contributions:

First, research on legged robots is synthesized, with a focus on hexapod robot motion control. Insect anatomy analysis informs the hexagonal robot body and three-joint single robotic leg design, which is assembled using SolidWorks. Different gaits are …

Multi-Agent Learning For Game-Theoretical Problems, Kshitija Taywade

Theses and Dissertations--Computer Science

Multi-agent systems are prevalent in the real world in various domains. In many multi-agent systems, interaction among agents is inevitable, and cooperation in some form is needed among agents to deal with the task at hand. We model the type of multi-agent systems where autonomous agents inhabit an environment with no global control or global knowledge, decentralized in the true sense. In particular, we consider game-theoretical problems such as the hedonic coalition formation games, matching problems, and Cournot games. We propose novel decentralized learning and multi-agent reinforcement learning approaches to train agents in learning behaviors and adapting to the environments. …

Peer-To-Peer Energy Trading In Smart Residential Environment With User Behavioral Modeling, Ashutosh Timilsina

Theses and Dissertations--Computer Science

Electric power systems are transforming from a centralized unidirectional market to a decentralized open market. With this shift, the end-users have the possibility to actively participate in local energy exchanges, with or without the involvement of the main grid. Rapidly reducing prices for Renewable Energy Technologies (RETs), supported by their ease of installation and operation, with the facilitation of Electric Vehicles (EV) and Smart Grid (SG) technologies to make bidirectional flow of energy possible, has contributed to this changing landscape in the distribution side of the traditional power grid.

Trading energy among users in a decentralized fashion has been referred …

Benchmarking Model Predictive Control And Reinforcement Learning For Legged Robot Locomotion, Shivayogi Akki

Dissertations, Master's Theses and Master's Reports

This research delves into the realm of quadrupedal robotics, focusing on the comparative analysis of Model Predictive Control (MPC) and Reinforcement Learning (RL) as predominant control strategies. Through the comprehensive dataset compiled and the insights derived from this analysis, this research aims to serve as a valuable resource for the legged robotics community, guiding researchers and practitioners in the selection and implementation of control strategies. The ultimate goal is to contribute to the advancement of legged robot capabilities and facilitate their successful deployment in real-world applications.

In this study, we employ the Unitree Go1 quadrupedal robot as a testbed, subjecting …

Development And Deployment Of A Dynamic Soaring Capable Uav Using Reinforcement Learning, Jacob Adamski

Doctoral Dissertations and Master's Theses

Dynamic soaring (DS) is a bio-inspired flight maneuver in which energy can be gained by flying through regions of vertical wind gradient such as the wind shear layer. With reinforcement learning (RL), a fixed wing unmanned aerial vehicle (UAV) can be trained to perform DS maneuvers optimally for a variety of wind shear conditions. To accomplish this task, a 6-degreesof- freedom (6DoF) flight simulation environment in MATLAB and Simulink has been developed which is based upon an off-the-shelf unmanned aerobatic glider. A combination of high-fidelity Reynolds-Averaged Navier-Stokes (RANS) computational fluid dynamics (CFD) in ANSYS Fluent and low-fidelity vortex lattice (VLM) …

Using Reinforcement Learning To Improve Network Reliability Through Optimal Resource Allocation, Henley Wells

Graduate Theses and Dissertations

Networks provide a variety of critical services to society (e.g. power grid, telecommunication, water, transportation) but are prone to disruption. With this motivation, we study a sequential decision problem in which an initial network is improved over time (e.g., by adding or increasing the reliability of edges) and rewards are gained over time as a function of the network’s all-terminal reliability. The actions during each time period are limited due to availability of resources such as time, money, or labor. To solve this problem, we utilized a Deep Reinforcement Learning (DRL) approach implemented within OpenAI-Gym using Stable Baselines. A Proximal …

Adaptive Multi-Scale Place Cell Representations And Replay For Spatial Navigation And Learning In Autonomous Robots, Pablo Scleidorovich

USF Tampa Graduate Theses and Dissertations

Place cells are one of the most widely studied neurons thought to play a vital role in spatial cognition. Extensive studies show that their activity in the rodent hippocampus is highly correlated with the animal’s spatial location, forming “place fields” of smaller sizes near the dorsal pole and larger sizes near the ventral pole. Despite advances, it is yet unclear how this multi-scale representation enables navigation in complex environments.

In this dissertation, we analyze the place cell representation from a computational point of view, evaluating how multi-scale place fields impact navigation in large and cluttered environments. The objectives are to …

Developing Novel Optimization And Machine Learning Frameworks To Improve And Assess The Safety Of Workplaces, Amin Aghalari

Theses and Dissertations

This study proposes several decision-making tools utilizing optimization and machine learning frameworks to assess and improve the safety of the workplaces. The first chapter of this study presents a novel mathematical model to optimally locate a set of detectors to minimize the expected number of casualties in a given threat area. The problem is formulated as a nonlinear binary integer programming model and then solved as a linearized branch-and-bound algorithm. Several sensitivity analyses illustrate the model's robustness and draw key managerial insights. One of the prevailing threats in the last decades, Active Shooting (AS) violence, poses a serious threat to …

Decentralized Intelligent Decision Making In Cyber Physical Social Systems, Nathan Patrizi

Electrical and Computer Engineering ETDs

The accelerated evolution towards jointly considering the physical, cyber, and social space is expected to dramatically increase the interest of the research and industrial community to build efficient, resilient, and secure Cyber Physical Social Systems. In this dissertation, we focus our research activities on devising decentralized intelligent decision making models, frameworks, and algorithms to support the smooth operation of Cyber Physical Social Systems. The proposed decentralized intelligent decision making models are jointly exploiting theories from the field of Economics, such as Game Theory and Contract Theory, and from the field of Computer Science, such as Reinforcement Learning concepts. Reinforcement learning …

Decision-Analytic Models Using Reinforcement Learning To Inform Dynamic Sequential Decisions In Public Policy, Seyedeh Nazanin Khatami

Doctoral Dissertations

We developed decision-analytic models specifically suited for long-term sequential decision-making in the context of large-scale dynamic stochastic systems, focusing on public policy investment decisions. We found that while machine learning and artificial intelligence algorithms provide the most suitable frameworks for such analyses, multiple challenges arise in its successful adaptation. We address three specific challenges in two public sectors, public health and climate policy, through the following three essays. In Essay I, we developed a reinforcement learning (RL) model to identify optimal sequence of testing and retention-in-care interventions to inform the national strategic plan “Ending the HIV Epidemic in the US”. …

Analyzing Decision-Making In Robot Soccer For Attacking Behaviors, Justin Rodney

USF Tampa Graduate Theses and Dissertations

In robotics soccer, decision-making is critical to the performance of a team’s SoftwareSystem. The University of South Florida’s (USF) RoboBulls team implements behavior for the robots by using traditional methods such as analytical geometry to path plan and determine whether an action should be taken. In recent works, Machine Learning (ML) and Reinforcement Learning (RL) techniques have been used to calculate the probability of success for a pass or goal, and even train models for performing low-level skills such as traveling towards a ball and shooting it towards the goal[1, 2]. Open-source frameworks have been created for training Reinforcement Learning …

Machine Learning Based Dynamic Power Dispatching And Smoothing Using Hybrid Energy Storage System For Renewable Energy Systems, Bhuvaneshwarr Ramalingam

Electrical and Computer Engineering ETDs

The stochastic fluctuations from Renewable Energy Resources (RER) have a great influence on power quality and off-grid communities. A combination of the different storage systems is accessible for RER generation intermittency and to bring about finest smoothing operating cycle compared to sole Energy Storage System (ESS). Additionally, energy management in Hybrid Energy Storage System (HESS) creates an uncertainty during power smoothing operation. This research materializes, an intelligent mechanism for power smoothing and dispatch with the introduction of hybridized storage that can accommodate the unpredictable behavior of RER under dynamic load. A feed-forward neural network is proposed as a power smoothing …

Rebalancing Shared Mobility Systems By User Incentive Scheme Via Reinforcement Learning, Matthew Brian Schofield

Theses and Dissertations

Shared mobility systems regularly suffer from an imbalance of vehicle supply within the system, leading to users being unable to receive service. If such imbalance problems are not mitigated some users will not be serviced. There is an increasing interest in the use of reinforcement learning (RL) techniques for improving the resource supply balance and service level of systems. The goal of these techniques is to produce an effective user incentivization policy scheme to encourage users of a shared mobility system to slightly alter their travel behavior in exchange for a small monetary incentive. These slight changes in user behavior …

Artificial Intelligent Risk-Aware Autonomous Decision-Making In Resource-Constrained Computing Systems, Pavlos Athanasios Apostolopoulos

Electrical and Computer Engineering ETDs

Artificial Intelligent autonomous systems are becoming increasingly ubiquitous in daily life. Mobile devices for example provide mechanical-generated intelligent support to humans, with various degrees of autonomy, and are a key part of the recent autonomous revolution. Autonomous intelligent systems aim to understand and interact with their users in a timely manner, while many of them are characterized by constrained resources. Despite that, the average person does not act in a formulaic and risk-neutral manner but instead exhibits risk-aware attitudes when performing a task that includes sources of uncertainties. When humans make decisions, they explore their surroundings, understand the emerging risks, …

A Study Of Deep Reinforcement Learning In Autonomous Racing Using Deepracer Car, Mukesh Ghimire

Honors Theses

Reinforcement learning is thought to be a promising branch of machine learning that has the potential to help us develop an Artificial General Intelligence (AGI) machine. Among the machine learning algorithms, primarily, supervised, semi supervised, unsupervised and reinforcement learning, reinforcement learning is different in a sense that it explores the environment without prior knowledge, and determines the optimal action. This study attempts to understand the concept behind reinforcement learning, the mathematics behind it and see it in action by deploying the trained model in Amazon's DeepRacer car. DeepRacer, a 1/18^th scaled autonomous car, is the agent which is trained …

Reinforcement Learning-Based Access Schemes In Cognitive Radio Networks, Ehab Maged Elguindy

Theses and Dissertations

In this thesis, we propose different MAC protocols based on three Reinforcement Learning (RL) approaches, namely Q-Learning, Deep Q-Network (DQN), and Deep Deterministic Policy Gradient (DDPG). We exploit the primary user (PU) feedback, in the form of ARQ and CQI bits, to enhance the performance of the secondary user (SU) MAC protocols. Exploiting the PU feedback information can be applied on the top of any SU sensing-based MAC protocol. Our proposed model relies on two main pillars, namely, an infinite-state Partially Observable Markov Decision Process (POMDP) to model the system dynamics besides a queuing-theoretic model for the PU queue; the …

Neural Network Supervised And Reinforcement Learning For Neurological, Diagnostic, And Modeling Problems, Donald Wunsch Iii

Masters Theses

“As the medical world becomes increasingly intertwined with the tech sphere, machine learning on medical datasets and mathematical models becomes an attractive application. This research looks at the predictive capabilities of neural networks and other machine learning algorithms, and assesses the validity of several feature selection strategies to reduce the negative effects of high dataset dimensionality. Our results indicate that several feature selection methods can maintain high validation and test accuracy on classification tasks, with neural networks performing best, for both single class and multi-class classification applications. This research also evaluates a proof-of-concept application of a deep-Q-learning network (DQN) to …

Eliciting & Visualizing Bias In Hiring Practices, Tsitsi Mambo

Senior Projects Fall 2021

This project seeks to develop a way to elicit and visualize bias in the hiring process through the use of Markov Decision Processes, a mathematical framework for modeling decision processes. Three forms of the simulation: User-defined, Random, and Q-learning, were created and their policies were analyzed and compared. Heat Map and Donut Pie visualizations are utilized to present the Policies created from the Models. This project is designed to display the decisions as a form of countering bias during the hiring process.

Scheduling Allocation And Inventory Replenishment Problems Under Uncertainty: Applications In Managing Electric Vehicle And Drone Battery Swap Stations, Amin Asadi

Graduate Theses and Dissertations

In this dissertation, motivated by electric vehicle (EV) and drone application growth, we propose novel optimization problems and solution techniques for managing the operations at EV and drone battery swap stations. In Chapter 2, we introduce a novel class of stochastic scheduling allocation and inventory replenishment problems (SAIRP), which determines the recharging, discharging, and replacement decisions at a swap station over time to maximize the expected total profit. We use Markov Decision Process (MDP) to model SAIRPs facing uncertain demands, varying costs, and battery degradation. Considering battery degradation is crucial as it relaxes the assumption that charging/discharging batteries do not …

Reinforcement Learning For Mobile Robot Collision Avoidance In Navigation Tasks, Zilong Jiao

Dissertations - ALL

Collision avoidance is fundamental for mobile robot navigation. In general, its solutions include: {\it map-based} and {\it mapless approaches.} In the map-based approach, robots pre-plan collision-free paths based on an environment map and follow their paths during navigation. On the other hand, the mapless approach requires robots to avoid collisions without referencing to an environment map. This thesis first studies the map-based approach for multiple robots to collectively build environment maps. In this study, a robot following a pre-planned path may encounter unexpected obstacles, such as other moving robots and obstacles inaccurately presented on an environment map. This motivates us …

Reinforcement Learning Approach For Inspect/Correct Tasks, Hoda Nasereddin

LSU Doctoral Dissertations

In this research, we focus on the application of reinforcement learning (RL) in automated agent tasks involving considerable target variability (i.e., characterized by stochastic distributions); in particular, learning of inspect/correct tasks. Examples include automated identification & correction of rivet failures in airplane maintenance procedures, and automated cleaning of surgical instruments in a hospital sterilization processing department. The location of defects and the corrective action to be taken for each varies from task episode. What needs to be learned are optimal stochastic strategies rather than optimization of any one single defect type and location. RL has been widely applied in robotics …

Artificial Intelligence Enabled Distributed Edge Computing For Internet Of Things Applications, Georgios Fragkos

Electrical and Computer Engineering ETDs

Artificial Intelligence (AI) based techniques are typically used to model decision-making in terms of strategies and mechanisms that can conclude to optimal payoffs for a number of interacting entities, often presenting competitive behaviors. In this thesis, an AI-enabled multi-access edge computing (MEC) framework is proposed, supported by computing-equipped Unmanned Aerial Vehicles (UAVs) to facilitate Internet of Things (IoT) applications. Initially, the problem of determining the IoT nodes optimal data offloading strategies to the UAV-mounted MEC servers, while accounting for the IoT nodes’ communication and computation overhead, is formulated based on a game-theoretic model. The existence of at least one Pure …

Socially Aware Network User Mobility Analysis And Novel Approaches On Aerial Mobile Wireless Network Deployment, Ismail Uluturk

USF Tampa Graduate Theses and Dissertations

Service demand patterns for wireless networks are evolving with the technological developments in areas such as personal computing, unmanned vehicles, and internet-of-things, where increasing mobile service demand is one of the significant challenges introduced. In addition to these new intrinsic dynamics, natural disasters and societal upheaval are also disrupting the conventional patterns of network demand. Situations like damaged infrastructure due to a natural disaster or large numbers of displaced people caused by political strife and social upheaval demand flexible, rapidly deployable network architectures. The increasing demands of next-generation communication services are straining the capabilities of the traditional approach of the …

A Comparative Analysis Of Reinforcement Learning Applied To Task-Space Reaching With A Robotic Manipulator With And Without Gravity Compensation, Jonathan Fugal

Theses and Dissertations--Electrical and Computer Engineering

Advances in computing power in recent years have facilitated developments in autonomous robotic systems. These robotic systems can be used in prosthetic limbs, wearhouse packaging and sorting, assembly line production, as well as many other applications. Designing these autonomous systems typically requires robotic system and world models (for classical control based strategies) or time consuming and computationally expensive training (for learning based strategies). Often these requirements are difficult to fulfill. There are ways to combine classical control and learning based strategies that can mitigate both requirements. One of these ways is to use a gravity compensated torque control with reinforcement …

Landing Throttleable Hybrid Rockets With Hierarchical Reinforcement Learning In A Simulated Environment, Francesco Alessandro Stefano Mikulis-Borsoi

Honors Theses and Capstones

In this paper, I develop a hierarchical Markov Decision Process (MDP) structure for completing the task of vertical rocket landing. I start by covering the background of this problem, and formally defining its constraints. In order to reduce mistakes while formulating different MDPs, I define and develop the criteria for a standardized MDP definition format. I then decompose the problem into several sub-problems of vertical landing, namely velocity control and vertical stability control. By exploiting MDP coupling and symmetrical properties, I am able to significantly reduce the size of the state space compared to a unified MDP formulation. This paper …

A Comprehensive And Modular Robotic Control Framework For Model-Less Control Law Development Using Reinforcement Learning For Soft Robotics, Charles Sullivan

Open Access Theses & Dissertations

Soft robotics is a growing field in robotics research. Heavily inspired by biological systems, these robots are made of softer, non-linear, materials such as elastomers and are actuated using several novel methods, from fluidic actuation channels to shape changing materials such as electro-active polymers. Highly non-linear materials make modeling difficult, and sensors are still an area of active research. These issues have rendered typical control and modeling techniques often inadequate for soft robotics. Reinforcement learning is a branch of machine learning that focuses on model-less control by mapping states to actions that maximize a specific reward signal. Reinforcement learning has …

Robot Motion Planning In Dynamic Environments, Hao-Tien Lewis Chiang

Computer Science ETDs

Robot motion planning in dynamic environments is critical for many robotic applications, such as self-driving cars, UAVs and service robots operating in changing environments. However, motion planning in dynamic environments is very challenging as this problem has been shown to be NP-Hard and in PSPACE, even in the simplest case. As a result, the lack of safe, efficient planning solutions for real-world robots is one of the biggest obstacles for ubiquitous adoption of robots in everyday life. Specifically, there are four main challenges facing motion planning in dynamic environments: obstacle motion uncertainty, obstacle interaction, complex robot dynamics and noise, and …

Satisfaction-Aware Data Offloading In Surveillance Systems, Marcos Paul Torres

Electrical and Computer Engineering ETDs

In this thesis, exploiting Fully Autonomous Aerial Systems' (FAAS) and Mobile Edge Computing (MEC) servers' computing capabilities to introduce a novel data offloading framework to support the energy and time-efficient video processing in surveillance systems based on satisfaction games. A surveillance system is introduced consisting of Areas of Interest (AoIs), where a MEC server is associated with each AoI, and a FAAS is flying above the AoIs to support the IP cameras' computing demands. Each IP camera adopts a utility function capturing its Quality of Service (QoS) considering the experienced time and energy overhead to offload and process remotely or …