Computer Sciences Commons^™

Keyframe Selection From Motion Capture Data With Dual-Agent Reinforcement Learning, Kun Hu, Wang, Clinton Mo, Mingyang Ma, Shaohui Mei, Zebin Chen, Zhiyong Wang

Research outputs 2022 to 2026

Animation production workflows centered around motion capture techniques require animators to edit motions based on a set of keyframes. However, most existing keyframe selection methods are optimization-based, which suffer from the issues of flexibility and efficiency. In this paper, a novel deep reinforcement learning method with dual agents are proposed for unsupervised keyframe selection. First, an S-Agent and an R-Agent evaluate the actions of selection and refinement, respectively. A deep spatio-temporal network, namely graph keyframe evaluation network (GKEN), is proposed for the agents. Then, an animation specified reward is devised based on reconstruction, which fulfills three important properties of the …

A Statistical Mechanics Approach To Reinforcement Learning, Jacob Adamczyk

Graduate Doctoral Dissertations

Reinforcement learning (RL), the study of optimal decision-making over long timescales in stochastic systems, has recently seen remarkable advances due in large part to the efforts of the deep learning community. RL has witnessed great success in solving problems in video games, robotics, biological control, and language modeling. However, a unified statistical mechanics framework to understand and develop the corresponding algorithms is lacking. To address this issue, we begin by showing that the reinforcement learning problem can be formulated and solved using the tools of statistical mechanics. Drawing on physical principles of free energy minimization and invariance, we address important …

Assessing Flaws In Captcha Security Through Progress In Ai, Jaydon Stanislowski

Scholarly Horizons: University of Minnesota, Morris Undergraduate Journal

Protecting the internet from the threat of malicious bot activity is an important problem as AI tools become more powerful and commonplace over time. To that end, security measures are employed across websites in the form of CAPTCHAs, short challenges designed to identify and block fake web traffic. Yet, they become less effective over time as AI becomes more powerful, and thus more capable of solving them. This paper examines recent research on the threat to CAPTCHA security posed by current AI models and how this security can be reinforced over time, focusing primarily on Google’s reCAPTCHA v3.

Simulation Study Of Elevator Group Control Scheduling Based On Real-Time Occupancy Perception, Yiyong Han, Shiyu Wang, Yang Ye, Zhen Zhang, Fengque Pei, Minghai Yuan

Journal of System Simulation

Abstract: To address the conflict between car space allocation and peak passenger flow response efficiency in elevator group control scheduling, a multi-objective scheduling method based on proximal policy optimization (PPO) with real-time occupancy perception was proposed. A simulation environment considering car capacity constraints was constructed, and a reward-penalty mechanism with average passenger waiting time, system energy consumption, and car congestion as optimization objectives was designed. Based on this, state and action spaces were defined to form a PPO-based scheduling framework; a simulation platform integrating traffic flow visualization, policy scheduling, and performance evaluation was developed. Simulation results show that this method …

Research On Output Feedback Control Based On Reinforcement Learning For Overhead Crane, Minghui Li, Daoxiang Gao

Journal of System Simulation

An output feedback control algorithm is designed based on reinforcement learning for the optimal control problem of overhead crane system. A high gain observer (HGO) is designed using output data to estimate the unmeasurable states of the overhead crane system. Based on the estimated states from the high-gain observer, a policy iteration (PI) method is designed with integral reinforcement learning, which uses Critic and Actor neural networks to approximate the optimal value function and control strategy, and adjusts the neural network weights in real time through online adaptive algorithms. According to the Lyapunov stability theory, the uniform ultimate boundedness of …

Improved Nsga-Ii For Dual-Resource Flexible Job Shop Scheduling Considering Worker Load, Guohui Zhang, Yuan Ren, Changjun Wu, Xiaofei Kou

Journal of System Simulation

For the dual-resource-constrained flexible job shop scheduling problem considering worker load, an evolutionary algorithm integrating reinforcement learning was proposed. A three-stage encoding conforming to the problem characteristics was designed, and three initialization methods were combined to improve the population quality; a left-insertion decoding method based on worker load was designed to ensure that the completion time of the operation is less than the maximum processable time of the worker on the current day; two neighborhood structures based on the critical path were constructed to enhance the local exploration ability of the population; reinforcement learning was integrated to enable the …

Perceptual And Geometric Advances In Crowd Simulation, Bilas Talukdar

Dissertations

Simulating realistic crowd motion remains a fundamental challenge in computer graphics and multi-agent systems, as it requires modeling both physically plausible interactions and perceptually natural behaviors. Existing crowd simulation methods typically employ simplified geometric abstractions, most commonly circular agent representations, and model navigation using either analytical interaction formulations (e.g., force, velocity, or constraint-based methods) or learned policies derived through reinforcement learning. Despite their effectiveness, these approaches often overlook detailed geometric structure and do not explicitly account for perceptual realism. This dissertation addresses these challenges by improving the realism of virtual crowd simulation through two key advancements: perceptual preference learning and …

Shaping Emergent Competitive And Cooperative Behaviors In Multi-Agent General-Sum Games, Ethan F. Erickson

Honors Projects

Reinforcement learning (RL) algorithms can train agents to solve problems in environments using complex behaviors that are not explicitly programmed, known as emergent behaviors. The goal of our research is to investigate how different RL reward values influence the emergence of competitive and cooperative behaviors in games with teams of multiple agents. Specifically, we focus on general-sum games, in which the sum of gains and losses of each team may be non-zero, allowing situations for agents to mutually benefit or mutually fail. Using Unity’s ML-Agents Toolkit to train agents with RL self-play in bounded 2D environments, we identify high-level behaviors …

Review On Optimization Of Simulation Modeling Strategies For Spacecraft Orbit Avoidance, Guozheng Li, Rui Wang, Shichao Fan, Xintong Cai, Xinyue Zhai

Journal of System Simulation

Abstract: The number of on-orbit spacecraft increases exponentially; the space environment becomes more complex, and the collision risk of on-orbit spacecraft increases significantly. On-orbit safety is thus severely threatened, posing higher requirements for orbit avoidance methods. The costs and risks of space activities are extremely high, making simulation an effective method to solve complex problems of orbit avoidance. The modeling, solution, and simulation methods for the two core issues of spacecraft orbit avoidance, "collision avoidance" and "pursuit-evasion games", were systematically reviewed, and the existing shortcomings were analyzed. The applications of technologies such as deep reinforcement learning in promoting orbit avoidance …

Robot Path Planning By Reinforcement Learning Based On Sac3q-Hdm, Dequan Li, Wan Xiong

Journal of System Simulation

Abstract: To address the issues of overestimated and underestimated biases, low sample utilization rate, and the inability to balance exploration and exploitation in reinforcement learning for path planning, an improved SAC method was proposed. The size balance of entropy was explored and utilized through adaptive temperature coefficient adjustment; on the basis of the SAC framework, a triple Critic architecture was introduced to dynamically weight and fuse the minimum and average values through Qvalue uncertainty, balancing overestimated and underestimated biases. A mixed dynamic sampling experience replay buffer was designed; experience data was partitioned based on reward thresholds; sampling ratios were dynamically …

Multiple Q-Dqn Algorithm Based Rumor Control In Social Networks, Zainab Hasan, Huda Naji Nawaf

Journal of Intelligent Informatics, Networking, and Cybersecurity

Malicious rumours on social media platforms like Facebook, Twitter, and others can be widely disseminated because of social problems. It is challenging to manage a rumour once it gains growth and quickly moves throughout a network. One of the main issues with information dissemination is figuring out how to reduce the propagation of rumours within a social network. A deep reinforcement learning technique might be an effective strategy to manage the rumour issue. Deep Q learning network (DQN) has been used in the literature to mitigate rumors by selecting a blocker at each time step. In this work, the proposed …

Learning To Search For Vehicle Routing With Multiple Time Windows, Kuan Xu, Zhiguang Cao, Chenlong Zheng, Lindong Liu

Research Collection School Of Computing and Information Systems

In this study, we propose a reinforcement learning-based adaptive variable neighborhood search (RL-AVNS) method designed for effectively solving the Vehicle Routing Problem with Multiple Time Windows (VRPMTW). Unlike traditional adaptive approaches that rely solely on historical operator performance, our method integrates a reinforcement learning framework to dynamically select neighborhood operators based on real-time solution states and learned experience. We introduce a fitness metric that quantifies customers’ temporal flexibility to improve the shaking phase, and employ a transformer-based neural policy network to intelligently guide operator selection during the local search. Extensive computational experiments are conducted on realistic scenarios derived from the …

Reinforcement Learning Based Method For Uav Team Orienteering Optimization Under Multi-Constraint Condition, Can Yang, Kai Chen, Feng Zhu

Journal of System Simulation

Abstract: Traditional optimization methods struggle with efficiency, while reinforcement learning approaches often yield low solution quality and high training costs. In response, this paper proposes an attention mechanism-based reinforcement learning method. A dynamic attention strategy network with multi-information fusion is designed to improve solution quality. A visibility-graph approach is employed to simplify threat zone constraints and speed up convergence, and a decoding sequence reordering mechanism is introduced for further performance optimization of the solution. The simulation results show that the method generates high-quality solutions within milliseconds, achieving total rewards that approach or even surpass those obtained by traditional solvers …

Aura: An Ai-Powered Multimodal Prototype For Adaptive Apraxia Of Speech Therapy And Communication Support, Omotayo Omoyemi, Rachel K. Johnson

Speech-Language Pathology Faculty Publications

Apraxia of Speech (AOS) is a motor speech disorder that significantly limits communication and requires intensive, long-term therapy. Access to consistent treatment is often constrained by shortages of speech-language pathologists, high costs, and limited opportunities for continuous monitoring outside clinical settings. Recent advances in Artificial Intelligence (AI) provide new opportunities to support scalable and personalized speech therapy.

This paper presents AURA (Adaptive Understanding and Relearning Assistant for Apraxia), a multimodal AI framework designed to support speech therapy, progress monitoring, and communication for individuals with AOS. The system integrates speech analysis, machine learning–based error detection, reinforcement learning for adaptive therapy, and …

Reinforcement Learning-Enabled Control And Design Of Rigid-Link Robotic Fish: A Comprehensive Review, Nhat Dinh, Darion Vosbein, Yuehua Wang, Qingsong Cui

Faculty Publications

With the rising demand for maritime surveys of infrastructure, energy resources, and environmental conditions, autonomous robotic fish have emerged as a promising solution with their biomimetic propulsion, agile motion, efficiency, and capacity for underwater inspection, monitoring, data collection, and exploration tasks in complex aquatic environments. Inspired by fish spines, rigid-link fish robots (RLFRs), a category of robotic fish, are widely utilized in robotics research and applications. Their rigid, actuated joints enable them to reproduce the undulatory locomotion and high maneuverability of biological fishes, while the modular nature of rigid links between joints makes them cost-effective and easy to assemble. This …

Explainable Physics-Based Constraints On Reinforcement Learning For Accelerator Optimization, Jonathan Colen, Malachi Schram, Kishansingh Rajput, Armen Kasparian

Data Science Faculty Publications

We present a reinforcement learning (RL) framework for optimizing particle accelerator experiments that builds explainable physics-based constraints on agent behavior. The goal is to increase transparency and trust by letting users verify that the agent’s decision-making process incorporates suitable physics. Our algorithm uses a learnable surrogate function for physical observables, such as energy, and uses them to fine-tune how actions are chosen. This surrogate can be represented by a neural network or by an interpretable sparse dictionary model. We test our algorithm on a range of particle accelerator optimization environments designed to emulate the Continuous Electron Beam Accelerator Facility at …

Self-Calibrating Uav Navigation: Reinforcement Learning Approaches For Horizontal Trajectory Estimation, Shirin Nasr-Esfahani, S. Jagannathan

Electrical and Computer Engineering Faculty Research & Creative Works

Accurate unmanned aerial vehicle (UAV) trajectory estimation is essential for autonomous navigation, particularly in GPS-denied environments. Visualodometry and simultaneous localization and mapping (SLAM) approaches require precise camera intrinsic parameters, which are typically obtained through predefined or offline calibration. Instead, in this work, we propose a reinforcement learning (RL)-based self-calibration framework that estimates camera intrinsic parameters directly from monocular video sequences, without requiring prior knowledge of the camera, environment, or calibration targets. This intrinsic parameter estimation is then leveraged to achieve robust UAV trajectory estimation using only video data. We formulate the problem as a sequential decision-making task, where an RL …

ℵ-Ipomdp: Mitigating Deception In A Cognitive Hierarchy With Off-Policy Counterfactual Anomaly Detection, Nitay Alon, Joseph M. Barnby, Stefan Sarkadi, Lion Schulz, Jeffrey S. Rosenschein, Peter Dayan

Research outputs 2022 to 2026

Social agents with finitely nested opponent models are vulnerable to manipulation by agents with deeper recursive capabilities. This imbalance, rooted in logic and the theory of recursive modelling frameworks, cannot be solved directly. We propose a computational framework called ℵ-IPOMDP, which augments the Bayesian inference of model-based RL agents with an anomaly detection algorithm and an out-of-belief policy. Our mechanism allows agents to realize that they are being deceived, even if they cannot understand how, and to deter opponents via a credible threat. We test this framework in both a mixed-motive and a zero-sum game. Our results demonstrate the ℵ-mechanism’s …

Online Lifelong Optimal Adaptive Control Of Partially Uncertain Strict Feedback Discrete-Time Systems With Application To Quadrotor Uavs, Maxwell Geiger, Sarangapani Jagannathan

Electrical and Computer Engineering Faculty Research & Creative Works

This article considers the infinite time horizon optimal adaptive tracking control of partially uncertain strict feedback discrete-time (DT) systems with application to quadrotor uncrewed aerial vehicles (UAVs). First, the strict feedback DT system is transformed into an equivalent affine nonlinear DT system in terms of the tracking error dynamics. The optimal adaptive tracking control problem is solved using an augmented system approach, where a horizon of future bounded reference trajectory points is used in the augmented state, when compared to using a single point. It is assumed that the internal dynamics of the strict feedback system are unknown, but the …

Design And Analysis Of Modern Quantum Neural Network Architectures For Intelligent Systems, Lakshmi Chandrakanth Kasireddy, Prabhakara Rao Kapula, Dineshkumar Rajendran, Neha Bharani, Srikanth Pulipeti, Islombek Khushvaktov

Computer Science Faculty Publications

Quantum neural networks (QNNs) offer a principled pathway for integrating quantum computation with machine learning through superposition- and entanglement-based representations. This chapter proposes an architecture-aware design and evaluation framework for modern QNNs, emphasizing robustness and system feasibility alongside predictive performance. Multiple architectures variational QNNs, quantum convolutional neural networks, tensor-network hybrids, and fully quantum models—are assessed under a unified protocol. Experimental analysis shows that the proposed architecture-search–guided QNN achieves 91.8% classification accuracy and an F1-score of 0.914, outperforming fixed-template variational QNNs by approximately 5.6 percentage points. Under depolarizing noise with probability p = 0.10, the proposed model retains 85.3% accuracy, whereas …

Online Decision Mamba, Trenton W. Ruf

Dissertations and Theses

Online in-context reinforcement learning enhances offline-trained policies through online fine-tuning. We introduce Online Decision Mamba (ODM), an architecture that replaces the attention mechanism in Online Decision Transformers (ODT) with the Mamba module to improve long-context sequence modeling and overall RL performance. We performed in-depth evaluations on MuJoCo (OpenAI Gym) and Atari benchmarks, comparing ODM against state-of-the-art offline and online baselines—including Decision Mamba (DM) and ODT. Our results show that ODM achieves competitive or superior performance, with particularly robust gains when initial datasets lack expert demonstrations. In the Qbert Atari environment, ODM shows context-length sensitivity similar to offline DM; however, we …

Large Language Models As End-To-End Combinatorial Optimization Solvers, Xia Jiang, Yaoxin Wu, Minshuo Li, Zhiguang Cao, Yingqian Zhang

Research Collection School Of Computing and Information Systems

Combinatorial optimization (CO) problems, central to decision-making scenarios like logistics and manufacturing, are traditionally solved using problem-specific algorithms requiring significant domain expertise. While large language models (LLMs) have shown promise in automating CO problem solving, existing approaches rely on intermediate steps such as code generation or solver invocation, limiting their generality and accessibility. This paper introduces a novel framework that empowers LLMs to serve as end-to-end CO solvers by directly mapping natural language problem descriptions to solutions. We propose a two-stage training strategy: supervised fine-tuning (SFT) imparts LLMs with solution generation patterns from domain-specific solvers, while a feasibility-and-optimality-aware reinforcement learning …

Reinforcement Learning Based Intelligent Optimisation For Bin Packing Problems: A Review, Nadia Dahmani, Amril Nazir, Ikbal Taleb, Syed M.Salman Bukhari

All Works

The convergence of Reinforcement Learning (RL) and Bin Packing Problems (BPP) is a critical field of study that has profound ramifications in logistics, manufacturing, computer, and retail industries. This paper thoroughly examines the progression from simple rule-based tactics to advanced Deep Reinforcement Learning (DRL) techniques in solving BPPs. By conducting a thorough review of 231 papers conducted between 2019 and 2024, we address and provide answers to important research inquiries, such as “To what extent has academic research explored the use of RL for BPP during this time frame?” and “Which specific areas of application and methodologies have been predominantly …

Multi-Task Vehicle Routing Solver Via Mixture Of Specialized Experts Under State-Decomposable Mdp, Yuxin Pan, Zhiguang Cao, Chengyang Gu, Liu Liu, Peilin Zhao, Yize Chen, Fangzhen Lin

Research Collection School Of Computing and Information Systems

Existing neural methods for multi-task vehicle routing problems (VRPs) typically learn unified solvers to handle multiple constraints simultaneously. However, they often underutilize the compositional structure of VRP variants, each derivable from a common set of basis VRP variants. This critical oversight causes unified solvers to miss out the potential benefits of basis solvers, each specialized for a basis VRP variant. To overcome this limitation, we propose a framework that enables unified solvers to perceive the shared-component nature across VRP variants by proactively reusing basis solvers, while mitigating the exponential growth of trained neural solvers. Specifically, we introduce a State-Decomposable MDP …

Auv Path Planning Based On Behavior Cloning And Improved Dqn In Partially Unknown Environments, Lijing Xing, Min Li, Xiangguang Zeng, Ping Zhang, Bei Peng

Journal of System Simulation

Abstract: To address the problems of large randomness and slow convergence of the DQN dynamic path planning algorithm for a single autonomous underwater vehicle (AUV) in a partially unknown environment, a path planning method combining behavior cloning with A* algorithm and DQN (BA_DQN) was proposed. Based on the known environmental information, an improved A* algorithm incorporating ocean current resistance was proposed to guide DQN, thereby reducing the randomness of the DQN algorithm. By considering the complexity of the marine environment, the sampling probability was improved again after expanding the positive experience pool to enhance the training success rate. To address …

Modular Architecture For High-Performance And Low Overhead Data Transfers, Rasman Mubtasim Swargo, Engin Arslan, Md Arifuzzaman

Computer Science Faculty Research & Creative Works

High-performance applications necessitate rapid and dependable transfer of massive datasets across geographically dispersed locations. Traditional file transfer tools often suffer from resource underutilization and instability due to fixed configurations or monolithic optimization methods. We propose AutoMDT, a novel Modular Data Transfer Architecture, to address these issues by employing a deep reinforcement learning agent to simultaneously optimize concurrency levels for read, network, and write operations. This solution incorporates a lightweight network-system simulator, enabling offline training of a Proximal Policy Optimization (PPO) agent in approximately 45 minutes on average, thereby overcoming the impracticality of lengthy online training in production networks. AutoMDT's modular …

Modeling And Simulation Of Traffic Signal Control Based On Mlp With Improved Gcn-Td3, Deqi Huang, Yating Tu, Zhenhua Zhang, Xin Guo

Journal of System Simulation

Abstract: To address the issues of uneven traffic flow at urban intersections, limited road capacity, and the poor coordination of existing traffic signal control algorithms, a traffic signal control algorithm based on graph convolutional reinforcement learning was proposed. By utilizing a multilayer perceptron, the dynamic features of vehicles and phase information at the controlled intersection and its neighboring intersections were extracted. A graph convolutional neural network was then employed to aggregate these vehicle dynamic features into potential features representing regional traffic. The control strategy was derived through multiple iterations of an improved twin delayed deep deterministic policy gradient (TD3) algorithm. …

Digging Deeper With Deep Ram Networks, Andrew J. Wagner

Dissertations and Theses

While Deep Neural Networks (DNNs) have driven major breakthroughs in artificial intelligence, their internal complexity often makes their behavior hard to explain, resulting in the well-known “black box” dilemma. This thesis addresses the challenge of interpretability in DNNs and deep reinforcement learning (DRL) through two main contributions.

In Part I, we revisit and extend the use of Deep RAM Networks (DRNs) within the Arcade Learning Environment (ALE), showing that, with modern architectures and careful hyperparameter tuning, RAM-based agents can achieve performance competitive with established pixel-based baselines on Atari 2600 games, while offering additional advantages for research and analysis. We also …

Robust Ai Solutions For Financial Markets Through Generative Modeling, Dynamic Graph Learning, And Reinforcement-Based Portfolio Optimization, Jingyi Gu

Dissertations

Financial markets are inherently uncertain and dynamic, driven by complex factors such as macroeconomic signals, investor sentiment, and evolving inter-asset relationships. While machine learning has advanced financial modeling, existing approaches often fall short in addressing the real-world intricacies of finance. This dissertation confronts two critical challenges, human-driven stochasticity and risk-intensive decision-making under real-world trading constraints, while seizing a pivotal opportunity, the structural dynamics of evolving financial systems. These elements are foundational to advancing robust and practical financial intelligence.

To this end, this dissertation develops a unified framework for robust financial modeling and decision-making. The framework is architected as a progressive, …

Rl4co: An Extensive Reinforcement Learning For Combinatorial Optimization Benchmark, Federico Berto, Et. Al

Research Collection School Of Computing and Information Systems

Combinatorial optimization (CO) is fundamental to several real-world applications, from logistics and scheduling to hardware design and resource allocation. Deep reinforcement learning (RL) has recently shown significant benefits in solving CO problems, reducing reliance on domain expertise and improving computational efficiency. However, the absence of a unified benchmarking framework leads to inconsistent evaluations, limits reproducibility, and increases engineering overhead, raising barriers to adoption for new researchers. To address these challenges, we introduce RL4CO, a unified and extensive benchmark with in-depth library coverage of 27 CO problem environments and 23 state-of-the-art baselines. Built on efficient software libraries and best practices in …