Robotics Commons^™

Development And Testing Of A Computer Vision Pose Estimation System For Planar Mobile Robots, Andrew Jones

Master's Theses

This thesis details the development and testing of a computer vision-based real-time pose estimation system for differential drive robots. A single camera with a fisheye lens is used to locate ArUco markers placed at fixed locations and attached to robots. Using the OpenCV library, the Perspective-n-Point (PnP) problem is solved to facilitate the transformation of 2-D robot positions in an image to world-frame coordinates. An analytical solution is presented to estimate robot poses based on a single PnP solution, rather than solving the PnP problem for each pose estimate. Pose information is relayed to individual robots using a multi-microcontroller architecture …

Deep Multimodal Sensor Fusion For Autonomous Driving: A Real-World Deployment On Little Blue, Raphael T. Pamie-George

Electrical & Computer Engineering Theses & Dissertations

Accurate perception and control systems are essential for autonomous vehicles to safely interpret and navigate complex environments. Individual sensing modalities such as cameras, LiDAR, and inertial measurement units (IMUs) provide valuable environmental and vehicle state information; however, each sensor exhibits inherent limitations when used independently. Camera performance can degrade under varying lighting conditions, LiDAR measurements may be sparse or affected by environmental noise, and IMU measurements are susceptible to drift over time. Little Blue, a research autonomous vehicle platform from ODU (Old Dominion University), previously relied primarily on camera-based data for neural network training, limiting the system’s ability to exploit …

Robust Real-Time Uav Target Tracking With Onboard Vision-Based Yaw Control, Rylan Malarchick, Jose Castelblanco, Enrique Amaya, Carmen Dimario, Graysen Brinkman, Chirag Kumar, Kiwon Yoon, Sajid Berhane

Beyond: Undergraduate Research Journal

Autonomous tracking of agile unmanned aerial vehicles (UAVs) presents significant challenges for real-time perception and control systems. This work presents AIRHOUND (Autonomous Intelligent Rotorcraft for Hostile Object Unified Navigation and Detection), a UAV platform implementing vision-based yaw tracking through a modular ROS2 software architecture. The system employs YOLOv8 object detection optimized with NVIDIA TensorRT for embedded deployment on an NVIDIA Jetson Orin companion computer. Detected targets are processed through a geometric tracking module that converts pixel coordinates to angular yaw errors using pinhole camera intrinsics, with a proportional controller generating rate-limited yaw commands. These commands are streamed to a PX4 …

Vision-Guided Motion Planning For Autonomous Catch With A Robotic Arm, Kayla Go-Oco, Ashik K. Islam, Raegan Gritzmacher

Electrical Engineering

Modern robotic systems are often evaluated using static and highly controlled tasks, such as pick-and-place demonstrations, which do not fully represent the uncertainty and adaptability required in real-world environments. A major challenge in robotics research is enabling robotic systems to perceive, track, and respond to dynamic objects in real time while maintaining accurate and reliable motion control. This project addresses these challenges by developing an autonomous robotic platform in which an OpenMANIPULATOR-Y robotic arm detects, tracks, and attempts to catch a moving tennis ball using a vision-guided control system. The system integrates YOLO-based object detection with an Intel RealSense D455 …

Using Ai To Predict Energy Expenditure In Lower Limb Prosthesis Users, Nelly Diaz, Siem Hadish

Posters - 2026

• Computer vision has evolved from simple image classification and object detection to analyzing human motion and biomechanics (1). • CNN’s are usually focused on image classification, but, in this case, we are not asking the model if a person is walking. • Many real-world problems require regression: Predicting a continuous number like energy expenditure of walking is a complex task. • It is essential for Prosthetists to understand energy expenditure of their prosthetic patients (2). • An amputee may use 20-30% more energy to walk. • In this project, we developed an AI model to analyze human motion and …

Seeing What Matters: Safety-Critical Semantic Segmentation Via Transfer Learning On Construction Sites, Obiora J. Odugu

LSU Master's Theses

Autonomous robots are increasingly deployed on construction sites for tasks such as progress monitoring, inspection, and safety assessment. For these robots to operate effectively, they must perceive and interpret complex, dynamic environments populated by workers, machinery, and unstructured terrain. Achieving reliable perception depends on high performing semantic segmentation models trained on large volumes of annotated data—an expensive and logistically challenging requirement in construction due to privacy restrictions, variable site access, and slow digitalization. This research addresses the challenge of limited labeled data by investigating transfer learning as a label-efficient approach for construction-site segmentation. Specifically, it explores whether road construction imagery—abundant …

Intelligent Motion Tracking: A Low-Cost Surveillance Framework Using Soc Devices And Sensor Fusion, Aung Myat Khaung, Nicholas Michael Stiffler

Research from the Berry Summer Thesis Institute, 2025

This thesis presents the design and implementation of a lightweight surveillance system capable of realtime motion detection, object tracking, and behavioral history reconstruction in controlled environments. The system uses System-on-Chip devices such as Raspberry Pi boards equipped with NOIR cameras, monocular cameras, and break-beam sensors that work together to detect and track single or multiple moving objects like colored balls. The prototype is validated in structured settings with the goal of eventual deployment in more dynamic environments, addressing the challenge of reliably tracking visually similar objects with minimal distinguishing features. The architecture integrates computer vision with sensor fusion by combining …

Algorithms & Design Behind Autonomous Uavs And Ugvs Coordinated System, Aashish Dhakal

Honors Theses

Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs), when coordinated effectively, offer substantial potential for automating large-scale tasks—from search and rescue operations to precision agriculture. However, synchronizing these autonomous systems remains challenging, especially in time-sensitive missions requiring precision. This thesis investigates the design and algorithmic coordination of autonomous UAVs and UGVs, examining both single-vehicle scenarios and multi-agent (swarming) approaches. Using the Robot Operating System (ROS) as a communication backbone, I integrate GPS positioning with computer vision techniques through OpenCV, enabling accurate localization and object detection. During the development phase, I validate my methods using ArduPilot Software-in-the-Loop (SITL) simulations within …

Pseudo Gps For Romi, Emmanuel Baez, Owen Guinane, Gabriel Coria, Conor Schott

Mechanical Engineering

The Pseudo-GPS system for Romi robots addresses the need for precise real-time location tracking in Cal Poly's Mechatronics lab. This project, developed by Emmanuel Baez, Gabriel Coria, Owen Guinane, and Conor Schott, under the guidance of instructor Charlie Refvem, provides a proof-of-concept system to enhance the Romi robots' geolocation capabilities for advanced robotic algorithms.

The proposed system uses a Raspberry Pi 4 equipped with a Pi camera module and ArUco markers to track the position and orientation of Romi robots within a lab environment. Custom 3D-printed stands secure markers on the robots, and a designated origin marker defines the coordinate …

Robotic Odor Source Localization Using Vision And Olfaction Sensing, Sunzid Hassan

Master's Theses

Robotic Odor Source Localization (ROSL) technology allows autonomous agents like robots to find an odor source in unknown environments. A successful odor source location depends crucially on an effective navigation algorithm that directs the robot towards the odor source. This thesis is a combination of three projects. First, we detail development of a versatile multi-modal robotic platform for ROSL real-world ROSL experimentation and discussed real-world validation of a traditional olfactionbased ROSL algorithm. Secondly, we introduced vision in ROSL by proposing a fusion navigation algorithm that integrates deep-learning enabled vision and olfaction-based navigation. This hybrid approach tackles challenges such as turbulent …

Implementation Of Adas And Autonomy On Unlv Campus, Zillur Rahman

UNLV Theses, Dissertations, Professional Papers, and Capstones

The integration of Advanced Driving Assistance Systems (ADAS) and autonomous driving functionalities into contemporary vehicles has notably surged, driven by the remarkable progress in artificial intelligence (AI). These AI systems, capable of learning from real-world data, now exhibit the capability to perceive their surroundings via a suite of sensors, create optimal routes from source to destination, and execute vehicle control akin to a human driver.

Within the context of this thesis, we undertake a comprehensive exploration of three distinct yet interrelated ADAS and Autonomy projects. Our central objective is the implementation of autonomous driving(AD) technology at UNLV campus, culminating in …

An In-Depth Analysis Of Domain Adaptation In Computer And Robotic Vision, Muhammad Hassan Tanveer, Zainab Fatima, Shehnila Zardari, David A. Guerra-Zubiaga

Faculty Articles

This review article comprehensively delves into the rapidly evolving field of domain adaptation in computer and robotic vision. It offers a detailed technical analysis of the opportunities and challenges associated with this topic. Domain adaptation methods play a pivotal role in facilitating seamless knowledge transfer and enhancing the generalization capabilities of computer and robotic vision systems. Our methodology involves systematic data collection and preparation, followed by the application of diverse assessment metrics to evaluate the efficacy of domain adaptation strategies. This study assesses the effectiveness and versatility of conventional, deep learning-based, and hybrid domain adaptation techniques within the domains of …

Autonomous Shipwreck Detection & Mapping, William Ard

LSU Master's Theses

This thesis presents the development and testing of Bruce, a low-cost hybrid Remote Operated Vehicle (ROV) / Autonomous Underwater Vehicle (AUV) system for the optical survey of marine archaeological sites, as well as a novel sonar image augmentation strategy for semantic segmentation of shipwrecks. This approach takes side-scan sonar and bathymetry data collected using an EdgeTech 2205 AUV sensor integrated with an Harris Iver3, and generates augmented image data to be used for the semantic segmentation of shipwrecks. It is shown that, due to the feature enhancement capabilities of the proposed shipwreck detection strategy, correctly identified areas have a 15% …

Human-Robot Collaboration Enabled By Real-Time Vision Tracking, Travis Deegan

Electronic Theses and Dissertations

The number of robotic systems in the world is growing rapidly. However, most industrial robots are isolated in caged environments for the safety of users. There is an urgent need for human-in-the-loop collaborative robotic systems since robots are very good at performing precise and repetitive tasks but lack the cognitive ability and soft skills of humans. To fill this need, a key challenge is how to enable a robot to interpret its human co-worker’s motion and intention. This research addresses this challenge by developing a collaborative human-robot interface via innovations in computer vision, robotics, and system integration techniques. Specifically, this …

American Sign Language Recognition Using Machine Learning And Computer Vision, Kshitij Bantupalli, Ying Xie

Master of Science in Computer Science Theses

Speech impairment is a disability which affects an individual’s ability to communicate using speech and hearing. People who are affected by this use other media of communication such as sign language. Although sign language is ubiquitous in recent times, there remains a challenge for non-sign language speakers to communicate with sign language speakers or signers. With recent advances in deep learning and computer vision there has been promising progress in the fields of motion and gesture recognition using deep learning and computer vision-based techniques. The focus of this work is to create a vision-based application which offers sign language translation …

Enhancing 3d Visual Odometry With Single-Camera Stereo Omnidirectional Systems, Carlos A. Jaramillo

Dissertations, Theses, and Capstone Projects

We explore low-cost solutions for efficiently improving the 3D pose estimation problem of a single camera moving in an unfamiliar environment. The visual odometry (VO) task -- as it is called when using computer vision to estimate egomotion -- is of particular interest to mobile robots as well as humans with visual impairments. The payload capacity of small robots like micro-aerial vehicles (drones) requires the use of portable perception equipment, which is constrained by size, weight, energy consumption, and processing power. Using a single camera as the passive sensor for the VO task satisfies these requirements, and it motivates the …

Baseball Shagger, Anthony Velasquez, Nick Walker, Nathaniel Kai Paresa

Computer Engineering

The purpose of our project is to allow players to hit baseballs on a baseball field and not have to worry about picking them up. By combining our knowledge of software and hardware, we developed the first design of a robot that “shags” baseballs. Our endeavor was only partially successful. The device was tested on grass, turf, and concrete. The motors did not have enough torque to get moving on grass. The device faired better on turf where it could move, but was quite jerky as the motor drive needed to be high to start moving, but once it was …

Control Of A Powered Ankle-Foot Prosthesis: From Perception To Impedance Modulation, Guilherme Aramizo Ribeiro

Dissertations, Master's Theses and Master's Reports

Active ankle prostheses controllers are demonstrating gaining smart features to improve the safety and comfort offor users. The perception of user intention to modulate the ankle dynamics is a well-known example of such feature. But not much work focused on the perception of the environment, nor how the environment should be included in the mechanical design and control of the prosthesisprostheses. The proposed work aims to improve the feasibility of integrate the environment perception integration intoto the prostheses controllersler, and to define the desired ankle dynamics, as mechanical impedance, duringof the human walk on different environmental settings. As a preliminary …

Vision-Based Motion For A Humanoid Robot, Khalid Abdullah Alkhulayfi

Dissertations and Theses

The overall objective of this thesis is to build an integrated, inexpensive, human-sized humanoid robot from scratch that looks and behaves like a human. More specifically, my goal is to build an android robot called Marie Curie robot that can act like a human actor in the Portland Cyber Theater in the play Quantum Debate with a known script of every robot behavior. In order to achieve this goal, the humanoid robot need to has degrees of freedom (DOF) similar to human DOFs. Each part of the Curie robot was built to achieve the goal of building a complete humanoid …

Senior Project: Control System For An Underwater Remotely Operated Vehicle, Tyler Mau, Joseph Mahoney

Computer Engineering

No abstract provided.

Learning In Vision And Robotics, Daniel P. Barrett

Open Access Dissertations

I present my work on learning from video and robotic input. This is an important problem, with numerous potential applications. The use of machine learning makes it possible to obtain models which can handle noise and variation without explicitly programming them. It also raises the possibility of robots which can interact more seamlessly with humans rather than only exhibiting hard-coded behaviors. I will present my work in two areas: video action recognition, and robot navigation. First, I present a video action recognition method which represents actions in video by sequences of retinotopic appearance and motion detectors, learns such models automatically …

Dynamic Surgical Tool Tracking And Delivery System Using Baxter Robot, Sthitapragyan Parida, Juan Pablo Wachs, Maria Eugenia Cabrera

The Summer Undergraduate Research Fellowship (SURF) Symposium

Several hospitals face nurse staffing shortages for surgeries. This research focuses on building a system with Baxter robot capable of identifying surgical tools using computer vision and delivering them to the surgeon on demand. This would deal with the issue of nurse unavailability during simple surgical procedures. The key aspects of the project were: testing the accuracies of various Artificial Neural Networks (ANNs) in classifying surgical instruments, and programming Baxter to implement a surgical tool delivery system using magnets at the tip of its 7-DOF robotic arms. The methodology consisted of, first, implementing algorithms to enable Baxter to do pick …

Reasoning Across Language And Vision In Machines And Humans, Andrei Barbu

Open Access Dissertations

Humans not only outperform AI and computer-vision systems, but use an unknown computational mechanism to perform tasks for which no suitable approaches exist. I present work investigating both novel tasks and how humans approach them in the context of computer vision and linguistics. I demonstrate a system which, like children, acquires high-level linguistic knowledge about the world. Robots learn to play physically-instantiated board games and use that knowledge to engage in physical play. To further integrate language and vision I develop an approach which produces rich sentential descriptions of events depicted in videos. I then show how to simultaneously detect …

Mobile Robot Navigation For Person Following In Indoor Environments, Ninad Pradhan

All Dissertations

Service robotics is a rapidly growing area of interest in robotics research. Service robots inhabit human-populated environments and carry out specific tasks. The goal of this dissertation is to develop a service robot capable of following a human leader around populated indoor environments. A classification system for person followers is proposed such that it clearly defines the expected interaction between the leader and the robotic follower. In populated environments, the robot needs to be able to detect and identify its leader and track the leader through occlusions, a common characteristic of populated spaces. An appearance-based person descriptor, which augments the …

An Approach To Stereo-Point Cloud Registration Using Image Homographies, Damian M. Lyons, Stephen D. Fox

Faculty Publications

No abstract provided.

Autonomous Golf Cart Vision Using Hsv Image Processing And Commercial Webcam, John D. Fulton

Electrical Engineering

Using openCV I was able to use a store bought webcam and my laptop to build code that could detect the lane lines and the curb on the road and use them to calculate a trajectory for an autonomous vehicle to follow. The code does have some flaws, as discovered in testing, and definitely room for improvement but it still functions as a useful basis for development. Developed primarily for the Autonomous Golf Car Project at Cal Poly the principles of how it functions could be applied to other projects that want to use computer vision.

A Relaxed Fusion Of Information From Real And Synthetic Images To Predict Complex Behavior, Damian M. Lyons, D. Paul Benjamin

Faculty Publications

An important component of cognitive robotics is the ability to mentally simulate physical processes and to compare the expected results with the information reported by a robot's sensors. In previous work, we have proposed an approach that integrates a 3D game-engine simulation into the robot control architecture. A key part of that architecture is the Match-Mediated Difference (MMD) operation, an approach to fusing sensory data and synthetic predictions at the image level. The MMD operation insists that simulated and predicted scenes are similar in terms of the appearance of the objects in the scene. This is an overly restrictive constraint …

Integrating Perception And Problem Solving To Predict Complex Object Behaviors, Damian M. Lyons, Sirhan Chaudhry, Marius Agica, John Vincent Monaco

Faculty Publications

One of the objectives of Cognitive Robotics is to construct robot systems that can be directed to achieve realworld goals by high-level directions rather than complex, low-level robot programming. Such a system must have the ability to represent, problem-solve and learn about its environment as well as communicate with other agents. In previous work, we have proposed ADAPT, a Cognitive Architecture that views perception as top-down and goaloriented and part of the problem solving process.

Our approach is linked to a SOAR-based problem-solving and learning framework. In this paper, we present an architecture for the perceptive and world modelling components …

Synchronizing Real And Predicted Synthetic Video Imagery For Localization Of A Robot To A 3d Environment, Damian M. Lyons, Sirhan Chaudhry, D. Paul Benjamin

Faculty Publications

A mobile robot moving in an environment in which there are other moving objects and active agents, some of which may represent threats and some of which may represent collaborators, needs to be able to reason about the potential future behaviors of those objects and agents. In previous work, we presented an approach to tracking targets with complex behavior, leveraging a 3D simulation engine to generate predicted imagery and comparing that against real imagery. We introduced an approach to compare real and simulated imagery using an affine image transformation that maps the real scene to the synthetic scene in a …

Electronic Image Stabilization For Mobile Robotic Vision Systems, Michael John Smith

Theses and Dissertations

When a camera is affixed on a dynamic mobile robot, image stabilization is the first step towards more complex analysis on the video feed. This thesis presents a novel electronic image stabilization (EIS) algorithm for small inexpensive highly dynamic mobile robotic platforms with onboard camera systems. The algorithm combines optical flow motion parameter estimation with angular rate data provided by a strapdown inertial measurement unit (IMU). A discrete Kalman filter in feedforward configuration is used for optimal fusion of the two data sources. Performance evaluations are conducted by a simulated video truth model (capturing the effects of image translation, rotation, …