Robotics Commons^™

Development And Testing Of A Computer Vision Pose Estimation System For Planar Mobile Robots, Andrew Jones

Master's Theses

This thesis details the development and testing of a computer vision-based real-time pose estimation system for differential drive robots. A single camera with a fisheye lens is used to locate ArUco markers placed at fixed locations and attached to robots. Using the OpenCV library, the Perspective-n-Point (PnP) problem is solved to facilitate the transformation of 2-D robot positions in an image to world-frame coordinates. An analytical solution is presented to estimate robot poses based on a single PnP solution, rather than solving the PnP problem for each pose estimate. Pose information is relayed to individual robots using a multi-microcontroller architecture …

Evaluation Of General Voronoi Diagram Decomposition For Harmonic Fields In Navigation Of Dynamic Environments, Franco Abullarade

Master's Theses

Harmonic potential fields provide provably minimum-free navigation, but any change to the workspace geometry invalidates the field and forces a costly global recomputation, typically restricting them to static environments. This thesis extends the harmonic map framework of Vlantis et al., which maps the free workspace onto a unit disk and uses an atlas of per-region transformations, to dynamic indoor settings. First, we replace their manually annotated room partition with an automatic decomposition based on the Generalized Voronoi Diagram, allowing the atlas to be built from an arbitrary occupancy grid in an automated way. Second, we introduce a localized repair procedure …

Vision-Guided Motion Planning For Autonomous Catch With A Robotic Arm, Kayla Go-Oco, Ashik K. Islam, Raegan Gritzmacher

Electrical Engineering

Modern robotic systems are often evaluated using static and highly controlled tasks, such as pick-and-place demonstrations, which do not fully represent the uncertainty and adaptability required in real-world environments. A major challenge in robotics research is enabling robotic systems to perceive, track, and respond to dynamic objects in real time while maintaining accurate and reliable motion control. This project addresses these challenges by developing an autonomous robotic platform in which an OpenMANIPULATOR-Y robotic arm detects, tracks, and attempts to catch a moving tennis ball using a vision-guided control system. The system integrates YOLO-based object detection with an Intel RealSense D455 …

Autonomous Vision-Based Litter Collection Rover, Dante Michael Benedetti, Benjamin Scott Tavares, Nathan Heil

Electrical Engineering

This report documents the design, implementation, and testing of an autonomous litter-collection rover developed as a Senior Project Design Lab (EE 460/463/464) at California Polytechnic State University. The rover integrates autonomy, computer vision, embedded real-time control, mecanum-wheel omnidirectional mobility, and a two-degree-of-freedom robotic arm to detect, approach, and collect small ground-level litter such as bottles, wrappers, and paper fragments.

The system uses a two-layer compute architecture: an NVIDIA Jetson Orin Nano running ROS 2 for perception, SLAM, and path planning, paired with an STM32L4A6ZG microcontroller for real-time motor control and odometry. The robot is built on a multi-level aluminum frame …

Integration Of Robotic Arm And Conveyor With Programmable Logic Controller, Drake A. Small

Master's Theses

To meet new curriculum demands brought on by Cal Poly's upcoming switch to semesters, a new, cross-disciplinary lab module was developed for EE 435 (Industrial Power Control and Automation). The module emphasizes career-applicable skills, preparing students for the field of controls engineering within the manufacturing industry. These skills include robotic control, computer networking and configuration, and embedded systems. The work focused on integrating a collaborative robot arm to pick-and-place boxes on a conveyor. A myCobot 320 Pi and an Ultimation Powered Roller MDR Conveyor were integrated with the existing PLC system using Modbus RTU and EtherNet/IP communication protocols, respectively. A …

Design And Parametric Study Of A Mems-Based Reservoir Computer For Reinforcement Learning, Andrew P. Carr

Master's Theses

Single-node reservoir computing (RC) is a hardware-efficient approach to machine learning, leveraging the dynamics of physical systems. In this work, two reinforcement learning algorithms, Q-learning and Proximal Policy Optimization (PPO), are applied to a simulated micro-electro-mechanical system (MEMS)-based reservoir computer to solve both discrete and continuous control tasks. MEMS-based reservoirs are low-power, compact, and their natural frequencies (kHz to MHz) pair well with real-time control loops. To explore the relationship between reservoir dynamics and learning performance, a parametric study is conducted on two reservoir hyperparameters, reservoir size and neuron separation, using CartPole-v1 and MountainCar-v0. The RC successfully learns multiple tasks …

Stretch Web Teleoperation Keyboard Final Report, Aaron Chou, Francisco Irazaba

Computer Engineering

This project added keyboard controls to the Hello Robot Stretch Web Teleop interface. The goal was to make the robot easier to control from a web browser without needing a gamepad or separate terminal-based keyboard program. The new interface allows users to control different parts of the robot with customizable key bindings.

Hallucination Techniques For Self-Supervised Synthetic Datasets For Mobile Robots, Wyatt D. Colburn

Master's Theses

Classical techniques in autonomous navigation struggle in tightly constrained spaces. Machine learning has been shown to perform better in these difficult environments but most techniques require large amounts of navigation experience for training. Using a new machine learning paradigm learning from hallucination (LfH), training data can be collected in a safe environment and not require supervision. Data is collected in real time while an agent performs a random walk in free space, supervision is not required as there are no obstacles for the robot to run into. After a random walk a post processing pipeline will hallucinate a safety corridor …

Cal Poly (Cp) Legged Robot, Sebastian Barboza, Jonathan Mchale, Isabella Sorensen, Isaac Golan

Mechanical Engineering

The Navy spends $60 billion annually on dangerous ship maintenance performed by sailors. To save lives and resources, the Naval Surface Warfare Center (NSWC) is looking for robots to replace sailors and navigate ships to perform various tasks. Robots with tracks and wheels have been most recently explored by NSWC, however they have encountered significant problems navigating the ships, especially through naval ship doorways with a significant ledge. By using a legged robot, our team hopes to solve these problems and have a robot that can navigate the ship with relative ease and stability.

Mixed Reality In Human–Robot Interaction For Collaborative Applications, Evan Reid, Caitlin Osorio

College of Engineering Summer Undergraduate Research Program

This project explores the use of Extended Reality (XR) technologies to enhance human- robot interaction in industrial contexts. Building upon prior research in affective and cognitive state recognition during human-cobot collaboration, this study investigates how natural hand and head gestures, captured through Meta Quest passthrough mode, can be used to communicate human intent to a Universal Robotics e-Series collaborative robot. The XR system provides users with an immersive, real-world visual interface while tracking motion and position in real time. The captured gestures are interpreted through a custom software pipeline that integrates machine learning models and rule-based logic to trigger adaptive …

Development Of A Control System For An 8-Dof Quadrupedal Robotic Research Platform, Jack Butler

Master's Theses

Quadrupedal robots offer a versatile locomotion option that can extend the operating space of a robot into uneven terrains. However, controlling these systems presents significant challenges due to nonlinearities introduced by various factors.

In this thesis, model-predictive control (MPC) is applied to an 8-DOF legged robot developed by Cal Poly’s Legged Robotics group. The MPC framework employs a lumped rigid-body model that treats the robot as a single rigid body with forces applied directly at the foot contact points. The controller is developed within the ROS2 environment, with integration of state estimation and gait-pattern generation, to provide maximum modularity and …

Spy Bot, Nathanael Bruce Farris

Computer Engineering

This project set out to develop a compact, mobile-controlled robotic platform designed for remote video surveillance and control via a smartphone. Using a Raspberry Pi as the core controller, the robot integrates a Flask-based server, HTML/JavaScript-based user interface, and an infrared-capable camera to provide real-time video streaming and directional control through a web browser. Key design challenges included managing power delivery, integrating motor control with live video, and modernizing web technologies to ensure smooth communication between client and server. The final system demonstrates a responsive and portable proof of concept that highlights the potential for future upgrades, such as night …

A Turtlebot3 Hardware Testbed For Distributed Kalman Filter Localization, Dmitri Dobrynin, Indigo T. Garcia

Electrical Engineering

This report presents the preliminary design for a distributed localization framework for a multi-robot system. Many robotics research papers provide simulations of proposed algorithms in regards to formation control and task allocation. However, it is often that these proposals are without hardware experiments, being limited only to simulation. The objective of this framework is to provide a hardware implementation of a distributed Kalman filtering algorithm for multi-agent localization, as well as provide grounds for future multi-agent experiments. The framework is implemented on a swarm of three Turtlebot3 mobile robots. The robots can accurately localize themselves with respect to other agents …

Conversational Social Robot, Julianna M. Christopoulos, Mackenzie Goldman, Cece E. Hujanen, Jared Hunter

Mechanical Engineering

Background: Social robots are used in various settings to reduce burden on human workers and expand opportunities for people in need of assistance.

Challenge: Design a humanoid robot head and torso capable of holding conversations and interacting with a user using a LLM and motion. Conversations are limited to discussing Cal Poly resources and opportunities with visitors to the Bently Research Center on campus.

Framework For Multi-Agent Coordination And Distributed Localization In Micro-Uavs, Minwoo Park, Nikolas Tambornini

Electrical Engineering

Micro-UAVs (unmanned aerial vehicles) due to their inexpensive nature and compact form factor have shown an increase in prevalence throughout a multitude of applications including, but not limited to: search and rescue, military reconnaissance, and agriculture monitoring. However, for a majority of these high impact applications, a swarm of micro-UAVs are required and furthermore mandate that they are able to cooperatively and autonomously coordinate with each other. For long, controlling and communicating between a user and a singular micro-UAV has been a well known and solved problem, however the same can't be said for swarms of micro-UAVs. This project seeks …

Autonomous Mapping Rover, Garrett Jones, Timothy Kyle Chu, Eugenio Caruso Pasos

Electrical Engineering

This report details the design, implementation, and evaluation of "Rovero," an autonomous mapping rover developed as a senior project. Rovero integrates state-of-the-art technologies, including ROS2 for communication, SLAM algorithms for mapping, and sensor fusion for accurate navigation. The project aimed to achieve robust autonomous operation in indoor environments, leveraging a combination of LiDAR, IMU, and encoder data for real-time decision-making. Key challenges addressed include path planning, obstacle avoidance, and integration of multiple sensor inputs. Results demonstrate successful mapping capabilities and efficient navigation performance.

Design Principles For Robotic-Controlled 3d Printing Of Soft Tissue-Mimicking Materials, Karina Mealey

Master's Theses

Geometrically and physically accurate 3D models are emerging as essential tools for surgeons in pre-operative planning and training, but current methods of producing these models are not sufficient. Existing solutions have downsides such as high costs, long and tedious production times, high complexity resulting in a lack of scalability, or an inability to meet material requirements for accurately simulating human tissue.

The objective of this work is to develop principles for a low-cost, easy-to-use 3D printing system capable of prototyping with soft tissue-mimicking materials. To pursue this goal, a MyCobot280 robot arm was used as the mechanism to explore the …

Fault Tolerant Dynamic Task Allocation For Heterogeneous Multi-Robot Systems, Jack R. Cline

Master's Theses

This research presents a novel approach to dynamic task allocation in heterogeneous multi-robot systems with integrated fault detection capabilities. As multiple industries are becoming more reliant on multi-robot systems for tasks, maintaining operational efficiency despite robot failures becomes critical. We propose a framework that combines optimization-based task allocation with a Kalman filter that estimates task progress for anomaly detection to identify unreliable agents and dynamically redistribute tasks. Observing values such as the normalized innovation squared (NIS), covariance, and progress rate, the algorithm can designate a robot as faulty. Embedding information about which robots are faulty in the task algorithm allows …

Adversarial Deep Reinforcement Learning For Tank Duel Simulation Using Lidar-Based Observations, Braedan Kennedy

Master's Theses

Previous research has demonstrated that reinforcement learning agents can learn to steer differential-drive robots around obstacles using 2D lidar scans as observations. However, these studies typically treat all range returns as undifferentiated obstacles—objects to avoid—without distinguishing between different object types. This thesis builds upon previous research by introducing an adversarial task in which an agent must interpret raw range readings to both avoid static obstacles and identify, pursue, and engage a hostile target.

To investigate this problem, this thesis introduces TankGame, a novel, lightweight 2D tank duel simulator. Each agent receives a 360° lidar scan, controls its motion via tread …

Distributed Formation Control Of Nonholonomic Mobile Robots: Safety-Critical Leader-Follower Approach With Obstacle Avoidance And Dynamic Reconfiguration, Kelvin C. Villago

Master's Theses

Networked control systems for multi-agent robotics have emerged as a critical paradigm for executing complex coordinated tasks in diverse environments. While formation control serves as the backbone of such systems, real-world deployment introduces significant challenges including communication constraints, environmental obstacles, and the need for adaptive reconfiguration. This research addresses these challenges by developing a novel unified framework that seamlessly integrates obstacle avoidance algorithms with dynamic formation reconfiguration capabilities, specifically designed for communication-limited networked control architectures. The proposed framework represents a significant advancement over existing approaches by simultaneously handling both static and dynamic obstacles while maintaining system cohesion under communication constraints. …

Pseudo Gps For Romi, Emmanuel Baez, Owen Guinane, Gabriel Coria, Conor Schott

Mechanical Engineering

The Pseudo-GPS system for Romi robots addresses the need for precise real-time location tracking in Cal Poly's Mechatronics lab. This project, developed by Emmanuel Baez, Gabriel Coria, Owen Guinane, and Conor Schott, under the guidance of instructor Charlie Refvem, provides a proof-of-concept system to enhance the Romi robots' geolocation capabilities for advanced robotic algorithms.

The proposed system uses a Raspberry Pi 4 equipped with a Pi camera module and ArUco markers to track the position and orientation of Romi robots within a lab environment. Custom 3D-printed stands secure markers on the robots, and a designated origin marker defines the coordinate …

Advanced Grasping Sensor Technologies For Autonomous Robotic Apple Harvesting Using Tactile Data And Cnns, Chris Bae

College of Engineering Summer Undergraduate Research Program

This research investigates how to achieve an optimal grasp of an apple using a four-finger soft robotic grasper equipped with force-resistive sensors. Specifically, we sought to determine whether a convolutional neural network (CNN) could accurately classify the grasper's state and recommend adjustments ("in," "out," or "good" grasp) based on tactile data from the sensors. Spatiotemporal tactile images were developed from the sensors and fed into our CNN, achieving near 100% accuracy on unseen test data. This work suggests that CNN-based processing of tactile images can be a powerful tool for real-time control of soft robotic grippers.

Computer Vision In A Robotic Arm, Jack Maxwell

College of Engineering Summer Undergraduate Research Program

We used a machine learning-based object detection algorithm to give a robotic arm the ability to "see" with its camera.

Dynamic Maze Puzzle Navigation Using Deep Reinforcement Learning, Luisa Shu Yi Chiu

Master's Theses

The implementation of deep reinforcement learning in mobile robotics offers a great solution for the development of autonomous mobile robots to efficiently complete tasks and transport objects. Reinforcement learning continues to show impressive potential in robotics applications through self-learning and biological plausibility. Despite its advancements, challenges remain in applying these machine learning techniques in dynamic environments. This thesis explores the performance of Deep Q-Networks (DQN), using images as an input, for mobile robot navigation in dynamic maze puzzles and aims to contribute to advancements in deep reinforcement learning applications for simulated and real-life robotic systems. This project is a step …

Design And Implementation Of An Inverted Short Baseline Acoustic Positioning System, Jakob Frabosilio

Master's Theses

This document details the design, implementation, testing, and analysis of an inverted short baseline acoustic positioning system. The system presented here is an above-water, air-based prototype for an underwater acoustic positioning system; it is designed to determine the position of remotely-operated underwater vehicles (ROVs) and autonomous underwater vehicles (AUVs) in the global frame using a method that does not drift over time.

A ground-truth positioning system is constructed using a stacked hexapod platform actuator, which mimics the motion of an AUV and provides the true position of an ultrasonic microphone array. An ultrasonic transmitter sends a pulse of sound towards …

Drone Swarm Search And Rescue, Rushabh Shah, Christopher Short, Anderson Macmillan, Wyatt Colburn

Electrical Engineering

Drone swarms offer the potential to drastically reduce search times and improve the effectiveness of search and rescue operations. This senior project explores the development of a drone swarm system for search and rescue missions, focusing on two key challenges: (1) precise localization of each drone within the swarm relative to one another and (2) accurate localization of a target beacon relative to the drones. The project utilizes Real Time Kinematic (RTK) processing to enhance the accuracy of drone localization, achieving centimeter-level precision. Target localization is achieved through a triangulation-based approach using Received Signal Strength Indication (RSSI) data from a …

Autonomous Apple Harvester Robot, Jack Ryan Cline, Tyus Green, Devon Woolston

Electrical Engineering

As agricultural demands rise and manual labor costs increase, there has become a dire need to automate apple harvesting. However, the precision and speed necessary for cost-efficient apple harvesting pose a significant challenge for robotic automation. To maintain cost-effective production, a harvester must be able to operate fast enough and long enough to compete with human labor. It must also be able to navigate and traverse apple orchards autonomously and pick apples without damaging the fruit or tree. This project presents an apple harvesting robot that uses a Mask R-CNN vision system with an RGB-D camera to detect the location …

Deep Learning Using Vision And Lidar For Global Robot Localization, Brett E. Gowling

Master's Theses

As the field of mobile robotics rapidly expands, precise understanding of a robot’s position and orientation becomes critical for autonomous navigation and efficient task performance. In this thesis, we present a snapshot-based global localization machine learning model for a mobile robot, the e-puck, in a simulated environment. Our model uses multimodal data to predict both position and orientation using the robot’s on-board cameras and LiDAR sensor. In an effort to minimize localization error, we explore different sensor configurations by varying the number of cameras and LiDAR layers used. Additionally, we investigate the performance benefits of different multimodal fusion strategies while …

Drones For Marine Science And Agriculture, David Caldera, Sai Murthy

College of Engineering Summer Undergraduate Research Program

Our research project was launched at Cal Poly in 2019 with the goal of assisting researchers at the CSULB Shark Lab in detecting sharks from aerial images. Under the guidance of Dr. Franz J. Kurfess, students trained an object detection algorithm using shark images and were able to achieve high rate of detection. Following this success, the team has constructed multiple drones and expanded their research to include applications in the fields of agriculture and ecology. This summer the goal is to use a iPhone 14 Pro in lieu of a traditional camera system for real-time object recognition. Object detection …

Exploring Cognition And Affect During Human-Cobot Interaction, Angelika T. Canete, Javier Gonzalez-Sanchez, Rafael Guerra Silva

College of Engineering Summer Undergraduate Research Program

Collaborative robots (Cobots) have recently gained popularity due to their capability to work collaboratively with human operators. This collaborative relationship has been named under the robotics discipline of Human-Robot Collaboration (HRC), in which humans and robots work together to accomplish a common task while also being in the same physical space. An important part of collaboration is the human's decision-making, which is largely affected by their affective and cognitive state. A cobot lacks this fundamental understanding of the human operator. In this research, we utilize a server-client program to communicate the affective states of a human user to a Raspberry …