Engineering Commons^™

Relative Vectoring Using Dual Object Detection For Autonomous Aerial Refueling, Derek B. Worth, Jeffrey L. Choate, James Lynch, Scott L. Nykl, Clark N. Taylor

Faculty Publications

Once realized, autonomous aerial refueling will revolutionize unmanned aviation by removing current range and endurance limitations. Previous attempts at establishing vision-based solutions have come close but rely heavily on near perfect extrinsic camera calibrations that often change midflight. In this paper, we propose dual object detection, a technique that overcomes such requirement by transforming aerial refueling imagery directly into receiver aircraft reference frame probe-to-drogue vectors regardless of camera position and orientation. These vectors are precisely what autonomous agents need to successfully maneuver the tanker and receiver aircraft in synchronous flight during refueling operations. Our method follows a common 4-stage process …

Visual Navigation And Control For Spacecraft Proximity Operations With Unknown Targets, Wyatt J. Harris

Theses and Dissertations

Many current and future spacecraft missions must conduct rendezvous and proximity operations (RPO) with resident space objects (RSOs). An important subset of spacecraft RPO that is yet to be demonstrated on-orbit involves final approach maneuvers with respect to RSOs where no information (such as geometry, inertia, relative velocity, etc.) is known about the target a priori, and no information is actively provided by the target during maneuvering. Such operation with respect to ‘unknown’ targets represents an important possible mission set for Department of Defense spacecraft and is the subject of this research. Two visual servoing frameworks capable of autonomously controlling …

Stereo Camera Calibrations With Optical Flow, Joshua D. Larson

Theses and Dissertations

Remotely Piloted Aircraft (RPA) are currently unable to refuel mid-air due to the large communication delays between their operators and the aircraft. AAR seeks to address this problem by reducing the communication delay to a fast line-of-sight signal between the tanker and the RPA. Current proposals for AAR utilize stereo cameras to estimate where the receiving aircraft is relative to the tanker, but require accurate calibrations for accurate location estimates of the receiver. This paper improves the accuracy of this calibration by improving three components of it: increasing the quantity of intrinsic calibration data with CNN preprocessing, improving the quality …

Accelerating Point Set Registration For Automated Aerial Refueling, Ryan M. Raettig

Theses and Dissertations

The goal of AAR is to control the tanker boom to safely refuel a receiving aircraft with no input or aid from the boom operator. To achieve this, the pose of the receiver relative to the tanker must be known. Point set registration is a fundamental issue used to estimate the relative pose of an object in an environment. However, it's likely a computational bottleneck of a vision processing pipeline. In addition, the matching of each sensed point with a closest truth point, nearest neighbor matching, is the most costly portion of the point set registration process. For this reason, …

Using Motion Capture And Augmented Reality To Test Aar With Boom Occlusion, Vincent J. Bownes

Theses and Dissertations

The operational capability of drones is limited by their inability to perform aerial refueling. This can be overcome by automating the process with a computer vision solution. Previous work has demonstrated the feasibility of automated aerial refueling (AAR) in simulation. To progress this technique to the real world, this thesis conducts experiments using real images of a physical aircraft replica and a motion capture system for truth data. It also compares the error between the real and virtual experiments to validate the fidelity of the simulation. Results indicate that the current technique is effective on real images and that the …

Object Detection With Deep Learning To Accelerate Pose Estimation For Automated Aerial Refueling, Andrew T. Lee

Theses and Dissertations

Remotely piloted aircraft (RPAs) cannot currently refuel during flight because the latency between the pilot and the aircraft is too great to safely perform aerial refueling maneuvers. However, an AAR system removes this limitation by allowing the tanker to directly control the RP A. The tanker quickly finding the relative position and orientation (pose) of the approaching aircraft is the first step to create an AAR system. Previous work at AFIT demonstrates that stereo camera systems provide robust pose estimation capability. This thesis first extends that work by examining the effects of the cameras' resolution on the quality of pose …

Maximizing Accuracy Through Stereo Vision Camera Positioning For Automated Aerial Refueling, Kirill A. Sarantsev

Theses and Dissertations

Aerial refueling is a key component of the U.S. Air Force strategic arsenal. When two aircraft interact in an aerial refueling operation, the accuracy of relative navigation estimates are critical for the safety, accuracy and success of the mission. Automated Aerial Refueling (AAR) looks to improve the refueling process by creating a more effective system and allowing for Unmanned Aerial Vehicle(s) (UAV) support. This paper considers a cooperative aerial refueling scenario where stereo cameras are used on the tanker to direct a \boom" (a large, long structure through which the fuel will ow) into a port on the receiver aircraft. …

Use Of Lidar In Automated Aerial Refueling To Improve Stereo Vision Systems, Michael R. Crowl

Theses and Dissertations

The United States Air Force (USAF) executes five Core Missions, four of which depend on increased aircraft range. To better achieve global strike and reconnaissance, unmanned aerial vehicles (UAVs) require aerial refueling for extended missions. However, current aerial refueling capabilities are limited to manned aircraft due to technical difficulties to refuel UAVs mid-flight. The latency between a UAV operator and the UAV is too large to adequately respond for such an operation. To overcome this limitation, the USAF wants to create a capability to guide the refueling boom into the refueling receptacle. This research explores the use of light detection …

Infrared And Electro-Optical Stereo Vision For Automated Aerial Refueling, William E. Dallmann

Theses and Dissertations

Currently, Unmanned Aerial Vehicles are unsafe to refuel in-flight due to the communication latency between the UAVs ground operator and the UAV. Providing UAVs with an in-flight refueling capability would improve their functionality by extending their flight duration and increasing their flight payload. Our solution to this problem is Automated Aerial Refueling (AAR) using stereo vision from stereo electro-optical and infrared cameras on a refueling tanker. To simulate a refueling scenario, we use ground vehicles to simulate a pseudo tanker and pseudo receiver UAV. Imagery of the receiver is collected by the cameras on the tanker and processed by a …

Satellite Articulation Sensing Using Computer Vision, David H. Curtis

Theses and Dissertations

Autonomous on-orbit satellite servicing benefits from an inspector satellite that can gain as much information as possible about the primary satellite. This includes performance of articulated objects such as solar arrays, antennas, and sensors. A method for building an articulated model from monocular imagery using tracked feature points and the known relative inspection route is developed. Two methods are also developed for tracking the articulation of a satellite in real-time given an articulated model using both tracked feature points and image silhouettes. Performance is evaluated for multiple inspection routes and the effect of inspection route noise is assessed. Additionally, a …

Integrity Monitoring For Automated Aerial Refueling: A Stereo Vision Approach, Thomas R. Stuart

Theses and Dissertations

Unmanned aerial vehicles (UAVs) increasingly require the capability to y autonomously in close formation including to facilitate automated aerial refueling (AAR). The availability of relative navigation measurements and navigation integrity are essential to autonomous relative navigation. Due to the potential non-availability of the global positioning system (GPS) during military operations, it is highly desirable that relative navigation can be accomplished without the use of GPS. This paper develops two algorithms designed to provide relative navigation measurements solely from a stereo image pair. These algorithms were developed and analyzed in the context of AAR using a stereo camera system modeling that …

Improving Real-World Performance Of Vision Aided Navigation In A Flight Environment, Donald T. Venable

Theses and Dissertations

The motivation of this research is to fuse information from an airborne imaging sensor with information extracted from satellite imagery in order to provide accurate position when GPS is unavailable for an extended duration. A corpus of existing geo-referenced satellite imagery is used to create a key point database. A novel algorithm for recovering coarse pose using by comparing key points extracted from the airborne imagery to the reference database is developed. This coarse position is used to bootstrap a local-area geo-registration algorithm, which provides GPS-level position estimates. This research derives optimizations for existing local-area methods for operation in flight …

Evaluating Storm Sewer Pipe Condition Using Autonomous Drone Technology, Maria T. Meeks

Theses and Dissertations

The United States Air Force (USAF) owns a total of 30.9 million linear feet (LF) of storm sewer pipes valued at approximately $2.3B in its vast portfolio of built infrastructure. Current inventory records reveal that 78% of the inventory (24.1 million LF) is over 50 years old and will soon exceed its estimated service life. Additionally, the USAF depends on contract support while its business processes undervalue in-service evaluations from long-term funding plans. Ultimately, this disconnect negatively impacts infrastructure performance and overall strategic success, and the USAF risks making uninformed decisions in a fiscally constrained environment.

This research presents a …

Towards Automated Aerial Refueling: Real Time Position Estimation With Stereo Vision, Bradley D. Denby

Theses and Dissertations

Aerial refueling is essential to the United States Air Force (USAF) core mission of rapid global mobility. However, in-flight refueling is not available to remotely piloted aircraft (RPA) or unmanned aerial systems (UAS). As reliance on drones for intelligence, surveillance, and reconnaissance (ISR) and other USAF core missions grows, the ability to automate aerial refueling for such systems becomes increasingly critical. New refueling platforms include sensors that could be used to estimate the relative position of an approaching aircraft. Relative position estimation is a key component to solving the automated aerial refueling (AAR) problem. Analysis of data from a one-seventh …

Multidimensional Analysis Of Nuclear Detonations, Robert C. Slaughter

Theses and Dissertations

The focus of this research is to develop the methods for 3-dimensional reconstructions and temperature distributions of early time nuclear fireballs. Initial developments have resulted in the first 2-dimensional temperature distribution of a nuclear fireball using digitized film. This same process was then extended to perform the first limb darkening calculations of nuclear fireballs. Limb darkening coefficients were then used to derive a temperature profile of the nuclear fireball as a function of optical path length. A 3-dimensional reconstruction was performed using a variation of a structure from illuminance technique in combination with limb darkening coefficients to produce structure using …

A Monocular Slam Method To Estimate Relative Pose During Satellite Proximity Operations, Scott J. Kelly

Theses and Dissertations

Automated satellite proximity operations is an increasingly relevant area of mission operations for the US Air Force with potential to significantly enhance space situational awareness (SSA). Simultaneous localization and mapping (SLAM) is a computer vision method of constructing and updating a 3D map while keeping track of the location and orientation of the imaging agent inside the map. The main objective of this research effort is to design a monocular SLAM method customized for the space environment. The method developed in this research will be implemented in an indoor proximity operations simulation laboratory. A run-time analysis is performed, showing near …

Terrain Referenced Navigation Using Sift Features In Lidar Range-Based Data, Matthew T. Leines

Theses and Dissertations

The use of GNSS in aiding navigation has become widespread in aircraft. The long term accuracy of INS are enhanced by frequent updates of the highly precise position estimations GNSS provide. Unfortunately, operational environments exist where constant signal or the requisite number of satellites are unavailable, significantly degraded, or intentionally denied. This thesis describes a novel algorithm that uses scanning LiDAR range data, computer vision features, and a reference database to generate aircraft position estimations to update drifting INS estimates. The algorithm uses a single calibrated scanning LiDAR to sample the range and angle to the ground as an aircraft …

Enhanced Image-Aided Navigation Algorithm With Automatic Calibration And Affine Distortion Prediction, Juan D. Jurado

Theses and Dissertations

This research aims at improving two key steps within the image aided navigation process: camera calibration and landmark tracking. The camera calibration step is improved by automating the point correspondence calculation within the standard camera calibration algorithm, thereby reducing the required time for calibration while maintaining the output model accuracy. The feature landmark tracking step is improved by digitally simulating affine distortions on input images in order to calculate more accurate feature descriptors for improved feature matching in high relative viewpoint change. These techniques are experimentally demonstrated in an outdoor environment with a consumer-grade inertial sensor and three imaging sensors, …

Covariance Analysis Of Vision Aided Navigation By Bootstrapping, Andrew L. Relyea

Theses and Dissertations

Inertial Navigation System (INS) aiding using bearing measurements taken over time of stationary ground features is investigated. A cross country flight, in two and three dimensional space, is considered, as well as a vertical drop in three dimensional space. The objective is to quantify the temporal development of the uncertainty in the navigation states of an aircraft INS which is aided by taking bearing measurements of ground objects which have been geolocated using ownship position. It is shown that during wings level flight at constant speed and a fixed altitude, an aircraft that tracks ground objects and over time sequentially …

Design Of A Monocular Multi-Spectral Skin Detection, Melanin Estimation, And False-Alarm Suppression System, Keith R. Peskosky

Theses and Dissertations

A real-time skin detection, false-alarm reduction, and melanin estimation system is designed targeting search and rescue (SAR) with application to special operations for manhunting and human measurement and signatures intelligence. A mathematical model of the system is developed and used to determine how the physical system performs under illumination, target-to-sensor distance, and target-type scenarios. This aspect is important to the SAR community to gain an understanding of the deployability in different operating environments. A multi-spectral approach is developed and consists of two short-wave infrared cameras and two visible cameras. Through an optical chain of lenses, custom designed and fabricated dichroic …

Electronic Image Stabilization For Mobile Robotic Vision Systems, Michael John Smith

Theses and Dissertations

When a camera is affixed on a dynamic mobile robot, image stabilization is the first step towards more complex analysis on the video feed. This thesis presents a novel electronic image stabilization (EIS) algorithm for small inexpensive highly dynamic mobile robotic platforms with onboard camera systems. The algorithm combines optical flow motion parameter estimation with angular rate data provided by a strapdown inertial measurement unit (IMU). A discrete Kalman filter in feedforward configuration is used for optimal fusion of the two data sources. Performance evaluations are conducted by a simulated video truth model (capturing the effects of image translation, rotation, …

Deeply-Integrated Feature Tracking For Embedded Navigation, Jeffery R. Gray

Theses and Dissertations

The Air Force Institute of Technology (AFIT) is investigating techniques to improve aircraft navigation using low-cost imaging and inertial sensors. Stationary features tracked within the image are used to improve the inertial navigation estimate. These features are tracked using a correspondence search between frames. Previous research investigated aiding these correspondence searches using inertial measurements (i.e., stochastic projection). While this research demonstrated the benefits of further sensor integration, it still relied on robust feature descriptors (e.g., SIFT or SURF) to obtain a reliable correspondence match in the presence of rotation and scale changes. Unfortunately, these robust feature extraction algorithms are computationally …

Model-Based Control Using Model And Mechanization Fusion Techniques For Image-Aided Navigation, Constance D. Hendrix

Theses and Dissertations

Unmanned aerial vehicles are no longer used for just reconnaissance. Current requirements call for smaller autonomous vehicles that replace the human in high-risk activities. Many times these activities are performed in GPS-degraded environments. Without GPS providing today's most accurate navigation solution, autonomous navigation in tight areas is more difficult. Today, image-aided navigation is used and other methods are explored to more accurately navigate in such areas (e.g., indoors). This thesis explores the use of inertial measurements and navigation solution updates using cameras with a model-based Linear Quadratic Gaussian controller. To demonstrate the methods behind this research, the controller will provide …

Using Predictive Rendering As A Vision-Aided Technique For Autonomous Aerial Refueling, Adam D. Weaver

Theses and Dissertations

This research effort seeks to characterize a vision-aided approach for an Unmanned Aerial System (UAS) to autonomously determine relative position to another aircraft in a formation, specifically to address the autonomous aerial refueling problem. A system consisting of a monocular digital camera coupled with inertial sensors onboard the UAS is analyzed for feasibility of using this vision-aided approach. A three-dimensional rendering of the tanker aircraft is used to generate predicted images of the tanker as seen by the receiver aircraft. A rigorous error model is developed to model the relative dynamics between an INS-equipped receiver and the tanker aircraft. A …

Communication Free Robot Swarming, Zachary C. Gray

Theses and Dissertations

As the military use of unmanned aerial vehicles increases, a growing need for novel strategies to control these systems exists. One such method for controlling many unmanned aerial vehicles simultaneously is the through the use of swarm algorithms. This research explores a swarm robotic algorithm developed by Kadrovach implemented on Pioneer Robots in a real-world environment. An adaptation of his visual sensor is implemented using stereo vision as the primary method of sensing the environment. The swarm members are prohibited from explicitly communicating other than passively through the environment. The resulting implementation produces a communication free swarming algorithm. The algorithm …

Situational Awareness And Synthetic Vision For Unmanned Aerial Vehicle Flight Testing, Joseph M. Dugan

Theses and Dissertations

The Advanced Navigation Technology (ANT) Center at the Air Force Institute of Technology (AFIT) is currently exploring ways to develop and advance the employment of autonomous Unmanned Aerial Vehicles (UAV) by the Department of Defense for military purposes. The research in this thesis describes the development of a tool that enhances situational awareness and provides synthetic vision in a program called the Aviator Visual Display Simulator (AVDS) during UAV flight. During flight testing, the Situational Awareness and Synthetic Vision Relay Tool (SASVRT) developed provides the test coordinator and pilot, as well as the safety observers, with the most pertinent information …

Determination Of Structure From Motion Using Aerial Imagery, Paul R. Graham

Theses and Dissertations

The structure from motion process creates three-dimensional models from a sequence of images. Until recently, most research in this field has been restricted to land-based imagery. This research examines the current methods of land-based structure from motion and evaluates their performance for aerial imagery. Current structure from motion algorithms search the initial image for features to track though the subsequent images. These features are used to create point correspondences between the two images. The correspondences are used to estimate the motion of the camera and then the three-dimensional structure of the scene. This research tests current algorithms using synthetic data …

Discovering The Merit Of The Wavelet Transform For Object Classification, Matthew D. Eyster

Theses and Dissertations

Vision is the primary sense by which most biological systems collect information about their environment. Computer vision is a branch of artificial intelligence concerned with endowing machines with the ability to understand images. Object recognition is a key part of machine vision with far reaching benefits ranging from target recognition, surveillance systems, to automation systems. Extraction of salient features from an image is one of the key steps in object recognition. Typically, geometric primitives are extracted from an image using local analysis. However, the wavelet transform provides a global approach with good locality. Additionally, the directional and multiresolution properties may …

Realtime Color Stereovision Processing, Byron P. Formwalt

Theses and Dissertations

Recent developments in aviation have made micro air vehicles (MAVs) a reality. These featherweight palm-sized radio-controlled flying saucers embody the future of air-to-ground combat. No one has ever successfully implemented an autonomous control system for MAVs. Because MAVs are physically small with limited energy supplies, video signals offer superiority over radar for navigational applications. This research takes a step forward in real time machine vision processing. It investigates techniques for implementing a real time stereovision processing system using two miniature color cameras. The effects of poor-quality optics are overcome by a robust algorithm, which operates in real time and achieves …