Robotics Commons^™

Drone Detection Using Yolov5, Burchan Aydin, Subroto Singha

Faculty Publications

The rapidly increasing number of drones in the national airspace, including those for recreational and commercial applications, has raised concerns regarding misuse. Autonomous drone detection systems offer a probable solution to overcoming the issue of potential drone misuse, such as drug smuggling, violating people’s privacy, etc. Detecting drones can be difficult, due to similar objects in the sky, such as airplanes and birds. In addition, automated drone detection systems need to be trained with ample amounts of data to provide high accuracy. Real-time detection is also necessary, but this requires highly configured devices such as a graphical processing unit (GPU). …

A Monte Carlo Framework For Incremental Improvement Of Simulation Fidelity, Damian M. Lyons, James Finocchiaro, Misha Novitzky, Chris Korpela

Faculty Publications

Robot software developed in simulation often does not be- have as expected when deployed because the simulation does not sufficiently represent reality - this is sometimes called the `reality gap' problem. We propose a novel algorithm to address the reality gap by injecting real-world experience into the simulation. It is assumed that the robot program (control policy) is developed using simulation, but subsequently deployed on a real system, and that the program includes a performance objective monitor procedure with scalar output. The proposed approach collects simulation and real world observations and builds conditional probability functions. These are used to generate …

A Monte Carlo Framework For Incremental Improvement Of Simulation Fidelity, Damian Lyons, James Finocchiaro, Misha Novitsky, Chris Korpela

Faculty Publications

Robot software developed in simulation often does not be- have as expected when deployed because the simulation does not sufficiently represent reality - this is sometimes called the `reality gap' problem. We propose a novel algorithm to address the reality gap by injecting real-world experience into the simulation. It is assumed that the robot program (control policy) is developed using simulation, but subsequently deployed on a real system, and that the program includes a performance objective monitor procedure with scalar output. The proposed approach collects simulation and real world observations and builds conditional probability functions. These are used to generate …

Visual Homing For Robot Teams: Do You See What I See?, Damian Lyons, Noah Petzinger

Faculty Publications

Visual homing is a lightweight approach to visual navigation which does not require GPS. It is very attractive for robot platforms with a low computational capacity. However, a limitation is that the stored home location must be initially within the field of view of the robot. Motivated by the increasing ubiquity of camera information we propose to address this line-of-sight limitation by leveraging camera information from other robots and fixed cameras. To home to a location that is not initially within view, a robot must be able to identify a common visual landmark with another robot that can be used …

Visual Homing For Robot Teams: Do You See What I See?, Damian Lyons, Noah Petzinger

Faculty Publications

Visual homing is a lightweight approach to visual navigation which does not require GPS. It is very attractive for robot platforms with a low computational capacity. However, a limitation is that the stored home location must be initially within the field of view of the robot. Motivated by the increasing ubiquity of camera information we propose to address this line-of-sight limitation by leveraging camera information from other robots and fixed cameras. To home to a location that is not initially within view, a robot must be able to identify a common visual landmark with another robot that can be used …

A Meta-Level Approach For Multilingual Taint Analysis, Damian Lyons, Dino Becaj

Faculty Publications

It is increasingly common for software developers to leverage the features and ease-of-use of different languages in building software systems. Nonetheless, interaction between different languages has proven to be a source of software engineering concerns. Existing static analysis tools handle the software engineering concerns of monolingual software but there is little general work for multilingual systems despite the increasing visibility of these systems. While recent work in this area has greatly extended the scope of multilingual static analysis systems, the focus has still been on a primary, host language interacting with subsidiary, guest language functions. In this paper we propose …

Wall Detection Via Imu Data Classification In Autonomous Quadcopters, Jason Hughes, Damian Lyons

Faculty Publications

Abstract—An autonomous drone flying near obstacles needs to be able to detect and avoid the obstacles or it will collide with them. In prior work, drones can detect and avoid walls using data from camera, ultrasonic or laser sensors mounted either on the drone or in the environment. It is not always possible to instrument the environment, and sensors added to the drone consume payload and power - both of which are constrained for drones. This paper studies how data mining classification techniques can be used to predict where an obstacle is in relation to the drone based only on …

Evaluating The Potential Of Drone Swarms In Nonverbal Hri Communication, Kasper Grispino, Damian Lyons, Truong-Huy Nguyen

Faculty Publications

Human-to-human communications are enriched with affects and emotions, conveyed, and perceived through both verbal and nonverbal communication. It is our thesis that drone swarms can be used to communicate information enriched with effects via nonverbal channels: guiding, generally interacting with, or warning a human audience via their pattern of motions or behavior. And furthermore that this approach has unique advantages such as flexibility and mobility over other forms of user interface. In this paper, we present a user study to understand how human participants perceived and interpreted swarm behaviors of micro-drone Crazyflie quadcopters flying three different flight formations to bridge …

Using Taint Analysis And Reinforcement Learning (Tarl) To Repair Autonomous Robot Software, Damian Lyons, Saba Zahra

Faculty Publications

It is important to be able to establish formal performance bounds for autonomous systems. However, formal verification techniques require a model of the environment in which the system operates; a challenge for autonomous systems, especially those expected to operate over longer timescales. This paper describes work in progress to automate the monitor and repair of ROS-based autonomous robot software written for an a-priori partially known and possibly incorrect environment model. A taint analysis method is used to automatically extract the data-flow sequence from input topic to publish topic, and instrument that code. A unique reinforcement learning approximation of MDP utility …

Drone Proximity Detection Via Air Disturbance Analysis, Qian Zhao, Jason Hughes

Faculty Publications

The use of unmanned aerial vehicles (drones) is expanding to commercial, scientific, and agriculture applications, including surveillance, product deliveries and aerial photography. One challenge for applications of drones is detecting obstacles and avoiding collisions. A typical solution to this issue is the use of camera sensors or ultrasonic sensors for obstacle detection or sometimes just manual control (teleoperation). However, these solutions have costs in battery lifetime, payload, operator skill. We note that there will be an air disturbance in the vicinity of the drone when it’s moving close to obstacles or other drones. Our objective is to detect obstacles from …

A New Ectotherm 3d Tracking And Behavior Analytics System Using A Depth-Based Approach With Color Validation, With Preliminary Data On Kihansi Spray Toad (Nectophrynoides Asperginis) Activity, Philip Bal, Damian Lyons, Avishai Shuter

Faculty Publications

The Kihansi spray toad (Nectophrynoides asperginis), classified as Extinct in the Wild by the IUCN, is being bred at the Wildlife Conservation Society’s (WCS) Bronx Zoo as part of an effort to successfully reintroduce the species into the wild. Thousands of toads live at the Bronx Zoo presenting an opportunity to learn more about their behaviors for the first time, at scale. It is impractical to perform manual observations for long periods of time. This paper reports on the development of a RGB-D tracking and analytics approach that allows researchers to accurately and efficiently gather information about the toads’ behavior. …

A Monte Carlo Approach To Closing The Reality Gap, Damian Lyons, James Finocchiaro, Michael Novitzky, Christopher Korpela

Faculty Publications

We propose a novel approach to the ’reality gap’ problem, i.e., modifying a robot simulation so that its performance becomes more similar to observed real world phenomena. This problem arises whether the simulation is being used by human designers or in an automated policy development mechanism. We expect that the program/policy is developed using simulation, and subsequently deployed on a real system. We further assume that the program includes a monitor procedure with scalar output to determine when it is achieving its performance objectives. The proposed approach collects simulation and real world observations and builds conditional probability functions. These are …

An Approach To Fast Multi-Robot Exploration In Buildings With Inaccessible Spaces, Matt Mcneill, Damian Lyons

Faculty Publications

The rapid exploration of unknown environments is a common application of autonomous multi-robot teams. For some types of exploration missions, a mission designer may possess some rudimentary knowledge about the area to be explored. For example, the dimensions of a building may be known, but not its floor layout or the location of furniture and equipment inside. For this type of mission, the Space- Based Potential Field (SBPF) method is an approach to multirobot exploration which leverages a priori knowledge of area bounds to determine robot motion. Explored areas and obstacles exert a repulsive force, and unexplored areas exert an …

A Comparison Of Contextual Bandit Approaches To Human-In-The-Loop Robot Task Completion With Infrequent Feedback, Matt Mcneill, Damian Lyons

Faculty Publications

Artificially intelligent assistive agents are playing an increased role in our work and homes. In contrast with currently predominant conversational agents, whose intelligence derives from dialogue trees and external modules, a fully autonomous domestic or workplace robot must carry out more complex reasoning. Such a robot must make good decisions as soon as possible, learn from experience, respond to feedback, and rely on feedback only as much as necessary. In this research, we narrow the focus of a hypothetical robot assistant to a room tidying task in a simulated domestic environment. Given an item, the robot chooses where to put …

Evaluation Of Field Of View Width In Stereo-Vision-Based Visual Homing, Damian Lyons, Benjamin Barriage, Luca Del Signore

Faculty Publications

Visual homing is a local navigation technique used to direct a robot to a previously seen location by comparing the image of the original location with the current visual image. Prior work has shown that exploiting depth cues such as image scale or stereo-depth in homing leads to improved homing performance. While it is not unusual to use a panoramic field of view (FOV) camera in visual homing, it is unusual to have a panoramic FOV stereo-camera. So, while the availability of stereo-depth information may improve performance, the concomitant-restricted FOV may be a detriment to performance, unless specialized stereo hardware …

A Neuromorphic Quadratic, Integrate, And Fire Silicon Neuron With Adaptive Gain, David Parent, Eric Basham

Faculty Publications

An integrated circuit implementation of a silicon neuron was designed, manufactured, and tested. The circuit was designed using the Quadratic, Integrate, and Fire (QIF) neuron model in 0.5 µm silicon technology. The neuron implementation was optimized for low current consumption, drawing only 1.56 mA per QIF circuit and utilized hysteretic reset, non-inverting integrator, and voltage-squarer circuits. The final area of each circuit in silicon was 268 µm height × 400 µm width. This design is the first IC of its kind for this neuron model and is successfully able to output true spiking that follows the behaviors of bistability, monotonic, …

Formal Performance Guarantees For An Approach To Human In The Loop Robot Missions, Damian Lyons, Ron Arkin, Shu Jiang, Matt O'Brien, Feng Tang, Peng Tang

Faculty Publications

Abstract— A key challenge in the automatic verification of robot mission software, especially critical mission software, is to be able to effectively model the performance of a human operator and factor that into the formal performance guarantees for the mission. We present a novel approach to modelling the skill level of the operator and integrating it into automatic verification using a linear Gaussians model parameterized by experimental calibration. Our approach allows us to model different skill levels directly in terms of the behavior of the lumped, robot plus operator, system.

Using MissionLab and VIPARS (a behavior-based robot mission verification …

An Approach To Robust Homing With Stereovision, Fuqiang Fu, Damian Lyons

Faculty Publications

Visual Homing is a bioinspired approach to robot navigation which can be fast and uses few assumptions. However, visual homing in a cluttered and unstructured outdoor environment offers several challenges to homing methods that have been developed for primarily indoor environments. One issue is that any current image during homing may be tilted with respect to the home image. The second is that moving through a cluttered scene during homing may cause obstacles to interfere between the home scene and location and the current scene and location. In this paper, we introduce a robust method to improve a previous developed …

Performance Verification For Robot Missions In Uncertain Environments, Damian Lyons, Ron Arkin, Shu Jiang, Matt O'Brien, Feng Tang, Peng Tang

Faculty Publications

Abstract—Certain robot missions need to perform predictably in a physical environment that may have significant uncertainty. One approach is to leverage automatic software verification techniques to establish a performance guarantee. The addition of an environment model and uncertainty in both program and environment, however, means the state-space of a model-checking solution to the problem can be prohibitively large. An approach based on behavior-based controllers in a process-algebra framework that avoids state-space combinatorics is presented here. In this approach, verification of the robot program in the uncertain environment is reduced to a filtering problem for a Bayesian Network. Validation results …

Establishing A-Priori Performance Guarantees For Robot Missions That Include Localization Software, Damian Lyons, Ron Arkin, Shu Jiang, Matt O'Brien, Feng Tang, Peng Tang

Faculty Publications

One approach to determining whether an automated system is performing correctly is to monitor its performance, signaling when the performance is not acceptable; another approach is to automatically analyze the possible behaviors of the system a-priori and determine performance guarantees. Thea authors have applied this second approach to automatically derive performance guarantees for behaviorbased, multi-robot critical mission software using an innovative approach to formal verification for robotic software. Localization and mapping algorithms can allow a robot to navigate well in an unknown environment. However, whether such algorithms enhance any specific robot mission is currently a matter for empirical validation. Several …

Formal Performance Guarantees For Behavior-Based Localization Missions, Damian Lyons, Ron Arkin, Shu Jiang, Matt O'Brien, Feng Tang, Peng Tang

Faculty Publications

Abstract— Localization and mapping algorithms can allow a robot to navigate well in an unknown environment. However, whether such algorithms enhance any specific robot mission is currently a matter for empirical validation. In this paper we apply our MissionLab/VIPARS mission design and verification approach to an autonomous robot mission that uses probabilistic localization software.

Two approaches to modeling probabilistic localization for verification are presented: a high-level approach, and a sample-based approach which allows run-time code to be embedded in verification. Verification and experimental validation results are presented for two different missions, each using each method, demonstrating the accuracy …

Landmark Detection With Surprise Saliency Using Convolutional Neural Networks, Feng Tang, Damian Lyons, Daniel Leeds

Faculty Publications

Abstract—Landmarks can be used as reference to enable people or robots to localize themselves or to navigate in their environment. Automatic definition and extraction of appropriate landmarks from the environment has proven to be a challenging task when pre-defined landmarks are not present. We propose a novel computational model of automatic landmark detection from a single image without any pre-defined landmark database. The hypothesis is that if an object looks abnormal due to its atypical scene context (what we call surprise saliency), it then may be considered as a good landmark because it is unique and easy to spot by …

Establishing Performance Guarantees For Behavior-Based Robot Missions Using An Smt Solver, Feng Tang, Damian M. Lyons, Ronald Arkin

Faculty Publications

In prior work we developed an approach to formally representing behavior-based multi-robot programs, and the uncertain environments in which they operate, as process networks. We automatically extract a set of probabilistic equations governing program execution in that environment using a static analysis module called VIPARS, and solve these using a Dynamic Bayesian Network (DBN) to establish whether stated performance guarantees hold for the program in that environment. In this paper we address the challenge of expanding the range of performance guarantees that are possible by using an SMT-solver instead of a DBN. We translate flow functions, which are recursive probabilistic …

Thorough Exploration Of Complex Environments With A Space-Based Potential Field, Kenealy Alina, Nicholas Primiano, Alex Keyes, Lyons Damian

Faculty Publications

Robotic exploration, for the purposes of search and rescue or explosive device detection, can be improved by using a team of multiple robots. Potential field navigation methods offer natural and efficient distributed exploration algorithms in which team members are mutually repelled to spread out and cover the area efficiently. However, they also suffer from field minima issues. Liu and Lyons proposed a Space-Based Potential Field (SBPF) algorithm that disperses robots efficiently and also ensures they are driven in a distributed fashion to cover complex geometry. In this paper, the approach is modified to handle two problems with the original SBPF …

Homing With Stereovision, Paramesh Nirmal, Damian Lyons

Faculty Publications

Visual Homing is a navigation method based on comparing a stored image of a goal location to the current image to determine how to navigate to the goal location. It is theorized that insects such as ants and bees employ visual homing techniques to return to their nest or hive, and inspired by this, several researchers have developed elegant robot visual homing algorithms. Depth information, from visual scale, or other modality such as laser ranging, can improve the quality of homing. While insects are not well equipped for stereovision, stereovision is an effective robot sensor. We describe the challenges involved …

Evaluation Of Parallel Reduction Strategies For Fusion Of Sensory Information From A Robot Team., Damian M. Lyons, Joseph Leroy

Faculty Publications

The advantage of using a team of robots to search or to map an area is that by navigating the robots to different parts of the area, searching or mapping can be completed more quickly. A crucial aspect of the problem is the combination, or fusion, of data from team members to generate an integrated model of the search/mapping area. In prior work we looked at the issue of removing mutual robots views from an integrated point cloud model built from laser and stereo sensors, leading to a cleaner and more accurate model. This paper addresses a further challenge: Even …

Leveraging Area Bounds Information For Autonomous Decentralized Multi-Robot Exploration, Tsungming Liu, Damian Lyons

Faculty Publications

This paper proposes a simple and uniform, decentralized approach to the problem of dispersing a team of robots to explore an area quickly. The Decentralized Space-Based Potential Field (D-SBPF) algorithm is a potential field approach that leverages knowledge of the overall bounds of the area to be explored. It includes a monotonic coverage factor in the potential field to avoid minima, realistic sensor bounds, and a distributed map exchange protocol. The D-SBPF approach yields a simple potential field control strategy for all robots but nonetheless has good dispersion and overlap performance in exploring areas with convex geometry while avoiding potential …

Performance Verification For Behavior-Based Robot Missions, Damian M. Lyons, Ron Arkin, Shu Jiang, Tsungming Liu, Paramesh Nirmal

Faculty Publications

Certain robot missions need to perform predictably in a physical environment that may have significant uncertainty. One approach is to leverage automatic software verification techniques to establish a performance guarantee. The addition of an environment model and uncertainty in both program and environment, however, means the state-space of a model-checking solution to the problem can be prohibitively large. An approach based on behavior-based controllers in a process-algebra framework that avoids state-space combinatorics is presented here. In this approach, verification of the robot program in the uncertain environment is reduced to a filtering problem for a Bayesian Network. Validation results are …

Probabilistic Verification Of Multi-Robot Missions In Uncertain Environments, Damian M. Lyons, Ronald Arkin, Shu Jiang, Dagan Harrington, Feng Tang, Peng Tang

Faculty Publications

The effective use of autonomous robot teams in highly-critical missions depends on being able to establish performance guarantees. However, establishing a guarantee for the behavior of an autonomous robot operating in an uncertain environment with obstacles is a challenging problem. This paper addresses the challenges involved in building a software tool for verifying the behavior of a multi-robot waypoint mission that includes uncertain environment geometry as well as uncertainty in robot motion. One contribution of this paper is an approach to the problem of a-priori specification of uncertain environments for robot program verification. A second contribution is a novel method …

A Kinect-Based System For Automatic Recording Of Some Pigeon Behaviors, Damian Lyons, James Macdonall, Kelly Cunningham

Faculty Publications

Contact switches and touch screens are the state of the art for recording pigeons’ pecking behavior. Recording other behavior, however, requires a different sensor for each behavior, and some behaviors cannot easily be recorded. We present a flexible and inexpensive image-based approach to detecting and counting pigeon behaviors that is based on the Kinect sensor from Microsoft. Although the system is as easy to set up and use as the standard approaches, it is more flexible because it can record behaviors in addition to key pecking. In this article, we show how both the fast, fine motion of key pecking …