Physical Sciences and Mathematics Commons^™

Situate: An Agent-Based System For Situation Recognition, Max Henry Quinn

Dissertations and Theses

Computer vision and machine learning systems have improved significantly in recent years, largely based on the development of deep learning systems, leading to impressive performance on object detection tasks. Understanding the content of images is considerably more difficult. Even simple situations, such as "a handshake", "walking the dog", "a game of ping-pong", or "people waiting for a bus", present significant challenges. Each consists of common objects, but are not reliably detectable as a single entity nor through the simple co-occurrence of their parts.

In this dissertation, toward the goal of developing machine learning systems that demonstrate properties associated with understanding, …

View Synthesis Of Dynamic Scenes Based On Deep 3d Mask Volume, Kai-En Lin, Guowei Yang, Lei Xiao, Feng Liu, Ravi Ramamoorthi

Computer Science Faculty Publications and Presentations

Image view synthesis has seen great success in reconstructing photorealistic visuals, thanks to deep learning and various novel representations. The next key step in immersive virtual experiences is view synthesis of dynamic scenes. However, several challenges exist due to the lack of high-quality training datasets, and the additional time dimension for videos of dynamic scenes. To address this issue, we introduce a multi-view video dataset, captured with a custom 10-camera rig in 120FPS. The dataset contains 96 high-quality scenes showing various visual effects and human interactions in outdoor scenes. We develop a new algorithm, Deep 3D Mask Volume, which enables …

Novel View Synthesis - A Neural Network Approach, Hoang Le

Dissertations and Theses

Novel view synthesis is an important research problem in computer vision and computational photography. It enables a wide range of applications including re-cinematography, video enhancement, virtual reality, etc. These algorithms leverage a pre-acquired set of images taken from a set of viewpoints to synthesize another image at a novel viewpoint as if it was captured by a real camera. To synthesize a high-quality novel view, these algorithms often assume a static scene, or the images were captured synchronously. However, the scenes in practice are often dynamic, and taking a dense set of images of these scenes at the same moment …

Leveraging Model Flexibility And Deep Structure: Non-Parametric And Deep Models For Computer Vision Processes With Applications To Deep Model Compression, Anthony D. Rhodes

Dissertations and Theses

My dissertation presents several new algorithms incorporating non-parametric and deep learning approaches for computer vision and related tasks, including object localization, object tracking and model compression. With respect to object localization, I introduce a method to perform active localization by modeling spatial and other relationships between objects in a coherent "visual situation" using a set of probability distributions. I further refine this approach with the Multipole Density Estimation with Importance Clustering (MIC-Situate) algorithm. Next, I formulate active, "situation" object search as a Bayesian optimization problem using Gaussian Processes. Using my Gaussian Process Context Situation Learning (GP-CL) algorithm, I demonstrate improved …

Novel View Synthesis In Time And Space, Simon Niklaus

Dissertations and Theses

Novel view synthesis is a classic problem in computer vision. It refers to the generation of previously unseen views of a scene from a set of sparse input images taken from different viewpoints. One example of novel view synthesis is the interpolation of views in between the two images of a stereo camera. Another classic problem in computer vision is video frame interpolation, which is important for video processing. It refers to the generation of video frames in between existing ones and is commonly used to increase the frame rate of a video or to match the frame rate to …

An Application Of Deep Learning Models To Automate Food Waste Classification, Alejandro Zachary Espinoza

Dissertations and Theses

Food wastage is a problem that affects all demographics and regions of the world. Each year, approximately one-third of food produced for human consumption is thrown away. In an effort to track and reduce food waste in the commercial sector, some companies utilize third party devices which collect data to analyze individual contributions to the global problem. These devices track the type of food wasted (such as vegetables, fruit, boneless chicken, pasta) along with the weight. Some devices also allow the user to leave the food in a kitchen container while it is weighed, so the container weight must also …

Bounding Box Improvement With Reinforcement Learning, Andrew Lewis Cleland

Dissertations and Theses

In this thesis, I explore a reinforcement learning technique for improving bounding box localizations of objects in images. The model takes as input a bounding box already known to overlap an object and aims to improve the fit of the box through a series of transformations that shift the location of the box by translation, or change its size or aspect ratio. Over the course of these actions, the model adapts to new information extracted from the image. This active localization approach contrasts with existing bounding-box regression methods, which extract information from the image only once. I implement, train, and …

Cox Processes For Counting By Detection, Purnima Rajan, Yongming Ma, Bruno Jedynak

Portland Institute for Computational Science Publications

In this work, doubly stochastic Poisson (Cox) processes and convolutional neural net (CNN) classifiers are used to estimate the number of instances of an object in an image. Poisson processes are well suited to model events that occur randomly in space, such as the location of objects in an image or the enumeration of objects in a scene. The proposed algorithm selects a subset of bounding boxes in the image domain, then queries them for the presence of the object of interest by running a pre-trained CNN classifier. The resulting observations are then aggregated, and a posterior distribution over the …

Gaussian Processes With Context-Supported Priors For Active Object Localization, Bruno Jedynak

Portland Institute for Computational Science Publications

We devise an algorithm using a Bayesian optimization framework in conjunction with contextual visual data for the efficient localization of objects in still images. Recent research has demonstrated substantial progress in object localization and related tasks for computer vision. However, many current state-of-the-art object localization procedures still suffer from inaccuracy and inefficiency, in addition to failing to provide a principled and interpretable system amenable to high-level vision tasks. We address these issues with the current research.

Our method encompasses an active search procedure that uses contextual data to generate initial bounding-box proposals for a target object. We train a convolutional …

Improved Scoring Models For Semantic Image Retrieval Using Scene Graphs, Erik Timothy Conser

Dissertations and Theses

Image retrieval via a structured query is explored in Johnson, et al. [7]. The query is structured as a scene graph and a graphical model is generated from the scene graph's object, attribute, and relationship structure. Inference is performed on the graphical model with candidate images and the energy results are used to rank the best matches. In [7], scene graph objects that are not in the set of recognized objects are not represented in the graphical model. This work proposes and tests two approaches for modeling the unrecognized objects in order to leverage the attribute and relationship models to …

Refining Bounding-Box Regression For Object Localization, Naomi Lynn Dickerson

Dissertations and Theses

For the last several years, convolutional neural network (CNN) based object detection systems have used a regression technique to predict improved object bounding boxes based on an initial proposal using low-level image features extracted from the CNN. In spite of its prevalence, there is little critical analysis of bounding-box regression or in-depth performance evaluation. This thesis surveys an array of techniques and parameter settings in order to further optimize bounding-box regression and provide guidance for its implementation. I refute a claim regarding training procedure, and demonstrate the effectiveness of using principal component analysis to handle unwieldy numbers of features produced …

Bayesian Optimization For Refining Object Proposals, With An Application To Pedestrian Detection, Anthony D. Rhodes

Student Research Symposium

We devise an algorithm using a Bayesian optimization framework in conjunction with contextual visual data for the efficient localization of objects in still images. Recent research has demonstrated substantial progress in object localization and related tasks for computer vision. However, many current state-of-the-art object localization procedures still suffer from inaccuracy and inefficiency, in addition to failing to successfully leverage contextual data. We address these issues with the current research.

Our method encompasses an active search procedure that uses contextual data to generate initial bounding-box proposals for a target object. We train a convolutional neural network to approximate an offset distance …

Image Stitching: Handling Parallax, Stereopsis, And Video, Fan Zhang

Dissertations and Theses

Panorama stitching increases the field of view in an image by assembling multiple views together. Traditional stitching techniques are proven to be effective only when dealing with parallax-free monocular images. Many challenges that remain unsolved in the stitching research area include how to stitch monocular images with large parallax, how to stitch stereoscopic images to maintain their stereoscopic consistency and original disparity distribution, and how to create panoramic videos with temporally coherent content. To provide more powerful stitching techniques with more universality, we first develop a parallax-tolerant image stitching technique. With the help of it, we then effectively extend the …

Vision-Based Motion For A Humanoid Robot, Khalid Abdullah Alkhulayfi

Dissertations and Theses

The overall objective of this thesis is to build an integrated, inexpensive, human-sized humanoid robot from scratch that looks and behaves like a human. More specifically, my goal is to build an android robot called Marie Curie robot that can act like a human actor in the Portland Cyber Theater in the play Quantum Debate with a known script of every robot behavior. In order to achieve this goal, the humanoid robot need to has degrees of freedom (DOF) similar to human DOFs. Each part of the Curie robot was built to achieve the goal of building a complete humanoid …

Active Object Localization In Visual Situations, Max H. Quinn, Anthony Rhodes, Melanie Mitchell

Computer Science Faculty Publications and Presentations

—We describe a method for performing active localization of objects in instances of visual situations. A visual situation is an abstract concept—e.g., “a boxing match”, “a birthday party”, “walking the dog”, “waiting for a bus”—whose image instantiations are linked more by their common spatial and semantic structure than by low-level visual similarity. Our system combines given and learned knowledge of the structure of a particular situation, and adapts that knowledge to a new situation instance as it actively searches for objects. More specifically, the system learns a set of probability distributions describing spatial and other relationships among relevant objects. The …

Investigations Of An "Objectness" Measure For Object Localization, Lewis Richard James Coates

Dissertations and Theses

Object localization is the task of locating objects in an image, typically by finding bounding boxes that isolate those objects. Identifying objects in images that have not had regions of interest labeled by humans often requires object localization to be performed first. The sliding window method is a common naïve approach, wherein the image is covered with bounding boxes of different sizes that form windows in the image. An object classifier is then run on each of these windows to determine if each given window contains a given object. However, because object classification algorithms tend to be computationally expensive, it …

Collecting Image Cropping Dataset: A Hybrid System Of Machine And Human Intelligence, Uyen T. Mai, Feng Liu

Student Research Symposium

Image cropping is a common tool that exists in almost any image editor, yet automatic cropping is still a difficult problem in Computer Vision. Since images nowadays can be easily collected through the web, machine learning is a promising approach to solve this problem. However, an image cropping dataset is not yet available and gathering such a large-scale dataset is a non-trivial task. Although a crowdsourcing website such as Mechanical Turk seems to be a solution to this task, image cropping is a sophisticated task that is vulnerable to unreliable annotation; furthermore, collecting a large-scale high-quality dataset through crowdsourcing is …

The Performance Of Random Prototypes In Hierarchical Models Of Vision, Kendall Lee Stewart

Dissertations and Theses

I investigate properties of HMAX, a computational model of hierarchical processing in the primate visual cortex. High-level cortical neurons have been shown to respond highly to particular natural shapes, such as faces. HMAX models this property with a dictionary of natural shapes, called prototypes, that respond to the presence of those shapes. The resulting set of similarity measurements is an effective descriptor for classifying images. Curiously, prior work has shown that replacing the dictionary of natural shapes with entirely random prototypes has little impact on classification performance. This work explores that phenomenon by studying the performance of random prototypes on …

Leveraging Contextual Relationships Between Objects For Localization, Clinton Leif Olson

Dissertations and Theses

Object localization is currently an active area of research in computer vision. The object localization task is to identify all locations of an object class within an image by drawing a bounding box around objects that are instances of that class. Object locations are typically found by computing a classification score over a small window at multiple locations in the image, based on some chosen criteria, and choosing the highest scoring windows as the object bounding-boxes. Localization methods vary widely, but there is a growing trend towards methods that are able to make localization more accurate and efficient through the …

Using Gist Features To Constrain Search In Object Detection, Joanna Browne Solmon

Dissertations and Theses

This thesis investigates the application of GIST features [13] to the problem of object detection in images. Object detection refers to locating instances of a given object category in an image. It is contrasted with object recognition, which simply decides whether an image contains an object, regardless of the object's location in the image.

In much of computer vision literature, object detection uses a "sliding window" approach to finding objects in an image. This requires moving various sizes of windows across an image and running a trained classifier on the visual features of each window. This brute force method can …

The Role Of Prototype Learning In Hierarchical Models Of Vision, Michael David Thomure

Dissertations and Theses

I conduct a study of learning in HMAX-like models, which are hierarchical models of visual processing in biological vision systems. Such models compute a new representation for an image based on the similarity of image sub-parts to a number of specific patterns, called prototypes. Despite being a central piece of the overall model, the issue of choosing the best prototypes for a given task is still an open problem. I study this problem, and consider the best way to increase task performance while decreasing the computational costs of the model. This work broadens our understanding of HMAX and related hierarchical …

Object Detection And Recognition In Natural Settings, George William Dittmar

Dissertations and Theses

Much research as of late has focused on biologically inspired vision models that are based on our understanding of how the visual cortex processes information. One prominent example of such a system is HMAX [17]. HMAX attempts to simulate the biological process for object recognition in cortex based on the model proposed by Hubel & Wiesel [10]. This thesis investigates the ability of an HMAX-like system (GLIMPSE [20]) to perform object-detection in cluttered natural scenes. I evaluate these results using the StreetScenes database from MIT [1, 8]. This thesis addresses three questions: (1) Can the GLIMPSE-based object detection system replicate …

Interpreting Individual Classifications Of Hierarchical Networks, Will Landecker, Michael David Thomure, Luis M.A. Bettencourt, Melanie Mitchell, Garrett T. Kenyon, Steven P. Brumby

Computer Science Faculty Publications and Presentations

Hierarchical networks are known to achieve high classification accuracy on difficult machine-learning tasks. For many applications, a clear explanation of why the data was classified a certain way is just as important as the classification itself. However, the complexity of hierarchical networks makes them ill-suited for existing explanation methods. We propose a new method, contribution propagation, that gives per-instance explanations of a trained network's classifications. We give theoretical foundations for the proposed method, and evaluate its correctness empirically. Finally, we use the resulting explanations to reveal unexpected behavior of networks that achieve high accuracy on visual object-recognition tasks using well-known …

On The Role Of Shape Prototypes In Hierarchical Models Of Vision, Michael David Thomure, Melanie Mitchell, Garrett T. Kenyon

Computer Science Faculty Publications and Presentations

We investigate the role of learned shape-prototypes in an influential family of hierarchical neural-network models of vision. Central to these networks’ design is a dictionary of learned shapes, which are meant to respond to discriminative visual patterns in the input. While higher-level features based on such learned prototypes have been cited as key for viewpointinvariant object-recognition in these models [1], [2], we show that high performance on invariant object-recognition tasks can be obtained by using a simple set of unlearned, “shape-free” features. This behavior is robust to the size of the network. These results call into question the roles of …

Understanding Classification Decisions For Object Detection, Will Landecker, Michael David Thomure, Melanie Mitchell

Systems Science Friday Noon Seminar Series

Computer vision systems are traditionally tested in the object detection paradigm. In these experiments, a vision system is asked whether or not a specific object--for example an animal--occurs in a given image. A system that often answers correctly is said to be very accurate. In this talk, we will discuss some ambiguity that exists in this measure of accuracy. We will also propose a new measure of object-detection accuracy that addresses some of this ambiguity, and apply this measure to the hierarchical "standard model" of visual cortex.