Graphics and Human Computer Interfaces Commons^™

Metaformer Is Actually What You Need For Vision, Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, Shuicheng Yan

Research Collection School Of Computing and Information Systems

Transformers have shown great potential in computer vision tasks. A common belief is their attention-based token mixer module contributes most to their competence. However, recent works show the attention-based module in transformers can be replaced by spatial MLPs and the resulted models still perform quite well. Based on this observation, we hypothesize that the general architecture of the transformers, instead of the specific token mixer module, is more essential to the model's performance. To verify this, we deliberately replace the attention module in transformers with an embarrassingly simple spatial pooling operator to conduct only basic token mixing. Surprisingly, we observe …

Faithful Extreme Rescaling Via Generative Prior Reciprocated Invertible Representations, Zhixuan Zhong, Liangyu Chai, Yang Zhou, Bailin Deng, Jia Pan, Shengfeng He

Research Collection School Of Computing and Information Systems

This paper presents a Generative prior ReciprocAted Invertible rescaling Network (GRAIN) for generating faithful high-resolution (HR) images from low-resolution (LR) invertible images with an extreme upscaling factor (64×). Previous researches have leveraged the prior knowledge of a pretrained GAN model to generate high-quality upscaling results. However, they fail to produce pixel-accurate results due to the highly ambiguous extreme mapping process. We remedy this problem by introducing a reciprocated invertible image rescaling process, in which high-resolution information can be delicately embedded into an invertible low-resolution image and generative prior for a faithful HR reconstruction. In particular, the invertible LR features not …

Group Contextualization For Video Recognition, Yanbin Hao, Hao Zhang, Chong-Wah Ngo, Xiangnan He

Research Collection School Of Computing and Information Systems

Learning discriminative representation from the complex spatio-temporal dynamic space is essential for video recognition. On top of those stylized spatio-temporal computational units, further refining the learnt feature with axial contexts is demonstrated to be promising in achieving this goal. However, previous works generally focus on utilizing a single kind of contexts to calibrate entire feature channels and could hardly apply to deal with diverse video activities. The problem can be tackled by using pair-wise spatio-temporal attentions to recompute feature response with cross-axis contexts at the expense of heavy computations. In this paper, we propose an efficient feature refinement method that …

Revisiting Local Descriptor For Improved Few-Shot Classification, Jun He, Richang Hong, Xueliang Liu, Mingliang Xu, Qianru Sun

Research Collection School Of Computing and Information Systems

Few-shot classification studies the problem of quickly adapting a deep learner to understanding novel classes based on few support images. In this context, recent research efforts have been aimed at designing more and more complex classifiers that measure similarities between query and support images but left the importance of feature embeddings seldom explored. We show that the reliance on sophisticated classifiers is not necessary, and a simple classifier applied directly to improved feature embeddings can instead outperform most of the leading methods in the literature. To this end, we present a new method, named DCAP, for few-shot classification, in which …

Flavor-Videos: Enhancing The Flavor Perception Of Food While Eating With Videos, Meetha Nesam James, Nimesha Ranasinghe, Anthony Tang, Lora Oehlberg

Research Collection School Of Computing and Information Systems

People are typically involved in different activities while eating, particularly when eating alone, such as watching television or playing games on their phones. Previous research in Human-Food Interaction (HFI) has primarily focused on studying people’s motivation and analyzing of the media content watched while eating. However, their impact on human behavioral and cognitive processes, particularly flavor perception and its attributes, remains underexplored. We present a user study to investigate the influence of six types of videos, including mukbang – a new food video genre, on flavor perceptions (taste sensations, liking, and emotions) while eating plain white rice. Our findings revealed …

Out-Of-Core Gpu Path Tracing On Large Instanced Scenes Via Geometry Streaming, Jeremy Berchtold

Master's Theses

We present a technique for out-of-core GPU path tracing of arbitrarily large scenes that is compatible with hardware-accelerated ray-tracing. Our technique improves upon previous works by subdividing the scene spatially into streamable chunks that are loaded using a priority system that maximizes ray throughput and minimizes GPU memory usage. This allows for arbitrarily large scaling of scene complexity. Our system required under 19 minutes to render a solid color version of Disney's Moana Island scene (39.3 million instances, 261.1 million unique quads, and 82.4 billion instanced quads at a resolution of 1024x429 and 1024spp on an RTX 5000 (24GB memory …

Mlp-3d: A Mlp-Like 3d Architecture With Grouped Time Mixing, Zhaofan Qiu, Ting Yao, Chong-Wah Ngo, Tao Mei

Research Collection School Of Computing and Information Systems

Convolutional Neural Networks (CNNs) have been re-garded as the go-to models for visual recognition. More re-cently, convolution-free networks, based on multi-head self-attention (MSA) or multi-layer perceptrons (MLPs), become more and more popular. Nevertheless, it is not trivial when utilizing these newly-minted networks for video recognition due to the large variations and complexities in video data. In this paper, we present MLP-3D networks, a novel MLP-like 3D architecture for video recognition. Specifically, the architecture consists of MLP-3D blocks, where each block contains one MLP applied across tokens (i.e., token-mixing MLP) and one MLP applied independently to each token (i.e., channel MLP). …

Adaptive Temporal Grouping For Black-Box Adversarial Attacks On Videos, Zhipeng Wei, Jingjing Chen, Hao Zhang, Linxi Jiang, Yu-Gang Jiang

Research Collection School Of Computing and Information Systems

Deep-learning based video models, which have remarkable performance on action recognition tasks, are recently proved to be vulnerable to adversarial samples, even those generated in the black-box setting. However, these black-box attack methods are insufficient to attack videos models in real-world applications due to the requirement of lots of queries. To this end, we propose to boost the efficiency of black-box attacks on video recognition models. Although videos carry rich temporal information, they include redundant spatial information from adjacent frames. This motivates us to introduce the adaptive temporal grouping (ATG) method, which groups video frames by the similarity of their …

Multimodal Zero-Shot Hateful Meme Detection, Jiawen Zhu, Roy Ka-Wei Lee, Wen Haw Chong

Research Collection School Of Computing and Information Systems

Facebook has recently launched the hateful meme detection challenge, which garnered much attention in academic and industry research communities. Researchers have proposed multimodal deep learning classification methods to perform hateful meme detection. While the proposed methods have yielded promising results, these classification methods are mostly supervised and heavily rely on labeled data that are not always available in the real-world setting. Therefore, this paper explores and aims to perform hateful meme detection in a zero-shot setting. Working towards this goal, we propose Target-Aware Multimodal Enhancement (TAME), which is a novel deep generative framework that can improve existing hateful meme classification …

A Simple Data Mixing Prior For Improving Self-Supervised Learning, Sucheng Ren, Huiyu Wang, Zhengqi Gao, Shengfeng He, Alan Yuille, Yuyin Zhou, Cihang Xie

Research Collection School Of Computing and Information Systems

Data mixing (e.g., Mixup, Cutmix, ResizeMix) is an essential component for advancing recognition models. In this paper, we focus on studying its effectiveness in the self-supervised setting. By noticing the mixed images that share the same source images are intrinsically related to each other, we hereby propose SDMP, short for Simple Data Mixing Prior, to capture this straightforward yet essential prior, and position such mixed images as additional positive pairs to facilitate self-supervised representation learning. Our experiments verify that the proposed SDMP enables data mixing to help a set of self-supervised learning frameworks (e.g., MoCo) achieve better accuracy and out-of-distribution …

High-Resolution Face Swapping Via Latent Semantics Disentanglement, Yangyang Xu, Bailin Deng, Junle Wang, Yanqing Jing, Jia Pan, Shengfeng He

Research Collection School Of Computing and Information Systems

We present a novel high-resolution face swapping method using the inherent prior knowledge of a pre-trained GAN model. Although previous research can leverage generative priors to produce high-resolution results, their quality can suffer from the entangled semantics of the latent space. We explicitly disentangle the latent semantics by utilizing the progressive nature of the generator, deriving structure at-tributes from the shallow layers and appearance attributes from the deeper ones. Identity and pose information within the structure attributes are further separated by introducing a landmark-driven structure transfer latent direction. The disentangled latent code produces rich generative features that incorporate feature blending …

Cross-Lingual Adaptation For Recipe Retrieval With Mixup, Bin Zhu, Chong-Wah Ngo, Jingjing Chen, Wing-Kwong Chan

Research Collection School Of Computing and Information Systems

Cross-modal recipe retrieval has attracted research attention in recent years, thanks to the availability of large-scale paired data for training. Nevertheless, obtaining adequate recipe-image pairs covering the majority of cuisines for supervised learning is difficult if not impossible. By transferring knowledge learnt from a data-rich cuisine to a data-scarce cuisine, domain adaptation sheds light on this practical problem. Nevertheless, existing works assume recipes in source and target domains are mostly originated from the same cuisine and written in the same language. This paper studies unsupervised domain adaptation for image-to-recipe retrieval, where recipes in source and target domains are in different …

Reinforcement Learning-Based Interactive Video Search, Zhixin Ma, Jiaxin Wu, Zhijian Hou, Chong-Wah Ngo

Research Collection School Of Computing and Information Systems

Despite the rapid progress in text-to-video search due to the advancement of cross-modal representation learning, the existing techniques still fall short in helping users to rapidly identify the search targets. Particularly, in the situation that a system suggests a long list of similar candidates, the user needs to painstakingly inspect every search result. The experience is frustrated with repeated watching of similar clips, and more frustratingly, the search targets may be overlooked due to mental tiredness. This paper explores reinforcement learning-based (RL) searching to relieve the user from the burden of brute force inspection. Specifically, the system maintains a graph …

A Bidirectional Formulation For Walk On Spheres, Yang Qi

Dartmouth College Master’s Theses

Poisson’s equations and Laplace’s equations are important linear partial differential equations (PDEs)
widely used in many applications. Conventional methods for solving PDEs numerically often need to
discretize the space first, making them less efficient for complex shapes. The random walk on spheres
method (WoS) is a grid-free Monte-Carlo method for solving PDEs that does not need to discrete the
space. We draw analogies between WoS and classical rendering algorithms, and find that the WoS
algorithm is conceptually identical to forward path tracing.
We show that solving the Poisson’s equation is equivalent to solving the Green’s function for every
pair of …

Increasing Perceived Realism Of Objects In A Mixed Reality Environment Using 'Diminished Virtual Reality', Logan Scott Parker

Honors Theses

With the recent explosion of popularity of virtual and mixed reality, an important question has arisen: “Is there a way to create a better blend of real and virtual worlds in a mixed reality experience?” This research attempts to determine whether a visual filter can be created and applied to virtual objects to better convince the brain into interpreting a composite of virtual and real views as one seamless view. The method devised in this thesis is being called 'Diminished Virtual Reality'. The results found in this study show that when presented with a scene composed of a combination of …

Pluto In Hand: Design And Implementation Of A Location-Based Mobile Augmented Reality Application For Viewing Open Data, Matthew O. Ward

Theses and Dissertations

Immersive mobile augmented reality (AR) technology has improved while geolocational data volume has grown. City governments can utilize this technology to share their geospatial data with the public, promoting smart city aims. This research describes the design and implementation of a novel open-source ARGIS application to view property tax lot information in New York City. This proof-of-technology demonstrates web-based AR can visualize location-based spatial data.

Automated Filament Inking For Multi-Color Fff 3d Printing, Eammon Littler

Dartmouth College Master’s Theses

We propose a novel system for low-cost multi-color Fused Filament Fabrication (FFF) 3D printing, allowing for the creation of customizable colored filament using a pre-processing approach. We developed an open-source device to automatically ink filament using permanent markers. Our device can be built using 3D printed parts and off-the-shelf electronics. An accompanying web-based interface allows users to view GCODE toolpaths for a multi-color print and quickly generate filament color profiles. Taking a pre-processing approach makes this system compatible with the majority of desktop 3D printers on the market, as the processed filament behaves no differently from conventional filaments. Furthermore, inked …

Gauging The State-Of-The-Art For Foresight Weight Pruning On Neural Networks, Noah James

Computer Science and Computer Engineering Undergraduate Honors Theses

The state-of-the-art for pruning neural networks is ambiguous due to poor experimental practices in the field. Newly developed approaches rarely compare to each other, and when they do, their comparisons are lackluster or contain errors. In the interest of stabilizing the field of pruning, this paper initiates a dive into reproducing prominent pruning algorithms across several architectures and datasets. As a first step towards this goal, this paper shows results for foresight weight pruning across 6 baseline pruning strategies, 5 modern pruning strategies, random pruning, and one legacy method (Optimal Brain Damage). All strategies are evaluated on 3 different architectures …

An Investigation Into, And The Construction Of, An Operable Windows Notifier, Grey Hixson

Computer Science and Computer Engineering Undergraduate Honors Theses

The Office of Sustainability at the University of Arkansas identified that building occupants that have control over operable windows may open them at inappropriate times. Windows opened in a building with a temperature and air differential leads to increased HVAC operating costs and building occupant discomfort. This led the Associate Vice Chancellor of Facilities at the University of Arkansas to propose the construction of a mobile application that a building occupant can use to make an informed decision before opening their window. I have formulated a series of research objectives in conjunction with the Director of the Office of Sustainability …

Analysis Of Gpu Memory Vulnerabilities, Jarrett Hoover

Computer Science and Computer Engineering Undergraduate Honors Theses

Graphics processing units (GPUs) have become a widely used technology for various purposes. While their intended use is accelerating graphics rendering, their parallel computing capabilities have expanded their use into other areas. They are used in computer gaming, deep learning for artificial intelligence and mining cryptocurrencies. Their rise in popularity led to research involving several security aspects, including this paper’s focus, memory vulnerabilities. Research documented many vulnerabilities, including GPUs not implementing address space layout randomization, not zeroing out memory after deallocation, and not initializing newly allocated memory. These vulnerabilities can lead to a victim’s sensitive data being leaked to an …

Using A Bert-Based Ensemble Network For Abusive Language Detection, Noah Ballinger

Computer Science and Computer Engineering Undergraduate Honors Theses

Over the past two decades, online discussion has skyrocketed in scope and scale. However, so has the amount of toxicity and offensive posts on social media and other discussion sites. Despite this rise in prevalence, the ability to automatically moderate online discussion platforms has seen minimal development. Recently, though, as the capabilities of artificial intelligence (AI) continue to improve, the potential of AI-based detection of harmful internet content has become a real possibility. In the past couple years, there has been a surge in performance on tasks in the field of natural language processing, mainly due to the development of …

A Study Of Software Development Methodologies, Kendra Risener

Computer Science and Computer Engineering Undergraduate Honors Theses

Software development methodologies are often overlooked by software engineers as aspects of development that are handled by project managers alone. However, if every member of the team better understood the development methodology being used, it increases the likelihood that the method is properly implemented and ultimately used to complete the project more efficiently. Thus, this paper seeks to explore six common methodologies: the Waterfall Model, the Spiral Model, Agile, Scrum, Kanban, and Extreme Programming. These are discussed in two main sections in the paper. In the first section, the frameworks are isolated and viewed by themselves. The histories, unique features, …

Using Bluetooth Low Energy And E-Ink Displays For Inventory Tracking, David Whelan

Computer Science and Computer Engineering Undergraduate Honors Theses

The combination of Bluetooth Low energy and E-Ink displays allow for a low energy wire-less display. The application of this technology is far reaching especially given how the Bluetooth Low Energy specification can be extended. This paper proposes an extension to this specification specifically for inventory tracking. This extension combined with the low energy E-Ink display results in a smart label that can keep track of additional meta data and inventory counts for physical inventory. This label helps track the physical inventory and can help mitigate any errors in the logical organization of inventory.

Implementing The Cms+ Sports Rankings Algorithm In A Javafx Environment, Luke Welch

Industrial Engineering Undergraduate Honors Theses

Every year, sports teams and athletes get cut from championship opportunities because of their rank. While this reality is easier to swallow if a team or athlete is distant from the cut, it is much harder when they are right on the edge. Many times, it leaves fans and athletes wondering, “Why wasn’t I ranked higher? What factors when into the ranking? Are the rankings based on opinion alone?” These are fair questions that deserve an answer. Many times, sports rankings are derived from opinion polls. Other times, they are derived from a combination of opinion polls and measured performance. …

Designing Visuo-Haptic Illusions With Proxies In Virtual Reality: Exploration Of Grasp, Movement Trajectory And Object Mass, Martin Feick, Kora Persephone Regitz, Anthony Tang, Antonio Kruger

Research Collection School Of Computing and Information Systems

Visuo-haptic illusions are a method to expand proxy-based interactions in VR by introducing unnoticeable discrepancies between the virtual and real world. Yet how different design variables affect the illusions with proxies is still unclear. To unpack a subset of variables, we conducted two user studies with 48 participants to explore the impact of (1) different grasping types and movement trajectories, and (2) different grasping types and object masses on the discrepancy which may be introduced. Our Bayes analysis suggests that grasping types and object masses (≤ 500 g) did not noticeably affect the discrepancy, but for movement trajectory, results were …

Tourgether360: Exploring 360° Tour Videos With Others, Kartikaeya Kumar, Lev Poretski, Jiannan Li, Anthony Tang

Research Collection School Of Computing and Information Systems

Contemporary 360° video players do not provide ways to let people explore the videos together. Tourgether360 addresses this problem for 360° tour videos using a pseudo-spatial navigation technique that provides both an overhead “context” view of the environment as a minimap, as well as a shared pseudo-3D environment for exploring the video. Collaborators appear as avatars along a track depending on their position in the video timeline and can point and synchronize their playback. In this work, we describe the intellectual precedents for this work, our design goals, and our implementation approach of Tourgether360. Finally, we discuss future work based …

Mmekg: Multi-Modal Event Knowledge Graph Towards Universal Representation Across Modalities, Yubo Ma, Zehao Wang, Mukai Li, Yixin Cao, Meiqi Chen, Xinze Li, Wenqi Sun, Kunquan Deng, Kun Wang, Aixin Sun, Jing Shao

Research Collection School Of Computing and Information Systems

Events are fundamental building blocks of realworld happenings. In this paper, we present a large-scale, multi-modal event knowledge graph named MMEKG. MMEKG unifies different modalities of knowledge via events, which complement and disambiguate each other. Specifically, MMEKG incorporates (i) over 990 thousand concept events with 644 relation types to cover most types of happenings, and (ii) over 863 million instance events connected through 934 million relations, which provide rich contextual information in texts and/or images. To collect billion-scale instance events and relations among them, we additionally develop an efficient yet effective pipeline for textual/visual knowledge extraction system. We also develop …

Prompt For Extraction? Paie: Prompting Argument Interaction For Event Argument Extraction, Yubo Ma, Zehao Wang, Yixin Cao, Mukai Li, Meiqi Chen, Kun Wang, Jing Shao

Research Collection School Of Computing and Information Systems

In this paper, we propose an effective yet efficient model PAIE for both sentence-level and document-level Event Argument Extraction (EAE), which also generalizes well when there is a lack of training data. On the one hand, PAIE utilizes prompt tuning for extractive objectives to take the best advantages of Pre-trained Language Models (PLMs). It introduces two span selectors based on the prompt to select start/end tokens among input texts for each role. On the other hand, it captures argument interactions via multi-role prompts and conducts joint optimization with optimal span assignments via a bipartite matching loss. Also, with a flexible …

Designing And Evaluating Accessible E-Learning For Students With Visual Impairments In K-12 Computing Education, Earl W. Huff Jr

All Dissertations

This dissertation explores the pathways for making K-12 computing education more accessible for blind or visually impaired (BVI) learners. As computer science (CS) expands into K-12 education, more concerted efforts are required to ensure all students have equitable access to opportunities to pursue a career in computing. To determine their viability with BVI learners, I conducted three studies to assess current accessibility in CS curricula, materials, and learning environments. Study one was interviews with visually impaired developers; study two was interviews with K-12 teachers of visually impaired students; study three was a remote observation within a computer science course. My …

Stumbling Into Virtual Worlds. How Resolution Affects Users’ Immersion In Virtual Reality And Implications For Virtual Reality In Therapeutic Applications, Brianna Martinson

Undergraduate Honors Theses

Studies of how users experience Virtual Reality (VR) have thus far failed to address the extent to which rendering resolution and rendering frame rate affect users’ sense of immersion in VR, including applications of VR involving simulators, treatments for psychological and mental disorders, explorations of new and nonexistent structures, and ways to better understand the human body in medical applications.

This study investigated if rendering resolution affected users’ sense of immersion in VR. This was conducted by comparing the responses of two groups, relative to two measures of participant immersion: (a) participant’s sense of presence and (b) participant’s sense of …