Enhancing Place-Based Interaction With Emotion Ai And Augmented Reality,
2024
California Polytechnic State University, San Luis Obispo
Enhancing Place-Based Interaction With Emotion Ai And Augmented Reality, Jake Maier, Ivan Martinez
College of Engineering Summer Undergraduate Research Program
This project explores the integration of augmented reality (AR) and Emotion AI technologies to enhance user experiences in physical environments. By seamlessly merging virtual elements with real-world contexts, we aim to deepen individuals’ interactions and perceptions of their surroundings. Leveraging AR technology enables users to access contextual information, engage with interactive content, and navigate spaces with heightened immersion and understanding. Additionally, Emotion AI enhances these experiences by detecting and responding to users’ emotional states, fostering personalized and emotionally resonant interactions. We aim to integrate digital content within physical environments using mixed-reality headsets equipped with eye-tracking capabilities and consumer-grade wireless EEG …
Leveraging Tradespace-Exploration For A Senior Project Team Formation Application,
2024
California Polytechnic State University, San Luis Obispo
Leveraging Tradespace-Exploration For A Senior Project Team Formation Application, Miguel Saenz
College of Engineering Summer Undergraduate Research Program
This project revolves around the development of an app in MATLAB that leverages the VASSAR rule-based system and a genetic algorithm to form groups of teams for the Mechanical Engineering Senior Design project class. We leveraged the iterative design process to eventually attain a functional app with a reasonable runtime that works provided correctly formatted rulesheets describing student project preference and member preference.
Gradualreality : Enhancing Physical Object Interaction In Virtual Reality Via Interaction State-Aware Blending,
2024
Singapore Management University
Gradualreality : Enhancing Physical Object Interaction In Virtual Reality Via Interaction State-Aware Blending, Hyuna Seo, Juheon Yi, Rajesh Krishna Balan, Youngki Lee
Research Collection School Of Computing and Information Systems
We present GradualReality, a novel interface enabling a Cross Reality experience that includes gradual interaction with physical objects in a virtual environment and supports both presence and usability. Daily Cross Reality interaction is challenging as the user’s physical object interaction state is continuously changing over time, causing their attention to frequently shift between the virtual and physical worlds. As such, presence in the virtual environment and seamless usability for interacting with physical objects should be maintained at a high level. To address this issue, we present an Interaction State-Aware Blending approach that (i) balances immersion and interaction capability and (ii) …
Improving Out-Of-Distribution Detection With Disentangled Foreground And Background Features,
2024
Singapore Management University
Improving Out-Of-Distribution Detection With Disentangled Foreground And Background Features, Choubo Ding, Guansong Pang
Research Collection School Of Computing and Information Systems
Detecting out-of-distribution (OOD) inputs is a principal task for ensuring the safety of deploying deep-neural-network classifiers in open-set scenarios. OOD samples can be drawn from arbitrary distributions and exhibit deviations from in-distribution (ID) data in various dimensions, such as foreground features (e.g., objects in CIFAR100 images vs. those in CIFAR10 images) and background features (e.g., textural images vs. objects in CIFAR10). Existing methods can confound foreground and background features in training, failing to utilize the background features for OOD detection. This paper considers the importance of feature disentanglement in out-of-distribution detection and proposes the simultaneous exploitation of both foreground and …
Enhancing Recipe Retrieval With Foundation Models: A Data Augmentation Perspective,
2024
Singapore Management University
Enhancing Recipe Retrieval With Foundation Models: A Data Augmentation Perspective, Fangzhou Song, Bin Zhu, Yanbin Hao, Shuo Wang
Research Collection School Of Computing and Information Systems
Learning recipe and food image representation in common embedding space is non-trivial but crucial for cross-modal recipe retrieval. In this paper, we propose a new perspective for this problem by utilizing foundation models for data augmentation. Leveraging on the remarkable capabilities of foundation models (i.e., Llama2 and SAM), we propose to augment recipe and food image by extracting alignable information related to the counterpart. Specifically, Llama2 is employed to generate a textual description from the recipe, aiming to capture the visual cues of a food image, and SAM is used to produce image segments that correspond to key ingredients in …
Onerestore : A Universal Restoration Framework For Composite Degradation,
2024
Singapore Management University
Onerestore : A Universal Restoration Framework For Composite Degradation, Yu Guo, Yuan Gao, Yuxu Lu, Huilin Zhu, Ryan Wen Liu, Shengfeng He
Research Collection School Of Computing and Information Systems
In real-world scenarios, image impairments often manifest as composite degradations, presenting a complex interplay of elements such as low light, haze, rain, and snow. Despite this reality, existing restoration methods typically target isolated degradation types, thereby falling short in environments where multiple degrading factors coexist. To bridge this gap, our study proposes a versatile imaging model that consolidates four physical corruption paradigms to accurately represent complex, composite degradation scenarios. In this context, we propose OneRestore, a novel transformer-based framework designed for adaptive, controllable scene restoration. The proposed framework leverages a unique cross-attention mechanism, merging degraded scene descriptors with image features, …
An End-To-End Bi-Objective Approach To Deep Graph Partitioning,
2024
Singapore Management University
An End-To-End Bi-Objective Approach To Deep Graph Partitioning, Pengcheng Wei, Yuan Fang, Zhihao Wen, Zheng Xiao, Binbin Chen
Research Collection School Of Computing and Information Systems
Graphs are ubiquitous in real-world applications, such as computation graphs and social networks. Partitioning large graphs into smaller, balanced partitions is often essential, with the biobjective graph partitioning problem aiming to minimize both the“cut” across partitions and the imbalance in partition sizes. However, existing heuristic methods face scalability challenges or overlook partition balance, leading to suboptimal results. Recent deep learning approaches, while promising, typically focus only on node-level features and lack a truly end-to-end framework, resulting in limited performance. In this paper, we introduce a novel method based on graph neural networks (GNNs) that leverages multilevel graph features and addresses …
Video Editing For Video Retrieval,
2024
Singapore Management University
Video Editing For Video Retrieval, Bin Zhu, Kevin Flanagan, Adriano Fragomeni, Michael Wray, Dima Damen
Research Collection School Of Computing and Information Systems
Though pre-training vision-language models have demonstrated significant benefits in boosting video-text retrieval performance from large-scale web videos, fine-tuning still plays a critical role with manually annotated clips with start and end times, which requires considerable human effort. To address this issue, we explore an alternative cheaper source of annotations, single timestamps, for video-text retrieval. We initialise clips from timestamps in a heuristic way to warm up a retrieval model. Then a video clip editing method is proposed to refine the initial rough boundaries to improve retrieval performance. A student-teacher network is introduced for video clip editing: the teacher model is …
Densetrack : Drone-Based Crowd Tracking Via Density-Aware Motion-Appearance Synergy,
2024
Singapore Management University
Densetrack : Drone-Based Crowd Tracking Via Density-Aware Motion-Appearance Synergy, Yi Lei, Huilin Zhu, Jingling Yuan, Guangli Xiang, Xian Zhong, Shengfeng He
Research Collection School Of Computing and Information Systems
Drone-based crowd tracking faces difficulties in accurately identifying and monitoring objects from an aerial perspective, largely due to their small size and close proximity to each other, which complicates both localization and tracking. To address these challenges, we present the Density-aware Tracking (DenseTrack) framework. DenseTrack capitalizes on crowd counting to precisely determine object locations, blending visual and motion cues to improve the tracking of small-scale objects. It specifically addresses the problem of cross-frame motion to enhance tracking accuracy and dependability. DenseTrack employs crowd density estimates as anchors for exact object localization within video frames. These estimates are merged with motion …
Zero-Shot Object Counting With Good Exemplars,
2024
Singapore Management University
Zero-Shot Object Counting With Good Exemplars, Huilin Zhu, Jingling Yuan, Zhengwei Yang, Yu Guo, Zheng Wang, Xian Zhong, Shengfeng He
Research Collection School Of Computing and Information Systems
Zero-shot object counting (ZOC) aims to enumerate objects in images using only the names of object classes during testing, without the need for manual annotations. However, a critical challenge in current ZOC methods lies in their inability to identify high-quality exemplars effectively. This deficiency hampers scalability across diverse classes and undermines the development of strong visual associations between the identified classes and image content. To this end, we propose the Visual Association-based Zero-shot Object Counting (VA-Count) framework. VACount consists of an Exemplar Enhancement Module (EEM) and a Noise Suppression Module (NSM) that synergistically refine the process of class exemplar identification …
Beat-It : Beat-Synchronized Multi-Condition 3d Dance Generation,
2024
Singapore Management University
Beat-It : Beat-Synchronized Multi-Condition 3d Dance Generation, Zikai Huang, Xuemiao Xu, Cheng Xu, Huaidong Zhang, Chenxi Zheng, Jing Qin, Shengfeng He
Research Collection School Of Computing and Information Systems
Dance, as an art form, fundamentally hinges on the precise synchronization with musical beats. However, achieving aesthetically pleasing dance sequences from music is challenging, with existing methods often falling short in controllability and beat alignment. To address these shortcomings, this paper introduces Beat-It, a novel framework for beat-specific, key pose-guided dance generation. Unlike prior approaches, Beat-It uniquely integrates explicit beat awareness and key pose guidance, effectively resolving two main issues: the misalignment of generated dance motions with musical beats, and the inability to map key poses to specific beats, critical for practical choreography. Our approach disentangles beat conditions from music …
Graph Continual Learning With Debiased Lossless Memory Replay,
2024
Singapore Management University
Graph Continual Learning With Debiased Lossless Memory Replay, Chaoxi Niu, Guansong Pang, Ling Chen
Research Collection School Of Computing and Information Systems
Real-life graph data often expands continually, rendering the learning of graph neural networks (GNNs) on static graph data impractical. Graph continual learning (GCL) tackles this problem by continually adapting GNNs to the expanded graph of the current task while maintaining the performance over the graph of previous tasks. Memory replay-based methods, which aim to replay data of previous tasks when learning new tasks, have been explored as one principled approach to mitigate the forgetting of the knowledge learned from the previous tasks. In this paper we extend this methodology with a novel framework, called Debiased Lossless Memory replay (DeLoMe). Unlike …
Pvp-Ssd: Point-Voxel Fusion With Partitioned Point Cloud Sampling For Anchor-Free Single-Stage Small 3d Object Detection,
2024
Singapore Management University
Pvp-Ssd: Point-Voxel Fusion With Partitioned Point Cloud Sampling For Anchor-Free Single-Stage Small 3d Object Detection, Xinlin Wu, Yibin Tian, Yin Pan, Zhiyuan Zhang, Xuesong Wu, Ruisheng Wang, Zhi Zeng
Research Collection School Of Computing and Information Systems
Single-stage object detection from 3D point clouds in autonomous driving faces significant challenges, particularly in accurately detecting small objects. To address this issue, we propose a novel method called Point-Voxel dual-branch feature extraction with Partitioned point cloud sampling for anchor-free Single-Stage Detection of 3D objects (PVP-SSD). The network comprises two branches: a point branch and a voxel branch. In the point branch, a partitioned point cloud sampling strategy leverages axial features to divide the point cloud. Then, it assigns different sampling weights to various segments to enhance the sampling accuracy. Additionally, a local feature enhancement module explicitly calculates the correlation …
Efficient Cascaded Multiscale Adaptive Network For Image Restoration,
2024
Singapore Management University
Efficient Cascaded Multiscale Adaptive Network For Image Restoration, Yichen Zhou, Pan Zhou, Teck Khim Ng
Research Collection School Of Computing and Information Systems
Image restoration, encompassing tasks such as deblurring, denoising, and super-resolution, remains a pivotal area in computer vision. However, efficiently addressing the spatially varying artifacts of various low-quality images with local adaptiveness and handling their degradations at different scales poses significant challenges. To efficiently tackle these issues, we propose the novel Efficient Cascaded Multiscale Adaptive (ECMA) Network. ECMA employs Local Adaptive Module, LAM, which dynamically adjusts convolution kernels across local image regions to efficiently handle varying artifacts. Thus, LAM addresses the local adaptiveness challenge more efficiently than costlier mechanisms like self-attention, due to its less computationally intensive convolutions. To construct a …
Text-Driven Video Prediction,
2024
Fudan University
Text-Driven Video Prediction, Xue Song, Jingjing Chen, Bin Zhu, Yu-Gang Jiang
Research Collection School Of Computing and Information Systems
Current video generation models usually convert signals indicating appearance and motion received from inputs (e.g., image and text) or latent spaces (e.g., noise vectors) into consecutive frames, fulfilling a stochastic generation process for the uncertainty introduced by latent code sampling. However, this generation pattern lacks deterministic constraints for both appearance and motion, leading to uncontrollable and undesirable outcomes. To this end, we propose a new task called Text-driven Video Prediction (TVP). Taking the first frame and text caption as inputs, this task aims to synthesize the following frames. Specifically, appearance and motion components are provided by the image and caption …
Getting To The Point: Contrasting Directness And Warmth In Motivational Embodied Conversational Agents,
2024
Technological University Dublin
Getting To The Point: Contrasting Directness And Warmth In Motivational Embodied Conversational Agents, Michael O'Mahony, Cathy Ennis, Robert Ross
Conference papers
Enhancing long-term engagement with conversational agents remains a significant challenge. Controlling the perceived warmth or directness of an agent’s personality through the style of its generated text could be used to increase user likeability. This paper reports an investigation of a Wizard-of-Oz (WoZ) mediated study of two variants of a motivational embodied conversational agent to measure user perception of and attitudes towards warmth in interaction style. Results show a significant effect of users preferring an agent with a "more direct" personality for this scenario, though this effect is in many ways nuanced.
Imbalanced Graph Classification With Multi-Scale Oversampling Graph Neural Networks,
2024
Singapore Management University
Imbalanced Graph Classification With Multi-Scale Oversampling Graph Neural Networks, Rongrong Ma, Guansong Pang, Ling Chen
Research Collection School Of Computing and Information Systems
One main challenge in imbalanced graph classification is to learn expressive representations of the graphs in under-represented (minority) classes. Existing generic imbalanced learning methods, such as oversampling and imbalanced learning loss functions, can be adopted for enabling graph representation learning models to cope with this challenge. However, these methods often directly operate on the graph representations, ignoring rich discriminative information within the graphs and their interactions. To tackle this issue, we introduce a novel multi-scale oversampling graph neural network (MOSGNN) that learns expressive minority graph representations based on intra- and inter-graph semantics resulting from oversampled graphs at multiple scales - …
Robust Image Classification System Via Cloud Computing, Aligned Multimodal Embeddings, Centroids And Neighbours,
2024
Singapore Management University
Robust Image Classification System Via Cloud Computing, Aligned Multimodal Embeddings, Centroids And Neighbours, Wei Lun Koh, Boon Yong Koh, Bing Tian Dai
Research Collection School Of Computing and Information Systems
We propose a framework for a cloud-based application of an image classification system that is highly accessible, maintains data confidentiality, and robust to incorrect training labels. The end-to-end system is implemented using Amazon Web Services (AWS), with a detailed guide provided for replication, enhancing the ways which researchers can collaborate with a community of users for mutual benefits. A front-end web application allows users across the world to securely log in, contribute labelled training images conveniently via a drag-and-drop approach, and use that same application to query an up-to-date model that has knowledge of images from the community of users. …
Adavis: Adaptive And Explainable Visualization Recommendation For Tabular Data,
2024
Singapore Management University
Adavis: Adaptive And Explainable Visualization Recommendation For Tabular Data, Songheng Zhang, Haotian Li, Huamin Qu, Yong Wang
Research Collection School Of Computing and Information Systems
Automated visualization recommendation facilitates the rapid creation of effective visualizations, which is especially beneficial for users with limited time and limited knowledge of data visualization. There is an increasing trend in leveraging machine learning (ML) techniques to achieve an end-to-end visualization recommendation. However, existing ML-based approaches implicitly assume that there is only one appropriate visualization for a specific dataset, which is often not true for real applications. Also, they often work like a black box, and are difficult for users to understand the reasons for recommending specific visualizations. To fill the research gap, we propose AdaVis, an adaptive and explainable …
Granular3d: Delving Into Multi-Granularity 3d Scene Graph Prediction,
2024
Singapore Management University
Granular3d: Delving Into Multi-Granularity 3d Scene Graph Prediction, Kaixiang Huang, Jingru Yang, Jin Wang, Shengfeng He, Zhan Wang, Haiyan He, Qifeng Zhang, Guodong Lu
Research Collection School Of Computing and Information Systems
This paper addresses the significant challenges in 3D Semantic Scene Graph (3DSSG) prediction, essential for understanding complex 3D environments. Traditional approaches, primarily using PointNet and Graph Convolutional Networks, struggle with effectively extracting multi-grained features from intricate 3D scenes, largely due to a focus on global scene processing and single-scale feature extraction. To overcome these limitations, we introduce Granular3D, a novel approach that shifts the focus towards multi-granularity analysis by predicting relation triplets from specific sub-scenes. One key is the Adaptive Instance Enveloping Method (AIEM), which establishes an approximate envelope structure around irregular instances, providing shape-adaptive local point cloud sampling, thereby …
