M3sa: Multimodal Sentiment Analysis Based On Multi-Scale Feature Extraction And Multi-Task Learning,
2024
Singapore Management University
M3sa: Multimodal Sentiment Analysis Based On Multi-Scale Feature Extraction And Multi-Task Learning, Changkai Lin, Hongju Cheng, Qiang Rao, Yang Yang
Research Collection School Of Computing and Information Systems
Sentiment analysis plays an indispensable part in human-computer interaction. Multimodal sentiment analysis can overcome the shortcomings of unimodal sentiment analysis by fusing multimodal data. However, how to extracte improved feature representations and how to execute effective modality fusion are two crucial problems in multimodal sentiment analysis. Traditional work uses simple sub-models for feature extraction, and they ignore features of different scales and fuse different modalities of data equally, making it easier to incorporate extraneous information and affect analysis accuracy. In this paper, we propose a Multimodal Sentiment Analysis model based on Multi-scale feature extraction and Multi-task learning (M 3 SA). …
From Canteen Food To Daily Meals: Generalizing Food Recognition To More Practical Scenarios,
2024
Fudan University
From Canteen Food To Daily Meals: Generalizing Food Recognition To More Practical Scenarios, Guoshan Liu, Yang Jiao, Jingjing Chen, Bin Zhu, Yu-Gang Jiang
Research Collection School Of Computing and Information Systems
The precise recognition of food categories plays a pivotal role for intelligent health management, attracting significant research attention in recent years. Prominent benchmarks, such as Food-101 and VIREO Food-172, provide abundant food image resources that catalyze the prosperity of research in this field. Nevertheless, these datasets are well-curated from canteen scenarios and thus deviate from food appearances in daily life. This discrepancy poses great challenges in effectively transferring classifiers trained on these canteen datasets to broader daily-life scenarios encountered by humans. Toward this end, we present two new benchmarks, namely DailyFood-172 and DailyFood-16, specifically designed to curate food images from …
Out-Of-Distribution Detection In Long-Tailed Recognition With Calibrated Outlier Class Learning,
2024
Singapore Management University
Out-Of-Distribution Detection In Long-Tailed Recognition With Calibrated Outlier Class Learning, Wenjun Miao, Guansong Pang, Xiao Bai, Tianqi Li, Jin Zheng
Research Collection School Of Computing and Information Systems
Existing out-of-distribution (OOD) methods have shown great success on balanced datasets but become ineffective in long-tailed recognition (LTR) scenarios where 1) OOD samples are often wrongly classified into head classes and/or 2) tail-class samples are treated as OOD samples. To address these issues, current studies fit a prior distribution of auxiliary/pseudo OOD data to the long-tailed in-distribution (ID) data. However, it is difficult to obtain such an accurate prior distribution given the unknowingness of real OOD samples and heavy class imbalance in LTR. A straightforward solution to avoid the requirement of this prior is to learn an outlier class to …
Vadclip: Adapting Vision-Language Models For Weakly Supervised Video Anomaly Detection,
2024
Singapore Management University
Vadclip: Adapting Vision-Language Models For Weakly Supervised Video Anomaly Detection, Peng Wu, Xuerong Zhou, Guansong Pang, Lingru Zhou, Qingsen Yan, Peng Wang, Yanning Zhang
Research Collection School Of Computing and Information Systems
The recent contrastive language-image pre-training (CLIP) model has shown great success in a wide range of image-level tasks, revealing remarkable ability for learning powerful visual representations with rich semantics. An open and worthwhile problem is efficiently adapting such a strong model to the video domain and designing a robust video anomaly detector. In this work, we propose VadCLIP, a new paradigm for weakly supervised video anomaly detection (WSVAD) by leveraging the frozen CLIP model directly without any pre-training and fine-tuning process. Unlike current works that directly feed extracted features into the weakly supervised classifier for frame-level binary classification, VadCLIP makes …
Hackles: Simulating And Visually Representing The Anxiety Of Walking Alone,
2024
Singapore Management University
Hackles: Simulating And Visually Representing The Anxiety Of Walking Alone, Sydney Pratte, Anthony Tang, Shannon Hoover, Lora Oehlberg
Research Collection School Of Computing and Information Systems
In this work, we compare the designs of two fashion-tech garments that communicate the anxiety felt when walking alone. While the two garments share a common vision, they are designed to be worn in two radically different settings and to communicate to different audiences: one directly communicates an empathetic experience to its wearer; the other a model wears at a runway show and must share its story to a general audience. We used Research Through Design (RtD) methods to design both fashion-tech garments. Then, we recorded and analyzed the design process for both garments via an annotated portfolio to compare …
Foodmask: Real-Time Food Instance Counting, Segmentation And Recognition,
2024
Singapore Management University
Foodmask: Real-Time Food Instance Counting, Segmentation And Recognition, Huu-Thanh Nguyen, Yu Cao, Chong-Wah Ngo, Wing-Kwong Chan
Research Collection School Of Computing and Information Systems
Food computing has long been studied and deployed to several applications. Understanding a food image at the instance level, including recognition, counting and segmentation, is essential to quantifying nutrition and calorie consumption. Nevertheless, existing techniques are limited to either category-specific instance detection, which does not reflect precisely the instance size at the pixel level, or category-agnostic instance segmentation, which is insufficient for dish recognition. This paper presents a compact and fast multi-task network, namely FoodMask, for clustering-based food instance counting, segmentation and recognition. The network learns a semantic space simultaneously encoding food category distribution and instance height at pixel basis. …
Catnet: Cross-Modal Fusion For Audio-Visual Speech Recognition,
2024
Singapore Management University
Catnet: Cross-Modal Fusion For Audio-Visual Speech Recognition, Xingmei Wang, Jianchen Mi, Boquan Li, Yixu Zhao, Jiaxiang Meng
Research Collection School Of Computing and Information Systems
Automatic speech recognition (ASR) is a typical pattern recognition technology that converts human speeches into texts. With the aid of advanced deep learning models, the performance of speech recognition is significantly improved. Especially, the emerging Audio–Visual Speech Recognition (AVSR) methods achieve satisfactory performance by combining audio-modal and visual-modal information. However, various complex environments, especially noises, limit the effectiveness of existing methods. In response to the noisy problem, in this paper, we propose a novel cross-modal audio–visual speech recognition model, named CATNet. First, we devise a cross-modal bidirectional fusion model to analyze the close relationship between audio and visual modalities. Second, …
Delving Into Multimodal Prompting For Fine-Grained Visual Classification,
2024
Singapore Management University
Delving Into Multimodal Prompting For Fine-Grained Visual Classification, Xin Jiang, Hao Tang, Junyao Gao, Xiaoyu Du, Shengfeng He, Zechao Li
Research Collection School Of Computing and Information Systems
Fine-grained visual classification (FGVC) involves categorizing fine subdivisions within a broader category, which poses challenges due to subtle inter-class discrepancies and large intra-class variations. However, prevailing approaches primarily focus on uni-modal visual concepts. Recent advancements in pre-trained vision-language models have demonstrated remarkable performance in various high-level vision tasks, yet the applicability of such models to FGVC tasks remains uncertain. In this paper, we aim to fully exploit the capabilities of cross-modal description to tackle FGVC tasks and propose a novel multimodal prompting solution, denoted as MP-FGVC, based on the contrastive language-image pertaining (CLIP) model. Our MP-FGVC comprises a multimodal prompts …
Simple Image-Level Classification Improves Open-Vocabulary Object Detection,
2024
Singapore Management University
Simple Image-Level Classification Improves Open-Vocabulary Object Detection, Ruohuan Fang, Guansong Pang, Xiao Bai
Research Collection School Of Computing and Information Systems
Open-Vocabulary Object Detection (OVOD) aims to detect novel objects beyond a given set of base categories on which the detection model is trained. Recent OVOD methods focus on adapting the image-level pre-trained vision-language models (VLMs), such as CLIP, to a region-level object detection task via, eg., region-level knowledge distillation, regional prompt learning, or region-text pre-training, to expand the detection vocabulary. These methods have demonstrated remarkable performance in recognizing regional visual concepts, but they are weak in exploiting the VLMs' powerful global scene understanding ability learned from the billion-scale image-level text descriptions. This limits their capability in detecting hard objects of …
What Does One Billion Dollars Look Like?: Visualizing Extreme Wealth,
2024
CUNY Graduate Center
What Does One Billion Dollars Look Like?: Visualizing Extreme Wealth, William Mahoney Luckman
Dissertations, Theses, and Capstone Projects
The word “billion” is a mathematical abstraction related to “big,” but it is difficult to understand the vast difference in value between one million and one billion; even harder to understand the vast difference in purchasing power between one billion dollars, and the average U.S. yearly income. Perhaps most difficult to conceive of is what that purchasing power and huge mass of capital translates to in terms of power. This project blends design, text, facts, and figures into an interactive narrative website that helps the user better understand their position in relation to extreme wealth: https://whatdoesonebilliondollarslooklike.website/
The site incorporates …
Leveraging Llms And Generative Models For Interactive Known-Item Video Search,
2024
Singapore Management University
Leveraging Llms And Generative Models For Interactive Known-Item Video Search, Zhixin Ma, Jiaxin Wu, Chong-Wah Ngo
Research Collection School Of Computing and Information Systems
While embedding techniques such as CLIP have considerably boosted search performance, user strategies in interactive video search still largely operate on a trial-and-error basis. Users are often required to manually adjust their queries and carefully inspect the search results, which greatly rely on the users’ capability and proficiency. Recent advancements in large language models (LLMs) and generative models offer promising avenues for enhancing interactivity in video retrieval and reducing the personal bias in query interpretation, particularly in the known-item search. Specifically, LLMs can expand and diversify the semantics of the queries while avoiding grammar mistakes or the language barrier. In …
Learning From Machines: How Negative Feedback From Machines Improves Learning Between Humans,
2024
Zhejiang University
Learning From Machines: How Negative Feedback From Machines Improves Learning Between Humans, Tengjian Zou, Gokhan Ertug, Thomas Roulet
Research Collection Lee Kong Chian School Of Business
Prior studies on learning from failure primarily focus on how individuals learn from failure feedback given by other individuals. It is unclear whether and how the advent of machine feedback may influence individuals’ learning from failures. We suggest that failure feedback provided by machines facilitates learning in two ways. First, it focuses individuals’ attention on their failures, leading them to learn from these failures. Second, it serves as a catalyzer, motivating individuals to learn more from failure feedback given to them by other individuals as well. In addition, this catalyzing effect is stronger if the failure feedback from machines and …
Digitizing Delphi: Educating Audiences Through Virtual Reconstruction,
2024
Purdue University
Digitizing Delphi: Educating Audiences Through Virtual Reconstruction, Kate Koury
The Journal of Purdue Undergraduate Research
Implementing a 3D model into a virtual space allows the general public to engage critically with archaeological processes. There are many unseen decisions that go into reconstructing an ancient temple. Analysis of available materials and techniques, predictions of how objects were used, decisions of what sources to reference, puzzle piecing broken remains together, and even educated guesses used to fill gaps in information often go unobserved by the public. This work will educate users about those choices by allowing the side-by-side comparison of conflicting theories on the reconstruction of the Tholos at Delphi, which is an ideal site because of …
Piecing Together Performance: Collaborative, Participatory Research-Through-Design For Better Diversity In Games,
2024
Chapman University
Piecing Together Performance: Collaborative, Participatory Research-Through-Design For Better Diversity In Games, Daniel L. Gardner, Louanne Boyd, Reginald T. Gardner
Engineering Faculty Articles and Research
Digital games are a multi-billion-dollar industry whose production and consumption extend globally. Representation in games is an increasingly important topic. As those who create and consume the medium grow ever more diverse, it is essential that player or user-experience research, usability, and any consideration of how people interface with their technology is exercised through inclusive and intersectional lenses. Previous research has identified how character configuration interfaces preface white-male defaults [39, 40, 67]. This study relies on 1-on-1 play-interviews where diverse participants attempt to create “themselves” in a series of games and on group design activities to explore how participants may …
Are Academia And Industry Listening To Each Other? A Citation Analysis Of Ux Research Methods Resources,
2024
Minnesota State University, Mankato
Are Academia And Industry Listening To Each Other? A Citation Analysis Of Ux Research Methods Resources, Abigail Bakke, Leonard Dibono
English Department Publications
Technical and Professional Communication (TPC) has been facing concerns of viability, in both its relationship with industry and its ability to build a relevant and valid body of research. TPC’s disconnection with industry may be reflected in its relationship to UX as well, despite both fields’ shared values. To better understand how TPC and User Experience (UX) are relating to each other, we conducted a citation analysis of a sample of SIGDOC papers and a sample of Nielsen Norman Group (NN/g) practitioner articles focused on research methods. The SIGDOC papers tended to cite TPC sources, while the NN/g articles cited …
Dung Dkar Cloak: Exploring Soft Interfaces For Sonic Interactions,
2024
EJ Tech
Dung Dkar Cloak: Exploring Soft Interfaces For Sonic Interactions, Judit Eszter Kárpáti, Esteban De La Torre
Textile Society of America: Symposium Proceedings
The importance of crossmodal interaction within the contemporary cultural, technological and scientific panorama has evidently gained significant attention due to its remarkable advantages in creating a meaningful, interwoven, and integrated experience. The use and recontextualization of textiles in such exploratory quest into the human senses has proven to be critical. Computational science, algorithmic logic and digital devices have always been rooted and closely interwoven with textile crafts and practices. Recent technological advancements have further combined technology and textile, generating interactive textile surfaces, constructing endless possibilities for multisensorial experiences. In this presentation we will examine how we can weave a sensitive …
How Economically Marginalized Adolescents Of Color Negotiate Critical Pedagogy In A Computing Classroom,
2024
Carleton College
How Economically Marginalized Adolescents Of Color Negotiate Critical Pedagogy In A Computing Classroom, Jean Salac, Lena Armstrong, Megumi Kivuva, Jayne Everson, Amy J. Ko
Computer Science Faculty Work
Background and Context: With the growing movement to adopt critical framings of computing, scholars have worked to reframe computing education from the narrow development of programming skills to skills in identifying and resisting oppressive structures in computing. However, we have little guidance on how these framings may manifest in classroom practice. Objectives: To better understand the processes and practice of critical pedagogy in a computing classrooms, we taught a critically conscious computing elective within a summer academic program at a northwest United States university targeted at secondary students (ages 14–18) from low-income backgrounds and would be the first …
Advancing Policy Insights: Opinion Data Analysis And Discourse Structuring Using Llms,
2024
University of Central Florida
Advancing Policy Insights: Opinion Data Analysis And Discourse Structuring Using Llms, Aaditya Bhatia
Graduate Thesis and Dissertation 2023-2024
The growing volume of opinion data presents a significant challenge for policymakers striving to distill public sentiment into actionable decisions. This study aims to explore the capability of large language models (LLMs) to synthesize public opinion data into coherent policy recommendations. We specifically leverage Mistral 7B and Mixtral 8x7B models for text generation and have developed an architecture to process vast amounts of unstructured information, integrate diverse viewpoints, and extract actionable insights aligned with public opinion. Using a retrospective data analysis of the Polis platform debates published by the Computational Democracy Project, this study examines multiple datasets that span local …
Living Datasets: Towards Data-Centric Ai Explainability And Bias Mitigation,
2024
University of Texas at Arlington
Living Datasets: Towards Data-Centric Ai Explainability And Bias Mitigation, Akib Zaman
Computer Science and Engineering Dissertations - Archive
Benchmark datasets are critical to the evolution of AI efforts yet often embed unintended biases that influence the models that drive human-AI interactions. A deeper inspection and awareness of data is needed to understand the biases datasets may contain. In this dissertation, I introduce the Tag-and-Release method, inspired from wildlife research, that treats data as an organism and examines how different environments (i.e., CNNs) select for unique traits or characteristics that ultimately impact data's survival. Using the canonical MNIST handwritten digit dataset as a case study, I describe how the Tag-and-Release method can be used to analyze how dataset imbalance …
A Unified Cross-Modal Interactive System For Assisting Vision Impaired In Human Navigation And Indoor Based Human Robot Interaction,
2024
University of Texas at Arlington
A Unified Cross-Modal Interactive System For Assisting Vision Impaired In Human Navigation And Indoor Based Human Robot Interaction, Harish Ram Nambiappan
Computer Science and Engineering Dissertations - Archive
People who are blind and vision impaired often require assistance in performing various tasks. With new technologies emerging in the recent years, vision impaired people either require assistance in accessing those technologies or in using those technologies to perform different tasks in real life. Previous works have focused on assisting vision impaired people in different scenarios such as navigation, accessing smartphone interfaces etc. With the recent developments in robotics, a new research has emerged where new systems can be developed for vision impaired people to interact with robots to perform various human robot interactive tasks. But with developing new and …
