Physical Sciences and Mathematics Commons^™

Concept-Driven Multi-Modality Fusion For Video Search, Xiao-Yong Wei, Yu-Gang Jiang, Chong-Wah Ngo

Research Collection School Of Computing and Information Systems

As it is true for human perception that we gather information from different sources in natural and multi-modality forms, learning from multi-modalities has become an effective scheme for various information retrieval problems. In this paper, we propose a novel multi-modality fusion approach for video search, where the search modalities are derived from a diverse set of knowledge sources, such as text transcript from speech recognition, low-level visual features from video frames, and high-level semantic visual concepts from supervised learning. Since the effectiveness of each search modality greatly depends on specific user queries, prompt determination of the importance of a modality …