Physical Sciences and Mathematics Commons^™

Learning From The Master: Distilling Cross-Modal Advanced Knowledge For Lip Reading, Sucheng Ren, Yong Du, Jianming Lv, Guoqiang Han, Shengfeng He

Research Collection School Of Computing and Information Systems

Lip reading aims to predict the spoken sentences from silent lip videos. Due to the fact that such a vision task usually performs worse than its counterpart speech recognition, one potential scheme is to distill knowledge from a teacher pretrained by audio signals. However, the latent domain gap between the cross-modal data could lead to a learning ambiguity and thus limits the performance of lip reading. In this paper, we propose a novel collaborative framework for lip reading, and two aspects of issues are considered: 1) the teacher should understand bi-modal knowledge to possibly bridge the inherent cross-modal gap; 2) …