Aerospace Engineering Commons^™

Fine-Grained Activity Classification In Assembly Based On Multi-Visual Modalities, Haodong Chen, Niloofar Zendehdel, Ming-Chuan Leu, Zhaozheng Yin

Mechanical and Aerospace Engineering Faculty Research & Creative Works

Assembly activity recognition and prediction help to improve productivity, quality control, and safety measures in smart factories. This study aims to sense, recognize, and predict a worker's continuous fine-grained assembly activities in a manufacturing platform. We propose a two-stage network for workers' fine-grained activity classification by leveraging scene-level and temporal-level activity features. The first stage is a feature awareness block that extracts scene-level features from multi-visual modalities, including red, green blue (RGB) and hand skeleton frames. We use the transfer learning method in the first stage and compare three different pre-trained feature extraction models. Then, we transmit the feature information …