Engineering Commons^™

Training Thinner And Deeper Neural Networks: Jumpstart Regularization, Carles Riera, Camilo Rey, Thiago Serra, Eloi Puertas, Oriol Pujol

Faculty Conference Papers and Presentations

Neural networks are more expressive when they have multiple layers. In turn, conventional training methods are only successful if the depth does not lead to numerical issues such as exploding or vanishing gradients, which occur less frequently when the layers are sufficiently wide. However, increasing width to attain greater depth entails the use of heavier computational resources and leads to overparameterized models. These subsequent issues have been partially addressed by model compression methods such as quantization and pruning, some of which relying on normalization-based regularization of the loss function to make the effect of most parameters negligible. In this work, …