Other Applied Mathematics Commons^™

Scaling Up Exact Neural Network Compression By Relu Stability, Thiago Serra, Xin Yu, Abhinav Kumar, Srikumar Ramalingam

Faculty Conference Papers and Presentations

We can compress a rectifier network while exactly preserving its underlying functionality with respect to a given input domain if some of its neurons are stable. However, current approaches to determine the stability of neurons with Rectified Linear Unit (ReLU) activations require solving or finding a good approximation to multiple discrete optimization problems. In this work, we introduce an algorithm based on solving a single optimization problem to identify all stable neurons. Our approach is on median 183 times faster than the state-of-art method on CIFAR-10, which allows us to explore exact compression on deeper (5 x 100) and wider …