Open Access. Powered by Scholars. Published by Universities.®
Physical Sciences and Mathematics Commons™
Open Access. Powered by Scholars. Published by Universities.®
Articles 1 - 4 of 4
Full-Text Articles in Physical Sciences and Mathematics
Softprop: Softmax Neural Network Backpropagation Learning, Tony R. Martinez, Michael E. Rimer
Softprop: Softmax Neural Network Backpropagation Learning, Tony R. Martinez, Michael E. Rimer
Faculty Publications
Multi-layer backpropagation, like many learning algorithms that can create complex decision surfaces, is prone to overfitting. Softprop is a novel learning approach presented here that is reminiscent of the softmax explore-exploit Q-learning search heuristic It fits the problem while delaying settling into error minima to achieve better generalization and more robust learning. This is accomplished by blending standard SSE optimization with lazy training, a new objective function well suited to learning classification tasks, to form a more stable learning model. Over several machine learning data sets, softprop reduces classification error by 17.1 percent and the variance in results by 38.6 …
The Need For Small Learning Rates On Large Problems, Tony R. Martinez, D. Randall Wilson
The Need For Small Learning Rates On Large Problems, Tony R. Martinez, D. Randall Wilson
Faculty Publications
In gradient descent learning algorithms such as error backpropagation, the learning rate parameter can have a significant effect on generalization accuracy. In particular, decreasing the learning rate below that which yields the fastest convergence can significantly improve generalization accuracy, especially on large, complex problems. The learning rate also directly affects training speed, but not necessarily in the way that many people expect. Many neural network practitioners currently attempt to use the largest learning rate that still allows for convergence, in order to improve training speed. However, a learning rate that is too large can be as slow as a learning …
Lazy Training: Improving Backpropagation Learning Through Network Interaction, Timothy L. Andersen, Tony R. Martinez, Michael E. Rimer
Lazy Training: Improving Backpropagation Learning Through Network Interaction, Timothy L. Andersen, Tony R. Martinez, Michael E. Rimer
Faculty Publications
Backpropagation, similar to most high-order learning algorithms, is prone to overfitting. We address this issue by introducing interactive training (IT), a logical extension to backpropagation training that employs interaction among multiple networks. This method is based on the theory that centralized control is more effective for learning in deep problem spaces in a multi-agent paradigm. IT methods allow networks to work together to form more complex systems while not restraining their individual ability to specialize. Lazy training, an implementation of IT that minimizes misclassification error, is presented. Lazy training discourages overfitting and is conducive to higher accuracy in multiclass problems …
The Inefficiency Of Batch Training For Large Training Sets, Tony R. Martinez, D. Randall Wilson
The Inefficiency Of Batch Training For Large Training Sets, Tony R. Martinez, D. Randall Wilson
Faculty Publications
Multilayer perceptrons are often trained using error backpropagation (BP). BP training can be done in either a batch or continuous manner. Claims have frequently been made that batch training is faster and/or more "correct" than continuous training because it uses a better approximation of the true gradient for its weight updates. These claims are often supported by empirical evidence on very small data sets. These claims are untrue, however, for large training sets. This paper explains why batch training is much slower than continuous training for large training sets. Various levels of semi-batch training used on a 20,000-instance speech recognition …