Open Access. Powered by Scholars. Published by Universities.®
Physical Sciences and Mathematics Commons™
Open Access. Powered by Scholars. Published by Universities.®
- Publication Type
Articles 1 - 11 of 11
Full-Text Articles in Physical Sciences and Mathematics
Replication And Knowledge Production In Empirical Software Engineering Research, Jonathan L. Krein
Replication And Knowledge Production In Empirical Software Engineering Research, Jonathan L. Krein
Theses and Dissertations
Although replication is considered an indispensable part of the scientific method in software engineering, few replication studies are published each year. The rate of replication, however, is not surprising given that replication theory in software engineering is immature. Not only are replication taxonomies varied and difficult to reconcile, but opinions on the role of replication contradict. In general, we have no clear sense of how to build knowledge via replication, particularly given the practical realities of our research field. Consequently, most replications in software engineering yield little useful information. In particular, the vast majority of external replications (i.e., replications performed …
Softprop: Softmax Neural Network Backpropagation Learning, Tony R. Martinez, Michael E. Rimer
Softprop: Softmax Neural Network Backpropagation Learning, Tony R. Martinez, Michael E. Rimer
Faculty Publications
Multi-layer backpropagation, like many learning algorithms that can create complex decision surfaces, is prone to overfitting. Softprop is a novel learning approach presented here that is reminiscent of the softmax explore-exploit Q-learning search heuristic It fits the problem while delaying settling into error minima to achieve better generalization and more robust learning. This is accomplished by blending standard SSE optimization with lazy training, a new objective function well suited to learning classification tasks, to form a more stable learning model. Over several machine learning data sets, softprop reduces classification error by 17.1 percent and the variance in results by 38.6 …
Improving Speech Recognition Learning Through Lazy Training, Tony R. Martinez, Michael E. Rimer, D. Randall Wilson
Improving Speech Recognition Learning Through Lazy Training, Tony R. Martinez, Michael E. Rimer, D. Randall Wilson
Faculty Publications
Multi-layer backpropagation, like most learning algorithms that can create complex decision surfaces, is prone to overfitting. We present a novel approach, called lazy training, for reducing the overfit in multiple-layer networks. Lazy training consistently reduces generalization error of optimized neural networks by more than half on a large OCR dataset and on several real world problems from the UCI machine learning database repository. Here, lazy training is shown to be effective in a multi-layered adaptive learning system, reducing the error of an optimized backpropagation network in a speech recognition system by 50.0% on the TIDIGITS corpus.
Speed Training: Improving The Rate Of Backpropagation Learning Through Stochastic Sample Presentation, Timothy L. Andersen, Tony R. Martinez, Michael E. Rimer
Speed Training: Improving The Rate Of Backpropagation Learning Through Stochastic Sample Presentation, Timothy L. Andersen, Tony R. Martinez, Michael E. Rimer
Faculty Publications
Artificial neural networks provide an effective empirical predictive model for pattern classification. However, using complex neural networks to learn very large training sets is often problematic, imposing prohibitive time constraints on the training process. We present four practical methods for dramatically decreasing training time through dynamic stochastic sample presentation, a technique we call speed training. These methods are shown to be robust to retaining generalization accuracy over a diverse collection of real world data sets. In particular, the SET technique achieves a training speedup of 4278% on a large OCR database with no detectable loss in generalization.
The Need For Small Learning Rates On Large Problems, Tony R. Martinez, D. Randall Wilson
The Need For Small Learning Rates On Large Problems, Tony R. Martinez, D. Randall Wilson
Faculty Publications
In gradient descent learning algorithms such as error backpropagation, the learning rate parameter can have a significant effect on generalization accuracy. In particular, decreasing the learning rate below that which yields the fastest convergence can significantly improve generalization accuracy, especially on large, complex problems. The learning rate also directly affects training speed, but not necessarily in the way that many people expect. Many neural network practitioners currently attempt to use the largest learning rate that still allows for convergence, in order to improve training speed. However, a learning rate that is too large can be as slow as a learning …
The Little Neuron That Could, Timothy L. Andersen, Tony R. Martinez
The Little Neuron That Could, Timothy L. Andersen, Tony R. Martinez
Faculty Publications
SLPs (single layer perceptrons) oflen exhibit reasonable generalization performance on many problems of interest. However, due to the well known limitations of SLPs very little effort has been made to improve their performance. This paper proposes a method for improving the performance of SLPs called "wagging" (weight averaging). This method involves training several different SLPs on the same training data, and then averaging their weights to obtain a single SLP. The performance of the wagged SLP is compared with other more complex learning algorithms (bp, c4.5, ibl, MML, etc) on 15 data sets from real world problem domains. Surprisingly, the …
Constructing High Order Perceptrons With Genetic Algorithms, Timothy L. Andersen, Tony R. Martinez
Constructing High Order Perceptrons With Genetic Algorithms, Timothy L. Andersen, Tony R. Martinez
Faculty Publications
Constructive induction, which is defined to be the process of constructing new and useful features from existing ones, has been extensively studied in the literature. Since the number of possible high order features for any given learning problem is exponential in the number of input attributes (where the order of a feature is defined to be the number of attributes of which it is composed), the main problem faced by constructive induction is in selecting which features to use out of this exponentially large set of potential features. For any feature set chosen the desirable characteristics are minimality and generalization …
Bias And The Probability Of Generalization, Tony R. Martinez, D. Randall Wilson
Bias And The Probability Of Generalization, Tony R. Martinez, D. Randall Wilson
Faculty Publications
In order to be useful, a learning algorithm must be able to generalize well when faced with inputs not previously presented to the system. A bias is necessary for any generalization, and as shown by several researchers in recent years, no bias can lead to strictly better generalization than any other when summed over all possible functions or applications. This paper provides examples to illustrate this fact, but also explains how a bias or learning algorithm can be “better” than another in practice when the probability of the occurrence of functions is taken into account. It shows how domain knowledge …
Towards A General Distributed Platform For Learning And Generalization, Brent W. Hughes, Tony R. Martinez
Towards A General Distributed Platform For Learning And Generalization, Brent W. Hughes, Tony R. Martinez
Faculty Publications
Different learning models employ different styles of generalization on novel inputs. This paper proposes the need for multiple styles of generalization to support a broad application base. The Priority ASOCS model (Priority Adaptive Self-organizing Concurrent System) is overviewed and presented as a potential platform which can support multiple generalization styles. PASOCS is an adaptive network composed of many simple computing elements operating asynchronously and in parallel. The PASOCS can operate in either a data processing mode or a learning mode. During data processing mode, the system acts as a parallel hardware circuit. During leaming mode, the PASOCS incorporates rules, with …
The Importance Of Using Multiple Styles Of Generalization, Tony R. Martinez, D. Randall Wilson
The Importance Of Using Multiple Styles Of Generalization, Tony R. Martinez, D. Randall Wilson
Faculty Publications
There are many ways for a learning system to generalize from training set data. There is likely no one style of generalization which will solve all problems better than any other style, for different styles will work better on some applications than others. This paper presents several styles of generalization and uses them to suggest that a collection of such styles can provide more accurate generalization than any one style by itself. Empirical results of generalizing on several real-world applications are given, and comparisons are made on the generalization accuracy of each style of generalization. The empirical results support the …
Consistency And Generalization In Incrementally Trained Connectionist Networks, Tony R. Martinez
Consistency And Generalization In Incrementally Trained Connectionist Networks, Tony R. Martinez
Faculty Publications
This paper discusses aspects of consistency and generalization in connectionist networks which learn through incremental training by examples or rules. Differences between training set learning and incremental rule or example learning are presented. Generalization, the ability to output reasonable mappings when presented with novel input patterns, is discussed in light of the above learning methods. In particular, the contrast between humming distance generalization and generalizing by high order combinations of critical variables is overviewed. Examples of detailed rules for an incremental learning model are presented for both consistency and generalization constraints.