Physical Sciences and Mathematics Commons^™

Why Rectified Linear Neurons Are Efficient: Symmetry-Based, Complexity-Based, And Fuzzy-Based Explanations, Olac Fuentes, Justin Parra, Elizabeth Y. Anthony, Vladik Kreinovich

Departmental Technical Reports (CS)

Traditionally, neural networks used a sigmoid activation function. Recently, it turned out that piecewise linear activation functions are much more efficient -- especially in deep learning applications. However, so far, there have been no convincing theoretical explanation for this empirical efficiency. In this paper, we show that, by using different uncertainty techniques, we can come up with several explanations for the efficiency of piecewise linear neural networks. The existence of several different explanations makes us even more confident in our results -- and thus, in the efficiency of piecewise linear activation functions.

Z-Numbers: How They Describe Student Confidence And How They Can Explain (And Improve) Laplacian And Schroedinger Eigenmap Dimension Reduction In Data Analysis, Vladik Kreinovich, Olga Kosheleva, Michael Zakharevich

Departmental Technical Reports (CS)

Experts have different degrees of confidence in their statements. To describe these different degrees of confidence, Lotfi A. Zadeh proposed the notion of a Z-number: a fuzzy set (or other type of uncertainty) supplemented by a degree of confidence in the statement corresponding to fuzzy sets. In this chapter, we show that Z-numbers provide a natural formalization of the competence-vs-confidence dichotomy, which is especially important for educating low-income students. We also show that Z-numbers provide a natural theoretical explanation for several empirically heuristic techniques of dimension reduction in data analysis, such as Laplacian and Schroedinger eigenmaps, and, moreover, show how …

Why Deep Learning Methods Use Kl Divergence Instead Of Least Squares: A Possible Pedagogical Explanation, Olga Kosheleva, Vladik Kreinovich

Departmental Technical Reports (CS)

In most applications of data processing, we select the parameters that minimize the mean square approximation error. The same Least Squares approach has been used in the traditional neural networks. However, for deep learning, it turns out that an alternative idea works better -- namely, minimizing the Kullback-Leibler (KL) divergence. The use of KL divergence is justified if we predict probabilities, but the use of this divergence has been successful in other situations as well. In this paper, we provide a possible explanation for this empirical success. Namely, the Least Square approach is optimal when the approximation error is normally …

Why Triangular Membership Functions Are Often Efficient In F-Transform Applications: Relation To Interval Uncertainty\\ And Haar Wavelets, Olga Kosheleva, Vladik Kreinovich

Departmental Technical Reports (CS)

Fuzzy techniques describe expert opinions. At first glance, we would therefore expect that the more accurately the corresponding membership functions describe the expert's opinions, the better the corresponding results. In practice, however, contrary to these expectations, the simplest -- and not very accurate -- triangular membership functions often work the best. In this paper, on the example of the use of membership functions in F-transform techniques, we provide a possible theoretical explanation for this surprising empirical phenomenon.

Beyond Integration: A Symmetry-Based Approach To Reaching Stationarity In Economic Time Series, Songsak Sriboonchitta, Olga Kosheleva, Vladik Kreinovich

Departmental Technical Reports (CS)

Many efficient data processing techniques assume that the corresponding process is stationary. However, in areas like economics, most processes are not stationery: with the exception of stagnation periods, economies usually grow. A known way to apply stationarity-based methods to such processes -- integration -- is based on the fact that often, while the process itself is not stationary, its first or second differences are stationary. This idea works when the trend polynomially depends on time. In practice, the trend is usually non-polynomial: it is often exponentially growing, with cycles added. In this paper, we shod how integration techniques can be …

Why Sparse?, Thongchai Dumrongpokaphan, Olga Kosheleva, Vladik Kreinovich, Aleksandra Belina

Departmental Technical Reports (CS)

In many situations, a solution to a practical problem is sparse, i.e., corresponds to the case when most of the parameters describing the solution are zeros, and only a few attain non-zero values. This surprising empirical phenomenon helps solve the corresponding problems -- but it remains unclear why this phenomenon happens. In this paper, we provide a possible theoretical explanation for this mysterious phenomenon.

How To Best Apply Neural Networks In Geosciences: Towards Optimal "Averaging" In Dropout Training, Afshin Gholamy, Justin Parra, Vladik Kreinovich, Olac Fuentes, Elizabeth Y. Anthony

Departmental Technical Reports (CS)

The main objectives of geosciences is to find the current state of the Earth -- i.e., solve the corresponding inverse problems -- and to use this knowledge for predicting the future events, such as earthquakes and volcanic eruptions. In both inverse and prediction problems, often, machine learning techniques are very efficient, and at present, the most efficient machine learning technique is deep neural training. To speed up this training, the current learning algorithms use dropout techniques: they train several sub-networks on different portions of data, and then "average" the results. A natural idea is to use arithmetic mean for this …

Why Taylor Models And Modified Taylor Models Are Empirically Successful: A Symmetry-Based Explanation, Mioara Joldes, Christoph Lauter, Martine Ceberio, Olga Kosheleva, Vladik Kreinovich

Departmental Technical Reports (CS)

In this paper, we show that symmetry-based ideas can explain the empirical success of Taylor models and modified Taylor models in representing uncertainty.

How To Store Tensors In Computer Memory: An Observation, Martine Ceberio, Vladik Kreinovich

Departmental Technical Reports (CS)

In this paper, after explaining the need to use tensors in computing, we analyze the question of how to best store tensors in computer memory. Somewhat surprisingly, with respect to a natural optimality criterion, the standard way of storing tensors turns out to be one of the optimal ones.

How To Make A Proof Of Halting Problem More Convincing: A Pedagogical Remark, Benjamin W. Robertson, Olga Kosheleva, Vladik Kreinovich

Departmental Technical Reports (CS)

As an example of an algorithmically undecidable problem, most textbooks list the impossibility to check whether a given program halts on given data. A usual proof of this result is based on the assumption that the hypothetical halt-checker works for all programs. To show that a halt-checker is impossible, we design an auxiliary program for which the existence of such a halt-checker leads to a contradiction. However, this auxiliary program is usually very artificial. So, a natural question arises: what if we only require that the halt-checker work for reasonable programs? In this paper, we show that even with such …

Propagation Of Probabilistic Uncertainty: The Simplest Case (A Brief Pedagogical Introduction), Olga Kosheleva, Vladik Kreinovich

Departmental Technical Reports (CS)

The main objective of this text is to provide a brief introduction to formulas describing the simplest case of propagation of probabilistic uncertainty -- for students who have not yet taken a probability course.

Sudoku App: Model-Driven Development Of Android Apps Using Ocl?, Yoonsik Cheon, Aditi Barua

Departmental Technical Reports (CS)

Model driven development (MDD) shifts the focus of software development from writing code to building models by developing an application as a series of transformations on models including eventual code generation. Can the key ideas of MDD be applied to the development of Android apps, one of the most popular mobile platforms of today? To answer this question, we perform a small case study of developing an Android app for playing Sudoku puzzles. We use the Object Constraint Language (OCL) as the notation for creating precise models and translate OCL constraints to Android Java code. Our findings are mixed in …

Impacts Of Java Language Features On The Memory Performances Of Android Apps, Yoonsik Cheon, Adriana Escobar De La Torre

Departmental Technical Reports (CS)

Android apps are written in Java, but unlike Java applications they are resource-constrained in storage capacity and battery lifetime. In this document, we perform an experiment to measure quantitatively the impact of Java language and standard API features on the memory efficiency of Android apps. We focus on garbage collection because it is a critical process for performance affecting user experience. We learned that even Java language constructs and standard application programming interfaces (APIs) may be a source of a performance problem causing a significant memory overhead for Android apps. Any critical section of code needs to be scrutinized on …

Need For A Large-N Array (And Wavelets And Differences) To Determine The Assumption-Free 3-D Earth Model, Solymar Ayala Cortez, Aaron A. Velasco, Vladik Kreinovich

Departmental Technical Reports (CS)

One of the main objectives of geophysical seismic analysis is to determine the Earth's structure. Usually, to determine this structure, geophysicists supplement the measurement results with additional geophysical assumptions. An important question is: when is it possible to reconstruct the Earth's structure uniquely based on the measurement results only, without the need to use any additional assumptions? In this paper, we show that for this, one needs to use large-N arrays -- 2-D arrays of seismic sensors. To actually perform this reconstruction, we need to use differences between measurements by neighboring sensor and we need to apply wavelet analysis to …

Efficient Parameter-Estimating Algorithms For Symmetry-Motivated Models: Econometrics And Beyond, Vladik Kreinovich, Anh H. Ly, Olga Kosheleva, Songsak Sriboonchitta

Departmental Technical Reports (CS)

It is known that symmetry ideas can explain the empirical success of many non-linear models. This explanation makes these models theoretically justified and thus, more reliable. However, the models remain non-linear and thus, identification or the model's parameters based on the observations remains a computationally expensive nonlinear optimization problem. In this paper, we show that symmetry ideas can not only help to select and justify a nonlinear model, they can also help us design computationally efficient almost-linear algorithms for identifying the model's parameters.

Practical Need For Algebraic (Equality-Type) Solutions Of Interval Equations And For Extended-Zero Solutions, Ludmila Dymova, Pavel Sevastjanov, Andrzej Pownuk, Vladik Kreinovich

Departmental Technical Reports (CS)

One of the main problems in interval computations is solving systems of equations under interval uncertainty. Usually, interval computation packages consider united, tolerance, and control solutions. In this paper, we explain the practical need for algebraic (equality-type) solutions, when we look for solutions for which both sides are equal. In situations when such a solution is not possible, we provide a justification for extended-zero solutions, in which we ignore intervals of the type [−a, a].

How To Use Absolute-Error-Minimizing Software To Minimize Relative Error: Practitioner's Guide, Afshin Gholamy, Vladik Kreinovich

Departmental Technical Reports (CS)

In many engineering and scientific problems, there is a need to find the parameters of a dependence from the experimental data. There exist several software packages that find the values for these parameters -- values for which the mean square value of the absolute approximation error is the smallest. In practice, however, we are often interested in minimizing the mean square value of the relative approximation error. In this paper, we show how we can use the absolute-error-minimizing software to minimize the relative error.

Granular Approach To Data Processing Under Probabilistic Uncertainty, Andrzej Pownuk, Vladik Kreinovich

Departmental Technical Reports (CS)

In many real-life situations, uncertainty can be naturally described as a combination of several components, components which are described by probabilistic, fuzzy, interval, etc. granules. In such situations, to process this uncertainty, it is often beneficial to take this granularity into account by processing these granules separately and then combining the results.

In this paper, we show that granular computing can help even in situations when there is no such natural decomposition into granules: namely, we can often speed up processing of uncertainty if we first (artificially) decompose the original uncertainty into appropriate granules.

A Thought On Refactoring Java Loops Using Java 8 Streams, Khandoker Rahad, Zejing Cao, Yoonsik Cheon

Departmental Technical Reports (CS)

Java 8 has introduced a new abstraction called a stream to represent an immutable sequence of elements and to provide a variety of operations to be executed on the elements in series or in parallel. By processing a collection of data in a declarative way, it enables one to write more concise and clean code that can also leverage multi-core architectures without needing a single line of multithread code to be written. In this document, we describe our preliminary work on systematically refactoring loops with Java 8 streams to produce more concise and clean code. Our idea is to adapt …

How Better Are Predictive Models: Analysis On The Practically Important Example Of Robust Interval Uncertainty, Vladik Kreinovich, Hung T. Nguyen, Songsak Sriboonchitta, Olga Kosheleva

Departmental Technical Reports (CS)

One of the main applications of science and engineering is to predict future value of different quantities of interest. In the traditional statistical approach, we first use observations to estimate the parameters of an appropriate model, and then use the resulting estimates to make predictions. Recently, a relatively new predictive approach has been actively promoted, the approach where we make predictions directly from observations. It is known that in general, while the predictive approach requires more computations, it leads to more accurate predictions. In this paper, on the practically important example of robust interval uncertainty, we analyze how more accurate …

How To Gauge Accuracy Of Processing Big Data: Teaching Machine Learning Techniques To Gauge Their Own Accuracy, Vladik Kreinovich, Thongchai Dumrongpokaphan, Hung T. Nguyen, Olga Kosheleva

Departmental Technical Reports (CS)

When the amount of data is reasonably small, we can usually fit this data to a simple model and use the traditional statistical methods both to estimate the parameters of this model and to gauge this model's accuracy. For big data, it is often no longer possible to fit them by a simple model. Thus, we need to use generic machine learning techniques to find the corresponding model. The current machine learning techniques estimate the values of the corresponding parameters, but they usually do not gauge the accuracy of the corresponding general non-linear model. In this paper, we show how …

Kuznets Curve: A Simple Dynamical System-Based Explanation, Thongchai Dumrongpokaphan, Vladik Kreinovich

Departmental Technical Reports (CS)

In the 1950s, a future Nobelist Simon Kuznets discovered the following phenomenon: as a country's economy improves, inequality first grows but then decreases. In this paper, we provide a simple dynamical system-based explanation for this empirical phenomenon.

Taking Into Account Interval (And Fuzzy) Uncertainty Can Lead To More Adequate Statistical Estimates, Ligang Sun, Hani Dbouk, Steffen Schön, Vladik Kreinovich

Departmental Technical Reports (CS)

Traditional statistical data processing techniques (such as Least Squares) assume that we know the probability distributions of measurement errors. Often, we do not have full information about these distributions. In some cases, all we know is the bound of the measurement error; in such cases, we can use known interval data processing techniques. Sometimes, this bound is fuzzy; in such cases, we can use known fuzzy data processing techniques.

However, in many practical situations, we know the probability distribution of the random component of the measurement error and we know the upper bound -- numerical or fuzzy -- on the …

Entropy As A Measure Of Average Loss Of Privacy, Luc Longpre, Vladik Kreinovich, Thongchai Dumrongpokaphan

Departmental Technical Reports (CS)

Privacy means that not everything about a person is known, that we need to ask additional questions to get the full information about the person. It therefore seems to reasonable to gauge the degree of privacy in each situation by the average number of binary ("yes"-"no") questions that we need to ask to determine the full information -- which is exactly Shannon's entropy. The problem with this idea is that it is possible, by asking two binary questions -- and thus, strictly speaking, getting only two bits of information -- to sometimes learn a large amount of information. In this …

Maximum Entropy As A Feasible Way To Describe Joint Distributions In Expert Systems, Thongchai Dumrongpokaphan, Vladik Kreinovich, Hung T. Nguyen

Departmental Technical Reports (CS)

In expert systems, we elicit the probabilities of different statements from the experts. However, to adequately use the expert system, we also need to know the probabilities of different propositional combinations of the experts' statements -- i.e., we need to know the corresponding joint distribution. The problem is that there are exponentially many such combinations, and it is not practically possible to elicit all their probabilities from the experts. So, we need to estimate this joint distribution based on the available information. For this purpose, many practitioners use heuristic approaches -- e.g., the t-norm approach of fuzzy logic. However, this …

Why Student Distributions? Why Matern's Covariance Model? A Symmetry-Based Explanation, Steffen Schön, Gaël Kermarrec, Boris Kargoll, Ingo Neumann, Olga Kosheleva, Vladik Kreinovich

Departmental Technical Reports (CS)

In this paper, we show that empirical successes of Student distribution and of Matern's covariance models can be indirectly explained by a natural requirement of scale invariance -- that fundamental laws should not depend on the choice of physical units. Namely, while neither the Student distributions nor Matern's covariance models are themselves scale-invariant, they are the only one which can be obtained by applying a scale-invariant combination function to scale-invariant functions.

Markowitz Portfolio Theory Helps Decrease Medicines' Side Effect And Speed Up Machine Learning, Thongchai Dumrongpokaphan, Vladik Kreinovich

Departmental Technical Reports (CS)

In this paper, we show that, similarly to the fact that distributing the investment between several independent financial instruments decreases the investment risk, using a combination of several medicines can decrease the medicines' side effects. Moreover, the formulas for optimal combinations of medicine are the same as the formulas for the optimal portfolio, formulas first derived by the Nobel-prize winning economist H. M. Markowitz. A similar application to machine learning explains a recent success of a modified neural network in which the input neurons are also directly connected to the output ones.

What If We Do Not Know Correlations?, Michael Beer, Zitong Gong, Ingo Neumann, Songsak Sriboonchitta, Vladik Kreinovich

Departmental Technical Reports (CS)

It is well know how to estimate the uncertainty of the result y of data processing if we know the correlations between all the inputs. Sometimes, however, we have no information about the correlations. In this case, instead of a single value σ of the standard deviation of the result, we get a range [σ] of possible values. In this paper, we show how to compute this range.

Fuzzy Sets As Strongly Consistent Random Sets, Kittawit Autchariyapanitkul, Hung T. Nguyen, Vladik Kreinovich

Departmental Technical Reports (CS)

It is known that from the purely mathematical viewpoint, fuzzy sets can be interpreted as equivalent classes of random sets. This interpretations helps to teach fuzzy techniques to statisticians and also enables us to apply results about random sets to fuzzy techniques. The problem with this interpretation is that it is too complicated: a random set is not an easy notion, and classes of random sets are even more complex. This complexity goes against the spirit of fuzzy sets, whose purpose was to be simple and intuitively clear. From this viewpoint, it is desirable to simplify this interpretation. In this …

From Fuzzy Universal Approximation To Fuzzy Universal Representation: It All Depends On The Continuum Hypothesis, Mahdokhat Michelle Afravi, Vladik Kreinovich

Departmental Technical Reports (CS)

It is known that fuzzy systems have a universal approximation property. A natural question is: can this property be extended to a universal representation property? Somewhat surprisingly, the answer to this question depends on whether the following Continuum Hypothesis: every infinite subset of the real line has either the same number of elements as the real line itself or as many elements as natural numbers.