Social and Behavioral Sciences Commons^™

The Andersen Likelihood Ratio Test With A Random Split Criterion Lacks Power, Georg Krammer

Journal of Modern Applied Statistical Methods

The Andersen LRT uses sample characteristics as split criteria to evaluate Rasch model fit, or theory driven hypothesis testing for a test. The power and Type I error of a random split criterion was evaluated with a simulation study. Results consistently show a random split criterion lacks power.

Outlier Impact And Accommodation On Power, Hongjing Liao, Yanju Li, Gordon P. Brooks

Journal of Modern Applied Statistical Methods

The outliers’ influence on power rates in ANOVA and Welch tests at various conditions was examined and compared with the effectiveness of nonparametric methods and Winsorizing in minimizing the impact of outliers. Results showed that, considering both power and Type I error, a nonparametric test is the safest choice to control the inflation of Type I error with a decent sample size and yield relatively high power.

Are Per-Family Type I Error Rates Relevant In Social And Behavioral Science?, Andrew V. Frane

Journal of Modern Applied Statistical Methods

The familywise Type I error rate is a familiar concept in hypothesis testing, whereas the per‑family Type I error rate is rarely addressed. This article uses Monte Carlo simulations and graphics to make a case for the relevance of the per‑family Type I error rate in research practice and pedagogy.

Per Family Or Familywise Type I Error Control: "Eether, Eyether, Neether, Nyther, Let's Call The Whole Thing Off!", H. J. Keselman

Journal of Modern Applied Statistical Methods

Frane (2015) pointed out the difference between per-family and familywise Type I error control and how different multiple comparison procedures control one method but not necessarily the other. He then went on to demonstrate in the context of a two group multivariate design containing different numbers of dependent variables and correlations between variables how the per-family rate inflates beyond the level of significance. In this article I reintroduce other newer better methods of Type I error control. These newer methods provide more power to detect effects than the per-family and familywise techniques of control yet maintain the overall rate of …

P-Values Versus Significance Levels, Phillip I. Good

Journal of Modern Applied Statistical Methods

In this article Phillip Good responds to Richard Anderson's article Conceptual Distinction between the Critical p Value and the Type I Error Rate in Permutation Testing.

Conceptual Distinction Between The Critical P Value And The Type I Error Rate In Permutation Testing: Author Response To Peer Comments, Richard B. Anderson

Journal of Modern Applied Statistical Methods

Richard Anderson responds to comments regarding his target article Conceptual Distinction between the Critical p Value and the Type I Error Rate in Permutation Testing.

A Response To Anderson's (2013) Conceptual Distinction Between The Critical P Value And Type I Error Rate In Permutation Testing, Fortunato Pesarin, Stefano Bonnini

Journal of Modern Applied Statistical Methods

Pesarin and Bonnini respond to Anderson's (2013) Conceptual Distinction between the Critical p value and Type I Error Rate in Permutation Testing

Conceptual Distinction Between The Critical P Value And The Type I Error Rate In Permutation Testing, Richard B. Anderson

Journal of Modern Applied Statistical Methods

To counter past assertions that permutation testing is not distribution-free, this article clarifies that the critical p value (alpha) in permutation testing is not a Type I error rate and that a test's validity is independent of the concept of Type I error.

Robust Modifications Of The Levene And O’Brien Tests For Spread, Abdul R. Othman, The Sin Yan, H. J. Keselman, Rand R. Wilcox, James Algina

Journal of Modern Applied Statistical Methods

Variants of Levene’s and O’Brien’s procedures not investigated by Keselman, Wilcox & Algina (2008) were examined. Simulations indicate that a new O’Brien variant provides very good Type I error control and is simpler for applied researchers to compute than the method recommended by Keselman, et al.

Type I Error Rates Of The Two-Sample Pseudo-Median Procedure, Nor Aishah Ahad, Abdul Rahman Othman, Sharipah Soaad Syed Yahaya

Journal of Modern Applied Statistical Methods

The performance of the pseudo-median based procedure is examined in terms of controlling Type I error for a two independent groups test. The procedure is a modification of the one-sample Wilcoxon statistic using the pseudo-median of differences between group values as the central measure of location. The proposed procedure was shown to have good control of Type I error rates under the study conditions regardless of distribution type.

Approximate Vs. Monte Carlo Critical Values For The Winsorized T-Test, Michael Lance

Wayne State University Dissertations

Historically, it has been accepted practice for critical values for the Winsorized t test for independent samples to be based on adjusted degrees of freedom depending on the number of total non-Winsorized (approximate) values. Recently, a new such table of Winsorized critical values has been developed via approximate randomization by Monte Carlo simulation.

Based on eight common data distributions estimated from Psychology and Education along with the normal and five Mathematical distributions, these two tables of values were compared with respect to robustness to types I and II errors through Monte Carlo simulations for one and 10% Winsorized values per …

Notes On Hypothesis Testing Under A Single-Stage Design In Phase Ii Trial, Kung-Jong Lui

Journal of Modern Applied Statistical Methods

A primary objective of a phase II trial is to determine future development is warranted for a new treatment based on whether it has sufficient activity against a specified type of tumor. Limitations exist in the commonly-used hypothesis setting and the standard test procedure for a phase II trial. This study reformats the hypothesis setting to mirror the clinical decision process in practice. Under the proposed hypothesis setting, the critical points and the minimum required sample size for a desired power of finding a superior treatment at a given α -level are presented. An example is provided to illustrate how …

Generalized Variances Ratio Test For Comparing K Covariance Matrices From Dependent Normal Populations, Marcelo Angelo Cirillo, Daniel Furtado Ferreira, Thelma Sáfadi, Eric Batista Ferreira

Journal of Modern Applied Statistical Methods

New tests based on the ratio of generalized variances are presented to compare covariance matrices from dependent normal populations. Monte Carlo simulation concluded that the tests considered controlled the Type I error, providing empirical probabilities that were consistent with the nominal level stipulated.

Detecting Lag-One Autocorrelation In Interrupted Time Series Experiments With Small Datasets, Clare Riviello, S. Natasha Beretvas

Journal of Modern Applied Statistical Methods

The power and type I error rates of eight indices for lag-one autocorrelation detection were assessed for interrupted time series experiments (ITSEs) with small numbers of data points. Performance of Huitema and McKean’s (2000) z_HM statistic was modified and compared with the z_HM, five information criteria and the Durbin-Watson statistic.

A Randomization Method To Control The Type I Error Rates In Best Subset Regression, Yasser A. Shehata, Paul White

Journal of Modern Applied Statistical Methods

A randomization method for the assessment of statistical significance for best subsets regression is given. The procedure takes into account the number of potential predictors and the inter-dependence between predictors. The approach corrects a non-trivial problem with Type I errors and can be used to assess individual variable significance.

Comparing Different Methods For Multiple Testing In Reaction Time Data, Massimiliano Pastore, Massimo Nucci, Giovanni Galfano

Journal of Modern Applied Statistical Methods

Reaction times were simulated for examining the power of six methods for multiple testing, as a function of sample size and departures from normality. Power estimates were low for all methods for non-normal distributions. With normal distributions, even for small sample sizes, satisfactory power estimates were observed, especially for FDR-based procedures.

Corrections For Type I Error In Social Science Research: A Disconnect Between Theory And Practice, Kenneth Lachlan, Patric R. Spence

Journal of Modern Applied Statistical Methods

Type I errors are a common problem in factorial ANOVA and ANOVA based analyses. Despite decades of literature offering solutions to the Type I error problems associated with multiple significance tests, simple solutions such as Bonferroni corrections have been largely ignored by social scientists. To examine this discontinuity between theory and practice, a content analysis was performed on 5 flagship social science journals. Results indicate that corrections for Type I error are seldom utilized, even in designs so complicated as to almost guarantee erroneous rejection of null hypotheses.

Type I Error Rates For A One Factor Within-Subjects Design With Missing Values, Miguel A. Padilla, James Algina

Journal of Modern Applied Statistical Methods

Missing data are a common problem in educational research. A promising technique, that can be implemented in SAS PROC MIXED and is therefore widely available, is to use maximum likelihood to estimate model parameters and base hypothesis tests on these estimates. However, it is not clear which test statistic in PROC MIXED performs better with missing data. The performance of the Hotelling- Lawley-McKeon and Kenward-Roger omnibus test statistics on the means for a single factor withinsubject ANOVA are compared. The results indicate that the Kenward-Roger statistic performed better in terms of keeping the Type I error close to the nominal …

Type I Error Rates For Rank-Based Tests Of Homogeneity Of Slopes, Alan J. Klockars, Tim P. Moses

Journal of Modern Applied Statistical Methods

The purpose of this study was to explicate two issues concerning the standard and rank based test of homogeneity of slopes. Two alternative ranking methods intended to address nonnormality and additive treatment effect patterns were developed and compared in terms of their ability to control Type I error. The results replicated previous findings of inflated Type I error rates with leptokurtic curves and with rank based tests with some patterns of additive treatment effects. The new nonparametric procedures generally control Type I error although they were slightly inflated with skewed distributions.

Exact Level And Power Of Permutation, Bootstrap, And Asymptotic Tests Of Trend, Christopher D. Corcoran, Cyrus R. Mehta

Journal of Modern Applied Statistical Methods

We develop computational tools that can evaluate the exact size and power of three tests of trend (e.g., permutation, bootstrap and asymptotic) without resorting to large-sample theory or simulations. We then use these tools to compare the operating characteristics of the three tests. It is seen that the bootstrap test is ultra-conservative relative to the other two tests and as a result suffers from a severe deterioration in power. The power of the asymptotic test is uniformly larger than that of the other two tests, but it fails to preserve the Type I error for most of the range of …