Physical Sciences and Mathematics Commons^™

A Review Of Frequentist Tests For The 2x2 Binomial Trial, Chris Lloyd

Chris J. Lloyd

The 2x2 binomial trial is the simplest of data structures yet its statistical analysis and the issues it raises have been debated and revisited for over 70 years. Which analysis should biomedical researchers use in applications? In this review, we consider frequentist tests only, specifically tests with control size either exactly or very close to exactly. These procedures can be classified as conditional and unconditional. Amongst tests motivated by a conditional model, Lancaster’s mid-p and Liebermeister’s test are less conservative than Fisher’s classical test, but do not control type 1 error. Within the conditional framework, only Fisher’s test can be …

Importance Accelerated Robbins-Monro Recursion With Applications To Parametric Confidence Limits, Zdravjko I. Botev, Chris Lloyd

Chris J. Lloyd

Monro (1951) to calculating confidence limits leads to poor efficiency and difficulties in estimating the appropriate governing constants as well as the standard error. We suggest sampling instead from an alternative importance distribu- tion and modifying the Robbins-Monro recursion accordingly. This can reduce the asymptotic variance by the usual importance sampling factor. It also allows the standard error and optimal step length to be estimated from the simulation. The methodology is applied to computing almost exact confidence limits in a generalised linear model.

On The Exact Size Of Multiple Comparison Tests, Chris Lloyd

Chris J. Lloyd

No abstract provided.

On The Size Accuracy Of Combination Tests, Chris Lloyd

Chris J. Lloyd

One element of the analysis of adaptive clinical trials is combining the evidence from several (often two) stages. When the endpoint is binary, standard single stage tests statistics do not control size well. Yet the combined test might not be valid if the single stage tests are not. The purpose of this paper is to numerically and theoretically examine the extent to which combining basic tests statistics mitigates or magnifies the size violation of the final test.

Aging Population Scenarios: An Australian Experience, Chris Lloyd

Chris J. Lloyd

One element of the analysis of adaptive clinical trials is combining the evidence from several (often two) stages. When the endpoint is binary, standard single stage tests statistics do not control size well. Yet the combined test might not be valid if the single stage tests are not. The purpose of this paper is to numerically and theoretically examine the extent to which combining basic tests statistics mitigates or magnifies the size violation of the final test.

Computing Highly Accurate Or Exact P-Values Using Importance Sampling, Chris Lloyd

Chris J. Lloyd

Especially for discrete data, standard first order P-values can suffer from poor accuracy, even for quite large sample sizes. Moreover, different test statistics can give practically different results. There are several approaches to computing P-values which do not suffer these defects, such as parametric bootstrap P-values or the partially maximised P-values of Berger and Boos (1994).

Both these methods require computing the exact tail probability of the approximate P-value as a function of the nuisance parameter/s, known as the significance profile. For most practical problems, this is not computationally feasible. I develop an importance sampling approach to this problem. A …

Some Non-Asymptotic Properties Of Parametric Bootstrap P-Values, Chris Lloyd

Chris J. Lloyd

The bootstrap P-value is the exact tail probability of a test statistic, cal-culated assuming the nuisance parameter equals the null maximum likelihood (ML) estimate. For discrete data, bootstrap P-values perform amazingly well even for small samples, even as standard first order methods perform surprisingly poorly. Why is this? Detailed numerical calculations in Lloyd (2012a) strongly suggest that the good performance of bootstrap is not explained by asymptotics. In this paper, I establish several desirable non-asymptotic properties of bootstrap P-values. The most important of these is that bootstrap will correct ‘bad’ ordering of the sample space which leads to a more …

A Practical Ad-Hoc Adjustment To The Simes P-Value, Chris Lloyd

Chris J. Lloyd

The Simes P-value is more powerful than Bonferroni but still suffers from some conservatism when the tests are correlated. Based on a massive simulation study, I develop a formula that corrects for this conservatism. it requires the number of experimental arms which is known. It also requires the correlation and skewness of the underlying test statistics, which will need analytic approximation in practice.

Computing Highly Accurate Confidence Limits From Discrete Data Using Importance Sampling, Chris Lloyd

Chris J. Lloyd

For discrete parametric models, approximate confidence limits perform poorly from a strict frequentist perspective. In principle, exact and optimal confidence limits can be computed using the formula of Buehler (1957), Lloyd and Kabaila (2003). So-called profile upper limits (Kabaila \& Lloyd, 2001) are closely related to Buehler limits and have extremely good properties. Both profile and Buehler limits depend on the probability of a certain tail set as a function of the unknown parameters. Unfortunately, this probability surface is not computable for realistic models. In this paper, importance sampling is used to estimate the surface and hence the confidence limits. …

Computing Highly Accurate Or Exact P-Values Using Importance Sampling (Revised), Chris Lloyd

Chris J. Lloyd

Especially for discrete data, standard first order P-values can suffer from poor accuracy, even for quite large sample sizes. Moreover, different test statistics can give practically different results. There are several approaches to computing P-values which do not suffer these defects, such as parametric bootstrap P-values or the partially maximised P-values of Berger & Boos (1994).

Both these methods require computing the exact tail probability of the approximate P-value as a function of the nuisance parameter/s, known as the significance profile. For most practical problems this is not computationally feasible. I develop an importance sampling approach to this problem. A …

Bootstrap P-Values In Discrete Models: Asymptotic And Non-Asymptotic Effects, Chris Lloyd

Chris J. Lloyd

(This paper is a major revision of http://works.bepress.com/chris_lloyd/15/.) Standard first order P-values suffer from two important drawbacks. First, even for quite large sample sizes they can misrepresent the exact significance which depends on nuisance parameters unspecified under the null. For most discrete models is that accuracy is variable and breaks down completely at the boundary. Second, different test statistics can give practically different results.

The bootstrap P-value is the exact significance with the null maximum estimate (ML) of the nuisance parameter substituted. We show that bootstrap P-values based on different first order statistics differ to second order. We also show …

Bootstrap And Second Order Tests Of Risk Difference, Chris Lloyd

Chris J. Lloyd

Standard approximate tests of the difference of two probabilities have type 1 error that can differ significantly from nominal, even for quite large sample sizes. There are two modern methods of reducing type 1 error. One is to use so-called higher order asymptotics (Reid, 2003) to provide an explicit adjustment to the likelihood ratio statistic. The second is to replace the nuisance parameter in an exact calculation with a null estimate (Young and Lee, 2005), which is a kind of bootstrap. The purpose of this paper is to explain and evaluate these two methods, for testing whether a difference in …

More Powerful Exact Noninferiority And Equivalence Tests Based On Binary Matched Pairs, Chris Lloyd

Chris J. Lloyd

Assessing the therapeutic noninferiority or equivalence of one medical treatment compared to another is often based on the difference of response rates from a matched binary pairs design. This paper develops new exact unconditional tests for noninferiority and equivalence that are more powerful than available alternatives. There are three new elements presented in this paper. First we introduce the LR statistic as an alternative to the previously proposed score statistic of Nam (1997). Second, we eliminate the nuisance parameter by estimation followed by maximization as an alternative to the partial maximization of Berger and Boos (1994) or traditional full maximization. …

The Economic Value Of Improved Environmental Health In Victorian Rivers, Chris Lloyd

Chris J. Lloyd

The non market valuation technique known as choice modelling was used to general benefit estimates for a selection of hypothetical environmental improvements Victorian Rivers. Monetary values were estimated for four attributed of improvements. The relevance of the approach to management and policy issues is demonstrated.

Estimated P-Values In Discrete Models: Asymptotic And Non-Asymptotic Effects, Chris Lloyd

Chris J. Lloyd

The exact null distribution of a P-value typically depends on nuisance parameters unspecified under the null. For discrete models and standard approximate P-values, this dependence can be quite strong. The estimated (or bootstrap) P-value is the exact probability of the P-value being no larger than its observed value, with the null estimate of the nuisance parameter substituted. For continuous models, it is known that such `bootstrap' P-values deviate from uniformity by terms of order m^{-3/2}, where m is a measure of sample size. The main difficulty with discrete models is the breakdown of asymptotics near the boundary. The aim of …

More Powerful Exact Tests Of Equivalence, Chris Lloyd

Chris J. Lloyd

In randomized clinical trials, it is often required to demonstrate that a new medical treatment is neither substantially worse nor better than a standard reference treatment. Formal testing of such `equivalence hypotheses' is typcialyl done by combining two one-sided tests (TOST). A quite diferent strand of research has demonstrated that maximising P-values over nuisance parameters produces optimal tests (Rohmel and Mansmann (1999) and Lloyd (2008a)). In this paper we point out that, even if the one-sided tests are exact and optimal, the TOST will generally be conservative and requires a further adjustment to remove this conservatism. The appropriate procedure is …

On Approximate Conditioning And Higher Order Asymptotics For 2x2 Tables, Chris Lloyd

Chris J. Lloyd

For testing canonical parameters in a continuous exponential family, P-values based on higher order asymptotic formulas such as p* approximate the exact conditional P-value with great accuracy. For discrete models, the conditional distribution can be extremely discrete or even degenerate which raises the questions (a) should one try to approximate the conditional P-value, (b) what does p* approximate? Pierce and Peters (1999) have argued that p* approximates an approximately conditional P-value and that this approximately conditional P-value is an inferentially sensible quantity worth approximating. Their arguments and numerical results are oriented towards problems where the conditioning variable has 3 or …

Multinomial Logistic Regression: An Application To Estimating Performance Of A Multiple Screening Test For Bowel Cancer When Negatives Are Unverified., Chris Lloyd, Don Frommer

Chris J. Lloyd

This paper describes a method of estimating the performance of a multiple screening test where those who test negative do not have their true disease status determined. The methodology is motivated by a dataset on 49,927 subjects who were given K=6 binary tests for bowel cancer. A complicating factor is that individuals may have polyps present in the bowel, a condition that the screening test is not designed to detect but which may be worth diagnosing. The methodology is based on a multinomial logit model for Pr(S|R_6), the probability distribution of patient status S (healthy, polyps or diseased) conditional on …

A New Exact And More Powerful Unconditional Test Of No Treatment Effect From Binary Matched Pairs, Chris Lloyd

Chris J. Lloyd

We consider the problem of testing for a difference in the probability of success from matched binary pairs. Starting with three standard inexact tests, the nuisance parameter is first estimated and then the residual dependence is eliminated by maximisation, producing what I call an E+M P-value. The E+M P-value based on McNemar's statistic is shown numerically to dominate previous suggestions, including partially maximised P-values as described in Berger and Sidik (2003). The latter method however may have computational advantages for large samples.

Exact Tests Of Non-Inferiority From Independent Binomial Data Based On Second Order Test Statistics, Chris Lloyd

Chris J. Lloyd

Recent advances in likelihood asymptotics (Reid, 2003) lead to pivotal quantities that are closer to standard normal than standard pivotals and also respect some kind of conditionality. It is less clear the extent to which these methods work for discrete models. On the other hand, in the context of binomial trials conditional pivotals can lead to more efficient unconditional inferences, see Boschloo (1970) and Lloyd and Moldovan (2007). This suggests that second order pivotals that respect local conditionality might provide more powerful exact tests. For testing the rate ratio from independent binomial samples, we investigate 5 first order pivotals and …

Exact Confidence Bounds For The Risk Ratio In 2x2 Tables With Structural Zero, Chris J. Lloyd, Max Moldovan

Chris J. Lloyd

This paper examines exact one-sided confidence limits for the risk ratio in a 2x2 table with structural zero. Starting with four approximate lower and upper limits, we adjust each using the algorithm of Buehler (1957) to arrive at lower (upper) limits that have exact coverage properties and are as large (small) as possible subject to coverage, as well as an ordering, constraint. Different Buehler limits are compared by their mean size, since all are exact in their coverage. Buehler limits based on the signed root likelihood ratio statistic are found to have the best performance and recommended for practical use.

Exact One-Sided Confidence Limits For The Difference Between Two Correlated Proportions, Chris Lloyd, Max V. Moldovan

Chris J. Lloyd

We construct exact and optimal one-sided upper and lower confidence bounds for the difference between two probabilities based on matched binary pairs using well-established optimality theory of Buehler (1957). Starting with five different approximate loer and upper limits, we adjust them to have coverage probability exactly equal to the desired nominal level and then compare the resulting exact limits by their mean size. Exact limits based on the signed root likelihood ratio statistic are preferred and recommended for practical use.

Exact P-Values For Discrete Models Obtained By Estimation And Maximisation, Chris Lloyd

Chris J. Lloyd

In constructing exact tests, one must deal with the possible dependence of the P-value on the nuisance parameter/s psi as well as discreteness of the sample space. A classical but heavy handed approach is to maximise over psi. We prove what has previously been understood informally, namely that maximisation produces the uinique and smallest possible P-value subject to the ordering induced by the underlying test statistic and test validity. On the other hand, allowing for the worst case will be more attractive when the P-value is less dependent on psi. We investigate the extent to which estimating psi under the …

Unconditional Efficient One-Sided Confidence Limits For The Odds Ratio Based On Conditional Likelihood, Chris Lloyd, Max Moldovan

Chris J. Lloyd

We compare various one-sided confidence limits for the odds ratio in a 2x2 table. The first group of limits relies on first order asymptotic approximations and includes limits based on the (signed) likelihood ratio, score and Wald statistics. The second group of limits is based on the conditional tilted hypergeometric distribution, with and without mid-P correction. All these limits have poor unconditional coverage properties and so we apply the general transformation of Buehler (1957) to obtain limits which are unconditionally exact. The performance of these competing exact limits is assessed across a range of sample sizes and parameter values by …

Efficient And Exact Tests Of The Risk Ratio In A Correlated 2x2 Table With Structural Zero, Chris Lloyd

Chris J. Lloyd

For a correlated 2x2 table where the (01) cell is empty by design, the parameter of interest is typically the ratio of the probability of secondary response conditional on primary response to the probability of primary response, also known as a risk ratio. It is common to test whether or not the risk ratio equals one. One method of obtaining an exact P-value is to maximise the tail probability of the test statistic over the nuisance parameter. It is argued that better results are obtained by first replacing the nuisance parameter by its profile estimate in the calculation of its …

Improved Buehler Limits Based On Refined Designated Statistics , Chris Lloyd, Paul Kabaila

Chris J. Lloyd

The Buehler upper confidence limit is as small as possible, subject to the constraints that (a) its coverage probability never falls below nimonal and (b) it is a non-decreasing function of a designated statistic. The designated statistic may have ties among its possible values.

We prove that breaking such ties by a sufficiently small

modification can never increase the Buehler limit. We also prove that, under commonly satisfied conditions, breaking ties by a sufficiently small modification will result in an improved i.e. smaller Buehler limit. We conclude that designated statistics should not contain ties, apart from ties justified by symmetry …

More Powerful Unconditional Tests Of No Treatment Effect From Binary Matched Pairs, Chris Lloyd

Chris J. Lloyd

This is the workign paper version that preceeded the paper "A New Exact and More Powerful Unconditional Test of no Treatment Effect from Binary Matched Pairs" published in Biometrics 76 (also on this site:http://works.bepress.com/chris_lloyd/3/