Education Commons^™

Self-Report Measures Of The Home Learning Environment In Large Scale Research: Measurement Properties And Associations With Key Developmental Outcomes, Frank Niklas, Cuc Nguyen, Daniel S. Cloney, Collette Tayler, Ray Adams

Prof Ray Adams

A Learning Assessment Program Designed To Meet The Post-2015 Development Goals, Juliette Mendelovits, Maurice Walker, Ray Adams

Prof Ray Adams

No abstract provided.

Educational Progress For All, Ray Adams

Prof Ray Adams

ACER’s ongoing work in global educational monitoring is supporting educational progress for all learners, worldwide, as Ray Adams explains.

Effects Of Student Reading Proficiency On Pisa Mathematics Items – A Dif Study Across Countries, Ray Adams, Luc Le

Prof Ray Adams

No abstract provided.

Computer Adaptive Testing : A Feasibility Study, Siek Khoo, Geoff Masters, Ray Adams

Prof Ray Adams

The Australian National Office of Overseas Skills Recognition (NOOSR) commissioned ACER to investigate the feasibility of implementing computer adaptive testing (CAT) in NOOSR’s screening examinations for overseas trained professionals. The National Office administers screening examinations in seven professions (Dentistry, Dietetics, Occupational Therapy, Podiatry, Pharmacy, Physiotherapy and Veterinary Science) at venues throughout Australia and in up to fifty centres around the world. Surveys were conducted to gain an overview of the current methods and procedures used by NOOSR in the screening examinations. Issues related to the application of CAT to NOOSR’s screening examinations and the possible improvement of NOOSR’s assessment program …

Mapping Student Achievement, Geoff Masters, Ray Adams, Jan Lokan

Prof Ray Adams

The analyses, publications and reports of quantitative data in education and the social sciences usually omit basic information on the construct measured. Probabilistic models for test data make it possible to delineate coherent and richly described measurement continua that facilitate interpretation of student achievement. The potential of conjoint measurement to bring about fundamental advances in educational testing practice lies in part in the opportunities it provides to build useful maps of learning domains and to use those maps in communicating student achievements. This chapter presents two applications of conjoint measurement aimed at constructing and describing achievement variables, developing insights into …

Displaying Test Analyses With Titan And Bigscale, Siek Khoo, Ray Adams, J Linacre

Prof Ray Adams

No abstract provided.

Quest: The Interactive Test Analysis System, Ray Adams, Siek Khoo

Prof Ray Adams

This is a guide to using Quest. Quest offers a comprehensive test and questionnaire analysis environment by providing a data analyst with access to the most recent developments in Rasch measurement theory, as well as a range of traditional analysis procedures. It includes an easy to use control language with flexible and informative output. Quest can be used to construct and validate variables based on both dichotomous and polychotomous observations. It scores and analyses such instruments as multiple choice tests, Likert type rating scales, short answer items, and partial credit items.

The Programme For International Student Assessment: An Overview, Ross Turner, Ray Adams

Prof Ray Adams

Pisa : Frequently Answered Criticisms, Ray Adams

Prof Ray Adams

Studies such as PISA that attempt to compare outcomes across educational systems are expensive and difficult to implement. Further, the results of such studies are routinely criticized by educational commentators - particularly when the results are not consistent with their preconceived ideas about the relative merits and efficiencies of various educational practices and systems. This chapter discusses what is done to ameliorate the threats to the validity of PISA in five areas often targeted by commentators and reviewers as sources of invalidity in international comparisons. The five areas discussed are: (1) sampling - are the samples of students who undertake …

The Influence Of Equating Methodology On Reported Trends In Pisa, Eveline Gebhardt, Ray Adams

Prof Ray Adams

In 2005 PISA published trend indicators that compared the results of PISA 2000 and PISA 2003. This paper explores the extent to which the outcomes of these trend analyses are sensitive to the choice of test equating methodologies, the choice of regression models and the choice of linking items. To establish trends, PISA equated its 2000 and 2003 tests using a methodology based on Rasch Modelling that involved estimating linear transformations that mapped 2003 Rasch-scaled scores to the previously established PISA 2000 Rasch-scaled scores. This paper compares the outcomes of this approach with an alternative, which involves the joint Rasch …

The Impact Of Differential Investment Of Student Effort On The Outcomes Of International Studies, J Butler, Ray Adams

Prof Ray Adams

International comparative assessments of student achievement, such as Trends in Mathematics and Science (TIMSS) and Programme for International Student Achievement (PISA) are becoming increasingly important in the development of evidence-based education policy. The potentially far-reaching influence of such studies underscores the need for these assessments to be valid and reliable. In education, increasing recognition is being given to motivational factors which impact on student learning. This research considers a possible threat to the validity of such studies by investigating the influence the amount of effort invested by test-takers has on their outcomes. Reassuringly, it is found that the reported expenditure …

Modelling Mathematics Problem Solving Item Responses Using A Multidimensional Irt Model, Margaret Wu, Ray Adams

Prof Ray Adams

This research examined students' responses to mathematics problem- solving tasks and applied a general multidimensional IRT model at the response category level. In doing so, cognitive processes were identified and modelled through item response modelling to extract more information than would be provided using conventional practices in scoring items. More specifically, the study consisted of two parts. The first part involved the development of a mathematics problem-solving framework that was theoretically grounded, drawing upon research in mathematics education and cognitive psychology. The framework was then used as the basis for item development. The second part of the research involved the …

Self-Review For Higher Education Institutions, Nathan Zoanetti, Patrick Griffin, Ray Adams

Prof Ray Adams

This paper describes a plausible values imputation approach for deriving population scores on several language proficiency domains. The approach harnessed a multi-dimensional item response analysis combining student responses, rater judgements and student background variables. The target population was grade one and grade two primary school students enrolled in the Hong Kong schooling system. The raters were local teachers of English employed within the sampled schools. The primary objective of this research was to impute plausible values for data where no data was provided or where rater data was deemed suspect. By necessity, a secondary objective of this study was to …

Application Of Multivariate Rasch Models In International Large-Scale Educational Assessments, Ray Adams, Margaret Wu, C Carstensen

Prof Ray Adams

In large-scale educational assessments, such as the Programme for International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS), a primary concern is with the estimation of the population-level characteristics of a number of latent variables and the relationships between latent variables and other variables. Typically these studies are undertaken in contexts in which there are constraints on sample size and individual student response time, yet there are high expectations with regard to the breadth of content coverage. These demands and constraints have resulted in such studies using rotated-booklet designs, with each student responding to a …

The Mixed-Coefficients Multinomial Logit Model : A Generalized Form Of The Rasch Model, Ray Adams, Margaret Wu

Prof Ray Adams

Since Rasch's introduction of his item response models, there has been a proliferation of extensions and alternatives, each of which has a different name and different matching software package. The proliferation of models has, in some ways, been a hindrance to practitioners. This paper presents a generalised item response model that provides a unifying framework for a large class of Rasch-type models. The advantages of a single framework include mathematical elegance, generality in a single software package, and a facilitation of the development, testing, and comparison of new models. The unified model is a multidimensional item response model, the specification …

Reliability As A Measurement Design Effect, Ray Adams

Prof Ray Adams

Test reliability is a concept central to classical test theory and it is commonly stated as a requirement that a test attain a certain level of reliability before it be considered of sufficient quality for practical use. This article discusses the role of reliability in item response theory, and in particular the role of reliability in contexts where matrix sampling designs are used and concern is with the estimation of population parameters rather than the measurement of individuals. The concept of a measurement design effect is introduced. This concept parallels the concept of sampling design effects, in that it describes …

Response To 'Cautions On Oecd's Recent Educational Survey (Pisa), Ray Adams

Prof Ray Adams

This paper refers to the work of S J Prais who questioned the outcomes of the Organisation for Economic Cooperation and Development's PISA survey of the reading, mathematics and science attainments of fifteen year-olds. Prais suggested that methodological flaws in Programme for International Student Assessment (PISA) had resulted in an apparent improvement in the attainment of British students, particularly when compared to their Swiss and German counterparts. This paper responds to Prais's criticisms, noting that when Prais's conjectures are tested with empirical data they are not supported. The author argues that many of Prais's criticisms are due to an incomplete …

The Implications Of Halo Effects And Item Dependencies For Objective Measurement, T Mcnamara, Ray Adams

Prof Ray Adams

Analysis of data from performance-based language tests presents the problem of dealing appropriately with bundles of items that are linked to particular tasks. The authors use recent developments in Rasch modelling which allow analysis of bundles of items (or sets of ratings) to examine the extent of dependence between items and ratings, and they explore the implications of those dependences on the assessments. Data from two performance assessments are used in two studies of this problem.

Interpreting The Parameters Of A Multidimensional Rasch Model, W Wang, M Wilson, Ray Adams

Prof Ray Adams

This chapter introduces the multidimensional random coefficient multinomial logit model (MRCML) which can be applied to multidimensional polytomous test items. Two examples of real data analyses are followed to demonstrates the applications of the MRCML. A detailed discussion is made of interpretation of item parameters, in comparision to those derived from their corresponding unidimensional Rasch measurement models. Finally, it provides some guidelines for constructing scoring matrices and the corresponding models to aid parameter interpretation.

Measuring Student Knowledge And Skills : The Pisa 2000 Assessment Of Reading, Mathematical And Scientific Literacy, Andreas Schieicher, Claudia Tamassia, Geoff Masters, Ray Adams

Prof Ray Adams

Broadening Assessment, Improving Fairness? Some Examples From School Science, Jan Lokan, Ray Adams, Brian Doig

Prof Ray Adams

Following a review of equity issues in testing, especially in relation to science, two assessment programmes in school science, each of which was designed both to probe a range of aspects of student performance and for use in a large-scale study, are discussed. Both have been developed and used in the 1990s, one is Australia only and the other in many countries. The programmes and the range of aspects they attempt to assess are described and results are cited for several groups of students. Issues of fairness through broadening of large-scale assessments to include tasks requiring students to construct their …

International Student Assessment For The New Millennium, Ray Adams

Prof Ray Adams

The author briefly describes the OECD initiated project, 'Program for International Student Assessment' (PISA), in which over 100,000 15 year old students from 4000 schools in 26 countries will participate. A major focus of the project is the assessment of wider knowledge, skills and competencies in literacy, mathematics and science areas.

Acer Conquest: Generalised Item Response Modelling Software, Margaret Wu, Ray Adams, M Wilson

Prof Ray Adams

ACER Conquest is a computer program for fitting item response and latent regression models. It provides a comprehensive and flexible range of item response models to analysts, allowing them to examine the properties of performance assessment, traditional assessments and rating scales. This manual examines ACER ConQuest in detail, includes a tutorial and coverage of technical matters.

Measuring Individual Differences In Change With Multidimensional Rasch Models, W Wang, M Wilson, Ray Adams

Prof Ray Adams

Item response models have been developed to explore change measurement, including those proposed by Fischer and his colleagues. In this article the authors propose another multidimensional Rasch model, the multidimensional random coefficient multinomial logit (MRCML) model. All these models are briefly reviewed and compared. The MRCML can be applied to not only polytomous items but also investigation of variations in item difficulties. Based on variations in difficulties across occasions and items, five kinds of models are proposed. Some simulation studies were conducted to examine parameter recovery of the MRCML model under various testing situations. All the parameters were recovered very …

The Multidimensional Random Coefficients Multinomial Logit Model, Ray Adams, M Wilson, W Wang

Prof Ray Adams

A multidimensional Rasch-type item response model, the multidimensional random coefficients multinomial logit model, is presented as an extension to the Adams and Wilson (1996) random coefficients multinomial logit model. The model is developed in a form that permits generalization to the multidimensional case of a wide class of Rasch models, including the simple logistic model, Masters' partial credit model, Wilson's ordered partition model, and Fischer's linear logistic model. Moreover, the model includes several existing multidimensional models as special cases, including Whitely's multicomponent latent trait model, Andersen's multidimensional Rasch model for repeated testing, and Embretson's multidimensional Rasch model for learning and …

Applications Of Item Response Theory In The Third International Mathematics And Science Study, E Gonzalez, Ray Adams, Marg Wu

Prof Ray Adams

The Third International Mathematics and Science Study (TIMSS) is a survey of mathematics and science achievement sponsored by the International Association for the Evaluation of Educational Achievement. With more than 45 participating countries, three target populations, more than thirty languages, 15,000 schools, more than half a million students, TIMSS is the largest and most ambitious international study of educational achievement ever undertaken. Students participating in the assessment were administered one of eight achievement tests that contain items in mathematics and science, using a variety of item formats. As TIMSS is an international study, many audiences will look at the results …

Rasch Models Of Multidimensionality Between Items And Within Items, W Wen-Chung, M Wilson, Ray Adams

Prof Ray Adams

This chapter focuses on Rasch measurement and extends it to multidimensional Rasch models. The author first clarifies multidimensionality at two levels: the item level and the test level. With this two way classification, three kinds of tests are formed: undimensional tests, multidimensional between-item tests, and multidimensional within-item tests. The author describes a multidimensional Rasch model, the multidimensional random coefficients multinomial logit (MRCML) model, for multidimensional tests. The model, when applied to multidimensional between-item tests, is more efficient than are unidimensional models in terms of model fit and reliability indices. The model can also be applied to investigate scale interaction of …

Multilevel Item Response Models: An Approach To Errors In Variables Regression, Ray Adams, M Wilson, Margaret Wu

Prof Ray Adams

In this article the authors show how certain analytic problems that arise when one attempts to use latent variables as outcomes in regression analyses can be addressed by taking a multilevel perspective on item response modelling.

Bias In Weighted Likelihood Estimation When Using The Rasch Model, D Roberts, Ray Adams

Prof Ray Adams

There are three methods of maximum likelihood estimation that are commonly used with Rasch item response models: Joint maximum likelihood, which provides point estimates for both person and item parameters; conditional maximum likelihood, which produces estimates for the item parameters only, and marginal maximum likelihood, which produces estimates of the item parameters and parameters of the distribution from which it has been assumed that the persons have been sampled. A number of approaches have been suggested for obtaining ability estimates for subsequent use in reporting or analysis when either conditional or marginal methods have been used to obtain the item …