Education Commons^™

Reduce Randomly Guessing Effects In A University Generic Skills Test, Luc T. Le

Dr Luc Tu Le

National Assessment Program : Ict Literacy 2008 : Year 6 And Year 10 : Technical Report, Chris Freeman, John Ainley, Julian Fraillon, Eveline Gebhardt, Kate O'Malley, Wolfram Schulz

Dr Wolfram Schulz

This report describes the technical aspects of the National ICT Literacy Sample Assessment and summarises the main activities involved in the data collection, the data collection instruments and the analysis and reporting of the data. Chapter 2 summarises the development of the assessment domain and describes the process of item development and construction of the instruments. Chapter 3 reviews the sample design and describes the sampling process. Chapter 3 also describes the weighting procedures that were implemented to derive population estimates. Chapter 4 summarises the field administration and data management procedures, including quality control and the cleaning and coding of …

Evaluation Of Item Parameter Recovery Estimation By Acer Conquest Software, Luc T. Le, Ray Adams

Dr Luc Tu Le

ACER ConQuest (Adams, Wu, and Wilson, 2012) has been popularly used for analysing testing and assessment data. Two of the most common estimation methods for Rasch measurement models (Rasch, 1960/1980) are available in this software, marginal maximum likelihood estimation (MML) and joint maximum likelihood estimation (JML). This study is concerned with item parameter recovery for the dichotomous Rasch model. Our primary focus is on comparing JML and MML when the assumptions of MML are violated, that is the abilities are not sampled from the distribution that is assumed in the estimation.

Evaluation Of Item Parameter Recovery Estimation By Acer Conquest Software, Luc T. Le, Ray Adams

Professor Ray Adams

ACER ConQuest (Adams, Wu, and Wilson, 2012) has been popularly used for analysing testing and assessment data. Two of the most common estimation methods for Rasch measurement models (Rasch, 1960/1980) are available in this software, marginal maximum likelihood estimation (MML) and joint maximum likelihood estimation (JML). This study is concerned with item parameter recovery for the dichotomous Rasch model. Our primary focus is on comparing JML and MML when the assumptions of MML are violated, that is the abilities are not sampled from the distribution that is assumed in the estimation.

National Assessment Program : Ict Literacy 2008 : Year 6 And Year 10 : Technical Report, Chris Freeman, John Ainley, Julian Fraillon, Eveline Gebhardt, Kate O'Malley, Wolfram Schulz

Julian Fraillon

This report describes the technical aspects of the National ICT Literacy Sample Assessment and summarises the main activities involved in the data collection, the data collection instruments and the analysis and reporting of the data. Chapter 2 summarises the development of the assessment domain and describes the process of item development and construction of the instruments. Chapter 3 reviews the sample design and describes the sampling process. Chapter 3 also describes the weighting procedures that were implemented to derive population estimates. Chapter 4 summarises the field administration and data management procedures, including quality control and the cleaning and coding of …

National Assessment Program : Ict Literacy 2008 : Year 6 And Year 10 : Technical Report, Chris Freeman, John Ainley, Julian Fraillon, Eveline Gebhardt, Kate O'Malley, Wolfram Schulz

Dr John Ainley

This report describes the technical aspects of the National ICT Literacy Sample Assessment and summarises the main activities involved in the data collection, the data collection instruments and the analysis and reporting of the data. Chapter 2 summarises the development of the assessment domain and describes the process of item development and construction of the instruments. Chapter 3 reviews the sample design and describes the sampling process. Chapter 3 also describes the weighting procedures that were implemented to derive population estimates. Chapter 4 summarises the field administration and data management procedures, including quality control and the cleaning and coding of …

Extended Mantel-Haenszel Procedure For Dif Detection : A Note On Its Implementation In Acer Conquest, Xiaoxun Sun

Xiaoxun Sun

Many methods have been developed for studying Differential Item Functioning (DIF). An approach developed by Holland and Thayer (1988) adapted the Mantel-Haenszel (MH) statistic (Mantel and Haenszel 1959). The author of this chapter has implemented MH procedure and its extension in an experimental version of ACER ConQuest which can handle DIF analysis for multiple focal groups and polytomous items. This chapter compares the extended MH procedure in DIF detection to the standardized item difficulty difference method based on the Rasch estimates for both dichotomous and polytomous items. The use of MH method in DIF analysis with multiple focal groups is …

National Assessment Program : Ict Literacy Technical Report, Eveline Gebhardt, Julian Fraillon, Wolfram Schulz, Kate O'Malley, Chris Freeman, Martin Murphy, John Lindsey

Martin Murphy

This report is concerned with procedures, processes and technical aspects of the National Assessment Program - ICT Literacy 2011 Literacy (NAP - ICTL 2011) and should be read in conjunction with the Public Report from Literacy NAP - ICTL 2011 which focuses on results and interpretation of results from that assessment. The first cycle of the NAP - ICTL was held in 2005 and provided the baseline against which future performance would be compared. The second cycle of the program was conducted in 2008 and was the first cycle where trends in performance were able to be examined.

Investigating Item Difficulty Change By Item Positions Under The Rasch Model, Luc T. Le, Van Nguyen

Dr Van Nguyen

Investigating Item Difficulty Change By Item Positions Under The Rasch Model, Luc T. Le, Van Nguyen

Dr Luc Tu Le

Multiple-Choice Questions Compared To Short-Answer Response : Which Assesses Understanding Of Chemistry More Effectively?, Ross Hudson

Dr Ross Hudson

The research inquires into the effectiveness of the two predominant forms of questions that are used on the Victorian Certificate of Education (VCE) Victoria Chemistry examination. These are multiple- choice questions and short-answer questions. This research examines not only the style of chemistry question but also the content type examined (recall and application questions) along with gender differences in students' responses to such questions. The research involved three phases, i) analysis of five years results from the VCE Chemistry examinations, ii) class trial testing students of both genders with structured questions that examined the same material content with each type …

Ameliorating Culturally Based Extreme Response Tendencies To Attitude Items, Maurice Walker

Maurice Walker

No abstract provided.

The Influence Of Equating Methodology On Reported Trends In Pisa, Eveline Gebhardt, Ray Adams

Prof Ray Adams

In 2005 PISA published trend indicators that compared the results of PISA 2000 and PISA 2003. This paper explores the extent to which the outcomes of these trend analyses are sensitive to the choice of test equating methodologies, the choice of regression models and the choice of linking items. To establish trends, PISA equated its 2000 and 2003 tests using a methodology based on Rasch Modelling that involved estimating linear transformations that mapped 2003 Rasch-scaled scores to the previously established PISA 2000 Rasch-scaled scores. This paper compares the outcomes of this approach with an alternative, which involves the joint Rasch …

Application Of Multivariate Rasch Models In International Large-Scale Educational Assessments, Ray Adams, Margaret Wu, C Carstensen

Prof Ray Adams

In large-scale educational assessments, such as the Programme for International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS), a primary concern is with the estimation of the population-level characteristics of a number of latent variables and the relationships between latent variables and other variables. Typically these studies are undertaken in contexts in which there are constraints on sample size and individual student response time, yet there are high expectations with regard to the breadth of content coverage. These demands and constraints have resulted in such studies using rotated-booklet designs, with each student responding to a …

The Mixed-Coefficients Multinomial Logit Model : A Generalized Form Of The Rasch Model, Ray Adams, Margaret Wu

Prof Ray Adams

Since Rasch's introduction of his item response models, there has been a proliferation of extensions and alternatives, each of which has a different name and different matching software package. The proliferation of models has, in some ways, been a hindrance to practitioners. This paper presents a generalised item response model that provides a unifying framework for a large class of Rasch-type models. The advantages of a single framework include mathematical elegance, generality in a single software package, and a facilitation of the development, testing, and comparison of new models. The unified model is a multidimensional item response model, the specification …

The Effect Of Test-Taker Gender, Audience And Topic On Task Performance In Tape-Mediated Assessment Of Speaking., Tom Lumley, Barry O'Sullivan

Dr Tom Lumley

Performance in tests of spoken language can be influenced by a wide range of features of both task and participants. This article hypothesises that there may be effects on performance attributable to an interaction of variables such as the task topic, the gender of the person presenting the topic and the gender of the candidate. In contrast to previous studies, which have examined speaking tests involving face-to-face interaction, this study considered the issue in a tape-mediated test delivered in a language laboratory, with no interlocutor present, but where stimulus material is presented by one or more speakers, one of whom …

The Implications Of Halo Effects And Item Dependencies For Objective Measurement, T Mcnamara, Ray Adams

Prof Ray Adams

Analysis of data from performance-based language tests presents the problem of dealing appropriately with bundles of items that are linked to particular tasks. The authors use recent developments in Rasch modelling which allow analysis of bundles of items (or sets of ratings) to examine the extent of dependence between items and ratings, and they explore the implications of those dependences on the assessments. Data from two performance assessments are used in two studies of this problem.

Interpreting The Parameters Of A Multidimensional Rasch Model, W Wang, M Wilson, Ray Adams

Prof Ray Adams

This chapter introduces the multidimensional random coefficient multinomial logit model (MRCML) which can be applied to multidimensional polytomous test items. Two examples of real data analyses are followed to demonstrates the applications of the MRCML. A detailed discussion is made of interpretation of item parameters, in comparision to those derived from their corresponding unidimensional Rasch measurement models. Finally, it provides some guidelines for constructing scoring matrices and the corresponding models to aid parameter interpretation.

Measuring Individual Differences In Change With Multidimensional Rasch Models, W Wang, M Wilson, Ray Adams

Prof Ray Adams

Item response models have been developed to explore change measurement, including those proposed by Fischer and his colleagues. In this article the authors propose another multidimensional Rasch model, the multidimensional random coefficient multinomial logit (MRCML) model. All these models are briefly reviewed and compared. The MRCML can be applied to not only polytomous items but also investigation of variations in item difficulties. Based on variations in difficulties across occasions and items, five kinds of models are proposed. Some simulation studies were conducted to examine parameter recovery of the MRCML model under various testing situations. All the parameters were recovered very …

The Multidimensional Random Coefficients Multinomial Logit Model, Ray Adams, M Wilson, W Wang

Prof Ray Adams

A multidimensional Rasch-type item response model, the multidimensional random coefficients multinomial logit model, is presented as an extension to the Adams and Wilson (1996) random coefficients multinomial logit model. The model is developed in a form that permits generalization to the multidimensional case of a wide class of Rasch models, including the simple logistic model, Masters' partial credit model, Wilson's ordered partition model, and Fischer's linear logistic model. Moreover, the model includes several existing multidimensional models as special cases, including Whitely's multicomponent latent trait model, Andersen's multidimensional Rasch model for repeated testing, and Embretson's multidimensional Rasch model for learning and …

Reading Comprehension Sub-Skills: Teachers' Perceptions Of Content In An Eap Test, Tom Lumley

Dr Tom Lumley

This study examines the place of sub skills in English as second language (ESL) syllabus and test design, with particular attention to the enduring influence of Munby (1978). Rasch analysis of item difficulty and the use of Rasch Item Response Theory (IRT) are discussed.

Mapping Abilities And Skill Levels Using Rasch Techniques, Annie Brown, Catherine Elder, Tom Lumley, Tim Mcnamara, Joy Mcqueen

Dr Tom Lumley

The ongoing interest in investigating the use of Item Response Theory (IRT) methods, involving the Rasch model, in language testing research and the development of language tests has encouraged the authors to explore the nature of this approach and to introduce some research on its validation.

The Analysis Of Partial Credit Scoring, Geoff Masters

Prof Geoff Masters AO

This article discusses a range of issues in the practical application of an item response theory (IRT) method for partial credit scoring. After a brief discussion of partial credit scoring as an alternative to right-wrong scoring in the measurment of educational achievement, an IRT model for partial credit analysis is developed and described. This model is presented as a straightforward and logical application of Rasch's dichotomous model to a sequence of ordered response alternatives. The distinctive nature of the item parameters in the model is described and these parameters are contrasted with two more familiar sets of parameters: Thurstone thresholds …

Banking Non-Dichotomously Scored Items, Geoff Masters, John Evans

Prof Geoff Masters AO

A method for constructing a bank of items scored in two or more ordered response categories is described and illustrated. This method enables multistep problems, rating scale items, question 'clusters', and other items using partial credit scoring to be calibrated and incorporated into an item bank, and it provides a mechanism for computer adaptive testing with items of this type. Procedures are described for calibrating an initial set of items, for testing the fit of items to the underlying measurement model, and for linking new items to an existing item bank. The method is illustrated using items from the Watson-Glaser …

Common Person Equating With The Rasch Model, Geoff Masters

Prof Geoff Masters AO

Two procedures, one based on item difficulties, the other based on person abilities, were used to equate 14 forms of a reading comprehension test using the Rasch model. These forms had no items in common. For practical purposes, the two procedures produced equivalent results. An advantage of common person equating for testing the unidimensionality assumption is pointed out, and the need for caution in interpreting tests of common item invariance is stressed.

Dicot: Analysing Classroom Tests With The Rasch Model, Geoff Masters

Prof Geoff Masters AO

A computer program (DICOT) for the Rasch analysis of classroom tests is described. Results are presented in a simple, self-explanatory form, and person ability and item difficulty estimates are expressed in a metric like the one with which teachers and parents are already familiar. Person and item fit statistics provide an opportunity to diagnose strengths and weaknesses of individual children and to identify items which are problematic.

Defining A 'Fear-Of-Crime' Variable: A Comparison Of Two Rasch Models, Geoff Masters

Prof Geoff Masters AO

No abstract provided.