Education Commons^™

Item Differential In Computer Based And Paper Based Versions Of A High Stakes Tertiary Entrance Test: Diagrams And The Problem Of Annotation, Brad Jackel

Dr Brad Jackel

This paper presents the results from a tertiary entrance test that was delivered to two groups of candidates, one as a paper based test and the other as a computer based test. Item level differential reveals a pattern that appears related to item type: questions based on diagrammatic stimulus show a pattern of increased difficulty when delivered on computer. Differential in performance was not present in other sections of the test and it would appear unlikely to be explained by demographic differences between the groups. It is suggested this differential is due to the inability of the candidates to freely …

What Information From Pisa Is Useful For Teachers? How Can Pisa Help Our Students To Become More Proficient?, Juliette Mendelovits, Dara Searle, Tom Lumley

Juliette Mendelovits

A frequent objection to large-scale testing programs, both national and international, is that they are used as an instrument of control, rather than as a means of providing information to effect change. Moreover, concerns about large-scale testing often take the form of objection to the specific characteristics of the assessments as being prescriptive and proscriptive, leading to a narrowing of the curriculum and the spectre of 'teaching to the test' to the exclusion of more important educational content. Taking PISA reading literacy as its focus, this paper proposes, on the contrary, that a coherent assessment system is valuable in so …

Is There A Relationship Between Chemistry Performance And Question Type, Question Content And Gender?, Ross Hudson

Dr Ross Hudson

This research inquires into the effectiveness of the two predominant forms of questions - multiple-choice questions and short-answer questions - used in the State University Entrance Examination for Chemistry including the relationship between performance and gender. It examines not only the style of question but also the content type examined (recall and application questions). The research involves class trial testing of students with structured questions that examine the same material content with each type of question (multiple-choice or short-answer) and also examines the different type of content (recall or application) and finally the influence of student gender. Rasch analysis of …

Cartoons And Linguistic Context, Brad Jackel

Dr Brad Jackel

This paper presents the results of a trial conducted during the development of The Graduate Australian Medical School Admission Test (GAMSAT) 2006, where two versions of a cartoon unit were trialled; in both cases the cartoon and the question itself were identical. The only difference was in the way the cartoon was introduced within the test. Psychometrically, across the 650 candidates involved in the trial, one version ‘worked’ very well, the other not at all.

Technological Issues For Computer-Based Assessment, Beno Csapó, John Ainley, Randy Bennett, Thibaud Latour, Nancy Law

Dr John Ainley

This chapter reviews the contribution of new information-communication technologies to the advancement of educational assessment. Improvements can be described in terms of precision in detecting the actual values of the observed variables, efficiency in collecting and processing information, and speed and frequency of feedback given to the participants and stakeholders. The chapter reviews previous research and development in two ways, describing the main tendencies in four continents (Asia, Australia, Europe and the US) as well as summarising research on how technology advances assessment in certain crucial dimensions (assessment of established constructs, extension of assessment domains, assessment of new constructs and …

Multiple-Choice Questions Compared To Short-Answer Response : Which Assesses Understanding Of Chemistry More Effectively?, Ross Hudson

Dr Ross Hudson

The research inquires into the effectiveness of the two predominant forms of questions that are used on the Victorian Certificate of Education (VCE) Victoria Chemistry examination. These are multiple- choice questions and short-answer questions. This research examines not only the style of chemistry question but also the content type examined (recall and application questions) along with gender differences in students' responses to such questions. The research involved three phases, i) analysis of five years results from the VCE Chemistry examinations, ii) class trial testing students of both genders with structured questions that examined the same material content with each type …

What Information From Pisa Is Useful For Teachers? How Can Pisa Help Our Students To Become More Proficient?, Juliette Mendelovits, Dara Searle, Tom Lumley

Dr Tom Lumley

A frequent objection to large-scale testing programs, both national and international, is that they are used as an instrument of control, rather than as a means of providing information to effect change. Moreover, concerns about large-scale testing often take the form of objection to the specific characteristics of the assessments as being prescriptive and proscriptive, leading to a narrowing of the curriculum and the spectre of 'teaching to the test' to the exclusion of more important educational content. Taking PISA reading literacy as its focus, this paper proposes, on the contrary, that a coherent assessment system is valuable in so …

Learning Mathematics For Life : A Perspective From Pisa, Jan De Lange, Alla Routitsky, Kaye Stacey, Ross Turner, Margaret Wu, Andreas Schleicher, Claire Shewbridge, Pablo Zoido, Nicola Clements

Dr Alla Routitsky

People from many countries have expressed interest in the tests students take for the Programme for International Student Assessment (PISA). This book examines the link between the PISA test requirements and student performance. It focuses specifically on the proportions of students who answer questions correctly across a range of difficulty. The questions are classified by content, competencies, context and format, and the connections between these and student performance are then analysed. This analysis has been carried out in an effort to link PISA results to curricular programs and structures in participating countries and economies. Results from the student assessment reflect …

Affective Engagement : A Person-Centred Approach To Understanding The Structure Of Subjective Learning Experiences, Sarah Buckley, Galit Hasen, Mary Ainley

Dr Sarah Buckley

Accounts of students' learning have increasingly emphasised the role of affective engagement in achievement settings. Although most studies have focused on negative emotional experiences such as anxiety, more recent studies have investigated the role of positive emotions. This study examines the structure of students' subjective learning experiences in relation to individual interest profiles. It measured two components of affect: activation as positive arousal that indicates engagement, and valence as an evaluative quality of the students' experience. Senior secondary students (females, N=162) completed measures of individual interests, curiosity and prior knowledge, read three social issues texts and then answered some questions …

The Effect Of Test-Taker Sex, Audience And Topic On Task Performance In Tape-Mediated Assessment Of Speaking, Tom Lumley, Barry O'Sullivan

Dr Tom Lumley

There is growing interest in the effect on candidate performance of characteristics of the interlocutor in tests of speaking. A range of variables associated with the interlocutor may cause systematic variation in linguistic performance, and consequently in scores awarded. This paper hypothesises that there may be effects on performance attributable to an interaction of these variables; in the present study, the task topic, the sex of the person presenting the topic and the sex of the candidate. This investigated in the context of a tape-mediated test of speaking, where no interlocutor is actually present; instead, stimulus material is presented by …

Enhanced Mathematics Learning: Does Technology Make A Difference?, Katherine Dix

Dr Katherine Dix

Linguistic And Cultural Norms In Language Testing : A Case Study, Annie Brown, Tom Lumley

Dr Tom Lumley

Users of English in Asia may not have the same need to aspire to a ' standard' form of English. It may be more appropriate for them to develop a communicative competence employing the sociolinguistic and cultural norms of the region. Taking such an approach in the development of tests of English proficiency involves rethinking basic assumptions. This paper considers these issues in the context of a test of English proficiency developed for use with English teachers in Indonesia

Mapping Abilities And Skill Levels Using Rasch Techniques, Annie Brown, Catherine Elder, Tom Lumley, Tim Mcnamara, Joy Mcqueen

Dr Tom Lumley

The ongoing interest in investigating the use of Item Response Theory (IRT) methods, involving the Rasch model, in language testing research and the development of language tests has encouraged the authors to explore the nature of this approach and to introduce some research on its validation.

Reforming The Assessment Of Student Achievement In The Senior Secondary School, Geoff Masters, P Hill

Prof Geoff Masters AO

The challenge that confronts agencies responsible for assessment and reporting in the senior secondary school is to extend systematic assessment procedures to broader range of learning outcomes than those currently assessed by public examination, to develop methods of reporting which are more descriptive of individual achievement and which provide a better basis for describing and maintaining standards, and to provide results which are sufficiently comparable across schools to enable fair comparisons of applicants for tertiary study. Some recent developments in assessment and reporting practice are considered with a view to identifying methods and approaches capable of satisfying this diverse set …

Banking Non-Dichotomously Scored Items, Geoff Masters, John Evans

Prof Geoff Masters AO

A method for constructing a bank of items scored in two or more ordered response categories is described and illustrated. This method enables multistep problems, rating scale items, question 'clusters', and other items using partial credit scoring to be calibrated and incorporated into an item bank, and it provides a mechanism for computer adaptive testing with items of this type. Procedures are described for calibrating an initial set of items, for testing the fit of items to the underlying measurement model, and for linking new items to an existing item bank. The method is illustrated using items from the Watson-Glaser …

Dicot: Analysing Classroom Tests With The Rasch Model, Geoff Masters

Prof Geoff Masters AO

A computer program (DICOT) for the Rasch analysis of classroom tests is described. Results are presented in a simple, self-explanatory form, and person ability and item difficulty estimates are expressed in a metric like the one with which teachers and parents are already familiar. Person and item fit statistics provide an opportunity to diagnose strengths and weaknesses of individual children and to identify items which are problematic.