Education Commons^™

Some Drivers Of Test Item Difficulty In Mathematics, Ross Turner

Ross Turner

This paper is one of four contributions to the symposium session at the AERA’s 2012 Annual Meeting titled Exploring Reading and Mathematics Item Difficulty: Teaching and Learning Implications of PISA Survey Data. The author presents a rubric used to analyse mathematics test items developed for use in the OECD’s PISA survey. The rubric focuses on a set of mathematical competencies that are components of mathematical literacy. The work on which this report is based suggests that demand for activation of these competencies functions as a significant driver of item difficulty, which potentially has implications for the teaching and learning of …

How Well Do Young People Deal With Contradictory And Unreliable Information On Line? What The Pisa Digital Reading Assessment Tells Us, Tom Lumley, Juliette Mendelovits

Juliette Mendelovits

There is sometimes an assumption that young people, as ‘digital natives’, are able to use online information effectively, including selecting and negotiating digital texts that are not only relevant for what they need, but also are likely to provide reliable information. This paper examines the question of how well young people are in fact able to recognise whether information is likely to be trustworthy. While some small-scale work has been done in this area, this paper draws on data from the first large-scale international assessment of online reading, the Digital Reading Assessment (DRA) that was part of the Organisation for …

Evaluation Of Booklet Effect In Pisa Mathematics Across Countries, Luc Le

Dr Luc Tu Le

In PISA 2003, 13 main linked test booklets were constructed by item cluster rotation design. Each student was randomly assigned one of the test booklets. Each of the selected clusters was located in four of the test booklets in different positions. There were seven mathematics clusters with a total of 85 items. Booklet effect was identified and was used to adjust the measure of student performance in PISA (see PISA 2003 technical report; OECD, 2004). This study was designed to explore which factors could account for this booklet effect in mathematics across 41 PISA countries. A three-faceted partial credit model …

A Framework For Predicting Item Difficulty In Reading Tests, Tom Lumley, Alla Routitsky, Juliette Mendelovits, Dara Ramalingam

Dr Tom Lumley

Results on reading tests are typically reported on scales composed of levels, each giving a statement of student achievement or proficiency. The PISA reading scales provide broad descriptions of skill levels associated with reading items, intended to communicate to policy makers and teachers about the reading proficiency of students at different levels. However, the described scales are not explicitly tied to features that predict difficulty. Difficulty is thus treated as an empirical issue, using a post hoc solution, while a priori estimates of item difficulty have tended to be unreliable. Understanding features influencing the difficulty of reading tasks has the …

A Slow Burn: The Impact Of Ials In Australia, Juliette Mendelovits

Juliette Mendelovits

Australia Australia has put its hand up to participate in all the international literacy surveys on offer: IALS in the second wave (1996), the ALL Survey in the second wave (2006), and PIAAC in 2011-12 – in the field right now. Nevertheless, until very recently Australia’s general response to the surveys can best be described as inert: more in the vein of our habitual “she’ll be right” than the “clever country” that our prime minister of the 1980s urged us to become. For most of the period since the IALS results were published in 1997, Australia paid little attention to …

Differential Performance Of Poor Readers In International Mathematics Tests By Item Formats, Luc Le

Dr Luc Tu Le

The important role of language, including reading, on student performance on assessments in school academic areas such as mathematics has been addressed in number of studies. Although positive relations between different aspects of reading and mathematics performance have been investigated, there have been few studies focusing on the interaction between the context of the mathematics item and student reading proficiency, and most of those there were related only to the verbal loading. Recently, a low correlation between verbal loading and item difficulty was found for PISA mathematics items. This study used PISA 2003 mathematics data with 41 countries with about …

Pisa And The ‘Latest Model’: Can An International Survey Have Anything Useful To Say To Canberra’S Mathematics Teachers?, Ross Turner

Ross Turner

Identifying Cognitive Processes Important To Mathematics Learning But Often Overlooked, Ross Turner

Ross Turner

Pisa In Brief : Highlights From The Full Australian Report : Challenges For Australian Education : Results From Pisa 2009 : The Pisa 2009 Assessment Of Students' Reading, Mathematical And Scientific Literacy., Sue Thomson, Lisa De Bortoli, Marina Nicholas, Kylie Hillman, Sarah Buckley

Lisa De Bortoli

No abstract provided.

Pisa Mathematics - An Insider's View, Ross Turner

Ross Turner

Learning Mathematics For Life : A Perspective From Pisa, Jan De Lange, Alla Routitsky, Kaye Stacey, Ross Turner, Margaret Wu, Andreas Schleicher, Claire Shewbridge, Pablo Zoido, Nicola Clements

Dr Alla Routitsky

People from many countries have expressed interest in the tests students take for the Programme for International Student Assessment (PISA). This book examines the link between the PISA test requirements and student performance. It focuses specifically on the proportions of students who answer questions correctly across a range of difficulty. The questions are classified by content, competencies, context and format, and the connections between these and student performance are then analysed. This analysis has been carried out in an effort to link PISA results to curricular programs and structures in participating countries and economies. Results from the student assessment reflect …

A New Role For Item Aficionados In High-Stakes Testing Programs, Gabrielle Matters

Dr Gabrielle Matters

The purpose of this paper is to point teachers, test analysts, and users of test results to the significance of student responses at the item level and considering what it is that each item purports to measure and actually measures before taking the evidence of a low score on a test − just a score derived from a collection of items − and coming to the seemingly obvious but not necessarily accurate conclusion that the student has no knowledge or understanding of the domain being tested. An item aficionado does not approach test items and test results at the level …

Investigating Gender Differential Item Functioning Across Countries And Test Languages For Pisa Science Items, Luc Le

Dr Luc Tu Le

This study uses PISA cycle 3 field trial data to investigate the relationships between gender differential item functioning (DIF) across countries and test languages for science items and their formats and the four other dimensions defined in PISA framework: focus, context, competency, and scientific knowledge. The data used were collected from 60 test language groups by 50 participating countries with a total of about 83,000 15-year-old students. An IRT method is used to detect the gender uniform DIF for each of the language groups and the whole international sample. Gender directions have been drawn for each of the five item …

Learning About Students' Mathematical Literacy From Pisa 2003, S Mccrone, J Dossey, Ross Turner, M Lindquist

Ross Turner

Pisa 2003—Mathematical Literacy And Learning In The Americas, John Dossey, Sharon Mccrone, Ross Turner, Mary Lindquist

Ross Turner

Ameliorating Culturally Based Extreme Response Tendencies To Attitude Items, Maurice Walker

Maurice Walker

No abstract provided.

Pisa In Brief From Australia's Perspective : Highlights From The Full Australian Report : Exploring Scientific Literacy : How Australia Measures Up : The Pisa 2006 Assessment Of Students' Scientific, Reading And Mathematical Literacy Skills., Sue Thomson, Lisa De Bortoli

Lisa De Bortoli

No abstract provided.

The Influence Of Equating Methodology On Reported Trends In Pisa, Eveline Gebhardt, Ray Adams

Prof Ray Adams

In 2005 PISA published trend indicators that compared the results of PISA 2000 and PISA 2003. This paper explores the extent to which the outcomes of these trend analyses are sensitive to the choice of test equating methodologies, the choice of regression models and the choice of linking items. To establish trends, PISA equated its 2000 and 2003 tests using a methodology based on Rasch Modelling that involved estimating linear transformations that mapped 2003 Rasch-scaled scores to the previously established PISA 2000 Rasch-scaled scores. This paper compares the outcomes of this approach with an alternative, which involves the joint Rasch …

Developing Mathematical Literacy, Ross Turner, L Steen, H Burkhardt

Ross Turner

Modelling And Applications In Pisa, Ross Turner

Ross Turner

Pisa – The Programme For International Student Assessment – An Overview, Ross Turner

Ross Turner

El Programa Internacional Para La Evaluación De Los Alumnos (Pisa). Una Perspectiva General, Ross Turner

Ross Turner

Application Of Multivariate Rasch Models In International Large-Scale Educational Assessments, Ray Adams, Margaret Wu, C Carstensen

Prof Ray Adams

In large-scale educational assessments, such as the Programme for International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS), a primary concern is with the estimation of the population-level characteristics of a number of latent variables and the relationships between latent variables and other variables. Typically these studies are undertaken in contexts in which there are constraints on sample size and individual student response time, yet there are high expectations with regard to the breadth of content coverage. These demands and constraints have resulted in such studies using rotated-booklet designs, with each student responding to a …

Sampling For National Surveys In Education, Martin Murphy, Wolfram Sculz

Martin Murphy

The aim of this paper is to provide PMRT and its subgroups with information about the sampling processes used in surveys conducted under its National Assessment Plan (NAP). The design of national surveys in education in recent years has been strongly influenced by the methodologies used in major international surveys, particularly TIMSS and PISA. In the discussion below, reference will be made to the methods used in these surveys, and their application in the context of Australian national surveys. The intention of this paper is to explain the major steps in the survey sampling process, the reasons for the methods …

Response To 'Cautions On Oecd's Recent Educational Survey (Pisa), Ray Adams

Prof Ray Adams

This paper refers to the work of S J Prais who questioned the outcomes of the Organisation for Economic Cooperation and Development's PISA survey of the reading, mathematics and science attainments of fifteen year-olds. Prais suggested that methodological flaws in Programme for International Student Assessment (PISA) had resulted in an apparent improvement in the attainment of British students, particularly when compared to their Swiss and German counterparts. This paper responds to Prais's criticisms, noting that when Prais's conjectures are tested with empirical data they are not supported. The author argues that many of Prais's criticisms are due to an incomplete …

Pisa 2000 Technical Report, Ray Adams (Ed), Margaret Wu (Ed)

Professor Ray Adams

The Programme for International Student Assessment (PISA) was launched by the OECD to provide internationally comparable evidence on student performance. This technical report is the product of a collaborative effort between the countries participating in PISA and the experts and the institutions working within the framework of the International PISA Consortium.

The Pisa Item Formats & The Development Of The Pisa Marking Guides, Joy Mcqueen, Juliette Mendelovits, Gayl O'Connor, Margaret Wu

Gayl O'Connor

No abstract provided.

Measuring Student Knowledge And Skills : The Pisa 2000 Assessment Of Reading, Mathematical And Scientific Literacy, Andreas Schieicher, Claudia Tamassia, Geoff Masters, Ray Adams

Prof Ray Adams

Item Panelling, Or Cognitive Walk-Through, Ross Turner

Ross Turner

International Student Assessment For The New Millennium, Ray Adams

Prof Ray Adams

The author briefly describes the OECD initiated project, 'Program for International Student Assessment' (PISA), in which over 100,000 15 year old students from 4000 schools in 26 countries will participate. A major focus of the project is the assessment of wider knowledge, skills and competencies in literacy, mathematics and science areas.