Education Commons^™

On The Use Of Rotated Context Questionnaires In Conjunction With Multilevel Item Response Models, Raymond J. Adams, Petra Lietz, Alla Berezner

Dr Petra Lietz

On The Use Of Rotated Context Questionnaires In Conjunction With Multilevel Item Response Models, Raymond Adams, Petra Lietz, Alla Berezner

Professor Ray Adams

Background: While rotated test booklets have been employed in large-scale assessments to increase the content coverage of the assessments, rotation has not yet been applied to the context questionnaires administered to respondents.

Methods: This paper describes the development of a methodology that uses rotated context questionnaires in conjunction with multilevel item response models and plausible values. In order to examine the impact of this methodology on the continuity of the results, PISA 2006 data for nine heterogeneous countries were rescaled after having been restructured to simulate the outcomes of the use of different rotated context questionnaire designs.

Results: Results revealed …

Measure For Measure : A Review Of Outcomes Of School Education In Australia, John Ainley, Eveline Gebhardt

Dr John Ainley

There have been a plethora of research reports providing information about the achievements of students in Australian schools and how those achievements differ among jurisdictions and among groups of students. However, each report has often been viewed in isolation from other similar studies. This report is intended to provide an integrated appraisal of the results of the international and national achievement surveys conducted since 1994 but with some references to earlier studies. In addition to limiting the task to manageable proportions, this time span corresponds to the widespread introduction of modern measurement methods so that there is a better basis …

A Teacher's Guide To Pisa Scientific Literacy, Sue Thomson, Kylie Hillman, Lisa De Bortoli

OECD Programme for International Student Assessment (PISA) Australia

This report focuses on Australian students' performance on the PISA items that have been released in Scientific Literacy. It includes an overview of the PISA Scientific Literacy Framework, Australian students' results in the 2006 international assessment, examples of responses and marking guides and the context behind achievement, e.g. attitudes, engagement and motivation, and recognition of environmental issues.

Measure For Measure : A Review Of Outcomes Of School Education In Australia, John Ainley, Eveline Gebhardt

Assessment and Reporting

There have been a plethora of research reports providing information about the achievements of students in Australian schools and how those achievements differ among jurisdictions and among groups of students. However, each report has often been viewed in isolation from other similar studies. This report is intended to provide an integrated appraisal of the results of the international and national achievement surveys conducted since 1994 but with some references to earlier studies. In addition to limiting the task to manageable proportions, this time span corresponds to the widespread introduction of modern measurement methods so that there is a better basis …

A Teacher's Guide To Pisa Reading Literacy, Sue Thomson, Kylie Hillman, Lisa De Bortoli

OECD Programme for International Student Assessment (PISA) Australia

This report focuses on Australian students' performance on the PISA items that have been released in reading literacy. It includes an overview of the PISA Reading Framework, Australia's results in the PISA 2009 international assessment , examples of responses and marking guides, and the context behind achievement, e.g. enjoyment of reading, how often they read and high-level strategies for learning.

A Teacher's Guide To Pisa Mathematical Literacy, Sue Thomson, Kylie Hillman, Lisa De Bortoli

OECD Programme for International Student Assessment (PISA) Australia

This report focuses on mathematical literacy and is one of a series of three reports on Australian students' performance in PISA. The report provides an overview of the PISA mathematics framework and Australia's results in the PISA 2003 international assessment. Also included are mathematics items released for public viewing after the PISA 2003 assessment and examples of responses, marking guides and comparisons of results with other countries. The context behind achievement, e.g. attitudes, engagement and learning strategies, is also presented.

Print And Digital Reading In Pisa 2009 : Comparison And Contrast, Juliette Mendelovits, Dara Ramalingam, Tom Lumley

Dara Ramalingam

PISA was administered for the fourth time in 2009. Since in each administration, one of reading, maths or science is chosen as the major domain, the 2009 survey marked the first time that a domain (in this case, reading) was revisited as the major focus of the assessment. This allowed a full review of the framework for reading literacy and the inclusion of new elements to reflect the way that reading has changed since 2000 (OECD, 2009). One such change is the increasing prevalence of digital texts. The assessment of digital reading in the PISA 2009 cycle, undertaken by 19 …

How Well Do Young People Deal With Contradictory And Unreliable Information On Line? What The Pisa Digital Reading Assessment Tells Us, Tom Lumley, Juliette Mendelovits

Dr Tom Lumley

There is sometimes an assumption that young people, as ‘digital natives’, are able to use online information effectively, including selecting and negotiating digital texts that are not only relevant for what they need, but also are likely to provide reliable information. This paper examines the question of how well young people are in fact able to recognise whether information is likely to be trustworthy. While some small-scale work has been done in this area, this paper draws on data from the first large-scale international assessment of online reading, the Digital Reading Assessment (DRA) that was part of the Organisation for …

International Update (No. 3) June 2012

International Update archive (2010-2012)

Table of contents for this issue: (a) Comparing school learning outcomes internationally; (b) Girls outperform boys in reading in PISA 2009+; (c) Investigating computer and information literacy; (d) Discovering civic knowledge, attitudes and engagement; (e) Expertise and networks from Australia to PNG; (d) Education systems must bridge Indigenous and Western worlds; (e) AHELO fieldwork goes live; (f) Innovative international assessment communities evolve; (g) Quality assurance research and its relevance to China; (h) Professional certification can improve teaching quality.

A Framework For Predicting Item Difficulty In Reading Tests, Tom Lumley, Alla Routitsky, Juliette Mendelovits, Dara Ramalingam

Juliette Mendelovits

Results on reading tests are typically reported on scales composed of levels, each giving a statement of student achievement or proficiency. The PISA reading scales provide broad descriptions of skill levels associated with reading items, intended to communicate to policy makers and teachers about the reading proficiency of students at different levels. However, the described scales are not explicitly tied to features that predict difficulty. Difficulty is thus treated as an empirical issue, using a post hoc solution, while a priori estimates of item difficulty have tended to be unreliable. Understanding features influencing the difficulty of reading tasks has the …

Print And Digital Reading In Pisa 2009 : Comparison And Contrast, Juliette Mendelovits, Dara Ramalingam, Tom Lumley

Juliette Mendelovits

PISA was administered for the fourth time in 2009. Since in each administration, one of reading, maths or science is chosen as the major domain, the 2009 survey marked the first time that a domain (in this case, reading) was revisited as the major focus of the assessment. This allowed a full review of the framework for reading literacy and the inclusion of new elements to reflect the way that reading has changed since 2000 (OECD, 2009). One such change is the increasing prevalence of digital texts. The assessment of digital reading in the PISA 2009 cycle, undertaken by 19 …

How Well Do Young People Deal With Contradictory And Unreliable Information On Line? What The Pisa Digital Reading Assessment Tells Us, Tom Lumley, Juliette Mendelovits

Juliette Mendelovits

There is sometimes an assumption that young people, as ‘digital natives’, are able to use online information effectively, including selecting and negotiating digital texts that are not only relevant for what they need, but also are likely to provide reliable information. This paper examines the question of how well young people are in fact able to recognise whether information is likely to be trustworthy. While some small-scale work has been done in this area, this paper draws on data from the first large-scale international assessment of online reading, the Digital Reading Assessment (DRA) that was part of the Organisation for …

Some Drivers Of Test Item Difficulty In Mathematics, Ross Turner

OECD Programme for International Student Assessment (PISA)

This paper is one of four contributions to the symposium session at the AERA’s 2012 Annual Meeting titled Exploring Reading and Mathematics Item Difficulty: Teaching and Learning Implications of PISA Survey Data. The author presents a rubric used to analyse mathematics test items developed for use in the OECD’s PISA survey. The rubric focuses on a set of mathematical competencies that are components of mathematical literacy. The work on which this report is based suggests that demand for activation of these competencies functions as a significant driver of item difficulty, which potentially has implications for the teaching and learning of …

How Well Do Young People Deal With Contradictory And Unreliable Information On Line? What The Pisa Digital Reading Assessment Tells Us, Tom Lumley, Juliette Mendelovits

OECD Programme for International Student Assessment (PISA)

There is sometimes an assumption that young people, as ‘digital natives’, are able to use online information effectively, including selecting and negotiating digital texts that are not only relevant for what they need, but also are likely to provide reliable information. This paper examines the question of how well young people are in fact able to recognise whether information is likely to be trustworthy. While some small-scale work has been done in this area, this paper draws on data from the first large-scale international assessment of online reading, the Digital Reading Assessment (DRA) that was part of the Organisation for …

A Framework For Predicting Item Difficulty In Reading Tests, Tom Lumley, Alla Routitsky, Juliette Mendelovits, Dara Ramalingam

OECD Programme for International Student Assessment (PISA)

Results on reading tests are typically reported on scales composed of levels, each giving a statement of student achievement or proficiency. The PISA reading scales provide broad descriptions of skill levels associated with reading items, intended to communicate to policy makers and teachers about the reading proficiency of students at different levels. However, the described scales are not explicitly tied to features that predict difficulty. Difficulty is thus treated as an empirical issue, using a post hoc solution, while a priori estimates of item difficulty have tended to be unreliable. Understanding features influencing the difficulty of reading tasks has the …

Print And Digital Reading In Pisa 2009 : Comparison And Contrast, Juliette Mendelovits, Dara Ramalingam, Tom Lumley

OECD Programme for International Student Assessment (PISA)

PISA was administered for the fourth time in 2009. Since in each administration, one of reading, maths or science is chosen as the major domain, the 2009 survey marked the first time that a domain (in this case, reading) was revisited as the major focus of the assessment. This allowed a full review of the framework for reading literacy and the inclusion of new elements to reflect the way that reading has changed since 2000 (OECD, 2009). One such change is the increasing prevalence of digital texts. The assessment of digital reading in the PISA 2009 cycle, undertaken by 19 …

Pisa 2009 Technical Report, John Cresswell (Ed)

OECD Programme for International Student Assessment (PISA)

PISA is methodologically highly complex, requiring intensive collaboration among many stakeholders. The successful implementation of PISA depends on the use, and sometimes further development, of state-of-the-art methodologies and technologies. The PISA 2009 Technical Report describes those methodologies, along with other features that have enabled PISA to provide high quality data to support policy formation and review. The descriptions are provided at a level that will enable review and, potentially, replication of the implemented procedures and technical solutions to problems. This report contains a description of the theoretical underpinning of the complex techniques used to create the PISA 2009 Database, which …

A Slow Burn: The Impact Of Ials In Australia, Juliette Mendelovits

Juliette Mendelovits

Australia Australia has put its hand up to participate in all the international literacy surveys on offer: IALS in the second wave (1996), the ALL Survey in the second wave (2006), and PIAAC in 2011-12 – in the field right now. Nevertheless, until very recently Australia’s general response to the surveys can best be described as inert: more in the vein of our habitual “she’ll be right” than the “clever country” that our prime minister of the 1980s urged us to become. For most of the period since the IALS results were published in 1997, Australia paid little attention to …

The Performance Of Students In The Australian Capital Territory On Pisa: Report To The Act Department Of Education And Training, Sue Thomson, Lisa De Bortoli

OECD Programme for International Student Assessment (PISA) Australia

The purpose of this report is to examine the performance of ACT students in the Programme for International Student Assessment (PISA) over the three cycles: Reading literacy in 2000, Mathematical literacy in 2003 and Scientific literacy in 2006. The report provides an opportunity to examine achievement within the territory, in comparison to the rest of the country and internationally.

Effects Of Item Positions On Their Difficulty And Discrimination : A Study In Pisa Science Data Across Test Language And Countries, Luc T. Le

OECD Programme for International Student Assessment (PISA)

This study was based on a four-cluster rotation design of 13 linked test booklets from PISA 2006 science data. It investigated effects of item positions on their difficulty and discrimination parameter estimates obtained from one and two parameter IRT Partial Credit models. The analyses were done separately for 57 test language groups from 53 countries with a total of about 340,000 students.

The results revealed that for all of the test language groups the items tended to become more difficult when they were located later in the test. However, a high linear relationship between the item difficulty estimates by the …

Pisa 2000 Technical Report, Ray Adams (Ed), Margaret Wu (Ed)

Professor Ray Adams

The Programme for International Student Assessment (PISA) was launched by the OECD to provide internationally comparable evidence on student performance. This technical report is the product of a collaborative effort between the countries participating in PISA and the experts and the institutions working within the framework of the International PISA Consortium.

Item Panelling, Or Cognitive Walk-Through, Ross Turner

Ross Turner