Education Commons^™

National Assessment Program : Ict Literacy Years 6 & 10 Report 2014, Julian Fraillon, Wolfram Schulz, Eveline Gebhardt, John Ainley

Julian Fraillon

Literacy in information and communication technology (ICT) is critical to students as they progress through schooling and enter a world in which information technology will be ubiquitous. Work, health care, family finances, learning and social interaction will all depend on competence in ICT. To assess progress in this crucial part of student learning, ACARA conducts a National Assessment Program (NAP) aimed at measuring ICT literacy. Every three years since 2005, a sample of Year 6 and Year 10 students from across Australia have been tested to determine their ICT knowledge, understanding and skills and their ability to use ICT creatively, …

Assessing The Intended Participation Of Young Adolescents As Future Citizens, John Ainley, Wolfram Schulz, Julian Fraillon

Julian Fraillon

In many Western countries there is concern about the level of participation of their citizens in civic life and the apparent lack of interest and involvement among young people in public and political life (Curtice & Seyd, 2003). The development of knowledge, understanding, skills, and dispositions that prepare young people to comprehend the world, hold productive employment, and be informed active citizens are among the characteristics that educational systems, schools, and teachers value and attempt to foster. However, countries vary in the status accorded to civic and citizenship education as part of school education and the relative importance of developing …

Group Of National Experts On The Ahelo Feasibility Study : Engineering Assessment Framework : Ahelo Feasibility Study, Hamish Coates, Sarah Richardson, Jacob Pearce, Julian Fraillon

Julian Fraillon

This AHELO Engineering Assessment Framework describes and illustrates the domain of engineering competency that is used in the AHELO Feasibility Study. The framework gives an organisational structure for the domain in terms of engineering knowledge, processes and contexts; describes the types of assessment items that have been developed; and details how reporting will be carried out. - p.4

International Computer And Information Literacy Study: Icils 2013: Technical Report, Julian Fraillon, Wolfram Schulz, Tim Friedman, John Ainley, Eveline Gebhardt

Julian Fraillon

The International Computer and Information Literacy Study (ICILS) studied the extent to which young people have developed computer and information literacy (CIL) to support their capacity to participate in the digital age. ICILS systematically investigated differences in CIL outcomes across the participating countries and looked at how these countries provide CIL-related education. The study also explored differences within and across countries with respect to the relationship between the outcomes of CIL education and student characteristics and school contexts. Chapter 1 of this report presents an overview of the International Computer and Information Literacy Study. Chapters 2 and 3 are concerned …

The Rationale For And Use Of Assessment Frameworks: Improving Assessment And Reporting Quality In Medical Education, Jacob Pearce, Daniel Edwards, Julian Fraillon, Hamish Coates, Benedict Canny, David Wilkinson

Julian Fraillon

An assessment framework provides a structured conceptual map of the learning outcomes of a programme of study along with details of how achievement of the outcomes can be measured. The rationale for using frameworks to underpin the targeting of essential content components is especially relevant for the medical education community. Frameworks have the capacity to improve validity and reliability in assessment, allowing test developers to more easily create robust assessment instruments. The framework used by the Australian Medical Assessment Collaboration (AMAC) is an interesting and relevant case study for the international community as it draws and builds on established processes …

Study Flags Civics And Citizenship Knowledge, Julian Fraillon

Julian Fraillon

The percentage of Australian school students with the skills and knowledge required to be active and informed citizens has remained stable between 2010 and 2013, but there is room for improvement, as Julian Fraillon explains.

Computer-Based Assessment Systems, Julian Fraillon, Ralph Carstens

Julian Fraillon

ICILS was the first IEA study to collect student achievement and questionnaire data on computer rather than on paper. This chapter describes the key aspects of the computerbased test delivery system used in ICILS. It also details some of the challenges of using computer-based systems to collect student data in a crossnational large-scale assessment such as ICILS.

Icils Test Development, Julian Fraillon

Julian Fraillon

The ICILS assessment was developed over a 20-month period from April 2010 to December 2012. Most of this work was conducted by the international study center (ISC) at the Australian Council for Educational Research (ACER) in collaboration with national research coordinators (NRCs) and the project advisory committee (PAC). SoNET Systems conducted the software development for the test modules. This chapter provides a detailed description of the test development process and review procedures as well as the test design implemented for the ICILS field trial and main survey.

Australian Students Prepare For Digital Future, Julian Fraillon

Julian Fraillon

The world’s first computer-based international study into computer and information literacy shows that Australian students are developing the skills necessary for successful participation in the digital world. Julian Fraillon reports.

National Assessment Program : Ict Literacy 2008 : Year 6 And Year 10 : Technical Report, Chris Freeman, John Ainley, Julian Fraillon, Eveline Gebhardt, Kate O'Malley, Wolfram Schulz

Julian Fraillon

This report describes the technical aspects of the National ICT Literacy Sample Assessment and summarises the main activities involved in the data collection, the data collection instruments and the analysis and reporting of the data. Chapter 2 summarises the development of the assessment domain and describes the process of item development and construction of the instruments. Chapter 3 reviews the sample design and describes the sampling process. Chapter 3 also describes the weighting procedures that were implemented to derive population estimates. Chapter 4 summarises the field administration and data management procedures, including quality control and the cleaning and coding of …

Measuring Young People’S Understanding Of Civics And Citizenship In A Cross-National Study., Wolfram Schulz, Julian Fraillon, John Ainley

Julian Fraillon

The development of knowledge and understanding is widely reported as a key aim of civic and citizenship education and as an important aspect of being equipped for active citizenship. Measuring knowledge and understanding of this domain has grown as a focus of educational research both as an outcome in its own right and to underpin the systematic investigation of factors associated with the development of effective active citizenship. Given that civics and citizenship is often seen as a set of cross-curricular capabilities rather than an established discipline, it is crucial to articulate clearly what is being measured and to be …

Assessing Higher Education Learning Outcomes In Civil Engineering: The Oecd Ahelo Feasibility Study, Roger Hadgraft, Jacob Pearce, Daniel Edwards, Julian Fraillon, Hamish Coates

Julian Fraillon

BACKGROUND

Higher education is increasingly a global business. At present, university ranking schemes are heavily reliant on research indicators, while students are likely looking for an excellent teaching and learning environment. Aware of this discrepancy, the OECD has funded the AHELO project.

PURPOSE

AHELO was designed to test the feasibility of an international assessment of higher education learning outcomes. The two test disciplines are civil engineering and economics, together with an assessment of generic skills. The civil engineering test will be reported in this paper.

More than a ranking, AHELO is a direct evaluation of student performance. It is intended …

National Assessment Program : Information Communication And Technology Literacy : Year 6 And Year 10 : Technical Report 2005, John Ainley, Julian Fraillon, Chris Freeman, Martin Murphy

Julian Fraillon

This report describes the technical aspects of the National ITC Literacy Sample Assessment and summarises the main activities involved in the data collection, the data collection instruments and the analysis and reporting of the data. Chapter 2 summarises the development of the assessment domain and describes the process of item development and construction of the instruments. Chapter 3 reviews the sample design and describes the sampling process. Chapter 3 also describes the process of weighting to derive population estimates. Chapter 4 summarises the field administration and data management procedures, including quality control and the cleaning and coding of the data. …

National Assessment Program : Ict Literacy 2011 : Year 6 And Year 10 : Technical Report, Eveline Gebhardt, Julian Fraillon, Wolfram Schulz, Kate O'Malley, Chris Freeman, Martin Murphy, John Ainley

Julian Fraillon

This report describes the technical aspects of the National Assessment Program – ICT Literacy (NAP – ICTL) 2011 sample assessment and summarises the main activities involved in the data collection, the data collection instruments and the analysis and reporting of the data. Chapter 2 summarises the development of the assessment domain and describes the process of item development and construction of the instruments. Chapter 3 reviews the sample design and describes the sampling process. It also describes the weighting procedures that were implemented to derive population estimates and calculation of participation rates. Chapter 4 summarises the field administration of the …

Scaling Procedures For Iccs Test Items, Wolfram Schulz, Julian Fraillon

Julian Fraillon

This chapter describes the procedures used to analyse and scale the International Civic and Citizenship Education Study (ICCS) international and regional test items that were administered to measure students’ civic knowledge. The chapter covers the following topics: the scaling model used to analyze and scale the test items; test coverage and item dimensionality; assessment of item fit; assessment of scorer reliabilities for open-ended items; differential item functioning by gender; review of cross-national measurement equivalence; international item adjudication; international item calibration and test reliability; international ability estimates (plausible values and weighted likelihood estimates); estimation of changes in civic content knowledge between …

National Assessment Program : Ict Literacy Years 6 And 10 Report 2011, John Ainley, Julian Fraillon, Eveline Gebhardt, Wolfram Schulz

Julian Fraillon

Under the National Assessment Program, national samples of Year 6 and Year 10 students were assessed to determine their levels of confidence, creativity and skill development in the use of information and communication technologies. This report compares the results of Australian school students by state and territory and student sub-groups, and provides details of their achievement against an ICT literacy scale. It also enables the most recent achievements of students to be compared against those from the previous national assessments of ICT literacy, conducted first in 2005 and again in 2008. In addition, a survey of student access to and …

The Development Of Regional Instruments, David Kerr, Wolfram Schulz, Julian Fraillon

Julian Fraillon

The International Civic and Citizenship Education Study (ICCS) included regional modules for Europe, Latin America, and Asia. Thirty-five of the 38 countries that participated in ICCS opted to be involved in these regional modules. Twenty-four countries participated in the European module, six in the Latin American module, and five countries in the Asian module. Regional instruments were developed for each module and were administered to students after they had completed the international assessment. These instruments were designed to address aspects of civic and citizenship education specific to each region. The results of these additional data collections are reported in a …

Students’ Participation In And Valuing Of Civic Engagement At School, Wolfram Schulz, Julian Fraillon

Julian Fraillon

The IEA International Civic and Citizenship Education Study studied the ways in which young people in lower secondary schools are prepared to undertake their roles as citizens in a wide range of countries including Europe, Latin America, and the Asian-Pacific region. ICCS was the third IEA study designed to measure contexts and outcomes of civic and citizenship education and was linked to the 1999 IEA Civic Education Study (CIVED). A central aspect of the study was the assessment of student knowledge about a wide range of civic-related issues. ICCS gathered data from more than 140,000 Grade 8 (or equivalent) students …

The Analysis Of Measurement Equivalence In International Studies Using The Rasch Model, Wolfram Schulz, Julian Fraillon

Julian Fraillon

When comparing data derived from tests or questionnaires in cross-national studies, researchers commonly assume measurement invariance in their underlying scaling models. However, different cultural contexts, languages, and curricula can have powerful effects on how students respond in different countries. This article illustrates how the application of the Rasch item response theory (IRT) model (Rasch, 1960) can be used for assessing differences in measurement properties of tests and questionnaires with reference to examples from the field trial analyses for the International Association for the Evaluation of Educational Achievement (IEA) International Civic and Citizenship Education Study (ICCS). It also discusses the general …

Iccs Test Development, Julian Fraillon

Julian Fraillon

The ICCS civic knowledge assessment was developed over an 18-month period from October 2006 to April 2008. Most of the ICCS test-item development was conducted by the international study center (ISC) at the Australian Council for Educational Research (ACER) in collaboration with the study’s national research coordinators (NRCs) and the Project Advisory Committee (PAC). This chapter provides a detailed description of the test-development process, review procedures, and the test design implemented for the ICCS field trial and main survey. Table 2.1 provides an overview of the test-development processes and timeline.

The Analysis Of Measurement Equivalence In International Studies Using The Rasch Model., Wolfram Schulz, Julian Fraillon

Julian Fraillon

This paper outlines how Rasch modelling can be used for testing assumption about measurement equivalence in international studies. Based on the Rasch model (Rasch, 1960) for dichotomous items and the Partial Credit model for polytomous items (Masters & Wright, 1982), so-called item-by-country interactions can be estimated that provide useful information about the level of measurement invariance in parameter estimates. It should be noted that item-by-country interactions are one form of differential item functioning (DIF) where groups vary in their probability of answering questions even after controlling for their levels of ability (see Hambleton, Swaminathan and Rogers, 1991). Item-by-country interactions can …

National Assessment Program : Ict Literacy Years 6 & 10 Report 2005, John Ainley, Julian Fraillon, Chris Freeman

Julian Fraillon

Australia's national goals for schooling assert that when students leave school they should be: confident, creative and productive users of new technologies, particularly information and communication technologies, and understand the impact of those technologies on society. The Australian National Assessment Program includes the systematic assessment of the extent to which this goal is being achieved through triennial sample surveys of students in Years 6 and 10. This report is based on the assessment of ICT literacy conducted in October 2005. It describes the development of a computer-based tool for assessing ICT literacy among school students and the application of that …

National Assessment Program : Ict Literacy Years 6 & 10 Report 2008, John Ainley, Julian Fraillon, Chris Freeman

Julian Fraillon

This report presents the findings from the National Assessment Program –ICT literacy assessment conducted in 2008 under the auspices of the national council of education ministers, the Ministerial Council for Education, Early Childhood Development and Youth Affairs (MCEECDYA). National samples of Year 6 and Year 10 students were assessed to determine their levels of confidence, creativity and skill development in the use of information and communication technologies. This report compares the results of Australian school students by state and territory and student sub-groups, and provides details of their achievement against an ICT literacy scale. It also enables the most recent …

International Civic And Citizenship Education Study : Assessment Framework, Wolfram Schulz, Julian Fraillon, John Ainley, Bruno Losito, David Kerr

Julian Fraillon

The International Civic and Citizenship Study (ICCS) is the third lEA project investigating the role of schooling in preparing young people for their roles as citizens in society. The first study in this area began in 1971. The second Civic Education Study was undertaken in 1999. Almost 10 years on, global change has again prompted a new survey of civic and citizenship education. Amongst other considerations, the growing impact of the processes of globalization, external threats to civic societies and their freedoms, and the limited interest and involvement of young generations in public and political life. [p. 5, ed]

National Assessment Program : Civics And Citizenship Years 6 And 10 Report 2004, Suzanne Mellor, John Ainley, Julian Fraillon, Nicole Wernert

Julian Fraillon

The National Civics and Citizenship Sample Assessment was conducted in October 2004 with 10,712 Year 6 students from 318 schools and 9,536 Year 10 students from 249 schools. At both year levels, a sample of schools was selected with a probability proportional to size and then a sample of up to two classrooms was selected at random from those schools. The sample design and procedures, the high response rates (more than 90 percent) and the low levels of exclusions ensured that there was very little bias in the sample. The assessment was representative of the elements identified in the assessment …