Education Commons^™

Measurement Of Young Children's Learning For Program Evaluation, Jen Jackson, Syeda Kashfee Ahmed, Toby Carslake, Petra Lietz

Dr Jen Jackson

The global commitment to early learning has been expressed in the United Nations (UN) Sustainable Development Goals Agenda (SDG) (United Nations, 2016) and access to support for early learning is considered a human right for all children, whether provided by the family, community, or institutional programs (UNESCO, 2013). Inadequate cognitive stimulation has been identified as one of the key psychosocial risk factors associated with poor child development – a factor that is modifiable, with the right interventions (Walker et al., 2007). Thus, insights into how early learning supports may be delivered effectively in various contexts are essential. To explore the …

Review Of The Piaac Numeracy Assessment Framework: Final Report, Dave Tout, Diana Coben, Vince Geiger, Lynda Ginsburg, Kees Hoogland, Terry Maguire, Sue Thomson, Ross Turner

Ross Turner

The Programme for the International Assessment of Adult Competencies (PIAAC) is an international assessment of the proficiency of adults (aged 16-65 years) in key information processing skills (reading, numeracy and Problem Solving in Technology-Rich Environments). The PIAAC Survey of Adult Skills has revealed that a considerable number of adults in OECD countries possess only limited literacy and numeracy skills. The OECD is currently reviewing the content of the frameworks and cognitive assessment instruments for the 2nd cycle of PIAAC ready for delivery in 2021-22. This report is the result of a review of the numeracy construct and assessment in PIAAC. …

Review Of The Piaac Numeracy Assessment Framework: Final Report, Dave Tout, Diana Coben, Vince Geiger, Lynda Ginsburg, Kees Hoogland, Terry Maguire, Sue Thomson, Ross Turner

David (Dave) Tout

The Programme for the International Assessment of Adult Competencies (PIAAC) is an international assessment of the proficiency of adults (aged 16-65 years) in key information processing skills (reading, numeracy and Problem Solving in Technology-Rich Environments). The PIAAC Survey of Adult Skills has revealed that a considerable number of adults in OECD countries possess only limited literacy and numeracy skills. The OECD is currently reviewing the content of the frameworks and cognitive assessment instruments for the 2nd cycle of PIAAC ready for delivery in 2021-22. This report is the result of a review of the numeracy construct and assessment in PIAAC. …

Final Report: Development Of An Online Engagement Scale, Darren Matthews, Ling Tan, Daniel Edwards

Dr Daniel Edwards

The Commonwealth Department of Education and Training commissioned the Australian Council for Educational Research (ACER) to review the Learner Engagement Scale (LES), a key measure of student experience. It is administered as part of the Student Experience Survey, an annual survey of a sample of the 400 000 first and final year undergraduates studying at Australian Higher Education institutions – 80 per cent on-campus and around 8 per cent off-campus. Over the years, LES results have consistently shown substantially lower engagement levels in off-campus students – far in excess of the variation between teaching modes that might be expected – …

Fostering Understanding Of Early Numeracy Development, Kate Reid, Nicola Andrews

Dr Kate Reid

In 2012, the Australian Council for Educational Research (ACER) began the Longitudinal Literacy and Numeracy Study: Transitions from Preschool to School (LLANS:TPS). The study is part of a program of longitudinal literacy and numeracy research at ACER that started with a seven-year longitudinal study of children’s developing literacy and numeracy throughout primary school, which began in 1999 with a cohort of 1000 children from 100 schools around Australia (Meiers et al., 2006). The original Longitudinal Literacy and Numeracy Study (LLANS) developed new instruments for assessing children’s literacy and numeracy understanding in the first three years of primary school and described …

Can Engagement Be Compared? Measuring Academic Engagement For Comparison, Ling Tan, Xiaoxun Sun, Siek Toon Khoo

Dr Siek Toon Khoo

Student engagement is a reflection of active involvement in learning. In digital learning environment, research studies on engagement have been focused on detecting behavioral and psychological engagement indicators from the patterns of activities using feature engineering, but student engagement estimates were rarely compared across sessions or across domains of learning. This paper describes how this could be done by revisiting engagement instrument, diagnosing engagement indicators, estimating engagement parameters, and equating. This study illustrates how engagement reliability can be improved by refining engagement indictors. We demonstrated through DataShop data that student engagement levels can be compared across domains of learning.

Toward A Unified Validation Framework In Mixed Methods Research, Amy B. Dellinger, Nancy L. Leech

Nancy Leech

The primary purpose of this article is to further discussions of validity in mixed methods research by introducing a validation framework to guide thinking about validity in this area. To justify the use of this framework, the authors discuss traditional terminology and validity criteria for quantitative and qualitative research, as well as present recently published validity terminology for mixed methods research. The authors discuss the rationale for their framework and how it unifies thinking about validity in mixed methods research. Finally, they discuss how the framework can be used.

What Makes A Difference? Measuring Non-Academic Outcomes, Prue Anderson, Julian Fraillon

Prue Anderson

There are some very real challenges that educators face in attempting to define, measure and improve the non-academic outcomes of students in Australian schools but there is plenty that can be done to better measure and influence them. There are many challenges in measuring non-academic outcomes but the overarching one is to determine what they are, and whether they are actually 'non-academic' at all. This article addresses each of three major challenges: (1) Defining the outcomes, followed by a recommendation; (2) measuring the outcomes; recommendation; and (3) improving outcomes; recommendation.

Group Of National Experts On The Ahelo Feasibility Study : Engineering Assessment Framework : Ahelo Feasibility Study, Hamish Coates, Sarah Richardson, Jacob Pearce, Julian Fraillon

Julian Fraillon

This AHELO Engineering Assessment Framework describes and illustrates the domain of engineering competency that is used in the AHELO Feasibility Study. The framework gives an organisational structure for the domain in terms of engineering knowledge, processes and contexts; describes the types of assessment items that have been developed; and details how reporting will be carried out. - p.4

Assessing Vocational Competencies In Civil Engineering: Lessons From Ahelo For Future Practice, Jacob Pearce

Dr Jacob Pearce

There has been much interest in the notion of vocational competencies of late. The pressing question, however, is how to measure vocational competencies. In order to contribute to this ‘how’, the results of the Civil Engineering strand of the recent Assessment of Higher Education Learning Outcomes (AHELO) Feasibility Study are discussed. The focus of the assessment instrument was explicitly on “above content” areas of the domain, and the assessment framework provides an example of a construct which articulates certain domain-specific competence components. This paper draws specific attention the Constructed Response Tasks (CRTs) in the instrument and discusses the results of …

Scaling Procedures For Icils Test Items, Eveline Gebhardt, Wolfram Schulz

Dr Wolfram Schulz

This chapter describes the procedures used to analyze and scale the ICILS test items that were administered to measure students’ computer and information literacy (CIL).

Measuring Learning Growth In A World Of Universal Education, Ross Turner

Ross Turner

ACER is leading the development of new ‘learning metrics’ – tools for measuring learning growth – as Ross Turner explains.

National Assessment Program : Ict Literacy 2008 : Year 6 And Year 10 : Technical Report, Chris Freeman, John Ainley, Julian Fraillon, Eveline Gebhardt, Kate O'Malley, Wolfram Schulz

Dr Wolfram Schulz

This report describes the technical aspects of the National ICT Literacy Sample Assessment and summarises the main activities involved in the data collection, the data collection instruments and the analysis and reporting of the data. Chapter 2 summarises the development of the assessment domain and describes the process of item development and construction of the instruments. Chapter 3 reviews the sample design and describes the sampling process. Chapter 3 also describes the weighting procedures that were implemented to derive population estimates. Chapter 4 summarises the field administration and data management procedures, including quality control and the cleaning and coding of …

Can Engagement Be Compared? Measuring Academic Engagement For Comparison, Ling Tan, Xiaoxun Sun, Siek Toon Khoo

Dr Siek Toon Khoo

Student engagement is a reflection of active involvement in learning. In digital learning environment, research studies on engagement have been focused on detecting behavioral and psychological engagement indicators from the patterns of activities using feature engineering, but student engagement estimates were rarely compared across sessions or across domains of learning. This paper describes how this could be done by revisiting engagement instrument, diagnosing engagement indicators, estimating engagement parameters, and equating. This study illustrates how engagement reliability can be improved by refining engagement indictors. We demonstrated through DataShop data that student engagement levels can be compared across domains of learning.

Can Engagement Be Compared? Measuring Academic Engagement For Comparison, Ling Tan, Xiaoxun Sun, Siek Toon Khoo

Dr Xiaoxun Sun

Student engagement is a reflection of active involvement in learning. In digital learning environment, research studies on engagement have been focused on detecting behavioral and psychological engagement indicators from the patterns of activities using feature engineering, but student engagement estimates were rarely compared across sessions or across domains of learning. This paper describes how this could be done by revisiting engagement instrument, diagnosing engagement indicators, estimating engagement parameters, and equating. This study illustrates how engagement reliability can be improved by refining engagement indictors. We demonstrated through DataShop data that student engagement levels can be compared across domains of learning.

Can Engagement Be Compared? Measuring Academic Engagement For Comparison, Ling Tan, Xiaoxun Sun, Siek Toon Khoo

Dr Ling Tan

Student engagement is a reflection of active involvement in learning. In digital learning environment, research studies on engagement have been focused on detecting behavioral and psychological engagement indicators from the patterns of activities using feature engineering, but student engagement estimates were rarely compared across sessions or across domains of learning. This paper describes how this could be done by revisiting engagement instrument, diagnosing engagement indicators, estimating engagement parameters, and equating. This study illustrates how engagement reliability can be improved by refining engagement indictors. We demonstrated through DataShop data that student engagement levels can be compared across domains of learning.

An Exercise In Institutional Reflection: The Learning Analytics Readiness Instrument (Lari), Kimberly E. Arnold, Steven Lonn, Matthew Pistilli

Matthew Pistilli

Developing Large-Scale Assessments, Ray Philpot

Ray Philpot

A seminar on large-scale assessment given to a post-graduate class at Boston College, USA. In this seminar I discussed concepts in measurement theory, explained the principles for developing large-scale assessments and gave practical advice on how high quality tests can be constructed.

Children’S Enjoyment Of Play During School Lunchtime Breaks: An Examination Of Intraday And Interday Reliability, Brendon P. Hyndman, Amanda Telford, Caroline Finch, Shahid Ullah, Amanda Benson

Dr Brendon P Hyndman

Background: Enjoyment and play during school lunchtime are correlated with children’s physical activity. Despite this, there is an absence of studies reporting children’s enjoyment of play during school lunchtime breaks. The purpose of this study was to examine the intraday and interday reliability of children’s enjoyment of school lunchtime play. Methods: Surveys used to assess children’s enjoyment of lunchtime play were distributed to and completed by 197 children (112 males, 85 females), aged 8–12 years attending an elementary school in Victoria, Australia. Children completed the surveys during class before lunch (expected enjoyment) and after lunch (actual enjoyment) for 5 days. …

National Assessment Program : Ict Literacy 2008 : Year 6 And Year 10 : Technical Report, Chris Freeman, John Ainley, Julian Fraillon, Eveline Gebhardt, Kate O'Malley, Wolfram Schulz

Julian Fraillon

This report describes the technical aspects of the National ICT Literacy Sample Assessment and summarises the main activities involved in the data collection, the data collection instruments and the analysis and reporting of the data. Chapter 2 summarises the development of the assessment domain and describes the process of item development and construction of the instruments. Chapter 3 reviews the sample design and describes the sampling process. Chapter 3 also describes the weighting procedures that were implemented to derive population estimates. Chapter 4 summarises the field administration and data management procedures, including quality control and the cleaning and coding of …

Measure For Measure : A Review Of Outcomes Of School Education In Australia, John Ainley, Eveline Gebhardt

Dr John Ainley

There have been a plethora of research reports providing information about the achievements of students in Australian schools and how those achievements differ among jurisdictions and among groups of students. However, each report has often been viewed in isolation from other similar studies. This report is intended to provide an integrated appraisal of the results of the international and national achievement surveys conducted since 1994 but with some references to earlier studies. In addition to limiting the task to manageable proportions, this time span corresponds to the widespread introduction of modern measurement methods so that there is a better basis …

National Assessment Program : Ict Literacy 2008 : Year 6 And Year 10 : Technical Report, Chris Freeman, John Ainley, Julian Fraillon, Eveline Gebhardt, Kate O'Malley, Wolfram Schulz

Dr John Ainley

This report describes the technical aspects of the National ICT Literacy Sample Assessment and summarises the main activities involved in the data collection, the data collection instruments and the analysis and reporting of the data. Chapter 2 summarises the development of the assessment domain and describes the process of item development and construction of the instruments. Chapter 3 reviews the sample design and describes the sampling process. Chapter 3 also describes the weighting procedures that were implemented to derive population estimates. Chapter 4 summarises the field administration and data management procedures, including quality control and the cleaning and coding of …

Mapping Student Achievement, Geoff Masters, Ray Adams, Jan Lokan

Prof Ray Adams

The analyses, publications and reports of quantitative data in education and the social sciences usually omit basic information on the construct measured. Probabilistic models for test data make it possible to delineate coherent and richly described measurement continua that facilitate interpretation of student achievement. The potential of conjoint measurement to bring about fundamental advances in educational testing practice lies in part in the opportunities it provides to build useful maps of learning domains and to use those maps in communicating student achievements. This chapter presents two applications of conjoint measurement aimed at constructing and describing achievement variables, developing insights into …

The Development Of The Lunchtime Enjoyment Of Activity And Play Questionnaire, Brendon P. Hyndman, Amanda Telford, Shahid Ullah, Caroline Finch, Amanda Benson

Dr Brendon P Hyndman

BACKGROUND: Enjoyment of physical activity is as an important determinant of children’s participation in physical activity. Despite this, there is an absence of reliable measures for assessing children’s enjoyment of play activities during school lunchtime. The purpose of this study was to develop and assess the reliability of the Lunchtime Enjoyment of Activity and Play (LEAP) Questionnaire. METHODS: Questionnaire items were categorized employing a social-ecological framework including intrapersonal (20 items), interpersonal (2 items), and physical environment/policy (17 items) components to identify the broader influences on children’s enjoyment. An identical questionnaire was administered on 2 occasions, 10 days apart, to 176 …

Extended Mantel-Haenszel Procedure For Dif Detection : A Note On Its Implementation In Acer Conquest, Xiaoxun Sun

Xiaoxun Sun

Many methods have been developed for studying Differential Item Functioning (DIF). An approach developed by Holland and Thayer (1988) adapted the Mantel-Haenszel (MH) statistic (Mantel and Haenszel 1959). The author of this chapter has implemented MH procedure and its extension in an experimental version of ACER ConQuest which can handle DIF analysis for multiple focal groups and polytomous items. This chapter compares the extended MH procedure in DIF detection to the standardized item difficulty difference method based on the Rasch estimates for both dichotomous and polytomous items. The use of MH method in DIF analysis with multiple focal groups is …

Adaptive Testing For Psychological Assessment: How Many Items Are Enough To Run An Adaptive Testing Algorithm?, Michaela Wagner-Menghin, Geoff Masters

Prof Geoff Masters AO

Although the principles of adaptive testing were established in the psychometric literature many years ago (e.g., Weiss, 1977), and practice of adaptive testing is established in educational assessment, it is not yet widespread in psychological assessment. One obstacle to adaptive psychological testing is a lack of clarity about the necessary number of items to run an adaptive algorithm. The study explores the relationship between item bank size, test length and measurement precision. Simulated adaptive test runs (allowing a maximum of 30 items per person) out of an item bank with 10 items per ability level (covering .5 logits, 150 items …

Using The Partial Credit Model To Identify And Adjust For Differences Among Assessors, Geoff Masters, Siek Khoo

Prof Geoff Masters AO

No abstract provided.

National Assessment Program : Ict Literacy Technical Report, Eveline Gebhardt, Julian Fraillon, Wolfram Schulz, Kate O'Malley, Chris Freeman, Martin Murphy, John Lindsey

Martin Murphy

This report is concerned with procedures, processes and technical aspects of the National Assessment Program - ICT Literacy 2011 Literacy (NAP - ICTL 2011) and should be read in conjunction with the Public Report from Literacy NAP - ICTL 2011 which focuses on results and interpretation of results from that assessment. The first cycle of the NAP - ICTL was held in 2005 and provided the baseline against which future performance would be compared. The second cycle of the program was conducted in 2008 and was the first cycle where trends in performance were able to be examined.

Group Of National Experts On The Ahelo Feasibility Study : Engineering Assessment Framework : Ahelo Feasibility Study, Hamish Coates, Sarah Richardson, Jacob Pearce, Julian Fraillon

Dr Jacob Pearce

This AHELO Engineering Assessment Framework describes and illustrates the domain of engineering competency that is used in the AHELO Feasibility Study. The framework gives an organisational structure for the domain in terms of engineering knowledge, processes and contexts; describes the types of assessment items that have been developed; and details how reporting will be carried out. - p.4

What Makes A Difference? Measuring Non-Academic Outcomes, Prue Anderson, Julian Fraillon

Julian Fraillon

There are some very real challenges that educators face in attempting to define, measure and improve the non-academic outcomes of students in Australian schools but there is plenty that can be done to better measure and influence them. There are many challenges in measuring non-academic outcomes but the overarching one is to determine what they are, and whether they are actually 'non-academic' at all. This article addresses each of three major challenges: (1) Defining the outcomes, followed by a recommendation; (2) measuring the outcomes; recommendation; and (3) improving outcomes; recommendation.