Educational Assessment, Evaluation, and Research Commons^™

International Developments (No.9) 2019

International Developments

In this issue of International Developments, we summarise the research evidence gathered through the work of the Australian Council for Educational Research (ACER) and the impact our projects make around the globe.

Wcag 2.1 Meets Stem: Application, Interpretation, And Opportunities For Further Standard Development, Jason White

Journal of Science Education for Students with Disabilities

This paper offers an overview of changes introduced in the recent Web Content Accessibility Guidelines (WCAG) 2.1 standard that have implications for the accessibility of content in science, technology, engineering and mathematics (STEM) disciplines. In reviewing new provisions of WCAG 2.1, interpretive questions are raised regarding their application to mathematical and scientific materials. The paper is concluded with a brief discussion of opportunities for further enhancing accessibility of STEM materials in future revisions of the Guidelines.

How Teachers Engage With Student Assessment Data: Understanding Antecedents To Data-Driven Decision Making, Cynthia P. Raffe, Dennis Alonzo, University Of New South Wales

2009 - 2019 ACER Research Conferences

In recent years, education systems internationally have been encouraging data-use initiatives with the aim of improving student learning through data-driven decision making (DDDM). The use of student assessment data in particular has been promoted to guide teaching practices and progress student learning. Despite this, the adoption of data-use practices by New South Wales classroom teachers has been slow. The study qualitatively examined two core foundational activities that affect New South Wales classroom teachers’ use of student assessment data, referred to as assessment data collection (ADC) practices, and assessment data analysis (ADA) practices. Specifically, the study sought to answer: What are …

Building The Capacity Of Teachers For Supporting 21st-Century Learning, Jenny Gore, Michelle Ware, Sui-Linn White, Lee-Anne Collins, Lloyd Bowen, Carole Hansen

2009 - 2019 ACER Research Conferences

Quality Teaching Rounds (QTR) was first conceptualised by Julie Bowe and Jenny Gore in 2007. It involves teachers working in professional learning communities (PLCs) to reflect on their classroom practice through the lens of the quality teaching model. This teacher-led process builds capacity for quality teaching with novice and experienced teachers alike. Following a set of protocols and adhering to essential features of the approach, one PLC member teaches a lesson, observed by all others. The lesson is coded individually and then collaboratively analysed, using the shared language of the model. This poster presentation graphically highlights evidence from several research …

Variations In Implementation Of Specifications Grading In Stem Courses, Mai Yin Tsoi, Mary E. Anzovino, Amy H. Lin Erickson, Edward R. Forringer, Emily Henary, Angela Lively, Michael S. Morton, Karen Perell-Gerson, Stan Perrine, Omar Villanueva, Marygeorge Whitney, Cynthia M. Woodbridge

Georgia Journal of Science

Specifications grading is an assessment strategy based on mastery learning, clear learning objectives, and frequent evaluations and feedback. Twelve instructors at a southeastern four-year public college implemented the specifications grading method across eight discrete courses in four STEM areas. In this modified assessment strategy, the students controlled their grades through multiple attempts, with limitations, on assessments of course objectives. The instructors designed and executed specifications grading in unique ways that aligned with their content areas, teaching beliefs, and individual teaching styles.

Preliminary observations suggest that, regardless of subject area, specifications grading can be used as an alternative to traditional assessment …

Assessing Computational Thinking, Daniel Duckworth

2009 - 2019 ACER Research Conferences

This paper provides some context for the role of computation thinking (CT) in the Australian Curriculum, an abridged literature review of CT as a problem-solving framework from the International Computer and Information Literacy Study (ICILS) 2018 assessment framework and some examples of how CT has been used to solve real-world problems. Finally, this paper presents ways to teach and assess CT.

What Can Early Childhood Education And Care Settings Teach Us About Skills For The 21st Century?, Dan Cloney, Kellie Picker

2009 - 2019 ACER Research Conferences

Early childhood education and care (ECEC) settings are naturally oriented towards promoting 21st century skills. This can be seen in Australia, where learning is defined as the development of identity, social and emotional skills, problem-solving, and communication skills. A 21st century orientation is also seen in the playbased pedagogies implemented in ECEC settings. A gap, however, exists in the ability of the ECEC sector to communicate its successes. This gap relates to the lack of measurement tools to quantify the quality of the adult–child interactions in ECEC settings, and children’s growth in these 21 century skills and abilities. This paper …

Key Skills For The 21st Century: An Evidence-Based Review, Esther Doecke, Quentin Maire

2009 - 2019 ACER Research Conferences

It is vital that education systems deliver quality outcomes for all young people and prepare them well for their future in the economy and society. To do so, many systems have traditionally had a strong focus on developing academic skills, particularly in literacy and numeracy. In recent years, education systems have developed greater expectations that schools will also equip young people with a broader set of skills for the 21st century (e.g. creativity, critical thinking, problem-solving). This paper addresses these developments and the challenges they present. Building on an evidence-based review, this paper asks what are the key skills required …

Assessment In The Interpersonal Domain: Experiences From Empathy Assessment In Medical Education, Neville Chiavaroli

2009 - 2019 ACER Research Conferences

Frameworks for the teaching and assessment of 21st-century skills commonly recognise the importance of learning and skill development in the interpersonal domain. They also usually acknowledge the challenge of reliably and validly assessing students in this domain. In the field of medical education and in selecting students for medical courses, the concept of empathy has become central to representing the particular interpersonal understandings and skills expected of students and practising doctors. Attempts to assess these attributes during medical training are just as challenging as in school contexts. This presentation draws on several years’ experience of working with medical educators to …

Digital Literacy: Myths And Realities, Julian Fraillon

2009 - 2019 ACER Research Conferences

Digital literacy, under a wide variety of names, is routinely classified as a 21st-century skill and is frequently reported as an area of high priority in school education systems internationally. In comparison with students in other countries, Australian students have high levels of access to digital technologies both at and outside of school. With this access comes the expectations that students will be highly-proficient users of digital technologies and that schools will use digital technologies in transformative ways to support student learning. This session will examine how concepts of digital literacy have developed over time, what data from large-scale assessments …

Assessing And Understanding Social And Emotional Skills: The Oecd Study On Social And Emotional Skills, Sue Thomson

2009 - 2019 ACER Research Conferences

In an increasingly fast-changing and diverse world, the importance of developing social and emotional skills is becoming more evident. The large body of accumulated evidence shows that these skills have strong relationships with life outcomes and they have been referred to as a key component of 21st century skills. The Organisation for Economic Co-operation and Development (OECD) Study on Social and Emotional Skills is a new international assessment of these skills in students at primary and secondary schools. This study also gathers information on students’ families, schools and community learning contexts, aiming to provide information about the conditions or practices …

Teaching And Assessing The General Capabilities In A Secondary School Context, Loren Clarke, Melissa Hughes

2009 - 2019 ACER Research Conferences

Education researchers, policymakers and private enterprise agree that, in addition to content knowledge, students in the 21st century need to acquire particular skills to equip them for active citizenship in the modern world. This is a real challenge for teachers today: how do they teach and assess the skills needed to live and work in the 21st century? This paper will explore the development of Eltham High School’s focus on teaching and assessment of collaboration, problem-solving, and critical thinking over the past seven years. It will explore the development of the assessment program, its connection to the school and state …

Using Learning Analytics To Measure 21st-Century Skills, Dragan Gašević

2009 - 2019 ACER Research Conferences

The unprecedented opportunities to collect data about learning and contexts in which learning occurs has attracted great attention in education. The use of data analytics and machine learning methods have offered much potential to address many relevant questions in education. This talk will focus on the use of learning analytics to measure 21st-century skills in education and outline the types of data commonly used. It will also discuss approaches that are used for analysis and modelling of relevant learning processes and outline the ways in which learning analytics can be used to track learning progression and how the validity of …

The Science Behind The Art Of Teaching: Evaluation As Inspiration, Michele Bruniges Am

2009 - 2019 ACER Research Conferences

Teachers across Australia inspire students to love learning. Our best teachers are constantly evaluating their impact on learning outcomes and adapting their practice – balancing the art and science of teaching. As we move rapidly towards the third decade of the 21st century, there is more pressure than ever for all teachers to deliver both deep discipline knowledge and the skills students need to survive and thrive in the workplace of the future. We need to use technology and data to support teachers to maximise learning outcomes for their students. This has to be done in a way that helps …

Digital Literacy Skill Development: Prescriptive Learning Analytics Assessment Model, Elspeth Mckay

2009 - 2019 ACER Research Conferences

There is a broad awareness of how information communications technology (ICT) digital literacy impacts everyday life. In schools, use of ICT tools has become mandatory. These tools include computers, tablets and mobile phones. These smart devices are used to send emails, browse the internet and make video calls. It is essential for teachers to identify student digital literacy levels through classroom activities and when to implement flexible ePedagogies for students who need help. This presentation will provide easy-to-follow steps to manage learning analytics to determine digital literacy skill levels. Learning analytics can be used for a range of purposes: to …

Educational Reform – Scottish Style!, David Leng

2009 - 2019 ACER Research Conferences

The government in Scotland has made education their defining mission, setting out a clear framework for improvement based on four key aims as articulated in the National Improvement Framework (2015). One improvement driver is Assessing Children’s Progress. This new approach to assessment has been developed to integrate curriculum, assessment (particularly classroom assessment) and pedagogy. Teacher professional judgement has been central to this, and national initiatives have focused on supporting and strengthening it. In this context, the Scottish National Standardised Assessments (SNSA) was launched in 2017, and is a national assessment tool to support improvement in classroom practice while still providing …

Infographic: School Attendance And Mental Health Disorders, Kristen Owen

Teacher infographics

Mental disorders are a significant cause of absence, particularly in high school, a new examination of data from the Australian Child and Adolescent Survey of Mental Health and Wellbeing shows. Today’s infographic looks at some of the results across Years 1-12.

Context And Regulation Of Homeschooling: Issues, Evidence, And Assessment Practices, Janet F. Carlson

Buros Center: Professional Staff Publications

The article discusses salient factors that influence the current context within which homeschooling occurs. Individual states have applied various approaches to establish regulations that both preserve the rights of homeschooling parents and fulfill the state’s obligation to ensure that its residents receive the education to which they are constitutionally entitled. Case and ethnographic studies or research involving small and selected samples often appear in outlets associated with homeschool advocacy groups or in outlets that are not mainstream. The paucity of empirical evidence derived from methodologically strong research paradigms has led to little certainty about many aspects of homeschooling including its …

Mathematical Literacy And Video Games, Craig Carrigee

Dissertations

Mathematical literacy refers to the ability of a person to be able to comprehend and conceptualize real-world data and mathematical situations to make informed decisions. Minimal research exists compared to the explosion of video games available to the public over the past two decades. This study investigates the relationships between playing video games and the ability of a person to answer real-world questions about mathematics as well as the person’s attitude toward mathematics, including their self-confidence, perceived value, enjoyment, and motivation. Previous studies have shown mixed results related to the influence of playing video games on cognitive math ability, but …

Social-Emotional Difference Given Disability, Grade, And School-Wide Positive Behavioral Interventions And Supports, Edith A. Hayles

Dissertations

Students with disabilities as a group remain behind in academic achievement when compared to students without disabilities. Without the right school-based interventions, many students with disabilities will experience academic failure, disciplinary infractions, social isolation, self-doubt, school disengagement, and school dropout. Additionally, social-emotional intervention helps older students to improve executive functioning, develop self-regulation skills, and score better on achievement tests than the students not receiving any social-emotional programming. Moreover, students enrolled in schools that implement evidence-based educational interventions to facilitate students’ social-emotional competencies demonstrate more positive behaviors and social-emotional interactions. Students with social-emotional competencies have also described feeling safer and happier …

Traffic Noise And Sexual Selection: Studies Of Anthropogenic Impact On Bird Songs And Undergraduate Student Reasoning Of Evolutionary Mechanisms, Sarah Spier

School of Natural Resources: Dissertations, Theses, and Student Research

Humans have transformed much of the natural landscape and are continuing to do so at an accelerated rate, compromising natural areas that serve as important habitat for many species. Roads impact much of the environment as they fragment habitat and introduce traffic noise into the acoustic environment, deferentially affecting wildlife in roadside habitat. I explored how traffic noise affects the detection of birds based on whether their vocalizations were masked by traffic noise. Masked species detection was not affected by an increase in traffic noise amplitude, while there was a negative effect of traffic noise amplitude on unmasked species detection, …

Traffic Noise And Sexual Selection: Studies Of Anthropogenic Impact On Bird Songs And Undergraduate Student Reasoning Of Evolutionary Mechanisms, Sarah Spier

School of Natural Resources: Dissertations, Theses, and Student Research

Humans have transformed much of the natural landscape and are continuing to do so at an accelerated rate, compromising natural areas that serve as important habitat for many species. Roads impact much of the environment as they fragment habitat and introduce traffic noise into the acoustic environment, deferentially affecting wildlife in roadside habitat. I explored how traffic noise affects the detection of birds based on whether their vocalizations were masked by traffic noise. Masked species detection was not affected by an increase in traffic noise amplitude, while there was a negative effect of traffic noise amplitude on unmasked species detection, …

Improving Young Children's Learning In Economically Developing Countries: What Works, Why, And Where? Scoping Review, Jen Jackson, Syeda Kashfee Ahmed, Toby Carslake, Petra Lietz

Monitoring Learning

This scoping review responds to the increasing interest in improving early childhood education and care (ECEC) in economically developing countries. It examines available research in relation to the question: What effective interventions have been implemented recently in economically developing countries to improve children’s learning in the years before school? As much of the research underpinning ECEC interventions has focused on economically developed countries, it is timely to review available research about the effectiveness of interventions in the economically developing world. The ECEC interventions were categorised as Income supplementation (n=8); Parent-focused interventions (n=37); Child-focused education and nurturing care (n=35); Integrated interventions …

Values In Evaluations Of Maternal And Child Health Programs, Erica Lynn Dodds

Dissertations

Values are a central component in evaluation and are prominently discussed in evaluation theory, but relatively little is known about how they are used in practice. The theory-practice gap in evaluation may cause unsound evaluation practice or irrelevant or impractical evaluation theory. However, little is known about this gap as it relates to values and valuing. Failing to take certain values into account in an evaluation risks reinforcing the status quo to the detriment of vulnerable populations, so identifying how the theory-practice gap plays out in the use of values is important.

This study focuses on the maternal and child …

An Exploratory Investigation Into The Role Of Evaluation Theory In Evaluative Thinking, Lyssa Wilson Becho

Dissertations

The field of evaluation generally agrees that evaluation theory is important. Evaluation theorists and scholars have written prolifically about the role of evaluation theory in practice. However, the empirical literature associated with how and why evaluation theory is important is still emerging. In particular, how evaluation theory fosters high quality evaluation practice through transparent logical reasoning, attention to values and valuing, and contextual responsiveness, and how these inform evaluative thinking are understudied phenomena in the field.

This study intends to provide insights into the relationship between evaluation theory and evaluation practice to answer two major research questions. The first research …

Ethnicity And Punishment: A State-Level Investigation On Hispanic Representation In School Discipline, Candace Nicole Fowles

Theses and Dissertations

Racial disproportionality in exclusionary discipline measures among public school students has been well-established for African American youth in the United States. The research literature has included limited and inconsistent research findings providing information on the representation patterns among Hispanic students in school discipline. Previous studies on Hispanic representation in school discipline have established a need for data to be analyzed at the state level. Using a large dataset acquired from the Department of Education's Office of Civil Rights - Civil Rights Data Collection (CRDC) for the 2013-2014 school year, this paper examines if Hispanic students are disproportionally represented in exclusionary …

Faculty Interactions With Black Male Students At Hbcus And Community Colleges As Predictors Of Academic Achievement In Stem, Antoine Gates

Dissertations

A myriad of studies in STEM education aim to identify the underlying reasons behind poor achievement of Black males in higher education. Research studies have indicated that HBCUs make significant advances in various desired outcomes such as graduation rates for Blacks, especially males. However, Black males at community colleges do not always fare similarly to their counterparts at HBCUs. There are limited studies on faculty interactions with Black males in STEM at community colleges. Therefore, the goal of this study was to identify whether significant differences existed in the interactions STEM faculty formed with Black males at community colleges and …

The Influence Of Tape Diagrams And Bar Models On Middle School Students' Proportional Reasoning, Katie Paulding

Boise State University Theses and Dissertations

Proportional reasoning is an integral component of adolescent mathematical cognitive development and a foundational concept for students to understand in order to be successful in higher level mathematics and science courses. Yet research indicates students struggle to proportionally reason. Task features of proportional reasoning problems are known to influence student cognition and success in problem solving, including familiarity with problem context, problem type, numerical content, and mode of task representation. The purpose of this study was to examine the influence of two iconic representations (tape diagrams and bar models) and three ratio relationships (6:3, 8:2, and 5:2) on student cognition …

Using Playable Case Studies To Influence Teen Girls' Self-Efficacy And Interest In Cybersecurity, Desiree Marie Winters

Theses and Dissertations

Various factors dissuade women from the field of cybersecurity. Educational interventions are needed to mitigate the negative effects of stereotypes and low perceived self-efficacy and help girls gain interest in learning about cybersecurity. This thesis focuses on an intervention targeted to increase teenage girls' interest and self-efficacy in cybersecurity: the Cybermatics Playable Case Study. Findings from a mixed-methods study in which a focus group was conducted with 7 middle school girls, interviews were conducted with 2 high school girls, and pre- and post- simulation survey was collected from all 9 participants reveal tensions between enjoyment and frustration in the girls' …

Theory And Practice In Doctoral Dissertation Research, 2007-2017: A Content Analysis By Degree Type, William Friel

Seton Hall University Dissertations and Theses (ETDs)

In response to the overwhelming presence of professional practice doctoral degrees in the early 2000s, universities offering Doctor of Education (EdD) and/or Doctor of Philosophy (PhD) degrees in higher education studies began to question the purpose and value of each degree. These universities asked: What is the value of academic research if it does not aim to improve the condition of what is being studied? This led to a theory-practice debate among departments offering the EdD and the PhD from the perspective of the goal of the dissertation research: Truth or improvement. To address this tension, the Carnegie Project on …