Educational Assessment, Evaluation, and Research Commons^™

An Exploration Of Extension Professionals' Evaluation Experiences In Indigenous Contexts, Sara M. Tomis, J.C. Bunch, Amy Harder, T. Grady Roberts

Journal of Extension

Extension professionals evaluating programs that engage Indigenous youth face unique opportunities and challenges. The purpose of this study was to explore these professionals' evaluation experiences, with a focus on culturally responsive practice. We conducted a focus group comprised of professionals associated with the Federally Recognized Tribes Extension Program (FRTEP). Study findings revealed a need for enhanced support for FRTEP professionals conducting evaluations of programs for Indigenous youth audiences, as well as for more culturally responsive practice within the Cooperative Extension System.

Evaluating The Impact Of The Kentucky Read To Achieve (Rta) Initiative On School Reading Performance, Nicholas Gage, Erin Hogan, Todd Whitney, Terrance M. Scott, Amy S. Lingo

Kentucky Teacher Education Journal: The Journal of the Teacher Education Division of the Kentucky Council for Exceptional Children

Kentucky reading achievement has remained below statewide goals, which prompted a revision to the Read to Achieve (RTA) initiative, a state-level grant-funded program aimed at improving student literacy outcomes. The revision prioritized the professional learning of administrators and K-3 reading instructional staff in the science of reading and the adoption of high-quality structured literacy-based resources and aligned intervention resources. This study uses nine years of panel data (2014–2015 through 2023–2024) from all public schools in Kentucky to evaluate whether the revised RTA initiative had a differential effect on school-level reading outcomes. We employ propensity score matching and a heterogeneous difference-in-differences …

Measuring Civic And Citizenship Education Through Large-Scale Assessments: Insights From The International Civic And Citizenship Education Study (Iccs), Wolfram Schulz, Valeria Damiani

Civics and Citizenship Assessment

This chapter from the open access book ‘Promoting K-12 Civic Learning and Engagement Through Assessment’ edited  by Hamilton, Rikoon and  Kidd, discusses ways of measuring aspects of civic and citizenship education (CCE) cross-nationally with a specific focus on the International Civic and Citizenship Education Study (ICCS) commissioned by the IEA (International Association for the Evaluation of Educational Achievement). The chapter starts with a brief overview of the main characteristics of the study, the legacy with previous IEA surveys of civic education, and the assessment design including a description of the cognitive and affective-behavioral dimensions that are measured in the …

Leadership Development Programs For Elected Officials: Measuring The Impact Of Extension On County Leadership Capacity, Nathan Palardy, Matthew Sowcik

Journal of Human Sciences and Extension

Although Extension-led leadership development programs for elected officials are widespread, few evaluations assess their effectiveness. This study uses 2 years of survey data to quantitatively and qualitatively evaluate several programs for Florida county commissioners, focusing on reaction, knowledge transfer, and behavior change. Results indicate high satisfaction with the program content and quality and substantial improvement in leadership competencies. Commissioners reported frequent application of the skills taught and cited networking opportunities as a key benefit. Findings suggest that Extension-led programs can strengthen local governments by enhancing the leadership capabilities of elected officials and offering insights to guide the design of similar …

Farm Tour Evaluation Tool For Agricultural Awareness, Hannah Eason, Jessica Ryals, Yvette Goodiel, Lauren Butler, John Diaz, Marilyn Swisher

Journal of Extension

Educational farm tours offer learning opportunities that cannot be duplicated in a classroom setting. Developing a consistent farm tour evaluation tool is paramount to quantifying individual Extension farm tours while also providing a common platform to measure outcomes and impacts nationally and over time. An IRB-approved farm tour evaluation tool was developed to capture agricultural awareness and used to evaluate the 2022 National Association of County Agricultural Agents Annual Meeting/Professional Improvement Conference (NACAA AM/PIC) tours. The evaluation tool attempts to quantify farm tour outcomes and impacts and demonstrate the value of agriculture to decision makers, consumers, and community members.

Establishing Cultivating Dialogic Feedback (Cdf): Enhancing Teacher Evaluation Through The Humanization Of Teachers, Adrian Michael Blumenthal

Electronic Theses and Dissertations

Current practices in K-12 teacher evaluation look to mechanize the evaluation model, and effectuate better teaching by enforcing compliance to the evaluation model. This practice treats teachers as disseminators of instructional practice and thereby of information to students which dehumanizes teachers. The following research takes a different approach to understanding teacher evaluation by working away from fixing the evaluation model by replacing it with another model. Instead, the research examines the breadth of compliance culture, called performativity, and examines the relationship between teacher and evaluator. At the root of the research is discovering how the relationship between teacher and evaluator …

Developing A Program Theory To Guide Program Evaluation, Amanda D. Ali

Journal of Extension

A program theory specifies how a program is intended to work. It is different from a logic model and bears importance in program evaluation by considering a program’s processes. This article shares an application for developing a program theory, a worksheet that assists with this process, and a graphical illustration that visually communicates program flow.

Evaluating Learning Outcomes In A Serious Game: A Practical And Model Checking Approach, Matthew D. Douglas

Theses and Dissertations

This research introduces a novel computational framework to evaluate and predict the educational impact of serious games during development. By using finite state machines (FSM) and model-checking techniques, this study evaluates two serious games. Traditional evaluation approaches, often reliant on resource-intensive human trials, lack scalability and fail to provide early insight into the alignment of game mechanics with learning objectives. This study addresses these challenges of traditional evaluation methods.

Remodeling Pedagogical Evaluation Tools To Incorporate Student Self-Efficacy And Sense Of Belonging In Scientific Research, Richelle L. Tanner, Nicholas P. Burnett, Emily E. King, Anne E. Todgham

Biology, Chemistry, and Environmental Sciences Faculty Articles and Research

Curated undergraduate research experiences have been widely used at colleges and universities for decades to build student interest, technical preparation, and confidence in the pursuit of scientific careers. Educators often employ standardized survey instruments to evaluate learning outcomes for research experiences, but many of these assessments consider only technical skill development and career interests and are not rooted in discrete pedagogical theories. As higher education aims to create inclusive and equitable learning experiences for students, we argue that pedagogical assessment tools for undergraduate research experiences need to expand to consider outcomes such as increased science literacy, confidence in relational “soft” …

Evaluation Of Ste(A)M Education: An Analysis Of Research And Practices From 2014 To 2023, Billy T.M. Wong, Kam Cheong Li, Mengjin Liu

Journal of Educational Technology Development and Exchange (JETDE)

STEM (Science, Technology, Engineering, and Mathematics) and STEAM (by including Arts) education (hereafter STE(A)M education) has been widely studied and practiced. Evaluation plays an important role in assessing and improving the effectiveness of relevant initiatives. Although there has been extensive research on evaluating STE(A)M education, little attention has been paid to examining the patterns of said evaluations. To address the research gap, this paper analyses relevant work on the evaluation of STE(A)M education practices with respect to their countries/regions, levels of education, discipline domains, pedagogical approaches, objectives of evaluation, evaluation areas, as well as data collection and analysis methods. It …

Game-Based Learning With Augmented Reality, Kantida Nanon

Dissertations

Augmented Reality (AR) technology provides an effective and controllable learning environment, especially for complicated, potentially dangerous, or critical content. Effective implementation of interaction and platforms for educational tools can maximize learner engagement and optimize the learning experience. This dissertation explores the potential of AR technology to enhance learning experiences across multiple platforms and subject areas, particularly within chemistry and mathematics. The dissertation is grounded in the field of Human-Computer Interaction (HCI) and explores the potential of AR to increase learner engagement, improve learning outcomes, and transform traditional pedagogical practices.

The dissertation is divided into two parts. The first part examines …

The Alchemy Of Assessment And Evaluation, Jessica Kahlow

Mavs Open Press Open Educational Resources - Archive

This comprehensive guide illuminates the art and science of educational assessment, blending ancient alchemical wisdom with modern AI innovations. Part one lays the foundation, exploring essential principles of assessment and evaluation and its pivotal role in shaping learning outcomes. Part two delves into diverse assessment types, from formative assessments driving real-time adjustments to summative evaluations measuring overall achievement. Part three offers expert guidance on crafting effective assessment designs, ensuring authenticity and inclusivity. Part four examines the alchemy of feedback and grading, empowering educators with strategies for constructive engagement and growth. Part five focuses on evaluations, showcasing methods to assess program …

Assessing Community Needs And Concerns In Iron County, Meagan Johnston, Lendel Narine

Outcomes and Impact Quarterly

Utah State University Extension conducted a needs assessment to identify priority needs in Iron County. The needs assessment employed a three-phase approach involving interviews, surveys, and data analysis. Results revealed top priority issues, including youth mental health, preparing youth for successful careers, and employment opportunities. The findings will guide Extension programming in addressing community needs effectively.

A Process Evaluation Of Utah’S Food Security Council, Palak Gupta, Heidi Leblanc

Outcomes and Impact Quarterly

Utah’s Food Security Council (FSC) brings together diverse stakeholders to address food security across the state. As FSC concluded its inaugural year of work, a process evaluation assessed its progress. Findings will inform FSC’s future strategic improvements, partnerships, and community outreach activities.

Evaluation Practices In National Science Foundation Research Experiences For Undergraduates Programs, Aly Busse

USF Tampa Graduate Theses and Dissertations

The purpose of this study was to assess and describe the status of program evaluation within the National Science Foundation’s (NSF) Research Experiences for Undergraduates (REU) sites. The National Science Foundation's REU program supports thousands of undergraduates annually in hands-on, immersive research experiences under the mentorship of science professionals. To date, there is a lack of published or accessible evaluation results from these REU programs and it is unknown what evaluation approaches are being used in the REU programs, how they are designed and implemented, how evaluation findings are utilized, and what barriers to evaluation activities exist. A self-administered electronic …

Estimating And Applying Parameters Necessary To Plan Cluster Randomized Trials (Crts) And Multisite Cluster Randomized Trials (Mscrts), Dea Mulolli

Dissertations

Cluster randomized trials (CRTs) are commonly used to study the effectiveness of educational interventions. During the design phase of a study, it is critical for researchers to ensure their studies are adequately powered to detect meaningful treatment effects, including both main and moderator effects. Designing CRTs with adequate power to detect main and moderator effects requires accurate estimates of design parameters. This research aims to advance the literature on design parameters for power analyses, specifically focusing on empirical estimates of intraclass correlations (ICCs). The work consists of three research papers that examine the role of including the teacher level in …

Using An Interrupted Case Study To Engage Undergraduates’ Critical Thinking Style And Enhance Content Knowledge, Kelsey Hall, Katherine Starzec

Journal on Empowering Teaching Excellence

Access the online Pressbooks version of this article here.

The interrupted case study is a structured way to engage students in active learning. Interruptions, or pauses for reflection and discussion scheduled within the case-study presentation, provide students with a chance to collaborate and engage in critical thinking. Critical thinking style, which is a measure of how one tends to think critically, provides insight into how one tackles problem solving. This article describes a pilot project that combined critical-thinking style and an interrupted case study, delivered over a two-class-period time frame, to four college courses. The project’s goals were to assess …

Phenomenological Research Examining Administrators And Special Education Teachers' Perception, Experience & Knowledge Of The Mississippi Special Education Growth Rubric, Tricia Pittman

Dissertations

In the current era of accountability, public school administrators’ responsibilities may include evaluating special education teachers and providing them with feedback and support to improve their effectiveness. The problem that exists is that administrators who do not have a background in special education are charged with the responsibility of evaluating special education teachers, resulting in special education teachers not getting the support or feedback necessary to improve their instructional practice.

In addition to administrators not having the background knowledge necessary to effectively evaluate and support special education teachers, the problem of developing an effective evaluation model to evaluate special education …

Application Of A Risk Assessment Framework To Evaluate Organizational Capacity In Extension, Lendel K. Narine, Andree' Walker Bravo

Journal of Human Sciences and Extension

This brief report outlines a risk assessment conducted at Utah State University (USU) Extension in 2023. The report provides a discussion on the need, value, and use of risk assessments. It also presents an adaptable methodological framework for conducting risk assessments in Extension and an overview of the risk assessment results. Results show the highest risk to USU Extension is related to recruitment and retention of employees, while other noteworthy risks include internal conflicts and low employee morale. The results of the risk assessment will be used to inform strategic planning in Extension. We recommend periodic risk assessments to ensure …

Profess 12: Project Report 5: The Profess 12 Summer School - Evaluation, Una Beagon, Ruchita Jani, Janet Mckennedy, Brian Bowe, Margaret Morgan, Rosalind M Henry

Reports

This report outlines the evaluation of the summer school which was a key element of the PROFESS 12 project. The key steps involved in the 5-day summer school along with the method by which the feedback and the reflections of the participants of the summer school were collected is described here. Thereafter, the reflections received in the form of audio and text files are analysed and themes identified from the experience of the summer school participants are explained in detail.

The survey responses from the feedback on the summer school participation are also shown graphically. This analysis helped us identify …

Conceptualizations Of Data Visualization Use Beyond Efficiency In Evaluation, Sarah Douville

CGU Theses & Dissertations

Data visualization (data viz) is a valuable tool within evaluation for its ability to aid cognitive efficiency over text-based presentation of data (Card et. al., 1999; Evergreen, 2017, 2018; Few, 2012; Nussbaumer Knaflic, 2015; Tufte 2001). This exploratory multi-phase mixed methods research study considers purposes for using data viz in evaluation that can be achieved with that increased efficiency through the research question: “What conceptualizations of data viz use do program evaluators have beyond increased efficiency?”

In Phase I, secondary analysis of existing interview data with experts in both data viz and evaluation was used to better understand conceptualizations of …

Student Assessment In Madhya Pradesh, Acer India

ACER India

ACER will build a student assessment system in Madhya Pradesh as part of Rajya Shiksha Kendra (RSK)’s School Certification Programme. Under this initiative, schools will be certified as gold, silver, and bronze based on their performance. We will develop assessments for students in grades 3, 5, 8, and 10 in schools that meet the qualifying criteria.

Defining Success For An Introductory Mathematics Non-Stem Pathway Course, Clark D. Metz

Graduate Theses, Dissertations, and Problem Reports (ETD)

The enduring debate surrounding college algebra, traditionally a gateway for STEM pathways, underscores the necessity for curricular evolution in response to the rising popularity of non-STEM majors. This dissertation explores the success of an applied college algebra course designed for non-STEM students. The study aimed to determine whether this alternative course could meet diverse student needs while promoting academic achievement and fostering positive attitudes towards mathematics. Findings revealed through a novel approach of a difference-in-differences analysis that students enrolled in the applied college algebra course exhibited higher pass rates due to the introduction of the new course. The lower failure …

Evaluation Of English Language Improvement Program For Information System Graduates Using A Comparative Analysis Method, Ratih Purwasih, Rahimullaily Rahimullaily, Zikri Zikri

REID (Research and Evaluation in Education)

STMIK Indonesia Padang seeks to improve the quality of its graduates by providing several academic training in information systems science and soft skills. One of them is training to improve graduates' English skills, including English I and II, English for career, and TOEFL training. The purpose of the study was to evaluate the success of the English language improvement program given to graduates of STMIK Indonesia Padang by comparing the scores for all English programs. This research method is explanatory. Data processing and analysis used descriptive statistics and comparative mean analysis techniques with Friedman test statistics on 4 data groups …

An Evaluation Of Clemson University’S Cooperative Extension Services Online Learning Programs And Framework For Program Development And Evaluation, Christina Leard

All Dissertations

The COVID-19 pandemic brought online learning to the forefront with the need to define online learning with best practices. Clemson University Extension conducted a retrospective observational study compiling survey data from virtual extension agents and supervisor’s conducting online programs. A literature review analyzed terminology for online learning to obtain a standardized definition to define online learning for Clemson Extension Services programs; evaluate and identify through literature strengths, weaknesses, opportunities, and challenges (SWOC) of on-line learning; determine and identify online learning competencies for instructors, assessment and evaluation; and analyze and evaluate response data from virtual extension agents delivering online programs in …

Evaluation And Improvement Of Processes And Metrics In A Dual Credit Program, Kristian E. Fischer

Education Theses and Dissertations

This dissertation used convergent mixed methods design in a program implementation evaluation of a dual credit program. A binary logistic regression was performed on dual credit enrollment persistence. A stakeholder survey was deployed to parents, administrators, instructors and counselors associated with the dual credit program. A full review of district documents related to dual credit was included as part of the qualitative component. The results of the regression indicated non-significance in the predictor variables for the dependent variable of dual credit persistence. The emergent themes from the survey included concerns about student readiness for dual credit coursework and a need …

Research Conference 2023: Becoming Lifelong Learners. Proceedings And Program, Kylie Burns (Ed.)

2021-2030 ACER Research Conferences

We know learning in the early years sets the foundations for people’s ongoing academic, cognitive and socio-emotional development. Education across those formative first 12 years is currently segmented, with children often starting fresh in new systems. How can each of us make learning in the early childhood and primary years more effective? Research Conference 2023 will examine research around how we can improve the continuity of learning from birth to 12 years. It will bring together leading international and Australian researchers to provide insights into the best ways to identify, conceptualise, develop and assess these new linkages for learning.

A Qualitative Inquiry Of Research Administrator Perceptions Of Job Satisfaction In A Central Research Administration Unit At The University Of Denver, Noelle L. Strom

Electronic Theses and Dissertations

The Sponsored Programs Administration (SPA) team provides critical research administration services for the research community at the University of Denver. As research funding continues to rise at the University of Denver, retention of research administrators is critically important. Job satisfaction is key to employee engagement and retention. Higher job satisfaction contributes to several positive outcomes for institutions, including lower turnover, higher productivity, lower costs, and employee loyalty. The need to retain experienced research administration professionals is highlighted by the fact that few formal educational routes exist that prepare employees for a career in research administration; rather, developing expertise takes a …

The Remote Online Initiative: A Legislative Strategy For Rural Workforce Development In Utah, Amanda D. Ali, Paul A. Hill

Outcomes and Impact Quarterly

The Remote Online Initiative (ROI) was a legislative response to growing unemployment and limited job opportunities in rural Utah. The ROI program seeks to reduce unemployment in rural Utah communities via remote job placements. Since 2018, the ROI program has expanded to include three flagship certificate courses – Certified Remote Work Professional (CRWP), Certified Remote Work Leader (CRWL), and Certified Tech Sales Professional (CTSP). To date, evaluation results show that 411 Utah graduates found remote jobs after completing the CRWP course, 55% of participants said their department created a remote work environment and hired, on average, 19 remote employees upon …

Evaluating The “What’S Up Down South” Economic Summit: A Platform For Business Leaders To Share Ideas, Paul A. Hill, Amanda D. Ali

Outcomes and Impact Quarterly

The What’s Up Down South economic summit is southern Utah’s largest business gathering. The event provides the business community with economic insights to assist them in making informed decisions. USU Extension evaluated the 27^th annual What's Up Down South economic summit. Results showed the event was an effective platform for disseminating information and advancing knowledge on economic and business topics. Participants felt the summit provided a valuable experience to the business community. These findings support the continuation of the summit's information dissemination activities in the future.